Start at the top. Logging and budgets take an afternoon and remove most of the risk. The rest are optimizations you make once the logs tell you where the money goes.

How do you log token usage per user?

Every major provider returns token counts on the response. Store them in one table, with the price you used at the time, so later price changes do not rewrite history.

create table llm_usage (
  id bigint generated always as identity primary key,
  user_id uuid not null references auth.users (id),
  feature text not null,          -- e.g. 'summarize', 'chat', 'extract'
  model text not null,
  input_tokens int not null,
  output_tokens int not null,
  cached_input_tokens int not null default 0,
  cost_usd numeric(12, 6) not null,
  created_at timestamptz not null default now()
);

create index llm_usage_user_day on llm_usage (user_id, created_at);

Keep prices in code or config, not hard coded inside queries, and update them when your provider changes its pricing page.

// lib/llm/pricing.ts
// Fill these from your provider's current pricing page (USD per 1M tokens).
export const PRICES: Record<string, { input: number; cachedInput: number; output: number }> = {
  "small-model": { input: 0, cachedInput: 0, output: 0 },
  "large-model": { input: 0, cachedInput: 0, output: 0 },
};

export function costUsd(model: string, input: number, cached: number, output: number) {
  const p = PRICES[model];
  if (!p) throw new Error(`No price for ${model}`);
  const uncached = Math.max(input - cached, 0);
  return (uncached * p.input + cached * p.cachedInput + output * p.output) / 1_000_000;
}

How do you enforce a per-user token budget?

Check the budget on the server, before the model call, inside the same route that makes the call. Never trust a limit enforced only in the browser.

// lib/llm/budget.ts
import { createClient } from "@supabase/supabase-js";

const db = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_SERVICE_ROLE_KEY!);

const DAILY_TOKEN_LIMIT: Record<string, number> = {
  free: 50_000,
  pro: 1_000_000,
};

export async function tokensUsedToday(userId: string) {
  const since = new Date();
  since.setUTCHours(0, 0, 0, 0);
  const { data, error } = await db
    .from("llm_usage")
    .select("input_tokens, output_tokens")
    .eq("user_id", userId)
    .gte("created_at", since.toISOString());
  if (error) throw error;
  return (data ?? []).reduce((sum, r) => sum + r.input_tokens + r.output_tokens, 0);
}

export async function checkBudget(userId: string, plan: string) {
  const used = await tokensUsedToday(userId);
  const limit = DAILY_TOKEN_LIMIT[plan] ?? DAILY_TOKEN_LIMIT.free;
  return { allowed: used < limit, used, limit };
}

The limits above are placeholders. Set yours from your own logs: look at what a normal user spends in a day, then set the free limit a little above that and the paid limit well above it. When a user hits the limit, say so plainly in the UI and offer the upgrade, instead of returning a vague error.

For high traffic, replace the sum query with a running counter (a daily row per user, or a key in Redis) so the check stays fast.

How do you route requests to a cheaper model?

Most features have easy and hard requests. Classifying a support ticket, extracting a date, or rewriting a short sentence rarely needs the largest model. Long reasoning over many documents might.

// app/api/ai/route.ts
import { checkBudget } from "@/lib/llm/budget";
import { callModel } from "@/lib/llm/call";
import { getSessionUser } from "@/lib/auth";

function pickModel(task: string, inputChars: number) {
  if (task === "classify" || task === "extract") return "small-model";
  if (inputChars < 2_000) return "small-model";
  return "large-model";
}

export async function POST(req: Request) {
  const user = await getSessionUser(req);
  if (!user) return new Response("unauthorized", { status: 401 });

  const { task, input } = await req.json();
  const budget = await checkBudget(user.id, user.plan);
  if (!budget.allowed) {
    return Response.json({ error: "daily_limit", used: budget.used, limit: budget.limit }, { status: 429 });
  }

  const model = pickModel(task, input.length);
  const result = await callModel({ userId: user.id, feature: task, model, input, maxOutputTokens: 800 });
  return Response.json(result);
}

Routing is where quality can slip without anyone noticing. Before you move a task to a smaller model, run your eval set on both models and compare. If you do not have evals yet, set them up first; our guide on adding evals to an LLM feature before launch walks through a lightweight setup.

How do caching and prompt caching cut costs?

There are two different kinds of caching, and they solve different problems.

Response caching is yours. If the same input gets the same answer, such as summarizing the same public article or classifying the same product title, hash the normalized input with the model name and prompt version, and store the result.

// lib/llm/call.ts (excerpt)
import { createHash } from "node:crypto";

const PROMPT_VERSION = "summarize-v3";

function cacheKey(model: string, input: string) {
  const normalized = input.trim().replace(/\s+/g, " ");
  return createHash("sha256").update(`${PROMPT_VERSION}:${model}:${normalized}`).digest("hex");
}

Put the prompt version in the key so that changing the prompt invalidates old answers. Do not cache answers that depend on private user data across users.

Prompt caching is the provider's. If every call starts with the same long prefix, such as a big system prompt, tool definitions, or a reference document, providers like Anthropic and OpenAI can reuse that prefix and bill those cached input tokens at a lower rate. Anthropic asks you to mark the cacheable block explicitly. OpenAI applies it automatically to long prompts that share a prefix. In both cases the rule is the same: put stable content first and anything that changes per request, like the user's message or today's date, last. One changed character early in the prompt breaks the cache.

What should you track every week?

If cost per active user is more than a small slice of what that user pays you, fix the biggest feature in the cost-by-feature list before you touch anything else.

What mistakes make LLM bills spike?

1. Unbounded chat history. Sending the full conversation on every turn grows cost with each message. Keep the last few turns plus a short running summary.
2. Retry loops without a cap. A schema check that fails and retries forever can call the model dozens of times. Cap retries at one or two, then fail clearly.
3. No output cap. Without a max output setting, a model can write far more than the UI shows.
4. Calling the model for things code can do. Date parsing, simple validation, and keyword matching do not need an LLM.
5. Client-side API keys. A key in the browser lets anyone spend your budget. Every model call must go through your server.
6. Background jobs with no owner. Cron jobs and agent loops need their own budget and logging, tagged with a system user, or they become invisible spend.

How does Luminix Studio approach this?

Luminix Studio builds web and mobile apps, and the AI features and automations around them, for startups worldwide. When we add an LLM feature to a product, usage logging and a server-side budget go in with the first version, not after the first surprising invoice, and model routing comes only after there are evals to prove the cheaper path still works. If you want an AI feature that stays inside its budget, or want an existing one audited, see luminixstudio.com.