LLM Budget Enforcement & AI Cost Control

Why LLM Costs Spiral Quickly

LLM APIs bill per token — roughly ¾ of a word. Costs are quoted per million tokens, which sounds cheap until you multiply by the number of developers, requests per day, and average conversation length:

The Need for Budget Enforcement

Without a cost-governance layer, most organizations discover the same painful pattern:

The solution is a budget enforcement engine that sits between your application and the LLM provider — checking every request against project-level quotas before it's forwarded. This is exactly what OpenSourceAIHub provides, with three layers of protection that work together.

Budget Enforcement Architecture

Every request flows through a multi-stage pipeline before reaching the LLM provider. The budget engine adds enforcement checks at the gateway level — no provider-side configuration needed:

Example Budget Policy

Each project can have its own budget configuration that combines all three layers. Here's a complete example for a customer support bot:

{
  "project_id": "proj_customer_support",
  "budget": {
    "monthly_limit_credits": 100000000,
    "per_request_max_tokens": 4096,
    "alert_thresholds": [0.50, 0.80, 1.00],
    "on_exhausted": "block",

    "budget_mode": {
      "enabled": true,
      "trigger_threshold": 0.80,
      "allowed_models": [
        "oah/llama-3-70b",
        "oah/gpt-4.1-mini",
        "oah/gemini-2.5-flash"
      ]
    },

    "loop_protection": {
      "max_requests_per_window": 50,
      "window_seconds": 60,
      "max_tokens_per_minute": 500000
    },

    "notifications": {
      "webhook_url": "https://hooks.slack.com/services/...",
      "channels": ["webhook", "dashboard"]
    }
  }
}

Real-Time Cost Tracking

Budget enforcement is only half the picture. You also need visibility into where the money is going. The Hub dashboard provides real-time cost analytics per project:

Cost Visibility Response Headers

Every successful response includes headers for programmatic cost tracking:

Preventing Runaway Prompts & Agent Loops

The most expensive AI incidents aren't caused by high usage — they're caused by accidents. Here are the three most common patterns the Hub detects and stops automatically:

The Future: Graceful Downgrade

Graceful Downgrade is part of Phase 1.1 of our roadmap. Join the early access program above to help shape the implementation.

Per-Million-Token Pricing (Managed Mode)

Understanding per-model costs is critical for setting realistic budgets. Here are the current rates in Managed Mode:

Using OpenSourceAIHub for Cost Control

The Hub offers two billing modes. Both include the full AI Firewall (DLP) security layer at no additional cost.

Project-Level Cost Isolation

OpenSourceAIHub supports project-scoped API keys (oah_*) that separate billing and policies by team, application, or environment:

import OpenAI from "openai";

// Project key (oah_*) — scoped to "Customer Support Bot"
const client = new OpenAI({
  apiKey: "oah_proj_support_xxxxx",
  baseURL: "https://api.opensourceaihub.ai/v1",
});

// This request inherits:
// - The project's monthly budget ($100 / 100M credits)
// - The project's DLP policy (BLOCK SSN, REDACT EMAIL)
// - The project's loop protection settings
// - The project's BYOK keys (if configured)
const response = await client.chat.completions.create({
  model: "oah/gpt-4.1-mini",
  messages: [{ role: "user", content: "Summarize this patient record..." }],
  max_tokens: 512,
});

Handling Budget Errors in Application Code

When the pre-flight check determines the budget can't cover the request, the Hub returns a structured JSON error. Your application should handle this gracefully:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "os_hub_your_key_here",
  baseURL: "https://api.opensourceaihub.ai/v1",
});

try {
  const response = await client.chat.completions.create({
    model: "oah/gpt-4.1",
    messages: [{ role: "user", content: prompt }],
  });
  console.log(response.choices[0].message.content);
} catch (err: any) {
  if (err.status === 402) {
    const detail = err.error;
    console.error(
      `Budget exceeded: need $${detail.required_balance}, ` +
      `have $${detail.current_balance}`
    );
    // Option A: Retry with a cheaper model (oah/llama-3-70b)
    // Option B: Prompt user to top up wallet
    // Option C: Queue for later processing
  } else if (err.status === 429) {
    console.error("Rate limited — possible agent loop detected");
    // Back off and investigate
  } else {
    throw err;
  }
}

from openai import OpenAI, APIStatusError

client = OpenAI(
    api_key="os_hub_your_key_here",
    base_url="https://api.opensourceaihub.ai/v1",
)

try:
    response = client.chat.completions.create(
        model="oah/gpt-4.1",
        messages=[{"role": "user", "content": prompt}],
    )
    print(response.choices[0].message.content)
except APIStatusError as e:
    if e.status_code == 402:
        print(f"Budget exceeded: {e.body}")
        # Fallback to cheaper model or notify user
    elif e.status_code == 429:
        print("Rate limited — possible agent loop detected")
    else:
        raise

Cost Optimization Strategies

Start Enforcing LLM Budgets

Sign up, top up your wallet with $5, and every request is automatically budget-checked from your very first API call. No configuration required. Existing wallet enforcement is live today — advanced features (threshold alerts, loop protection, Budget Mode) are coming in Phase 1.1.