bifrost-gateway – Agents & Automation skill for AI agents

Bifrost AI Gateway

Overview

[Bifrost](https://github.com/maximhq/bifrost) is an open-source (Apache-2.0), self-hosted AI gateway written in Go. It puts 20+ providers (OpenAI, Anthropic, AWS Bedrock, Google Vertex and Gemini, Azure OpenAI, Mistral, Groq, Ollama, vLLM and others) behind one OpenAI-compatible API, and adds fallbacks, weighted load balancing, virtual keys with budgets and rate limits, semantic caching, request logging, Prometheus and OpenTelemetry export, and an MCP gateway. Applications keep their existing SDKs and change only the base URL. Documentation: https://docs.getbifrost.ai

When to Use This Skill

• One application or agent needs several LLM providers behind a single endpoint
• Provider failover, retries or key load balancing should live in config, not application code
• Teams, customers or apps need their own keys with spend budgets and rate limits
• Repeated or similar prompts should be served from a cache
• LLM traffic needs request logs, cost tracking, Prometheus metrics or OpenTelemetry traces
• Several MCP servers should be exposed to agents through one governed endpoint
• Claude Code, Codex CLI or another coding agent should run through a self-hosted gateway

Not a fit when the user cannot run a service at all (Bifrost is self-hosted), or for model training and fine-tuning.

How It Works

1. Start the gateway

# Node.js: download and run the binary
npx -y @maximhq/bifrost

# Docker: persist config and logs in ./data
docker run -p 8080:8080 -v $(pwd)/data:/app/data maximhq/bifrost

• The gateway and web UI listen on port 8080 (-port with npx; -e APP_PORT=... plus -p with Docker). Open http://localhost:8080.
• Pin a version for anything beyond local evaluation: maximhq/bifrost:<version> or npx -y @maximhq/bifrost --transport-version <version>.
• A fresh instance has no dashboard password. Before exposing it beyond localhost, start it with BIFROST_SETUP_TOKEN set and enable password protection under Workspace → Config → Security.

2. Add providers

Use the web UI (Model Providers page), the management API, or config.json in the app directory (the mounted ./data folder for Docker; ~/.config/bifrost for npx unless -app-dir is set):

{
  "$schema": "https://www.getbifrost.ai/schema",
  "providers": {
    "openai": {
      "keys": [{ "name": "openai-1", "value": "env.OPENAI_API_KEY", "models": ["*"], "weight": 1.0 }]
    },
    "anthropic": {
      "keys": [{ "name": "anthropic-1", "value": "env.ANTHROPIC_API_KEY", "models": ["*"], "weight": 1.0 }]
    }
  }
}

curl -X POST http://localhost:8080/api/providers \
  -H "Content-Type: application/json" \
  -d '{"provider": "openai", "keys": [{"name": "openai-1", "value": "env.OPENAI_API_KEY", "models": ["*"], "weight": 1.0}]}'

• env.VAR_NAME reads the secret from the gateway's environment. Never inline real keys.
• "models": ["*"] allows every model on that key; an empty list denies all models.

3. Send requests

Address models as provider/model on the OpenAI-compatible endpoint:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Hello"}]}'

Existing SDKs only change their base URL:

| Client | Base URL |

|---|---|

| Any OpenAI-compatible client | http://localhost:8080/v1 |

| OpenAI SDK | http://localhost:8080/openai |

| Anthropic SDK | http://localhost:8080/anthropic |

| Google GenAI SDK | http://localhost:8080/genai |

| LangChain | http://localhost:8080/langchain |

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/openai",
    api_key=os.environ["BIFROST_VIRTUAL_KEY"],
)
reply = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-6",  # any configured provider through the same client
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
)

4. Add fallbacks, retries and load balancing

• Fallbacks: list backup models in the request body. Each is tried in order after the primary fails, with its own retry budget.

  {"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Hi"}], "fallbacks": ["anthropic/claude-sonnet-4-6", "gemini/gemini-2.5-flash"]}
  

• Retries: set per provider in network_config ("max_retries": 3, "retry_backoff_initial": 500, "retry_backoff_max": 5000). The default is 0 retries. Keys rejected for credentials, quota or model access always move on to the next key; rotating away from a rate-limited (429) key needs max_retries above 0 and more than one key.
• Key load balancing: give a provider several keys with different weight values; Bifrost picks keys by weighted random selection.
• Provider load balancing: weight providers inside a virtual key's provider_configs (next step).

5. Govern access with virtual keys

Virtual keys (sk-bf-...) carry provider and model allowlists, budgets and rate limits:

curl -X POST http://localhost:8080/api/governance/virtual-keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-bot",
    "provider_configs": [
      {"provider": "openai", "weight": 0.7, "allowed_models": ["gpt-4o-mini"], "key_ids": ["*"]},
      {"provider": "anthropic", "weight": 0.3, "allowed_models": ["claude-sonnet-4-6"], "key_ids": ["*"]}
    ],
    "budgets": [{"max_limit": 100.00, "reset_duration": "1M"}],
    "rate_limit": {"request_max_limit": 100, "request_reset_duration": "1m", "token_max_limit": 10000, "token_reset_duration": "1h"},
    "is_active": true
  }'

• Clients send the key as x-bf-vk, Authorization: Bearer, x-api-key (Anthropic style) or x-goog-api-key (Gemini style). In examples, use visible placeholders such as <BIFROST_VIRTUAL_KEY> rather than realistic secret-shaped values.
• Virtual keys are optional until you enforce them: set "client": {"enforce_auth_on_inference": true} in config.json, or turn on Enforce Virtual Keys on Inference in Settings → Security.
• A virtual key with no provider_configs denies every provider. Budgets can also be set on teams and customers above the key.

6. Enable semantic caching

Caching needs a vector store (Redis/Valkey, Weaviate, Qdrant or Pinecone) and the semantic_cache plugin:

{
  "vector_store": {"enabled": true, "type": "redis", "config": {"addr": "localhost:6379"}},
  "plugins": [{
    "name": "semantic_cache",
    "enabled": true,
    "config": {"provider": "openai", "embedding_model": "text-embedding-3-small", "dimension": 1536, "threshold": 0.85, "ttl": 300}
  }]
}

• A request is cached only when it carries an x-bf-cache-key header (or the plugin sets default_cache_key). This is the most common reason nothing gets cached.
• For exact-match caching with no embedding calls, use hash mode: "config": {"dimension": 1, "ttl": 1800}.

7. Observe traffic

• Every request is logged with provider, model, tokens, cost and latency in the web UI at http://localhost:8080/logs.
• Prometheus metrics are served at /metrics.
• The otel plugin exports GenAI-convention spans to any OTLP collector (Grafana, Datadog, Honeycomb, Langfuse and others):

  {"plugins": [{"name": "otel", "enabled": true, "config": {"collector_url": "http://otel-collector:4318/v1/traces", "trace_type": "genai_extension", "protocol": "http"}}]}
  

8. Use the MCP gateway

Bifrost connects to MCP servers (STDIO, HTTP or SSE) and re-exposes their tools at /mcp:

{
  "mcp": {
    "client_configs": [
      {"name": "docs", "connection_type": "http", "connection_string": "https://mcp.example.com/mcp", "auth_type": "none", "tools_to_execute": ["*"]}
    ]
  }
}

# Connect Claude Code to every tool Bifrost exposes
claude mcp add --transport http bifrost http://localhost:8080/mcp

• When the gateway enforces virtual keys, pass one as a header (claude mcp add-json bifrost '{"type":"http","url":"http://localhost:8080/mcp","headers":{"Authorization":"Bearer <BIFROST_VIRTUAL_KEY>"}}'). The key's MCP tool permissions decide which tools the client sees.
• On /v1/chat/completions, tool calls are returned to the caller and run only through an explicit POST /v1/mcp/tool/execute, unless Agent Mode auto-execution is configured.

9. Route coding agents through Bifrost

Claude Code (Anthropic-compatible route), in ~/.claude/settings.json or the shell:

{"env": {"ANTHROPIC_BASE_URL": "http://localhost:8080/anthropic", "ANTHROPIC_AUTH_TOKEN": "<BIFROST_VIRTUAL_KEY>"}}

Codex CLI (Responses API provider), in ~/.codex/config.toml, with the virtual key exported as OPENAI_API_KEY:

model = "openai/gpt-5"
model_provider = "bifrost"

[model_providers.bifrost]
name = "Bifrost"
base_url = "http://localhost:8080/openai/v1"
wire_api = "responses"
env_key = "OPENAI_API_KEY"
requires_open

… (full content is in the download package)