multi-tenant-llm-hosting – skill Thiết kế & UI cho agent AI

Multi-Tenant LLM Hosting

Host many teams/customers on shared inference infrastructure without sacrificing security, performance, or cost governance.

Prerequisites

• Kubernetes cluster with GPU node pools
• API gateway or LLM gateway (LiteLLM, Bifrost, Envoy, Kong)
• Prometheus + Grafana for per-tenant observability
• Redis or equivalent for rate limiting state
• Billing system or cost attribution database

Isolation Model

• Strong tenant identity on every request
• Per-tenant API keys and scoped model access
• Namespace/workload isolation plus enforced network and storage boundaries; shared

engine/GPU memory may require separate processes or nodes for high-risk tenants

• Strict data retention and log partitioning controls

vLLM Multi-Model Serving

# vllm-deployment.yaml - Multi-model serving with vLLM
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-premium
  namespace: llm-serving
  labels:
    app: vllm
    model-tier: premium
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vllm
      model-tier: premium
  template:
    metadata:
      labels:
        app: vllm
        model-tier: premium
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
    spec:
      containers:
        - name: vllm
          image: YOUR_APPROVED_VLLM_IMAGE@sha256:REPLACE_WITH_VERIFIED_DIGEST
          args:
            - "--model=/models/llama-3.1-70b"
            - "--tensor-parallel-size=2"
            - "--max-model-len=8192"
            - "--gpu-memory-utilization=0.90"
            - "--max-num-seqs=128"
          ports:
            - containerPort: 8000
              name: inference
          resources:
            requests:
              nvidia.com/gpu: 2
              cpu: "8"
              memory: "64Gi"
            limits:
              nvidia.com/gpu: 2
              cpu: "16"
              memory: "128Gi"
          volumeMounts:
            - name: model-weights
              mountPath: /models
              readOnly: true
      volumes:
        - name: model-weights
          persistentVolumeClaim:
            claimName: premium-model-weights
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        gpu-type: a100
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-economy
  namespace: llm-serving
  labels:
    app: vllm
    model-tier: economy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
      model-tier: economy
  template:
    metadata:
      labels:
        app: vllm
        model-tier: economy
    spec:
      containers:
        - name: vllm
          image: YOUR_APPROVED_VLLM_IMAGE@sha256:REPLACE_WITH_VERIFIED_DIGEST
          args:
            - "--model=/models/llama-3.1-8b"
            - "--max-model-len=4096"
            - "--gpu-memory-utilization=0.85"
            - "--max-num-seqs=256"
          ports:
            - containerPort: 8000
              name: inference
          resources:
            requests:
              nvidia.com/gpu: 1
              cpu: "4"
              memory: "32Gi"
            limits:
              nvidia.com/gpu: 1
              cpu: "8"
              memory: "64Gi"
          volumeMounts:
            - name: model-weights
              mountPath: /models
              readOnly: true
      volumes:
        - name: model-weights
          persistentVolumeClaim:
            claimName: economy-model-weights
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

Per-Tenant Quota Configuration

# tenant-quotas-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tenant-quotas
  namespace: llm-serving
data:
  quotas.yaml: |
    tenants:
      acme-corp:
        tier: enterprise
        models_allowed:
          - llama-3.1-70b
          - llama-3.1-8b
          - nomic-embed-text
        rate_limits:
          requests_per_minute: 300
          tokens_per_minute: 500000
          concurrent_requests: 50
        budget:
          daily_limit_usd: 500.00
          monthly_limit_usd: 10000.00
          alert_threshold_percent: 80
        priority: high

      startup-xyz:
        tier: standard
        models_allowed:
          - llama-3.1-8b
          - nomic-embed-text
        rate_limits:
          requests_per_minute: 60
          tokens_per_minute: 100000
          concurrent_requests: 10
        budget:
          daily_limit_usd: 50.00
          monthly_limit_usd: 1000.00
          alert_threshold_percent: 80
        priority: medium

      internal-dev:
        tier: free
        models_allowed:
          - llama-3.1-8b
        rate_limits:
          requests_per_minute: 20
          tokens_per_minute: 50000
          concurrent_requests: 5
        budget:
          daily_limit_usd: 10.00
          monthly_limit_usd: 200.00
          alert_threshold_percent: 90
        priority: low

Namespace Isolation for High-Risk Tenants

# tenant-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme-corp
  labels:
    tenant: acme-corp
    isolation: strict
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-isolation
  namespace: tenant-acme-corp
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: llm-gateway
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: llm-serving
      ports:
        - port: 8000
          protocol: TCP
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: tenant-acme-corp
spec:
  hard:
    requests.cpu: "16"
    requests.memory: "64Gi"
    limits.cpu: "32"
    limits.memory: "128Gi"
    requests.nvidia.com/gpu: "4"
    pods: "20"

Request Routing and Rate Limiting

The previous sample trusted x_tenant_id while never checking x_api_key,

used racy check-then-increment counters, and treated missing usage as free. Do not

expose an inference service with that pattern. Implement this server-side contract:

1. Authenticate a hashed, revocable virtual key or verified gateway identity; derive

the tenant from that record. Ignore caller-supplied tenant headers. Apply this to

every route, including embeddings, models, metrics and streaming endpoints.

2. Validate a bounded request schema, model allowlist, context/output-token limits

and allowed parameters. Remove caller override URLs/credentials. The endpoint

table is operator-owned. Reject streaming until accounting/cancellation support

is implemented; never parse an SSE stream as one JSON object.

3. Atomically reserve RPM, TPM, a concurrency lease and worst-case request spend

against both daily and monthly budgets in a shared transaction/Lua operation.

Fail closed when the quota store is unavailable. Use expiring uniquely owned

leases and an idempotent finalizer so crashes cannot leak or double-release slots.

4. Forward only to the fixed private endpoint with TLS where required, explicit

deadlines, bounded response size and status checks. Preserve upstream failures;

never return a provider error body as a successful completion.

5. Reconcile reservation with authoritative usage and an append-only request-ID

ledger. Missing usage is unresolved accounting, not zero cost. Finalize or retain

a conservative hold after disconnect/timeouts and reconcile asynchronously.

Worked reservation example (illustrative rates, not provider prices): a tenant

has $1.00 remaining today, $20 monthly and one free concurrency slot. A request

reserves $0.06 and 2,000 total tokens atomically; a simultaneous request must see

$0.94 remaining and the occupied slot. Final verified cost $0.04 refunds $0.02

once by request ID. Repeating finalization must not refund again.

The ConfigMap above is policy data only: it enforces no quotas by itself. The

Deployment fragments are one model per server, not a multi-model process. Supply

a tested image/model/GPU-compatible release, namespace, Service, probes, secrets,

TLS and NetworkPolicies before deployment; placeholders intentionally cannot run.

Prefix caching is disabled until the chosen engine's tenant cache isolation has

been tested. Never infer cache confidentiality from Kubernetes namespaces alone.

Contents

• [Rate Limiting with Envoy](references/details.md)
• [Billing Integration](references/details.md)
• [Noisy-Neighbor Controls](references/details.md)
• [Per-Tenant Monitoring](references/details.md)
• [Security Baseline](reference

… (nội dung đầy đủ có trong gói tải về)