graceful-shutdown – skill Khác cho agent AI

Graceful Shutdown

Overview

A skill for implementing graceful shutdown in servers, workers, and long-running processes. Ensures in-flight requests complete, background jobs finish or checkpoint, database connections close cleanly, and the process exits with a proper status code. Essential for zero-downtime deployments in container orchestrators (Kubernetes, ECS, Docker Compose) and bare-metal process managers (systemd, PM2).

When to Use This Skill

• Use when building an HTTP server that must not drop active connections during deploys
• Use when writing a background worker that processes jobs from a queue
• Use when deploying to Kubernetes, Docker, or any environment that sends SIGTERM before killing
• Use when the user says "graceful shutdown", "drain connections", "handle SIGTERM", "zero downtime", or "don't kill active requests"
• Use when implementing health check endpoints (/healthz, /readyz) for orchestrators

How It Works

Step 1: Register signal handlers early

Trap SIGTERM (orchestrator shutdown) and SIGINT (Ctrl+C) at process startup. Set a flag so the application knows it is shutting down.

let isShuttingDown = false;

function onShutdownSignal(signal: string): void {
  if (isShuttingDown) return; // prevent double-shutdown
  isShuttingDown = true;
  console.log(`Received ${signal}, starting graceful shutdown...`);
  void shutdown().catch(() => { process.exitCode = 1; });
}

process.on("SIGTERM", () => onShutdownSignal("SIGTERM"));
process.on("SIGINT", () => onShutdownSignal("SIGINT"));

Step 2: Stop accepting new work

Immediately stop the server from accepting new connections. For HTTP servers, call server.close(). For queue workers, stop polling for new jobs.

async function shutdown(): Promise<void> {
  // 1. Stop accepting new connections
  server.close(() => {
    console.log("Server closed — no new connections accepted");
  });

  // 2. Mark health check as not-ready so load balancers stop routing
  //    (readiness probe returns 503 from this point)
}

Step 3: Drain in-flight work with a deadline

Wait for active requests and background tasks to finish, but enforce a hard deadline so the process never hangs indefinitely.

const DRAIN_TIMEOUT_MS = 25_000; // must be less than orchestrator's terminationGracePeriodSeconds

async function drainAndExit(): Promise<void> {
  const deadline = setTimeout(() => {
    console.error("Drain timeout reached — forcing exit");
    process.exit(1);
  }, DRAIN_TIMEOUT_MS);
  deadline.unref(); // don't keep the event loop alive just for the timer

  try {
    // Wait for active connections to finish
    await waitForActiveConnections();

    // Flush buffered data (logs, metrics, queues)
    await flushBuffers();

    // Close external resource handles
    await closeResources();

    console.log("Graceful shutdown complete");
    process.exit(0);
  } catch (err) {
    console.error("Error during shutdown:", err);
    process.exit(1);
  }
}

Step 4: Implement readiness and liveness probes

Orchestrators use these to decide whether to route traffic and whether to restart the container. Liveness proves the process is alive; readiness controls whether traffic is routed. While the listener is still available during a drain, keep liveness healthy and return 503 only from readiness. After the listener closes, new probes cannot connect, so do not promise that HTTP liveness remains reachable for the entire termination window.

import { createServer, IncomingMessage, ServerResponse } from "node:http";

function handleHealthCheck(req: IncomingMessage, res: ServerResponse): void {
  if (req.url === "/healthz") {
    // Keep liveness distinct from readiness while the listener is available.
    // Drain-rejection middleware must not turn this endpoint into a 503.
    res.writeHead(200).end("ok");
    return;
  }

  if (req.url === "/readyz") {
    // Readiness: 503 during shutdown so the load balancer stops routing.
    if (isShuttingDown) {
      res.writeHead(503).end("shutting down");
    } else {
      res.writeHead(200).end("ready");
    }
    return;
  }
}

Step 5: Track active connections

Maintain a count of in-flight requests so you know when draining is complete. Use a once guard covering both finish and close events so that client disconnects (aborted requests) correctly decrement the counter.

let activeConnections = 0;
let drainResolve: (() => void) | null = null;

function trackRequest(res: ServerResponse): void {
  activeConnections++;
  let counted = true;
  function release(): void {
    if (!counted) return;
    counted = false;
    activeConnections--;
    if (isShuttingDown && activeConnections === 0 && drainResolve) {
      drainResolve();
    }
  }
  res.on("finish", release);
  res.on("close", release);
}

function waitForActiveConnections(): Promise<void> {
  if (activeConnections === 0) return Promise.resolve();
  return new Promise((resolve) => {
    drainResolve = resolve;
  });
}

Examples

Example 1: Express.js server with graceful shutdown

import express from "express";
import { createServer } from "node:http";

const app = express();
const server = createServer(app);
let isShuttingDown = false;
let activeRequests = 0;
let drainResolve: (() => void) | null = null;

// Health endpoints — registered BEFORE the drain-rejection middleware so it
// cannot turn liveness into a 503 while the listener is still available.
app.get("/healthz", (_, res) => res.send("ok"));
app.get("/readyz", (_, res) => {
  res.status(isShuttingDown ? 503 : 200).send(isShuttingDown ? "draining" : "ready");
});

// Track in-flight requests and reject new application work during drain.
app.use((req, res, next) => {
  if (isShuttingDown) {
    res.setHeader("Connection", "close");
    res.status(503).json({ error: "Server is shutting down" });
    return;
  }

  activeRequests++;
  let counted = true;
  function release(): void {
    if (!counted) return;
    counted = false;
    activeRequests--;
    if (isShuttingDown && activeRequests === 0 && drainResolve) {
      drainResolve();
    }
  }
  // Listen for both finish (normal) and close (client abort) so the
  // counter always decrements. The once guard prevents double-decrement
  // when both events fire.
  res.on("finish", release);
  res.on("close", release);
  next();
});

// Application routes
app.get("/api/data", async (req, res) => {
  const data = await fetchData();
  res.json(data);
});

// Graceful shutdown
function shutdown(signal: string): void {
  if (isShuttingDown) return;
  isShuttingDown = true;
  console.log(`${signal} received — draining ${activeRequests} active requests`);

  server.close();

  const forceExit = setTimeout(() => {
    console.error("Forced exit — drain timeout exceeded");
    process.exit(1);
  }, 25_000);
  forceExit.unref();

  if (activeRequests === 0) {
    console.log("No active requests — exiting cleanly");
    process.exit(0);
  }

  drainResolve = () => {
    console.log("All requests drained — exiting cleanly");
    process.exit(0);
  };
}

process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("SIGINT", () => shutdown("SIGINT"));

server.listen(3000, () => console.log("Server ready on :3000"));

Example 2: Python FastAPI under Uvicorn

Uvicorn owns SIGTERM handling and request draining. It stops accepting new connections, asks existing connections to shut down, waits for connections and tasks up to --timeout-graceful-shutdown, and only then sends the ASGI lifespan shutdown event. Do not replace its signal handler or wait for requests again inside lifespan; use that hook to release application resources after Uvicorn's drain.

from contextlib import asynccontextmanager
from fastapi import FastAPI


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.db_pool = await open_database_pool()
    try:
        yield
    finally:
        # Uvicorn has already completed or timed out its request drain.
        await app.state.db_pool.close()


app = FastAPI(lifespan=lifespan)


@app.get("/healthz")
async def healthz():
    return {"status": "ok"}


@app.get("/readyz")
async def readyz():
    return {"status": "ready"}

Run Uvicorn with a deadline shorter than the orchestrator's kill timeout:

uvicorn app:app --timeout-graceful-shutdown 25

If readiness must turn 503 before SIGTERM, coordinate a separately secured and tested pre-stop drain signal plus a propagation delay. FastAPI's lifespan shutdown hook is too late for that transition because Uvicorn invokes it after request draining.

Example 3: Process-level worker with a bounded drain

JobQueue is an application-owned adapter. Its poll must honor the timeout, and each job must be idempotent because a crash after work but before ack causes redelivery. This is a process, not a Node worker thread (process signals are not delivered to

… (nội dung đầy đủ có trong gói tải về)