Monitoring and runtime triage

Codex Pooler exposes Prometheus metrics at /metrics. Metrics auth is managed from /admin/system; when bearer protection is configured, Prometheus must use the matching Kubernetes Secret reference rather than placing the credential in Helm values. /metrics is not protected by the runtime firewall.

Metrics exposure states

The endpoint has exactly three states:

• Open: no metrics bearer is configured, so scrapes are allowed without it
• Bearer-protected: a metrics bearer is configured, so Prometheus must provide it
• Unavailable: metrics settings cannot be read, so /metrics fails closed

Restore the previous metrics bearer state in /admin/system to roll back a metrics exposure change.

Use monitoring for runtime evidence only. Dashboards, alerts, logs, tickets, and copied queries must not include prompts, response bodies, uploaded files, websocket frames, cookies, bearer tokens, upstream secrets, or raw Pool API keys.

Collector path

The Helm chart can render a Prometheus Operator ServiceMonitor for the app service:

monitoring:
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prometheus-stack
    interval: 10s
    scrapeTimeout: 5s

The release label and scrape interval should match your own Prometheus Operator selectors and scrape budget. For fast OOM investigations, prefer a short interval for the app ServiceMonitor and keep broader Kubernetes collectors at their normal cadence. Worker and scheduler roles don’t start the Prometheus reporter because they don’t expose /metrics; use Kubernetes cgroup metrics and sampler logs for those pods.

Runtime triage dashboard

Build your Grafana or Prometheus dashboard around the signals needed to correlate runtime pressure across memory, request handling, gateway admission, database access, and restarts:

1. Kubernetes cgroup working set versus pod memory limit, RSS/cache split, and cgroup memory not explained by BEAM total
2. BEAM memory total, processes, binary, ETS, code, atom, atom used, and system memory by pod
3. BEAM process count, port count, total run queue, CPU run queue, and IO run queue
4. app restarts, OOM events, and current pod last terminated reason
5. request rate, HTTP status class rate, and p95 endpoint/router latency
6. gateway admission accepted, queued, rejected, timed-out, queue-time p95, and per-pod running or queued saturation by route class
7. bounded stream outcomes, websocket bridge fallbacks, and stream-buffer oversized or truncated events
8. Ecto query rate, queries per request, hot query sources by safe source and SQL command, unknown query rate, and DB queue p95
9. routing circuit transition rate by transition, route class, and bounded reason class
10. saved-reset post-consume convergence rate by bounded source and outcome, plus p95 timing from application to canonical evidence, canonical evidence to lifecycle finish, and application to lifecycle finish

Read the dashboard as a correlation view. If cgroup memory climbs while BEAM total stays flat, look outside normal BEAM heap attribution. If vm_memory_binary_bytes climbs with cgroup memory, inspect streaming response retention, file bodies, and upstream transport buffering. If process count, ports, or run queue climb, inspect stuck request processes, websocket ownership, and overloaded route classes.

Admission saturation is local to each app pod. codex_pooler_gateway_admission_running and codex_pooler_gateway_admission_queued report the current state by route class on the reporting pod, not a cluster-wide queue. Every route class reports an explicit value, including zero. Normal exposition is 18 series per app pod, with a bounded ceiling of 20 including the invalid-label fallback. Select or aggregate pods deliberately when comparing capacity across replicas.

The new gateway metrics are bounded at the source. Stream outcomes are succeeded, failed, settlement_failed, or interrupted; invalid values become unknown. Their downstream and upstream transport labels are http_sse, websocket, or unknown, giving at most 45 series per app pod across five effective outcomes and two three-value transport vocabularies. settlement_failed is attempt-scoped, so concurrent failed terminal settlement attempts can each emit it. Websocket bridge fallbacks have at most 20 reason series per app pod, while precommit buffer overflow is a separate untagged counter. Together with the admission gauges and circuit additions, this work adds approximately 266 bounded time series per app pod.

Oversized stream-buffer histogram observations saturate at 128 MiB for Prometheus export, so p95 remains finite. The runtime’s raw telemetry event and stream-buffer behavior are unchanged.

Saved-reset post-consume convergence

codex_pooler_saved_reset_convergence_count is a best-effort view of committed post-consume saved-reset transitions observed by scraped web nodes. It is not a cluster-wide inventory and does not prove that every persisted transition has been exported: worker and scheduler roles do not run the Prometheus reporter. A convergence performed on another role can still persist its lifecycle metadata for later cross-role reconciliation without appearing in this metric.

The metric and its three timing histograms use only the bounded source and outcome labels. source is one of reconciliation, runtime_headers, runtime_websocket_upgrade_headers, runtime_websocket_frame_headers, runtime_event, runtime_error, finalizer, or unknown; outcome is confirmed_by_quota, reblocked, expired, or unknown. No Pool, upstream, account, identity, request, payload, or credit identifier is exported. Missing or malformed source and outcome values become unknown; absent, malformed, or time-inconsistent timestamps omit the corresponding timing observation rather than inventing a duration.

The bounded histograms are codex_pooler_saved_reset_convergence_applied_to_canonical_seconds, codex_pooler_saved_reset_convergence_canonical_to_lifecycle_seconds, and codex_pooler_saved_reset_convergence_applied_to_lifecycle_seconds. They describe the observed event timeline on a reporting web node, not delay for the persisted cross-role lifecycle metadata to become visible. Use the rate panel to see which bounded transition category occurred, then read the p95 panels only for categories with observations.

The downloadable starter dashboard includes these queries now. Apply the application release, then add the matching panels to your live dashboard and refresh the published screenshot. Don’t treat the screenshot above or any existing deployed dashboard as evidence that these new panels are already live.

Download the starter Grafana dashboard JSON

Useful PromQL

max by (pod) (
  container_memory_working_set_bytes{namespace="codex-pooler", container="app", image!=""}
)

vm_memory_total_bytes{namespace="codex-pooler", job="codex-pooler-app"}
vm_memory_binary_bytes{namespace="codex-pooler", job="codex-pooler-app"}
vm_memory_processes_bytes{namespace="codex-pooler", job="codex-pooler-app"}
vm_memory_ets_bytes{namespace="codex-pooler", job="codex-pooler-app"}
vm_memory_system_bytes{namespace="codex-pooler", job="codex-pooler-app"}

clamp_min(
  max by (pod) (
    container_memory_working_set_bytes{
      namespace="codex-pooler",
      container="app",
      image!=""
    }
  )
  - on (pod)
  max by (pod) (
    vm_memory_total_bytes{namespace="codex-pooler", job="codex-pooler-app"}
  ),
  0
)

increase(kube_pod_container_status_restarts_total{
  namespace="codex-pooler",
  exported_container="app"
}[15m])

rate(codex_pooler_gateway_stream_buffer_oversized_count[5m])
rate(codex_pooler_gateway_stream_buffer_truncated_count[5m])

sum by (pod) (
  rate(codex_pooler_repo_query_count{namespace="codex-pooler", job="codex-pooler-app"}[5m])
)

topk(10,
  sum by (source, command) (
    rate(codex_pooler_repo_query_count{namespace="codex-pooler", job="codex-pooler-app"}[5m])
  )
)

sum by (pod, command) (
  rate(codex_pooler_repo_query_count{
    namespace="codex-pooler",
    job="codex-pooler-app",
    source="unknown"
  }[5m])
)

histogram_quantile(0.95,
  sum by (le, source, command) (
    rate(codex_pooler_repo_query_total_time_seconds_bucket{
      namespace="codex-pooler",
      job="codex-pooler-app"
    }[5m])
  )
)

codex_pooler_gateway_admission_running{
  namespace="codex-pooler",
  job="codex-pooler-app"
}
codex_pooler_gateway_admission_queued{
  namespace="codex-pooler",
  job="codex-pooler-app"
}

sum by (outcome, downstream_transport, upstream_transport) (
  rate(codex_pooler_gateway_stream_outcome_count{
    namespace="codex-pooler",
    job="codex-pooler-app"
  }[5m])
)

sum by (reason) (
  rate(codex_pooler_gateway_websocket_bridge_fallback_count{
    namespace="codex-pooler",
    job="codex-pooler-app"
  }[5m])
)
sum(rate(codex_pooler_gateway_websocket_bridge_precommit_overflow_count{
  namespace="codex-pooler",
  job="codex-pooler-app"
}[5m]))

sum by (method, status_class) (
  rate(codex_pooler_http_request_count{
    namespace="codex-pooler",
    job="codex-pooler-app"
  }[5m])
)

sum by (route_class, transport) (
  rate(codex_pooler_gateway_admission_enqueued_count{
    namespace="codex-pooler",
    job="codex-pooler-app"
  }[5m])
)

histogram_quantile(0.95,
  sum by (le, route_class, transport) (
    rate(codex_pooler_gateway_admission_dequeued_time_seconds_bucket{
      namespace="codex-pooler",
      job="codex-pooler-app"
    }[5m])
  )
)

histogram_quantile(0.95,
  sum by (le, pod) (
    rate(codex_pooler_repo_query_queue_time_seconds_bucket{
      namespace="codex-pooler",
      job="codex-pooler-app"
    }[5m])
  )
)

sum by (transition, route_class, reason_class) (
  rate(codex_pooler_gateway_routing_circuit_transition_count{
    namespace="codex-pooler",
    job="codex-pooler-app"
  }[5m])
)

sum by (source, outcome) (
  rate(codex_pooler_saved_reset_convergence_count{
    namespace="codex-pooler",
    job="codex-pooler-app"
  }[5m])
)

histogram_quantile(0.95,
  sum by (le, source, outcome) (
    rate(codex_pooler_saved_reset_convergence_applied_to_canonical_seconds_bucket{
      namespace="codex-pooler",
      job="codex-pooler-app"
    }[5m])
  )
)
histogram_quantile(0.95,
  sum by (le, source, outcome) (
    rate(codex_pooler_saved_reset_convergence_canonical_to_lifecycle_seconds_bucket{
      namespace="codex-pooler",
      job="codex-pooler-app"
    }[5m])
  )
)
histogram_quantile(0.95,
  sum by (le, source, outcome) (
    rate(codex_pooler_saved_reset_convergence_applied_to_lifecycle_seconds_bucket{
      namespace="codex-pooler",
      job="codex-pooler-app"
    }[5m])
  )
)

Memory sampler logs

The in-process memory sampler is enabled by default in every release role. When BEAM total memory or cgroup usage crosses the configured threshold, it logs a sanitized snapshot with role metadata, memory categories, cgroup memory stats, process and port counts, top processes by memory, top processes by message queue length, and top ETS tables by memory. It never logs ETS table contents, messages, request bodies, prompts, bearer tokens, websocket frames, or upstream payloads.

Emergency tuning environment variables:

CODEX_POOLER_MEMORY_SAMPLER_ENABLED=true
CODEX_POOLER_MEMORY_SAMPLER_THRESHOLD_RATIO=0.70
CODEX_POOLER_MEMORY_SAMPLER_MIN_INTERVAL_MS=60000
CODEX_POOLER_MEMORY_SAMPLER_TOP_PROCESSES=20
CODEX_POOLER_MEMORY_SAMPLER_TOP_ETS_TABLES=20
CODEX_POOLER_MEMORY_SAMPLER_LIMIT_BYTES=1073741824

Use a lower threshold or shorter interval only during active investigation, because logs are the only signal likely to capture a worker or scheduler spike that reaches OOM before the next Prometheus scrape. Worker and scheduler pods do not expose the app /metrics endpoint and do not start the Prometheus reporter, so combine their sampler logs with Kubernetes cgroup memory, restart, OOM, and memory-limit metrics.