Observe and probe the service

Alcarta exposes application traffic and operational telemetry on separate listeners.

Endpoint	Meaning
Application /healthz	Process liveness; intentionally does not check dependencies
Application /readyz	Dependency, migration, and draining readiness
Metrics /metrics	Prometheus metrics
Metrics /healthz/detail	Detailed operational health

A draining or dependency-impaired instance can still return 200 from /healthz; remove it from traffic when /readyz returns 503.

The metrics listener defaults to 127.0.0.1:9090. Keep it private. If you configure its optional operations token, send that token from the scraper rather than exposing the listener publicly.

Logs are structured JSON in normal operation. Preserve the request ID so an API error can be correlated with gateway and worker logs. Avoid recording message bodies, credentials, bearer tokens, or webhook secrets in downstream log processors.

OpenTelemetry export is disabled until GW_OTLP_ENDPOINT is set. Both gRPC and HTTP/protobuf are supported; the default trace sample ratio is intentionally low for production traffic. Increase it temporarily and deliberately when diagnosing a problem.

Rate limits are fleet-wide when Redis is the event bus. If Redis coordination is unavailable, the gateway retains a process-local organization guard and fails open for distributed coordination so that a Redis outage does not silently become a total API outage.