Monitoring

Monitoring

The orchestrator emits three observability streams:

• Prometheus metrics — /metrics (95+ metrics under the s3o_ prefix); see the full reference table below.
• OpenTelemetry traces — every HTTP request, manager operation, and backend S3 call; OTLP-exported to Tempo or compatible collector.
• Structured JSON logs — slog to stdout, including a dedicated audit subset with "audit": true.

A ready-to-import Grafana dashboard covering every metric ships at grafana/s3-orchestrator.json.

Web dashboard

When ui.enabled is true, the dashboard at {path}/ shows a live snapshot of:

• Storage summary — total bytes used/capacity across all backends
• Backend quota — bytes used/limit with progress bars per backend, object counts, active multipart uploads
• Monthly usage — API requests, egress, and ingress per backend with limits
• Object tree — interactive collapsible file browser. Buckets and directories are collapsed by default; click to expand. Each directory shows a rollup file count and total size.
• Configuration — virtual bucket names, write routing strategy, replication factor, rebalance status, rate limiting, encryption status
• Logs — recent structured log output from an in-memory ring buffer (last 5,000 entries). Filter by severity level and search by text. Logs are available immediately on page load — no need to SSH into the host.

The dashboard also provides management actions:

• Upload — upload files to any virtual bucket directly from the browser (up to 512 MiB per file)
• Download — download individual objects by clicking the download icon on any file in the tree
• Delete — delete individual objects by clicking the delete icon on any file in the tree
• Rebalance — trigger an on-demand rebalance using the configured strategy and settings
• Clean Excess — remove over-replicated copies that exceed the replication factor
• Sync — import pre-existing objects from a backend’s S3 bucket into the proxy database. Select a backend and a virtual bucket — objects already in the database are skipped, and objects belonging to other virtual buckets are excluded.

On-demand reconciliation

When a backend loses data (expired credentials, provider outage, accidental deletion), the metadata database retains stale entries that cause log noise in the rebalancer, replicator, and scrubber. The reconcile endpoint walks both the backend (paginated ListObjects) and the metadata DB (paginated cursor over object_locations ordered by key) as ascending key streams, then merges them in lockstep. Memory is bounded by a 1000-entry page size on each side regardless of total object count, so a backend holding millions of objects reconciles without OOM. Keys belonging to sibling virtual buckets stored on the same backend are skipped — each virtual bucket reconciles in its own pass.

# Reconcile all backends
curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/reconcile

# Reconcile a single backend
curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/reconcile?backend=g3

Response:

{"status":"ok","imported":0,"removed":57,"backends_scanned":1}

• imported: objects on the backend but not in the DB (brought under management)
• removed: DB entries whose objects no longer exist on the backend (stale metadata cleaned up)
• backends_scanned: how many backends were processed

The background reconciler (reconcile.enabled: true) runs the same logic on a timer. The admin endpoint is for immediate use after incidents.

The dashboard requires authentication. Users log in at {path}/login with the admin_key and admin_secret configured in the ui section. Sessions last 24 hours.

The dashboard is server-rendered HTML. The object tree uses JavaScript for lazy-loaded directory expansion — directories fetch their children on click via the /ui/api/tree endpoint.

JSON endpoints at {path}/api/dashboard, {path}/api/tree, and {path}/api/logs return data for programmatic access or integration with other tools. The logs endpoint accepts optional query parameters: level (minimum severity: DEBUG, INFO, WARN, ERROR), since (RFC3339 timestamp), component, and limit. Management endpoints ({path}/api/delete, {path}/api/upload, {path}/api/rebalance, {path}/api/clean-excess, {path}/api/sync) accept POST requests. The download endpoint ({path}/api/download?key=...) accepts GET requests. All API endpoints require authentication.

Health endpoints

Two health endpoints serve different purposes:

Liveness (/health) — always returns HTTP 200. Use this for liveness probes (Consul checks, K8s livenessProbe). The service stays in rotation during temporary database outages.

curl http://localhost:9000/health
# {"status":"ok"}
# or {"status":"degraded"} when the database circuit breaker is open

Readiness (/health/ready) — returns HTTP 200 when the service is ready to handle traffic, HTTP 503 during startup (before migrations and backend initialization complete) and during shutdown drain. Use this for readiness probes (K8s readinessProbe, Nomad on_update = "require_healthy").

curl http://localhost:9000/health/ready
# {"status":"ready"}      — 200
# {"status":"not ready"}  — 503

The HTTP response body is intentionally minimal — only the status field is returned, so log aggregators can grep on a fixed pattern. To identify which instance answered a probe in a multi-instance deployment, query GET /admin/api/workers (which includes the instance identifier) or correlate by source IP.

Background worker health

/health only reflects database breaker state. Background services (replicator, cleanup queue, lifecycle, pending reaper, …) are supervised by the lifecycle manager and recover on their own, but a service that is running yet failing every tick looks identical to a healthy one in /health.

GET /admin/api/workers returns a JSON snapshot of every registered background service’s last-tick health, including last_success, last_failure, last_error, and consecutive_failures. Use it during incidents to distinguish:

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/workers

{
  "workers": [
    {"name": "cleanup_queue", "last_success": "2026-05-12T18:42:01Z", "consecutive_failures": 0},
    {"name": "replicator", "last_failure": "2026-05-12T18:45:14Z", "last_error": "connection refused", "consecutive_failures": 3}
  ]
}

Workers in proxy-only deployments return 503 from this endpoint because no worker pool is registered.

The same data flows into Prometheus as s3o_worker_last_success_timestamp_seconds, s3o_worker_consecutive_failures, and s3o_worker_ticks_total{result} so alerting can run without scraping the admin endpoint. Suggested alert shapes are in the metrics table below.

Grafana dashboard

A comprehensive Grafana dashboard is included at grafana/s3-orchestrator.json. Import it via Grafana’s UI (Dashboards → Import → Upload JSON file) or provision it from disk. It expects a Prometheus datasource with UID prometheus.

The dashboard covers all emitted metrics, organised by domain into rows: overview, quota & storage, request performance, backend operations, manager operations, circuit breaker & degraded mode, replication, usage tracking, rate limiting & rejections, rebalancer, drain & lifecycle, cleanup queue & audit, encryption, object data cache, integrity verification, Redis, over-replication cleanup, pending PUT intents, and authentication (streaming SigV4). Rows for less frequently inspected domains are collapsed by default.

Key Prometheus metrics

If telemetry.metrics.enabled is true, metrics are exposed at /metrics. Two listener modes:

• Inline (telemetry.metrics.listen empty, the default): /metrics is served on the same listener as the public S3 API. Convenient for single-port deployments and the docker-compose / local-dev workflow. In this mode /debug/pprof/* endpoints are not registered — mounting pprof on the public S3 listener would leak runtime internals (de-anonymized stack frames via /debug/pprof/heap, the command line and flag values via /debug/pprof/cmdline) and offer a DoS amplifier (/debug/pprof/profile?seconds=300 triggers minutes of CPU profiling on demand).
• Dedicated listener (telemetry.metrics.listen set, e.g. 127.0.0.1:9001): /metrics and /debug/pprof/* are both mounted on the dedicated listener. Bind to 127.0.0.1 or a private network interface so the surface is only reachable from operators inside the trust boundary. The nomad demo uses 0.0.0.0:9001 so the docker-compose Prometheus container can scrape via the bridge gateway; production deployments should tighten this to 127.0.0.1:9001 or a private network address.

Once the dedicated listener is configured, captures look like:

# 60-second allocation profile during a load spike
curl -o /tmp/allocs.pb.gz "http://127.0.0.1:9001/debug/pprof/allocs?seconds=60"
go tool pprof -top -cum -alloc_space /tmp/allocs.pb.gz

# Instantaneous live-heap snapshot
curl -o /tmp/heap.pb.gz "http://127.0.0.1:9001/debug/pprof/heap"
go tool pprof -top -cum -inuse_space /tmp/heap.pb.gz

The standard net/http/pprof endpoints are available: /debug/pprof/, /debug/pprof/profile, /debug/pprof/heap, /debug/pprof/allocs, /debug/pprof/goroutine, /debug/pprof/block, /debug/pprof/mutex, /debug/pprof/trace, /debug/pprof/cmdline, /debug/pprof/symbol.

Key metrics to alert on:

Full metric reference

All metrics are prefixed with s3o_. Exposed at /metrics when telemetry.metrics.enabled is true.

Quota metrics are refreshed from PostgreSQL every 30 seconds (no backend API calls).

OpenTelemetry tracing

Spans are emitted for every HTTP request, manager operation, and backend S3 call. The service registers as s3-orchestrator (resource.service.name). Traces propagate via W3C traceparent headers. Configured to export via gRPC OTLP to Tempo or any OTLP-compatible collector.

Trace-to-log correlation — every JSON log line emitted within an active span automatically includes trace_id and span_id fields. Log aggregators (Loki, etc.) can use these fields to link logs to their corresponding traces in Tempo or any OpenTelemetry-compatible tracing backend. Only log calls that receive a context.Context with an active span include trace context; application-level logs without a span context are unaffected.

Structured logs

All logs are JSON to stdout. Key fields: msg, level, error, backend, operation.

Audit logs are a subset of structured logs with "audit": true. Every S3 API request and significant internal operation emits an audit entry with a request_id for correlation. Filter audit entries in your log pipeline with a JSON query on the audit field.

Key audit events:

Each S3 API request produces two correlated audit entries (HTTP-level and storage-level) sharing the same request_id. Internal operations (rebalance, replication) generate their own correlation IDs. The request_id also appears as a s3o.request_id attribute on OpenTelemetry spans.

HTTP panic recovery

A panic inside an HTTP handler is caught by the panic-recovery middleware applied to the S3 and admin route groups. The recovery contract:

• The client gets a structured response, not a connection reset. S3 routes receive an XML <Error><Code>InternalError</Code><Message>...Request ID: ...</Message></Error> body with HTTP 500; admin routes receive the same shape as a JSON {"error": "...Request ID: ..."}.
• The Prometheus counter s3o_http_panic_recovered_total{route} increments. A non-zero value is an immediate alert candidate.
• A slog.ErrorContext line is written at level ERROR with component=httputil, the route, the method and path, the panic value, the captured Go stack, and a trace_id / span_id if a span was active when the panic occurred.
• An http.PanicRecovered audit entry is emitted with the same correlation request_id as the failing request.
• If an OpenTelemetry span was active on the request, it is marked as failed via SetStatus(Error) + RecordError so traces in Tempo highlight the failure.

UI routes are intentionally not wrapped in the first iteration (mounted on the same mux as the S3 catch-all; the bulk of panic risk is on S3 and admin anyway). The recovery message deliberately does not echo the panic value to the client; only the request ID is returned so support tickets can be correlated back to the orchestrator log line.

Clients can supply their own correlation ID via the X-Request-Id request header; otherwise the orchestrator generates one. The ID is returned in the X-Amz-Request-Id response header.

Trace-to-log correlation — JSON log output includes trace_id and span_id fields on every line emitted within an active OpenTelemetry span. Log aggregators like Grafana Loki can extract these fields to link directly from a log entry to the corresponding trace in Tempo, and vice versa.