Admin Guide

This guide walks through deploying, configuring, and operating the S3 Orchestrator from scratch. For architecture and feature details, see the README. For client-side usage (AWS CLI, rclone, SDKs), see the User Guide.

Prerequisites

• PostgreSQL — any recent version. The orchestrator auto-applies its schema on startup.
• At least one S3-compatible storage backend — OCI Object Storage, Backblaze B2, AWS S3, MinIO, Wasabi, etc. You need a bucket and access credentials on that backend.
• The orchestrator binary — a Docker image (via make push VERSION=vX.Y.Z), a .deb package (via make deb VERSION=X.Y.Z), or built from source (make run).
• Redis (optional) — for shared usage counters in multi-instance deployments. See usage_flush for details.

Quickstart

Get a minimal single-bucket, single-backend orchestrator running in five steps.

1. Generate a config file

Run the interactive config generator:

s3-orchestrator init

This prompts for a database driver (SQLite by default), backend credentials, and virtual bucket settings. Or create a config manually:

2. Create a minimal config

server:
  listen_addr: "0.0.0.0:9000"

database:
  driver: sqlite
  path: "s3-orchestrator.db"

buckets:
  - name: "my-files"
    credentials:
      - access_key_id: "AKID_MYFILES"
        secret_access_key: "changeme-replace-with-random-secret"

backends:
  - name: "primary"
    endpoint: "https://namespace.compat.objectstorage.us-phoenix-1.oraclecloud.com"
    region: "us-phoenix-1"
    bucket: "my-actual-bucket"
    access_key_id: "backend-access-key"
    secret_access_key: "backend-secret-key"
    force_path_style: true
    quota_bytes: 21474836480  # 20 GB

Save this as config.yaml.

3. Start the orchestrator

# From source
s3-orchestrator -config config.yaml

# Or via Docker
docker run -v $(pwd)/config.yaml:/etc/s3-orchestrator/config.yaml -p 9000:9000 s3-orchestrator

4. Verify it’s running

curl http://localhost:9000/health
# {"status":"ok","instance":"hostname"}

5. Test with a quick upload/download

aws --endpoint-url http://localhost:9000 \
    s3 cp /etc/hostname s3://my-files/test.txt

aws --endpoint-url http://localhost:9000 \
    s3 cp s3://my-files/test.txt -

Configuration Walkthrough

This section covers each config section in detail. See packaging/config.yaml for a complete template.

All config values support ${ENV_VAR} expansion — the orchestrator calls os.Expand on the entire YAML file before parsing. Use this for secrets:

database:
  password: "${DB_PASSWORD}"

server

server:
  listen_addr: "0.0.0.0:9000"    # required
  log_level: "info"               # debug, info, warn, error (default: info, reloadable via SIGHUP)
  max_object_size: 5368709120     # 5 GB default
  max_concurrent_requests: 0      # max concurrent S3 requests (0 = unlimited, default)
  # max_concurrent_reads: 0       # separate limit for GET/HEAD (0 = use global limit)
  # max_concurrent_writes: 0      # separate limit for PUT/POST/DELETE (0 = use global limit)
  # load_shed_threshold: 0        # active shedding threshold (0.0-1.0, 0 = disabled)
  # admission_wait: "0s"          # brief wait before rejection (0 = instant)
  backend_timeout: "30s"          # per-operation timeout for backend S3 calls
  read_header_timeout: "10s"      # max time to read request headers (default: 10s)
  read_timeout: "5m"              # max time to read entire request including body (default: 5m)
  write_timeout: "5m"             # max time to write response (default: 5m)
  idle_timeout: "120s"            # max time to wait for next request on keep-alive (default: 120s)
  shutdown_delay: "5s"            # delay before HTTP drain on SIGTERM (default: 0, no delay)

• listen_addr is the only required field.
• max_object_size caps single-PUT uploads. Larger objects should use multipart upload (most clients do this automatically). PutObject buffers the entire body in memory to support write failover across backends, so peak memory from uploads is approximately max_object_size x max_concurrent_writes.
• max_concurrent_requests limits the number of S3 requests processed simultaneously. When the limit is reached, new requests are rejected with 503 SlowDown and Retry-After: 1. Set to 2-3x database.max_conns for load shedding. 0 disables the limit.
• max_concurrent_reads and max_concurrent_writes provide separate concurrency limits for reads (GET, HEAD) and writes (PUT, POST, DELETE). When both are set, they replace max_concurrent_requests with independent pools so write storms cannot starve reads. Background workers contend with HTTP writes, not reads — cleanup, replication, rebalance, pending reaper, and over-replication acquire admission slots from the same pool sized to max_concurrent_writes. In merged mode (max_concurrent_requests only), every HTTP request and every background worker shares the single global pool. Size max_concurrent_writes to accommodate both peak HTTP write traffic and the worst-case overlap of background worker activity (typically the replication factor × replicator concurrency for the dominant case). See issue #835 for the design rationale.
• load_shed_threshold enables active load shedding. When in-flight requests exceed this fraction of pool capacity (e.g. 0.8), new requests are probabilistically rejected before the hard limit, providing smooth degradation instead of a cliff.
• admission_wait adds a brief wait before rejecting when the semaphore is full (e.g. 50ms). Smooths micro-bursts without adding latency during sustained overload. Default 0 means instant rejection.
• backend_timeout bounds individual S3 API calls to backends. Increase if you have slow backends or large objects.
• read_header_timeout protects against slow-read attacks that hold connections open by sending headers slowly. The 10-second default is generous for any legitimate client.
• read_timeout and write_timeout bound the total time for reading/writing entire requests and responses. The 5-minute defaults accommodate large object transfers.
• idle_timeout controls how long keep-alive connections stay open waiting for the next request.
• shutdown_delay adds a pause between marking the instance as not-ready and starting the HTTP drain on SIGTERM. Set this to ~5s in environments where service deregistration is asynchronous (Consul, Kubernetes) so load balancers stop routing before connections are closed. Default 0 means no delay.

buckets

Each bucket defines a virtual namespace with one or more credential sets.

buckets:
  - name: "app1-files"
    # max_multipart_uploads: 100  # optional; limit active multipart uploads (0 = unlimited)
    credentials:
      - access_key_id: "AKID_APP1"
        secret_access_key: "secret1"

  - name: "app2-files"
    credentials:
      - access_key_id: "AKID_APP2_WRITER"
        secret_access_key: "secret2"
      - access_key_id: "AKID_APP2_READER"
        secret_access_key: "secret3"

Generating credentials: Use openssl rand to produce random keys:

# Generate an access key ID (20 chars, uppercase + digits)
openssl rand -hex 10 | tr '[:lower:]' '[:upper:]'

# Generate a secret access key (40 chars, base64)
openssl rand -base64 30

Validation rules:

• Bucket names must not contain /.
• Bucket names must be unique across the config.
• Access key IDs must be globally unique across all buckets.
• Each bucket must have at least one credential set.
• Each credential needs either access_key_id + secret_access_key (SigV4) or token (legacy).

Multiple credentials on the same bucket let different services share a namespace with independent keys. This is useful when you want a writer service and a reader service accessing the same files.

SigV4 credentials also support presigned URLs automatically. Clients can generate time-limited presigned URLs using any AWS SDK presign client — no additional configuration is needed on the orchestrator side.

database

The driver field selects between SQLite (embedded, zero-dependency) and PostgreSQL (required for multi-instance deployments). When driver is omitted, the orchestrator infers postgres if host is set, otherwise sqlite.

SQLite (default for single-instance):

database:
  driver: sqlite
  path: "s3-orchestrator.db"     # default: s3-orchestrator.db

SQLite requires no external dependencies. The database file is created automatically on first start. Advisory lock-based leader election is replaced by a process-local mutex, so multi-instance deployments are not supported with SQLite.

PostgreSQL (required for multi-instance):

database:
  driver: postgres
  host: "db.example.com"        # required
  port: 5432                     # default: 5432
  database: "s3orchestrator"     # required
  user: "s3orchestrator"         # required
  password: "${DB_PASSWORD}"
  ssl_mode: "require"            # default: require (use "disable" for local dev)
  max_conns: 50                  # default: 50; size to 2-3x max_concurrent_requests
  min_conns: 10                  # default: 5
  max_conn_lifetime: "5m"        # default: 5m

Pool settings (max_conns, min_conns, max_conn_lifetime) control the pgx connection pool. Size max_conns to 2-3x your max_concurrent_requests setting. See Performance Tuning - Connection Pool Sizing for detailed guidance.

routing_strategy

Controls how the orchestrator selects a backend when writing new objects.

routing_strategy: "pack"       # "pack" or "spread" (default: pack)

• pack (default) — fills the first backend in config order until its quota is full, then overflows to the next. Best for stacking free-tier allocations sequentially.
• spread — places each object on the backend with the lowest utilization ratio ((bytes_used + orphan_bytes) / bytes_limit). Best for distributing storage evenly across backends.

Both strategies respect quota limits and usage limits — full or over-limit backends are always skipped.

backends

Each backend is an S3-compatible storage service with its own credentials and optional quota.

backends:
  - name: "oci"
    endpoint: "https://namespace.compat.objectstorage.us-phoenix-1.oraclecloud.com"
    region: "us-phoenix-1"
    bucket: "my-oci-bucket"
    access_key_id: "${OCI_ACCESS_KEY}"
    secret_access_key: "${OCI_SECRET_KEY}"
    force_path_style: true
    quota_bytes: 21474836480     # 20 GB

Endpoint URLs by provider:

Quota: Set quota_bytes to limit how much data a backend can hold. Set to 0 or omit for unlimited. Quota is tracked in PostgreSQL and updated atomically with every write/delete. Note that multipart uploads do not reserve quota upfront — temporary parts consume backend storage without being counted against the quota until CompleteMultipartUpload records the final object size. A client uploading many large parts could temporarily exceed a backend’s quota before completion.

Max object size: Some providers impose per-object size limits (e.g. Supabase rejects uploads over 50 MB with 413 EntityTooLarge). Set max_object_size to prevent the orchestrator from routing writes, rebalance moves, or replication copies to a backend when the object exceeds the limit:

    max_object_size: 52428800    # 50 MB (0 = unlimited)

Usage limits: Optional monthly caps on API requests, egress, and ingress per backend:

    api_request_limit: 20000     # monthly API calls (0 = unlimited)
    egress_byte_limit: 1073741824  # 1 GB monthly egress (0 = unlimited)
    ingress_byte_limit: 0        # unlimited ingress

When a backend exceeds a usage limit, writes overflow to the next eligible backend. Limits reset each month automatically.

Unsigned payload: By default, uploads stream directly to backends without buffering the entire body in memory. The AWS SDK normally buffers the request body to compute a SigV4 payload hash (SHA-256), but the orchestrator uses UNSIGNED-PAYLOAD to skip this. Without streaming, large uploads (multipart completion, replication) can cause out-of-memory kills.

For HTTPS endpoints, unsigned payload is enabled by default. For plain HTTP endpoints, it is auto-disabled unless explicitly set — AWS S3 rejects unsigned payloads over HTTP, but most S3-compatible backends (MinIO, R2, etc.) accept them. Set unsigned_payload: true on HTTP backends to enable streaming:

    unsigned_payload: true   # stream uploads without buffering (auto-enabled for HTTPS)

Set unsigned_payload: false to force payload hashing. This buffers the entire object in memory before uploading — only use this if you have a specific compliance requirement for end-to-end payload integrity independent of TLS.

Disable checksum: AWS SDK v2 defaults to sending streaming checksums (CRC64NVME) on uploads. Some S3-compatible providers — notably Google Cloud Storage — reject these with SignatureDoesNotMatch. Set disable_checksum: true on backends that don’t support the AWS checksum headers:

    disable_checksum: true   # required for GCS HMAC interoperability

This sets the SDK’s RequestChecksumCalculation and ResponseChecksumValidation to WhenRequired, disabling automatic checksum injection without affecting SigV4 request signing.

Strip SDK headers: AWS SDK v2 adds headers (amz-sdk-invocation-id, amz-sdk-request, accept-encoding) and a query parameter (x-id) that are included in the SigV4 signed header set. Google Cloud Storage does not include these when verifying the signature, causing SignatureDoesNotMatch errors. Set strip_sdk_headers: true to remove them before request signing:

    strip_sdk_headers: true   # required for GCS HMAC interoperability

For GCS backends, you typically need both disable_checksum: true and strip_sdk_headers: true:

  - name: "gcs"
    endpoint: "https://storage.googleapis.com"
    region: "auto"
    bucket: "my-bucket"
    access_key_id: "GOOG..."
    secret_access_key: "..."
    force_path_style: true
    disable_checksum: true
    strip_sdk_headers: true

Credential source: credential_source selects how the orchestrator obtains credentials for the backend. Default is static, which uses the access_key_id / secret_access_key fields above. Set to default_chain to delegate to the AWS SDK’s default credential chain (env vars, EC2 IMDS, SSO, ~/.aws/credentials, STS assume-role). When default_chain is set, the two key fields must be omitted — leaving stale keys behind is rejected at validation so they cannot silently shadow the SDK-resolved credentials.

Use default_chain when:

• The orchestrator runs on an EC2 instance with an IAM role attached (IMDS-vended credentials rotate every ~6 hours and cannot be tracked by YAML).
• Local development uses SSO (aws sso login) instead of long-lived keys.
• You want the SDK to resolve credentials via STS assume-role chains.

  - name: "aws-prod"
    endpoint: "https://s3.amazonaws.com"
    region: "us-east-1"
    bucket: "my-prod-bucket"
    credential_source: "default_chain"
    # access_key_id / secret_access_key intentionally omitted

Note: the config loader already expands ${ENV_VAR} references at load time, so access_key_id: ${AWS_ACCESS_KEY_ID} covers the env-var case under credential_source: static. Use default_chain for credential sources the loader cannot reach (IMDS, SSO, STS) and for cases where refresh matters.

telemetry

telemetry:
  metrics:
    enabled: true
    path: "/metrics"             # default: /metrics
    # listen: "127.0.0.1:9091"  # serve on separate address (keeps /metrics off the public port)
  tracing:
    enabled: false
    endpoint: "localhost:4317"   # OTLP gRPC endpoint
    insecure: true               # no TLS to collector
    sample_rate: 1.0             # fraction of requests that generate traces (use 0.01–0.1 in production)

Metrics are served on the same port as the S3 API. Tracing exports spans via gRPC OTLP (e.g., to Tempo or Jaeger).

Production sample rate guidance: A sample_rate of 1.0 traces every request, which is appropriate for development and low-traffic deployments. For production workloads above ~100 RPS, reduce to 0.01–0.1 to avoid overwhelming the trace backend with storage, network, and CPU overhead. Metrics and logs are unaffected by sample rate.

circuit_breaker

The circuit breaker is always active. These settings tune its sensitivity.

circuit_breaker:
  failure_threshold: 3           # consecutive DB failures before opening (default: 3)
  open_timeout: "15s"            # delay before probing recovery (default: 15s)
  cache_ttl: "60s"               # key→backend cache TTL during degraded reads (default: 60s)
  parallel_broadcast: false      # fan-out reads to all backends in parallel (default: false)
  degraded_broadcast_parallelism: 0 # cap concurrent probes during parallel broadcast; 0 = no cap (default: 0)

When the database is unreachable, the orchestrator enters degraded mode: reads broadcast to all backends (with caching), writes return 503. The circuit automatically recovers when the database comes back.

By default, degraded reads try each backend sequentially. When parallel_broadcast is enabled, all backends are tried concurrently and the first success wins — reducing worst-case read latency from N * backend_timeout to roughly the fastest backend’s response time. Enable this if read latency during outages is critical, but note that each parallel broadcast sends API requests to all backends simultaneously, which counts against monthly usage limits.

For fleets with many configured backends, set degraded_broadcast_parallelism to cap how many backends are probed at once. With a positive value, probes run as a rolling window: the first N launch immediately and each failure replenishes the slot with the next pending backend, so at most N goroutines (and at most N concurrent backend API calls / TLS handshakes) are in flight at any time. The default of 0 preserves the historical “fan out to every backend at once” behaviour.

The other defaults are sensible for most deployments. Increase cache_ttl if you have many read-heavy clients and want fewer backend round-trips during outages.

backend_circuit_breaker

Per-backend circuit breakers isolate failures at the individual backend level. When a backend’s credentials expire or the provider becomes unreachable, the circuit opens after consecutive failures and the backend is excluded from request routing. A single probe request tests recovery after the timeout elapses. Disabled by default.

backend_circuit_breaker:
  enabled: true
  failure_threshold: 5             # consecutive failures before opening (default: 10)
  open_timeout: "5m"               # delay before probing recovery (default: 5m)

Unlike the database circuit breaker, which triggers degraded mode for the entire system, backend circuit breakers affect only the individual backend. Reads fall back to other replicas, and writes route to other backends with available quota. No extra API calls are made — the breaker trips purely on organic traffic failures.

The s3o_circuit_breaker_state{name="<backend>"} metric tracks each backend’s circuit state (0=closed, 1=open, 2=half-open). Alert on > 0 for individual backends to detect credential or provider issues. Requires a restart to change (not hot-reloadable).

rebalance

Moves objects between backends to optimize storage distribution. Disabled by default — enabling it will generate egress/ingress traffic on your backends.

rebalance:
  enabled: true
  strategy: "pack"               # "pack" or "spread" (default: pack)
  interval: "6h"                 # default: 6h
  batch_size: 100                # objects per run (default: 100)
  threshold: 0.1                 # min utilization spread to trigger (default: 0.1)
  concurrency: 5                 # parallel moves per run (default: 10)

• pack — fills backends in config order, consolidating free space onto the last backend. Good for maximizing free-tier allocations.
• spread — equalizes utilization ratios across all backends. Good for distributing load.

Object moves run concurrently within each batch, bounded by concurrency. Increase for faster rebalancing; decrease to reduce backend load.

replication

Creates additional copies of objects on different backends for redundancy.

replication:
  factor: 2                      # copies per object (default: 1 = no replication)
  worker_interval: "5m"          # replication cycle (default: 5m)
  batch_size: 50                 # objects per cycle (default: 50)
  concurrency: 5                 # parallel object replications per cycle (default: 10)
  unhealthy_threshold: "10m"     # grace period before replacing copies on circuit-broken backends (default: 10m)

The replication factor must be <= number of backends. The worker runs once at startup to catch up on any pending replicas, then continues at the configured interval. Reads automatically fail over to replicas if the primary copy is unavailable.

Replication is asynchronous — writes go to a single backend and the replicator creates additional copies in the background. When a client overwrites an existing key, all old copies (including replicas) are removed and a single new copy is written. The replication factor drops to 1 until the next replicator cycle creates the additional copies. If the single backend holding the new copy fails before replication runs, the new version of the object is at risk. For most workloads this window (up to worker_interval) is acceptable. Lowering worker_interval reduces the exposure at the cost of more frequent DB queries and backend I/O.

Health-aware replication: When backend circuit breakers are enabled, the replicator monitors backend health. If a backend’s circuit breaker has been open longer than unhealthy_threshold, copies on that backend are treated as unavailable and replacement copies are created on healthy backends. This prevents sustained outages from silently reducing redundancy. The threshold prevents churn during brief transient failures. Set to 0 to disable health-aware replication (copies on down backends are still counted).

Cleanup Queue

The cleanup queue is always active. Tunables:

cleanup_queue:
  concurrency: 10                # parallel cleanup deletions per tick (default: 10)
  claim_grace_period: 5m         # reclaim stale per-row claims older than this (default: 5m)
  multipart_stale_timeout: 24h   # abort multipart uploads older than this (default: 24h)

multipart_stale_timeout is consumed by the hourly CleanupStaleMultipartUploads sweep — uploads that have been open longer than this are aborted, their parts deleted from the backend, and the multipart rows removed. The default 24h matches the AWS S3 SDK’s default abort behavior; lower it on backends with tight free-tier headroom to recover quota faster.

When any backend object deletion fails during normal operations (PutObject orphan cleanup, DeleteObject, overwrite displaced copies, multipart part cleanup, rebalancer, replicator), the failed deletion is automatically enqueued for retry.

Each enqueued item tracks the object’s size_bytes. On enqueue, the backend’s orphan_bytes counter is incremented so that write routing and replication target selection account for the physically unreleased space. On successful cleanup the row is removed and orphan_bytes is decremented in a single atomic CTE; a worker crash between the two operations cannot leave the counter inconsistent.

Per-row claim pattern. Every row carries claimed_at and claimed_by columns. When a worker tick fetches a batch it stamps each row with the current instance’s identifier and timestamp, gated by FOR UPDATE SKIP LOCKED (Postgres) or SQLite’s intrinsic single-writer serialisation. Two instances ticking concurrently always see disjoint row sets, so a connection death or rolling-deploy overlap that would otherwise let two workers process the same row is now structurally impossible. A claim older than claim_grace_period (default 5m) is reclaimable so a worker that died mid-process does not leave the row stuck; reclaims emit s3o_cleanup_queue_stale_claims_recovered_total and a cleanup_queue.claim_recovered audit event.

The background worker runs every minute and retries with exponential backoff (1 minute to 24 hours). Scheduling a retry clears the row’s claim so it is immediately re-eligible for the next tick. After 10 failed attempts, the row is graduated to the cleanup_dlq table via core.MoveCleanupToDLQ (single transaction: read the row, insert it into cleanup_dlq, delete it from cleanup_queue). orphan_bytes is intentionally NOT decremented during the move because the backend object is still on disk. The DLQ entry retains the full row payload (key, backend, size, reason, last_error) plus an original_id correlation column so an operator can find the original queue entry.

Monitoring:

• s3o_cleanup_queue_depth staying elevated — orphaned objects are accumulating in the active queue.
• s3o_cleanup_queue_processed_total{status="exhausted"} — counter increments each time an item exhausts retries.
• s3o_cleanup_queue_processed_total{status="success_absent"} — counter increments each time a backend DELETE returned 404 and the row was dropped as idempotent success (the backend already agrees the object is gone). A sustained rate here is benign and just means upstream PUTs are silently failing somewhere; spikes are worth correlating with backend health.
• s3o_cleanup_queue_stale_claims_recovered_total{backend} — non-zero rate means a worker died mid-process or the grace period is too short for realistic worst-case processing time.
• s3o_cleanup_dlq_depth > 0 — the DLQ holds at least one unrecoverable orphan; alerting here gives operators a direct signal instead of a counter delta.
• s3o_cleanup_dlq_enqueued_total{backend} — rate of graduations per backend; a single backend dominating means that backend’s delete path is broken.
• s3o_cleanup_enqueue_failures_total{backend,reason,stage} — orphan-leak blind spot signal. The cleanup-queue itself is durable, but its enqueue path is best-effort: when a backend write succeeds and the DB is then unreachable, the orphan cannot be recorded in cleanup_queue and the only signal is this counter plus the matching storage.OrphanEnqueueFailed audit event. stage="enqueue" is the worst case (the cleanup-queue worker will never see this orphan); stage="orphan_bytes" means the row landed but the quota counter drifts. See the runbook below.
• s3o_quota_orphan_bytes — elevated values mean backends have significant physically unreleased space (DLQ entries are the long-tail contributors).

Untracked-orphan recovery (cleanup enqueue failed during DB outage). A non-zero rate of s3o_cleanup_enqueue_failures_total{stage="enqueue"} means at least one orphan exists on a backend with no cleanup_queue row. The cleanup-queue worker will not retry it; the storage will leak until reconciled. Recovery workflow:

1. Query the audit log for event="storage.OrphanEnqueueFailed" to enumerate the specific backend/key/size of each affected orphan during the outage window.
2. Once DB connectivity is restored, run POST /admin/api/reconcile[?backend=name]. The reconciler walks each backend’s actual key list against object_locations using a bounded-memory sorted-merge and emits S3-only keys to the cleanup path (with a fresh cleanup_queue row this time). This is the same diff machinery that runs on the nightly reconcile interval.
3. If the audit log indicates more than a handful of failures, target the reconciler at the affected backends specifically rather than waiting for the next scheduled scan.

stage="orphan_bytes" failures do not need step 2 — the cleanup_queue row landed and the worker will eventually delete the object. The quota counter drift is reset when backend_quotas.orphan_bytes is reconciled against cleanup_queue (a periodic safety pass; not yet automated).

Manual cleanup: Inspect DLQ entries and resolve them deliberately. The bytes are still on the backend, so the workflow is delete the object out-of-band, then write off the row + adjust orphan_bytes by the row’s size:

-- View unrecoverable orphans needing manual intervention
SELECT id, original_id, backend_name, object_key, reason, attempts,
       size_bytes, first_enqueued_at, moved_at, last_error
FROM cleanup_dlq
ORDER BY moved_at;

-- After confirming the object is gone (manual S3 delete, reconciler sweep, etc.):
BEGIN;
UPDATE backend_quotas
   SET orphan_bytes = GREATEST(0, orphan_bytes - (SELECT size_bytes FROM cleanup_dlq WHERE id = 42))
 WHERE backend_name = (SELECT backend_name FROM cleanup_dlq WHERE id = 42);
DELETE FROM cleanup_dlq WHERE id = 42;
COMMIT;

-- Or, to push a DLQ entry back through automatic retry (e.g. after fixing the backend):
INSERT INTO cleanup_queue (backend_name, object_key, reason, size_bytes, next_retry, attempts, last_error)
SELECT backend_name, object_key, reason, size_bytes, NOW(), 0, last_error
  FROM cleanup_dlq WHERE id = 42;
DELETE FROM cleanup_dlq WHERE id = 42;

write_path

The write path can run in two modes. Direct mode (enabled: false) writes to the backend and commits the metadata immediately afterward; a crash between the two leaks bytes onto the backend with no DB record. Pending-intent mode (enabled: true, the default) inserts a row into pending_objects before the backend PUT and atomically deletes that row when the metadata commits — so a crash between the PUT and the commit leaves a recoverable intent the background reaper can resolve.

write_path:
  pending_pattern:
    enabled: true        # default: true; PUT-before-COMMIT crash-recovery pattern
    reaper_tick: 1m      # how often PendingReaper sweeps unresolved intents (default: 1m)
    min_age: 5m          # only intents older than this are eligible (default: 5m) — avoids racing in-flight PUTs
    batch_size: 50       # rows claimed per tick (default: 50)

How recovery works. On every tick the PendingReaper worker (internal/worker/pending.go) claims a batch of pending_objects rows older than min_age, HEADs the backend at the recorded key, and resolves each one:

• HEAD 200 → the backend received the bytes. Promote the intent to a committed object_locations row (pending_reaper.promoted audit event).
• HEAD 404 → the backend never received the bytes. Drop the intent (pending_reaper.dropped audit event). No orphan exists.
• Non-404 HEAD error → leave the intent for the next tick. A sustained backend reachability problem here surfaces as s3o_pending_intents_resolved_total{status="ambiguous"}.
• A later write for the same key already committed → drop the intent as superseded (pending_reaper.superseded).

Why min_age matters. The reaper must not race the foreground write path; if min_age is too short the reaper can interrogate an intent whose backend PUT is still in flight and either prematurely commit it or churn ambiguous resolutions. The 5-minute default is generous; lower it only if you have measured the p99 PUT duration and accept the operational tradeoff.

Monitoring:

• s3o_pending_intents_enqueued_total — should track the PutObject rate closely.
• s3o_pending_intents_resolved_total{status} — committed is the happy path (synchronous commit succeeded); promoted + dropped are reaper resolutions; ambiguous is the alert.
• s3o_pending_intents_depth — gauge of unresolved intents. Alert when consistently above batch_size — the reaper is not keeping up (raise batch_size, lower reaper_tick, or add concurrency).
• Audit events: pending_reaper.promoted / pending_reaper.dropped / pending_reaper.superseded.

When to disable. Don’t, unless you are running an embedded SQLite single-instance demo and trust the OS to flush. The pattern adds one DB write per PUT (cheap) and saves you from one entire class of write-path crash leak.

rate_limit

Per-IP token bucket rate limiting. When enabled, rate limiting applies to both the S3 proxy and the admin API. Requests exceeding the limit receive 429 SlowDown.

rate_limit:
  enabled: true
  requests_per_sec: 100          # token refill rate (default: 100)
  burst: 200                     # max burst size (default: 200)
  cleanup_interval: "1m"         # stale entry eviction interval (default: 1m)
  cleanup_max_age: "5m"          # evict entries not seen within this window (default: 5m)
  trusted_proxies:               # CIDRs whose X-Forwarded-For is trusted
    - "10.0.0.0/8"
    - "172.16.0.0/12"

A background goroutine evicts per-IP entries not seen within cleanup_max_age every cleanup_interval. Under high source-IP cardinality (e.g., DDoS), the map can hold up to cleanup_max_age worth of unique IPs — tune both values down if memory pressure is a concern.

When trusted_proxies is configured, the orchestrator extracts the real client IP from the X-Forwarded-For header using rightmost-untrusted extraction: it walks the XFF chain from right to left, skipping addresses within trusted CIDRs, and uses the first untrusted address for rate limiting. If the direct peer is not in a trusted CIDR, X-Forwarded-For is ignored entirely to prevent spoofing. Without trusted_proxies, the direct connection IP is always used.

Multi-instance note: Rate limits are enforced per-instance. Behind a load balancer with round-robin routing, the effective rate for a given client is requests_per_sec * instance_count. Divide your desired aggregate rate by the number of API instances when configuring.

ui

Built-in web dashboard for operational visibility and management. Disabled by default. Requires authentication via an admin key/secret pair — sessions are HMAC-signed cookies with a 24-hour TTL.

ui:
  enabled: true
  path: "/ui"                          # URL prefix (default: /ui)
  admin_key: "${UI_ADMIN_KEY}"         # access key for dashboard login
  admin_secret: "${UI_ADMIN_SECRET}"   # secret key — plaintext or bcrypt hash
  admin_token: "${UI_ADMIN_TOKEN}"     # separate token for admin API (defaults to admin_key)
  session_secret: "${UI_SESSION_SECRET}" # required — HMAC key for session cookies
  force_secure_cookies: true           # always set Secure flag on cookies (for behind TLS proxy)

admin_key, admin_secret, and session_secret are all required when enabled is true. Generate credentials the same way as bucket credentials:

echo "Admin Key: $(openssl rand -hex 10 | tr '[:lower:]' '[:upper:]')"
echo "Admin Secret: $(openssl rand -base64 30)"

Bcrypt-hashed secrets: For bare-metal deployments where the config file is at rest on disk, you can store admin_secret as a bcrypt hash instead of plaintext. The orchestrator detects bcrypt hashes automatically (they start with $2). Generate one with:

htpasswd -nbBC 10 "" 'your-secret' | cut -d: -f2

Both plaintext and bcrypt secrets are fully supported — no config migration needed.

Session secret: Session keys are derived deterministically from session_secret using HMAC-SHA256, so sessions survive restarts. For multi-instance deployments behind a load balancer, all instances sharing the same session_secret will accept each other’s sessions. Generate a value with:

openssl rand -hex 32

session_secret is independent of admin_secret — rotating the admin password does not invalidate active sessions, and vice versa.

usage_flush

Controls how often usage counters are flushed to the database. When adaptive flushing is enabled, the interval shortens automatically when any backend approaches a usage limit, improving enforcement accuracy.

usage_flush:
  interval: "30s"            # base flush interval (default: 30s)
  adaptive_enabled: true     # shorten interval when near limits (default: false)
  adaptive_threshold: 0.8    # usage ratio to trigger fast flush (default: 0.8)
  fast_interval: "5s"        # interval when near limits (default: 5s)

• interval — how often counters are flushed under normal conditions. Lower values reduce staleness but increase database writes.
• adaptive_enabled — when true, the flush interval drops to fast_interval whenever any backend’s effective usage exceeds adaptive_threshold of its configured limit.
• adaptive_threshold — the ratio (0–1 exclusive) at which fast flushing kicks in. At 0.8, a backend at 80% of any usage limit triggers the fast interval.
• fast_interval — must be less than interval. Used when adaptive flushing detects a backend near its limits.

Multi-instance note: Without Redis, each instance accumulates usage counters in memory between flushes. With N instances, the enforcement margin near limits is up to N * interval worth of unaccounted operations. Adaptive flushing reduces this near limits but doesn’t eliminate it. For tighter enforcement, configure Redis shared counters to eliminate the cross-instance blind spot entirely, or reduce interval and run fewer API instances.

redis

Optional shared usage counters for multi-instance deployments. When configured, all instances share usage counters via Redis instead of tracking them independently in memory. This eliminates the cross-instance blind spot between PostgreSQL flushes.

redis:
  address: "redis.example.com:6379"  # host:port (required when section is present)
  password: "${REDIS_PASSWORD}"       # AUTH password (omit for no auth)
  db: 0                               # Redis database number (default: 0)
  tls: false                          # enable TLS (default: false)
  key_prefix: "s3orch"                # namespace for multi-tenant Redis (default: s3orch)
  failure_threshold: 3                # consecutive failures before local fallback (default: 3)
  open_timeout: "15s"                 # delay before probing Redis recovery (default: 15s)

• address — required when the redis section is present. The orchestrator PINGs Redis on startup and fails hard if unreachable.
• key_prefix — namespaces all Redis keys. Use different prefixes if multiple orchestrator deployments share one Redis instance.
• failure_threshold and open_timeout — control the circuit breaker that falls back to local counters when Redis is unavailable.

When Redis is active, the usage flush service acquires a PostgreSQL advisory lock so only one instance performs the destructive GETSET + flush-to-PG operation. When Redis is in fallback (or not configured), each instance flushes independently without a lock.

A background health probe runs every 5 seconds while the breaker is open: it PINGs Redis and, on success, syncs the accumulated local-counter deltas back via an additive INCRBY pipeline (no DEL — keys from before the outage expire via TTL) and recloses the breaker. The breaker recovery is clean: the failure counter is zeroed so the system tolerates the configured failure_threshold of new transient errors before tripping again. No process restart is required after a Redis outage.

Redis is not reloadable — changing Redis settings requires a restart.

lifecycle

Automatically deletes objects whose key matches a prefix and whose age exceeds the configured expiration. Useful for temporary uploads, staging artifacts, or anything with a known retention period.

lifecycle:
  rules:
    - prefix: "tmp/"
      expiration_days: 7
    - prefix: "uploads/staging/"
      expiration_days: 1

• prefix — key prefix to match (required, must be non-empty).
• expiration_days — delete objects older than this many days (required, must be > 0).
• Omit the lifecycle section or leave rules empty to disable lifecycle entirely.
• Rules are evaluated every hour by a background worker with an advisory lock.
• Deletions go through the standard DeleteObject path — all copies removed, quotas decremented, failed deletes enqueued to the cleanup queue.
• Hot-reloadable via SIGHUP.

encryption

Server-side envelope encryption with chunked AES-256-GCM. When enabled, objects are encrypted before being stored on backends and decrypted transparently on read. Exactly one key source is required.

encryption:
  enabled: true
  chunk_size: 65536                    # default: 64KB (range: 4KB–1MB, must be power of 2)
  master_key: "${ENCRYPTION_KEY}"      # base64-encoded 256-bit key

Generating a master key:

openssl rand -base64 32

Key source options — exactly one of the following must be set:

Vault Transit configuration:

encryption:
  enabled: true
  vault:
    address: "http://vault.service.consul:8200"
    token: "${VAULT_TOKEN}"
    key_name: "s3-orchestrator"
    mount_path: "transit"     # default: transit

The Vault Transit engine handles wrapping and unwrapping DEKs — the orchestrator never sees the master key material. The key_name must reference an existing key in the Transit engine.

Key rotation support:

When rotating to a new master key, move the old key to previous_keys so existing objects can still be decrypted:

encryption:
  enabled: true
  master_key: "${NEW_ENCRYPTION_KEY}"         # new primary key
  previous_keys:
    - "${OLD_ENCRYPTION_KEY}"                 # old key, kept for unwrapping

After updating the config, call the rotate-encryption-key admin API to re-wrap all DEKs with the new key. See Rotating encryption keys below.

Important notes:

• Encryption is not reloadable — changing encryption settings requires a restart.
• The chunk_size must stay the same for the lifetime of the data. Changing it after objects are encrypted will make those objects unreadable.
• Encrypted objects are slightly larger than their plaintext (header + per-chunk overhead). The exact overhead is: 32 bytes (header) + 28 bytes per chunk (nonce + auth tag).

integrity

SHA-256 content hashing for data integrity verification. When enabled, objects are checksummed on write and the hash is stored alongside the object location in PostgreSQL.

integrity:
  enabled: true
  verify_on_read: true               # Hash-check every GET response as it streams
  verify_on_replicate: true          # Verify hash when creating replicas (default: true)
  scrubber_interval: "6h"            # Background verification interval (0 = disabled)
  scrubber_batch_size: 100           # Objects per scrub cycle

How it works:

• Write path: SHA-256 is computed on the plaintext body (before encryption) and stored in object_locations.content_hash.
• Read path (verify_on_read): A VerifyingReader wraps the response body and computes the hash as data streams to the client. On mismatch at EOF, the corrupted copy is enqueued for cleanup.
• Scrubber: A background worker periodically reads random objects from backends, decrypts if needed, and verifies their hash. Corrupted copies are enqueued for cleanup. Each read counts against the backend’s usage quota.
• Backfill: Objects written before integrity was enabled have no stored hash. Use admin backfill-checksums to read those objects and compute their hashes.

Integrity is hot-reloadable — changes take effect on SIGHUP without a restart.

cache

Optional in-memory LRU cache for full GET responses. Reduces backend API calls and egress by serving repeated reads from memory. Per-instance only — not shared across instances.

cache:
  enabled: true
  max_size: "256MB"            # total cache capacity (default: 256MB)
  max_object_size: "10MB"      # largest cacheable object (default: 10MB)
  ttl: "5m"                    # per-entry time-to-live (default: 5m)

• max_size — total memory the cache may consume. Size this based on available container memory after accounting for the Go heap, connection pools, and streaming buffers. A good starting point is 10-25% of the container’s memory allocation.
• max_object_size — objects larger than this are never admitted to the cache. Prevents a single large object from evicting many smaller frequently-accessed objects. Set this below the typical “hot” object size in your workload.
• ttl — maximum time an entry stays cached before automatic expiry. In multi-instance deployments, this bounds how stale a cached object can be when writes happen on another instance. Lower values reduce staleness at the cost of more backend requests.

Cache entries are automatically invalidated on PutObject, DeleteObject, CopyObject, DeleteObjects, and CompleteMultipartUpload. Range requests bypass the cache on miss but are served from cache on hit.

When to enable:

• Read-heavy workloads where the same objects are fetched repeatedly (thumbnails, config files, assets)
• Backends with per-request API charges or egress costs
• High-latency backends where caching improves P99 latency

When to skip:

• Write-heavy workloads with few repeated reads
• Objects are too large to fit meaningfully in memory
• Single-instance with very low read traffic

The cache is not hot-reloadable — changing cache settings requires a restart. When encryption is enabled, the cache stores post-decryption plaintext.

Metrics:

When enabled, the dashboard is served at {path}/ on the same port as the S3 API.

All dashboard responses include security headers (X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin, Content-Security-Policy). The dashboard requires authentication via the configured admin_key/admin_secret — unauthenticated requests are redirected to the login page (HTML) or receive 401 (API).

Multi-Backend Configurations

Single backend with quota

The simplest setup. One backend with a byte cap:

backends:
  - name: "oci"
    endpoint: "https://namespace.compat.objectstorage.region.oraclecloud.com"
    region: "us-phoenix-1"
    bucket: "my-bucket"
    access_key_id: "${OCI_KEY}"
    secret_access_key: "${OCI_SECRET}"
    force_path_style: true
    quota_bytes: 21474836480     # 20 GB

Multiple backends with quotas (pack routing)

Stack multiple free-tier allocations. With the default routing_strategy: "pack", when one backend fills up, writes overflow to the next. Use routing_strategy: "spread" instead to distribute objects evenly by utilization ratio:

backends:
  - name: "oci-free"
    endpoint: "https://namespace.compat.objectstorage.region.oraclecloud.com"
    region: "us-phoenix-1"
    bucket: "free-tier-bucket"
    access_key_id: "${OCI_KEY}"
    secret_access_key: "${OCI_SECRET}"
    force_path_style: true
    quota_bytes: 21474836480     # 20 GB (OCI free tier)

  - name: "b2-free"
    endpoint: "https://s3.us-west-002.backblazeb2.com"
    region: "us-west-002"
    bucket: "free-tier-bucket"
    access_key_id: "${B2_KEY}"
    secret_access_key: "${B2_SECRET}"
    force_path_style: true
    quota_bytes: 10737418240     # 10 GB (B2 free tier)

This gives you 30 GB of combined storage across two providers.

Multiple backends without quotas (requires replication or spread)

When all backends are unlimited and using the default pack routing, only the first backend would receive writes. To distribute data, either set replication.factor >= 2 to replicate across backends, or use routing_strategy: "spread" to distribute writes by utilization ratio.

backends:
  - name: "oci"
    endpoint: "https://namespace.compat.objectstorage.region.oraclecloud.com"
    region: "us-phoenix-1"
    bucket: "bucket-a"
    access_key_id: "${OCI_KEY}"
    secret_access_key: "${OCI_SECRET}"
    force_path_style: true
    # no quota_bytes — unlimited

  - name: "aws"
    endpoint: "https://s3.us-east-1.amazonaws.com"
    region: "us-east-1"
    bucket: "bucket-b"
    access_key_id: "${AWS_KEY}"
    secret_access_key: "${AWS_SECRET}"
    # no quota_bytes — unlimited

replication:
  factor: 2

Validation rule: You cannot mix unlimited and quota-limited backends. Either all backends have quota_bytes set (quota routing) or all are unlimited (replication or spread routing required).

Onboarding a New Tenant

To add a new application to the orchestrator:

1. Generate credentials for the new tenant:

   echo "Access Key: $(openssl rand -hex 10 | tr '[:lower:]' '[:upper:]')"
   echo "Secret Key: $(openssl rand -base64 30)"

2. Add a bucket entry to your config:

   buckets:
     # ... existing buckets ...
     - name: "new-app"
       credentials:
         - access_key_id: "${NEW_APP_ACCESS_KEY}"
           secret_access_key: "${NEW_APP_SECRET_KEY}"

3. Reload the configuration by sending SIGHUP (kill -HUP $(pidof s3-orchestrator)) — or restart the orchestrator.

4. Hand the client four pieces of information:

   • Endpoint URL (e.g., http://s3-orchestrator.service.consul:9000)
   • Bucket name (e.g., new-app)
   • Access Key ID
   • Secret Access Key

5. Point them to the User Guide for client setup instructions.

CLI Subcommands

version

Prints the binary version, Go version, and platform:

s3-orchestrator version
# s3-orchestrator v0.41.7 go1.26.0 linux/amd64

validate

Validates a configuration file without starting the server. Exits 0 on success with a brief summary, or exits 1 with error details. Useful for CI pipelines or pre-deploy checks:

s3-orchestrator validate -config config.yaml

admin

Operational CLI for inspecting and controlling a running instance. Reads config.yaml to discover the server address and admin token (ui.admin_token, falling back to ui.admin_key), then makes HTTP requests to the admin API.

s3-orchestrator admin [flags] <command>

Flags:

Commands:

# Show backend health, usage, and circuit breaker state
s3-orchestrator admin status

# List all copies of an object across backends
s3-orchestrator admin object-locations -key "my-bucket/path/to/file.txt"

# Show cleanup queue depth and pending items
s3-orchestrator admin cleanup-queue

# Force flush usage counters to the database
s3-orchestrator admin usage-flush

# Drop every entry from the in-memory object data cache
# (returns 503 when caching is disabled in config)
s3-orchestrator admin cache-flush

# Inspect cache size and entry count
s3-orchestrator admin cache-stats

# Drop a single key from the cache
s3-orchestrator admin cache-invalidate -key bucket/path/object.txt

# Drop every cached key under a prefix
s3-orchestrator admin cache-invalidate-prefix -prefix bucket/path/

# Trigger one replication cycle (creates missing replicas)
s3-orchestrator admin replicate

# Show count of over-replicated objects
s3-orchestrator admin over-replication

# Clean over-replicated objects (remove excess copies)
s3-orchestrator admin over-replication --execute

# Clean with a custom batch size
s3-orchestrator admin over-replication --execute --batch-size 200

# View the current log level
s3-orchestrator admin log-level

# Change log level at runtime (no restart or SIGHUP needed)
s3-orchestrator admin log-level -set debug

# Start draining a backend (migrates all objects to other backends)
s3-orchestrator admin drain <backend-name>

# Check drain progress
s3-orchestrator admin drain-status <backend-name>

# Cancel an active drain (objects already moved are not rolled back)
s3-orchestrator admin drain-cancel <backend-name>

# Remove a backend's database records (S3 objects preserved, reversible via sync)
s3-orchestrator admin remove-backend <backend-name>

# Preview what --purge would destroy (dry-run)
s3-orchestrator admin remove-backend <backend-name> --purge

# Remove a backend AND delete its S3 objects (requires --confirm)
s3-orchestrator admin remove-backend <backend-name> --purge --confirm

# Encrypt all unencrypted objects in-place (requires encryption enabled)
s3-orchestrator admin encrypt-existing

# Decrypt all encrypted objects back to plaintext (requires encryption enabled for key access)
s3-orchestrator admin decrypt-existing

# Re-wrap all DEKs encrypted with a specific key ID (key rotation)
s3-orchestrator admin rotate-encryption-key --old-key-id config-0

# Trigger an on-demand integrity scrub cycle (verify stored hashes)
s3-orchestrator admin scrub

# Scrub with a custom batch size
s3-orchestrator admin scrub -batch-size 500

# Compute and store content hashes for all unhashed objects
s3-orchestrator admin backfill-checksums

# Backfill with a custom batch size (controls pace of backend reads)
s3-orchestrator admin backfill-checksums -batch-size 50

# Reconcile all backends (import untracked objects, remove stale DB entries)
s3-orchestrator admin reconcile

# Reconcile a single backend
s3-orchestrator admin reconcile -backend g3

The admin API requires ui.admin_token (or ui.admin_key as fallback) to be set in the configuration. All requests are authenticated via the X-Admin-Token header.

Importing Existing Data

The sync subcommand imports objects from an existing backend bucket into the orchestrator’s metadata database. Use this when bringing a bucket that already has data under orchestrator management.

Dry run first

Always preview what would be imported before committing:

s3-orchestrator sync \
  --config config.yaml \
  --backend oci \
  --bucket my-files \
  --dry-run

Run the import

s3-orchestrator sync \
  --config config.yaml \
  --backend oci \
  --bucket my-files

The --bucket flag specifies which virtual bucket the imported objects belong to. Keys are stored internally as {bucket}/{key}, so this determines the namespace.

Partial import with –prefix

Import only objects under a specific key prefix:

s3-orchestrator sync \
  --config config.yaml \
  --backend oci \
  --bucket my-files \
  --prefix "photos/"

Objects already tracked in the database for that backend are automatically skipped. The command logs per-page progress and a final summary.

Sync flags

Monitoring

Web dashboard

When ui.enabled is true, the dashboard at {path}/ shows a live snapshot of:

• Storage summary — total bytes used/capacity across all backends
• Backend quota — bytes used/limit with progress bars per backend, object counts, active multipart uploads
• Monthly usage — API requests, egress, and ingress per backend with limits
• Object tree — interactive collapsible file browser. Buckets and directories are collapsed by default; click to expand. Each directory shows a rollup file count and total size.
• Configuration — virtual bucket names, write routing strategy, replication factor, rebalance status, rate limiting, encryption status
• Logs — recent structured log output from an in-memory ring buffer (last 5,000 entries). Filter by severity level and search by text. Logs are available immediately on page load — no need to SSH into the host.

The dashboard also provides management actions:

• Upload — upload files to any virtual bucket directly from the browser (up to 512 MiB per file)
• Download — download individual objects by clicking the download icon on any file in the tree
• Delete — delete individual objects by clicking the delete icon on any file in the tree
• Rebalance — trigger an on-demand rebalance using the configured strategy and settings
• Clean Excess — remove over-replicated copies that exceed the replication factor
• Sync — import pre-existing objects from a backend’s S3 bucket into the proxy database. Select a backend and a virtual bucket — objects already in the database are skipped, and objects belonging to other virtual buckets are excluded.

On-demand reconciliation

When a backend loses data (expired credentials, provider outage, accidental deletion), the metadata database retains stale entries that cause log noise in the rebalancer, replicator, and scrubber. The reconcile endpoint walks both the backend (paginated ListObjects) and the metadata DB (paginated cursor over object_locations ordered by key) as ascending key streams, then merges them in lockstep. Memory is bounded by a 1000-entry page size on each side regardless of total object count, so a backend holding millions of objects reconciles without OOM. Keys belonging to sibling virtual buckets stored on the same backend are skipped — each virtual bucket reconciles in its own pass.

# Reconcile all backends
curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/reconcile

# Reconcile a single backend
curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/reconcile?backend=g3

Response:

{"status":"ok","imported":0,"removed":57,"backends_scanned":1}

• imported: objects on the backend but not in the DB (brought under management)
• removed: DB entries whose objects no longer exist on the backend (stale metadata cleaned up)
• backends_scanned: how many backends were processed

The background reconciler (reconcile.enabled: true) runs the same logic on a timer. The admin endpoint is for immediate use after incidents.

The dashboard requires authentication. Users log in at {path}/login with the admin_key and admin_secret configured in the ui section. Sessions last 24 hours.

The dashboard is server-rendered HTML. The object tree uses JavaScript for lazy-loaded directory expansion — directories fetch their children on click via the /ui/api/tree endpoint.

JSON endpoints at {path}/api/dashboard, {path}/api/tree, and {path}/api/logs return data for programmatic access or integration with other tools. The logs endpoint accepts optional query parameters: level (minimum severity: DEBUG, INFO, WARN, ERROR), since (RFC3339 timestamp), component, and limit. Management endpoints ({path}/api/delete, {path}/api/upload, {path}/api/rebalance, {path}/api/clean-excess, {path}/api/sync) accept POST requests. The download endpoint ({path}/api/download?key=...) accepts GET requests. All API endpoints require authentication.

Health endpoints

Two health endpoints serve different purposes:

Liveness (/health) — always returns HTTP 200. Use this for liveness probes (Consul checks, K8s livenessProbe). The service stays in rotation during temporary database outages.

curl http://localhost:9000/health
# {"status":"ok"}
# or {"status":"degraded"} when the database circuit breaker is open

Readiness (/health/ready) — returns HTTP 200 when the service is ready to handle traffic, HTTP 503 during startup (before migrations and backend initialization complete) and during shutdown drain. Use this for readiness probes (K8s readinessProbe, Nomad on_update = "require_healthy").

curl http://localhost:9000/health/ready
# {"status":"ready"}      — 200
# {"status":"not ready"}  — 503

The HTTP response body is intentionally minimal — only the status field is returned, so log aggregators can grep on a fixed pattern. To identify which instance answered a probe in a multi-instance deployment, query GET /admin/api/workers (which includes the instance identifier) or correlate by source IP.

Background worker health

/health only reflects database breaker state. Background services (replicator, cleanup queue, lifecycle, pending reaper, …) are supervised by the lifecycle manager and recover on their own, but a service that is running yet failing every tick looks identical to a healthy one in /health.

GET /admin/api/workers returns a JSON snapshot of every registered background service’s last-tick health, including last_success, last_failure, last_error, and consecutive_failures. Use it during incidents to distinguish:

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/workers

{
  "workers": [
    {"name": "cleanup_queue", "last_success": "2026-05-12T18:42:01Z", "consecutive_failures": 0},
    {"name": "replicator", "last_failure": "2026-05-12T18:45:14Z", "last_error": "connection refused", "consecutive_failures": 3}
  ]
}

Workers in proxy-only deployments return 503 from this endpoint because no worker pool is registered.

The same data flows into Prometheus as s3o_worker_last_success_timestamp_seconds, s3o_worker_consecutive_failures, and s3o_worker_ticks_total{result} so alerting can run without scraping the admin endpoint. Suggested alert shapes are in the metrics table below.

Grafana dashboard

A comprehensive Grafana dashboard is included at grafana/s3-orchestrator.json. Import it via Grafana’s UI (Dashboards → Import → Upload JSON file) or provision it from disk. It expects a Prometheus datasource with UID prometheus.

The dashboard covers all emitted metrics, organised by domain into rows: overview, quota & storage, request performance, backend operations, manager operations, circuit breaker & degraded mode, replication, usage tracking, rate limiting & rejections, rebalancer, drain & lifecycle, cleanup queue & audit, encryption, object data cache, integrity verification, Redis, over-replication cleanup, pending PUT intents, and authentication (streaming SigV4). Rows for less frequently inspected domains are collapsed by default.

Key Prometheus metrics

If telemetry.metrics.enabled is true, metrics are exposed at /metrics. Two listener modes:

• Inline (telemetry.metrics.listen empty, the default): /metrics is served on the same listener as the public S3 API. Convenient for single-port deployments and the docker-compose / local-dev workflow. In this mode /debug/pprof/* endpoints are not registered — mounting pprof on the public S3 listener would leak runtime internals (de-anonymized stack frames via /debug/pprof/heap, the command line and flag values via /debug/pprof/cmdline) and offer a DoS amplifier (/debug/pprof/profile?seconds=300 triggers minutes of CPU profiling on demand).
• Dedicated listener (telemetry.metrics.listen set, e.g. 127.0.0.1:9001): /metrics and /debug/pprof/* are both mounted on the dedicated listener. Bind to 127.0.0.1 or a private network interface so the surface is only reachable from operators inside the trust boundary. The nomad demo uses 0.0.0.0:9001 so the docker-compose Prometheus container can scrape via the bridge gateway; production deployments should tighten this to 127.0.0.1:9001 or a private network address.

Once the dedicated listener is configured, captures look like:

# 60-second allocation profile during a load spike
curl -o /tmp/allocs.pb.gz "http://127.0.0.1:9001/debug/pprof/allocs?seconds=60"
go tool pprof -top -cum -alloc_space /tmp/allocs.pb.gz

# Instantaneous live-heap snapshot
curl -o /tmp/heap.pb.gz "http://127.0.0.1:9001/debug/pprof/heap"
go tool pprof -top -cum -inuse_space /tmp/heap.pb.gz

The standard net/http/pprof endpoints are available: /debug/pprof/, /debug/pprof/profile, /debug/pprof/heap, /debug/pprof/allocs, /debug/pprof/goroutine, /debug/pprof/block, /debug/pprof/mutex, /debug/pprof/trace, /debug/pprof/cmdline, /debug/pprof/symbol.

Key metrics to alert on:

Structured logs

All logs are JSON to stdout. Key fields: msg, level, error, backend, operation.

Audit logs are a subset of structured logs with "audit": true. Every S3 API request and significant internal operation emits an audit entry with a request_id for correlation. Filter audit entries in your log pipeline with a JSON query on the audit field.

Key audit events:

Each S3 API request produces two correlated audit entries (HTTP-level and storage-level) sharing the same request_id. Internal operations (rebalance, replication) generate their own correlation IDs. The request_id also appears as a s3o.request_id attribute on OpenTelemetry spans.

HTTP panic recovery

A panic inside an HTTP handler is caught by the panic-recovery middleware applied to the S3 and admin route groups. The recovery contract:

• The client gets a structured response, not a connection reset. S3 routes receive an XML <Error><Code>InternalError</Code><Message>...Request ID: ...</Message></Error> body with HTTP 500; admin routes receive the same shape as a JSON {"error": "...Request ID: ..."}.
• The Prometheus counter s3o_http_panic_recovered_total{route} increments. A non-zero value is an immediate alert candidate.
• A slog.ErrorContext line is written at level ERROR with component=httputil, the route, the method and path, the panic value, the captured Go stack, and a trace_id / span_id if a span was active when the panic occurred.
• An http.PanicRecovered audit entry is emitted with the same correlation request_id as the failing request.
• If an OpenTelemetry span was active on the request, it is marked as failed via SetStatus(Error) + RecordError so traces in Tempo highlight the failure.

UI routes are intentionally not wrapped in the first iteration (mounted on the same mux as the S3 catch-all; the bulk of panic risk is on S3 and admin anyway). The recovery message deliberately does not echo the panic value to the client; only the request ID is returned so support tickets can be correlated back to the orchestrator log line.

Clients can supply their own correlation ID via the X-Request-Id request header; otherwise the orchestrator generates one. The ID is returned in the X-Amz-Request-Id response header.

Trace-to-log correlation — JSON log output includes trace_id and span_id fields on every line emitted within an active OpenTelemetry span. Log aggregators like Grafana Loki can extract these fields to link directly from a log entry to the corresponding trace in Tempo, and vice versa.

Admin API Reference

All /admin/api/* endpoints require the X-Admin-Token header. Set the token via the S3O_ADMIN_TOKEN environment variable or the admin.token config key. All request and response bodies are JSON; request bodies are capped at 1 MiB.

Common error responses are documented under each section; for any unexpected server-side failure the response is {"error": "<short message>"} with HTTP 500 and the original error is logged with the request_id.

Health & status

GET /admin/api/status

Backend health snapshot: per-backend bytes used, bytes limit, object count, API requests, egress, ingress, plus the database circuit-breaker state.

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/status

Response (200):

{
  "db_healthy": true,
  "backends": [
    {
      "name": "oci",
      "bytes_used": 12345678,
      "bytes_limit": 10737418240,
      "object_count": 4231,
      "api_requests": 18234,
      "egress_bytes": 92340876,
      "ingress_bytes": 12345678
    }
  ],
  "usage_period": "2026-05"
}

GET /admin/api/reload-status

Most recent config-reload result. Returns {"status": "no_reload_yet"} on a freshly started instance that has not yet had a SIGHUP. After a reload, returns the structured result (generation, summary of diffs, validation outcome, errors).

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/reload-status

GET /admin/api/workers

JSON snapshot of every registered background service’s last-tick state. See Background worker health above for the response schema and full operator workflow. Returns 503 when the lifecycle manager is not wired (proxy-only deployments).

GET /admin/api/log-level · PUT /admin/api/log-level

Inspect or change the runtime log level without restarting. Accepted levels: debug, info, warn, error.

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/log-level
# {"level":"info"}

curl -X PUT -H "X-Admin-Token: $TOKEN" -H "Content-Type: application/json" \
  -d '{"level":"debug"}' \
  http://localhost:9000/admin/api/log-level
# {"level":"debug"}

The change is logged via an INFO audit entry. Useful during incidents — flip to debug for a minute, capture the failure mode, then flip back.

Object inspection

GET /admin/api/object-locations?key=<key>

Returns all backend copies of a single object key, with backend name, size, and encryption metadata. Useful for debugging “where did this object actually land?” and for verifying that the replication factor is satisfied.

curl -H "X-Admin-Token: $TOKEN" "http://localhost:9000/admin/api/object-locations?key=reports/2026-05.pdf"

400 when key is missing.

Cleanup & quota

GET /admin/api/cleanup-queue

Current cleanup queue depth plus a sample of up to 50 pending rows. The web dashboard’s cleanup panel uses this endpoint.

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/cleanup-queue

Response includes depth (total pending count) and items (sample with id, backend, key, reason, attempts, next_retry).

POST /admin/api/usage-flush

Forces an out-of-band flush of the in-memory usage counters to the database. Use after a manual quota adjustment to make the new value visible immediately instead of waiting up to usage_flush.interval seconds. Same effect as the scheduled flush; the operation is idempotent.

curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/usage-flush
# {"status":"flushed"}

Replication & over-replication

POST /admin/api/replicate

Triggers a single replication cycle synchronously. Useful after a backend recovery to fast-forward repair without waiting for the next scheduled tick.

curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/replicate
# {"status":"ok","copies_created":42}
# or {"status":"skipped","copies_created":0,"reason":"replication factor is 1"}

GET /admin/api/over-replication · POST /admin/api/over-replication[?batch_size=N]

The GET form reports how many object keys currently have more copies than the configured replication.factor requires. The POST form runs one cleanup pass that removes surplus replicas (preferring drained, circuit-broken, and most-utilised backends — see the OverReplicationCleaner section in the Background Services diagram).

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/over-replication
# {"factor":3,"pending":127}

curl -X POST -H "X-Admin-Token: $TOKEN" "http://localhost:9000/admin/api/over-replication?batch_size=500"
# {"status":"ok","copies_removed":127}

batch_size defaults to the configured value; the POST form clamps user input to 10000.

Drain & backend lifecycle

POST /admin/api/backends/{name}/drain

Begins draining {name}. Subsequent writes filter the backend out of eligibility (drain race re-check in objects_write.go covers the in-flight case), and the drain worker starts migrating existing objects to other healthy backends.

curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/backends/oci-old/drain
# {"status":"drain started","backend":"oci-old"} (202)

Returns 400 if the backend is unknown or already draining.

GET /admin/api/backends/{name}/drain

Returns the drain’s current state: total objects/bytes moved, total remaining, started_at, last_progress_at, and any per-batch errors. Poll this during a drain to track progress.

DELETE /admin/api/backends/{name}/drain

Cancels the active drain. In-flight migrations finish; new migrations stop. The backend’s eligibility filter is re-enabled. Useful when a drain was triggered by accident or when the intended target’s free space turns out to be insufficient.

DELETE /admin/api/backends/{name}[?purge=true&confirm=<token>]

Non-purge form (default): drops the backend’s DB records (object_locations, quota, usage). The backend’s actual S3 objects are left in place — reversible by re-adding the backend and running s3-orchestrator admin sync. Use this when retiring a backend whose data has already been migrated.

curl -X DELETE -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/backends/oci-old
# {"status":"backend removed","backend":"oci-old"}

Purge form (two-phase, irreversible): also deletes every S3 object on the backend.

# Phase 1: preview
curl -X DELETE -H "X-Admin-Token: $TOKEN" "http://localhost:9000/admin/api/backends/oci-old?purge=true"
# {"status":"confirmation required","backend":"oci-old","object_count":4231,"total_bytes":92340876,"confirm_token":"<token>"}

# Phase 2: confirm using the returned token within 5 minutes
curl -X DELETE -H "X-Admin-Token: $TOKEN" "http://localhost:9000/admin/api/backends/oci-old?purge=true&confirm=<token>"
# {"status":"backend purged","backend":"oci-old"}

The confirmation token is HMAC-bound to the backend name and expires after 5 minutes, so the preview cannot be replayed against a different backend or after a typo.

Encryption operations

POST /admin/api/rotate-encryption-key

Re-wraps every DEK still wrapped with the named old key using the current primary master key. Required after a key rotation (see Rotating encryption keys below).

curl -X POST -H "X-Admin-Token: $TOKEN" -H "Content-Type: application/json" \
  -d '{"old_key_id":"config-0"}' \
  http://localhost:9000/admin/api/rotate-encryption-key
# {"status":"complete","rotated":1423,"failed":0,"total":1423}

old_key_id formats: inline keys → config-0 (primary), config-1, …; file-based → file-0; Vault Transit → the key name. Returns 400 if encryption is not enabled or the body is malformed.

POST /admin/api/encrypt-existing

Bulk-encrypts every object that does not yet have encryption metadata. The orchestrator downloads each object, encrypts it, re-uploads the ciphertext, and updates the DB record. Idempotent — re-running only processes objects that are still plaintext.

curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/encrypt-existing
# {"status":"complete","encrypted":1423,"failed":0,"total":1423}

Counts against backend API quotas (one GET + one PUT per object).

POST /admin/api/decrypt-existing

Inverse of the above: downloads every encrypted object, decrypts it, re-uploads plaintext, and clears the encryption metadata. The orchestrator must still be configured with the key provider (the DEKs need to be unwrapped). Use only when permanently removing encryption from a deployment.

curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/decrypt-existing
# {"status":"complete","decrypted":1423,"failed":0,"total":1423}

Integrity

POST /admin/api/scrub[?batch_size=N]

Runs one integrity scrub pass: reads random objects from each backend, computes the SHA-256, and compares to the stored content_hash. On mismatch the bad copy is enqueued for cleanup with reason integrity_scrub_failed. The scrubber’s scheduled runs are independent — this endpoint just triggers an immediate pass.

curl -X POST -H "X-Admin-Token: $TOKEN" "http://localhost:9000/admin/api/scrub?batch_size=200"
# {"status":"ok","checked":200,"failed":0}
# or {"status":"skipped","reason":"integrity verification not enabled"}

POST /admin/api/backfill-checksums[?batch_size=N]

Computes and stores content_hash for every object that does not have one yet. Use after enabling integrity.enabled: true on an existing deployment so the scrubber has hashes to compare against. Paginates internally until done.

curl -X POST -H "X-Admin-Token: $TOKEN" "http://localhost:9000/admin/api/backfill-checksums?batch_size=500"
# {"status":"ok","processed":12345}
# or {"status":"skipped","reason":"integrity verification is not enabled"}

Reconciliation

POST /admin/api/reconcile[?backend=<name>]

On-demand reconciler pass. See On-demand reconciliation above for the operator workflow and the bounded-memory sorted-merge details.

Cache management

The object-data cache is optional (cache.enabled: true). All cache endpoints return 503 {"status":"disabled","reason":"object data cache is not enabled"} when the cache is not configured, so callers can distinguish “no cache” from “cache empty after flush”.

GET /admin/api/cache

Current utilization: entry count, bytes used, configured max. Mirrors the s3o_cache_* gauges so operators without Prometheus access can still inspect cache state.

curl -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/cache
# {"entries":2341,"size_bytes":83886080,"max_bytes":268435456}

POST /admin/api/cache/flush

Drops every entry from the cache. Used by load-test tooling to characterise cache-cold GET performance. The response carries entries_cleared so the caller can confirm the cache was actually populated before the flush.

curl -X POST -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/cache/flush
# {"status":"flushed","entries_cleared":2341}

DELETE /admin/api/cache/keys/{key...}

Drops a single object from the cache. The {key...} wildcard pattern accepts keys with embedded slashes. Always returns 200 — invalidating an unknown key is a no-op, matching the cache’s own contract.

curl -X DELETE -H "X-Admin-Token: $TOKEN" http://localhost:9000/admin/api/cache/keys/reports/2026-05.pdf

DELETE /admin/api/cache/prefix?prefix=<prefix>

Drops every cached key under the given prefix. Useful after a bulk update for one tenant or directory. Returns 400 if prefix is empty — use /admin/api/cache/flush to clear the whole cache.

curl -X DELETE -H "X-Admin-Token: $TOKEN" "http://localhost:9000/admin/api/cache/prefix?prefix=reports/"

Common Operations

Reloading configuration

Many settings can be updated without restarting the orchestrator by sending SIGHUP:

1. Edit the config file with your changes.
2. Send SIGHUP to the running process:

   kill -HUP $(pidof s3-orchestrator)

3. Check the logs to confirm the reload succeeded:

   {"level":"INFO","msg":"SIGHUP received, reloading configuration","path":"config.yaml"}
   {"level":"INFO","msg":"Configuration reload complete"}

What takes effect immediately:

• Log level (server.log_level) — can also be changed at runtime via s3-orchestrator admin log-level -set debug
• Bucket credentials (add/remove/rotate credentials without downtime)
• Rate limit settings (requests per second, burst)
• Backend quota limits (quota_bytes)
• Backend usage limits (api_request_limit, egress_byte_limit, ingress_byte_limit)
• Rebalance settings (strategy, interval, batch size, threshold, enable/disable)
• Replication settings (factor, worker interval, batch size, unhealthy threshold)
• Usage flush settings (interval, adaptive enabled/threshold/fast interval)

What requires a restart:

• server.listen_addr, server timeouts, server.shutdown_delay, database, telemetry, ui, routing_strategy, encryption, redis
• Backend structural changes (endpoint, S3 credentials, adding/removing backends)

If any of these fields change, the reload still proceeds for the reloadable settings, and warnings are logged:

{"level":"WARN","msg":"Config field changed but requires restart to take effect","field":"server.listen_addr"}

If the config file is invalid, the orchestrator keeps the current configuration entirely and logs the parse/validation error:

{"level":"ERROR","msg":"Config reload failed, keeping current config","error":"invalid config: ..."}

No partial reload happens — either all reloadable settings update, or none do.

Adding a new backend

Add the backend to the backends list in your config and restart the orchestrator. Backend count changes are not reloadable — a restart is required. Quota limits are synced to the database on startup.

Draining a backend

Draining migrates all objects off a backend to other backends without data loss. Use this when decommissioning a backend but preserving all stored objects.

1. Start the drain:

   s3-orchestrator admin drain <backend-name>

   This immediately excludes the backend from new writes (PutObject and CreateMultipartUpload skip it) and begins migrating objects in batches of 100. Any in-progress multipart uploads on the backend are aborted first.

2. Monitor progress:

   s3-orchestrator admin drain-status <backend-name>

   Returns objects remaining, bytes remaining, objects moved so far, and whether the drain is still active. Poll this periodically until active is false and objects_remaining is 0.

3. Wait for completion. The drain runs as a background goroutine. Each object is read from the source backend, written to the least-utilized eligible backend, and the database record is atomically swapped via compare-and-swap. Failed moves are logged but don’t stop the drain.

4. Remove the backend from config and restart:

   # Edit config.yaml — remove the backend entry
   # Restart or redeploy the orchestrator

   After drain completes, DeleteBackendData cleans up remaining database records (usage, quota, cleanup queue) automatically. Removing the backend from config on restart prevents it from being re-initialized.

Cancelling a drain:

s3-orchestrator admin drain-cancel <backend-name>

Objects already moved are not rolled back. The backend becomes eligible for new writes again.

Metrics to watch during drain:

Removing a backend

Removing deletes all database records for a backend. This is destructive — objects on that backend become inaccessible. Use drain first if you want to preserve data.

Drop database records only (objects remain on the backend’s S3 storage):

s3-orchestrator admin remove-backend <backend-name>

Preview what purge would destroy (dry-run):

s3-orchestrator admin remove-backend <backend-name> --purge

Drop database records AND delete S3 objects (requires confirmation):

s3-orchestrator admin remove-backend <backend-name> --purge --confirm

The --purge flag without --confirm shows a preview of what would be destroyed (object count, total bytes) and exits. With --confirm, the CLI obtains a signed confirmation token from the server (valid for 60 seconds) and executes the purge. Individual delete failures are logged but don’t stop the operation.

After removing, edit the config to remove the backend entry and restart.

Note: You cannot remove a backend that is currently draining. Cancel the drain first with drain-cancel.

Important: update the config after drain or remove

Drain and remove state is held in memory only — it is not persisted to the database. This means:

• If the service restarts with a drained/removed backend still in the config, SyncQuotaLimits re-creates the backend’s quota record and the backend is re-initialized as a fresh, empty backend eligible for new writes. No data is lost, but the decommissioned backend silently starts receiving traffic again.
• If the service crashes during an active drain, all drain progress is lost. The backend reverts to active on restart. You would need to restart the drain.
• SIGHUP does not remove backends — config reload only updates quota limits and usage limits. The in-memory backend map is set at startup and cannot be modified at runtime.

Always remove the backend from the config file and restart (or redeploy) after a drain or remove operation completes. The dashboard UI shows a pulsing “Draining” badge on backends with an active drain so you can monitor progress visually.

Adjusting quotas

Change quota_bytes in the config and send SIGHUP. Quota limits are synced to the database on reload. Alternatively, restart the orchestrator — SyncQuotaLimits also runs on startup.

Enabling replication after initial setup

Add a replication section with factor > 1 and send SIGHUP (or restart). When restarting, the replication worker runs immediately at startup to begin creating copies of existing objects, then continues at the configured interval. With SIGHUP, the new factor takes effect on the next worker tick.

Remember: the replication factor cannot exceed the number of backends.

Enabling encryption on existing data

If you enable encryption on an orchestrator that already has unencrypted objects, those objects remain unencrypted until you explicitly encrypt them. New objects are encrypted automatically; existing ones need the encrypt-existing admin API.

1. Enable encryption in the config and restart the orchestrator.

2. Encrypt existing objects:

   s3-orchestrator admin encrypt-existing

   This processes all unencrypted objects in batches of 100: downloads from the backend, encrypts, re-uploads the ciphertext (overwriting the plaintext), and updates the database record. The response shows progress:

   {"status": "complete", "encrypted": 1423, "failed": 0, "total": 1423}

3. Monitor via the s3o_encrypt_existing_objects_total metric (labels: success, error).

Failed objects are logged individually and can be retried by calling encrypt-existing again — it only processes objects without encryption metadata.

Disabling encryption / decrypting existing data

To remove encryption from all objects and restore plaintext on backends, use the decrypt-existing admin API. Encryption must still be configured when you run this (the orchestrator needs the key provider to unwrap DEKs). Disable encryption in the config after decryption completes.

1. Decrypt all encrypted objects:

   s3-orchestrator admin decrypt-existing

   This processes all encrypted objects in batches of 100: downloads ciphertext from the backend, decrypts, re-uploads plaintext (overwriting the ciphertext), and clears encryption metadata in the database. Each object costs 2 API calls (one GET, one PUT) plus egress and ingress against the backend’s usage quota. The response shows progress:

   {"status": "complete", "decrypted": 1423, "failed": 0, "total": 1423}

2. Disable encryption in the config and restart the orchestrator.

3. Monitor via the s3o_decrypt_existing_objects_total metric (labels: success, error).

Failed objects are logged individually and can be retried by calling decrypt-existing again — it only processes objects with encryption metadata.

Both encrypt-existing and decrypt-existing keep backend_quotas.bytes_used consistent with the on-disk byte count: each object is rewritten at a different size (encryption inflates by per-chunk overhead, decryption removes it), and the per-backend counter advances by the size delta inside the same transaction as the metadata update. No manual reconciliation against SUM(object_locations.size_bytes) is needed after a run.

Rotating encryption keys

Key rotation re-wraps DEKs with a new master key without re-encrypting object data. This is a metadata-only operation and is fast regardless of object sizes.

1. Generate a new master key:

   openssl rand -base64 32

2. Update the config — set the new key as master_key and move the old key to previous_keys:

   encryption:
     enabled: true
     master_key: "${NEW_ENCRYPTION_KEY}"
     previous_keys:
       - "${OLD_ENCRYPTION_KEY}"

3. Restart the orchestrator (encryption config is not reloadable).

4. Re-wrap all DEKs:

   s3-orchestrator admin rotate-encryption-key --old-key-id config-0

   The old-key-id identifies which key’s DEKs to re-wrap. For inline config keys, the ID is config-0 for the primary and config-1, config-2, etc. for previous keys in order. For file-based keys, the ID is file-0. For Vault Transit, it’s the key name.

   The response shows progress:

   {"status": "complete", "rotated": 1423, "failed": 0, "total": 1423}

5. After all DEKs are re-wrapped, you can optionally remove the old key from previous_keys and restart. Objects that were rotated no longer need the old key.

Metrics to watch during rotation:

Rotating client credentials

Update the credentials in the bucket config and send SIGHUP. The new credentials take effect immediately and old credentials stop working. Coordinate with the tenant to update their client configuration at the same time.

Example: rotating credentials without downtime

To perform a zero-downtime credential rotation, temporarily add both old and new credentials:

1. Add the new credential alongside the old one:

   buckets:
     - name: "app1-files"
       credentials:
         - access_key_id: "OLD_KEY"
           secret_access_key: "old_secret"
         - access_key_id: "NEW_KEY"
           secret_access_key: "new_secret"

2. Send SIGHUP — both credentials now work.
3. Update the client to use the new credentials.
4. Remove the old credential from the config and send SIGHUP again.

Deployment

Nomad and Kubernetes

Production-ready manifests for both platforms are in deploy/. Each includes a local demo script that stands up a complete environment in one command:

make kubernetes-demo   # k3d cluster with docker-compose backing services
make nomad-demo        # Nomad dev agent with docker-compose backing services

See deploy/README.md for production deployment instructions, Vault integration, TLS/mTLS configuration, and Ingress setup.

Multi-instance deployment

By default, every instance runs both the HTTP API and all background workers (--mode=all). For larger deployments, the --mode flag separates these roles:

# API instance — serves S3 requests, no background workers
s3-orchestrator -config config.yaml -mode api

# Worker instance — runs rebalancer, replicator, cleanup, lifecycle
s3-orchestrator -config config.yaml -mode worker

How it works:

• API instances serve S3 requests, the web UI, and rate limiting. They run the usage-flush service to avoid losing counters on restart, but skip all advisory-locked background tasks.
• Worker instances run all background services and expose /health, /health/ready, and /metrics for monitoring, but don’t serve S3 traffic or the web UI.
• Background tasks that modify state (rebalancer, replicator, cleanup, lifecycle, multipart cleanup) use PostgreSQL advisory locks — only one instance cluster-wide executes each task per cycle. Running multiple worker instances is safe; extra instances simply skip cycles when the lock is held. The circuit breaker watchdog runs on all instances without a lock (it operates on per-instance circuit state).

Recommended topology: N api instances behind a load balancer + 1–2 worker instances for redundancy. All instances share the same config file and PostgreSQL database.

Docker

# Local build
make build

# Multi-arch build and push to registry with version tag
make push VERSION=vX.Y.Z

The VERSION is baked into the binary via -ldflags and displayed in the web UI and /health endpoint. Use versioned tags (not latest) to avoid Docker layer caching issues on orchestration platforms.

docker run -d \
  --name s3-orchestrator \
  -v /path/to/config.yaml:/etc/s3-orchestrator/config.yaml \
  -p 9000:9000 \
  -e DB_PASSWORD=secret \
  -e OCI_ACCESS_KEY=... \
  -e OCI_SECRET_KEY=... \
  s3-orchestrator

The default entrypoint is s3-orchestrator -config /etc/s3-orchestrator/config.yaml. Mount your config file to that path, or override the command to use a different location.

Environment variables referenced in the config via ${VAR} syntax are expanded at startup, so pass secrets as -e flags or via your orchestration platform’s secret injection.

The listen_addr in your config determines which port the process binds to inside the container — make sure your -p mapping matches.

Debian Package (Systemd)

Build a .deb package for bare-metal or VM deployments:

# Build for host architecture
make deb VERSION=X.Y.Z

# Build for both amd64 and arm64
make deb-all VERSION=X.Y.Z

# Build and validate with lintian
make deb-lint VERSION=X.Y.Z

Install and configure:

sudo dpkg -i s3-orchestrator_X.Y.Z_amd64.deb

# Edit the config — set database, backends, buckets
sudo vim /etc/s3-orchestrator/config.yaml

# Set secrets as environment variables (referenced via ${VAR} in config)
sudo vim /etc/default/s3-orchestrator

# Start the service
sudo systemctl start s3-orchestrator

# Check status
sudo systemctl status s3-orchestrator
sudo journalctl -u s3-orchestrator -f

The package installs:

The systemd unit runs as a dedicated s3-orchestrator user with filesystem hardening (ProtectSystem=strict, ProtectHome=yes, NoNewPrivileges=yes). The service is enabled on install but not started automatically, allowing configuration before first start.

Config reload works via systemd:

sudo systemctl reload s3-orchestrator

This sends SIGHUP to the process, reloading bucket credentials, quota limits, rate limits, and rebalance/replication settings without downtime. See Reloading configuration for details on what is and isn’t reloadable.

Uninstall:

# Remove the package (preserves config and data)
sudo apt remove s3-orchestrator

# Remove everything including config, data, and system user
sudo apt purge s3-orchestrator