:: s3-orchestrator

S3-Compatible Envelope Encryption Multi-Cloud Replication Free-Tier Stacking Built-in Dashboard Quota Enforcement Prometheus Metrics Tempo Tracing Nomad HCL Kubernetes Manifests Object Data Cache

Quickstart Documentation Go API GitHub

Unified S3 storage across multiple backends

Combine free-tier storage from multiple providers into a single, larger pool - no cloud payment plans needed!

flowchart LR
    C([S3 Clients]):::client --> O[s3-orchestrator<br/>quota routing]:::orch
    O -->|"12/20 GB used"| B1[OCI Object Storage<br/>quota: 20 GB]:::backend
    O -->|"10/10 GB full"| B2[Backblaze B2<br/>quota: 10 GB]:::full
    O -->|"3/5 GB used"| B3[AWS S3<br/>quota: 5 GB]:::backend
    B1 & B2 & B3 -.- POOL([35 GB unified · 25 GB used · 10 GB free]):::pool

    classDef client fill:#6b4c2a,stroke:#d4a05a,color:#fff,font-weight:bold
    classDef orch fill:#7a5a30,stroke:#e8c070,color:#fff,font-weight:bold
    classDef backend fill:#3a2e20,stroke:#c4a35a,color:#e8dfd0
    classDef full fill:#3a2e20,stroke:#8b3a3a,color:#d4a0a0
    classDef pool fill:none,stroke:#d4a05a,color:#d4a05a,stroke-dasharray:5 5

Transparent multi-cloud replication keeps copies across providers with automatic failover on read

flowchart LR
    C([S3 Client]):::client --> O[s3-orchestrator<br/>replication factor: 2]:::orch
    O -->|write| B1[Backend A]:::backend
    O -.->|replicate| B2[Backend B]:::backend
    B1 -->|read fails| O
    O -->|failover read| B2

    classDef client fill:#6b4c2a,stroke:#d4a05a,color:#fff,font-weight:bold
    classDef orch fill:#7a5a30,stroke:#e8c070,color:#fff,font-weight:bold
    classDef backend fill:#3a2e20,stroke:#c4a35a,color:#e8dfd0

Drop-in S3 replacement - any tool that speaks S3 (aws cli, rclone, SDKs) works with zero code changes

flowchart TD
    CLI[aws cli]:::tool --> O[s3-orchestrator<br/>:9000]:::orch
    RC[rclone]:::tool --> O
    SDK[Python / Go / JS<br/>S3 SDKs]:::tool --> O
    TF[Terraform<br/>S3 backend]:::tool --> O

    classDef tool fill:#3a2e20,stroke:#c4a35a,color:#e8dfd0
    classDef orch fill:#7a5a30,stroke:#e8c070,color:#fff,font-weight:bold

Key Features

Multi-Backend Storage

Stack allocations from different providers into a single, larger storage target.

Combine free-tier allocations from OCI Object Storage, Backblaze B2, AWS S3, MinIO, Wasabi, or any S3-compatible provider. The orchestrator routes writes based on available quota and presents all backends as one unified endpoint.

Per-Backend Quotas

Cap each backend at the exact byte limit to avoid surprise bills.

Set a byte limit on each backend and the orchestrator enforces it atomically on every write. When a backend fills up, writes overflow to the next available backend automatically. Set quota to 0 to disable enforcement.

Cross-Backend Replication

Automatic multi-cloud redundancy with zero client-side changes.

Set a replication factor and a background worker ensures every object exists on that many backends. Objects are written to one backend on PUT; the replicator asynchronously copies them to reach the target factor.

Virtual Buckets

Isolated namespaces and independent credentials per application.

Each bucket has its own SigV4 access key and secret key, with support for presigned URLs (up to 7-day expiry) for direct browser uploads and downloads. Objects are stored with an internal key prefix so bucket isolation requires zero changes to the storage layer or database schema.

Server-Side Encryption

Envelope encryption with AES-256-GCM via inline keys, files, or Vault Transit.

Each object gets a unique data encryption key (DEK), wrapped by the master key. Supports inline config keys, file-based keys, or HashiCorp Vault Transit for HSM-backed key management. Key rotation re-wraps DEKs without touching object data.

S3-Compatible API

Works with aws cli, rclone, any standard S3 client or SDK.

Supports PutObject, GetObject, HeadObject, DeleteObject, CopyObject, ListObjectsV2, multipart uploads, range reads, and user metadata. Any tool that speaks S3 works with no modifications.

Web Dashboard

Real-time storage overview, directory browser, and admin operations.

Built-in web UI with storage summaries, per-backend quota bars, monthly usage charts, a lazy-loaded directory tree for browsing and deleting objects, and admin controls for rebalancing, syncing, and uploading.

Usage Limits

Cap monthly API requests, egress, and ingress per backend.

Set monthly caps on API requests, egress bytes, and ingress bytes per backend. When a backend exceeds a limit, writes overflow to other backends and reads fail over to replicas. Adaptive flushing shortens the tracking interval as limits approach.

Lifecycle Management

Automatic object expiration with configurable rules.

Define expiration rules that target specific key prefixes - for example, automatically clean up temporary uploads or cache objects after a set period. Only objects matching the configured prefix patterns are expired; everything else in the bucket is left untouched. A background worker handles deletion of both backend storage and database metadata.

Observability

Prometheus metrics, OpenTelemetry tracing, structured audit logging.

Exposes Prometheus metrics for all operations, quotas, and background tasks. Ships with a pre-built Grafana dashboard covering request rates, latency, backend health, quota usage, and replication status. OpenTelemetry tracing with configurable sampling. Structured JSON audit logs with request ID correlation across HTTP and storage layers.

Object Data Cache

In-memory LRU cache for read-heavy workloads.

Optional in-memory LRU cache that serves repeated reads from local storage, reducing backend API calls and egress. Configurable maximum cache size, per-object size limit, and TTL. Automatically invalidated on writes and deletes to ensure consistency. Ideal for read-heavy workloads where the same objects are fetched frequently.

Who Is This For?

Homelabbers

Stack free-tier allocations from multiple providers into usable storage without paying for a single plan.

Self-Hosters Running MinIO

Add automatic cloud backups to a local MinIO instance with one config change - no sync scripts or extra tooling.

Small Teams and Startups

Get multi-cloud redundancy and encryption without the cost or complexity of enterprise storage platforms.

Anyone Who Wants Provider Independence

Avoid vendor lock-in. Your applications talk S3 to one endpoint - swap, add, or remove backends without touching a line of code.

Admin Web Interface

A built-in web dashboard provides real-time storage summaries, per-backend quota and usage bars, monthly traffic charts, a lazy-loaded directory tree for browsing and managing objects, and admin controls for rebalancing, syncing, uploading, and deleting files and folders.

Built-in Monitoring

s3-orchestrator ships with a pre-built Grafana dashboard and Prometheus metrics out of the box. Track request rates, latency percentiles, backend health, quota usage, replication progress, and background task performance - all without writing a single query.