https://s3-orchestrator.munchbox.cc/godoc/index.htmlGenerated documentation for the Go packages in this project. Core proxy Multi-backend S3 object orchestration with quota enforcement, replication, and rebalancing. store PostgreSQL metadata store with circuit breaker wrapper and role-based interfaces. backend S3-compatible storage client abstraction with circuit breaker wrapper. config YAML configuration loading, defaults, and validation.Hugoen-ushttps://s3-orchestrator.munchbox.cc/godoc/accounting/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/accounting/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/accounting" Index type OperationRecordFunc type Recorder func New(usage *counter.UsageTracker, recordOp OperationRecordFunc) *Recorder func (r *Recorder) APICall(backend string) func (r *Recorder) APICalls(backend string, n int64) func (r *Recorder) Egress(backend string, sizeBytes int64) func (r *Recorder) GetSuccess(operation, backend string, sizeBytes int64, start time.Time) func (r *Recorder) Ingress(backend string, sizeBytes int64) func (r *Recorder) Operation(operation, backend string, start time.Time, err error) func (r *Recorder) OperationFailed(operation, backend string, start time.Time, err error) func (r *Recorder) PutSuccess(operation, backend string, sizeBytes int64, start time.Time)https://s3-orchestrator.munchbox.cc/godoc/admin/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/admin/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/transport/admin" Package admin provides the admin API handler for operational control endpoints. Index type BackendOps type BackfillChecksumsResult type BulkRewriteResult type Deps type Handler func New(d *Deps) *Handler func (h *Handler) BackfillChecksums(ctx context.Context, batchSize int) BackfillChecksumsResult func (h *Handler) EncryptExisting(ctx context.Context) BulkRewriteResult func (h *Handler) Register(mux *http.ServeMux) func (h *Handler) Replicate(ctx context.Context) (ReplicateResult, error) func (h *Handler) Scrub(ctx context.Context, batchSize int) ScrubResult func (h *Handler) SetReloadStatusProvider(fn func() any) type OverReplicationOps type Reconciler type ReplicateResult type ReplicatorOps type ScrubResult type ScrubberOps type WorkerHealthhttps://s3-orchestrator.munchbox.cc/godoc/adminctl/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/adminctl/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/cli/adminctl" Package adminctl implements the `s3-orchestrator admin …` family of subcommands. Each command is a thin HTTP client over the admin API exposed by the running server, formatting JSON responses for human consumption. Index func Command(cmd string, args []string, baseAddr, token string, stdout, stderr io.Writer) int func Run(args []string, stdout, stderr io.Writer) inthttps://s3-orchestrator.munchbox.cc/godoc/audit/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/audit/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/observe/audit" Package audit provides request ID propagation and structured audit logging for security-relevant operations. Index func Log(ctx context.Context, event string, attrs …slog.Attr) func NewID() string func RequestID(ctx context.Context) string func SetOnEvent(fn func(event string)) func WithRequestID(ctx context.Context, id string) context.Context func Log func Log(ctx context.Context, event string, attrs ...slog.Attr) Log emits a structured audit log entry at Info level. Automatically includes the request ID from context. If OnEvent is set, it is called with the event name for metrics integration.https://s3-orchestrator.munchbox.cc/godoc/auth/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/auth/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/transport/auth" Package auth provides S3 SigV4 (header and presigned URL) and token-based request authentication with multi-bucket credential resolution. Index Constants Variables func NewChunkReader(body io.ReadCloser, mat *StreamingMaterial) io.ReadCloser func VerifySigV4(r *http.Request, accessKeyID, secretAccessKey string) error type BucketRegistry func NewBucketRegistry(buckets []config.BucketConfig) *BucketRegistry func (br *BucketRegistry) AuthenticateAndResolveBucket(r *http.Request) (string, *StreamingMaterial, error) func (br *BucketRegistry) MaxMultipartUploads(bucket string) int type StreamingMaterial type StreamingVariant func DetectStreamingVariant(payloadHash string) StreamingVariant func (v StreamingVariant) Label() string Constants MaxChunkBytes is the largest single declared chunk size accepted by the reader. AWS SDKs use 64 KiB by default; the cap defends against adversarial declarations of huge chunks that would never be delivered.https://s3-orchestrator.munchbox.cc/godoc/backend/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/backend/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/backend" Package backend defines the ObjectBackend interface for S3-compatible storage providers and provides the S3Backend implementation (AWS SDK v2) with per-backend circuit breaker wrapping. Index Variables func IsCopyPhase(err error, phase CopyPhase) bool func IsNotFound(err error) bool type BackendCopier type CircuitBreakerBackend func NewCircuitBreakerBackend(real ObjectBackend, name string, threshold int, timeout time.Duration) *CircuitBreakerBackend func (cb *CircuitBreakerBackend) CopyObject(ctx context.Context, srcKey, dstKey, contentType string, metadata map[string]string) (string, error) func (cb *CircuitBreakerBackend) DeleteObject(ctx context.Context, key string) error func (cb *CircuitBreakerBackend) GetObject(ctx context.Context, key string, rangeHeader string) (*GetObjectResult, error) func (cb *CircuitBreakerBackend) HeadObject(ctx context.Context, key string) (*HeadObjectResult, error) func (cb *CircuitBreakerBackend) PutObject(ctx context.Context, key string, body io.Reader, size int64, contentType string, metadata map[string]string) (string, error) func (cb *CircuitBreakerBackend) Unwrap() ObjectBackend type CopyError func (e *CopyError) Error() string func (e *CopyError) Unwrap() error type CopyPhase type GetObjectResult type HeadObjectResult type ListedObject type ObjectBackend type S3Backend func NewS3Backend(ctx context.Context, cfg *config.BackendConfig) (*S3Backend, error) func (b *S3Backend) CopyObject(ctx context.Context, srcKey, dstKey, contentType string, metadata map[string]string) (string, error) func (b *S3Backend) DeleteObject(ctx context.Context, key string) error func (b *S3Backend) GetObject(ctx context.Context, key string, rangeHeader string) (*GetObjectResult, error) func (b *S3Backend) HeadObject(ctx context.Context, key string) (*HeadObjectResult, error) func (b *S3Backend) ListObjects(ctx context.Context, prefix string, fn func([]ListedObject) error) error func (b *S3Backend) PutObject(ctx context.Context, key string, body io.Reader, size int64, contentType string, metadata map[string]string) (string, error) Variables ErrCopyNotSupported signals that a backend (or a decorator wrapping one) cannot perform a server-side copy. Callers should fall back to materialized copy. Returned by decorators like CircuitBreakerBackend when their wrapped backend does not implement BackendCopier.https://s3-orchestrator.munchbox.cc/godoc/breaker/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/breaker/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/breaker" Package breaker implements a generic three-state circuit breaker (closed, open, half-open) with pluggable error filters and probe jitter. The breaker emits no metrics or events on its own - callers wire those via the optional OnStateChange callback so this package stays free of observability dependencies. Index Constants Variables func CBCall[T any](cb *CircuitBreaker, fn func() (T, error)) (T, error) func CBCallNoResult(cb *CircuitBreaker, fn func() error) error func NewWatchdog(registry *Registry) lifecycle.Runner type CircuitBreaker func NewCircuitBreaker(name string, threshold int, timeout time.Duration, isError func(error) bool, sentinel error) *CircuitBreaker func (cb *CircuitBreaker) AddOnStateChange(fn func(StateChangeInfo)) func (cb *CircuitBreaker) IsHealthy() bool func (cb *CircuitBreaker) Name() string func (cb *CircuitBreaker) OpenDuration() time.Duration func (cb *CircuitBreaker) PostCheck(err error) error func (cb *CircuitBreaker) PreCheck() error func (cb *CircuitBreaker) ProbeEligible() bool func (cb *CircuitBreaker) Recover() func (cb *CircuitBreaker) ResetStaleProbe() bool func (cb *CircuitBreaker) SetOnStateChange(fn func(StateChangeInfo)) func (cb *CircuitBreaker) State() State type Registry func NewRegistry(initial …StaleProbeResetter) *Registry func (r *Registry) Len() int func (r *Registry) Register(b StaleProbeResetter) func (r *Registry) ResetStaleProbes() type StaleProbeResetter type State func (s State) String() string type StateChangeInfo Constants DefaultWatchdogInterval is the cadence at which the watchdog inspects every registered breaker. Picked as half the breaker probe timeout so a stuck half-open state is detected within one full probe window.https://s3-orchestrator.munchbox.cc/godoc/bufpool/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/bufpool/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/util/bufpool" Package bufpool provides a shared pool of reusable byte buffers for streaming I/O, reducing GC pressure by replacing per-call allocations in io.Copy with pooled buffers via io.CopyBuffer. Index func Copy(dst io.Writer, src io.Reader) (int64, error) func Get() *[]byte func GetWriter(w io.Writer) *bufio.Writer func Put(b *[]byte) func PutWriter(bw *bufio.Writer)https://s3-orchestrator.munchbox.cc/godoc/cache/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/cache/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/cache" Package cache provides an object data cache layer that sits between the orchestrator and storage backends, reducing API calls and egress by serving repeated reads from local storage. Index type Entry func (e *Entry) Size() int64 type EntryMeta type MemoryCache func NewMemoryCache(cfg MemoryConfig) (*MemoryCache, error) func (c *MemoryCache) Admit(size int64) bool func (c *MemoryCache) Clear() int func (c *MemoryCache) Get(key string) (*Entry, bool) func (c *MemoryCache) Invalidate(key string) func (c *MemoryCache) InvalidatePrefix(prefix string) int func (c *MemoryCache) PutBytes(key string, data []byte, meta EntryMeta) func (c *MemoryCache) Stats() Stats type MemoryConfig type ObjectCache type Statshttps://s3-orchestrator.munchbox.cc/godoc/chunkframing/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/chunkframing/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/integration/chunkframing" Index Constants type Variant func Detect(head []byte) Variant Constants MaxChunkSizeBytes caps a single declared chunk during validation. AWS SDKs default to 64 KiB; values beyond this threshold are treated as a negative signal.https://s3-orchestrator.munchbox.cc/godoc/config/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/config/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/config" Package config provides YAML configuration loading with environment variable expansion, validation, and hot-reload support via SIGHUP. Index Constants Variables func NonReloadableFieldsChanged(old, new *Config) []string func ParseLogLevel(s string) slog.Level type BackendCircuitBreakerConfig type BackendConfig type BucketConfig type CacheConfig type CircuitBreakerConfig type CleanupQueueConfig type Config func LoadConfig(path string) (*Config, error) func (c *Config) SetDefaultsAndValidate() error type CredentialConfig type DatabaseConfig func (c *DatabaseConfig) ConnectionString() string type EncryptionConfig type IntegrityConfig func (ic *IntegrityConfig) ShouldVerifyOnReplicate() bool type LifecycleConfig type LifecycleRule type MetricsConfig type NotificationConfig type NotificationEndpoint type PendingPatternConfig func (p *PendingPatternConfig) IsEnabled() bool type RateLimitConfig type RebalanceConfig type ReconcileConfig type RedisConfig type ReplicationConfig type RoutingStrategy type ServerConfig type TLSConfig type TelemetryConfig type TracingConfig type UIConfig type UsageFlushConfig type VaultTransitConfig type WritePathConfig Constants CredentialSourceStatic and friends enumerate the supported credential_source values.https://s3-orchestrator.munchbox.cc/godoc/core/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/core/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/store/core" Package core holds the engine-agnostic orchestration that both store engines share: the TxAdapter seam, the Runner abstraction, narrow store role interfaces, and operations that span multiple statements within a single transaction. Index Constants Variables func DeleteObjectsBatch(ctx context.Context, runner Runner, keys []string) (map[string][]DeletedCopy, error) func GroupByKey(locations []ObjectLocation) map[string][]ObjectLocation func ImportObject(ctx context.Context, runner Runner, key, backend string, size int64) (bool, error) func MoveCleanupToDLQ(ctx context.Context, runner Runner, id int64, lastError string) (bool, error) func MoveObjectLocation(ctx context.Context, runner Runner, key, fromBackend, toBackend string) (int64, error) func PromotePending(ctx context.Context, runner Runner, p *PendingObject) (PendingPromoteResult, []DeletedCopy, error) func RecordReplica(ctx context.Context, runner Runner, key, targetBackend, sourceBackend string) (int64, bool, error) func RemoveExcessCopy(ctx context.Context, runner Runner, key, backendName string, size int64) error func SweepStaleCleanupQueueRows(ctx context.Context, runner Runner, objectKey, backend string) (int64, error) func WithTxVal[T any](ctx context.Context, runner Runner, fn func(ctx context.Context, tx TxAdapter) (T, error)) (T, error) type AdvisoryLocker type BackendLifecycleStore type CleanupItem type CleanupQueueRow type CleanupStore type CleanupTxAdapter type CreateMultipartUploadParams type DashboardStore type DecryptableLocation type DeletedCopy func DeleteObject(ctx context.Context, runner Runner, key string) ([]DeletedCopy, error) func RecordObject(ctx context.Context, runner Runner, key, backend string, size int64, enc *EncryptionMeta) ([]DeletedCopy, error) func RecordObjectAndClearPending(ctx context.Context, runner Runner, key, backend string, size int64, enc *EncryptionMeta, intentID string) ([]DeletedCopy, error) type DirEntry type DirectoryListResult type EncryptedLocation type EncryptionAdmin type EncryptionMeta type ExistingCopy type ExpiredObjectsLister type IntegrityStore type KeyedExistingCopy type LifecycleAdmin type ListObjectsResult type MetadataStore type MultipartPart type MultipartStore type MultipartUpload type NotificationOutbox type NotificationRow type ObjectLocation type ObjectStore type ObjectsTxAdapter type PendingObject type PendingPromoteResult type PendingStore type PendingTxAdapter type QuotaStat type QuotaStore type QuotaTxAdapter type Reader type ReplicationStore type Runner type S3Error func (e *S3Error) Error() string type TxAdapter type UnencryptedLocation type UsageFlusher type UsageLimits type UsageStat Constants Advisory lock IDs assigned to each background service.https://s3-orchestrator.munchbox.cc/godoc/counter/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/counter/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/counter" Package counter provides usage tracking with per-backend atomic counters and monthly limit enforcement. Supports local in-memory counters and Redis-backed shared counters for multi-instance deployments. Index Constants func CurrentPeriod() string type CounterBackend type LoadAllResult type LocalCounterBackend func NewLocalCounterBackend(backendNames []string) *LocalCounterBackend func (l *LocalCounterBackend) Add(backend, field string, delta int64) func (l *LocalCounterBackend) AddAll(backend string, apiReqs, egress, ingress int64) func (l *LocalCounterBackend) Backends() []string func (l *LocalCounterBackend) Load(backend, field string) int64 func (l *LocalCounterBackend) LoadAll(backend string) LoadAllResult func (l *LocalCounterBackend) Swap(backend, field string) int64 func (l *LocalCounterBackend) SwapAll(backend string) LoadAllResult func (l *LocalCounterBackend) SwapAllBackends() map[string]LoadAllResult type RedisClient type RedisCounterBackend func NewRedisCounterBackend(client RedisClient, cfg *config.RedisConfig, backendNames []string) (*RedisCounterBackend, error) func (r *RedisCounterBackend) Add(backend, field string, delta int64) func (r *RedisCounterBackend) AddAll(backend string, apiReqs, egress, ingress int64) func (r *RedisCounterBackend) Backends() []string func (r *RedisCounterBackend) Close() error func (r *RedisCounterBackend) IsHealthy() bool func (r *RedisCounterBackend) Load(backend, field string) int64 func (r *RedisCounterBackend) LoadAll(backend string) LoadAllResult func (r *RedisCounterBackend) Swap(backend, field string) int64 type UsageTracker func NewUsageTracker(backend CounterBackend, limits map[string]core.UsageLimits) *UsageTracker func (u *UsageTracker) Backend() CounterBackend func (u *UsageTracker) BackendsWithinLimits(order []string, apiCalls, egress, ingress int64) []string func (u *UsageTracker) FlushUsage(ctx context.Context, store usageFlusher, skip map[string]bool) error func (u *UsageTracker) GetLimits() map[string]core.UsageLimits func (u *UsageTracker) NearLimit(threshold float64) bool func (u *UsageTracker) Record(backendName string, apiCalls, egress, ingress int64) func (u *UsageTracker) ResetBaselines(names []string) func (u *UsageTracker) SetBaseline(name string, stat core.UsageStat) func (u *UsageTracker) UpdateLimits(limits map[string]core.UsageLimits) func (u *UsageTracker) WithinLimits(backendName string, apiCalls, egress, ingress int64) bool Constants Counter field names used as keys in CounterBackend operations.https://s3-orchestrator.munchbox.cc/godoc/dashboard/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/dashboard/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/dashboard" Package dashboard owns the read-only stats aggregation that the web UI renders. The proxy package wraps Aggregator results with cluster-state enrichment (drain progress, breaker health) before returning them to HTTP handlers. Index type Aggregator func New(store core.DashboardStore, usage *counter.UsageTracker, order []string) *Aggregator func (da *Aggregator) GetData(ctx context.Context) (*Data, error) func (da *Aggregator) GetDirectoryChildren(ctx context.Context, prefix, startAfter string, maxKeys int) (*core.DirectoryListResult, error) type Datahttps://s3-orchestrator.munchbox.cc/godoc/di/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/di/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/di" Package di is the single wiring point for the orchestrator. It uses samber/do/v2 to register every store role, backend, worker, and transport handler as a lazy provider; consumers resolve their dependencies through the injector at the moment they need them. Index func IsRegistered[T any](inj do.Injector) bool func NewInjector(cfg *config.Config, mode string, logLevel *slog.LevelVar, logBuffer *telemetry.LogBuffer) do.Injector func NewLifecycleService(manager lifecycleOps, locker tickrunner.AdvisoryLocker) lifecycle.Runner func NewUsageFlushService(manager usageFlushOps, locker tickrunner.AdvisoryLocker) lifecycle.Runner func ProvideAdminHandler(i do.Injector) (*admin.Handler, error) func ProvideBackendManager(i do.Injector) (*proxy.BackendManager, error) func ProvideBreakerRegistry(i do.Injector) (*breaker.Registry, error) func ProvideBucketAuth(i do.Injector) (*auth.BucketRegistry, error) func ProvideCleanupWorker(i do.Injector) (*worker.CleanupWorker, error) func ProvideDatabaseBreaker(i do.Injector) (*breaker.CircuitBreaker, error) func ProvideDrainManager(i do.Injector) (*drain.Manager, error) func ProvideEncryptionAdmin(i do.Injector) (core.EncryptionAdmin, error) func ProvideEncryptionProvider(i do.Injector) (encryption.KeyProvider, error) func ProvideEncryptor(i do.Injector) (*encryption.Encryptor, error) func ProvideInstanceID(_ do.Injector) (instanceid.ID, error) func ProvideLifecycleAdmin(i do.Injector) (core.LifecycleAdmin, error) func ProvideLifecycleManager(i do.Injector) (*lifecycle.Manager, error) func ProvideLoginThrottle(_ do.Injector) (*httputil.LoginThrottle, error) func ProvideMetadataStore(i do.Injector) (core.MetadataStore, error) func ProvideMetricsDeps(i do.Injector) (metrics.Deps, error) func ProvideNotificationOutbox(i do.Injector) (core.NotificationOutbox, error) func ProvideNotifier(i do.Injector) (*notify.Notifier, error) func ProvideObjectCache(i do.Injector) (objcache.ObjectCache, error) func ProvideOverReplicationCleaner(i do.Injector) (*worker.OverReplicationCleaner, error) func ProvidePendingReaper(i do.Injector) (*worker.PendingReaper, error) func ProvideRateLimiter(i do.Injector) (*s3api.RateLimiter, error) func ProvideRebalancer(i do.Injector) (*worker.Rebalancer, error) func ProvideReconciler(i do.Injector) (*worker.Reconciler, error) func ProvideRedisCounterBackend(i do.Injector) (*counter.RedisCounterBackend, error) func ProvideReplicator(i do.Injector) (*worker.Replicator, error) func ProvideS3Server(i do.Injector) (*s3api.Server, error) func ProvideScrubber(i do.Injector) (*worker.Scrubber, error) func ProvideUIHandler(i do.Injector) (*ui.Handler, error) func WireAuditMetrics() func WireManager(inj do.Injector) error type BackendsResult func ProvideBackends(i do.Injector) (*BackendsResult, error) type OptionalResult func Optional[T any](inj do.Injector) OptionalResult[T] func (r OptionalResult[T]) Applied() bool func (r OptionalResult[T]) Disabled() bool func (r OptionalResult[T]) Failed() bool type Resolutionhttps://s3-orchestrator.munchbox.cc/godoc/drain/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/drain/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/drain" Package drain owns the backend drain/remove lifecycle. It tracks the draining state map, runs the migration goroutine, and exposes IsDraining for the proxy core’s eligibility filters. Index type Core type Manager func New(infra Core, objects core.ObjectStore, quota core.QuotaStore, backendLifecycle core.BackendLifecycleStore, abortMultipartUploads func(ctx context.Context, backendName string), processCleanupQueue func(ctx context.Context) (processed, failed int)) *Manager func (d *Manager) CancelDrain(name string) error func (d *Manager) ClearState() func (d *Manager) CompletedBackends() map[string]bool func (d *Manager) DrainOneObject(ctx context.Context, srcBackend backend.ObjectBackend, srcName string, obj *core.ObjectLocation) bool func (d *Manager) GetDrainProgress(ctx context.Context, name string) (*Progress, error) func (d *Manager) IsDraining(name string) bool func (d *Manager) PurgeBackendObjects(ctx context.Context, be backend.ObjectBackend, name string) func (d *Manager) RemoveBackend(ctx context.Context, name string, purge bool) error func (d *Manager) SeedActiveForTest(name string) func (d *Manager) SeedCompletedForTest(name string) func (d *Manager) StartDrain(ctx context.Context, name string) error type Progresshttps://s3-orchestrator.munchbox.cc/godoc/encryption/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/encryption/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/encryption" Package encryption provides envelope encryption for object payloads. It frames plaintext into chunked AES-GCM segments addressable by byte range and supports pluggable key providers (config-embedded, file, and Vault transit). Index Constants func PackKeyData(baseNonce, wrappedDEK []byte) []byte func ParseHeader(r io.Reader) (chunkSize int, baseNonce []byte, err error) func UnpackKeyData(data []byte) (baseNonce, wrappedDEK []byte, err error) type ConfigKeyProvider func NewConfigKeyProvider(masterKeyB64, keyID string) (*ConfigKeyProvider, error) func (p *ConfigKeyProvider) KeyID() string func (p *ConfigKeyProvider) UnwrapDEK(_ context.Context, wrappedDEK []byte, _ string) ([]byte, error) func (p *ConfigKeyProvider) WrapDEK(_ context.Context, dek []byte) ([]byte, string, error) type EncryptResult func (r *EncryptResult) RawDEK() []byte type Encryptor func NewEncryptor(provider KeyProvider, chunkSize int) (*Encryptor, error) func (e *Encryptor) ChunkSize() int func (e *Encryptor) CiphertextSize(plaintextSize int64) int64 func (e *Encryptor) Decrypt(ctx context.Context, body io.Reader, wrappedDEK []byte, keyID string) (io.Reader, error) func (e *Encryptor) DecryptRange(ctx context.Context, body io.Reader, wrappedDEK []byte, keyID string, rng *RangeResult, baseNonce []byte) (io.Reader, int64, error) func (e *Encryptor) Encrypt(ctx context.Context, body io.Reader, plaintextSize int64) (*EncryptResult, error) func (e *Encryptor) EncryptWithDEK(body io.Reader, plaintextSize int64, dek, wrappedDEK []byte, keyID string) (*EncryptResult, error) func (e *Encryptor) GenerateAndWrapDEK(ctx context.Context) (dek, wrappedDEK []byte, keyID string, err error) func (e *Encryptor) Provider() KeyProvider type FileKeyProvider func NewFileKeyProvider(path, keyID string) (*FileKeyProvider, error) func (p *FileKeyProvider) KeyID() string func (p *FileKeyProvider) UnwrapDEK(_ context.Context, wrappedDEK []byte, _ string) ([]byte, error) func (p *FileKeyProvider) WrapDEK(_ context.Context, dek []byte) ([]byte, string, error) type KeyProvider func NewKeyProviderFromConfig(cfg *config.EncryptionConfig) (KeyProvider, error) type MultiKeyProvider func NewMultiKeyProvider(primary KeyProvider, previous []KeyProvider) *MultiKeyProvider func (p *MultiKeyProvider) KeyID() string func (p *MultiKeyProvider) UnwrapDEK(ctx context.Context, wrappedDEK []byte, keyID string) ([]byte, error) func (p *MultiKeyProvider) WrapDEK(ctx context.Context, dek []byte) ([]byte, string, error) type RangeResult func CiphertextRange(start, end int64, chunkSize int) (*RangeResult, error) type VaultKeyProvider func NewVaultKeyProvider(cfg *config.VaultTransitConfig) (*VaultKeyProvider, error) func (p *VaultKeyProvider) Close() func (p *VaultKeyProvider) KeyID() string func (p *VaultKeyProvider) UnwrapDEK(ctx context.Context, wrappedDEK []byte, _ string) ([]byte, error) func (p *VaultKeyProvider) WrapDEK(ctx context.Context, dek []byte) ([]byte, string, error) Constants HeaderSize and related constants used by this package.https://s3-orchestrator.munchbox.cc/godoc/event/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/event/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/observe/event" Package event defines the CloudEvents-shaped notification types the orchestrator emits when objects are created, deleted, or otherwise mutated, plus the package-level Emit hook other layers call to publish them. The package has zero internal dependencies so any caller can import it without creating a cycle. Index Constants Variables func MatchesFilter(eventType string, patterns []string) bool type Event Constants Operational events - infrastructure state changes.https://s3-orchestrator.munchbox.cc/godoc/httpserver/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/httpserver/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/transport/httpserver" Package httpserver constructs the HTTP listener the daemon serves S3, admin, UI, health, and metrics traffic on. It owns route registration, middleware composition, TLS config (including the cert reloader for SIGHUP rotation), and the optional separate metrics listener. Index Constants type Deps type HealthDeps type Server func New(deps Deps) (*Server, error) func (s *Server) CertReloader() *httputil.CertReloader func (s *Server) Run(ctx context.Context) error func (s *Server) Shutdown(ctx context.Context) Constants Daemon mode strings. registerS3Handler and registerUIHandler are only mounted in “api” and “all” modes; worker-only mode exposes neither.https://s3-orchestrator.munchbox.cc/godoc/httputil/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/httputil/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/transport/httputil" Package httputil houses cross-cutting HTTP helpers that several transport sub-packages share: trusted-proxy-aware client IP extraction, login-attempt throttling, and TLS certificate hot-reloading. Index Constants func DecodeJSONBody(w http.ResponseWriter, r *http.Request, dst any, maxBytes int64) bool func ExtractClientIP(r *http.Request, trustedProxies []*net.IPNet) string func IsTLSRequest(r *http.Request, trustedProxies []*net.IPNet) bool func PanicRecover(route string, writeErr ErrorWriter) func(http.Handler) http.Handler func ParseTrustedProxies(cidrs []string) []*net.IPNet func RequireMethod(w http.ResponseWriter, r *http.Request, methods …string) bool func WriteJSON(w http.ResponseWriter, status int, body any) func WriteJSONError(w http.ResponseWriter, status int, msg string) type CertReloader func NewCertReloader(certFile, keyFile string) (*CertReloader, error) func (cr *CertReloader) GetCertificate(_ *tls.ClientHelloInfo) (*tls.Certificate, error) func (cr *CertReloader) Reload() error type ErrorWriter type LoginThrottle func NewLoginThrottle(maxFailures int, lockoutDuration time.Duration) *LoginThrottle func (lt *LoginThrottle) Close() func (lt *LoginThrottle) IsLockedOut(ip string) bool func (lt *LoginThrottle) RecordFailure(ip string) func (lt *LoginThrottle) RecordSuccess(ip string) Constantshttps://s3-orchestrator.munchbox.cc/godoc/infra/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/infra/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/infra" Package infra exposes the proxy-package backend infrastructure (backend map, usage tracker, drain checker, metrics, admission, per-op timeouts) as an importable type so subpackages can share it without an import cycle back to the root proxy package. Index type Config type Core func New(cfg *Config) *Core func (c *Core) Acct() *accounting.Recorder func (c *Core) AcquireAdmission(ctx context.Context) bool func (c *Core) AdmissionSem() chan struct{} func (c *Core) BackendOrder() []string func (c *Core) Backends() map[string]backend.ObjectBackend func (c *Core) ClassifyWriteError(span trace.Span, operation string, err error) error func (c *Core) DeleteWithTimeout(ctx context.Context, be backend.ObjectBackend, key string) error func (c *Core) EligibleForWrite(apiCalls, egress, ingress int64) []string func (c *Core) ExcludeDraining(eligible []string) []string func (c *Core) ExcludeUnhealthy(eligible []string) []string func (c *Core) GetBackend(name string) (backend.ObjectBackend, error) func (c *Core) IsDraining(name string) bool func (c *Core) Log() *slog.Logger func (c *Core) MaxObjectSize(name string) int64 func (c *Core) MetricsCollector() *metrics.Collector func (c *Core) RecordOperation(operation, backend string, start time.Time, err error) func (c *Core) ReleaseAdmission() func (c *Core) RoutingStrategy() config.RoutingStrategy func (c *Core) SetDrainChecker(d DrainChecker) func (c *Core) SetMetricsCollector(m *metrics.Collector) func (c *Core) StreamCopy(ctx context.Context, src, dst backend.ObjectBackend, key string) error func (c *Core) UpdateQuotaMetrics(ctx context.Context) error func (c *Core) Usage() *counter.UsageTracker func (c *Core) WithTimeout(ctx context.Context) (context.Context, context.CancelFunc) type DrainCheckerhttps://s3-orchestrator.munchbox.cc/godoc/initcmd/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/initcmd/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/cli/initcmd" Package initcmd implements the `s3-orchestrator init` subcommand, which interactively walks an operator through writing a baseline config.yaml suitable for first boot. Index func GenerateConfig(params *Params) (string, error) func Run(args []string, in io.Reader, out, errOut io.Writer) int func RunInteractive(configPath string, in io.Reader, out io.Writer) error type Backend type Bucket type Paramshttps://s3-orchestrator.munchbox.cc/godoc/instanceid/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/instanceid/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/instanceid" Package instanceid produces a stable per-process identifier that the cleanup worker writes into cleanup_queue.claimed_by for operator visibility into which orchestrator process is currently holding a row. Index type ID func New() (ID, error) func (i ID) String() string type ID ID is the resolved instance identifier in the form hostname-XXXXXXXX.https://s3-orchestrator.munchbox.cc/godoc/internalkey/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/internalkey/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/internalkey" Package internalkey centralizes the “bucket/userkey” namespacing convention used between the S3 transport layer and the storage / metadata layer. Index Constants func Make(bucket, userKey string) string func Prefix(bucket string) string func Split(internalKey string) (bucket, userKey string) Constants Separator is the delimiter between the bucket name and the user-facing object key inside an internal storage key.https://s3-orchestrator.munchbox.cc/godoc/ioutilx/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/ioutilx/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/util/ioutilx" Package ioutilx provides small composition helpers around io.Reader and io.ReadCloser. Helpers preserve close ordering and ensure cancel callbacks fire exactly once. Index func ReadCloser(r io.Reader, c io.Closer) io.ReadCloser func WithCancel(rc io.ReadCloser, cancel func()) io.ReadCloser func ReadCloser func ReadCloser(r io.Reader, c io.Closer) io.ReadCloser ReadCloser returns an io.ReadCloser that reads from r and delegates Close to c. Use when a transformation wraps a reader (decryption, metric counting, tee) but the original closer still has to run so the underlying resource (HTTP body, file, etc.) is released.https://s3-orchestrator.munchbox.cc/godoc/lifecycle/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/lifecycle/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/lifecycle" Package lifecycle provides a service manager for registering and running background goroutines with coordinated graceful shutdown. Index type HealthReporter type Manager func NewManager() *Manager func (m *Manager) Health() []WorkerHealth func (m *Manager) Names() []string func (m *Manager) Register(name string, r Runner) func (m *Manager) Run(ctx context.Context) func (m *Manager) SetBackoff(initial, maximum, reset time.Duration) func (m *Manager) Stop(timeout time.Duration) type Runner type Stopper type WorkerHealthhttps://s3-orchestrator.munchbox.cc/godoc/logfmt/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/logfmt/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/observe/logfmt" Package logfmt provides typed slog.Attr constructors for the project’s logging conventions. Index Constants func Component(name string) slog.Attr func Err(err error) slog.Attr func LoggerFromCtx(ctx context.Context, base *slog.Logger) *slog.Logger func Outcome(value string) slog.Attr func RequestIDFromCtx(ctx context.Context) slog.Attr func SetRequestIDFunc(fn func(context.Context) string) func TransformAttr(a slog.Attr) slog.Attr type ErrAttrHandler func NewErrAttrHandler(inner slog.Handler) *ErrAttrHandler func (h *ErrAttrHandler) Enabled(ctx context.Context, lvl slog.Level) bool func (h *ErrAttrHandler) Handle(ctx context.Context, r slog.Record) error func (h *ErrAttrHandler) WithAttrs(attrs []slog.Attr) slog.Handler func (h *ErrAttrHandler) WithGroup(name string) slog.Handler Constants Outcome values for the canonical “outcome” attribute. Use these constants rather than ad-hoc strings so dashboards can rely on a closed value set.https://s3-orchestrator.munchbox.cc/godoc/metrics/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/metrics/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/metrics" Package metrics owns the Prometheus gauge/counter recording for the proxy package. It exposes a Collector that the orchestrator embeds to refresh gauge values from the metadata store and record per-operation metrics. Index type Collector func New(store Deps, usage *counter.UsageTracker, backendNames []string, replicationFactor func() int) *Collector func (mc *Collector) RecordOperation(operation, backend string, start time.Time, err error) func (mc *Collector) UpdateQuotaMetrics(ctx context.Context) error type Depshttps://s3-orchestrator.munchbox.cc/godoc/multipart/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/multipart/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/multipart" Index Constants func NewCleanupService(mgr *Manager, locker tickrunner.AdvisoryLocker, staleTimeout time.Duration) lifecycle.Runner type Manager func New(core MultipartCore, coord *writepath.Coordinator, stores core.MetadataStore, encryptor *encryption.Encryptor, objectCache objcache.ObjectCache, dekCacheTTL time.Duration, integrityCfg *syncutil.AtomicConfig[config.IntegrityConfig]) *Manager func (mp *Manager) AbortMultipartUpload(ctx context.Context, bucket, key, uploadID string) error func (mp *Manager) AbortMultipartUploadsOnBackend(ctx context.Context, backendName string) func (mp *Manager) CleanupStaleMultipartUploads(ctx context.Context, olderThan time.Duration) func (mp *Manager) Close() func (mp *Manager) CompleteMultipartUpload(ctx context.Context, bucket, key, uploadID string, partNumbers []int) (string, error) func (mp *Manager) CreateMultipartUpload(ctx context.Context, key, contentType string, metadata map[string]string) (string, string, error) func (mp *Manager) GetParts(ctx context.Context, bucket, key, uploadID string) ([]core.MultipartPart, error) func (mp *Manager) ListMultipartUploads(ctx context.Context, prefix string, maxUploads int) ([]core.MultipartUpload, error) func (mp *Manager) UnwrapUploadDEK(ctx context.Context, mu *core.MultipartUpload) (dek, wrappedDEK []byte, baseNonce []byte, err error) func (mp *Manager) UploadPart(ctx context.Context, bucket, key, uploadID string, partNumber int, body io.Reader, size int64) (string, error) type MultipartCore Constants Default cleanup parameters used when the config does not override them.https://s3-orchestrator.munchbox.cc/godoc/must/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/must/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/util/must" Index func NotNil(name string, v any) func NotNil func NotNil(name string, v any) NotNil panics when v is nil. Handles both the untyped-nil case (v == nil) and the typed-nil interface case (e.g. var x Store = (*concreteStore)(nil)) which the bare comparison misses. name should identify the dependency for the panic message; constructors typically pass the parameter name.https://s3-orchestrator.munchbox.cc/godoc/notify/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/notify/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/notify" Package notify drains the notification_outbox table and delivers each pending event to its configured webhook target with retries and exponential backoff under an advisory lock for multi-instance safety. Index type Notifier func NewNotifier(cfg *config.NotificationConfig, store OutboxStore) *Notifier func (n *Notifier) Run(ctx context.Context) error type OutboxStorehttps://s3-orchestrator.munchbox.cc/godoc/object/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/object/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/object" Index Variables func AdvancePastEmittedCommonPrefix(prefix, delimiter, cursor string, seen map[string]bool) string func HashBody(data []byte) string func ParsePlaintextRange(rangeHeader string, plaintextSize int64) (start, end int64, ok bool) type DeleteObjectResult type Deps type ListObjectsV2Result type LocationCache func NewLocationCache(ttl time.Duration) *LocationCache func (c *LocationCache) Clear() func (c *LocationCache) Close() func (c *LocationCache) Delete(key string) func (c *LocationCache) Get(key string) (string, bool) func (c *LocationCache) Len() int func (c *LocationCache) Set(key, backend string) type Manager func New(d *Deps) *Manager func (o *Manager) BackendCapacityStats(ctx context.Context) map[string]core.QuotaStat func (o *Manager) CanAcceptWrite(size int64) bool func (o *Manager) CopyObject(ctx context.Context, sourceKey, destKey string) (string, error) func (o *Manager) DeleteObject(ctx context.Context, key string) error func (o *Manager) DeleteObjects(ctx context.Context, keys []string) []DeleteObjectResult func (o *Manager) GetObject(ctx context.Context, key string, rangeHeader string) (*s3be.GetObjectResult, error) func (o *Manager) HeadObject(ctx context.Context, key string) (*s3be.HeadObjectResult, error) func (o *Manager) ListObjects(ctx context.Context, prefix, delimiter, startAfter string, maxKeys int) (*ListObjectsV2Result, error) func (o *Manager) LocationCache() *LocationCache func (o *Manager) ObjectExists(ctx context.Context, key string) (bool, error) func (o *Manager) PutObject(ctx context.Context, key string, body io.Reader, size int64, contentType string, metadata map[string]string) (string, error) type ObjectCoordinator type ObjectCore type VerifyingReader func NewVerifyingReader(r io.ReadCloser) *VerifyingReader func (vr *VerifyingReader) Close() error func (vr *VerifyingReader) Read(p []byte) (int, error) func (vr *VerifyingReader) SetVerification(expected string, onMismatch func(expected, actual string)) func (vr *VerifyingReader) Verify(expected string) error Variables ListObjectsMaxPages caps DB round trips per ListObjects request so a single client call cannot drag the database through unbounded scans on pathological prefix layouts. Exposed as a var (rather than const) so tests can lower it without generating hundreds of mock pages.https://s3-orchestrator.munchbox.cc/godoc/observe/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/observe/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/observe" Package observe centralizes per-operation completion observability - audit log, notification event, and span status - so storage paths in object/, multipart/, and writepath/ can mark success with one call and the audit/event attribute shapes stay defined in exactly one place. Index func CopyCompleted(ctx context.Context, span trace.Span, sourceKey, destKey, sourceBackend, destBackend string, size int64) func DeleteBatchCompleted(ctx context.Context, span trace.Span, totalKeys, deleted, errors int) func DeleteCompleted(ctx context.Context, span trace.Span, key string, copiesDeleted int) func GetCompleted(ctx context.Context, key, backend string, size int64) func HeadCompleted(ctx context.Context, key, backend string, size int64) func ListCompleted(ctx context.Context, prefix string, keyCount int, truncated bool) func MultipartAborted(ctx context.Context, span trace.Span, uploadID, key, backend string, partsCleaned int) func MultipartCompleted(ctx context.Context, span trace.Span, key, backend, uploadID string, totalSize int64, partsCount int) func MultipartCreated(ctx context.Context, span trace.Span, key, backend, uploadID string) func PutCompleted(ctx context.Context, span trace.Span, key, backend string, size int64) func UploadPartCompleted(ctx context.Context, span trace.Span, key, backend, uploadID string, partNumber int, size int64)https://s3-orchestrator.munchbox.cc/godoc/postgres/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/postgres/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/store/postgres" CB-aware sqlc DBTX wrapper for the postgres driver. Every sqlc query - pool-bound or tx-bound - flows through this single chokepoint, which calls breaker.PreCheck before the SQL and breaker.PostCheck after. Advisory locks bypass the breaker by going through pool.Acquire() directly (see advisory_lock.go). Package store provides PostgreSQL metadata persistence for the S3 orchestrator. metadata tracking, quota enforcement, circuit breaker protection, replication, and rebalancing.https://s3-orchestrator.munchbox.cc/godoc/proxy/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/proxy/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy" Package proxy is the domain orchestration layer that coordinates multi-backend S3 storage. It routes writes, manages failover reads, handles multipart uploads, drains backends, and exposes dashboard data. Workers receive the Ops interface instead of direct access. Index type BackendManager func NewBackendManager(cfg *BackendManagerConfig) *BackendManager func (m *BackendManager) AdmissionSem() chan struct{} func (m *BackendManager) ClearCache() func (m *BackendManager) ClearDrainState() func (m *BackendManager) Close() func (m *BackendManager) DeleteOrEnqueue(ctx context.Context, be backend.ObjectBackend, backendName, key, reason string, sizeBytes int64) func (m *BackendManager) FlushUsage(ctx context.Context) error func (m *BackendManager) GetDashboardData(ctx context.Context) (*dashboard.Data, error) func (m *BackendManager) GetDirectoryChildren(ctx context.Context, prefix, startAfter string, maxKeys int) (*core.DirectoryListResult, error) func (m *BackendManager) IntegrityConfig() *config.IntegrityConfig func (m *BackendManager) LifecycleConfig() *config.LifecycleConfig func (m *BackendManager) MoveObject(ctx context.Context, req *writepath.MoveRequest) (int64, error) func (m *BackendManager) NearUsageLimit(threshold float64) bool func (m *BackendManager) ProcessLifecycleRules(ctx context.Context, rules []config.LifecycleRule) (deleted, failed int) func (m *BackendManager) ReconcileBackend(ctx context.Context, backendName, bucket string, knownBuckets []string) (*worker.ReconcileResult, error) func (m *BackendManager) RecordUsage(backendName string, apiCalls, egress, ingress int64) func (m *BackendManager) RedisCounterConfigured() bool func (m *BackendManager) SelectReplicaTarget(ctx context.Context, size int64, exclusion map[string]bool) (string, error) func (m *BackendManager) SetIntegrityConfig(cfg *config.IntegrityConfig) func (m *BackendManager) SetLifecycleConfig(cfg *config.LifecycleConfig) func (m *BackendManager) SetUsageFlushConfig(cfg *config.UsageFlushConfig) func (m *BackendManager) Stores() core.MetadataStore func (m *BackendManager) SyncBackend(ctx context.Context, backendName, bucket string, knownBuckets []string) (imported, skipped int, err error) func (m *BackendManager) UpdateUsageLimits(limits map[string]core.UsageLimits) func (m *BackendManager) UsageFlushConfig() *config.UsageFlushConfig func (m *BackendManager) WireDrain(d *drain.Manager) type BackendManagerConfighttps://s3-orchestrator.munchbox.cc/godoc/proxytest/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/proxytest/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/proxytest" Package proxytest provides cross-package test helpers for the proxy package. Importing it from production code is not supported. Index func NewManager(t testing.TB, cfg *proxy.BackendManagerConfig) *proxy.BackendManager type Workers func BuildWorkers(mgr *proxy.BackendManager, m core.MetadataStore) *Workers func NewManager func NewManager(t testing.TB, cfg *proxy.BackendManagerConfig) *proxy.BackendManager NewManager builds a *proxy.BackendManager from cfg. Kept as a helper so cross-package call sites stay grouped under a single import even though the underlying constructor is a one-liner.https://s3-orchestrator.munchbox.cc/godoc/readpath/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/readpath/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/readpath" Index Variables func NoopCleanup() type Core type Failover func New(infraCore Core, stores core.MetadataStore, cache LocationCache, parallelBroadcast bool, degradedBroadcastParallelism int, degradedReadsEnabled bool) *Failover func (f *Failover) Read(ctx context.Context, operation, key string, probe Probe) (string, error) type LocationCache type Probe Variables ErrUsageLimitSkip is the sentinel a Probe returns when it declined to attempt a backend purely because its usage limit would be exceeded. Failover counts these separately from genuine failures so the final “all backends declined for usage limits” outcome can be reported as core.ErrUsageLimitExceeded rather than as the last underlying error.https://s3-orchestrator.munchbox.cc/godoc/reconcile/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/reconcile/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/reconcile" Index func DeleteHandler(log *slog.Logger, backendName string, deleter DeleterFn, result *Result) func(context.Context, string) error func ImportHandler(log *slog.Logger, backendName string, importer ImporterFn, result *Result) func(context.Context, Entry) error func SiblingPrefixes(knownBuckets []string, current string) []string func Sorted(ctx context.Context, s3, dbIter keySource, onImport func(ctx context.Context, e Entry) error, onDelete func(ctx context.Context, key string) error) error type DBCursorStream func NewDBCursorStream(s DBKeyLister, backendName, bucketPrefix string, otherPrefixes []string) *DBCursorStream func (d *DBCursorStream) Next(ctx context.Context) (Entry, bool, error) func (d *DBCursorStream) Stop() type DBKeyLister type DeleterFn type Entry type ImporterFn type ObjectLister type Result type S3KeyStream func NewS3KeyStream(ctx context.Context, s3b ObjectLister, bucketPrefix string, otherPrefixes []string, apiPages *int64) *S3KeyStream func (s *S3KeyStream) Next(ctx context.Context) (Entry, bool, error) func (s *S3KeyStream) Stop()https://s3-orchestrator.munchbox.cc/godoc/reload/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/reload/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/reload" Package reload owns SIGHUP-driven configuration reload. The coordinator runs a two-phase Check / Apply pass over a sequence of hooks, swaps the atomic config on success, and reports full / partial / validation / load outcomes via ReloadResult. Index type Coordinator func New(deps *Deps) *Coordinator func (c *Coordinator) Generation() int64 func (c *Coordinator) LastResult() *ReloadResult func (c *Coordinator) Reload() *ReloadResult func (c *Coordinator) Shutdown() func (c *Coordinator) Watch() type Deps type Hook type HookOutcome type HookStatus type ReloadResult type ReloadStatushttps://s3-orchestrator.munchbox.cc/godoc/runtime/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/runtime/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/runtime" Package runtime is the daemon composition root. It assembles every long-lived subsystem - observability, DI, the HTTP listener, background workers, the reload coordinator - and owns the shutdown order so the CLI entry point does not. Index type Observability type Options type Runtime func New(opts Options, cfg *config.Config) (*Runtime, error) func (r *Runtime) Run(ctx context.Context) errorhttps://s3-orchestrator.munchbox.cc/godoc/s3api/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/s3api/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/transport/s3api" Package server implements the S3-compatible HTTP API, routing requests to the storage backend manager with authentication, rate limiting, and tracing. Index func ExtractClientIP(r *http.Request, trustedProxies []*net.IPNet) string func ParseTrustedProxies(cidrs []string) []*net.IPNet func WriteS3Error(w http.ResponseWriter, code int, errCode, message string) type AdmissionController func NewAdmissionController(maxConcurrent int) *AdmissionController func NewAdmissionControllerFromSem(sem chan struct{}) *AdmissionController func NewSplitAdmissionController(maxReads, maxWrites int) *AdmissionController func NewSplitAdmissionControllerFromSem(readSem, writeSem chan struct{}) *AdmissionController func (ac *AdmissionController) Middleware(next http.Handler) http.Handler func (ac *AdmissionController) SetAdmissionWait(d time.Duration) func (ac *AdmissionController) SetShedThreshold(t float64) type RateLimiter func NewRateLimiter(cfg config.RateLimitConfig) *RateLimiter func (rl *RateLimiter) Allow(ip string) bool func (rl *RateLimiter) Close() func (rl *RateLimiter) Middleware(next http.Handler) http.Handler func (rl *RateLimiter) UpdateLimits(requestsPerSec float64, burst int) type Server func NewServer(manager *proxy.BackendManager, maxObjectSize int64) *Server func (s *Server) GetBucketAuth() *auth.BucketRegistry func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) func (s *Server) SetBucketAuth(br *auth.BucketRegistry)https://s3-orchestrator.munchbox.cc/godoc/serve/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/serve/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/cli/serve" Package serve implements the `s3-orchestrator serve` subcommand. It is a thin composition root: load the configuration file, construct a runtime.Runtime, and run it. All daemon assembly lives in internal/runtime. Index func Run(ctx context.Context, configPath, mode string, stdout io.Writer) error func Run func Run(ctx context.Context, configPath, mode string, stdout io.Writer) error Run is the daemon entry point. Loads config, hands control to the runtime package, and blocks until ctx is cancelled (SIGINT/SIGTERM) or the runtime returns an error. Errors are returned to the caller instead of calling os.Exit so this function is unit-testable.https://s3-orchestrator.munchbox.cc/godoc/sqlite/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/sqlite/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/store/sqlite" CB-aware *sql.DB wrapper for the sqlite driver. Every statement the store fires - direct or transaction-bound - flows through this single chokepoint, which calls breaker.PreCheck before the call and breaker.PostCheck after. Advisory locks emulate a process-local mutex and never touch *sql.DB, so they bypass the breaker. Package sqlite implements every core store role plus the admin roles using an embedded SQLite database via modernc.org/sqlite. WAL mode handles concurrent reads, a process-local mutex emulates advisory locks, and the schema migrates on first start.https://s3-orchestrator.munchbox.cc/godoc/store/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/store/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/store" Database circuit breaker factory plus the error filter used by the driver-level DBTX/DB wrappers. The CB instance is a first-class DI-registered value; consumers that need lifecycle controls (IsHealthy, ResetStaleProbe) invoke *breaker.CircuitBreaker directly. Index func NewDatabaseBreaker(cfg config.CircuitBreakerConfig) *breaker.CircuitBreaker func NewDatabaseBreaker func NewDatabaseBreaker(cfg config.CircuitBreakerConfig) *breaker.CircuitBreaker NewDatabaseBreaker returns the circuit breaker used to guard metadata store calls. It treats S3Error, ErrNoSpaceAvailable, and “no rows” sentinels as application-level signals that should not trip the circuit. Wires the telemetry hook so CircuitBreakerState / CircuitBreakerTransitionsTotal are populated.https://s3-orchestrator.munchbox.cc/godoc/storetest/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/storetest/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/store/storetest" Package storetest hosts the mockgen-generated MockMetadataStore used across the test suite as a single drop-in replacement for the hand-written wide store mocks that previously lived in internal/proxy, internal/store, internal/testutil, and internal/di. Package storetest is a generated GoMock package. Index func Permissive(m *MockMetadataStore) type MetadataStore type MockMetadataStore func NewMockMetadataStore(ctrl *gomock.Controller) *MockMetadataStore func (m *MockMetadataStore) BackendObjectStats(ctx context.Context, backendName string) (int64, int64, error) func (m *MockMetadataStore) ClaimPendingCleanups(ctx context.Context, limit int, instanceID string, graceCutoff time.Time) ([]core.CleanupItem, error) func (m *MockMetadataStore) CleanupDLQDepth(ctx context.Context) (int64, error) func (m *MockMetadataStore) CleanupQueueDepth(ctx context.Context) (int64, error) func (m *MockMetadataStore) Close() func (m *MockMetadataStore) CompleteCleanupItem(ctx context.Context, id int64) error func (m *MockMetadataStore) CompleteNotification(ctx context.Context, id int64) error func (m *MockMetadataStore) CountActiveMultipartUploads(ctx context.Context, bucketPrefix string) (int64, error) func (m *MockMetadataStore) CountOverReplicatedObjects(ctx context.Context, factor int) (int64, error) func (m *MockMetadataStore) CreateMultipartUpload(ctx context.Context, params *core.CreateMultipartUploadParams) error func (m *MockMetadataStore) DecrementOrphanBytes(ctx context.Context, backendName string, amount int64) error func (m *MockMetadataStore) DeleteBackendData(ctx context.Context, backendName string) error func (m *MockMetadataStore) DeleteMultipartUpload(ctx context.Context, uploadID string) error func (m *MockMetadataStore) DeleteObject(ctx context.Context, key string) ([]core.DeletedCopy, error) func (m *MockMetadataStore) DeleteObjectLocation(ctx context.Context, key, backendName string) error func (m *MockMetadataStore) DeleteObjectsBatch(ctx context.Context, keys []string) (map[string][]core.DeletedCopy, error) func (m *MockMetadataStore) DeletePending(ctx context.Context, intentID string) error func (m *MockMetadataStore) DeletePendingByBackend(ctx context.Context, backendName string) error func (m *MockMetadataStore) EXPECT() *MockMetadataStoreMockRecorder func (m *MockMetadataStore) EnqueueCleanup(ctx context.Context, backendName, objectKey, reason string, sizeBytes int64) error func (m *MockMetadataStore) FlushUsageDeltas(ctx context.Context, backendName, period string, apiRequests, egressBytes, ingressBytes int64) error func (m *MockMetadataStore) GetActiveMultipartCounts(ctx context.Context) (map[string]int64, error) func (m *MockMetadataStore) GetAllObjectLocations(ctx context.Context, key string) ([]core.ObjectLocation, error) func (m *MockMetadataStore) GetBackendWithSpace(ctx context.Context, size int64, backendOrder []string) (string, error) func (m *MockMetadataStore) GetLeastUtilizedBackend(ctx context.Context, size int64, eligible []string) (string, error) func (m *MockMetadataStore) GetMultipartUpload(ctx context.Context, uploadID string) (*core.MultipartUpload, error) func (m *MockMetadataStore) GetMultipartUploadsByBackend(ctx context.Context, backendName string) ([]core.MultipartUpload, error) func (m *MockMetadataStore) GetObjectBackendsForKeys(ctx context.Context, keys []string) (map[string][]string, error) func (m *MockMetadataStore) GetObjectCounts(ctx context.Context) (map[string]int64, error) func (m *MockMetadataStore) GetObjectsWithoutHash(ctx context.Context, limit, offset int) ([]core.ObjectLocation, error) func (m *MockMetadataStore) GetOverReplicatedObjects(ctx context.Context, factor, limit int) ([]core.ObjectLocation, error) func (m *MockMetadataStore) GetParts(ctx context.Context, uploadID string) ([]core.MultipartPart, error) func (m *MockMetadataStore) GetPendingCleanups(ctx context.Context, limit int) ([]core.CleanupItem, error) func (m *MockMetadataStore) GetPendingNotifications(ctx context.Context, limit int) ([]core.NotificationRow, error) func (m *MockMetadataStore) GetQuotaStats(ctx context.Context) (map[string]core.QuotaStat, error) func (m *MockMetadataStore) GetRandomHashedObjects(ctx context.Context, limit int) ([]core.ObjectLocation, error) func (m *MockMetadataStore) GetStaleMultipartUploads(ctx context.Context, olderThan time.Duration) ([]core.MultipartUpload, error) func (m *MockMetadataStore) GetStalePending(ctx context.Context, olderThan time.Time, limit int) ([]core.PendingObject, error) func (m *MockMetadataStore) GetUnderReplicatedObjects(ctx context.Context, factor, limit int) ([]core.ObjectLocation, error) func (m *MockMetadataStore) GetUnderReplicatedObjectsExcluding(ctx context.Context, factor, limit int, excludedBackends []string) ([]core.ObjectLocation, error) func (m *MockMetadataStore) GetUnverifiedObjectCounts(ctx context.Context) (map[string]int64, error) func (m *MockMetadataStore) GetUsageForPeriod(ctx context.Context, period string) (map[string]core.UsageStat, error) func (m *MockMetadataStore) ImportObject(ctx context.Context, key, backend string, size int64) (bool, error) func (m *MockMetadataStore) IncrementOrphanBytes(ctx context.Context, backendName string, amount int64) error func (m *MockMetadataStore) InsertNotification(ctx context.Context, eventType, payload, endpointURL string) error func (m *MockMetadataStore) InsertPending(ctx context.Context, p *core.PendingObject) error func (m *MockMetadataStore) ListAllEncryptedLocations(ctx context.Context, limit, offset int) ([]core.DecryptableLocation, error) func (m *MockMetadataStore) ListDirectoryChildren(ctx context.Context, prefix, startAfter string, maxKeys int) (*core.DirectoryListResult, error) func (m *MockMetadataStore) ListEncryptedLocations(ctx context.Context, keyID string, limit, offset int) ([]core.EncryptedLocation, error) func (m *MockMetadataStore) ListExpiredObjects(ctx context.Context, prefix string, cutoff time.Time, limit int) ([]core.ObjectLocation, error) func (m *MockMetadataStore) ListMultipartUploads(ctx context.Context, prefix string, maxUploads int) ([]core.MultipartUpload, error) func (m *MockMetadataStore) ListObjects(ctx context.Context, prefix, startAfter string, maxKeys int) (*core.ListObjectsResult, error) func (m *MockMetadataStore) ListObjectsByBackend(ctx context.Context, backendName string, limit int) ([]core.ObjectLocation, error) func (m *MockMetadataStore) ListObjectsByBackendKeyAsc(ctx context.Context, backendName, afterKey string, limit int) ([]core.ObjectLocation, error) func (m *MockMetadataStore) ListUnencryptedLocations(ctx context.Context, limit, offset int) ([]core.UnencryptedLocation, error) func (m *MockMetadataStore) MarkObjectDecrypted(ctx context.Context, objectKey, backendName string, plaintextSize int64) error func (m *MockMetadataStore) MarkObjectEncrypted(ctx context.Context, objectKey, backendName string, encryptionKey []byte, keyID string, plaintextSize, ciphertextSize int64) error func (m *MockMetadataStore) MoveCleanupToDLQ(ctx context.Context, id int64, lastError string) (bool, error) func (m *MockMetadataStore) MoveObjectLocation(ctx context.Context, key, fromBackend, toBackend string) (int64, error) func (m *MockMetadataStore) PendingDepth(ctx context.Context) (int64, error) func (m *MockMetadataStore) PromotePending(ctx context.Context, p *core.PendingObject) (core.PendingPromoteResult, []core.DeletedCopy, error) func (m *MockMetadataStore) RecordObject(ctx context.Context, key, backend string, size int64, enc *core.EncryptionMeta) ([]core.DeletedCopy, error) func (m *MockMetadataStore) RecordObjectAndClearPending(ctx context.Context, key, backend string, size int64, enc *core.EncryptionMeta, intentID string) ([]core.DeletedCopy, error) func (m *MockMetadataStore) RecordPart(ctx context.Context, uploadID string, partNumber int, etag string, size int64, enc *core.EncryptionMeta) error func (m *MockMetadataStore) RecordReplica(ctx context.Context, key, targetBackend, sourceBackend string) (int64, bool, error) func (m *MockMetadataStore) RemoveExcessCopy(ctx context.Context, key, backendName string, size int64) error func (m *MockMetadataStore) RetryCleanupItem(ctx context.Context, id int64, backoff time.Duration, lastError string) error func (m *MockMetadataStore) RetryNotification(ctx context.Context, id int64, backoff time.Duration, lastError string) error func (m *MockMetadataStore) RunMigrations(ctx context.Context) error func (m *MockMetadataStore) SweepStaleCleanupQueueRows(ctx context.Context, key, backend string) (int64, error) func (m *MockMetadataStore) SyncQuotaLimits(ctx context.Context, backends []config.BackendConfig) error func (m *MockMetadataStore) UpdateContentHash(ctx context.Context, key, backendName, hash string) error func (m *MockMetadataStore) UpdateEncryptionKey(ctx context.Context, objectKey, backendName string, newEncryptionKey []byte, newKeyID string) error func (m *MockMetadataStore) VerifySchemaVersion(ctx context.Context) error func (m *MockMetadataStore) WithAdvisoryLock(ctx context.Context, lockID int64, fn func(context.Context) error) (bool, error) type MockMetadataStoreMockRecorder func (mr *MockMetadataStoreMockRecorder) BackendObjectStats(ctx, backendName any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ClaimPendingCleanups(ctx, limit, instanceID, graceCutoff any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) CleanupDLQDepth(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) CleanupQueueDepth(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) Close() *gomock.Call func (mr *MockMetadataStoreMockRecorder) CompleteCleanupItem(ctx, id any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) CompleteNotification(ctx, id any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) CountActiveMultipartUploads(ctx, bucketPrefix any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) CountOverReplicatedObjects(ctx, factor any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) CreateMultipartUpload(ctx, params any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DecrementOrphanBytes(ctx, backendName, amount any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DeleteBackendData(ctx, backendName any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DeleteMultipartUpload(ctx, uploadID any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DeleteObject(ctx, key any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DeleteObjectLocation(ctx, key, backendName any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DeleteObjectsBatch(ctx, keys any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DeletePending(ctx, intentID any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) DeletePendingByBackend(ctx, backendName any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) EnqueueCleanup(ctx, backendName, objectKey, reason, sizeBytes any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) FlushUsageDeltas(ctx, backendName, period, apiRequests, egressBytes, ingressBytes any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetActiveMultipartCounts(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetAllObjectLocations(ctx, key any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetBackendWithSpace(ctx, size, backendOrder any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetLeastUtilizedBackend(ctx, size, eligible any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetMultipartUpload(ctx, uploadID any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetMultipartUploadsByBackend(ctx, backendName any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetObjectBackendsForKeys(ctx, keys any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetObjectCounts(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetObjectsWithoutHash(ctx, limit, offset any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetOverReplicatedObjects(ctx, factor, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetParts(ctx, uploadID any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetPendingCleanups(ctx, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetPendingNotifications(ctx, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetQuotaStats(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetRandomHashedObjects(ctx, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetStaleMultipartUploads(ctx, olderThan any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetStalePending(ctx, olderThan, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetUnderReplicatedObjects(ctx, factor, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetUnderReplicatedObjectsExcluding(ctx, factor, limit, excludedBackends any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetUnverifiedObjectCounts(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) GetUsageForPeriod(ctx, period any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ImportObject(ctx, key, backend, size any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) IncrementOrphanBytes(ctx, backendName, amount any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) InsertNotification(ctx, eventType, payload, endpointURL any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) InsertPending(ctx, p any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListAllEncryptedLocations(ctx, limit, offset any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListDirectoryChildren(ctx, prefix, startAfter, maxKeys any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListEncryptedLocations(ctx, keyID, limit, offset any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListExpiredObjects(ctx, prefix, cutoff, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListMultipartUploads(ctx, prefix, maxUploads any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListObjects(ctx, prefix, startAfter, maxKeys any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListObjectsByBackend(ctx, backendName, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListObjectsByBackendKeyAsc(ctx, backendName, afterKey, limit any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) ListUnencryptedLocations(ctx, limit, offset any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) MarkObjectDecrypted(ctx, objectKey, backendName, plaintextSize any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) MarkObjectEncrypted(ctx, objectKey, backendName, encryptionKey, keyID, plaintextSize, ciphertextSize any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) MoveCleanupToDLQ(ctx, id, lastError any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) MoveObjectLocation(ctx, key, fromBackend, toBackend any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) PendingDepth(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) PromotePending(ctx, p any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RecordObject(ctx, key, backend, size, enc any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RecordObjectAndClearPending(ctx, key, backend, size, enc, intentID any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RecordPart(ctx, uploadID, partNumber, etag, size, enc any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RecordReplica(ctx, key, targetBackend, sourceBackend any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RemoveExcessCopy(ctx, key, backendName, size any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RetryCleanupItem(ctx, id, backoff, lastError any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RetryNotification(ctx, id, backoff, lastError any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) RunMigrations(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) SweepStaleCleanupQueueRows(ctx, key, backend any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) SyncQuotaLimits(ctx, backends any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) UpdateContentHash(ctx, key, backendName, hash any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) UpdateEncryptionKey(ctx, objectKey, backendName, newEncryptionKey, newKeyID any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) VerifySchemaVersion(ctx any) *gomock.Call func (mr *MockMetadataStoreMockRecorder) WithAdvisoryLock(ctx, lockID, fn any) *gomock.Callhttps://s3-orchestrator.munchbox.cc/godoc/synccmd/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/synccmd/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/cli/synccmd" Package synccmd implements the `s3-orchestrator sync` subcommand, which imports objects already present on a backend bucket into the metadata store so the proxy can serve them. Index func Run(args []string, stderr io.Writer) int type Options func Run func Run(args []string, stderr io.Writer) int Run is the CLI entry point. It parses the sync flags, opens the database, and walks the backend, returning the process exit code.https://s3-orchestrator.munchbox.cc/godoc/syncutil/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/syncutil/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/util/syncutil" Package syncutil provides small concurrency primitives the rest of the codebase shares: AtomicConfig wraps atomic.Pointer[T] for hot-reloadable config, and TTLCache implements a generic time-bounded cache with background eviction. Index type AtomicConfig func (a *AtomicConfig[T]) Load() *T func (a *AtomicConfig[T]) Store(val *T) type TTLCache func NewTTLCache[K comparable, V any](ttl time.Duration) *TTLCache[K, V] func (c *TTLCache[K, V]) Clear() func (c *TTLCache[K, V]) Close() func (c *TTLCache[K, V]) Delete(key K) func (c *TTLCache[K, V]) Get(key K) (V, bool) func (c *TTLCache[K, V]) Len() int func (c *TTLCache[K, V]) Set(key K, value V) func (c *TTLCache[K, V]) SetWithTTL(key K, value V, ttl time.Duration)https://s3-orchestrator.munchbox.cc/godoc/telemetry/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/telemetry/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/observe/telemetry" Package telemetry provides Prometheus metrics registration and OpenTelemetry tracing initialization for the S3 orchestrator. Package telemetry provides Prometheus metrics registration and OpenTelemetry tracing initialization for the S3 orchestrator. Index Constants Variables func BackendAttributes(operation, backendName, endpoint, bucket, key string) []attribute.KeyValue func InitTracer(ctx context.Context, cfg config.TracingConfig) (func(context.Context) error, error) func NewCircuitBreakerHook(name string) func(breaker.StateChangeInfo) func NewDatabaseBreakerHook(name string) func(breaker.StateChangeInfo) func RequestAttributes(method, path, bucket, key, clientIP string) []attribute.KeyValue func StartClientSpan(ctx context.Context, name string, attrs …attribute.KeyValue) (context.Context, trace.Span) func StartServerSpan(ctx context.Context, name string, attrs …attribute.KeyValue) (context.Context, trace.Span) func StartSpan(ctx context.Context, name string, attrs …attribute.KeyValue) (context.Context, trace.Span) func Tracer() trace.Tracer type LogBuffer func NewLogBuffer() *LogBuffer func (b *LogBuffer) Add(entry LogEntry) func (b *LogBuffer) Entries(opts *LogQueryOpts) []LogEntry type LogEntry type LogQueryOpts type TeeHandler func NewTeeHandler(primary slog.Handler, buf *LogBuffer) *TeeHandler func (h *TeeHandler) Enabled(ctx context.Context, level slog.Level) bool func (h *TeeHandler) Handle(ctx context.Context, r slog.Record) error func (h *TeeHandler) WithAttrs(attrs []slog.Attr) slog.Handler func (h *TeeHandler) WithGroup(name string) slog.Handler type TraceHandler func NewTraceHandler(inner slog.Handler) *TraceHandler func (h *TraceHandler) Enabled(ctx context.Context, level slog.Level) bool func (h *TraceHandler) Handle(ctx context.Context, r slog.Record) error func (h *TraceHandler) WithAttrs(attrs []slog.Attr) slog.Handler func (h *TraceHandler) WithGroup(name string) slog.Handler Constants TracerName and related constants used by this package.https://s3-orchestrator.munchbox.cc/godoc/tickrunner/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/tickrunner/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/lifecycle/tickrunner" Index Constants func ComponentLogger(slug string) *slog.Logger func HandlePassResult(ctx context.Context, log *slog.Logger, manager QuotaMetricsRefresher, count int, err error, countKey string) error type AdvisoryLocker type Config type QuotaMetricsRefresher type Service func New(cfg Config) *Service func (s *Service) Health() lifecycle.WorkerHealth func (s *Service) Interval() time.Duration func (s *Service) LockID() int64 func (s *Service) Name() string func (s *Service) Run(ctx context.Context) error func (s *Service) Tick(ctx context.Context) Constants MsgPassFailed / MsgPassCompleted / MsgQuotaMetricsRefreshFailed are the canonical terminal-event log messages shared by the periodic “pass” workers (rebalance, replication, over-replication). Held as constants so the three nearly-identical work closures cannot drift.https://s3-orchestrator.munchbox.cc/godoc/ui/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/ui/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/transport/ui" Package ui provides the built-in web dashboard for operational visibility, serving HTML pages, JSON API endpoints, and static assets. Index type BackendOps type Deps type Handler func New(d *Deps) *Handler func (h *Handler) Register(mux *http.ServeMux, prefix string) func (h *Handler) UpdateConfig(cfg *config.Config) type BackendOps BackendOps is the narrow surface of *proxy.BackendManager that the UI dashboard depends on for operations not exposed via a named sub-manager. *proxy.BackendManager satisfies it.https://s3-orchestrator.munchbox.cc/godoc/worker/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/worker/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/worker" Index Constants func CleanupBackoff(attempts int32) time.Duration func ExceedsThreshold(stats map[string]core.QuotaStat, order []string, threshold float64) bool func NewCleanupQueueService(cleanup *CleanupWorker, locker tickrunner.AdvisoryLocker) lifecycle.Runner func NewOverReplicationService(manager tickrunner.QuotaMetricsRefresher, overRep *OverReplicationCleaner, locker tickrunner.AdvisoryLocker) lifecycle.Runner func NewPendingReaperService(reaper *PendingReaper, locker tickrunner.AdvisoryLocker, tick time.Duration) lifecycle.Runner func NewRebalancerService(manager tickrunner.QuotaMetricsRefresher, rebalancer *Rebalancer, locker tickrunner.AdvisoryLocker) lifecycle.Runner func NewReconcileService(reconciler *Reconciler, locker tickrunner.AdvisoryLocker, interval time.Duration) lifecycle.Runner func NewReplicatorService(manager tickrunner.QuotaMetricsRefresher, replicator *Replicator, locker tickrunner.AdvisoryLocker) lifecycle.Runner func NewScrubberService(scrubber *Scrubber, locker tickrunner.AdvisoryLocker) lifecycle.Runner func WithAdmission(ctx context.Context, ac AdmissionControl, name string, fn func()) type AdmissionControl type BackendAccess type BackendSyncer type CleanupOps type CleanupWorker func NewCleanupWorker(deps CleanupOps, store core.MetadataStore, concurrency int, instanceID string, claimGracePeriod time.Duration) *CleanupWorker func (w *CleanupWorker) ProcessCleanupQueue(ctx context.Context) (processed, failed int) type DataMover type Ops type OverReplicationCleaner func NewOverReplicationCleaner(ops Ops, store core.MetadataStore) *OverReplicationCleaner func (c *OverReplicationCleaner) Clean(ctx context.Context, cfg config.ReplicationConfig) (int, error) func (c *OverReplicationCleaner) Config() *config.ReplicationConfig func (c *OverReplicationCleaner) CountPending(ctx context.Context, factor int) (int64, error) func (c *OverReplicationCleaner) ScoreCopy(loc *core.ObjectLocation, stats map[string]core.QuotaStat) float64 func (c *OverReplicationCleaner) SetConfig(cfg *config.ReplicationConfig) type PendingReaper func NewPendingReaper(deps CleanupOps, store core.MetadataStore, concurrency int, minAge time.Duration, batchSize int) *PendingReaper func (r *PendingReaper) ProcessPendingQueue(ctx context.Context) (resolved, failed int) type RebalanceMove type Rebalancer func NewRebalancer(ops Ops, store core.MetadataStore) *Rebalancer func (r *Rebalancer) Config() *config.RebalanceConfig func (r *Rebalancer) ExecuteMoves(ctx context.Context, plan []RebalanceMove, strategy string, concurrency int) int func (r *Rebalancer) ExecuteOneMove(ctx context.Context, move RebalanceMove, strategy string) bool func (r *Rebalancer) PlanPackTight(ctx context.Context, stats map[string]core.QuotaStat, batchSize int) ([]RebalanceMove, error) func (r *Rebalancer) PlanSpreadEven(ctx context.Context, stats map[string]core.QuotaStat, batchSize int) ([]RebalanceMove, error) func (r *Rebalancer) Rebalance(ctx context.Context, cfg config.RebalanceConfig) (int, error) func (r *Rebalancer) SetConfig(cfg *config.RebalanceConfig) type ReconcileResult type Reconciler func NewReconciler(syncer BackendSyncer, bucketNames []string) *Reconciler func (r *Reconciler) Reconcile(ctx context.Context, backendName string) (*ReconcileResult, error) func (r *Reconciler) Run(ctx context.Context) type RecorderProvider type ReplicationOutcome func (o ReplicationOutcome) Failed() int type Replicator func NewReplicator(ops Ops, store core.MetadataStore) *Replicator func (r *Replicator) CleanupOrphan(ctx context.Context, backendName, key string, sizeBytes int64) func (r *Replicator) Config() *config.ReplicationConfig func (r *Replicator) CopyToReplica(ctx context.Context, key string, copies []core.ObjectLocation, target string) (string, int64, error) func (r *Replicator) FindReplicaTarget(ctx context.Context, key string, size int64, exclusion map[string]bool) string func (r *Replicator) IsBackendHealthy(name string) bool func (r *Replicator) Replicate(ctx context.Context, cfg config.ReplicationConfig) (int, error) func (r *Replicator) ReplicateObject(ctx context.Context, key string, existingCopies []core.ObjectLocation, needed int) ReplicationOutcome func (r *Replicator) SetConfig(cfg *config.ReplicationConfig) func (r *Replicator) UnhealthyBackends(threshold time.Duration) []string type Scrubber func NewScrubber(deps ScrubberOps, store core.MetadataStore, encryptor *encryption.Encryptor) *Scrubber func (s *Scrubber) Backfill(ctx context.Context, batchSize, offset int) (processed, nextOffset int) func (s *Scrubber) Config() *config.IntegrityConfig func (s *Scrubber) Scrub(ctx context.Context, batchSize int) (checked, failed int) func (s *Scrubber) SetConfig(cfg *config.IntegrityConfig) type ScrubberOps type UsageAccessor Constants Worker name labels used by admission-rejection metric and other worker-scoped telemetry. Constants prevent the silent label drift that would otherwise split a metric’s time series in Grafana when a typo lands.https://s3-orchestrator.munchbox.cc/godoc/workerpool/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/workerpool/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/util/workerpool" Package workerpool provides generic bounded-concurrency worker pool functions for parallel processing of work items. Context cancellation stops dispatching new work; in-flight items run to completion. Index func Run[T any](ctx context.Context, concurrency int, items []T, fn func(context.Context, T)) func Run func Run[T any](ctx context.Context, concurrency int, items []T, fn func(context.Context, T)) Run processes items concurrently with bounded parallelism. The fn callback is invoked once per item. If ctx is cancelled, remaining undispatched items are skipped. In-flight items run to completion.https://s3-orchestrator.munchbox.cc/godoc/writepath/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://s3-orchestrator.munchbox.cc/godoc/writepath/index.htmlimport "github.com/afreidah/s3-orchestrator/internal/proxy/writepath" Index Variables type Coordinator func New(core WritepathCore, stores core.MetadataStore, pendingEnabled bool) *Coordinator func (w *Coordinator) DeleteOrEnqueue(ctx context.Context, be backend.ObjectBackend, backendName, key, reason string, sizeBytes int64) func (w *Coordinator) EnqueueCleanup(ctx context.Context, backendName, objectKey, reason string, sizeBytes int64) func (w *Coordinator) InsertPendingIntent(ctx context.Context, key, backendName string, size int64, enc *core.EncryptionMeta) (string, error) func (w *Coordinator) MoveObject(ctx context.Context, req *MoveRequest) (int64, error) func (w *Coordinator) RecordObjectAndPromoteIntent(ctx context.Context, span trace.Span, key, backendName string, size int64, enc *core.EncryptionMeta, intentID string) error func (w *Coordinator) RecordObjectOrCleanup(ctx context.Context, span trace.Span, be backend.ObjectBackend, key, backendName string, size int64, enc *core.EncryptionMeta) error func (w *Coordinator) RecoverFromRecordFailure(ctx context.Context, be backend.ObjectBackend, backendName, key, cleanupReason string, size int64) func (w *Coordinator) SelectBackendForWrite(ctx context.Context, size int64, eligible []string) (string, error) func (w *Coordinator) SelectWriteTarget(ctx context.Context, span trace.Span, operation string, size int64) (string, error) func (w *Coordinator) SetPendingEnabledForTest(enabled bool) type MoveRequest type WritepathCore Variables ErrMoveStale signals MoveObject was raced: MoveObjectLocation returned movedSize=0, meaning another process (or the same caller from a prior tick) already moved or deleted the object. The destination has had its now-orphaned bytes enqueued for cleanup via req.StaleOrphanReason. Callers treat this as a no-op rather than a failure - increment a “stale” / “skipped” counter rather than an error counter.