breaker

import "github.com/afreidah/s3-orchestrator/internal/breaker"

Package breaker implements a generic three-state circuit breaker (closed, open, half-open) with pluggable error filters and probe jitter. The breaker emits no metrics or events on its own - callers wire those via the optional OnStateChange callback so this package stays free of observability dependencies.

Index

Constants

DefaultWatchdogInterval is the cadence at which the watchdog inspects every registered breaker. Picked as half the breaker probe timeout so a stuck half-open state is detected within one full probe window.

const DefaultWatchdogInterval = 1 * time.Minute

Variables

ErrBackendUnavailable is returned by CircuitBreakerBackend when the circuit is open (backend is known to be unreachable or returning errors).

var ErrBackendUnavailable = errors.New("backend unavailable")

func CBCall

func CBCall[T any](cb *CircuitBreaker, fn func() (T, error)) (T, error)

CBCall wraps a call that returns (T, error) with circuit breaker logic.

func CBCallNoResult

func CBCallNoResult(cb *CircuitBreaker, fn func() error) error

CBCallNoResult wraps a call that returns only error with circuit breaker logic.

func NewWatchdog

func NewWatchdog(registry *Registry) lifecycle.Runner

NewWatchdog constructs the watchdog background service. The registry holds every breaker that should be inspected on a tick - membership is decided once at DI construction time.

type CircuitBreaker

CircuitBreaker implements a three-state circuit breaker with a pluggable error filter. It is safe for concurrent use. Observability hooks live behind the optional OnStateChange callback - the breaker package itself imports nothing from observe/.

type CircuitBreaker struct {
    // contains filtered or unexported fields
}

func NewCircuitBreaker

func NewCircuitBreaker(name string, threshold int, timeout time.Duration, isError func(error) bool, sentinel error) *CircuitBreaker

NewCircuitBreaker creates a new circuit breaker.

name: identifier for logging and labels (e.g. “database”, “oci-backend”)
threshold: consecutive failures before opening
timeout: delay before probing recovery
isError: filter that returns true for errors that should count as failures
sentinel: the error returned when the circuit is open (e.g. ErrDBUnavailable)

Use SetOnStateChange after construction to install metric / event hooks.

func (*CircuitBreaker) AddOnStateChange

func (cb *CircuitBreaker) AddOnStateChange(fn func(StateChangeInfo))

AddOnStateChange appends a callback to the existing hook chain. Used to attach an additional listener (e.g. cache invalidation on recovery) without disturbing the telemetry hook already installed.

func (*CircuitBreaker) IsHealthy

func (cb *CircuitBreaker) IsHealthy() bool

IsHealthy returns true when the circuit is closed.

func (*CircuitBreaker) Name

func (cb *CircuitBreaker) Name() string

Name returns the breaker’s identifier - useful for callers wiring observability hooks that need the label.

func (*CircuitBreaker) OpenDuration

func (cb *CircuitBreaker) OpenDuration() time.Duration

OpenDuration returns how long the circuit has been open or half-open. Returns 0 when the circuit is closed (healthy).

func (*CircuitBreaker) PostCheck

func (cb *CircuitBreaker) PostCheck(err error) error

PostCheck records the result of a real call and transitions state. When an error causes the circuit to open (or reopen), the original error is replaced with the sentinel so callers always see the canonical error.

func (*CircuitBreaker) PreCheck

func (cb *CircuitBreaker) PreCheck() error

PreCheck returns the sentinel error when the circuit is open. Transitions open -> half-open when the timeout has elapsed, allowing one probe request. If a previous probe has been in flight longer than probeTimeout, it is considered abandoned and a new probe is allowed.

func (*CircuitBreaker) ProbeEligible

func (cb *CircuitBreaker) ProbeEligible() bool

ProbeEligible returns true when the circuit is open and the open timeout has elapsed, meaning the next request should be allowed through as a probe. This is a read-only check with no side effects - the actual state transition happens in PreCheck when the request is dispatched.

func (*CircuitBreaker) Recover

func (cb *CircuitBreaker) Recover()

Recover transitions the breaker back to Closed cleanly, regardless of the current state. The intended caller is an out-of-band liveness probe that has independently confirmed the dependency is reachable (e.g., the Redis counter recovery probe in internal/counter/redis.go). PostCheck(nil) is the wrong tool for this case: it models a single in-flight call’s success and only handles HalfOpen->Closed, leaving an Open breaker stuck. Recover clears probe state and zeroes the failure counter so the breaker tolerates the configured threshold of new failures before re-opening.

func (*CircuitBreaker) ResetStaleProbe

func (cb *CircuitBreaker) ResetStaleProbe() bool

ResetStaleProbe checks whether a half-open probe has exceeded probeTimeout and resets the circuit to open if so. Called by the background watchdog service to ensure stale probes are detected even when no new requests arrive. Returns true if a stale probe was reset.

func (*CircuitBreaker) SetOnStateChange

func (cb *CircuitBreaker) SetOnStateChange(fn func(StateChangeInfo))

SetOnStateChange installs a callback invoked on every closed/open/ half-open transition. The callback runs synchronously while the breaker holds its lock, so implementations must be cheap and non-blocking. Passing nil clears any previously installed callback.

func (*CircuitBreaker) State

func (cb *CircuitBreaker) State() State

State returns the current circuit state.

type Registry

Registry is a thread-safe collection of breakers swept by the watchdog.

type Registry struct {
    // contains filtered or unexported fields
}

func NewRegistry

func NewRegistry(initial ...StaleProbeResetter) *Registry

NewRegistry constructs a Registry preloaded with the given breakers. Nil entries are silently dropped.

func (*Registry) Len

func (r *Registry) Len() int

Len returns the number of registered breakers.

func (*Registry) Register

func (r *Registry) Register(b StaleProbeResetter)

func (*Registry) ResetStaleProbes

func (r *Registry) ResetStaleProbes()

ResetStaleProbes invokes ResetStaleProbe on every registered breaker. Safe for concurrent use.

type StaleProbeResetter

StaleProbeResetter is implemented by anything whose half-open probe can time out and need a manual reset. Both *CircuitBreaker and backend.CircuitBreakerBackend (which embeds *CircuitBreaker) satisfy it.

type StaleProbeResetter interface {
    // ResetStaleProbe resets a half-open probe that has exceeded its timeout.
    // Returns true when a probe was actually reset.
    ResetStaleProbe() bool
}

type State

State represents the current circuit breaker state.

type State int

StateClosed and related constants used by this package.

const (
    StateClosed   State = iota // healthy  -  all calls pass through
    StateOpen                  // down  -  return sentinel error
    StateHalfOpen              // probing  -  one call allowed through
)

func (State) String

func (s State) String() string

String returns the human-readable name of the circuit state.

type StateChangeInfo

StateChangeInfo captures the context the breaker shares with its OnStateChange callback. Failures and OpenDuration are read-only snapshots of the breaker’s state at the moment of the transition.

type StateChangeInfo struct {
    Name         string
    From         State
    To           State
    Failures     int
    Threshold    int
    OpenDuration time.Duration // time spent open or half-open before reaching this state; zero when From was Closed
}

Generated by gomarkdoc