s3-orchestrator

workerpool

import "github.com/afreidah/s3-orchestrator/internal/util/workerpool"

Package workerpool provides generic bounded-concurrency worker pool functions for parallel processing of work items. Context cancellation stops dispatching new work; in-flight items run to completion.

Index

func Run

func Run[T any](ctx context.Context, concurrency int, items []T, fn func(context.Context, T))

Run processes items concurrently with bounded parallelism. The fn callback is invoked once per item. If ctx is cancelled, remaining undispatched items are skipped. In-flight items run to completion.

Implementation note (#861): spawns at most min(concurrency, len(items)) worker goroutines that consume from a shared jobs channel until the dispatcher closes it. The earlier design spawned one goroutine per item bounded by a counting semaphore, which produced N goroutines for N items even though only `concurrency` ran at once - tens of thousands of transient goroutines on the larger cleanup/replication batches. The fixed-worker model keeps the same external semantics but caps goroutine churn at the worker count.

Generated by gomarkdoc