materialize

import "github.com/afreidah/s3-orchestrator/internal/util/materialize"

Package materialize turns a one-shot stream into a seekable, re-readable payload without scaling heap with object size: bytes below MemThreshold stay in memory, larger payloads spill to a self-unlinking tempfile. Body.Reader hands back a fresh io.ReadSeeker positioned at offset 0 on every call, so callers that must replay a body - PUT failover, encryption layers, and the AWS SDK’s SigV4 payload-hash pass for signed uploads - can consume it repeatedly. The sink is engine- and transport-neutral so both the backend adapters and the proxy write path share one implementation.

Index

• Constants
• type Body
  • func New(src io.Reader, size int64, hasher hash.Hash) (*Body, error)
  • func NewEmpty(size int64) (*Body, error)
  • func (b *Body) Cleanup()
  • func (b *Body) Reader() (io.ReadSeeker, error)
  • func (b *Body) Size() int64
  • func (b *Body) Writer() io.Writer

Constants

MemThreshold is the largest payload size kept entirely in memory before the sink spills to a tempfile. Sized to match the AWS SDK’s own internal heuristic for the PUT signing path. Not a config knob because the choice is an implementation detail, not an operator concern.

const MemThreshold = 32 * 1024 * 1024

type Body

Body holds a payload buffered into memory or onto disk, and serves io.Readers positioned at offset 0 on each call. The caller invokes Cleanup once the payload is no longer needed (always safe to defer, even on a materialization error).

type Body struct {
    // contains filtered or unexported fields
}

func New

func New(src io.Reader, size int64, hasher hash.Hash) (*Body, error)

New copies src into a memory buffer or a tempfile based on size, optionally tee’ing the bytes into hasher so the caller can compute a content hash in the same single pass instead of re-scanning the materialized body afterwards. Pass hasher=nil when hashing is not required. The caller must defer (*Body).Cleanup on the returned body so the tempfile fd is released when the body is no longer needed (safe even on the in-memory branch).

func NewEmpty

func NewEmpty(size int64) (*Body, error)

NewEmpty allocates the underlying sink without writing any bytes. Exposed for code paths that want to drive the write themselves (e.g. materializing a foreign GetObject body where the integrity hashing is owned by a different layer).

func (*Body) Cleanup

func (b *Body) Cleanup()

Cleanup releases the underlying tempfile when the sink spilled to disk; the in-memory branch has no fd to release and the buffer is reclaimed by the GC when the Body goes out of scope. Always safe to defer regardless of which branch backs the body.

func (*Body) Reader

func (b *Body) Reader() (io.ReadSeeker, error)

Reader returns a fresh io.ReadSeeker positioned at offset 0. Safe to call repeatedly so failover attempts and encryption layers can each consume the body independently. The in-memory variant returns a fresh *bytes.Reader; the tempfile variant rewinds the underlying file before returning.

func (*Body) Size

func (b *Body) Size() int64

Size returns the number of bytes written to the sink. For the in-memory sink this is len(buf); for the tempfile sink this is the byte count returned by the copy.

func (*Body) Writer

func (b *Body) Writer() io.Writer

Writer returns the io.Writer the caller streams source bytes into. Only meaningful when NewEmpty was used to construct the sink; New owns its own write loop.

Generated by gomarkdoc