perf: pre-size reads and skip redundant copy to cut sync memory

[reflog]

Problem

Several 1 GB proxy/VPS hosts were tripping the SigNoz "Host memory usage high" alert (10 m avg system.memory.usage > 85%). Heap pprof from both http-proxy and lantern-box binaries pointed at the same culprit — the dominant live allocation was io.ReadAll, and every byte of it came from keepcurrent:

io.ReadAll
io/ioutil.ReadAll (inline)
github.com/getlantern/keepcurrent.(*byteChannel).UpdateFrom
github.com/getlantern/keepcurrent.(*Runner).syncOnce
github.com/getlantern/keepcurrent.(*Runner).Start.func2

• http-proxy: 94.9 MB / 146 MB heap (65%)
• lantern-box: 75 MB / 173 MB heap (44%)

The traces showed multiple large retained generations (75 + 60 + 48 + 38 + 30 + 24 MB …).

The source is getlantern/geo, which syncs the MaxMind GeoLite2 mmdb (~75 MB) via keepcurrent.FromTarGz(keepcurrent.FromWeb(dbURL), name) into two sinks (ToChannel(chDB) + ToFile(path)). Per sync that payload was buffered with io.ReadAll up to three times:

1. extracting the file from the tarball (tarGzSource.Fetch),
2. in syncOnce (data, _ = ioutil.ReadAll(rc)),
3. again in byteChannel.UpdateFrom, re-reading the bytes.Reader the Runner already handed it into a brand-new slice.

io.ReadAll grows its buffer by repeated reallocation, so each read churns through a sequence of ever-larger backing arrays (the 75/60/48/38… generations) before settling — and step 3 is a wholly redundant full copy. On a 1 GB box this transient churn dominates RSS.

Fix

No public API change.

1. readAll() (new read.go) — reads into a buffer pre-sized from the reader's known length, turning a multi-realloc read into a single allocation. Recognizes:
   • sizedReadCloser (HTTP Content-Length, archive entry size — constructed internally),
   • standard in-memory readers via Len() (*bytes.Reader, *bytes.Buffer, *strings.Reader).
   • Falls back to io.ReadAll when size is unknown.
2. Size-aware sources — webSource.Fetch surfaces Content-Length; tarGzSource.Fetch pre-sizes extraction from the archive entry's uncompressed size and returns a size-aware reader, so syncOnce's read is pre-sized too.
3. bytesConsumer fast-path — byteChannel receives the payload the Runner already buffered directly (updateFromBytes) instead of reading it into a second copy. The slice is read-only and shared across sinks (documented on the interface). This removes copy #3 entirely.

Net effect for the geo case: three doubling-realloc reads of a 75 MB payload → one extraction read + one pre-sized syncOnce read, with the channel sink reusing that buffer.

Test

• go build ./..., go vet ./... clean.
• Existing suite passes; added read_test.go covering pre-sizing for sized readers, fallback for unsized readers, and knownSize.

Rollout

Downstream bumps to follow once merged: getlantern/lantern-box and getlantern/http-proxy-lantern (both pin keepcurrent indirectly).

[Copilot]

Pull request overview

This PR reduces sync-time memory churn by pre-sizing large reads when a reader’s remaining size is known and by avoiding a redundant full-buffer copy when sending the already-buffered payload to channel sinks.

Changes:

• Add readAll()/knownSize() helpers to pre-size buffers and introduce an internal bytesConsumer fast-path for sinks.
• Surface size information from webSource (HTTP Content-Length) and tarGzSource (archive entry size) so Runner reads can be pre-sized.
• Update byteChannel to consume the already-buffered payload directly and add tests around the new read helper behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
source.go	Wrap HTTP bodies / extracted archive entries in size-aware readers to enable pre-sized reads.
sink.go	Route byteChannel through readAll() and add a no-copy bytesConsumer path.
read.go	Introduce readAll(), knownSize(), and internal bytesConsumer optimization.
read_test.go	Add tests for readAll() pre-sizing and knownSize() behavior.
keepcurrent.go	Switch Runner to readAll() and use bytesConsumer when available to skip redundant copies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

source.go:55

• The 304 / non-200 early-return paths only call resp.Body.Close(), but the comment claims “Drain+close so the connection can be reused for keep-alive”. In Go’s net/http transport, connections are generally only eligible for reuse when the response body is read to EOF and closed. Either drain the body before closing (to match the comment and enable keep-alive reuse) or adjust the comment to avoid implying reuse.

	}
	if s.getETag() != "" {
		req.Header.Add("If-None-Match", s.etag)
	}
	resp, err := s.client.Do(req)
	if err != nil {