Pimp My IDE / Garage logBack to dispatches
Runtime and infra05 Oct 20267 min read
Garage log 197 / latency attribution

Pause provenance bay

A garbage collector can wear the blame for a pause that the storage path supplied. Keep the runtime event, page faults, memory pressure, and request impact on one clock.

The practical call: do not tune the collector until you can place the pause and the fault time inside the same measured interval.
Stop-the-world traceWorst observed run
228 major faults
39,902 usPause
39,013 usFault time
97.8%Measured share
What happened

The pause crossed a storage boundary.

Fernando Simoes built a controlled Go workload under cgroup memory pressure. The median collector pause was about 51 microseconds. One faulted run reached 39,902 microseconds.

The experiment put a Go process and a quiet HTTP server in one cgroup. The Go process built a pointer-rich object graph. Memory pressure pushed cold pages to swap. When the runtime entered a stop-the-world phase and touched collector metadata, the kernel had to fetch those pages from storage.

A BPF probe counted 228 page faults inside the worst pause. It attributed 39,013 of the 39,902 microseconds to those faults. That is the useful shape of the result. The runtime pause supplied the critical section. Evicted pages supplied most of its duration.

A pause label names the event. It does not name the cause.

The application path paid a separate bill

The same report measured a 511 KiB message that usually took 3 to 5 milliseconds to build. Under the tested conditions it took 105 milliseconds on NVMe and 903 milliseconds on a network volume. The author did not confirm where all of that time went. Keep it as an observed application delay, not a proved page-fault attribution.

This distinction prevents a common postmortem failure. One measured interval can support a causal claim when the fault probe covers that interval. A nearby slowdown needs its own trace.

Go and Linux expose different witnesses

The Go garbage collector guide recommends execution traces for rare latency events and documents GODEBUG=gctrace=1 for collector-specific records. Linux Pressure Stall Information reports CPU, memory, and I/O stall time for the system or a cgroup. Neither witness replaces the other.

Use the runtime trace to locate the collector phase. Use fault tracing to account for blocked time. Use pressure data to show whether the host or cgroup was starved. Then attach the affected request, queue, or message so the infrastructure event has a product consequence.

01 / EVENT

What stopped?

Record the exact collector phase, start time, end time, Go version, and process.

02 / FAULT

What blocked it?

Count faults inside the same interval. Save fault time, sites, backing device, and cgroup.

03 / PRESSURE

What was scarce?

Capture memory and I/O pressure, swap activity, limits, and the competing workload.

04 / IMPACT

What did users feel?

Link the pause to one request, message, deadline, queue, or service-level objective.

Interactive makeover / causal trace rack

Build the incident packet

A normal latency dashboard puts charts beside each other. This rack makes each selected witness join one comparison bus and writes the missing fields into a copyable plan.

Select sections for the trace plan

This control writes a trace plan. It does not inspect a host, collect BPF data, or prove that page faults caused your pause.

Physical state / one comparison bus

Join the clocks

Every rail enters the same bus. The final state means that the packet structure is complete. Real outputs and review are still required.

1 of 4 sections selectedTrace plan started
RuntimeSelected
FaultsOpen
PressureOpen
ImpactOpen
Packet openThree witness sections are still open.
Shop notes

Do not fix the label.

A collector knob can change cycle frequency or memory use. It cannot make a storage read disappear merely because the stall occurred during collection.

Start with one narrow replay under recorded memory limits. Capture a runtime trace and a fault trace against the same monotonic clock. Save the cgroup pressure files before, during, and after the spike. Tag the request or message that overlaps the pause.

Change one condition at a time. Test the current memory limit, a larger limit, the same limit without the competing process, and the same workload with swap behavior changed. Keep request results and fault counts beside every run.

A good mitigation closes the user-facing symptom and the measured mechanism. The exit record needs the binary revision, host policy, workload, pause distribution, pressure sample, fault attribution, and rollback condition.

Sources read

Open the receipts

The experiment supplies the measured incident. Go documents the runtime witnesses. Linux documents pressure accounting. The discussion records competing interpretations, not settled proof.

Garage boundary: we read the report, its public repository page, the Go collector guide, the Linux PSI documentation, and the exact discussion. We did not reproduce the benchmark, run its BPF probes, inspect a production host, or test a mitigation. The 39,902 microsecond pause, 228 faults, 39,013 microseconds in faults, and message timings are the author's measurements.