Tracing P99.9 Allocator Stalls in Go Services with eBPF Ring Buffers
Standard 100 Hz CPU profiles average out transient stalls. When a latency-sensitive API normally answers in 1.2 ms but spikes to 18 ms twice a minute, sampling profilers show a flat flamegraph because the stall occupies less than 0.05% of wall-clock samples.
Where Sampling Profilers Miss Micro-Stalls
Execution traces help in staging, yet capturing full Go runtime traces under 40k RPS production traffic adds measurable write amplification. Instead, we can monitor kernel scheduler off-CPU intervals and Go runtime mallocgc durations inside eBPF, emitting records to user space only when a single event crosses a 2 ms threshold.
Attaching Low-Overhead Tracepoints
By pairing a uprobe on the runtime STW marker with tracepoint/sched/sched_switch, every thread preemption and memory sweep records a start timestamp in a per-TID hash map. Events under 2,000,000 ns are discarded completely inside the kernel.
SEC("tp_btf/sched_switch")
int BPF_PROG(on_sched_switch, bool preempt, struct task_struct *prev, struct task_struct *next) {
u32 prev_tgid = prev->tgid;
if (prev_tgid != target_pid)
return 0;
u64 ts = bpf_ktime_get_ns();
u32 prev_pid = prev->pid;
bpf_map_update_elem(&offcpu_start, &prev_pid, &ts, BPF_ANY);
u32 next_pid = next->pid;
u64 *start = bpf_map_lookup_elem(&offcpu_start, &next_pid);
if (!start)
return 0;
u64 delta = ts - *start;
bpf_map_delete_elem(&offcpu_start, &next_pid);
if (delta < threshold_ns)
return 0;
struct stall_event *ev = bpf_ringbuf_reserve(&events, sizeof(*ev), 0);
if (ev) {
ev->pid = next_pid;
ev->duration_ns = delta;
bpf_ringbuf_submit(ev, 0);
}
return 0;
}
Streaming Events via BPF_MAP_TYPE_RINGBUF
Unlike legacy per-CPU perf event arrays, BPF_MAP_TYPE_RINGBUF preserves global chronological ordering across cores and avoids allocating separate pages per logical CPU. In a 16-core VM, a single 256 KiB ring buffer handles bursts with zero dropped events.
What Caused the 14ms Spike
- Trigger: Periodic transparent huge page (THP) compaction inside
madvise(MADV_DONTNEED)during scavenger sweeps. - Fix: Setting
/sys/kernel/mm/transparent_hugepage/enabledtomadviseand pinningGOMEMLIMITslightly below cgroup hard limits dropped P99.9 tail latency from 18.4 ms to 2.1 ms.