· 7 min read

Tracing P99.9 Allocator Stalls in Go Services with eBPF Ring Buffers

Kernel Performance

Standard 100 Hz CPU profiles average out transient stalls. When a latency-sensitive API normally answers in 1.2 ms but spikes to 18 ms twice a minute, sampling profilers show a flat flamegraph because the stall occupies less than 0.05% of wall-clock samples.

Where Sampling Profilers Miss Micro-Stalls

Execution traces help in staging, yet capturing full Go runtime traces under 40k RPS production traffic adds measurable write amplification. Instead, we can monitor kernel scheduler off-CPU intervals and Go runtime mallocgc durations inside eBPF, emitting records to user space only when a single event crosses a 2 ms threshold.

Attaching Low-Overhead Tracepoints

By pairing a uprobe on the runtime STW marker with tracepoint/sched/sched_switch, every thread preemption and memory sweep records a start timestamp in a per-TID hash map. Events under 2,000,000 ns are discarded completely inside the kernel.

c — stall_probe.bpf.c
SEC("tp_btf/sched_switch")
int BPF_PROG(on_sched_switch, bool preempt, struct task_struct *prev, struct task_struct *next) {
    u32 prev_tgid = prev->tgid;
    if (prev_tgid != target_pid)
        return 0;

    u64 ts = bpf_ktime_get_ns();
    u32 prev_pid = prev->pid;
    bpf_map_update_elem(&offcpu_start, &prev_pid, &ts, BPF_ANY);

    u32 next_pid = next->pid;
    u64 *start = bpf_map_lookup_elem(&offcpu_start, &next_pid);
    if (!start)
        return 0;

    u64 delta = ts - *start;
    bpf_map_delete_elem(&offcpu_start, &next_pid);
    if (delta < threshold_ns)
        return 0;

    struct stall_event *ev = bpf_ringbuf_reserve(&events, sizeof(*ev), 0);
    if (ev) {
        ev->pid = next_pid;
        ev->duration_ns = delta;
        bpf_ringbuf_submit(ev, 0);
    }
    return 0;
}

Streaming Events via BPF_MAP_TYPE_RINGBUF

Unlike legacy per-CPU perf event arrays, BPF_MAP_TYPE_RINGBUF preserves global chronological ordering across cores and avoids allocating separate pages per logical CPU. In a 16-core VM, a single 256 KiB ring buffer handles bursts with zero dropped events.

What Caused the 14ms Spike

  • Trigger: Periodic transparent huge page (THP) compaction inside madvise(MADV_DONTNEED) during scavenger sweeps.
  • Fix: Setting /sys/kernel/mm/transparent_hugepage/enabled to madvise and pinning GOMEMLIMIT slightly below cgroup hard limits dropped P99.9 tail latency from 18.4 ms to 2.1 ms.