Master Linux debugging with this definitive cheat sheet for strace, perf, and eBPF. Compare overhead, filtering flags, and one-liners. Read the full guide.
When to reach for each tool
strace, perf, and eBPF sit on a spectrum from “what did this process do?” to “what is the whole machine spending time on?” Use strace when a single process misbehaves: hanging syscalls, unexpected file paths, permission denials, or a child that exits before you can attach a debugger. Use perf when the question is about CPU time, cache behavior, or which functions dominate a hot path. Reach for eBPF when you need low-overhead, programmable visibility across many processes—network paths, scheduling, storage latency, or custom events—without rewriting the application.
A practical order of attack is: confirm the symptom with strace on one PID, then profile with perf if the issue looks CPU-bound, then write or reuse an eBPF program only when you need durable, filtered, system-wide instrumentation. Switching tools mid-investigation is normal; the goal is the smallest probe that answers the next question.
strace: syscall truth with real cost
strace attaches to a process and prints every (or filtered) system call with arguments and return values. That makes it unmatched for “why did open fail?” and “which path is it really using?” The tradeoff is overhead: tracing every syscall can slow a busy process enough to change timing and hide race conditions. Prefer filtered traces: follow a subset of calls (file, network, process lifecycle) and attach only after reproduction steps are ready.
Useful discipline: start with a short attach window, capture a hang or error once, then detach. For multi-threaded programs, be explicit about whether you follow children and threads. Treat the output as a timeline of kernel transitions, not as a full CPU profile—strace will not tell you which userspace loop burns cycles between calls.
perf: sampling the hot path
perf samples hardware and software events so you can see where time goes without logging every syscall. CPU cycles, instructions, cache misses, and software events give a map of hot functions and call stacks. It is the right default when latency is high but strace shows few or no blocking calls—classic CPU-bound or lock-heavy work that never enters the kernel often enough for strace to explain it.
Keep sessions short and scoped: a single binary or PID, a fixed duration, and symbols available so stacks resolve to meaningful names. Compare “on-CPU” samples with wall-clock symptoms; if CPU is idle while latency is high, look at blocking I/O, locks, or waits rather than optimizing pure compute. Use call-graph sampling when flat function lists do not show which caller path matters.
- strace: best for correctness, paths, and failed syscalls on one process
- perf: best for CPU and cache hotspots with manageable sampling overhead
- eBPF: best for custom, filtered, system-wide observability with lower steady-state cost
eBPF and one-liner mindset
eBPF lets you attach small programs to kernel hooks and events, aggregate in-kernel, and export only what you care about. Compared with full strace-style tracing, well-written eBPF tools add less continuous overhead because filtering and counting happen before data leaves the kernel. Compared with generic perf samples, eBPF shines when you need structured fields—latency histograms, per-cgroup counters, or “only this syscall on this mount”—rather than a function-time profile alone.
A useful cheat-sheet habit: keep a short set of one-liners for attach-by-PID strace filters, timed perf record/report cycles, and known eBPF tools that answer recurring questions (open latency, TCP retransmits, scheduler run queue). Document the overhead expectation of each: strace for brief forensic attaches, perf for fixed-length profiles, eBPF for longer production-safe watches. Mastery is not memorizing every flag—it is choosing the probe whose cost and resolution match the failure mode in front of you.