Mastering strace, perf, and eBPF [2026 Cheat Sheet]

Debugging Linux systems in 2026 requires a tiered approach: starting with high-level syscall tracing via strace, moving to sampling-based profiling with perf, and finally leveraging the near-zero overhead of eBPF. This cheat sheet provides a consolidated reference for the most effective CLI flags, keyboard shortcuts, and filtering strategies to diagnose bottlenecks in production environments without crashing your kernel or causing significant latency spikes.

Tool Selection Matrix

Choosing the wrong tool can slow down your application or provide misleading data. Use this matrix to select the right probe for your specific debugging scenario.

strace: The Syscall Surgeon

strace intercepts and records the system calls which are called by a process and the signals which are received by a process. It is the first tool to reach for when an application is behaving unexpectedly.

Essential Flags

• -c: Count time, calls, and errors for each system call and report a summary on program exit.
• -e trace=file: Trace only system calls that take a filename as an argument (e.g., open, stat, chmod).
• -e trace=network: Trace all network-related system calls.
• -p [PID]: Attach to a running process.
• -f: Follow child processes created by fork and vfork.
• -o [file]: Write the trace output to a file instead of stderr.

# Find out why a process is hanging
strace -p 1234 -T -tt -yy

# Summarize syscall overhead
strace -cp 1234

# Trace specific syscalls with timestamps
strace -e trace=open,connect -t -f ./my_app

perf: The Performance Profiler

perf is the standard Linux profiler. It interfaces with the Performance Monitoring Unit (PMU) in your CPU and kernel tracepoints to sample events.

Common Workflows

• perf stat: Provides a quick overview of CPU performance counters (instructions per cycle, cache misses).
• perf top: Real-time view of functions consuming the most CPU cycles across the entire system.
• perf record: Samples the system and saves data to perf.data for later analysis.
• perf report: Analyzes the perf.data file generated by perf record.

# Record CPU profile for 30 seconds across all CPUs with call graphs
perf record -F 99 -a -g -- sleep 30

# View the results interactively
perf report -n --stdio

# Check for L1 cache misses
perf stat -e L1-dcache-load-misses ./my_heavy_app

eBPF-top & bpftrace: Modern Observability

eBPF (Extended Berkeley Packet Filter) allows you to run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules. Tools like bpftop (ebpf-top) leverage this for real-time monitoring.

bpftrace One-Liners

1. Tracing file opens: bpftrace -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)); }'
2. Syscall count by process: bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
3. Disk latency histogram: bpftrace -e 'kprobe:vfs_read { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @ns = hist(nsecs - @start[tid]); delete(@start[tid]); }'

Essential Keyboard Shortcuts

Use these shortcuts while running perf top or bpftop to navigate the data effectively.

Advanced Filtering & Search

To find the needle in the haystack, you must master filtering. In strace, the -e flag is your best friend.

# Only show syscalls that took longer than 100ms
strace -T -e trace=all ./app 2>&1 | awk -F'<' '$2 > 0.1 {print $0}'

# Filter perf report to show only your application's shared library
perf report --dsos=libmymath.so

# Trace only write() calls that return an error (negative value)
strace -e trace=write -e fault=write:error=EACCES ./app