This is a eBPF simple primer post written with generous help from Claude.

ELI5 version

eBPF is like having magic glasses for your computer. These glasses let you see what’s happening inside your computer without stopping it or slowing it down. You can watch programs talk to each other, see how fast things are moving, and even catch bad behaviors. The best part is you can program these glasses to look for specific things and take action when they happen.

What is eBPF?

eBPF is a technology in the Linux kernel that allows you to run small programs in a safe, sandboxed environment directly in the kernel. It was originally designed for network packet filtering but has evolved into a powerful, general-purpose monitoring and tracing framework.

Key features:

• Runs safely inside the kernel without modifying kernel code
• High performance with minimal overhead
• Versatile application across networking, security, and observability
• JIT (Just-In-Time) compilation for near-native performance

eBPF Tools Ecosystem

1. BCC (BPF Compiler Collection): A toolkit for creating eBPF programs using Python and Lua frontends.
2. bpftrace: A high-level tracing language for eBPF, similar to awk or DTrace. It provides a simple, powerful scripting interface for writing eBPF programs.
3. Cilium: Uses eBPF for container networking, observability, and security.
4. Falco: Security monitoring tool that uses eBPF to detect anomalous behavior.
5. Hubble: Network and security observability platform built on eBPF.
6. Pixie: Observability platform for Kubernetes applications using eBPF.

What is bpftrace?

bpftrace is a high-level tracing language for eBPF that makes it easy to write small programs to trace and analyze system behavior. Think of bpftrace as the friendly interface to eBPF’s power.

Relationship to eBPF:

• bpftrace is to eBPF what SQL is to a database engine
• It compiles your human-readable scripts into eBPF bytecode
• Handles the complexity of loading and running your eBPF programs
• Provides built-in functions and easy syntax for common tracing needs

Simple bpftrace example:

# Count system calls by process name
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[comm] = count(); }'

This one-liner counts all system calls grouped by process name, demonstrating bpftrace’s concise yet powerful syntax.

Kprobes and Uprobes

Kprobes

Kprobes (Kernel Probes) are debugging mechanisms in the Linux kernel that allow you to dynamically break into any kernel routine and collect debugging and performance information non-disruptively. They’re essentially dynamic breakpoints you can insert anywhere in the kernel code.

Key features:

• Can be attached to virtually any instruction in the kernel
• Minimal performance impact when not triggered
• Collect register and memory state at the probe point
• Available in two flavors: kprobes (at function entry) and kretprobes (at function return)

Uprobes

Uprobes (User Probes) are similar to kprobes but work in userspace. They allow you to trace and instrument user applications by inserting breakpoints at specific functions or instructions.

Key features:

• Trace applications without modifying their source code
• Attach to specific functions in userspace programs
• Monitor application behavior in production
• Available as both uprobes (function entry) and uretprobes (function return)

Relationship to eBPF

Kprobes and uprobes provide the attachment points for eBPF programs to hook into kernel and user application code. The relationship works like this:

1. Attachment mechanism: eBPF programs use kprobes/uprobes as the “hooks” to insert themselves into kernel or application execution paths
2. Data collection: When a probe is triggered, the associated eBPF program executes, collecting data and potentially making decisions
3. Performance: eBPF added JIT compilation to make probe handlers extremely efficient
4. Programmability: Before eBPF, probes were limited in functionality; eBPF adds a programmable layer to determine what happens when a probe triggers

An example in bpftrace showing both kprobe and uprobe:

# Trace kernel function
bpftrace -e 'kprobe:do_sys_open { printf("Opening file: %s\n", str(arg1)); }'

# Trace user function in libc
bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { printf("malloc called, size: %d\n", arg0); }'

eBPF transformed kprobes and uprobes from simple debugging tools into a powerful, programmable observability framework, turning them from basic breakpoints into sophisticated monitoring tools with minimal performance impact.