Sysdig — Deep System Visibility and Container Troubleshooting for Linux

Introduction

Sysdig is a system-level exploration and troubleshooting tool for Linux that captures system calls and OS events at the kernel level. It was built as a universal system visibility tool that understands containers natively, making it essential for debugging Docker and Kubernetes workloads.

What Sysdig Does

• Captures every system call on a Linux host with low overhead
• Filters events by process, user, file, network connection, or container
• Provides csysdig, an ncurses-based interactive UI similar to htop
• Saves captures to trace files for offline analysis and sharing
• Includes chisels, scriptable Lua-based analysis modules for common tasks

Architecture Overview

Sysdig operates by loading a kernel module (or eBPF probe) that taps into the Linux kernel tracepoint infrastructure. Every system call is captured and enriched with container metadata from the Docker or CRI runtime. Events flow through a userspace library where filters and chisels process them. The architecture supports both live analysis and file-based replay.

Self-Hosting & Configuration

• Install the kernel module via the official install script or package repositories
• Use the eBPF probe as an alternative when kernel module loading is restricted
• Run in Docker with --privileged and mounted host paths for full visibility
• Configure output format with -p flags for custom event printing
• Write custom chisels in Lua to automate specific analysis patterns

Key Features

• Unified view of system calls, network traffic, and file I/O in one tool
• Native container and Kubernetes awareness without extra configuration
• Trace file capture and replay for post-mortem debugging
• Over 50 built-in chisels for common tasks like top processes by I/O or network
• Works alongside Falco for runtime security monitoring

Comparison with Similar Tools

• strace — traces a single process but cannot capture system-wide events or container metadata
• tcpdump — captures network packets but lacks system call and file I/O visibility
• perf — kernel profiler focused on CPU performance, not general system exploration
• bpftrace — eBPF tracing with custom scripts but steeper learning curve
• htop/btop — real-time resource monitors without system call level detail

FAQ

Q: Does sysdig require a kernel module? A: By default yes, but an eBPF probe is available as an alternative for environments where kernel modules cannot be loaded.

Q: Can I use sysdig in Kubernetes? A: Yes. Deploy sysdig as a DaemonSet to capture events across all nodes with automatic pod and namespace tagging.

Q: What is the performance overhead? A: Sysdig adds minimal overhead during normal operation. Heavy filtering at the kernel level keeps userspace processing efficient.

Q: How does sysdig relate to Falco? A: Falco uses the same kernel instrumentation as sysdig but focuses on runtime security rules and alerting rather than interactive exploration.

Sources

• https://github.com/draios/sysdig
• https://sysdig.com/opensource/