core-dumps

어떤 스킬인가요?

Core dump analysis skill for production crash triage. Use when loading core files in GDB or LLDB, enabling core dump generation on Linux/macOS, mapping symbols with debuginfo or debuginfod, or extracting backtraces from crashes without re-running the program. Activates on queries about core files, ulimit, coredumpctl, debuginfod, crash triage, or analyzing segfaults from production binaries.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

# Core Dump Cheatsheet

Source: <https://man7.org/linux/man-pages/man5/core.5.html>
Source: <https://sourceware.org/gdb/current/onlinedocs/gdb.html/Core-File-Generation.html>
Source: <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html-single/developing_c_and_cpp_applications_in_rhel_8/index>

## Table of Contents

1. [Enable core dumps](#enable-core-dumps)
2. [Core pattern configuration](#core-pattern-configuration)
3. [systemd / coredumpctl](#systemd--coredumpctl)
4. [Analyse with GDB](#analyse-with-gdb)
5. [Analyse with LLDB](#analyse-with-lldb)
6. [debuginfod for symbols](#debuginfod-for-symbols)
7. [Non-interactive triage](#non-interactive-triage)
8. [macOS cores](#macos-cores)
9. [Stripping and symbol management](#stripping-and-symbol-management)

---

## Enable core dumps

```bash
# Per-session (temporary)
ulimit -c unlimited

# Per-process (in code or shell script)
#include <sys/resource.h>
struct rlimit rl = { RLIM_INFINITY, RLIM_INFINITY };
setrlimit(RLIMIT_CORE, &rl);

# Persistent (all users): add to /etc/security/limits.conf
*   soft   core   unlimited
*   hard   core   unlimited

# Check current limit
ulimit -c              # current shell
cat /proc/self/limits  # current process (Linux)

# Check if cores are being generated
ls -la /tmp/ | grep core
ls -la $(cat /proc/sys/kernel/core_pattern | sed 's/%[epstu]//g' | xargs dirname 2>/dev/null)
```

---

## Core pattern configuration

```bash
# Show current pattern
cat /proc/sys/kernel/core_pattern

# Common useful pattern (temporary)
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t
# %e = executable name, %p = PID, %t = Unix timestamp
# Other tokens: %u=UID, %g=GID, %s=signal, %h=hostname

# Persistent: create /etc/sysctl.d/99-core.conf
[Unit]
kernel.core_pattern=/tmp/core-%e-%p-%t
kernel.core_uses_pid=1

sudo sysctl -p /etc/sysctl.d/99-core.conf

# Check if pipe handler is active (like systemd-coredump or apport)
cat /proc/sys/kernel/core_pattern
# If it starts with '|', a pipe handler is active:
# | /usr/lib/systemd/systemd-coredump ...   <- systemd
# | /usr/share/apport/apport %p ...         <- Ubuntu apport
```

**Core pattern tokens:**

| Token | Meaning |
|-------|---------|
| `%e` | Executable filename (without path) |
| `%E` | Executable path (/ → !) |
| `%p` | PID |
| `%P` | PID of dump process (tid if thread-specific) |
| `%u` | UID |
| `%g` | GID |
| `%s` | Signal number |
| `%t` | Unix timestamp |
| `%h` | Hostname |
| `%c` | Core file size soft limit |

---

## systemd / coredumpctl

```bash
# List all recorded crashes
coredumpctl list

# Show crashes for a specific executable
coredumpctl list myapp

# Show detailed info about latest crash
coredumpctl info

# Show info for specific PID
coredumpctl info 12345

# Open latest crash in GDB
coredumpctl gdb

# Open specific crash in GDB
coredumpctl gdb myapp
coredumpctl gdb PID

# Export core file
coredumpctl dump -o /tmp/myapp.core
coredumpctl dump PID -o /tmp/myapp.core

# Show journal log around crash
coredumpctl info --no-pager | grep -A5 "MESSAGE="

# Core storage location
ls /var/lib/systemd/coredump/

# Disable coredumpctl (revert to kernel default)
# In /etc/systemd/coredump.conf:
[Coredump]
Storage=none
```

---

## Analyse with GDB

```bash
# Load binary and core
gdb ./prog /tmp/core-prog-12345-1700000000

# If binary is stripped, use the debug build
gdb ./prog-with-debug-symbols /tmp/core

# Essential first session
(gdb) bt                         # call stack
(gdb) bt full                    # stack + locals in each frame
(gdb) info registers             # CPU registers at crash
(gdb) frame 2                    # jump to frame 2
(gdb) info locals                # local variables in that frame
(gdb) print ptr                  # inspect a pointer
(gdb) x/10wx $rsp                # examine memory near stack pointer

# All threads
(gdb) thread apply all bt full

# What signal caused the crash?
(gdb) info signals               # list signal handling
(gdb) print $_siginfo            # signal info struct (Linux)

# Check for SIGABRT (assertion failure)
(gdb) bt                         # look for __assert_fail or abort in stack

# Print a struct nicely
(gdb) set print pretty on
(gdb) print *my_struct_ptr

# Find where a pointer points
(gdb) info symbol 0x7fff12345678
(gdb) x/s 0x7fff12345678         # if it might be a string
```

---

## Analyse with LLDB

```bash
# Load core
lldb ./prog -c core.12345
# Or:
lldb
(lldb) target create ./prog --core core.12345

# Essential commands
(lldb) bt                         # backtrace
(lldb) bt all                     # all threads
(lldb) thread backtrace all
(lldb) frame select 2             # select frame
(lldb) frame variable             # locals
(lldb) register read              # CPU registers
(lldb) memory read -s8 -fx -c10 0x7fff0000  # examine memory

# Inspect signal
(lldb) thread info
```

---

## debuginfod for symbols

```bash
# Install client
sudo apt install debuginfod      # Debian/Ubuntu
sudo dnf install elfutils-debuginfod-client  # Fedora/RHEL

# Set URL (add to ~/.bashrc or /etc/environment)
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"

# GDB auto-uses debuginfod when DEBUGINFOD_URLS is set
gdb ./stripped-prog core

# Check build ID (needed for debuginfod lookup)
readelf -n ./prog | grep 'Build ID'
file ./prog | grep BuildID

# Manual lookup
debuginfod-find debuginfo <build-id-hex>
debuginfod-find source  <build-id-hex> /path/to/source.c

# Warm cache manually
DEBUGINFOD_PROGRESS=1 gdb ./prog core
```

**Public debuginfod servers:**

| Distro | URL |
|--------|-----|
| Ubuntu | `https://debuginfod.ubuntu.com` |
| Fedora | `https://debuginfod.fedoraproject.org` |
| Debian | `https://debuginfod.debian.net` |
| Arch Linux | `https://debuginfod.archlinux.org` |
| openSUSE | `https://debuginfod.opensuse.org` |
| Generic | `https://debuginfod.elfutils.org` |

---

## Non-interactive triage

```bash
# Quick backtrace from core (CI / automated)
gdb -batch \
    -ex 'set print thread-events off' \
    -ex 'thread apply all bt full' \
    -ex 'info registers' \
    -ex 'quit' \
    ./prog core 2>&1 | tee crash_report.txt

# One-liner: print backtrace and registers
gdb -batch -ex 'bt full' -ex 'info registers' ./prog core

# Check signal number from core metadata
eu-readelf -n core | grep -i signal
readelf -n core 2>/dev/null | head -30

# Print core metadata without GDB
file core                 # shows signal, PID, architecture
```

---

## macOS cores

```bash
# Enable
ulimit -c unlimited
# Cores go to /cores/core.<PID>

# Check
ls /cores/

# Load in LLDB
lldb ./prog -c /cores/core.12345

# Enable for a specific process
taskgated policy ...     # complex; usually just run with ulimit set

# Disable Crash Reporter (prevent dialog)
# sudo defaults write com.apple.CrashReporter DialogType none

# Check crash logs (Crash Reporter writes .crash files)
ls ~/Library/Logs/DiagnosticReports/
ls /Library/Logs/DiagnosticReports/
```

---

## Stripping and symbol management

```bash
# Best practice: keep an unstripped copy indexed by build ID
BUILD_ID=$(readelf -n prog | grep 'Build ID' | awk '{print $3}')
mkdir -p /srv/symbols/${BUILD_ID:0:2}
cp prog /srv/symbols/${BUILD_ID:0:2}/${BUILD_ID:2}.debug

# Strip for distribution
objcopy --only-keep-debug prog prog.debug
strip --strip-debug prog
objcopy --add-gnu-debuglink=prog.debug prog

# Install debug symbols (Debian)
sudo apt install myapp-dbgsym   # or myapp-dbg

# Install debug symbols (Fedora/RHEL)
sudo dnf install myapp-debuginfo

# Tell GDB where debug files live
(gdb) set debug-file-directory /usr/lib/debug:/srv/symbols
```

---
name: core-dumps
description: Core dump analysis skill for production crash triage. Use when loading core files in GDB or LLDB, enabling core dump generation on Linux/macOS, mapping symbols with debuginfo or debuginfod, or extracting backtraces from crashes without re-running the program. Activates on queries about core files, ulimit, coredumpctl, debuginfod, crash triage, or analyzing segfaults from production binaries.
---

# Core Dumps

## Purpose

Guide agents through enabling, collecting, and analysing core dumps for post-mortem crash investigation without rerunning the buggy program.

## Triggers

- "My program crashed in production — how do I analyse the core?"
- "How do I enable core dumps on Linux?"
- "I have a core file but no symbols / source"
- "How do I use debuginfod to get symbols for a core?"
- "coredumpctl show me the crash"

## Workflow

### 1. Enable core dumps (Linux)

```bash
# Per-session (lost on logout)
ulimit -c unlimited

# Persistent (add to /etc/security/limits.conf)
*   soft   core   unlimited
*   hard   core   unlimited

# Check current limit
ulimit -c

# Set core pattern (where and how cores are named)
# Default: 'core' in CWD — often not useful
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t
# %e = executable, %p = PID, %t = timestamp

# Persistent (add to /etc/sysctl.d/99-core.conf)
kernel.core_pattern=/tmp/core-%e-%p-%t
kernel.core_uses_pid=1
```

### 2. systemd/coredumpctl (modern Linux)

If systemd manages core dumps (common on Ubuntu 20+, Fedora, Arch):

```bash
# List recent crashes
coredumpctl list

# Show details of the latest crash
coredumpctl info

# Load latest crash in GDB
coredumpctl gdb

# Load specific PID crash
coredumpctl gdb 12345

# Export core file
coredumpctl dump -o myapp.core PID
```

Core storage location: `/var/lib/systemd/coredump/`.

### 3. Enable core dumps (macOS)

```bash
# macOS uses /cores by default (must be root-writable)
ulimit -c unlimited

# Check
ls /cores/

# launchd-launched services: set in plist
# <key>HardResourceLimits</key>
# <dict><key>Core</key><integer>9223372036854775807</integer></dict>
```

### 4. Analyse a core with GDB

```bash
# Load binary and core
gdb ./prog core.12345

# If the binary was stripped, provide the unstripped copy
gdb ./prog-with-symbols core.12345

# Essential first commands
(gdb) bt                    # call stack
(gdb) bt full               # stack + locals
(gdb) info registers        # CPU state at crash
(gdb) frame 2               # jump to interesting frame
(gdb) info locals           # local variables in frame
(gdb) print ptr             # inspect a pointer

# All threads (multi-threaded crash)
(gdb) thread apply all bt full
```

### 5. Analyse a core with LLDB

```bash
lldb ./prog -c core.12345

# Or
lldb
(lldb) target create ./prog --core core.12345

# Commands
(lldb) bt
(lldb) thread backtrace all
(lldb) frame select 2
(lldb) frame variable
```

### 6. Missing symbols: debuginfod

`debuginfod` serves debug symbols from a central server, mapping build IDs to DWARF data.

```bash
# Install client (Debian/Ubuntu)
sudo apt install debuginfod

# Enable (add to ~/.bashrc or /etc/environment)
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"

# GDB auto-fetches symbols when DEBUGINFOD_URLS is set
gdb ./prog core

# Manually query
debuginfod-find debuginfo <build-id>
debuginfod-find source <build-id> /path/to/file.c
```

### 7. Missing symbols: manual approach

```bash
# Check if binary has a build ID
readelf -n ./prog | grep Build

# Find the correct debug package
# Debian: apt install prog-dbg or prog-dbgsym
# RPM: dnf install prog-debuginfo

# Point GDB to debug symbols directory
(gdb) set debug-file-directory /usr/lib/debug

# Or use eu-readelf to dump build ID, then find .debug file
eu-readelf -n ./prog
find /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id>
```

### 8. Strip binaries and keep symbols

Best practice: build with symbols, strip for distribution, keep an unstripped copy.

```bash
# Build
gcc -g -O2 -o prog main.c

# Separate debug info
objcopy --only-keep-debug prog prog.debug
objcopy --strip-debug prog prog.stripped

# Add a debuglink so GDB finds the debug file automatically
objcopy --add-gnu-debuglink=prog.debug prog.stripped

# Deploy prog.stripped; keep prog.debug in a symbols store indexed by build-id
```

### 9. Quick triage from core without full debug session

```bash
# Print backtrace non-interactively
gdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt

# Print registers
gdb -batch -ex 'info registers' ./prog core

# Check signal that caused crash
gdb -batch -ex 'info signal' ./prog core
```

For a full cheatsheet covering core pattern tokens, coredumpctl, GDB/LLDB commands, debuginfod servers, and strip/symbol workflows, see [references/cheatsheet.md](references/cheatsheet.md).

## Related skills

- Use `skills/debuggers/gdb` for full GDB session details
- Use `skills/debuggers/lldb` for LLDB-based analysis
- Use `skills/runtimes/sanitizers` to catch the bug before it reaches production
- Use `skills/binaries/elf-inspection` for `readelf`, build IDs, and binary inspection