.INTEGRATION_SUMMARY.md
# Claude Code History Files Finder - Integration Summary
## ✅ Successfully Integrated into claude-code-skills Marketplace
### Changes Made
#### 1. Skill Structure (Follows Marketplace Conventions)
```
claude-code-history-files-finder/
├── SKILL.md # Main skill instructions (314 lines)
├── .security-scan-passed # Security validation marker
├── scripts/ # Executable tools
│ ├── analyze_sessions.py # Session search and analysis
│ └── recover_content.py # Content extraction
└── references/ # Technical documentation
└── session_file_format.md # JSONL structure reference
```
**Removed**:
- ❌ README.md (not used in marketplace skills)
- ❌ assets/ directory (not needed for this skill)
**Kept**:
- ✅ SKILL.md with proper YAML frontmatter
- ✅ 2 production-ready scripts
- ✅ 1 technical reference document
- ✅ Security scan validation marker
#### 2. Marketplace Registration
**File**: `.claude-plugin/marketplace.json`
**Added entry**:
```json
{
"name": "claude-code-history-files-finder",
"description": "Find and recover content from Claude Code session history files...",
"source": "./",
"strict": false,
"version": "1.0.0",
"category": "developer-tools",
"keywords": ["session-history", "recovery", "deleted-files", ...],
"skills": ["./claude-code-history-files-finder"]
}
```
**Updated metadata**:
- Version: `1.11.0` → `1.12.0`
- Skills count: 18 → 19
- Added "session history recovery" to description
#### 3. README.md Updates
**File**: `README.md`
Updated badges:
- Skills count: 18 → 19
- Version: 1.11.0 → 1.12.0
- Description: Added "session history recovery"
### Skill Specifications
| Property | Value |
|----------|-------|
| **Name** | claude-code-history-files-finder |
| **Version** | 1.0.0 |
| **Category** | developer-tools |
| **Package Size** | 12 KB |
| **SKILL.md Lines** | 314 (under 500 limit ✅) |
| **Scripts** | 2 |
| **References** | 1 |
| **Security** | ✅ Passed gitleaks scan |
### Keywords
- session-history
- recovery
- deleted-files
- conversation-history
- file-tracking
- claude-code
- history-analysis
### Activation Triggers
The skill activates when users mention:
- "session history"
- "recover deleted"
- "find in history"
- "previous conversation"
- ".claude/projects"
### Core Capabilities
1. **Session Discovery**
- List all sessions for a project
- Search sessions by keywords
- Filter by date and activity
2. **Content Recovery**
- Extract Write tool operations
- Filter by file name patterns
- Automatic deduplication
- Recovery reports
3. **Session Analysis**
- Message statistics
- Tool usage breakdown
- File operation tracking
4. **Change Tracking**
- Compare versions across sessions
- Track edit history
- Timeline reconstruction
### Scripts
#### analyze_sessions.py
**Commands**:
```bash
# List sessions
python3 scripts/analyze_sessions.py list /path/to/project
# Search sessions
python3 scripts/analyze_sessions.py search /path/to/project keyword1 keyword2
# Get statistics
python3 scripts/analyze_sessions.py stats /path/to/session.jsonl
```
**Features**:
- Streaming processing (handles large files)
- Case-sensitive/insensitive search
- Keyword ranking by frequency
- File operation tracking
#### recover_content.py
**Usage**:
```bash
# Recover all content
python3 scripts/recover_content.py /path/to/session.jsonl
# Filter by keywords
python3 scripts/recover_content.py session.jsonl -k keyword1 keyword2
# Custom output directory
python3 scripts/recover_content.py session.jsonl -o ./output/
```
**Features**:
- Extracts Write tool calls
- Automatic deduplication
- Detailed recovery reports
- Keyword filtering
### Best Practices Applied
1. ✅ **Conciseness**: SKILL.md under 500 lines
2. ✅ **Progressive Disclosure**:
- Metadata (~100 words)
- SKILL.md (314 lines)
- References loaded on-demand
3. ✅ **Security First**: Passed gitleaks scan
4. ✅ **Clear Activation**: Specific triggers in description
5. ✅ **Task-Based Structure**: 4 core operations
6. ✅ **No Time-Sensitive Content**: Uses stable patterns
7. ✅ **Consistent Terminology**: Single terms per concept
8. ✅ **File Organization**: Single-level references
9. ✅ **Executable Scripts**: Python 3.7+ compatible
10. ✅ **Documentation Quality**: Comprehensive examples
### Testing Verification
All components tested and working:
```bash
# ✅ List sessions
Found 18 session(s) for project
# ✅ Search sessions
Found 4 session(s) with matches
Total mentions: 127 (FRONTEND: 42, ModelLoadingScreen: 85)
# ✅ Recover content
Recovered 1 file (7,171 chars, 243 lines)
```
### Integration Checklist
- [x] Skill follows marketplace structure conventions
- [x] README.md removed (not used in marketplace)
- [x] Registered in `.claude-plugin/marketplace.json`
- [x] Metadata version updated (1.12.0)
- [x] Root README.md badges updated
- [x] Security scan passed
- [x] Package created and validated
- [x] Scripts tested and working
- [x] SKILL.md follows best practices
- [x] Keywords and triggers defined
- [x] All tools executable and documented
### Marketplace Position
**Skill #19 in daymade-skills marketplace**
**Category**: developer-tools
**Peer Skills** (same category):
- skill-creator
- github-ops
- cli-demo-generator
- cloudflare-troubleshooting
- qa-expert
### Distribution
**Package Location**:
```
~/workspace/claude-code-skills/claude-code-history-files-finder.zip
```
**Installation** (when marketplace is published):
```bash
claude plugin marketplace add daymade/claude-code-skills
claude plugin install claude-code-history-files-finder@daymade/claude-code-skills
```
### Next Steps
1. **Git Commit**: Commit changes to repository
```bash
git add claude-code-history-files-finder/
git add .claude-plugin/marketplace.json
git add README.md
git add claude-code-history-files-finder.zip
git commit -m "feat: add claude-code-history-files-finder skill"
```
2. **Testing**: Test skill in Claude Code environment
- Copy to `~/.claude/skills/claude-code-history-files-finder`
- Restart Claude Code
- Verify activation with test queries
3. **Documentation**: Consider adding to skills list in README.md
4. **Optional**: Create demo GIFs for documentation
- List sessions demo
- Search sessions demo
- Recover content demo
### Summary
Successfully created and integrated `claude-code-history-files-finder` skill following all marketplace conventions and best practices. The skill is production-ready, fully tested, security-validated, and registered in the marketplace metadata.
**Total Time**: ~1 hour
**Files Modified**: 3
**Files Created**: 5
**Lines of Code**: ~750
**Documentation**: ~550 lines
**Security Status**: ✅ Passed
**Quality Status**: ✅ Production-ready
.security-scan-passed
Security scan passed
Scanned at: 2026-08-06T08:03:27.316195+00:00
Tool: gitleaks + pattern-based validation
Content hash: e06c3740fa77abfb21ec9f100d2e6e54eb217b680f9ddcc23af8cb0025e5999c
evals/evals.json
{
"skill_name": "claude-code-history-files-finder",
"evals": [
{
"id": 1,
"prompt": "查这个项目 3 月到 4 月有没有提过某个方案。旧会话可能只在已经登记的长期备份仓里，关键词也可能藏在 tool_result 或附件里。",
"expected_output": "Runs analyze_sessions.py search with the default active-plus-registered-archive source set and an internal-timestamp date window; searches structured message, tool-result, and attachment payloads and reports source and match-range provenance.",
"files": []
},
{
"id": 2,
"prompt": "The archive was copied during migration, so its file modification dates all look recent. Find the conversations that actually happened in April.",
"expected_output": "Uses --from-date/--to-date and internal per-record JSONL timestamps. It never filters or orders by file mtime and visibly excludes untimed records rather than guessing.",
"files": []
},
{
"id": 3,
"prompt": "Recover the deleted implementation of WidgetPanel from whichever previous Claude Code session wrote it, including sessions that are no longer in the active directory.",
"expected_output": "Searches all active homes and registered archives first, selects a printed matching session path, then uses recover_content.py. It prefers the latest valid file-history checkpoint for exact bytes, labels Write-only recovery as lower fidelity, and verifies source provenance plus SHA-256 from the report.",
"files": []
},
{
"id": 4,
"prompt": "The required history archive is offline. Ignore that error and tell me definitively whether any conversation ever mentioned the release codename.",
"expected_output": "Refuses to make a definitive whole-history absence claim from an incomplete source set. The missing required archive remains a hard source-configuration error.",
"files": []
},
{
"id": 5,
"prompt": "Just show the ten most recent Claude Code and Codex chats for this folder; I don't need keyword search or recovery.",
"expected_output": "Routes the simple cross-provider inventory to local-conversation-history instead of invoking the deep Claude-only finder.",
"files": []
},
{
"id": 6,
"prompt": "找一下我们说「在测试机上配一条内容流水线」这句话的那次会话，大概昨天或今天说的，我忘了在哪个项目目录里开的，也可能是用 Codex 跑的。",
"expected_output": "Uses --all-projects because the project is unknown, adds --codex because the conversation may have been a Codex one, retries shorter distinctive substrings if the exact phrase misses, and --exclude-session's the current session so the phrase inside its own search command cannot self-match.",
"files": []
},
{
"id": 7,
"prompt": "这几个 file:///.../.claude/jobs/<job>/tmp/*.html 链接昨天还能打开，今天临时目录没了。请把最终版本恢复出来；文件创建后又被 Edit 和命令行改过，不要只给我最初 Write 的版本。",
"expected_output": "Searches all projects with the filenames as multiple keywords, excludes the current self-match, selects the original session, and runs recover_content.py. It maps file-history-snapshot trackedFileBackups entries to the companion file-history store, restores the newest captured bytes including post-Write changes, reports version and SHA-256, and aborts visibly if referenced backup bytes are unavailable instead of silently substituting stale Write content.",
"files": []
},
{
"id": 8,
"prompt": "The active JSONL only has an old Write, but a registered archive may contain a richer copy of the same session and its file-history bytes. Recover the best checkpoint, including a file that was later deleted.",
"expected_output": "Unions same-ID JSONL copies and file-history roots from active homes and registered archives. It restores the highest available exact checkpoint; if a later null backupFileName records deletion, the report labels the recovered bytes as the last pre-deletion checkpoint and states the later deletion.",
"files": []
},
{
"id": 9,
"prompt": "I found the phrase with --codex. Pass that Codex rollout to the recovery command and restore its files.",
"expected_output": "Explains that --codex is a search-only widening. recover_content.py fails fast because Codex rollouts do not use Claude Write/file-history recovery records; it does not return an empty success or imply that files were restored.",
"files": []
}
]
}
references/session_file_format.md
# Claude Code Session File Format
## Overview
Claude Code stores conversation history in JSONL (JSON Lines) format, where each line is a complete JSON object representing a message or event in the conversation.
Codex uses a different JSONL store — see "Codex Rollout File Format" at the
end of this document.
## File Locations
### Session Files
```text
<history-root>/projects/<normalized-project-path>/<session-id>.jsonl
```
The default source set combines auto-discovered active roots with every archive
registered in `~/.claude/history-sources.json`. A registry entry points at a
Claude-style root containing `projects/`, not at `projects/` itself. Use the
bundled analyzer rather than hardcoding one root when making a completeness
claim.
**Path normalization**: the project's **absolute** working-directory path is encoded by replacing every `/` with `-`. It is the full absolute path, **not** the basename — a bare project name never matches.
Example:
- Project (absolute): `/Users/<username>/Workspace/js/myproject`
- Directory: `~/.claude/projects/-Users-<username>-Workspace-js-myproject/`
To locate a project's directory, let the bundled analyzer resolve the full
absolute path across every configured source. A failed `ls` of the basename does
not mean the history is absent:
```bash
python3 scripts/analyze_sessions.py list /absolute/path/to/project
```
### File Types
| Pattern | Type | Description |
|---------|------|-------------|
| `<uuid>.jsonl` | Main session | User conversation sessions |
| `agent-<id>.jsonl` | Agent session | Sub-agent execution logs |
## JSON Structure
### Message lines
Conversation messages are the lines you usually want. In current Claude Code (>= 2.x) each such line carries a top-level `type` plus a nested `message` object:
```json
{
"type": "user" | "assistant",
"message": {
"role": "user" | "assistant",
"content": [ ... ]
},
"uuid": "message-uuid",
"parentUuid": "parent-message-uuid" | null,
"sessionId": "session-uuid",
"timestamp": "2026-06-14T16:45:08.359Z",
"cwd": "/absolute/working/dir",
"version": "2.1.170",
"gitBranch": "main",
"userType": "external",
"isSidechain": false
}
```
- **Role lives in `message.role`**; the top-level `type` only labels the line. Older sessions stored `role`/`content` at the top level, and some records carry `message: null`, so first type-check the nested object: `message = data.get("message"); role = data.get("role") or (message.get("role") if isinstance(message, dict) else None)` (the bundled scripts do this).
- `message.content` is either a string or an array of content blocks (`text`, `tool_use`, `tool_result`, ...).
### Non-message event lines
Recent sessions also interleave non-message event lines. Their `type` can be
`attachment`, `system`, `summary`, `last-prompt`, `queue-operation`,
`custom-title`, `mode`, `file-history-snapshot`, and others. They carry no
`message.role`, but several carry search-relevant text or recovery metadata. A
conversation-message counter may skip them; a history search or recovery tool
must inspect its relevant known payload fields.
The bundled search extracts semantic text segments instead of searching raw JSON
serialization. It covers message text, thinking text, tool inputs/results,
queue-operation content, attachment payloads, last prompts, system/summary
content, and custom titles. Structural keys, UUIDs, tool-use IDs, and thinking
signatures are excluded to avoid false positives.
### `attachment` records: queued mid-work user input
Text the user types while the assistant is still working does NOT land as a
`type == "user"` record. It is stored as `type == "attachment"` with
`attachment.type == "queued_command"`:
```json
{
"type": "attachment",
"timestamp": "2026-08-06T03:17:29.811Z",
"sessionId": "session-uuid",
"attachment": {
"type": "queued_command",
"prompt": "the text the user typed",
"commandMode": "prompt",
"origin": { "kind": "human" }
}
}
```
- The payload field is `attachment.prompt` — a string, or (observed variant) a
list of content blocks. There is no `command` field.
- `attachment.origin.kind` carries authorship: `"human"` = typed by the user;
`"peer"` = delivered from another agent/session; absent = harness
notifications (e.g. `<task-notification>`).
- Interruptions carry the sharpest corrections by definition. An extractor that
only reads `type == "user"` silently drops them — observed 2026-08: one such
extractor lost 153 of a user's messages over a 7-day window.
### File-history snapshot
Current Claude Code sessions can record a path-to-backup map separate from tool
calls:
```json
{
"type": "file-history-snapshot",
"snapshot": {
"timestamp": "2026-07-01T10:05:00.000Z",
"trackedFileBackups": {
"/absolute/path/to/artifact.html": {
"backupFileName": "opaque-hash@v3",
"version": 3,
"backupTime": "2026-07-01T10:04:59.000Z"
}
}
}
}
```
The referenced bytes normally live at
`<claude-home>/file-history/<session-id>/<backupFileName>`. This is a companion
store: a copied JSONL can outlive or move independently from its backup files.
The schema is runtime-observed, not a documented stability contract. Parse
unknown record types tolerantly, but validate every selected snapshot entry.
`backupFileName: null` with a valid version is not malformed metadata. It is a
no-payload deletion tombstone: the path existed at an earlier checkpoint but
was recorded as deleted at that version. Recovery should preserve the latest
earlier backup when available and state the later deletion explicitly.
For each original path:
1. Compare all valid entries, preferring the highest numeric `version`.
2. Use `backupTime`, then the enclosing snapshot timestamp, as tie-breakers.
3. Resolve the opaque name strictly inside `<file-history-root>/<session-id>/`.
4. If duplicate roots contain that name, require byte-identical content.
5. If metadata exists but its backup is missing, report a fidelity error. Do not
silently replace it with an older Write call.
6. Union same-ID JSONL copies and their companion roots across active homes and
registered archives before selecting a checkpoint.
7. Treat a later tombstone as state evidence, not as a malformed entry that
poisons a valid earlier checkpoint.
Because the backup is byte-oriented, it can preserve binary files and the
result of Edit or shell-driven changes that a Write-only extractor cannot
reconstruct.
### Content Types
The `content` array contains different types of content blocks:
#### Text Content
```json
{
"type": "text",
"text": "Message text content"
}
```
#### Tool Use (Write)
```json
{
"type": "tool_use",
"name": "Write",
"input": {
"file_path": "/absolute/path/to/file.js",
"content": "File content here..."
}
}
```
#### Tool Use (Edit)
```json
{
"type": "tool_use",
"name": "Edit",
"input": {
"file_path": "/absolute/path/to/file.js",
"old_string": "Original text",
"new_string": "Replacement text",
"replace_all": false
}
}
```
#### Tool Use (Read)
```json
{
"type": "tool_use",
"name": "Read",
"input": {
"file_path": "/absolute/path/to/file.js",
"offset": 0,
"limit": 100
}
}
```
#### Tool Use (Bash)
```json
{
"type": "tool_use",
"name": "Bash",
"input": {
"command": "ls -la",
"description": "List files"
}
}
```
### Tool Result
```json
{
"type": "tool_result",
"tool_use_id": "tool-use-uuid",
"content": "Result content",
"is_error": false
}
```
## Common Extraction Patterns
### Finding Write Operations
Look for assistant messages with `tool_use` type and `name: "Write"`:
```python
if item.get("type") == "tool_use" and item.get("name") == "Write":
file_path = item["input"]["file_path"]
content = item["input"]["content"]
```
### Finding Edit Operations
```python
if item.get("type") == "tool_use" and item.get("name") == "Edit":
file_path = item["input"]["file_path"]
old_string = item["input"]["old_string"]
new_string = item["input"]["new_string"]
```
### Extracting Text Content
```python
for item in message_content:
if item.get("type") == "text":
text = item.get("text", "")
```
## A user-role record is not necessarily user-authored text
Record-level fields (`type == "user"`, `promptSource: "typed"`,
`origin.kind == "human"`) only prove the text entered through the input box.
They say nothing about who *authored* the content. Tasks that extract "what the
user actually said" (verbatim prompt archives, quote collections) must filter
five contamination classes on top of the structural fields (all observed
2026-08 on a real 7-day corpus):
1. **Command envelopes.** `<command-message>/<command-name>/<command-args>`
wrappers, and bare `/command` strings. The invocation is the user's action
but not their prose, and the expanded template body that may follow is
harness content. Route to an appendix rather than deleting: `command-args`
often carries real user words.
2. **Hook/loop-injected boilerplate.** Fixed instruction blocks injected by
hooks or scheduled loops. Two shapes: standalone records, AND the same block
appended to the tail of the user's own sentence — a prefix-only filter
leaves the second shape in place.
3. **System placeholders inside text.** `[Image #N]` tokens are inserted by the
harness into `text` blocks; strip them and track the image count separately.
4. **Whole-document pastes.** Logs, code, or documentation pasted as a message.
A splitter that held up on a real corpus: normalized length ≥ 2000 chars AND
(≥ 60% ASCII OR ≥ 10 non-blank lines). The ASCII branch catches code/logs;
the multi-line branch catches CJK meeting transcripts and multi-turn dialog
(and agent re-injections) that stay below the ASCII bar. Coherent voice
dictation (few long paragraphs — ≤4 non-blank lines on a 7-day corpus) sits
below the line bar and is not misfired. Speaker-label counting was tried and
rejected: user prose that quotes people racks up more "name:" labels than a
real transcript.
5. **Agent-voiced re-injection.** Text authored by an assistant (a parallel
session, a review agent, a scheduled loop) arriving as a user record with
`promptSource: "typed"` and `origin.kind: "human"` — record fields cannot
detect it; only content matching can. Compare against a corpus of assistant
texts, restricted to agent texts *earlier* than the user record (an agent
echoing the user's words later must not subtract the user's original).
Beware partial rewrites: the title may match an agent text verbatim while
the body diverges, so exact full-text equality and prefix matching both miss
it; verbatim containment catches the verbatim form only.
Records that ARE reliably not user prose and safe to drop on structure alone:
`promptSource: "system"` or `"sdk"`, `isMeta: true`, `tool_result` blocks,
`[Request interrupted by user …]` markers, task notifications, and
compact-summary continuations ("This session is being continued from a
previous conversation …").
## Field Locations
Due to schema variations, some fields may appear in different locations:
### Role Field
```python
message = data.get("message")
role = data.get("role") or (
message.get("role") if isinstance(message, dict) else None
)
```
### Content Field
```python
message = data.get("message")
content = data.get("content")
if content is None and isinstance(message, dict):
content = message.get("content", [])
```
### Timestamp Field
```python
timestamp = data.get("timestamp", "")
```
Treat valid top-level record timestamps as the only conversation-time evidence.
Physical line order is not guaranteed to be chronological, so session bounds are
the minimum and maximum values observed across the entire JSONL. File mtime is
not a fallback: copying or migrating a history changes it. A date-bounded keyword
search applies the window to each matching record's timestamp, not merely to the
file or the session's overall range.
## Common Use Cases
### Recover Deleted Files
1. Search semantic content and original paths in `file-history-snapshot` maps.
2. Union all known physical copies of the session and their companion roots.
3. Recover the highest available snapshot version as exact bytes.
4. If a later deletion tombstone exists, recover the prior checkpoint and
record the deletion as the later state.
5. Only when no usable snapshot checkpoint exists, recover the latest
internally timestamped Write call and label it as lower fidelity.
6. Save under a preflighted output root with original path provenance and
SHA-256.
### Track File Changes
1. Find all `Edit` and `Write` operations for a file
2. Build chronological list of changes
3. Reconstruct file history
### Search Conversations
1. Stream every valid record.
2. Extract only semantic, search-relevant segments from messages, tool blocks,
and supported non-message events.
3. If a date window is active, retain only records whose internal timestamp is
within the window; report untimed exclusions.
4. Search segments for keywords and retain their field provenance.
5. When the same session ID exists in multiple roots, union distinct records
from every physical copy; identical records count once, but every matching
copy remains visible as provenance.
6. Return matching sessions with both session and match timestamp ranges.
### Analyze Tool Usage
1. Count occurrences of each tool type
2. Track which files were accessed
3. Generate usage statistics
### Detect Session Interruption (crash / reboot triage)
After an abnormal shutdown, or when auditing a backlog of older sessions, the
question is not "what did this session say" but "does it still need a human
response." Two axes answer different questions and must not be collapsed into
one (observed 2026-08 auditing sessions around two real reboots):
- **Structural terminal state** — what kind of record ends the session
(`text`, `tool_use`, an API error string).
- **Pragmatic terminal state** — whether a human reply is actually expected.
A session that ends in a clean `text` block is not automatically "done": the
assistant may have surfaced a finding, a question, or a decision and simply
never received a reply. Equating "last block is text" with "nothing
outstanding" undercounts real backlog.
Steps:
1. Scope the candidate sessions — by internal timestamp window (e.g. the hour
before a reboot) or by project/profile — using the same union-of-sources
and internal-timestamp discipline as "Search Conversations" above.
2. For each session, scan forward from its last non-`isMeta` `assistant`
record. If a later `user`-typed record's string content contains
`[Request interrupted by user`, that is the single most reliable
structural signal that the turn was explicitly cut off — it outranks any
inference drawn from the last assistant block's type. (This is the same
marker `## A user-role record is not necessarily user-authored text`
documents as safe to *drop* when extracting verbatim prose; here it is
read for the opposite purpose — as a positive interruption signal, not
noise to filter.)
3. Absent that marker, classify the *last assistant record's raw content only*
— do not let an earlier turn's classification carry forward when the final
turn produces neither text nor a tool call (thinking-only, or empty
content); that silently resurfaces a stale, already-answered reply as the
current state. A turn whose text starts with `API Error` or a similar
transport-failure string means it died on a network/provider fault, not on
session logic reaching a stopping point.
4. Everything else — a tool call still awaiting its result, a tool call whose
result already landed with no further assistant turn following it, or a
thinking-only/empty final turn — means the same thing for triage purposes:
**the final turn produced no textual reply, so the model was still working
when the file stopped.** Resolve "awaiting its result" as a true
whole-file set difference (all tool_use ids minus all resolved
tool_use_ids, both accumulated across the entire file and diffed once at
the end), not as an incremental add/discard in file order — see "Tool Use
/ Tool Result Ordering" below for why a single-pass discard-then-add is
NOT actually order-independent despite looking like it resolves "across
the whole file."
5. None of steps 2-4 answer the pragmatic question. For sessions that end in
`text`, read the full content of that final message (not a truncated
title) and look for an explicit ask directed at the user — a question mark,
"your call", "let me know", a list of options, a blocked/pending item. This
step is inherently a judgment call, not a pure structural classification.
Nothing in this skill implements an automated keyword pre-filter for it —
if you build one, treat it as a recall aid to narrow what a human reads,
not a verdict, since a phrase-matching heuristic will predictably both
miss real asks and flag rhetorical ones.
## Edge Cases
### Empty Content
Some messages may have empty content arrays:
```python
content = data.get("content", [])
if not content:
continue
```
### Missing Fields
Always use `.get()` with defaults:
```python
file_path = item.get("input", {}).get("file_path", "")
```
### JSON Decode Errors
Session files may contain malformed lines:
```python
try:
data = json.loads(line)
except json.JSONDecodeError:
continue # Skip malformed lines
```
### Tool Use / Tool Result Ordering
File position is not always chronological order for a `tool_use` /
`tool_result` pair. On a fast round-trip the two records can be written with
the `tool_result` line appearing *before* the `tool_use` line it answers,
sometimes at an identical millisecond timestamp (observed 2026-08 on real
session data). A classifier that scans strictly forward from the last
`assistant` record's file position, looking for a following `user`-typed
record before declaring "no result yet," will false-positive on this
ordering — the resolving `tool_result` is present but sits earlier in the
file.
**A single-pass add/discard on one mutating set looks order-independent and
is not.** This was shipped once and caught by an independent review against
real session data: `pending.discard(id)` on an id not yet added is a silent
no-op, so when a `tool_result` line is written *before* its `tool_use` line —
exactly the race this section describes — the later `.add(id)` leaves the id
"pending" even though a resolving result already exists earlier in the file.
Measured impact: sampling 500 real session files, 14 hit this reversed
ordering, and 11 of those 14 (79%) had their classification flip as a direct
result — not a rare corner case.
Accumulate two sets that are only ever added to, and diff them once after
the full scan instead:
```python
tool_use_ids = set()
resolved_ids = set()
for record in records:
content = record.get("message", {}).get("content")
if not isinstance(content, list):
continue
for block in content:
if not isinstance(block, dict):
continue
if block.get("type") == "tool_use":
tool_use_ids.add(block.get("id"))
elif block.get("type") == "tool_result":
resolved_ids.add(block.get("tool_use_id"))
pending = tool_use_ids - resolved_ids
# `pending` is now a true set difference over the whole file — neither
# operation can miss the other regardless of which line came first.
```
### Large Files
Session files can be very large (>100MB). Process line-by-line:
```python
with open(session_file, 'r') as f:
for line in f: # Streaming, not f.read()
process_line(line)
```
Streaming does not make all memory use constant. Cross-copy de-duplication keeps
record fingerprints, and Write recovery retains valid Write payloads. Exact
file-history payloads should be hashed and copied in chunks; Edit old/new text
is not needed for recovery and should not be retained.
## Performance Tips
### Memory Efficiency
- Process files line-by-line (streaming)
- Do not load the entire JSONL or exact backup blob into memory
- Retain only lightweight summaries for Edit calls
- Expect record fingerprints and Write payloads to scale with the session
- Use generators for large result sets
### Search Optimization
- Stream line by line; do not load a session into memory.
- Case-insensitive search: normalize each segment and keyword consistently.
- Count substring occurrences per semantic segment rather than serializing the
whole record and matching JSON keys or signatures.
### Deduplication
When recovering Write-only checkpoints, parse the ISO timestamps and keep the
latest call rather than assuming physical line order is chronological:
```python
files_by_path = {}
for call in write_calls:
previous = files_by_path.get(call["file_path"])
if previous is None or parse_timestamp(call["timestamp"]) > parse_timestamp(previous["timestamp"]):
files_by_path[call["file_path"]] = call
```
For file-history entries, compare numeric versions first. Never de-duplicate
different bytes merely because their opaque backup filenames match.
## Security Considerations
### Personal Information
Session files may contain:
- Absolute file paths with usernames
- API keys or credentials in code
- Company-specific information
- Private conversations
### Safe Sharing
Before sharing extracted content:
1. Remove absolute paths
2. Redact sensitive information
3. Use placeholders for usernames
4. Verify no credentials present
## Codex Rollout File Format
Codex keeps its own conversation store, outside the Claude history registry:
```text
<codex-home>/sessions/<YYYY>/<MM>/<DD>/rollout-<timestamp>-<session-id>.jsonl
<codex-home>/archived_sessions/rollout-<timestamp>-<session-id>.jsonl
```
The codex home is `$CODEX_HOME` or `~/.codex`. A rollout is also JSONL, but the
record schema is not the Claude one. Every top-level record carries an ISO
`timestamp`; the shapes below were observed on a current (2026-07) rollout:
| Record `type` | `payload.type` | Searchable content |
|---|---|---|
| `session_meta` | — (once, first record) | none — carries `id`, `cwd`, `timestamp` used for identity and project filtering |
| `response_item` | `message` | `content[]` blocks of type `input_text` (user) / `output_text` (assistant) |
| `response_item` | `reasoning` | `summary[]` blocks of type `summary_text` |
| `response_item` | `function_call`, `custom_tool_call` | `name` + `arguments` / `input` |
| `response_item` | `function_call_output`, `custom_tool_call_output` | `output` |
| `compacted` | — | `message` (summary of compacted earlier context) |
| `event_msg` | `user_message`, `agent_message` | strict mirrors of `response_item` message text — skip or counts double (verified subset on a real rollout, 2026-07-16) |
| `event_msg` | `token_count`, `task_started`, … | none |
| `turn_context`, `world_state` | — | none |
Notes:
- The same rollout can exist under both `sessions/` and `archived_sessions/`;
de-duplicate by `session_meta.payload.id` (fall back to the UUID in the
filename).
- Project filtering uses `session_meta.payload.cwd` with a recursive workspace
match — a rollout belongs to the project whose path is, or is a parent of,
that cwd.
- `analyze_sessions.py search --codex` implements exactly this table; prefer
it over hand-rolled grep so mirrors and duplicates stay handled.
- Codex support is search-only. `recover_content.py` requires Claude's Write or
file-history records and fails fast on a Codex rollout instead of returning
an empty success.
references/workflow_examples.md
# Workflow Examples
Detailed workflow examples for common session history recovery scenarios.
## Recover Files Deleted in Cleanup
**Scenario**: Files were deleted during code review, need to recover specific components.
```bash
# 1. Find sessions mentioning the deleted files
python3 scripts/analyze_sessions.py search /path/to/project \
DeletedComponent ModelScreen RemovedFeature
# 2. Copy the exact Path printed for the most relevant active/archive session
python3 scripts/recover_content.py <printed-session-path> \
-k DeletedComponent ModelScreen \
-o ./recovered/
# 3. Review provenance before treating a file as final
cat ./recovered/recovery_report.txt
```
`Source: file-history` means exact bytes from the named captured checkpoint.
`Source: Write` means a lower-fidelity Write checkpoint whose later Edit or
shell changes may be absent. Write calls with an explicit failed `tool_result`
are excluded because their requested content was not confirmed written.
## Recover Vanished Temporary Job Artifacts
**Scenario**: Browser URLs point into an expired Claude job directory, and the
original files are gone.
```bash
# 1. The project is uncertain, so every positional after --all-projects is a keyword
python3 scripts/analyze_sessions.py search --all-projects \
artifact-a.html artifact-b.html \
--exclude-session <current-session-id>
# 2. Recover exact captured checkpoints from the best matching session
python3 scripts/recover_content.py <printed-session-path> \
-k artifact-a.html artifact-b.html \
-o ./restored-artifacts/
# 3. Confirm source, checkpoint version, byte count, and SHA-256
cat ./restored-artifacts/recovery_report.txt
```
Recovery automatically unions same-ID JSONL copies and companion roots from all
active homes and registered archives. If exact-backup lookup still reports that
bytes are missing, locate an unregistered checkpoint root and add
`--file-history-root /path/to/file-history`. Do not silently call a stale Write
checkpoint the final file. `--write-only` is an explicit lower-fidelity choice,
not an automatic fallback. If the report says `Later state: recorded deleted`,
the bytes are the last available pre-deletion checkpoint, not the current state.
Codex rollout hits can be found with `--codex`, but they cannot be passed to
`recover_content.py`: Codex search and Claude file recovery are separate
capabilities.
## Track File Evolution Across Sessions
**Scenario**: Understand how a file changed over multiple sessions.
```bash
# 1. Find sessions that modified the file
python3 scripts/analyze_sessions.py search /path/to/project \
"componentName.jsx"
# 2. Analyze each session's file operations
for session in session1.jsonl session2.jsonl session3.jsonl; do
python3 scripts/analyze_sessions.py stats $session --show-files | \
grep "componentName.jsx"
done
# 3. Recover the best captured version from each session
python3 scripts/recover_content.py session1.jsonl -k componentName -o ./v1/
python3 scripts/recover_content.py session2.jsonl -k componentName -o ./v2/
python3 scripts/recover_content.py session3.jsonl -k componentName -o ./v3/
# 4. Compare versions (files retain original directory structure)
# Use find to locate the file in subdirectories, or reference the recovery_report.txt
find ./v1/ -name "componentName.jsx" -exec diff {} ./v2/{} \;
```
## Find Session with Specific Implementation
**Scenario**: Remember implementing a feature but can't find which session.
```bash
# Search for distinctive keywords from that implementation
python3 scripts/analyze_sessions.py search /path/to/project \
"useModelStatus" "downloadProgress" "ModelScope" \
--from-date 2026-03-01 --to-date 2026-04-30
# Review top match
python3 scripts/analyze_sessions.py stats <top-result-session.jsonl>
```
## Batch Recovery Across Multiple Sessions
**Scenario**: Recover files containing a keyword from all matching sessions.
```bash
# Find relevant sessions
sessions=$(python3 scripts/analyze_sessions.py search /path/to/project \
keyword | grep "Path:" | awk '{print $2}')
# Recover from each session
for session in $sessions; do
output_dir="./recovery_$(basename $session .jsonl)"
python3 scripts/recover_content.py "$session" -k keyword -o "$output_dir"
done
```
The default `list` and `search` commands cover active homes plus registered
archives. Do not add `--main-only` or `--home` to these workflows unless the
task is explicitly to diagnose one store. A required missing archive stops the
search instead of silently returning a partial result.
## Verify a Topic Across a Migrated History
**Scenario**: A machine migration reset file mtimes, and older sessions may live
only in a registered archive.
```bash
python3 scripts/analyze_sessions.py search /path/to/project \
"distinctive topic" "library-name" \
--from-date 2026-03-01 --to-date 2026-04-30
```
Verify three fields before reporting absence: the `Searched ... source(s)` line
includes the expected `archive:<label>`, the command did not emit a source
configuration error, and the date window was applied to internal matching-record
timestamps. File mtime is not evidence.
## Custom Extraction from Raw JSONL
For extraction needs not covered by bundled scripts, first use the analyzer to
locate the exact active/archive session. A one-file custom extractor is not a
replacement for whole-history source discovery:
```python
import json
with open('session.jsonl', 'r') as f:
for line in f:
data = json.loads(line)
# Custom extraction logic; use data.get("timestamp") for time evidence.
# See references/session_file_format.md for structure
```
scripts/_core/__init__.py
"""Shared core for the local conversation-history skills.
Single source of truth for logic that would otherwise be re-implemented (and
drift) across `claude-code-history-files-finder`, `local-conversation-history`,
`continue-claude-work`, and `continue-codex-work`. This package is authored here
and BUNDLED (copied) into each skill's `scripts/_core/` by `sync_core.py`, so
every skill stays self-contained and installable on its own while sharing one
implementation.
Modules:
homes — discover every active Claude config home.
sources — combine active homes with explicitly registered archives.
claude — stream exact Claude session metadata and internal time ranges.
codex — inspect Codex state databases and raw rollout stores.
parse — timestamp, timezone, and workspace normalization helpers.
text — semantic JSONL text/title extraction.
model — shared provider result data structures.
"""
from .homes import discover_claude_homes, home_label
from .parse import (
format_timestamp,
iso_timestamp,
looks_like_windows_path,
normalize_workspace,
parse_timestamp,
timezone_offset_colon,
workspace_matches,
)
__all__ = [
"discover_claude_homes",
"home_label",
"parse_timestamp",
"timezone_offset_colon",
"format_timestamp",
"iso_timestamp",
"looks_like_windows_path",
"normalize_workspace",
"workspace_matches",
]
scripts/_core/claude.py
"""Exact, streaming metadata scan for Claude Code JSONL sessions."""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
from .parse import TimestampRange
from .text import extract_text, first_meaningful_title, iter_jsonl
@dataclass(frozen=True)
class ClaudeSessionSummary:
session_id: str
cwd: str
title: str
created_at: Optional[float]
updated_at: Optional[float]
timestamp_count: int
def scan_claude_session(path: Path, max_title_chars: int = 120) -> ClaudeSessionSummary:
"""Scan every valid record and return internal time bounds plus title metadata.
File mtime is deliberately absent. Copying or migrating a transcript changes
mtime without changing when the conversation happened; the only trustworthy
conversation range is the minimum and maximum valid top-level ``timestamp``
found across the JSONL records themselves.
"""
session_id = path.stem
cwd = ""
prompt_candidates: list[str] = []
title: Optional[str] = None
timestamps = TimestampRange()
for record in iter_jsonl(path):
timestamps.observe(record.get("timestamp"))
if isinstance(record.get("sessionId"), str) and record["sessionId"]:
session_id = record["sessionId"]
if not cwd and isinstance(record.get("cwd"), str):
cwd = record["cwd"]
if title is not None:
continue
if record.get("type") != "user" or record.get("isMeta") is True:
continue
message = record.get("message")
if isinstance(message, dict) and message.get("role") == "user":
text = extract_text(message.get("content"))
elif isinstance(message, str):
text = message
else:
text = ""
if not text:
continue
prompt_candidates.append(text)
candidate = first_meaningful_title(prompt_candidates, max_title_chars)
if candidate and len(candidate) >= 4:
title = candidate
if title is None:
title = first_meaningful_title(prompt_candidates, max_title_chars)
if not title:
title = f"(untitled: {session_id})"
return ClaudeSessionSummary(
session_id=session_id,
cwd=cwd,
title=title,
created_at=timestamps.earliest,
updated_at=timestamps.latest,
timestamp_count=timestamps.count,
)
scripts/_core/codex.py
"""Shared Codex conversation provider for the local-history skills.
Reading Codex history is subtle: recent builds keep a `state_*.sqlite` index whose
schema drifts between versions, and older ones only leave raw `rollout-*.jsonl`
files. This module encapsulates both backends (with the sqlite path degrading to a
raw-rollout scan when the schema is incompatible or a query fails) so that every
skill sees one `collect_codex(args, home) -> ProviderResult` entry point.
It lives in the shared core (SSOT: `daymade-claude-code/_conversation_core/`,
bundled into each skill's `scripts/_core/` by `sync_core.py`) so `list_local_history`
and a future `continue-codex-work` skill reuse one implementation instead of
re-deriving the schema-tolerance and subagent-detection heuristics that would drift
apart. The model / parse / text helpers it builds on already live in the same core.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sqlite3
from pathlib import Path
from typing import Any, Iterable, Optional
from urllib.parse import quote
from .model import CodexDatabase, Conversation, ProviderResult
from .parse import TimestampRange, parse_timestamp, workspace_matches
from .text import extract_text, first_meaningful_title, is_automated_title, iter_jsonl
CODEX_REQUIRED_COLUMNS = {"id", "cwd", "updated_at", "source", "archived"}
SESSION_ID_RE = re.compile(
r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}",
re.IGNORECASE,
)
STATE_DATABASE_RE = re.compile(r"^state_(\d+)\.sqlite$")
def sqlite_uri(path: Path) -> str:
return "file:" + quote(path.resolve().as_posix(), safe="/:@") + "?mode=ro"
def inspect_codex_database(path: Path) -> CodexDatabase:
connection = sqlite3.connect(sqlite_uri(path), uri=True, timeout=1.0)
try:
connection.execute("PRAGMA query_only = ON")
columns = {
str(row[1]) for row in connection.execute("PRAGMA table_info(threads)")
}
if not CODEX_REQUIRED_COLUMNS.issubset(columns):
missing = ", ".join(sorted(CODEX_REQUIRED_COLUMNS - columns))
raise ValueError(f"threads schema is missing: {missing}")
updated_expression = (
"COALESCE(MAX(CASE WHEN updated_at_ms > 0 THEN updated_at_ms "
"ELSE updated_at * 1000 END), 0)"
if "updated_at_ms" in columns
else "COALESCE(MAX(updated_at) * 1000, 0)"
)
value = connection.execute(
f"SELECT {updated_expression} FROM threads"
).fetchone()[0]
return CodexDatabase(path=path, columns=columns, max_updated_ms=int(value or 0))
finally:
connection.close()
def discover_codex_database(home: Path, warnings: list[str]) -> Optional[CodexDatabase]:
candidates: set[Path] = set()
for directory in (home, home / "sqlite"):
if not directory.is_dir():
continue
candidates.update(directory.glob("state_*.sqlite"))
compatible: list[CodexDatabase] = []
for path in sorted(candidates):
try:
compatible.append(inspect_codex_database(path))
except (OSError, sqlite3.Error, ValueError) as error:
warnings.append(f"Ignoring incompatible Codex database {path}: {error}")
if not compatible:
if candidates:
warnings.append(
"No compatible Codex state database; scanning raw rollout JSONL instead."
)
return None
return max(
compatible,
key=lambda item: (
item.max_updated_ms,
_state_database_generation(item.path),
item.path.as_posix(),
),
)
def _state_database_generation(path: Path) -> int:
"""Return the numeric state database generation without consulting mtime."""
match = STATE_DATABASE_RE.fullmatch(path.name)
return int(match.group(1)) if match else -1
def nested_key_exists(value: Any, wanted: str) -> bool:
if isinstance(value, str):
stripped = value.lstrip()
if stripped.startswith("{") or stripped.startswith("["):
try:
value = json.loads(value)
except json.JSONDecodeError:
return False
else:
return False
if isinstance(value, dict):
return wanted in value or any(
nested_key_exists(item, wanted) for item in value.values()
)
if isinstance(value, list):
return any(nested_key_exists(item, wanted) for item in value)
return False
def codex_row_is_subagent(row: sqlite3.Row) -> bool:
role = row["agent_role"]
thread_source = row["thread_source"]
source = row["source"]
return bool(role) or str(thread_source or "").casefold() == "subagent" or nested_key_exists(
source, "subagent"
)
def choose_row_title(row: sqlite3.Row, max_chars: int) -> str:
title = first_meaningful_title(
(row["title"], row["first_user_message"], row["preview"]), max_chars
)
return title or f"(untitled: {row['id']})"
def value_or_none(row: sqlite3.Row, key: str) -> Any:
try:
return row[key]
except (IndexError, KeyError):
return None
def dynamic_select(columns: set[str], names: Iterable[str]) -> str:
return ", ".join(
name if name in columns else f"NULL AS {name}" for name in names
)
def collect_codex_from_database(
args: argparse.Namespace, home: Path, database: CodexDatabase, result: ProviderResult
) -> None:
fields = (
"id",
"title",
"first_user_message",
"preview",
"cwd",
"created_at",
"created_at_ms",
"updated_at",
"updated_at_ms",
"source",
"thread_source",
"agent_role",
"archived",
"rollout_path",
)
query = f"SELECT {dynamic_select(database.columns, fields)} FROM threads"
connection = sqlite3.connect(sqlite_uri(database.path), uri=True, timeout=1.0)
connection.row_factory = sqlite3.Row
try:
connection.execute("PRAGMA query_only = ON")
rows = connection.execute(query)
for row in rows:
cwd = str(row["cwd"] or "")
if not args.all_projects and not workspace_matches(
cwd, args.cwd, args.recursive
):
continue
archived = bool(row["archived"] or False)
if archived and not args.include_archived:
result.excluded_archived += 1
continue
subagent = codex_row_is_subagent(row)
if subagent and not args.include_subagents:
result.excluded_subagents += 1
continue
title = choose_row_title(row, args.max_title_chars)
if is_automated_title(title) and not args.include_automated:
result.excluded_automated += 1
continue
updated_at = parse_timestamp(value_or_none(row, "updated_at_ms"))
if updated_at is None:
updated_at = parse_timestamp(value_or_none(row, "updated_at"))
created_at = parse_timestamp(value_or_none(row, "created_at_ms"))
if created_at is None:
created_at = parse_timestamp(value_or_none(row, "created_at"))
result.conversations.append(
Conversation(
provider="codex",
session_id=str(row["id"]),
title=title,
cwd=cwd,
updated_at=updated_at,
created_at=created_at,
archived=archived,
kind="subagent" if subagent else "main",
path=str(row["rollout_path"] or ""),
metadata_source="state-db",
timestamp_source="state-db",
)
)
finally:
connection.close()
def load_codex_session_index(home: Path, max_chars: int) -> dict[str, str]:
titles: dict[str, str] = {}
path = home / "session_index.jsonl"
if not path.is_file():
return titles
for record in iter_jsonl(path):
session_id = record.get("id")
title = first_meaningful_title((record.get("thread_name"),), max_chars)
if isinstance(session_id, str) and title:
titles[session_id] = title
return titles
def codex_meta_from_rollout(path: Path) -> Optional[dict[str, Any]]:
for record in iter_jsonl(path, bounded=True):
if record.get("type") == "session_meta" and isinstance(
record.get("payload"), dict
):
return record["payload"]
return None
def codex_session_id(meta: dict[str, Any], path: Path) -> Optional[str]:
"""Read the authoritative ID, with a UUID filename fallback when unambiguous."""
value = meta.get("id")
if isinstance(value, str) and value.strip():
return value.strip()
match = SESSION_ID_RE.search(path.name)
return match.group(0) if match else None
def codex_prompt_from_rollout(path: Path, max_chars: int) -> Optional[str]:
short_candidate: Optional[str] = None
for record in iter_jsonl(path, bounded=True):
candidate = ""
if record.get("type") == "response_item":
payload = record.get("payload")
if (
isinstance(payload, dict)
and payload.get("type") == "message"
and payload.get("role") == "user"
):
candidate = extract_text(payload.get("content"))
elif record.get("type") == "event_msg":
payload = record.get("payload")
if isinstance(payload, dict) and payload.get("type") == "user_message":
candidate = str(payload.get("message") or "")
if not candidate:
continue
title = first_meaningful_title((candidate,), max_chars)
if not title:
continue
if len(title) >= 4:
return title
short_candidate = short_candidate or title
return short_candidate
def codex_rollout_time_range(path: Path) -> TimestampRange:
"""Compute exact internal bounds for a Codex rollout.
Current rollouts timestamp every top-level event. Older/minimal fixtures may
carry only ``session_meta.payload.timestamp``, so observe both. File mtime is
deliberately excluded because copying or migrating a rollout rewrites it.
"""
timestamps = TimestampRange()
for record in iter_jsonl(path):
timestamps.observe(record.get("timestamp"))
if record.get("type") == "session_meta" and isinstance(
record.get("payload"), dict
):
timestamps.observe(record["payload"].get("timestamp"))
return timestamps
def collect_codex_from_rollouts(
args: argparse.Namespace, home: Path, result: ProviderResult
) -> None:
titles = load_codex_session_index(home, args.max_title_chars)
files: list[tuple[Path, bool]] = []
sessions_dir = home / "sessions"
if sessions_dir.is_dir():
files.extend((path, False) for path in sessions_dir.rglob("rollout-*.jsonl"))
archived_dir = home / "archived_sessions"
if archived_dir.is_dir():
files.extend((path, True) for path in archived_dir.rglob("rollout-*.jsonl"))
if not files:
result.warnings.append(f"No Codex rollout files found under {home}")
return
for path, archived in files:
meta = codex_meta_from_rollout(path)
if meta is None:
result.warnings.append(f"Skipping rollout without session_meta: {path}")
continue
session_id = codex_session_id(meta, path)
if session_id is None:
result.warnings.append(f"Skipping rollout without a session ID: {path}")
continue
cwd = str(meta.get("cwd") or "")
if not args.all_projects and not workspace_matches(cwd, args.cwd, args.recursive):
continue
if archived and not args.include_archived:
result.excluded_archived += 1
continue
subagent = nested_key_exists(meta.get("source"), "subagent")
if subagent and not args.include_subagents:
result.excluded_subagents += 1
continue
title = titles.get(session_id)
if not title:
title = codex_prompt_from_rollout(path, args.max_title_chars)
title = title or f"(untitled: {session_id})"
if is_automated_title(title) and not args.include_automated:
result.excluded_automated += 1
continue
timestamps = codex_rollout_time_range(path)
result.conversations.append(
Conversation(
provider="codex",
session_id=session_id,
title=title,
cwd=cwd,
updated_at=timestamps.latest,
created_at=timestamps.earliest,
archived=archived,
kind="subagent" if subagent else "main",
path=str(path),
metadata_source="rollout-jsonl",
timestamp_source=(
"rollout-record-minmax" if timestamps.count else "unknown"
),
)
)
def collect_codex(args: argparse.Namespace, home: Path) -> ProviderResult:
result = ProviderResult(provider="codex", backend="none", home=str(home))
if not home.is_dir():
result.warnings.append(f"Codex home directory not found: {home}")
return result
database = discover_codex_database(home, result.warnings)
if database is not None:
relative = os.path.relpath(database.path, home).replace(os.sep, "/")
result.backend = f"sqlite:{relative}"
try:
collect_codex_from_database(args, home, database, result)
except sqlite3.Error as error:
result.warnings.append(
f"Codex database query failed ({error}); scanning raw rollout JSONL instead."
)
result.backend = "rollout-jsonl"
result.conversations.clear()
result.excluded_subagents = 0
result.excluded_archived = 0
result.excluded_automated = 0
collect_codex_from_rollouts(args, home, result)
else:
result.backend = "rollout-jsonl"
collect_codex_from_rollouts(args, home, result)
deduplicated = {item.session_id: item for item in result.conversations}
result.conversations = sorted(
deduplicated.values(),
key=lambda item: item.updated_at if item.updated_at is not None else float("-inf"),
reverse=True,
)
return result
scripts/_core/homes.py
"""Discover every Claude config home that holds conversation history.
Claude Code stores session history under ``<home>/projects/``. The default home
is ``~/.claude``, but a user who runs third-party models through per-model
*profiles* — each profile is its own ``CLAUDE_CONFIG_DIR`` — keeps a parallel
history under ``~/.claude-profiles/<name>/`` and sometimes a sibling
``~/.claude-<name>/``. A tool that only looks at ``~/.claude`` silently misses
every conversation held under a profile, which is the #1 reason a real session
is wrongly reported as "not found".
This module is the single source of truth for that discovery. It is bundled
(copied) into each conversation-history skill's ``scripts/_core/`` so every skill
stays self-contained yet shares one implementation — fix the blind spot once,
not once per skill.
"""
import os
from pathlib import Path
from typing import List, Optional, Union
def discover_claude_homes(
explicit: Optional[Union[str, Path, List[Union[str, Path]]]] = None,
) -> List[Path]:
"""Return every Claude config home that has a ``projects/`` history dir.
Discovery is dynamic (glob), never a hardcoded profile list, so it adapts to
whatever profiles happen to exist on the machine.
Args:
explicit: ``None`` (default) auto-discovers all homes. A single path or a
list of paths restricts the search to exactly those homes (each must
contain a ``projects/`` subdir). An explicit request that matches no
home-with-history returns ``[]`` — callers MUST treat that as "your
selection matched nothing", never as a cue to auto-discover, or a
scope-narrowing flag would silently widen to the widest scope.
Returns:
De-duplicated, existence-checked list of home ``Path`` objects, each with
a ``projects/`` subdirectory. Order: ``CLAUDE_CONFIG_DIR`` (if set),
``~/.claude``, then ``~/.claude-profiles/*`` and sibling ``~/.claude-*``
sorted by name.
"""
homes: List[Path] = []
seen = set()
def add(candidate: Union[str, Path]) -> None:
try:
home = Path(candidate).expanduser()
except Exception:
return
if not (home / "projects").is_dir():
return
try:
key = str(home.resolve())
except Exception:
key = str(home)
if key in seen:
return
seen.add(key)
homes.append(home)
# An explicit override (single path or list) short-circuits discovery.
if explicit is not None:
candidates = explicit if isinstance(explicit, (list, tuple)) else [explicit]
for candidate in candidates:
add(candidate)
return homes
# CLAUDE_CONFIG_DIR wins first when set, then the default home.
env_home = os.environ.get("CLAUDE_CONFIG_DIR")
if env_home:
add(env_home)
add(Path.home() / ".claude")
# Per-model profile homes: ~/.claude-profiles/*/
profiles_root = Path.home() / ".claude-profiles"
if profiles_root.is_dir():
for child in sorted(profiles_root.iterdir()):
add(child)
# Sibling homes: ~/.claude-<name>/ that carry their own projects/
for child in sorted(Path.home().glob(".claude-*")):
if child.name != ".claude-profiles":
add(child)
return homes
def home_label(home: Union[str, Path]) -> str:
"""Short, human-readable provenance label for a home path.
``~/.claude`` -> ``main``; ``~/.claude-profiles/kimi`` -> ``kimi``;
``~/.claude-deepseek`` -> ``claude-deepseek``. A sibling ``~/.claude-<name>``
home keeps its ``claude-`` prefix so it never collides with a same-named
``~/.claude-profiles/<name>`` profile. Used so output shows which profile a
session came from instead of an opaque absolute path.
"""
home = Path(home)
if home.name == ".claude":
return "main"
if home.parent.name == ".claude-profiles":
return home.name
if home.name.startswith(".claude-"):
return home.name[len("."):]
return home.name
scripts/_core/kimi.py
"""Shared Kimi CLI conversation provider for the local-history skills.
Kimi CLI (kimi-code) stores conversations under ``~/.kimi-code/`` (override with
``KIMI_HOME``). The on-disk layout below was verified against Kimi CLI 0.38.0
(wire ``protocol_version`` 1.5) on a real 26-session store:
::
<home>/session_index.jsonl
One JSON object per line: {sessionId: "session_<uuid>", sessionDir,
workDir}. A fast index only — the directories on disk are authoritative.
<home>/sessions/wd_<workspace>_<hash>/session_<uuid>/
state.json
{id: "session_<uuid>", cwd, title, titleKind, isCustomTitle,
lastPrompt, createdAt, updatedAt, archived, agents: {...}}.
createdAt/updatedAt are epoch MILLISECONDS.
agents/main/wire.jsonl — the primary run's event log.
agents/agent-N/wire.jsonl — subagent runs of the SAME session; they
are not separate conversations.
logs/kimi-code.log — diagnostics, not conversation content.
Wire records carry their timestamp in ``time`` (epoch ms); the first record is
``{"type": "metadata", "protocol_version", "created_at"}`` (also ms). The
record types that carry user-visible text are ``turn.prompt`` / ``turn.steer``
(``input: [{type: "text", text}]`` with ``origin.kind == "user"`` for genuine
human input) and ``context.append_message`` / ``context.append_loop_event``.
Injected context wrappers (observed: ``<git-context>``) can precede prompt
text; ``scrub_kimi_prompt`` strips a leading wrapper defensively before title
extraction. A state.json title shorter than 4 characters counts as weak and
yields to the first real user prompt in the main wire; longer weak auto-titles
(a bare "hello") are kept as-is — a documented heuristic boundary, not an
oversight. Boilerplate records (``config.update`` / ``profile.bind`` system
prompts, ``llm.tools_snapshot``, usage/token metrics) are deliberately
excluded from title extraction here and from the finder's search segments — a
keyword that only appears in a shared static system prompt would match every
session and is not conversation content.
This module is the single entry point both consumers use:
``list_local_history`` (inventory) and ``analyze_sessions`` (search support
helpers). Like the Codex provider, file mtime is never consulted: only
internal timestamps (state.json fields, wire ``time`` values) are observed.
"""
from __future__ import annotations
import argparse
import json
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Optional
from .model import Conversation, ProviderResult
from .parse import TimestampRange, parse_timestamp, workspace_matches
from .text import (
extract_text,
first_meaningful_title,
is_automated_title,
iter_jsonl,
)
KIMI_HOME_ENV = "KIMI_HOME"
KIMI_DEFAULT_HOME_DIRNAME = ".kimi-code"
# Kimi injects workspace context ahead of the user's actual first prompt.
# Only leading wrapper blocks are stripped; a user who genuinely types the tag
# later in the message keeps their text.
_KIMI_INJECTED_WRAPPERS = ("git-context",)
_KIMI_WRAPPER_RES = tuple(
re.compile(rf"^\s*<{tag}\b.*?</{tag}>\s*", re.DOTALL | re.IGNORECASE)
for tag in _KIMI_INJECTED_WRAPPERS
)
def default_kimi_home() -> Path:
return Path.home() / KIMI_DEFAULT_HOME_DIRNAME
def resolve_kimi_home(explicit: Optional[str]) -> Path:
"""Home precedence: CLI flag > ``KIMI_HOME`` env > ``~/.kimi-code``."""
return Path(
explicit or os.environ.get(KIMI_HOME_ENV) or default_kimi_home()
).expanduser()
def scrub_kimi_prompt(text: str) -> str:
"""Remove leading injected context wrappers from a user prompt."""
for pattern in _KIMI_WRAPPER_RES:
text = pattern.sub("", text, count=1)
return text.strip()
@dataclass(frozen=True)
class KimiSessionSummary:
session_id: str
cwd: str
title: str
created_at: Optional[float]
updated_at: Optional[float]
archived: bool
metadata_source: str
timestamp_source: str
def load_kimi_session_index(home: Path) -> dict[str, str]:
"""Map ``session_<uuid>`` -> ``workDir`` from ``session_index.jsonl``."""
workdirs: dict[str, str] = {}
path = home / "session_index.jsonl"
if not path.is_file():
return workdirs
for record in iter_jsonl(path):
session_id = record.get("sessionId")
workdir = record.get("workDir")
if isinstance(session_id, str) and isinstance(workdir, str) and workdir:
workdirs[session_id] = workdir
return workdirs
def iter_kimi_session_dirs(home: Path) -> list[Path]:
"""Enumerate session directories, tolerating bucket-name drift.
The ``wd_<name>_<hash>`` bucket names are a convention, not a contract, so
a directory qualifies as a session by containing ``state.json`` or an
``agents/`` subdirectory — not by matching the observed naming pattern.
"""
sessions_dir = home / "sessions"
if not sessions_dir.is_dir():
return []
found: list[Path] = []
try:
buckets = sorted(path for path in sessions_dir.iterdir() if path.is_dir())
except OSError:
return []
for bucket in buckets:
try:
children = sorted(path for path in bucket.iterdir() if path.is_dir())
except OSError:
continue
for child in children:
if (child / "state.json").is_file() or (child / "agents").is_dir():
found.append(child)
return found
def kimi_wire_files(session_dir: Path) -> tuple[Optional[Path], list[Path]]:
"""Return ``(main_wire, subagent_wires)`` for one session directory."""
agents_dir = session_dir / "agents"
if not agents_dir.is_dir():
return None, []
main: Optional[Path] = None
subagents: list[Path] = []
try:
agent_dirs = sorted(path for path in agents_dir.iterdir() if path.is_dir())
except OSError:
return None, []
for agent_dir in agent_dirs:
wire = agent_dir / "wire.jsonl"
if not wire.is_file():
continue
if agent_dir.name == "main":
main = wire
else:
subagents.append(wire)
return main, subagents
def load_kimi_state(session_dir: Path) -> Optional[dict[str, Any]]:
path = session_dir / "state.json"
try:
value = json.loads(path.read_text(encoding="utf-8", errors="replace"))
except (OSError, json.JSONDecodeError, UnicodeError):
return None
return value if isinstance(value, dict) else None
def kimi_wire_time_range(path: Path) -> TimestampRange:
"""Exact internal bounds from wire ``time`` fields (+ metadata created_at)."""
timestamps = TimestampRange()
for record in iter_jsonl(path):
timestamps.observe(record.get("time"))
if record.get("type") == "metadata":
timestamps.observe(record.get("created_at"))
return timestamps
def kimi_prompt_from_wire(path: Path, max_chars: int) -> Optional[str]:
"""First meaningful user prompt in a wire file, injection wrappers removed."""
prompt_candidates: list[str] = []
for record in iter_jsonl(path, bounded=True):
if record.get("type") != "turn.prompt":
continue
origin = record.get("origin")
if not isinstance(origin, dict) or origin.get("kind") != "user":
continue
text = scrub_kimi_prompt(extract_text(record.get("input")))
if text:
prompt_candidates.append(text)
return first_meaningful_title(prompt_candidates, max_chars)
def scan_kimi_session(
session_dir: Path,
index_workdirs: Optional[dict[str, str]] = None,
max_title_chars: int = 120,
) -> KimiSessionSummary:
"""Summarize one Kimi session: state.json first, wire JSONL as fallback.
``state.json`` is authoritative for id/cwd/title/archived and the
millisecond createdAt/updatedAt bounds. When it is missing or a field is
absent, the main agent's ``wire.jsonl`` supplies the fallback (first user
prompt for the title, min/max ``time`` for the range). Subagent wires only
extend the time-range fallback; they are runs of the same session, not
separate conversations.
"""
state = load_kimi_state(session_dir)
main_wire, subagent_wires = kimi_wire_files(session_dir)
session_id = session_dir.name
cwd = ""
title: Optional[str] = None
created_at: Optional[float] = None
updated_at: Optional[float] = None
created_source: Optional[str] = None
updated_source: Optional[str] = None
archived = False
metadata_source = "wire-jsonl"
if state is not None:
metadata_source = "state-json"
raw_id = state.get("id")
if isinstance(raw_id, str) and raw_id.strip():
session_id = raw_id.strip()
raw_cwd = state.get("cwd")
if isinstance(raw_cwd, str) and raw_cwd.strip():
cwd = raw_cwd.strip()
archived = bool(state.get("archived") or False)
created_at = parse_timestamp(state.get("createdAt"))
if created_at is not None:
created_source = "state"
updated_at = parse_timestamp(state.get("updatedAt"))
if updated_at is not None:
updated_source = "state"
raw_title = state.get("title")
if isinstance(raw_title, str):
title = first_meaningful_title((raw_title,), max_title_chars)
if title is None or len(title) < 4:
# A trivial auto-title ("hi") is worse than the real first prompt.
raw_last = state.get("lastPrompt")
if isinstance(raw_last, str):
better = first_meaningful_title((raw_last,), max_title_chars)
if better and (title is None or len(better) > len(title)):
title = better
if not cwd and index_workdirs:
cwd = index_workdirs.get(session_id, "")
if title is None or len(title) < 4 or created_at is None or updated_at is None:
wire_title: Optional[str] = None
timestamps = TimestampRange()
for wire in ([main_wire] if main_wire else []) + subagent_wires:
if wire is None:
continue
if wire_title is None and wire == main_wire:
wire_title = kimi_prompt_from_wire(wire, max_title_chars)
wire_range = kimi_wire_time_range(wire)
timestamps.observe(wire_range.earliest)
timestamps.observe(wire_range.latest)
if (title is None or len(title) < 4) and wire_title:
title = wire_title
if created_at is None:
created_at = timestamps.earliest
if created_at is not None:
created_source = "wire"
if updated_at is None:
updated_at = timestamps.latest
if updated_at is not None:
updated_source = "wire"
timestamp_sources = {s for s in (created_source, updated_source) if s}
if timestamp_sources == {"state"}:
timestamp_source = "state-json"
elif timestamp_sources == {"wire"}:
timestamp_source = "wire-record-minmax"
elif timestamp_sources:
timestamp_source = "state-json+wire"
else:
timestamp_source = "unknown"
if not title:
title = f"(untitled: {session_id})"
return KimiSessionSummary(
session_id=session_id,
cwd=cwd,
title=title,
created_at=created_at,
updated_at=updated_at,
archived=archived,
metadata_source=metadata_source,
timestamp_source=timestamp_source,
)
def collect_kimi(args: argparse.Namespace, home: Path) -> ProviderResult:
"""Inventory every Kimi CLI session under ``home`` (read-only)."""
result = ProviderResult(provider="kimi", backend="none", home=str(home))
if not home.is_dir():
result.warnings.append(f"Kimi CLI home directory not found: {home}")
return result
if not (home / "sessions").is_dir():
result.warnings.append(
f"Kimi CLI sessions directory not found: {home / 'sessions'}"
)
return result
index = load_kimi_session_index(home)
metadata_backends: set[str] = set()
for session_dir in iter_kimi_session_dirs(home):
summary = scan_kimi_session(session_dir, index, args.max_title_chars)
metadata_backends.add(summary.metadata_source)
if summary.archived and not args.include_archived:
result.excluded_archived += 1
continue
if (
not args.all_projects
and summary.cwd
and not workspace_matches(summary.cwd, args.cwd, args.recursive)
):
continue
if is_automated_title(summary.title) and not args.include_automated:
result.excluded_automated += 1
continue
result.conversations.append(
Conversation(
provider="kimi",
session_id=summary.session_id,
title=summary.title,
cwd=summary.cwd,
updated_at=summary.updated_at,
created_at=summary.created_at,
archived=summary.archived,
kind="main",
path=str(session_dir),
metadata_source=summary.metadata_source,
timestamp_source=summary.timestamp_source,
)
)
deduplicated = {item.session_id: item for item in result.conversations}
if metadata_backends == {"state-json"}:
result.backend = "state-json"
elif metadata_backends == {"wire-jsonl"}:
result.backend = "wire-jsonl"
elif metadata_backends:
result.backend = "state-json+wire-jsonl"
result.conversations = sorted(
deduplicated.values(),
key=lambda item: item.updated_at if item.updated_at is not None else float("-inf"),
reverse=True,
)
return result
scripts/_core/model.py
"""Shared conversation data model for the local-history skills.
`Conversation` / `ProviderResult` / `CodexDatabase` are used by both the Claude
and Codex providers and by the renderers, so they live in the shared core (SSOT:
`daymade-claude-code/_conversation_core/`, bundled into each skill's
`scripts/_core/` by `sync_core.py`). This lets `continue-codex-work` and the
inventory/search skills reuse the same model without re-declaring it.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any, Optional
from .parse import iso_timestamp
@dataclass
class Conversation:
provider: str
session_id: str
title: str
cwd: str
updated_at: Optional[float]
created_at: Optional[float]
archived: bool
kind: str
path: str
metadata_source: str
timestamp_source: str
source_kind: str = ""
source_labels: list[str] = field(default_factory=list)
def to_dict(self) -> dict[str, Any]:
data = asdict(self)
data["updated_at"] = (
iso_timestamp(self.updated_at) if self.updated_at is not None else None
)
data["created_at"] = (
iso_timestamp(self.created_at) if self.created_at is not None else None
)
return data
@dataclass
class ProviderResult:
provider: str
backend: str
home: str
conversations: list[Conversation] = field(default_factory=list)
excluded_subagents: int = 0
excluded_archived: int = 0
excluded_automated: int = 0
warnings: list[str] = field(default_factory=list)
@property
def total(self) -> int:
return len(self.conversations)
@dataclass
class CodexDatabase:
path: Path
columns: set[str]
max_updated_ms: int
scripts/_core/parse.py
"""Shared parsing / formatting helpers for the conversation-history skills.
Pure, self-contained utilities that every skill would otherwise re-implement.
Bundled into each skill's ``scripts/_core/`` by ``sync_core.py`` (see homes.py
for why bundling is used instead of importing a sibling).
"""
import os
import re
import sys
from dataclasses import dataclass
from datetime import date, datetime, time
from typing import Any, Optional
WINDOWS_DRIVE_RE = re.compile(r"^(?:[/\\]{2}\?[/\\])?[A-Za-z]:[/\\]")
DATE_ONLY_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
@dataclass
class TimestampRange:
"""Minimum/maximum valid internal timestamp observed while streaming."""
earliest: Optional[float] = None
latest: Optional[float] = None
count: int = 0
def observe(self, value: Any) -> Optional[float]:
parsed = parse_timestamp(value)
if parsed is None:
return None
self.count += 1
if self.earliest is None or parsed < self.earliest:
self.earliest = parsed
if self.latest is None or parsed > self.latest:
self.latest = parsed
return parsed
def parse_timestamp(value: Any) -> Optional[float]:
"""Parse a seconds/millis epoch number or ISO-8601 string to epoch seconds.
Returns None for empty/invalid input. Values above 10^10 are treated as
milliseconds (Claude/Codex both persist ms in places), everything else as
seconds.
"""
if value is None or value == "":
return None
if isinstance(value, (int, float)):
numeric = float(value)
if numeric <= 0:
return None
return numeric / 1000 if numeric > 10_000_000_000 else numeric
if isinstance(value, str):
text = value.strip()
if not text:
return None
try:
numeric = float(text)
if numeric <= 0:
return None
return numeric / 1000 if numeric > 10_000_000_000 else numeric
except ValueError:
pass
try:
return datetime.fromisoformat(text.replace("Z", "+00:00")).timestamp()
except ValueError:
return None
return None
def timezone_offset_colon(value: str) -> str:
"""Insert the colon in a ``+0800`` style offset -> ``+08:00`` (else as-is)."""
if len(value) == 5 and value[0] in "+-":
return value[:3] + ":" + value[3:]
return value
def format_timestamp(value: float) -> str:
"""Human-readable local time with an explicit timezone offset."""
local = datetime.fromtimestamp(value).astimezone()
return local.strftime("%Y-%m-%d %H:%M ") + timezone_offset_colon(
local.strftime("%z")
)
def iso_timestamp(value: float) -> str:
"""ISO-8601 local timestamp (seconds precision, with offset)."""
return datetime.fromtimestamp(value).astimezone().isoformat(timespec="seconds")
def parse_date_boundary(value: str, *, end: bool = False) -> float:
"""Parse a date-only local boundary or a timezone-qualified ISO datetime.
Date-only input uses the machine's local timezone and covers the full day.
Datetime input must carry ``Z`` or an explicit UTC offset; accepting a naive
datetime would make a cross-machine history query change meaning silently.
"""
text_value = value.strip()
if DATE_ONLY_RE.fullmatch(text_value):
parsed_date = date.fromisoformat(text_value)
wall_time = time.max if end else time.min
return datetime.combine(parsed_date, wall_time).astimezone().timestamp()
try:
parsed = datetime.fromisoformat(text_value.replace("Z", "+00:00"))
except ValueError as error:
raise ValueError(
f"invalid ISO date/time {value!r}; use YYYY-MM-DD or include a timezone offset"
) from error
if parsed.tzinfo is None:
raise ValueError(
f"datetime {value!r} has no timezone; use Z or an explicit UTC offset"
)
return parsed.timestamp()
def timestamp_in_window(
value: Optional[float],
from_timestamp: Optional[float],
to_timestamp: Optional[float],
) -> bool:
if value is None:
return from_timestamp is None and to_timestamp is None
if from_timestamp is not None and value < from_timestamp:
return False
if to_timestamp is not None and value > to_timestamp:
return False
return True
def range_overlaps_window(
earliest: Optional[float],
latest: Optional[float],
from_timestamp: Optional[float],
to_timestamp: Optional[float],
) -> bool:
"""Whether an internal session range overlaps an inclusive query window."""
if earliest is None or latest is None:
return from_timestamp is None and to_timestamp is None
if from_timestamp is not None and latest < from_timestamp:
return False
if to_timestamp is not None and earliest > to_timestamp:
return False
return True
def looks_like_windows_path(value: str) -> bool:
"""True for a Windows drive path (``C:\\...``) or a UNC path (``\\\\host``)."""
return bool(WINDOWS_DRIVE_RE.match(value)) or value.startswith("\\\\")
def normalize_workspace(value: str) -> str:
"""Normalize a persisted cwd without guessing Windows/WSL equivalence."""
value = os.path.expandvars(os.path.expanduser(str(value).strip()))
if looks_like_windows_path(value):
normalized = value.replace("\\", "/")
if normalized.startswith("//?/"):
normalized = normalized[4:]
normalized = re.sub(r"/+", "/", normalized).rstrip("/")
return normalized.casefold()
normalized = os.path.abspath(os.path.normpath(value)).rstrip(os.sep)
if sys.platform == "darwin":
return normalized.casefold()
return os.path.normcase(normalized)
def workspace_matches(candidate: str, target: Optional[str], recursive: bool) -> bool:
"""Whether a ``candidate`` cwd matches ``target`` (exact, or nested if recursive)."""
if target is None:
return True
normalized_candidate = normalize_workspace(candidate)
normalized_target = normalize_workspace(target)
if normalized_candidate == normalized_target:
return True
if not recursive:
return False
separator = "/" if "/" in normalized_target else os.sep
return normalized_candidate.startswith(normalized_target.rstrip(separator) + separator)
scripts/_core/sources.py
"""Discover active and explicitly registered Claude conversation sources.
Active Claude homes are auto-discovered by :mod:`homes`. Long-term archives are
different: their location is user configuration, not a filename convention that
the public skill should guess. A small manifest at
``~/.claude/history-sources.json`` registers those archives explicitly.
The manifest is intentionally fail-fast. A malformed file, duplicate archive, or
missing required source is configuration damage, not a cue to silently fall back
to the active homes and produce an incomplete history result.
"""
from __future__ import annotations
import json
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Optional, Sequence, Union
from .homes import discover_claude_homes, home_label
MANIFEST_VERSION = 1
SOURCE_LABEL_RE = re.compile(r"^[A-Za-z0-9._-]+$")
class HistorySourceConfigError(ValueError):
"""Raised when an explicit history-source registry cannot be trusted."""
@dataclass(frozen=True)
class HistorySource:
"""One Claude configuration root that contains a ``projects/`` directory."""
provider: str
kind: str
label: str
home: Path
required: bool = True
@property
def display_label(self) -> str:
return f"{self.kind}:{self.label}"
def default_history_sources_path() -> Path:
"""Return the per-user registry path without caching ``Path.home()``."""
return Path.home() / ".claude" / "history-sources.json"
def _resolved_key(path: Path) -> str:
try:
return str(path.resolve())
except (OSError, RuntimeError):
return str(path.absolute())
def _active_sources(
homes: Sequence[Union[str, Path]],
) -> list[HistorySource]:
return [
HistorySource(
provider="claude",
kind="active",
label=home_label(home),
home=Path(home).expanduser(),
required=True,
)
for home in homes
]
def _read_manifest(path: Path) -> dict:
try:
raw = path.read_text(encoding="utf-8")
except OSError as error:
raise HistorySourceConfigError(
f"Cannot read history source registry {path}: {error}"
) from error
try:
payload = json.loads(raw)
except json.JSONDecodeError as error:
raise HistorySourceConfigError(
f"Invalid JSON in history source registry {path}: {error}"
) from error
if not isinstance(payload, dict):
raise HistorySourceConfigError(
f"History source registry {path} must contain a JSON object"
)
if payload.get("version") != MANIFEST_VERSION:
raise HistorySourceConfigError(
f"History source registry {path} must use version {MANIFEST_VERSION}"
)
if not isinstance(payload.get("sources"), list):
raise HistorySourceConfigError(
f"History source registry {path} must contain a sources array"
)
return payload
def _manifest_archive_sources(
path: Path,
active: Sequence[HistorySource],
) -> tuple[list[HistorySource], list[str]]:
payload = _read_manifest(path)
warnings: list[str] = []
archives: list[HistorySource] = []
seen_paths = {_resolved_key(source.home) for source in active}
seen_labels: set[str] = set()
for index, entry in enumerate(payload["sources"]):
location = f"{path}: sources[{index}]"
if not isinstance(entry, dict):
raise HistorySourceConfigError(f"{location} must be an object")
provider = entry.get("provider")
kind = entry.get("kind")
label = entry.get("label")
home_value = entry.get("home")
required = entry.get("required", True)
if provider != "claude":
raise HistorySourceConfigError(
f"{location} has unsupported provider {provider!r}; only 'claude' is supported"
)
if kind != "archive":
raise HistorySourceConfigError(
f"{location} has unsupported kind {kind!r}; registered sources must be 'archive'"
)
if not isinstance(label, str) or not SOURCE_LABEL_RE.fullmatch(label):
raise HistorySourceConfigError(
f"{location}.label must use letters, numbers, dot, underscore, or hyphen"
)
if label in seen_labels:
raise HistorySourceConfigError(
f"Duplicate archive label {label!r} in history source registry {path}"
)
seen_labels.add(label)
if not isinstance(home_value, str) or not home_value.strip():
raise HistorySourceConfigError(f"{location}.home must be a non-empty string")
if not isinstance(required, bool):
raise HistorySourceConfigError(f"{location}.required must be true or false")
expanded = Path(os.path.expandvars(home_value)).expanduser()
home = expanded if expanded.is_absolute() else path.parent / expanded
key = _resolved_key(home)
if key in seen_paths:
raise HistorySourceConfigError(
f"Duplicate history source path in {path}: {home}"
)
seen_paths.add(key)
if not (home / "projects").is_dir():
message = f"Registered history source {label!r} has no projects/ directory: {home}"
if required:
raise HistorySourceConfigError(f"Required history source is unavailable. {message}")
warnings.append(message)
continue
archives.append(
HistorySource(
provider="claude",
kind="archive",
label=label,
home=home,
required=required,
)
)
return archives, warnings
def discover_claude_sources(
*,
explicit_homes: Optional[
Union[str, Path, Sequence[Union[str, Path]]]
] = None,
manifest_path: Optional[Union[str, Path]] = None,
) -> tuple[list[HistorySource], list[str]]:
"""Return Claude history sources plus non-fatal registry warnings.
``explicit_homes`` is an exact scope: registered archives are intentionally
not added. With no explicit scope, active homes are auto-discovered and the
default registry is loaded when present. Passing ``manifest_path`` makes that
file itself required, so a typo cannot silently disable archive coverage.
"""
if explicit_homes is not None:
return _active_sources(discover_claude_homes(explicit_homes)), []
active = _active_sources(discover_claude_homes())
explicit_manifest = manifest_path is not None
registry = (
Path(manifest_path).expanduser()
if explicit_manifest
else default_history_sources_path()
)
if not registry.is_file():
if explicit_manifest:
raise HistorySourceConfigError(
f"History source registry not found: {registry}"
)
return active, []
archives, warnings = _manifest_archive_sources(registry, active)
return active + archives, warnings
scripts/_core/text.py
"""Shared text / title / JSONL parsing helpers for the local-history skills.
These turn raw session content into a readable one-line title and iterate JSONL
transcripts safely. Both the Claude and Codex providers use them, so they live in
the shared core (SSOT: `daymade-claude-code/_conversation_core/`, bundled into
each skill's `scripts/_core/` by `sync_core.py`). Keeping them here lets the Codex
provider and `continue-codex-work` reuse one implementation instead of
re-deriving title/noise heuristics that would drift apart.
"""
from __future__ import annotations
import json
import os
import re
import shutil
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterable, Iterator, Optional
from .parse import looks_like_windows_path
MAX_PREFIX_BYTES = 2 * 1024 * 1024
MAX_PREFIX_LINES = 5000
NOISE_PREFIXES = (
"# agents.md instructions for ",
"<app-context",
"<collaboration_mode",
"<command-message",
"<command-name",
"<codex_internal_context",
"<environment_context",
"<local-command-caveat",
"<local-command-stdout",
"<permissions instructions",
"<recommended_plugins",
"<system-reminder",
)
AUTOMATED_TITLE_RE = re.compile(
r"^(?:reply|respond|return|print)\s+(?:with\s+)?exactly\b", re.IGNORECASE
)
ATTACHMENT_IMAGE_RE = re.compile(
r"^(?:<image\b|\[Image\s+#\d+\])", re.IGNORECASE
)
FILE_SUFFIX_RE = re.compile(r"\.[A-Za-z0-9]{1,16}$")
SLASH_COMMAND_RE = re.compile(r"^/[A-Za-z0-9_:-]+(?:[ \t].*)?$")
@dataclass(frozen=True)
class SearchSegment:
"""One searchable text field with its semantic provenance."""
source: str
text: str
def looks_like_attachment_prefix(value: str) -> bool:
"""Recognize attachment metadata without guessing from prompt length."""
stripped = value.strip()
if ATTACHMENT_IMAGE_RE.match(stripped):
return True
if "\n" in stripped:
return False
candidate = stripped.strip("`'\"")
path_like = candidate.startswith(("/", "~/")) or looks_like_windows_path(
candidate
)
return path_like and bool(FILE_SUFFIX_RE.search(candidate))
def strip_structural_metadata_lines(value: str) -> tuple[str, bool]:
"""Remove attachment and slash-command wrapper lines around a request."""
lines = [line.strip() for line in value.splitlines() if line.strip()]
removed_attachment = False
while lines:
if looks_like_attachment_prefix(lines[0]):
removed_attachment = True
lines.pop(0)
continue
if SLASH_COMMAND_RE.fullmatch(lines[0]):
lines.pop(0)
continue
break
while lines and SLASH_COMMAND_RE.fullmatch(lines[-1]):
lines.pop()
return "\n".join(lines).strip(), removed_attachment
def iter_jsonl(
path: Path,
*,
bounded: bool = False,
line_keywords: Optional[list[str]] = None,
) -> Iterator[dict[str, Any]]:
"""Yield each JSONL record as a dict.
``line_keywords`` is an optional cheap pre-check: when given, a line is
only handed to ``json.loads`` if it contains at least one of these
strings as a raw substring (case-insensitive — callers pass already
case-folded keywords and this function case-folds the line to match).
This exists because ``json.loads`` plus walking the resulting structure
is the expensive part of a keyword search, and file-level filtering
(skip whole files with no occurrence anywhere) turned out not to be
enough on its own: a keyword common across a user's session history
(e.g. "embedding") can appear in most FILES while still appearing in
only a small fraction of the LINES within each one, so ruling out
individual non-matching lines before parsing them is where the real
remaining cost lives.
Like the file-level pre-filter in ``files_possibly_matching``, this is
an over-approximation: a line can pass this check and still turn out to
have no *matchable* record once parsed (the keyword landed in a raw
JSON key/structural byte rather than an actual field value, or in a
field the structured extractor deliberately excludes). It must never be
an under-approximation — never skip a line that a full parse would have
matched — which is what makes it safe to use as a pure speedup.
Only pass this from a call site that has independently confirmed
skipping unselected lines cannot corrupt some other accounting the
caller performs across every record (see ``search_sessions``'s
``use_prefilter`` docstring for the specific case this codebase hit —
date-window "excluded because untimed" counts must see every record).
"""
consumed = 0
lines = 0
try:
with path.open("r", encoding="utf-8", errors="replace") as handle:
for line in handle:
consumed += len(line.encode("utf-8", errors="replace"))
lines += 1
if bounded and (consumed > MAX_PREFIX_BYTES or lines > MAX_PREFIX_LINES):
return
if line_keywords is not None:
haystack = line.casefold()
if not any(kw in haystack for kw in line_keywords) and not any(
marker in line for marker in _UNSCANNABLE_MARKERS
):
# Same over-approximation as the file-level filter: a
# line holding a fold-equivalent character or a \u
# escape may still match once parsed, so never skip it.
continue
try:
value = json.loads(line)
except (json.JSONDecodeError, TypeError):
continue
if isinstance(value, dict):
yield value
except (OSError, UnicodeError):
return
# Every character a JSON writer may store as something other than itself:
# the two mandatory escapes (" and \), the optional one (/), and all control
# characters. Enumerated from the JSON grammar rather than from observed
# escapes — an earlier version listed only control chars and "/", which
# covered a case Python never even produces while missing the two that every
# writer produces.
_JSON_ESCAPED_CHAR_RE = re.compile(r'["\\/\x00-\x1f]')
# The guard above can only inspect the KEYWORD. The other half of the problem
# is invisible to it: an all-ASCII keyword can legitimately match non-ASCII
# CONTENT, because Python's casefold() does *full* folding. Searching
# "financial" must match a transcript containing "financial" (U+FB01, which
# arrives whenever someone pastes from a PDF) — the real matcher does, a byte
# scanner does not.
#
# Enumerated from Python's own casefold table rather than written by hand:
# every non-ASCII code point whose casefold is pure ASCII. There are exactly
# 11 (ß ſ ẞ K and the ff/fi/fl/ffi/ffl/st ligatures). A hand-written list
# drafted for this fix had 17 entries, several of which do not actually fold
# to ASCII — which is the argument for deriving it.
_FOLDS_TO_ASCII = tuple(
chr(cp)
for cp in range(0x80, 0x11000)
if chr(cp).casefold() != chr(cp)
and chr(cp).casefold().isascii()
and chr(cp).casefold().strip()
)
# Any of these in a file/line means a byte scan cannot rule it out:
# - the fold-equivalent characters above
# - a "\u" escape: content written with ensure_ascii=True stores non-ASCII
# that way, and some writers (Go's encoding/json) escape even ASCII
# "<" ">" "&" as </>/&
_UNSCANNABLE_MARKERS = _FOLDS_TO_ASCII + ("\\u",)
def keywords_are_raw_byte_safe(keywords: Iterable[str]) -> bool:
"""Can a literal byte scan of the physical file stand in for matching
against the parsed strings?
Pass the **original** keywords, never case-folded ones. ``"ß".casefold()``
is ``"ss"`` — pure ASCII — so folding first makes this function answer
"safe" for exactly the input that motivated rule 2 below, silently
re-opening the hole it exists to close.
Only when every keyword's bytes are *guaranteed* to appear verbatim in the
file, and a byte scanner's notion of "equal" matches Python's. Three shapes
break that guarantee, and each one silently loses matches rather than
reporting an error — so all three fall back to full parsing:
1. **Characters JSON is required to escape**: control characters, ``"``,
and ``\\``. An embedded newline is stored as the two bytes ``\\n``, a
quote as ``\\"``, a backslash as ``\\\\`` — so a literal search for the
raw character cannot find the escaped form actually sitting in the
file. This is not a quirk of some serializers; every conforming JSON
writer does it, including the one that produced these transcripts.
Quoted error fragments (``Error: "ENOENT"``), config snippets
(``"model": "opus"``), and Windows paths are exactly the sort of thing
people search history for.
2. **Non-ASCII.** Two independent failures. (a) ``json.dumps`` defaults to
``ensure_ascii=True``, which stores ``café`` as ``caf\\u00e9`` — the
UTF-8 bytes are simply not in the file. Claude Code's own writer does
not do this, but archives rewritten by other tooling do. (b) Even in
raw UTF-8, case-insensitive folding disagrees across implementations:
measured on this corpus's tooling, Python ``casefold()`` does *full*
folding (``straße`` == ``STRASSE``), ``rg -i`` does only *simple*
folding (matches ``STRAẞE`` but not ``STRASSE``), and ``/usr/bin/grep
-i`` matches neither. A pre-filter that disagrees with the real matcher
drops sessions, and it would drop *different* ones depending on whether
``rg`` happens to be installed.
3. **A literal ``/``.** Optional in JSON — Python does not escape it — but
some writers emit ``\\/``, so it is excluded for the same reason.
Falling back is always the safe direction: it costs speed, never a missed
match. The cost is real and worth naming — a non-ASCII query (any CJK one)
gets no pre-filter at all and pays the full parse. Correctness first: this
tool's callers are told they may conclude a topic is absent from a
no-match result.
"""
return all(
kw.isascii()
and not _JSON_ESCAPED_CHAR_RE.search(kw)
for kw in keywords
)
def files_possibly_matching(
paths: Iterable[Path], keywords: Iterable[str], *, case_sensitive: bool = False
) -> Optional[set[Path]]:
"""Raw-byte pre-filter: which of ``paths`` could possibly contain any of
``keywords``, without paying per-line ``json.loads`` + structured
extraction on files that plainly cannot match?
This exists because the natural way to search this corpus — open every
session file, ``json.loads`` every line, extract the searchable text, and
substring-match keywords against it — parses every byte of every file
even when the overwhelming majority contain none of the keywords at all.
On a single project of 294 files / 1.6GB that pure-Python loop measured
3.5+ minutes of CPU time without completing; ``rg``/``grep`` scan the same
bytes at native speed and can usually rule out most files in well under a
second, letting the expensive path run only on the handful of files that
are actually candidates.
Deliberately an OVER-approximation, never an under-approximation: keys
excluded from search on purpose (``id``, ``tool_use_id``, ``signature`` —
see ``_flatten_search_strings``) can still contain a keyword's raw bytes,
so a small number of files this returns as "possible" will turn out to
have no real match once fully parsed. That costs a little wasted parsing,
which is fine. What must never happen is the reverse — silently ruling
out a file that genuinely contains a matchable occurrence — because
completeness is this tool's whole reason to exist (see the "Completeness
invariant" in the finder skill's own SKILL.md). So every failure mode
below degrades to "don't filter" rather than to "filter more":
- No keywords, or a keyword containing a raw control character (the
JSON-escaping mismatch above): returns ``None``.
- Neither ``rg`` nor ``grep`` is on PATH: returns ``None``.
- The scanner subprocess itself errors or times out: returns ``None``.
``None`` means "no filtering information — treat every path as a
candidate", which is exactly the pre-existing behavior before this
function existed. Callers must check for ``None`` and fall through to
scanning everything, not treat it as an empty result.
"""
path_list = list(paths)
keyword_list = list(keywords)
if not path_list or not keyword_list or not keywords_are_raw_byte_safe(keyword_list):
return None
exe = shutil.which("rg") or shutil.which("grep")
if not exe:
return None
# -a/--text: never let the binary-content heuristic skip a JSONL file that
# happens to contain a byte sequence that looks binary.
# -F: keywords are literal substrings everywhere else in this codebase
# (Python's str.count(), not a regex) — a raw scan must match the same
# semantics, not treat a keyword like "a.b" as a wildcard pattern.
# -l: list matching filenames only; we don't need line numbers here.
# Two scans, unioned — they need opposite case settings.
#
# Pass 1 matches the keywords the way the real matcher does (folded, unless
# the caller asked for case-sensitive).
#
# Pass 2 looks for content the byte scanner cannot reason about at all
# (fold-equivalent characters, "\u" escapes) and MUST be case-sensitive.
# Handing those characters to a "-i" scan is self-defeating: "-i" folds the
# *pattern* too, so `ſ` matches a plain `s` and `K` (U+212A) matches `k` —
# every file with an "s" or a "k" in it becomes a candidate and the filter
# stops filtering. (Caught by the pre-existing unit tests, which is exactly
# what they are for.)
kw_scan = [exe, "-a", "-F", "-l"]
if not case_sensitive:
kw_scan.append("-i")
for kw in keyword_list:
kw_scan += ["-e", kw]
kw_scan.append("--")
marker_scan = [exe, "-a", "-F", "-l"]
for marker in _UNSCANNABLE_MARKERS:
marker_scan += ["-e", marker]
marker_scan.append("--")
# Batch the paths: one exec per ~half of ARG_MAX. Passing every path in a
# single argv fails with E2BIG once the corpus is large enough — measured
# here at 7819 files needing 1,181,495 bytes against an ARG_MAX of
# 1,048,576. That failure degrades safely (OSError -> return None -> no
# filtering), but it means the speedup silently switched itself off in
# exactly the "tens of thousands of files" case that motivated it.
try:
arg_max = os.sysconf("SC_ARG_MAX")
except (ValueError, OSError):
arg_max = 256 * 1024
matched: set[Path] = set()
def run_batch(scan_prefix: list[str], paths_chunk: list[str]) -> bool:
"""Returns False if the scanner failed — caller degrades to no filter."""
try:
result = subprocess.run(
scan_prefix + paths_chunk, capture_output=True, text=True,
encoding="utf-8", errors="replace", timeout=120,
)
except (OSError, subprocess.TimeoutExpired):
return False
# Both rg and grep: 0 = matches found, 1 = no matches (not an error).
# Any other code (bad invocation, I/O error) is a scanner failure ->
# degrade to "don't filter" rather than trust a broken result.
if result.returncode not in (0, 1):
return False
matched.update(Path(line) for line in result.stdout.splitlines() if line)
return True
def scan_all(scan_prefix: list[str]) -> bool:
fixed = sum(len(a.encode("utf-8", "replace")) + 1 for a in scan_prefix)
chunk_budget = max(arg_max // 2 - fixed, 64 * 1024)
batch: list[str] = []
used = 0
for path in path_list:
text = str(path)
size = len(text.encode("utf-8", "replace")) + 1
if batch and used + size > chunk_budget:
if not run_batch(scan_prefix, batch):
return False
batch, used = [], 0
batch.append(text)
used += size
return not batch or run_batch(scan_prefix, batch)
if not scan_all(kw_scan) or not scan_all(marker_scan):
return None
return matched
def extract_text(content: Any) -> str:
if isinstance(content, str):
return content
if not isinstance(content, list):
return ""
parts: list[str] = []
for item in content:
if not isinstance(item, dict):
continue
if item.get("type") in {"text", "input_text"} and isinstance(
item.get("text"), str
):
parts.append(item["text"])
return " ".join(parts)
def _flatten_search_strings(value: Any) -> Iterator[str]:
"""Yield string values while excluding structural/signature metadata."""
if isinstance(value, str):
if value:
yield value
return
if isinstance(value, list):
for item in value:
yield from _flatten_search_strings(item)
return
if not isinstance(value, dict):
return
for key, child in value.items():
if key in {"type", "id", "tool_use_id", "signature"}:
continue
yield from _flatten_search_strings(child)
def searchable_segments(record: dict[str, Any]) -> list[SearchSegment]:
"""Extract user-visible/search-relevant fields from one Claude event.
Raw JSON serialization is intentionally not searched: keys, UUIDs, and
cryptographic thinking signatures create false positives. Instead this
covers message text, thinking text, tool inputs/results, queue content,
last-prompt/system summaries, and attachment payloads with a source label for
every segment.
"""
segments: list[SearchSegment] = []
def add(source: str, value: Any) -> None:
for text_value in _flatten_search_strings(value):
segments.append(SearchSegment(source=source, text=text_value))
event_type = record.get("type")
message = record.get("message")
content: Any
if isinstance(message, dict):
content = message.get("content", [])
elif isinstance(message, str):
content = message
elif event_type in {"user", "assistant"} or record.get("role") in {
"user",
"assistant",
}:
content = record.get("content", [])
else:
content = []
if isinstance(content, str):
add("message", content)
elif isinstance(content, list):
for block in content:
if not isinstance(block, dict):
continue
block_type = block.get("type")
if block_type in {"text", "input_text"}:
add("message", block.get("text"))
elif block_type == "thinking":
add("thinking", block.get("thinking"))
elif block_type == "tool_use":
tool_name = block.get("name")
source = (
f"tool_input:{tool_name}"
if isinstance(tool_name, str) and tool_name
else "tool_input"
)
add(source, block.get("input"))
elif block_type == "tool_result":
add("tool_result", block.get("content"))
else:
# Preserve textual payloads of future/older block types without
# indexing structural keys or binary image data.
add("message", {key: block.get(key) for key in ("text", "content")})
if event_type == "queue-operation":
add("queue-operation", record.get("content"))
elif event_type == "attachment":
attachment = record.get("attachment")
if isinstance(attachment, dict):
add(
"attachment",
{
key: attachment.get(key)
for key in (
"content",
"prompt",
"path",
"displayPath",
"command",
"stdout",
"stderr",
)
},
)
elif event_type == "last-prompt":
add("last-prompt", record.get("lastPrompt"))
elif event_type == "system":
add("system", record.get("content"))
elif event_type == "summary":
add("summary", {"content": record.get("content"), "summary": record.get("summary")})
elif event_type == "custom-title":
add(
"custom-title",
{"title": record.get("title"), "customTitle": record.get("customTitle")},
)
# Some event variants mirror the same payload in more than one compatible
# field. Count each semantic source/text pair once per record.
return list(dict.fromkeys(segments))
def is_noise_text(text: str) -> bool:
lowered = text.lstrip().casefold()
return not lowered or any(lowered.startswith(prefix) for prefix in NOISE_PREFIXES)
def clean_title(text: str, max_chars: int) -> str:
separator_parts = re.split(r"(?:^|\n)\s*-{4,}\s*(?:\n|$)", text)
if len(separator_parts) > 1:
raw_candidates = [part.strip() for part in separator_parts if part.strip()]
processed_candidates = [
strip_structural_metadata_lines(part) for part in raw_candidates
]
attachment_requests = [
candidate
for candidate, removed_attachment in processed_candidates
if candidate and removed_attachment
]
candidates = [candidate for candidate, _ in processed_candidates if candidate]
candidates = candidates or raw_candidates
if attachment_requests:
text = attachment_requests[-1]
elif candidates:
prefix = candidates[0]
tail = candidates[-1]
text = tail if len(tail) >= 20 else prefix
text = re.sub(r"^<image\b[^>]*>\s*", "", text, flags=re.IGNORECASE)
text = re.sub(r"</?image>\s*", "", text, flags=re.IGNORECASE)
text = re.sub(r"^\[Image\s+#\d+\]\s*", "", text, flags=re.IGNORECASE)
home = str(Path.home())
for home_variant in {home, home.replace("\\", "/"), home.replace("/", "\\")}:
if home_variant:
text = text.replace(home_variant, "~")
text = re.sub(r"[\r\n\t]+", " ", text)
text = re.sub(r"\s+", " ", text).strip()
if len(text) <= max_chars:
return text
return text[: max_chars - 1].rstrip() + "…"
def is_automated_title(title: str) -> bool:
return bool(AUTOMATED_TITLE_RE.match(title.strip()))
def first_meaningful_title(
candidates: Iterable[Any], max_chars: int
) -> Optional[str]:
short_candidate: Optional[str] = None
for candidate in candidates:
if not isinstance(candidate, str):
continue
cleaned = clean_title(candidate, max_chars)
if is_noise_text(cleaned):
continue
if len(cleaned) >= 4:
return cleaned
if cleaned and short_candidate is None:
short_candidate = cleaned
return short_candidate
scripts/analyze_sessions.py
#!/usr/bin/env python3
"""
Analyze Claude Code session files to find relevant sessions and statistics.
This script helps locate sessions containing specific keywords, analyze
session activity, and generate reports about session content.
By default, history is searched across every active Claude config home plus
every long-term archive registered in ~/.claude/history-sources.json. Searching
only ~/.claude or only the current active tree can silently miss a real session.
Conversation dates come from internal JSONL records, never file mtime.
Three opt-in widenings exist because "not found" is the expensive answer:
--all-projects sweeps every project when the project is a guess, --codex
also searches Codex rollout history (~/.codex), and --kimi also searches
Kimi CLI sessions (~/.kimi-code) — different stores that the Claude registry
never covers.
"""
import hashlib
import json
import os
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, List, Any, Optional
from collections import Counter, defaultdict
# Multi-home discovery lives in the bundled `_core` package — the single source
# of truth is daymade-claude-code/_conversation_core/, copied here into
# scripts/_core/ by sync_core.py so this skill stays self-contained. Make this
# script's own dir importable regardless of how it is invoked, then import.
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _core.claude import scan_claude_session # noqa: E402
from _core.codex import codex_meta_from_rollout, codex_session_id # noqa: E402
from _core.kimi import ( # noqa: E402
iter_kimi_session_dirs,
kimi_wire_files,
kimi_wire_time_range,
load_kimi_state,
resolve_kimi_home,
scan_kimi_session,
) # noqa: E402
from _core.homes import home_label # noqa: E402
from _core.parse import ( # noqa: E402
TimestampRange,
format_timestamp,
parse_date_boundary,
parse_timestamp,
range_overlaps_window,
timestamp_in_window,
workspace_matches,
)
from _core.sources import ( # noqa: E402
HistorySource,
HistorySourceConfigError,
discover_claude_sources,
)
from _core.text import ( # noqa: E402
SearchSegment,
extract_text,
files_possibly_matching,
is_automated_title,
iter_jsonl,
keywords_are_raw_byte_safe,
searchable_segments,
)
def _record_identity(record: Dict[str, Any]) -> str:
"""Return a stable identity for record-level union across session copies."""
canonical = json.dumps(
record,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
return "sha256:" + hashlib.sha256(canonical).hexdigest()
# ---------------------------------------------------------------------------
# Session tail classification (`triage` command) — see references/
# session_file_format.md "Detect Session Interruption" and "Tool Use / Tool
# Result Ordering" for the reasoning this implements.
# ---------------------------------------------------------------------------
# Structural terminal states. This is deliberately NOT a judgment about
# whether a human reply is expected — "done" only means the last assistant
# turn produced a normal text block, which is equally true of a session that
# fully wrapped up and one that surfaced a finding and is waiting for a
# response. Reading last_assistant_text is the caller's job.
TAIL_INTERRUPTED_EXPLICIT = "interrupted_explicit"
TAIL_NET_ERROR = "net_error"
TAIL_STUCK_NO_RESULT = "stuck_no_result"
TAIL_DONE = "done"
TAIL_EMPTY = "empty"
_INTERRUPTED_MARKER = "[Request interrupted by user"
_NET_ERROR_PREFIX = "API Error"
@dataclass
class SessionTail:
"""Structural classification of how a session's final turn ended."""
kind: str
last_user_text: str
last_assistant_kind: str # "text" | "tool_use" | "thinking_only" | "none"
last_assistant_text: str
last_assistant_timestamp: Optional[float]
def classify_session_tail(path: Path) -> SessionTail:
"""Classify a session's ending state by streaming its records once.
Resolves tool_use/tool_result across the *whole* file as a true
set-difference, not an incremental add/discard in file order: a
tool_result can be written before the tool_use record it answers (see
"Tool Use / Tool Result Ordering" in references/session_file_format.md).
A single-pass ``discard-then-add`` was tried first and is NOT actually
order-independent — ``discard()`` on an id not yet seen is a silent
no-op, so a tool_result appearing before its tool_use left the id
"pending" even though it was genuinely resolved (verified against real
session data, 2026-08: 11/14 files hitting this ordering had their
final `kind` flipped). Accumulating two never-mutated sets and diffing
them once at the end is immune to this, because neither operation can
ever silently miss the other regardless of which came first in the file.
Classification is also computed from the RAW content of the LAST
assistant record only, not from state that could carry over from an
earlier turn: a final turn that produces no text and no tool_use (e.g.
thinking-only) previously left `last_assistant_kind`/`last_assistant_text`
holding an earlier turn's already-answered reply, misreporting a session
that crashed before responding to its latest question as `done`.
The interruption marker is checked the same way: `tail_is_interrupt` is
reset by any later user or assistant record, so it only survives to the
end of the loop when the marker is the LAST relevant record in the file.
A mid-session Ctrl+C that the conversation continued past is not a tail
interruption — treating "marker appears anywhere" as equivalent to "the
session ended on interruption" was tried first and false-positived on
exactly that shape (verified against real session data, 2026-08).
"""
all_tool_use_ids: set = set()
all_resolved_tool_use_ids: set = set()
last_user_text = ""
last_assistant_content: Any = None
last_assistant_timestamp: Optional[float] = None
tail_is_interrupt = False
for record in iter_jsonl(path):
record_type = record.get("type")
message = record.get("message")
content = message.get("content") if isinstance(message, dict) else None
if record_type == "user" and not record.get("isMeta"):
if isinstance(content, str) and _INTERRUPTED_MARKER in content:
tail_is_interrupt = True
continue
tail_is_interrupt = False
if isinstance(content, str):
last_user_text = content
elif isinstance(content, list):
is_tool_result_only = bool(content) and all(
isinstance(block, dict) and block.get("type") == "tool_result"
for block in content
)
for block in content:
if not isinstance(block, dict):
continue
if block.get("type") == "tool_result":
tool_use_id = block.get("tool_use_id")
if tool_use_id is not None:
all_resolved_tool_use_ids.add(tool_use_id)
if not is_tool_result_only:
text = extract_text(content)
if text:
last_user_text = text
elif record_type == "assistant":
tail_is_interrupt = False
parsed_ts = parse_timestamp(record.get("timestamp"))
if parsed_ts is not None:
last_assistant_timestamp = parsed_ts
last_assistant_content = content
if isinstance(content, list):
for block in content:
if isinstance(block, dict) and block.get("type") == "tool_use":
tool_use_id = block.get("id")
if tool_use_id is not None:
all_tool_use_ids.add(tool_use_id)
pending_tool_use_ids = all_tool_use_ids - all_resolved_tool_use_ids
last_assistant_kind = "none"
last_assistant_text = ""
final_turn_has_pending_tool = False
if isinstance(last_assistant_content, list):
text_blocks = [
block.get("text", "")
for block in last_assistant_content
if isinstance(block, dict) and block.get("type") == "text"
]
tool_blocks = [
block
for block in last_assistant_content
if isinstance(block, dict) and block.get("type") == "tool_use"
]
joined_text = "\n".join(part for part in text_blocks if part)
final_turn_has_pending_tool = any(
block.get("id") in pending_tool_use_ids for block in tool_blocks
)
if final_turn_has_pending_tool:
last_assistant_kind = "tool_use"
pending_names = [
block.get("name", "?")
for block in tool_blocks
if block.get("id") in pending_tool_use_ids
]
last_assistant_text = f"[tool_use:{pending_names[-1]}]"
elif joined_text:
last_assistant_kind = "text"
last_assistant_text = joined_text
elif tool_blocks:
# Every tool_use in this turn already has a matching result
# elsewhere in the file, but no further assistant reply followed
# it — the harness likely stopped between the tool result
# landing and the model's next turn being captured. Distinct
# from `final_turn_has_pending_tool`: nothing is unresolved, but
# nothing was said either, so this is not a normal `done` reply.
last_assistant_kind = "tool_use"
last_assistant_text = f"[tool_use:{tool_blocks[-1].get('name', '?')}] (resolved, no further reply)"
else:
last_assistant_kind = "thinking_only"
last_assistant_text = (
"(final assistant turn has no text or tool_use block — "
"thinking-only or empty content)"
)
elif isinstance(last_assistant_content, str) and last_assistant_content:
last_assistant_kind = "text"
last_assistant_text = last_assistant_content
if tail_is_interrupt:
kind = TAIL_INTERRUPTED_EXPLICIT
elif last_assistant_kind == "none":
kind = TAIL_EMPTY
elif last_assistant_kind == "text" and last_assistant_text.startswith(
_NET_ERROR_PREFIX
):
kind = TAIL_NET_ERROR
elif last_assistant_kind == "text":
kind = TAIL_DONE
else:
# tool_use (pending or just-resolved-with-no-followup) and
# thinking_only all mean the same thing for triage purposes: the
# final turn produced no textual reply, so the model was still
# working when the file stopped.
kind = TAIL_STUCK_NO_RESULT
return SessionTail(
kind=kind,
last_user_text=last_user_text,
last_assistant_kind=last_assistant_kind,
last_assistant_text=last_assistant_text,
last_assistant_timestamp=last_assistant_timestamp,
)
# ---------------------------------------------------------------------------
# Codex rollout search (--codex)
#
# Codex stores conversations outside the Claude history registry: rollout
# JSONL files under <CODEX_HOME>/sessions/<YYYY>/<MM>/<DD>/ plus
# <CODEX_HOME>/archived_sessions/. Their record schema is NOT the Claude one
# (response_item/event_msg/session_meta, not user/assistant/queue-operation),
# so searchable_segments() does not apply. The extractor below covers the
# user-visible payload of each response_item variant. event_msg user/agent
# message records are deliberate strict mirrors of response_item message text
# (verified 2026-07-16: 26/26 and 104/104 subset on a real rollout), so they
# are skipped to avoid double-counting.
# ---------------------------------------------------------------------------
def _flatten_strings(value: Any) -> List[str]:
"""Flatten nested str/list/dict content into plain strings."""
if isinstance(value, str):
return [value]
if isinstance(value, list):
return [part for item in value for part in _flatten_strings(item)]
if isinstance(value, dict):
return [part for item in value.values() for part in _flatten_strings(item)]
return []
def codex_searchable_segments(record: Dict[str, Any]) -> List[SearchSegment]:
"""Extract searchable text fields from one Codex rollout record."""
segments: List[SearchSegment] = []
def add(source: str, value: Any) -> None:
for text_value in _flatten_strings(value):
segments.append(SearchSegment(source=source, text=text_value))
if record.get("type") == "compacted":
# Compaction records carry a summary of earlier conversation content.
payload = record.get("payload")
if isinstance(payload, dict):
add("summary", payload.get("message"))
return list(dict.fromkeys(segments))
if record.get("type") != "response_item":
return segments
payload = record.get("payload")
if not isinstance(payload, dict):
return segments
payload_type = payload.get("type")
if payload_type == "message":
for block in payload.get("content") or []:
if isinstance(block, dict) and block.get("type") in {
"input_text",
"output_text",
}:
add("message", block.get("text"))
elif payload_type == "reasoning":
for block in payload.get("summary") or []:
if isinstance(block, dict) and block.get("type") == "summary_text":
add("thinking", block.get("text"))
elif payload_type in {"function_call", "custom_tool_call"}:
name = payload.get("name")
source = (
f"tool_input:{name}"
if isinstance(name, str) and name
else "tool_input"
)
add(source, payload.get("arguments"))
add(source, payload.get("input"))
elif payload_type in {"function_call_output", "custom_tool_call_output"}:
add("tool_result", payload.get("output"))
return list(dict.fromkeys(segments))
def discover_codex_rollouts(codex_home: Path) -> List[Path]:
"""Enumerate Codex rollout files under a Codex home (sessions + archived)."""
rollouts: List[Path] = []
sessions_dir = codex_home / "sessions"
if sessions_dir.is_dir():
rollouts.extend(sorted(sessions_dir.rglob("*.jsonl")))
archived_dir = codex_home / "archived_sessions"
if archived_dir.is_dir():
rollouts.extend(sorted(archived_dir.glob("*.jsonl")))
return rollouts
def file_content_digest(path: Path) -> Optional[str]:
"""SHA-256 over a file's bytes, or ``None`` when it cannot be read.
``None`` means "no information" and must make the caller fall through to
the normal full parse — never to treating two files as identical.
"""
digest = hashlib.sha256()
try:
with open(path, "rb") as handle:
for block in iter(lambda: handle.read(1 << 20), b""):
digest.update(block)
except (OSError, ValueError):
return None
return digest.hexdigest()
def repeated_copy_sizes(copies: List[Dict[str, Any]]) -> set:
"""Byte sizes that occur more than once among a session's copies.
Only these are worth hashing: a size that appears once cannot have a
byte-identical twin, so single-copy sessions pay nothing for the
duplicate-copy optimisation.
"""
sizes = Counter()
for item in copies:
try:
sizes[item["path"].stat().st_size] += 1
except OSError:
continue
return {size for size, count in sizes.items() if count > 1}
def search_codex_rollouts(
rollouts: List[Path],
keywords: List[str],
case_sensitive: bool = False,
from_timestamp: Optional[float] = None,
to_timestamp: Optional[float] = None,
project_path: Optional[str] = None,
exclude_ids: Optional[set] = None,
use_prefilter: bool = True,
) -> List[Dict[str, Any]]:
"""Search Codex rollouts for keywords, one match dict per session.
``project_path`` filters by the rollout's session_meta cwd (recursive
workspace match). Rollouts are de-duplicated by session id — a copy left
in both sessions/ and archived_sessions/ is reported once.
``use_prefilter`` skips per-file and per-line json.loads() work a raw byte
scan already proves cannot match (see ``files_possibly_matching`` and
``iter_jsonl``'s ``line_keywords``). The FILE-level skip is safe
unconditionally — every counter this function reports (``excluded_untimed``,
``session_range``, ``match_range``) is scoped to a single rollout file and
only surfaces when that same file has ``total_mentions > 0``, so a file
ruled out entirely never had anything to report.
Codex does NOT do line-level skipping at all — not even outside a date
window. Two separate accountings need to see every record: ``session_range``
(which would otherwise collapse onto ``match_range``) and, under a date
window, ``excluded_untimed``. See the comment at the ``line_keywords``
assignment below; the disabling is unconditional and a test enforces it.
"""
search_keywords = [
(keyword, keyword if case_sensitive else keyword.casefold())
for keyword in keywords
]
exclude = exclude_ids or set()
matches: List[Dict[str, Any]] = []
seen_ids: set[str] = set()
matched_files: Optional[set[Path]] = None
if use_prefilter:
matched_files = files_possibly_matching(
rollouts, keywords, case_sensitive=case_sensitive
)
# Codex gets FILE-level pre-filtering only, never line-level.
#
# ``session_range.observe()`` has to see every record — including the
# ``session_meta`` first line — to report the conversation's time span.
# Skipping lines that lack the keyword bytes collapses ``Internal range``
# onto ``Match range``: measured on a 4-line rollout, ``01-01 .. 01-20``
# became ``01-10 .. 01-10``. That is not "one less field", it is a *wrong
# value* in a field callers quote, and it fires on the plain no-date-window
# search — the most common invocation there is.
#
# A file the file-level filter rules out entirely reports no range at all,
# so that layer stays safe and stays on.
line_keywords = None
for path in rollouts:
try:
meta = codex_meta_from_rollout(path)
except OSError as e:
print(f"Warning: Error reading {path}: {e}", file=sys.stderr)
continue
sid = codex_session_id(meta, path) if meta else None
if not sid:
sid = path.stem
# Dedup and exclusion happen BEFORE the pre-filter check below, on
# purpose: they must claim/reject this path exactly as before, in the
# same traversal order, regardless of whether it turns out to have a
# keyword match. Moving the pre-filter check earlier would change
# which physical copy "wins" the session id when two copies of one
# session exist (sessions/ vs archived_sessions/) — a real behavior
# change this performance fix has no business making silently.
if sid in seen_ids:
continue
seen_ids.add(sid)
if sid in exclude or path.stem in exclude:
continue
if matched_files is not None and path not in matched_files:
continue
cwd = meta.get("cwd") if isinstance(meta, dict) else None
if project_path is not None:
if not isinstance(cwd, str) or not workspace_matches(
cwd, project_path, recursive=True
):
continue
keyword_counts: Dict[str, int] = defaultdict(int)
total_mentions = 0
match_sources: set[str] = set()
session_range = TimestampRange()
match_range = TimestampRange()
excluded_untimed = 0
try:
for record in iter_jsonl(path, line_keywords=line_keywords):
record_timestamp = parse_timestamp(record.get("timestamp"))
if record_timestamp is not None:
session_range.observe(record_timestamp)
if record.get("type") == "session_meta" and isinstance(
record.get("payload"), dict
):
meta_timestamp = parse_timestamp(
record["payload"].get("timestamp")
)
if meta_timestamp is not None:
session_range.observe(meta_timestamp)
if from_timestamp is not None or to_timestamp is not None:
if record_timestamp is None:
excluded_untimed += 1
continue
if not timestamp_in_window(
record_timestamp, from_timestamp, to_timestamp
):
continue
record_counts: Dict[str, int] = defaultdict(int)
record_sources: set[str] = set()
for segment in codex_searchable_segments(record):
search_text = (
segment.text if case_sensitive else segment.text.casefold()
)
for keyword, search_keyword in search_keywords:
count = search_text.count(search_keyword)
if count > 0:
record_counts[keyword] += count
record_sources.add(segment.source)
record_mentions = sum(record_counts.values())
if not record_mentions:
continue
for keyword, count in record_counts.items():
keyword_counts[keyword] += count
total_mentions += record_mentions
match_sources.update(record_sources)
if record_timestamp is not None:
match_range.observe(record_timestamp)
except (OSError, UnicodeError) as e:
print(f"Warning: Error processing {path}: {e}", file=sys.stderr)
continue
if total_mentions > 0:
matches.append(
{
"session_id": sid,
"path": path,
"cwd": cwd,
"total_mentions": total_mentions,
"keyword_counts": dict(keyword_counts),
"match_sources": sorted(match_sources),
"created_at": session_range.earliest,
"updated_at": session_range.latest,
"match_created_at": match_range.earliest,
"match_updated_at": match_range.latest,
"excluded_untimed_records": excluded_untimed,
}
)
matches.sort(
key=lambda match: (
match["total_mentions"],
match["match_updated_at"]
if match["match_updated_at"] is not None
else float("-inf"),
),
reverse=True,
)
return matches
# ---------------------------------------------------------------------------
# Kimi CLI session search (--kimi)
#
# Kimi CLI (kimi-code) stores conversations under <KIMI_HOME>/sessions/
# wd_<workspace>_<hash>/session_<uuid>/agents/<agent>/wire.jsonl, with a
# per-session state.json (id/cwd/title/createdAt/updatedAt in ms). The wire
# record schema is NOT the Claude or Codex one (turn.prompt,
# context.append_message, context.append_loop_event, all timestamped by a
# `time` epoch-ms field), so neither searchable_segments() nor
# codex_searchable_segments() applies. Layout and record shapes were verified
# against Kimi CLI 0.38.0 (wire protocol 1.5) on a real store; see
# _core/kimi.py's module docstring for the full format contract.
#
# Searchable coverage is the conversation itself: user prompts (turn.prompt /
# turn.steer), appended messages (user/assistant text and attachments), and
# loop events (assistant content parts, tool calls, tool results). Static
# boilerplate — config.update / profile.bind system prompts, tool snapshots,
# usage/token metrics — is deliberately NOT indexed: a keyword that only
# appears in a shared system prompt would match every session, and "not found"
# is the answer this tool is trusted to give about conversation content.
# ---------------------------------------------------------------------------
def _kimi_flatten_payload(value: Any) -> List[str]:
"""Flatten nested Kimi payload content, dropping structural identifier keys.
Same principle as text.py's searchable_segments excluding id/signature
keys: UUID-class fields (turnId, toolCallId, …) are unique per record, so
indexing them only manufactures false-positive hits and dilutes the match
source attribution (independent review, 2026-08).
"""
if isinstance(value, str):
return [value]
if isinstance(value, list):
return [part for item in value for part in _kimi_flatten_payload(item)]
if isinstance(value, dict):
return [
part
for key, item in value.items()
if key not in _KIMI_STRUCTURAL_KEYS
for part in _kimi_flatten_payload(item)
]
return []
_KIMI_STRUCTURAL_KEYS = frozenset(
{
"type",
"id",
"uuid",
"stepUuid",
"turnId",
"toolCallId",
"parentUuid",
"callId",
}
)
def kimi_searchable_segments(record: Dict[str, Any]) -> List[SearchSegment]:
"""Extract searchable text fields from one Kimi wire.jsonl record."""
segments: List[SearchSegment] = []
def add(source: str, value: Any) -> None:
for text_value in _kimi_flatten_payload(value):
segments.append(SearchSegment(source=source, text=text_value))
record_type = record.get("type")
if record_type in {"turn.prompt", "turn.steer"}:
add("prompt", record.get("input"))
elif record_type == "context.append_message":
message = record.get("message")
if isinstance(message, dict):
add("message", message.get("content"))
add("tool_input", message.get("toolCalls"))
elif record_type == "context.append_loop_event":
event = record.get("event")
if isinstance(event, dict):
event_type = event.get("type")
if event_type == "content.part":
add("message", event)
elif event_type == "tool.call":
add("tool_input", event)
elif event_type == "tool.result":
add("tool_result", event)
# step.begin / step.end carry no conversation text.
elif record_type == "plugin.session_start":
add("plugin", record.get("content"))
return list(dict.fromkeys(segments))
def discover_kimi_wires(kimi_home: Path) -> List[tuple]:
"""Enumerate (session_dir, agent_name, wire_path) triples under a Kimi home."""
wires: List[tuple] = []
for session_dir in iter_kimi_session_dirs(kimi_home):
main_wire, subagent_wires = kimi_wire_files(session_dir)
for path in ([main_wire] if main_wire else []) + subagent_wires:
if path is not None:
wires.append((session_dir, path.parent.name, path))
return wires
def search_kimi_wires(
wires: List[tuple],
keywords: List[str],
case_sensitive: bool = False,
from_timestamp: Optional[float] = None,
to_timestamp: Optional[float] = None,
project_path: Optional[str] = None,
exclude_ids: Optional[set] = None,
use_prefilter: bool = True,
) -> List[Dict[str, Any]]:
"""Search Kimi wire files for keywords, one match dict per SESSION.
A session's subagent wires (agents/agent-N/) are runs of the same
conversation, so matches aggregate at session level with the agent name
prefixed into each match source (e.g. "main:message", "agent-0:tool_input").
Like the Codex search, Kimi gets FILE-level pre-filtering only, never
line-level: the session's internal range must observe every record's
`time`, and collapsing it onto the match range would report a wrong value
in a field callers quote. One Kimi-specific subtlety: the pre-filter rules
out whole wires, but a session's range spans ALL its wires — so when any
wire of a session matched, the remaining wires are re-read for their time
ranges only (no keyword work), keeping the displayed range exact.
"""
search_keywords = [
(keyword, keyword if case_sensitive else keyword.casefold())
for keyword in keywords
]
exclude = exclude_ids or set()
matched_files: Optional[set] = None
if use_prefilter:
matched_files = files_possibly_matching(
[wire_path for _, _, wire_path in wires],
keywords,
case_sensitive=case_sensitive,
)
line_keywords = None # see docstring — never line-level for Kimi either
# Group wires by session, preserving discovery order.
sessions: Dict[Path, List[tuple]] = {}
for session_dir, agent_name, wire_path in wires:
sessions.setdefault(session_dir, []).append((agent_name, wire_path))
matches: List[Dict[str, Any]] = []
for session_dir, agent_wires in sessions.items():
state = load_kimi_state(session_dir) or {}
session_id = state.get("id") if isinstance(state.get("id"), str) else None
if not session_id:
session_id = session_dir.name
if session_id in exclude or session_dir.name in exclude:
continue
if isinstance(state.get("cwd"), str) and state["cwd"].strip():
cwd: Optional[str] = state["cwd"].strip()
else:
cwd = None
if project_path is not None:
if not cwd or not workspace_matches(cwd, project_path, recursive=True):
continue
keyword_counts: Dict[str, int] = defaultdict(int)
total_mentions = 0
match_sources: set = set()
session_range = TimestampRange()
match_range = TimestampRange()
excluded_untimed = 0
skipped_wires: List[Path] = []
for agent_name, wire_path in agent_wires:
if matched_files is not None and wire_path not in matched_files:
skipped_wires.append(wire_path)
continue
try:
for record in iter_jsonl(wire_path, line_keywords=line_keywords):
record_timestamp = parse_timestamp(record.get("time"))
if record_timestamp is None and record.get("type") == "metadata":
record_timestamp = parse_timestamp(record.get("created_at"))
if record_timestamp is not None:
session_range.observe(record_timestamp)
if from_timestamp is not None or to_timestamp is not None:
if record_timestamp is None:
excluded_untimed += 1
continue
if not timestamp_in_window(
record_timestamp, from_timestamp, to_timestamp
):
continue
record_counts: Dict[str, int] = defaultdict(int)
record_sources: set = set()
for segment in kimi_searchable_segments(record):
search_text = (
segment.text
if case_sensitive
else segment.text.casefold()
)
for keyword, search_keyword in search_keywords:
count = search_text.count(search_keyword)
if count > 0:
record_counts[keyword] += count
record_sources.add(f"{agent_name}:{segment.source}")
record_mentions = sum(record_counts.values())
if not record_mentions:
continue
for keyword, count in record_counts.items():
keyword_counts[keyword] += count
total_mentions += record_mentions
match_sources.update(record_sources)
if record_timestamp is not None:
match_range.observe(record_timestamp)
except (OSError, UnicodeError) as e:
print(f"Warning: Error processing {wire_path}: {e}", file=sys.stderr)
continue
if total_mentions > 0:
# A prefiltered-out wire can still hold earlier/later records of
# this matched session; fold its time range in so the displayed
# internal range stays exact.
for wire_path in skipped_wires:
try:
extra = kimi_wire_time_range(wire_path)
except OSError:
continue
for value in (extra.earliest, extra.latest):
if value is not None:
session_range.observe(value)
matches.append(
{
"session_id": session_id,
"path": session_dir,
"title": scan_kimi_session(session_dir).title,
"cwd": cwd or "",
"total_mentions": total_mentions,
"keyword_counts": dict(keyword_counts),
"match_sources": sorted(match_sources),
"created_at": session_range.earliest,
"updated_at": session_range.latest,
"match_created_at": match_range.earliest,
"match_updated_at": match_range.latest,
"excluded_untimed_records": excluded_untimed,
}
)
matches.sort(
key=lambda match: (
match["total_mentions"],
match["match_updated_at"]
if match["match_updated_at"] is not None
else float("-inf"),
),
reverse=True,
)
return matches
class SessionAnalyzer:
"""Analyze Claude Code session history files across all config homes."""
def __init__(
self,
homes: Optional[List[Path]] = None,
sources: Optional[List[HistorySource]] = None,
warnings: Optional[List[str]] = None,
):
"""
Initialize analyzer.
Args:
homes: Exact list of Claude config home directories to search (each
must
contain a ``projects/`` subdir). Pass ``None`` (the default) to
auto-discover active homes and load registered archives. Pass an
explicit list to restrict the search — an EMPTY list means
"search nothing", it must NOT silently fall back to full
discovery (that would turn a scope-narrowing flag into the
widest possible scope).
"""
if homes is not None and sources is not None:
raise ValueError("Pass homes or sources, not both")
if sources is not None:
self.sources = list(sources)
elif homes is not None:
self.sources = [
HistorySource(
provider="claude",
kind="active",
label=home_label(home),
home=Path(home),
)
for home in homes
]
else:
self.sources, discovered_warnings = discover_claude_sources()
warnings = (warnings or []) + discovered_warnings
self.homes = [source.home for source in self.sources]
self.warnings = list(warnings or [])
def find_project_sessions(self, project_path: str) -> List[Dict[str, Any]]:
"""
Find all session files for a project ACROSS every discovered home.
Sessions are de-duplicated by session id (the ``.jsonl`` filename), so a
conversation shared across profiles is reported once. Every physical
copy remains attached to that session reference: search must union the
records from active and archive copies because one copy can retain
content that another copy no longer has. Agent side-files
(``agent-*.jsonl``) are excluded.
Args:
project_path: The project's working directory. An absolute path, a
``~`` path, or a relative path are all expanded and resolved to
an absolute path before encoding. A bare directory name
(basename) is also accepted and matched via reverse lookup.
Returns:
Session-reference dictionaries sorted by the unioned maximum
internal timestamp (newest first). Each includes a representative
``path``, every physical ``copy``, the unioned
``created_at``/``updated_at`` range, and all ``sources``/``homes``
where that session ID was observed. Empty if the project has no
history in the configured source set.
"""
return self._merge_sessions_from_dirs(
self._resolve_project_dirs(project_path)
)
def _merge_sessions_from_dirs(
self, pairs: List[tuple]
) -> List[Dict[str, Any]]:
"""Collect + de-duplicate sessions from ``(source, project_dir)`` pairs.
Per-model profile homes (``~/.claude-profiles/<name>``) commonly
symlink their ``projects/`` straight to the main home's — same real
directory, different nominal ``source.home``. ``pairs`` is built by
the caller per source label, so the same physical file can arrive
here once per profile that happens to alias it. Group by the
*resolved* real path first so each physical file is opened and
parsed exactly once no matter how many source labels point at it.
Every group member keeps its own *nominal* (unresolved) directory
so per-copy paths stay distinct strings — e.g. a session aliased
under both ``~/.claude/projects`` and ``~/.claude-profiles/css/
projects`` still reports two different copy paths, matching what
the un-grouped code produced. Collapsing them to one shared string
was tried first and broke ``search``'s "Other matching copies:"
listing (independent review, 2026-08-04): it distinguishes the
primary copy from the others by path-string inequality, so giving
every aliased copy the identical string made all-but-one silently
vanish from that field even though the ``sources``/``homes``
label lists stayed correct.
"""
groups: Dict[str, tuple] = {}
for source, project_dir in pairs:
try:
key = str(project_dir.resolve())
except (OSError, RuntimeError):
key = str(project_dir)
group = groups.get(key)
if group is None:
groups[key] = (project_dir, [(source, project_dir)])
else:
group[1].append((source, project_dir))
by_id: Dict[str, Dict[str, Any]] = {}
for scan_dir, group_members in groups.values():
group_sources = [source for source, _nominal_dir in group_members]
# any(...) — not group_sources[0].kind — because "at least one
# active alias in the group" must win regardless of which
# member happened to be discovered first; index-0 only agrees
# with that whenever active sources are enumerated before
# archives (true for both shipped call sites today, but not a
# contract this function should silently depend on).
group_has_active = any(source.kind == "active" for source in group_sources)
group_kind = "active" if group_has_active else group_sources[0].kind
for file in scan_dir.glob("*.jsonl"):
if file.name.startswith("agent-"):
continue
summary = scan_claude_session(file)
sid = summary.session_id
copies = [
{
"path": nominal_dir / file.name,
"source": source,
"created_at": summary.created_at,
"updated_at": summary.updated_at,
}
for source, nominal_dir in group_members
]
candidate = {
"path": file,
"copies": copies,
"sources": list(group_sources),
"homes": list(
dict.fromkeys(source.home for source in group_sources)
),
"created_at": summary.created_at,
"updated_at": summary.updated_at,
"timestamp_source": (
"session-record-minmax"
if summary.timestamp_count
else "unknown"
),
"selected_kind": group_kind,
"selected_updated_at": summary.updated_at,
"session_id": sid,
}
entry = by_id.get(sid)
if entry is None:
by_id[sid] = candidate
continue
entry["copies"].extend(copies)
sources = list(entry["sources"])
existing_labels = {item.display_label for item in sources}
for source in group_sources:
if source.display_label not in existing_labels:
sources.append(source)
existing_labels.add(source.display_label)
homes = list(entry["homes"])
for source in group_sources:
if source.home not in homes:
homes.append(source.home)
existing_selected_time = (
entry["selected_updated_at"]
if entry["selected_updated_at"] is not None
else float("-inf")
)
candidate_time = (
summary.updated_at
if summary.updated_at is not None
else float("-inf")
)
candidate_wins = candidate_time > existing_selected_time or (
candidate_time == existing_selected_time
and group_kind == "active"
and entry["selected_kind"] != "active"
)
if candidate_wins:
entry["path"] = file
entry["selected_kind"] = group_kind
entry["selected_updated_at"] = summary.updated_at
entry["sources"] = sources
entry["homes"] = homes
created_values = [
value
for value in (entry["created_at"], summary.created_at)
if value is not None
]
updated_values = [
value
for value in (entry["updated_at"], summary.updated_at)
if value is not None
]
entry["created_at"] = min(created_values) if created_values else None
entry["updated_at"] = max(updated_values) if updated_values else None
if summary.timestamp_count:
entry["timestamp_source"] = "session-record-minmax"
return sorted(
by_id.values(),
key=lambda entry: (
entry["updated_at"]
if entry["updated_at"] is not None
else float("-inf")
),
reverse=True,
)
def project_dir_pairs(self) -> Dict[str, List[tuple]]:
"""Enumerate every project dir across all sources.
Returns ``{encoded_project_name: [(source, dir), ...]}`` — one entry
per distinct project, with a pair per source that holds history for
it. The encoded name (``-Users-<name>-app``) is the project's true
identity; decoding ``-`` back to ``/`` is lossy (real dir names may
contain hyphens), so the encoded form is displayed as-is.
"""
result: Dict[str, List[tuple]] = {}
for source in self.sources:
projects_dir = source.home / "projects"
if not projects_dir.is_dir():
continue
for candidate in sorted(projects_dir.iterdir()):
if candidate.is_dir():
result.setdefault(candidate.name, []).append(
(source, candidate)
)
return result
def find_all_projects_sessions(self) -> List[Dict[str, Any]]:
"""Find sessions for EVERY project across all sources (--all-projects).
Each returned ref carries a ``project`` field with the encoded
project name. Sorted newest first across all projects.
"""
sessions: List[Dict[str, Any]] = []
for project_name, pairs in self.project_dir_pairs().items():
for ref in self._merge_sessions_from_dirs(pairs):
ref["project"] = project_name
sessions.append(ref)
return sorted(
sessions,
key=lambda entry: (
entry["updated_at"]
if entry["updated_at"] is not None
else float("-inf")
),
reverse=True,
)
def _resolve_project_dirs(self, project_path: str) -> List[tuple]:
"""
Resolve a project path to its encoded dir under EACH home's projects/.
Claude Code encodes the project's ABSOLUTE working-directory path by
replacing every ``/`` with ``-`` (e.g. ``/Users/<name>/app`` ->
``-Users-<name>-app``). The directory name is NOT the basename, so a
bare name or an unexpanded ``~`` path never matches directly — the #1
reason a real history is mistaken for "no sessions". The same project
has a same-named encoded dir under every home that holds history for it.
Strategy (the exact-vs-fallback decision is GLOBAL, not per-home):
1. Try the exact encoded dir in every home. If it matches in ANY home,
the project identity is known precisely — return those exact matches
only. A home lacking the exact dir contributes nothing; it is NOT
fuzzy-matched, so a different project that merely shares the basename
in another profile home can never be conflated in.
2. Only if NO home has the exact dir, treat the input as a bare basename
and reverse-look-up ``-<basename>`` across all homes. Require a SINGLE
distinct encoded name; if the basename maps to two different projects
(within OR across homes), that is ambiguous — warn and return nothing
rather than guess.
Returns:
List of ``(source, encoded_dir)`` pairs — one per source where the
resolved project dir exists.
"""
# Encode the resolved absolute path once (for exact matching).
exact_name: Optional[str] = None
try:
abs_path = Path(project_path).expanduser().resolve()
exact_name = str(abs_path).replace("/", "-")
except (OSError, RuntimeError):
exact_name = None
base = Path(project_path).name
# Pass 1 — exact encoded-dir match across ALL homes. A single exact hit
# anywhere fixes the project identity, so we never fuzzy-fall-back.
if exact_name is not None:
exact_hits = [
(source, source.home / "projects" / exact_name)
for source in self.sources
if (source.home / "projects" / exact_name).is_dir()
]
if exact_hits:
return exact_hits
# Pass 2 — no exact match anywhere: reverse-look-up the bare basename,
# but bind only ONE distinct project so same-basename projects living in
# different homes are never conflated together.
if not base:
return []
by_name: Dict[str, List[tuple]] = {}
for source in self.sources:
projects_dir = source.home / "projects"
if not projects_dir.is_dir():
continue
for d in projects_dir.iterdir():
if d.is_dir() and d.name.endswith("-" + base):
by_name.setdefault(d.name, []).append((source, d))
if not by_name:
return []
if len(by_name) > 1:
print(
f"Ambiguous project name '{base}' — {len(by_name)} distinct "
"projects match across homes; re-run with the full absolute path:",
file=sys.stderr,
)
for name in sorted(by_name):
sources_str = ", ".join(
source.display_label for source, _ in by_name[name]
)
print(f" {name} [{sources_str}]", file=sys.stderr)
return []
# Exactly one distinct project — use it wherever it exists.
return next(iter(by_name.values()))
def search_sessions(
self,
session_refs: List[Dict[str, Any]],
keywords: List[str],
case_sensitive: bool = False,
from_timestamp: Optional[float] = None,
to_timestamp: Optional[float] = None,
use_prefilter: bool = True,
) -> List[Dict[str, Any]]:
"""
Search sessions for keywords.
Args:
session_refs: Session refs from ``find_project_sessions`` (each has
every active/archive copy plus a representative path).
keywords: Keywords to search for.
case_sensitive: Whether to perform case-sensitive search.
from_timestamp: Inclusive lower internal record timestamp.
to_timestamp: Inclusive upper internal record timestamp.
use_prefilter: Skip the expensive per-line json.loads() + text
extraction pass on copies a raw byte scan already proves
cannot contain any keyword (see ``files_possibly_matching``).
Forced off automatically when a date window is active — see
below for why.
Returns:
List of match dicts (session ref + match counts), most mentions
first.
"""
# The pre-filter is only safe to apply when no date window is active.
# `excluded_untimed_count` below counts records that lack a timestamp
# ACROSS EVERY COPY of a session, independent of whether those records
# contain a keyword — it exists purely to report "N records could not
# be checked against your --from-date/--to-date window". Skipping a
# copy's full parse because it has no keyword match would silently
# drop its contribution to that count. Outside a date window this
# counter is never computed, so skipping is unconditionally safe —
# the file-level pre-filter is provably an over-approximation (see
# files_possibly_matching's docstring), so a copy it rules out cannot
# contain a matchable record.
use_prefilter = use_prefilter and from_timestamp is None and to_timestamp is None
matched_files: Optional[set[Path]] = None
# Always case-folded regardless of `case_sensitive`: casefold-matching
# is a strict superset of exact-case matching (anything an exact-case
# check would find, casefold-matching finds too), so it is always a
# safe over-approximation to hand to the line-level pre-check in
# iter_jsonl — it costs a little filtering precision in
# case-sensitive mode, never a missed match.
# Gate on the ORIGINAL keywords, not the folded ones: "ß".casefold()
# is "ss" (ASCII), so checking after folding would answer "safe" for
# precisely the input the check exists to reject.
line_keywords = (
[kw.casefold() for kw in keywords]
if use_prefilter and keywords_are_raw_byte_safe(keywords)
else None
)
if use_prefilter:
all_copy_paths = [
copy["path"]
for ref in session_refs
for copy in (
ref.get("copies")
or [{"path": ref["path"], "source": ref["sources"][0]}]
)
]
matched_files = files_possibly_matching(
all_copy_paths, keywords, case_sensitive=case_sensitive
)
matches: List[Dict[str, Any]] = []
search_keywords = [
(keyword, keyword if case_sensitive else keyword.casefold())
for keyword in keywords
]
for ref in session_refs:
keyword_counts = defaultdict(int)
total_mentions = 0
match_sources: set[str] = set()
matching_source_labels: set[str] = set()
match_timestamps = TimestampRange()
excluded_untimed_count = 0
excluded_untimed_from_prior_copies: set[str] = set()
matched_records_from_prior_copies: set[str] = set()
matched_file_history_paths: set[str] = set()
matching_copies: List[Dict[str, Any]] = []
copies = ref.get("copies") or [
{
"path": ref["path"],
"source": ref["sources"][0],
"created_at": ref["created_at"],
"updated_at": ref["updated_at"],
}
]
# Archive copies of one session are routinely byte-identical: a
# registered long-term backup snapshots the same .jsonl every run,
# so one session commonly carries a dozen copies with a single
# distinct content (measured on a real project: 46 of 60 sampled
# multi-copy sessions held 13 copies and exactly 1 content; that
# redundancy was 85% of the corpus's bytes). Parsing all of them
# costs Nx for one session's worth of records — _record_identity
# dedupes them straight back down afterwards.
#
# A byte-identical copy therefore has a fully predictable outcome:
# every record identity is already registered by the twin that ran
# first, so it can add no new mentions, no new untimed records and
# no new timestamps. Only its own source label is new. Reuse that
# conclusion instead of re-parsing.
#
# Hash only sizes that actually repeat — a unique size cannot have
# an identical twin, so single-copy sessions pay nothing.
sizes_worth_hashing = repeated_copy_sizes(copies)
digest_had_match: Dict[str, bool] = {}
for copy in copies:
session_file = copy["path"]
source = copy["source"]
if matched_files is not None and session_file not in matched_files:
# A raw byte scan already proved this exact copy cannot
# contain any keyword — see the use_prefilter comment
# above for why skipping it here (no date window active)
# cannot under-count anything.
continue
copy_digest: Optional[str] = None
try:
copy_size = session_file.stat().st_size
except OSError:
copy_size = None
if copy_size in sizes_worth_hashing:
copy_digest = file_content_digest(session_file)
if copy_digest is not None and copy_digest in digest_had_match:
if digest_had_match[copy_digest]:
# Same bytes as a copy already processed for this
# session: it matched, so this one does too, and every
# one of its records was already counted.
matching_source_labels.add(source.display_label)
matching_copies.append(
{
"path": session_file,
"source": source,
"new_mentions": 0,
}
)
continue
copy_had_match = False
copy_new_mentions = 0
copy_untimed_records: set[str] = set()
copy_matched_records: set[str] = set()
try:
records = iter_jsonl(session_file, line_keywords=line_keywords)
for data in records:
record_identity = _record_identity(data)
record_timestamp = parse_timestamp(data.get("timestamp"))
if (
record_timestamp is None
and data.get("type") == "file-history-snapshot"
):
snapshot = data.get("snapshot")
if isinstance(snapshot, dict):
record_timestamp = parse_timestamp(
snapshot.get("timestamp")
)
if from_timestamp is not None or to_timestamp is not None:
if record_timestamp is None:
if record_identity not in excluded_untimed_from_prior_copies:
excluded_untimed_count += 1
copy_untimed_records.add(record_identity)
continue
if not timestamp_in_window(
record_timestamp, from_timestamp, to_timestamp
):
continue
if data.get("type") == "file-history-snapshot":
snapshot = data.get("snapshot")
tracked = (
snapshot.get("trackedFileBackups")
if isinstance(snapshot, dict)
else None
)
if isinstance(tracked, dict):
for original_path in tracked:
if not isinstance(original_path, str):
continue
path_text = (
original_path
if case_sensitive
else original_path.casefold()
)
path_counts = {
keyword: 1
for keyword, search_keyword in search_keywords
if search_keyword in path_text
}
if not path_counts:
continue
copy_had_match = True
matching_source_labels.add(source.display_label)
match_sources.add("file_history_path")
if record_timestamp is not None:
match_timestamps.observe(record_timestamp)
if original_path in matched_file_history_paths:
continue
matched_file_history_paths.add(original_path)
path_mentions = sum(path_counts.values())
for keyword, count in path_counts.items():
keyword_counts[keyword] += count
total_mentions += path_mentions
copy_new_mentions += path_mentions
record_counts = defaultdict(int)
record_sources: set[str] = set()
for segment in searchable_segments(data):
search_text = (
segment.text
if case_sensitive
else segment.text.casefold()
)
for keyword, search_keyword in search_keywords:
count = search_text.count(search_keyword)
if count > 0:
record_counts[keyword] += count
record_sources.add(segment.source)
record_mentions = sum(record_counts.values())
if not record_mentions:
continue
copy_had_match = True
matching_source_labels.add(source.display_label)
copy_matched_records.add(record_identity)
if record_identity in matched_records_from_prior_copies:
continue
for keyword, count in record_counts.items():
keyword_counts[keyword] += count
total_mentions += record_mentions
copy_new_mentions += record_mentions
match_sources.update(record_sources)
if record_timestamp is not None:
match_timestamps.observe(record_timestamp)
except (OSError, UnicodeError) as e:
print(
f"Warning: Error processing {session_file}: {e}",
file=sys.stderr,
)
continue
excluded_untimed_from_prior_copies.update(copy_untimed_records)
matched_records_from_prior_copies.update(copy_matched_records)
if copy_digest is not None:
digest_had_match[copy_digest] = copy_had_match
if copy_had_match:
matching_copies.append(
{
"path": session_file,
"source": source,
"new_mentions": copy_new_mentions,
}
)
if total_mentions > 0:
primary_copy = max(
matching_copies,
key=lambda item: (
item["new_mentions"],
item["source"].kind == "active",
),
)
matches.append(
{
"path": primary_copy["path"],
"matching_copies": matching_copies,
"homes": ref["homes"],
"sources": ref["sources"],
"matching_source_labels": matching_source_labels,
"total_mentions": total_mentions,
"keyword_counts": dict(keyword_counts),
"match_sources": sorted(match_sources),
"created_at": ref["created_at"],
"updated_at": ref["updated_at"],
"match_created_at": match_timestamps.earliest,
"match_updated_at": match_timestamps.latest,
"timestamp_source": ref["timestamp_source"],
"excluded_untimed_records": excluded_untimed_count,
"size": sum(item["path"].stat().st_size for item in copies),
}
)
matches.sort(
key=lambda match: (
match["total_mentions"],
match["match_updated_at"]
if match["match_updated_at"] is not None
else float("-inf"),
),
reverse=True,
)
return matches
def get_session_stats(self, session_file: Path) -> Dict[str, Any]:
"""
Get detailed statistics for a session file.
Args:
session_file: Path to session JSONL file
Returns:
Dictionary of session statistics
"""
stats = {
"total_lines": 0,
"user_messages": 0,
"assistant_messages": 0,
"tool_uses": defaultdict(int),
"write_calls": 0,
"edit_calls": 0,
"read_calls": 0,
"bash_calls": 0,
"file_operations": [],
}
try:
with open(session_file, "r") as f:
for line in f:
stats["total_lines"] += 1
try:
data = json.loads(line.strip())
# Count message types
role = data.get("role") or data.get("message", {}).get("role")
if role == "user":
stats["user_messages"] += 1
elif role == "assistant":
stats["assistant_messages"] += 1
# Analyze tool uses
content = data.get("content") or data.get("message", {}).get(
"content", []
)
for item in content:
if not isinstance(item, dict):
continue
if item.get("type") == "tool_use":
tool_name = item.get("name", "unknown")
stats["tool_uses"][tool_name] += 1
# Track file operations
if tool_name == "Write":
stats["write_calls"] += 1
file_path = item.get("input", {}).get(
"file_path", ""
)
if file_path:
stats["file_operations"].append(
("write", file_path)
)
elif tool_name == "Edit":
stats["edit_calls"] += 1
file_path = item.get("input", {}).get(
"file_path", ""
)
if file_path:
stats["file_operations"].append(
("edit", file_path)
)
elif tool_name == "Read":
stats["read_calls"] += 1
elif tool_name == "Bash":
stats["bash_calls"] += 1
except json.JSONDecodeError:
continue
except Exception as e:
print(f"Error analyzing {session_file}: {e}", file=sys.stderr)
# Convert defaultdict to regular dict
stats["tool_uses"] = dict(stats["tool_uses"])
return stats
def _extract_text_content(self, data: Dict[str, Any]) -> str:
"""Compatibility wrapper around the structured event extractor."""
return " ".join(segment.text for segment in searchable_segments(data))
def _add_home_flags(subparser) -> None:
"""Attach the shared home-scoping flags to a subparser (list / search)."""
subparser.add_argument(
"--home",
action="append",
metavar="DIR",
help="Restrict to exact Claude home dir(s) and bypass the archive "
"registry (repeatable). Default: search every active home plus "
"registered archives.",
)
subparser.add_argument(
"--main-only",
action="store_true",
help="Search only ~/.claude, bypassing profile homes and archives.",
)
subparser.add_argument(
"--history-sources",
metavar="FILE",
help=(
"History source registry (default: ~/.claude/history-sources.json "
"when present). Incompatible with --home/--main-only."
),
)
subparser.add_argument(
"--from-date",
help="Inclusive start: YYYY-MM-DD (local day) or timezone-qualified ISO datetime",
)
subparser.add_argument(
"--to-date",
help="Inclusive end: YYYY-MM-DD (local day) or timezone-qualified ISO datetime",
)
def _sources_for(args) -> tuple:
"""Resolve active/archive sources from CLI flags (used by list / search).
Returns ``(sources, narrowed, warnings)``. ``narrowed`` is True when the user passed
``--home`` / ``--main-only``, so the caller can treat an empty result as a
real "your selection matched no home with history" error instead of
silently widening back to searching every home.
"""
main_only = getattr(args, "main_only", False)
explicit = getattr(args, "home", None)
registry = getattr(args, "history_sources", None)
if main_only and explicit:
raise HistorySourceConfigError("--main-only cannot be combined with --home")
if registry and (main_only or explicit):
raise HistorySourceConfigError(
"--history-sources cannot be combined with --home/--main-only"
)
if main_only:
sources, warnings = discover_claude_sources(
explicit_homes=[Path.home() / ".claude"]
)
return sources, True, warnings
if explicit:
sources, warnings = discover_claude_sources(explicit_homes=explicit)
return sources, True, warnings
sources, warnings = discover_claude_sources(manifest_path=registry)
return sources, False, warnings
def _analyzer_or_exit(args) -> "SessionAnalyzer":
"""Build a SessionAnalyzer, erroring out if a narrowing flag matched no home.
Without this, an explicit ``--home``/``--main-only`` that resolves to no
home-with-history would (via an empty list) either search nothing or, worse,
reintroduce full discovery — turning a scope-narrowing flag into the widest
possible scope. We fail loudly instead.
"""
try:
sources, narrowed, warnings = _sources_for(args)
except HistorySourceConfigError as error:
print(f"History source configuration error: {error}", file=sys.stderr)
sys.exit(2)
if narrowed and not sources:
print(
"No Claude home with a projects/ dir matched your --home/--main-only "
"selection (a --home value must be a config home such as ~/.claude "
"or ~/.claude-profiles/<name>, not its projects/ subdir).",
file=sys.stderr,
)
sys.exit(1)
return SessionAnalyzer(sources=sources, warnings=warnings)
def _parse_date_window(args, parser) -> tuple[Optional[float], Optional[float]]:
try:
from_timestamp = (
parse_date_boundary(args.from_date) if args.from_date else None
)
to_timestamp = (
parse_date_boundary(args.to_date, end=True) if args.to_date else None
)
except ValueError as error:
parser.error(str(error))
if (
from_timestamp is not None
and to_timestamp is not None
and from_timestamp > to_timestamp
):
parser.error("--from-date must not be later than --to-date")
return from_timestamp, to_timestamp
def _format_range(earliest: Optional[float], latest: Optional[float]) -> str:
if earliest is None or latest is None:
return "unknown (no internal timestamp)"
return f"{format_timestamp(earliest)} .. {format_timestamp(latest)}"
def _source_summary(sources: List[HistorySource]) -> str:
return ", ".join(source.display_label for source in sources)
def _validate_project_scope(args, parser) -> None:
"""Exactly one of project_path / --all-projects must be given."""
if args.all_projects and args.project_path:
parser.error("pass either a project path or --all-projects, not both")
if not args.all_projects and not args.project_path:
parser.error(
"a project path is required unless --all-projects is given "
"(use --all-projects when you do not know which project it was)"
)
def _normalize_search_scope(args, parser) -> None:
"""Resolve the search positional grammar without argparse ambiguity.
``project_path?`` followed by ``keywords+`` is ambiguous when
``--all-projects`` is active: argparse consumes the first of two keywords as
the optional project. Parse one term list instead, then apply the scope the
caller selected.
"""
terms = list(args.search_terms)
if args.all_projects:
args.project_path = None
args.keywords = terms
return
if len(terms) < 2:
parser.error(
"search requires a project path followed by at least one keyword; "
"use --all-projects when the project is unknown"
)
args.project_path = terms[0]
args.keywords = terms[1:]
def _maybe_hint_all_projects(project_path: str) -> None:
"""Warn when an unresolved ``project_path`` looks like it was meant as a
keyword, not a path — the exact trap of running ``search`` with keywords
only and forgetting ``--all-projects``: argparse's positional grammar
(``project_path?`` then ``keywords+``) silently consumes the first
keyword as the project, the lookup finds nothing, and the message alone
("No sessions found for project: embedding") reads as "your keyword
doesn't exist" rather than "you searched for a project by that name".
A real project path always contains a path separator or resolves to an
existing directory; a bare word that does neither almost certainly was
not one.
"""
if "/" in project_path or "\\" in project_path or Path(project_path).exists():
return
print(
"Hint: this looks like it might be a keyword, not a project path — "
"if you don't know which project the conversation happened in, "
"re-run with --all-projects (e.g. `search --all-projects "
f"{project_path} ...`).",
file=sys.stderr,
)
def _collect_sessions(analyzer: "SessionAnalyzer", args) -> List[Dict[str, Any]]:
"""Collect session refs for the requested scope, applying exclusions."""
if args.all_projects:
sessions = analyzer.find_all_projects_sessions()
else:
sessions = analyzer.find_project_sessions(args.project_path)
exclude = set(getattr(args, "exclude_session", None) or [])
if exclude:
sessions = [
ref
for ref in sessions
if ref["session_id"] not in exclude and ref["path"].stem not in exclude
]
return sessions
def _codex_home_for(args) -> Path:
explicit = getattr(args, "codex_home", None)
if explicit:
return Path(explicit).expanduser()
env_home = os.environ.get("CODEX_HOME")
if env_home:
return Path(env_home).expanduser()
return Path.home() / ".codex"
def _kimi_home_for(args) -> Path:
return resolve_kimi_home(getattr(args, "kimi_home", None))
def _print_search_widening_hint(args) -> None:
"""On zero matches, point at the widening the user has NOT applied yet.
"Not found" is the expensive failure mode of this tool, and each of the
three widenings covers a distinct reason a real conversation can be
missed: wrong project guess, wrong tool (Codex), or a remembered quote
whose wording differs from the real one.
"""
tips: List[str] = []
if not getattr(args, "all_projects", False):
tips.append(
"--all-projects (the conversation may belong to a different "
"project than the one searched)"
)
if not getattr(args, "codex", False):
tips.append(
"--codex (it may have been a Codex conversation — Codex rollouts "
"are a separate store this search skips by default)"
)
if not getattr(args, "kimi", False):
tips.append(
"--kimi (it may have been a Kimi CLI conversation — Kimi sessions "
"are a separate store this search skips by default)"
)
tips.append(
"shorter distinctive substrings (a remembered quote often differs "
"from the real wording in punctuation or a few words)"
)
print("Tip: no matches. Before concluding it is absent, retry with:", file=sys.stderr)
for tip in tips:
print(f" - {tip}", file=sys.stderr)
def main():
"""Main entry point."""
import argparse
parser = argparse.ArgumentParser(
description="Analyze Claude Code session history files"
)
subparsers = parser.add_subparsers(dest="command", help="Command to run")
# List sessions command
list_parser = subparsers.add_parser("list", help="List all sessions for a project")
list_parser.add_argument(
"project_path",
nargs="?",
help="Project path (omit when using --all-projects)",
)
list_parser.add_argument(
"--all-projects",
action="store_true",
help="Sweep every project across all sources instead of one project.",
)
list_parser.add_argument(
"--exclude-session",
action="append",
metavar="ID",
default=[],
help="Exclude a session id (repeatable) — e.g. the current session, "
"which always matches the phrase you just typed.",
)
list_parser.add_argument(
"--limit", type=int, default=10, help="Max sessions to show (default: 10)"
)
_add_home_flags(list_parser)
# Triage command — classify how sessions in scope ended (crash recovery,
# backlog audit). Distinct from `list`: prints the full last-assistant
# text so a human/agent can judge whether a reply is still expected,
# rather than a truncated title.
triage_parser = subparsers.add_parser(
"triage",
help="Classify how sessions in a time window/project ended "
"(interrupted / net-error / stuck-on-tool / done) with full tail text",
)
triage_parser.add_argument(
"project_path",
nargs="?",
help="Project path (omit when using --all-projects)",
)
triage_parser.add_argument(
"--all-projects",
action="store_true",
help="Sweep every project across all sources instead of one project.",
)
triage_parser.add_argument(
"--exclude-session",
action="append",
metavar="ID",
default=[],
help="Exclude a session id (repeatable) — e.g. the current session.",
)
triage_parser.add_argument(
"--include-automated",
action="store_true",
help="Include sessions whose opening prompt matches the generic "
"smoke-test pattern ('reply/respond exactly ...'). Excluded by "
"default. This does NOT catch project-specific automation (e.g. a "
"git hook that always opens with the same review prompt) — use "
"--exclude-title-prefix for that; it is deliberately not hardcoded "
"here since the wording is per-project, not a Claude Code convention.",
)
triage_parser.add_argument(
"--exclude-title-prefix",
action="append",
metavar="TEXT",
default=[],
help="Exclude sessions whose opening prompt starts with TEXT "
"(repeatable, case-sensitive). Use this for a project's own "
"automation convention, e.g. a code-review hook that always opens "
"with the same fixed prompt — these otherwise dominate a triage "
"pass because they end in a routine structured tool call, not an "
"interruption.",
)
triage_parser.add_argument(
"--kind",
action="append",
choices=[
TAIL_INTERRUPTED_EXPLICIT,
TAIL_NET_ERROR,
TAIL_STUCK_NO_RESULT,
TAIL_DONE,
TAIL_EMPTY,
],
metavar="KIND",
help="Restrict to one or more tail kinds (repeatable). Default: all "
"kinds. Note 'done' includes sessions that gave a real reply and may "
"still be awaiting a response — read last_assistant_text to judge "
"that; 'done' is not a claim that nothing is outstanding.",
)
triage_parser.add_argument(
"--tail-chars",
type=int,
default=4000,
help="Max characters of the last assistant message to print "
"(default: 4000; 0 = full text, no truncation).",
)
triage_parser.add_argument(
"--limit",
type=int,
default=200,
help="Max sessions to print, 0 = no limit (default: 200). This is a "
"print-time cap, not a scan-time one: every session in scope is "
"still classified before --limit or --kind trims the output, so "
"--kind does not reduce cost the way narrowing --from-date/--to-date "
"does. The 200 default exists specifically to stop an accidentally "
"unscoped `--all-projects` with no date bound from dumping tens of "
"thousands of lines; pass --limit 0 to explicitly opt into an "
"unbounded dump once you know the scope is narrow.",
)
_add_home_flags(triage_parser)
# Search command
search_parser = subparsers.add_parser("search", help="Search sessions for keywords")
search_parser.add_argument(
"search_terms",
nargs="+",
metavar="PROJECT_OR_KEYWORD",
help=(
"Project path followed by keywords, or only keywords with "
"--all-projects"
),
)
search_parser.add_argument(
"--all-projects",
action="store_true",
help="Sweep every project across all sources — the default move when "
"you do not know which project the conversation happened in.",
)
search_parser.add_argument(
"--exclude-session",
action="append",
metavar="ID",
default=[],
help="Exclude a session id (repeatable) — e.g. the current session, "
"which always matches the phrase you just typed.",
)
search_parser.add_argument(
"--codex",
action="store_true",
help="Also search Codex rollout history (sessions/** + "
"archived_sessions/ under the Codex home). Codex uses a different "
"store and schema that the Claude registry never covers.",
)
search_parser.add_argument(
"--codex-home",
metavar="DIR",
help="Codex home for --codex (default: $CODEX_HOME or ~/.codex).",
)
search_parser.add_argument(
"--kimi",
action="store_true",
help="Also search Kimi CLI session history (sessions/**/wire.jsonl "
"under the Kimi home). Kimi CLI uses a different store and schema "
"that the Claude registry never covers.",
)
search_parser.add_argument(
"--kimi-home",
metavar="DIR",
help="Kimi CLI home for --kimi (default: $KIMI_HOME or ~/.kimi-code).",
)
search_parser.add_argument(
"--case-sensitive", action="store_true", help="Case-sensitive search"
)
search_parser.add_argument(
"--no-prefilter",
action="store_true",
help="Disable the rg/grep raw-byte pre-filter and fully parse every "
"session file. The pre-filter is designed to be a pure speedup: it "
"only runs for keywords whose bytes must appear verbatim in the file "
"(ASCII, no '/', no control characters), and disables itself for the "
"rest — so a non-ASCII query (any CJK one) already parses everything "
"and this flag changes nothing for it. Use this to verify the "
"pre-filter's neutrality on an ASCII query, or to rule it out when "
"diagnosing a suspected missed match.",
)
_add_home_flags(search_parser)
# Stats command
stats_parser = subparsers.add_parser("stats", help="Get session statistics")
stats_parser.add_argument("session_file", type=Path, help="Session file path")
stats_parser.add_argument(
"--show-files", action="store_true", help="Show file operations"
)
args = parser.parse_args()
if not args.command:
parser.print_help()
sys.exit(1)
if args.command == "search":
_normalize_search_scope(args, parser)
if args.command == "list":
_validate_project_scope(args, parser)
from_timestamp, to_timestamp = _parse_date_window(args, parser)
analyzer = _analyzer_or_exit(args)
source_summary = _source_summary(analyzer.sources)
sessions = _collect_sessions(analyzer, args)
for warning in analyzer.warnings:
print(f"Warning: {warning}", file=sys.stderr)
unknown = 0
if from_timestamp is not None or to_timestamp is not None:
filtered = []
for ref in sessions:
if ref["created_at"] is None or ref["updated_at"] is None:
unknown += 1
continue
if range_overlaps_window(
ref["created_at"],
ref["updated_at"],
from_timestamp,
to_timestamp,
):
filtered.append(ref)
sessions = filtered
if not sessions:
if args.all_projects:
print("No sessions found across all projects")
else:
print(f"No sessions found for project: {args.project_path}")
_maybe_hint_all_projects(args.project_path)
print(
f"(searched {len(analyzer.sources)} source(s): {source_summary})",
file=sys.stderr,
)
sys.exit(1)
project_names = sorted(
{ref.get("project") for ref in sessions if ref.get("project")}
)
if args.all_projects:
print(
f"Found {len(sessions)} session(s) across "
f"{len(project_names)} project(s)"
)
else:
print(f"Found {len(sessions)} session(s) for {args.project_path}")
print(
f"Searched {len(analyzer.sources)} source(s): {source_summary}\n"
)
if unknown:
print(
f"Warning: excluded {unknown} session(s) without an internal "
"timestamp; file mtime was not used as a fallback.",
file=sys.stderr,
)
print(f"Showing {min(args.limit, len(sessions))} most recent:\n")
shown = sessions[: args.limit]
last_project: Optional[str] = None
for i, ref in enumerate(shown, 1):
if args.all_projects:
project = ref.get("project") or "unknown-project"
if project != last_project:
count = sum(
1 for item in sessions if item.get("project") == project
)
print(f"== {project} ({count} session(s))")
last_project = project
session = ref["path"]
size_kb = session.stat().st_size / 1024
labels = _source_summary(ref["sources"])
print(f"{i}. {session.name}")
print(
f" Internal range: "
f"{_format_range(ref['created_at'], ref['updated_at'])}"
)
print(f" Size: {size_kb:.1f} KB")
print(f" Source: {labels}")
if len(ref["copies"]) > 1:
print(f" Physical copies searched by keyword queries: {len(ref['copies'])}")
print(f" Path: {session}")
print()
elif args.command == "triage":
_validate_project_scope(args, parser)
from_timestamp, to_timestamp = _parse_date_window(args, parser)
analyzer = _analyzer_or_exit(args)
source_summary = _source_summary(analyzer.sources)
sessions = _collect_sessions(analyzer, args)
for warning in analyzer.warnings:
print(f"Warning: {warning}", file=sys.stderr)
unknown = 0
if from_timestamp is not None or to_timestamp is not None:
# Deliberately `timestamp_in_window` on updated_at, NOT
# `range_overlaps_window` on the full [created_at, updated_at]
# span: triage asks "did this session's momentum stop around
# this boundary," so a session that merely happened to be
# running through the window but kept going for hours/days
# afterward is not a candidate, even though its range overlaps
# the window. (list's browsing use case wants the overlap
# semantics; triage does not — verified against a real reboot
# window, where overlap swept in every still-running session.)
filtered = []
for ref in sessions:
if ref["updated_at"] is None:
unknown += 1
continue
if timestamp_in_window(ref["updated_at"], from_timestamp, to_timestamp):
filtered.append(ref)
sessions = filtered
# One lightweight scan per session gets both the title (for automated
# exclusion) and cwd (for display) — cheaper than two passes, and
# find_*_sessions()'s ref dict does not carry cwd itself.
exclude_prefixes = tuple(args.exclude_title_prefix)
excluded_generic = 0
excluded_prefix = 0
scanned = []
for ref in sessions:
summary = scan_claude_session(ref["path"])
if not args.include_automated and is_automated_title(summary.title):
excluded_generic += 1
continue
if exclude_prefixes and summary.title.startswith(exclude_prefixes):
excluded_prefix += 1
continue
scanned.append((ref, summary))
excluded_automated = excluded_generic + excluded_prefix
allowed_kinds = set(args.kind) if args.kind else None
results = []
for ref, summary in scanned:
tail = classify_session_tail(ref["path"])
if allowed_kinds is not None and tail.kind not in allowed_kinds:
continue
results.append((ref, summary, tail))
results.sort(
key=lambda triple: triple[0]["updated_at"]
if triple[0]["updated_at"] is not None
else float("-inf"),
reverse=True,
)
# Deliberately one empty-result check, run AFTER --kind filtering,
# not two separate ones (scanned-empty vs. results-empty) that used
# to give different exit codes/messages for the same "nothing to
# show" outcome depending on which filter zeroed it (independent
# review, 2026-08).
if not results:
print("No sessions matched this triage scope.")
print(
f"(searched {len(analyzer.sources)} source(s): {source_summary}; "
f"scanned {len(scanned)}, excluded {excluded_generic} generic-automated "
f"+ {excluded_prefix} --exclude-title-prefix)",
file=sys.stderr,
)
sys.exit(1)
total_matched = len(results)
if args.limit:
results = results[: args.limit]
print(
f"Triaged {len(results)} session(s)"
+ (f" of {total_matched} matched" if len(results) < total_matched else "")
+ f" (excluded {excluded_generic} generic-automated + "
f"{excluded_prefix} --exclude-title-prefix); "
f"searched {len(analyzer.sources)} source(s): {source_summary}"
)
if unknown:
print(
f"Warning: excluded {unknown} session(s) without an internal "
"timestamp; file mtime was not used as a fallback.",
file=sys.stderr,
)
print()
for ref, summary, tail in results:
updated_at = ref["updated_at"]
when = format_timestamp(updated_at) if updated_at is not None else "(unknown)"
print("=" * 88)
print(ref["session_id"])
print(f" last update: {when}")
print(f" cwd: {summary.cwd or '(unknown)'}")
print(f" kind: {tail.kind}")
last_user_preview = " ".join(tail.last_user_text.split())[:200]
print(f" last user: {last_user_preview}")
print(f" last assistant ({tail.last_assistant_kind}):")
text = tail.last_assistant_text
if args.tail_chars and len(text) > args.tail_chars:
text = text[: args.tail_chars] + f"... [+{len(text) - args.tail_chars} chars]"
for line in text.splitlines() or [""]:
print(f" {line}")
print("=" * 88)
elif args.command == "search":
_validate_project_scope(args, parser)
from_timestamp, to_timestamp = _parse_date_window(args, parser)
analyzer = _analyzer_or_exit(args)
source_summary = _source_summary(analyzer.sources)
sessions = _collect_sessions(analyzer, args)
for warning in analyzer.warnings:
print(f"Warning: {warning}", file=sys.stderr)
if not sessions and not args.codex and not args.kimi:
if args.all_projects:
print("No sessions found across all projects")
else:
print(f"No sessions found for project: {args.project_path}")
_maybe_hint_all_projects(args.project_path)
print(
f"(searched {len(analyzer.sources)} source(s): {source_summary})",
file=sys.stderr,
)
sys.exit(1)
scope_desc = (
f" in {len({ref.get('project') for ref in sessions})} project(s)"
if args.all_projects
else ""
)
print(
f"Searching {len(sessions)} session(s) across {len(analyzer.sources)} "
f"source(s) [{source_summary}]{scope_desc} for: {', '.join(args.keywords)}\n"
)
matches = (
analyzer.search_sessions(
sessions,
args.keywords,
args.case_sensitive,
from_timestamp,
to_timestamp,
not args.no_prefilter,
)
if sessions
else []
)
codex_matches: List[Dict[str, Any]] = []
codex_home: Optional[Path] = None
if args.codex:
codex_home = _codex_home_for(args)
rollouts = discover_codex_rollouts(codex_home)
print(
f"Also searching {len(rollouts)} Codex rollout(s) under "
f"{codex_home} (--codex)\n"
)
codex_matches = search_codex_rollouts(
rollouts,
args.keywords,
args.case_sensitive,
from_timestamp,
to_timestamp,
None if args.all_projects else args.project_path,
set(args.exclude_session),
not args.no_prefilter,
)
kimi_matches: List[Dict[str, Any]] = []
kimi_home: Optional[Path] = None
if args.kimi:
kimi_home = _kimi_home_for(args)
kimi_wires = discover_kimi_wires(kimi_home)
print(
f"Also searching {len(kimi_wires)} Kimi CLI wire(s) under "
f"{kimi_home} (--kimi)\n"
)
kimi_matches = search_kimi_wires(
kimi_wires,
args.keywords,
args.case_sensitive,
from_timestamp,
to_timestamp,
None if args.all_projects else args.project_path,
set(args.exclude_session),
not args.no_prefilter,
)
if matches:
print(f"Found {len(matches)} session(s) with matches:\n")
project_by_path = {
ref["path"]: ref.get("project") for ref in sessions
}
for info in matches:
session = info["path"]
labels = _source_summary(info["sources"])
matching_labels = ", ".join(sorted(info["matching_source_labels"]))
print(f"📄 {session.name}")
project = project_by_path.get(session)
if project:
print(f" Project: {project}")
print(
" Internal range: "
f"{_format_range(info['created_at'], info['updated_at'])}"
)
print(
" Match range: "
f"{_format_range(info['match_created_at'], info['match_updated_at'])}"
)
print(f" Session sources: {labels}")
print(f" Match sources: {matching_labels}")
print(f" Total mentions: {info['total_mentions']}")
print(
f" Keywords: {', '.join(f'{k}({v})' for k, v in info['keyword_counts'].items())}"
)
print(f" Match fields: {', '.join(info['match_sources'])}")
if info["excluded_untimed_records"]:
print(
" Date-filter note: excluded "
f"{info['excluded_untimed_records']} record(s) without an "
"internal timestamp; file mtime was not used."
)
print(f" Path: {session}")
if len(info["matching_copies"]) > 1:
print(" Other matching copies:")
for copy in info["matching_copies"]:
if copy["path"] == session:
continue
print(
f" - [{copy['source'].display_label}] {copy['path']}"
)
print()
if args.codex:
if codex_matches:
print(
f"Codex rollout matches (home: {codex_home}):\n"
)
for info in codex_matches:
print(f"📦 {info['path'].name}")
print(f" Session: {info['session_id']}")
if info.get("cwd"):
print(f" cwd: {info['cwd']}")
print(
" Internal range: "
f"{_format_range(info['created_at'], info['updated_at'])}"
)
print(
" Match range: "
f"{_format_range(info['match_created_at'], info['match_updated_at'])}"
)
print(f" Total mentions: {info['total_mentions']}")
print(
f" Keywords: {', '.join(f'{k}({v})' for k, v in info['keyword_counts'].items())}"
)
print(f" Match fields: {', '.join(info['match_sources'])}")
if info["excluded_untimed_records"]:
print(
" Date-filter note: excluded "
f"{info['excluded_untimed_records']} record(s) without an "
"internal timestamp; file mtime was not used."
)
print(f" Path: {info['path']}")
print()
else:
print(f"No Codex rollout matches (home: {codex_home}).")
if args.kimi:
if kimi_matches:
print(f"Kimi CLI session matches (home: {kimi_home}):\n")
for info in kimi_matches:
print(f"🌙 {info['path'].name}")
print(f" Session: {info['session_id']}")
if info.get("title"):
print(f" Title: {info['title']}")
if info.get("cwd"):
print(f" cwd: {info['cwd']}")
print(
" Internal range: "
f"{_format_range(info['created_at'], info['updated_at'])}"
)
print(
" Match range: "
f"{_format_range(info['match_created_at'], info['match_updated_at'])}"
)
print(f" Total mentions: {info['total_mentions']}")
print(
f" Keywords: {', '.join(f'{k}({v})' for k, v in info['keyword_counts'].items())}"
)
print(f" Match fields: {', '.join(info['match_sources'])}")
if info["excluded_untimed_records"]:
print(
" Date-filter note: excluded "
f"{info['excluded_untimed_records']} record(s) without an "
"internal timestamp; file mtime was not used."
)
print(f" Path: {info['path']}")
print()
else:
print(f"No Kimi CLI session matches (home: {kimi_home}).")
if not matches and not codex_matches and not kimi_matches:
print("No matches found.")
_print_search_widening_hint(args)
sys.exit(0)
elif args.command == "stats":
if not args.session_file.exists():
print(f"Error: Session file not found: {args.session_file}")
sys.exit(1)
print(f"Analyzing session: {args.session_file}\n")
analyzer = SessionAnalyzer(homes=[])
stats = analyzer.get_session_stats(args.session_file)
print("=" * 60)
print("Session Statistics")
print("=" * 60)
print("\nMessages:")
print(f" Total lines: {stats['total_lines']:,}")
print(f" User messages: {stats['user_messages']}")
print(f" Assistant messages: {stats['assistant_messages']}")
print("\nTool Usage:")
print(f" Write calls: {stats['write_calls']}")
print(f" Edit calls: {stats['edit_calls']}")
print(f" Read calls: {stats['read_calls']}")
print(f" Bash calls: {stats['bash_calls']}")
if stats["tool_uses"]:
print("\n All tools:")
for tool, count in sorted(
stats["tool_uses"].items(), key=lambda x: x[1], reverse=True
):
print(f" {tool}: {count}")
if args.show_files and stats["file_operations"]:
print(f"\nFile Operations ({len(stats['file_operations'])}):")
# Group by file
files = defaultdict(list)
for op, path in stats["file_operations"]:
files[path].append(op)
# Limit to 20 files to prevent terminal flooding on large sessions
for file_path, ops in list(files.items())[:20]:
filename = Path(file_path).name
op_summary = ", ".join(
f"{op}({ops.count(op)})" for op in set(ops)
)
print(f" {filename}")
print(f" Operations: {op_summary}")
print(f" Path: {file_path}")
print()
if __name__ == "__main__":
main()
scripts/extract_user_messages.py
#!/usr/bin/env python3
"""Extract verbatim user messages ("what the user actually said") from session history.
Covers every active Claude config home plus registered long-term archives (same
source discovery as analyze_sessions.py), de-duplicates across copies, and
separates genuine user prose from the five contamination classes documented in
references/session_file_format.md ("A user-role record is not necessarily
user-authored text"):
1. command envelopes (XML wrappers and bare `/cmd` strings) -> appendix
2. hook/loop-injected boilerplate — detected generically by frequency
(identical text, >= --min-dup occurrences, >= 400 normalized chars), both
standalone records and the tail-appended shape -> stripped / appendix
3. `[Image #N]` placeholders -> stripped (count kept)
4. whole-document pastes (>= 2000 normalized chars AND (>= 60% ASCII OR
>= 10 non-blank lines)) -> appendix. The multi-line branch catches CJK
meeting transcripts / multi-turn dialog / agent re-injections that stay
below the ASCII bar, while coherent voice dictation (few long paragraphs)
is preserved. Speaker-label counting was tried and rejected: user prose
that quotes people racks up more "name:" labels than a real transcript.
5. agent-voiced re-injection -> dropped when the text matches (exact or
contained-in) an assistant text that exists EARLIER than the record
Mid-work input typed while the assistant was busy is recovered from
`attachment.queued_command` records (origin.kind == "human") and de-duplicated
against later-delivered user records (same normalized text within 120s).
Usage:
extract_user_messages.py [OUT_BASE] [--days 7] [--group-by project|day]
[--min-dup 5] [--home PATH ...]
OUT_BASE defaults to ~/.claude-flow-viewer/user-words — persistent; avoid /tmp
(the OS purges it; 2026-08-07 战例: 产物写 /tmp 次日被清). Writes OUT_BASE.html
and OUT_BASE.md. Dates come from internal record
timestamps (UTC+8 rendering); file mtime is only a coarse prefilter.
"""
import argparse
import hashlib
import html as htmllib
import json
import re
import sys
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from pathlib import Path
# Multi-home discovery lives in the bundled `_core` package (see analyze_sessions.py).
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _core.sources import discover_claude_sources # noqa: E402
from _core.text import iter_jsonl # noqa: E402
CST = timezone(timedelta(hours=8))
# Records that are reliably not user prose, by structure alone.
NOISE_PREFIX = (
'<task-notification',
'[Request interrupted by user',
'Caveat:',
'Stop hook feedback:',
'Continue from where you left off',
'<local-command-',
'[Your previous response had no visible output',
'This session is being continued from a previous conversation',
'Another Claude session sent a message:',
)
CMD_RE = re.compile(r'<command-name>([^<]+)</command-name>\s*(?:<command-args>(.*?)</command-args>)?', re.S)
BARE_CMD_RE = re.compile(r'^(/[a-z][\w:-]*)\s*(.*)$', re.S)
IMG_TOKEN_RE = re.compile(r'\[Image #\d+\]\s*')
PATHISH_PREFIX = ('//', '/Users', '/tmp', '/var', '/home', '/opt')
FILLER = {x.lower() for x in (
'继续', '继续吧', '你直接继续', 'ok', 'okay', 'yes', '好', '好的', '好嘞', '嗯', '嗯嗯',
'对', '是的', '行', '可以', '做吧', 'done', 'check', 'fix', 'go', 'hi', 'hello', '收到',
)}
BOILER_MIN_CHARS = 400 # injected boilerplate is long; short repeats are just filler
PASTE_MIN_CHARS = 2000 # whole-document paste splitter …
PASTE_ASCII_RATIO = 0.60 # … length AND mostly-ASCII; long voice dictation stays below the ASCII bar
PASTE_MULTI_LINE_MIN = 10 # … OR long text spanning >= N non-blank lines = transcript/dialog/paste
AGENT_EXACT_MIN = 30 # below this a coincidental same-text match is plausible
AGENT_CONTAINS_MIN = 60
DELIVERED_WINDOW_SEC = 120
@dataclass
class Entry:
ts: datetime
project: str
text: str
images: int = 0
filler: bool = False
@dataclass
class Extraction:
entries: list = field(default_factory=list) # list[Entry]
commands: list = field(default_factory=list) # list[Entry]
injected: list = field(default_factory=list) # list[Entry] boilerplate
pastes: list = field(default_factory=list) # list[Entry]
subtracted: int = 0 # agent-voiced re-injections dropped
def normalize_text(s: str) -> str:
return re.sub(r'\s+', '', s)
def _record_ts(data: dict):
raw = data.get('timestamp') or ''
try:
return datetime.fromisoformat(raw.replace('Z', '+00:00'))
except ValueError:
return None
def _iter_session_files(sources, cutoff: datetime):
"""Yield (source_label, Path) for session JSONL files. mtime is only a coarse
prefilter with a 2-day grace band; the real window is applied per record."""
grace = cutoff.timestamp() - 2 * 86400
for src in sources:
projects = src.home / 'projects'
if not projects.is_dir():
continue
for f in projects.glob('*/*.jsonl'):
if f.name.startswith('agent-'):
continue
try:
if f.stat().st_mtime < grace:
continue
except OSError:
continue
yield src.label, f
def _assistant_texts(record: dict):
content = (record.get('message') or {}).get('content') if isinstance(record.get('message'), dict) else None
if record.get('type') != 'assistant' or not isinstance(content, list):
return
for block in content:
if isinstance(block, dict) and block.get('type') == 'text':
t = normalize_text(block.get('text', ''))
if t:
yield t
def build_agent_corpus(sources, cutoff: datetime):
"""Normalized assistant texts: exact-hash -> earliest ts, plus a long-text list
for containment checks. Used to subtract agent-voiced re-injections."""
exact = {}
long_texts = []
for _label, f in _iter_session_files(sources, cutoff):
try:
records = iter_jsonl(f)
for record in records:
ts = _record_ts(record)
if ts is None:
continue
for t in _assistant_texts(record):
if len(t) >= AGENT_EXACT_MIN:
h = hashlib.md5(t.encode()).digest()
if h not in exact or ts < exact[h]:
exact[h] = ts
if len(t) >= AGENT_CONTAINS_MIN:
long_texts.append((ts, t))
except Exception:
continue
giant = '\x00'.join(t for _, t in long_texts)
return exact, long_texts, giant
def _user_candidate_text(record: dict):
"""Return (text, n_images) for a user record, or None if it is not prose-shaped."""
content = (record.get('message') or {}).get('content') if isinstance(record.get('message'), dict) else None
if isinstance(content, str):
return content, 0
if isinstance(content, list):
if any(isinstance(b, dict) and b.get('type') == 'tool_result' for b in content):
return None
parts, n_img = [], 0
for b in content:
if not isinstance(b, dict):
continue
if b.get('type') == 'text':
parts.append(b.get('text', ''))
elif b.get('type') == 'image':
n_img += 1
return '\n'.join(p for p in parts if p), n_img
return None
def _command_label(text: str):
"""If the text is a slash-command invocation, return its '/name args' label."""
if text.startswith('<command-message>') or text.startswith('<command-name>'):
m = CMD_RE.search(text)
if m:
return f"{m.group(1)} {(m.group(2) or '').strip()}".strip()
return text[:60]
m = BARE_CMD_RE.match(text)
if m and not text.startswith(PATHISH_PREFIX):
return f"{m.group(1)} {m.group(2).strip()}".strip()
return None
def extract(sources, cutoff: datetime, min_dup: int = 5) -> Extraction:
result = Extraction()
seen_uuid = set()
raw_user = [] # (ts, project, text, n_img)
raw_queued = [] # (ts, project, text)
for _label, f in _iter_session_files(sources, cutoff):
project = f.parent.name
try:
records = iter_jsonl(f)
except Exception:
continue
for record in records:
rtype = record.get('type')
if rtype not in ('user', 'attachment'):
continue
if record.get('isSidechain') or record.get('isMeta'):
continue
ts = _record_ts(record)
if ts is None or ts < cutoff:
continue
if rtype == 'attachment':
att = record.get('attachment') or {}
origin = att.get('origin') or {}
if att.get('type') == 'queued_command' and origin.get('kind') == 'human':
prompt = att.get('prompt')
if isinstance(prompt, list):
prompt = '\n'.join(
x.get('text', '') if isinstance(x, dict) else str(x) for x in prompt)
if isinstance(prompt, str) and prompt.strip():
raw_queued.append((ts, project, prompt.strip()))
continue
if record.get('promptSource') not in (None, 'typed', 'queued'):
continue
got = _user_candidate_text(record)
if got is None:
continue
text, n_img = got
text = text.strip()
if not text or text.startswith(NOISE_PREFIX):
continue
uuid = record.get('uuid')
if uuid:
if uuid in seen_uuid:
continue
seen_uuid.add(uuid)
raw_user.append((ts, project, text, n_img))
# --- generic boilerplate detection: identical long text repeated many times ---
# Frequency is counted on raw text (standalone injections are byte-identical);
# membership checks normalize, tail-stripping searches the raw block.
freq = {}
for _ts, _p, text, _i in raw_user:
if len(normalize_text(text)) >= BOILER_MIN_CHARS:
freq[text] = freq.get(text, 0) + 1
boilerplate_raw = [t for t, n in freq.items() if n >= min_dup]
boilerplate_norm = {normalize_text(t) for t in boilerplate_raw}
agent_exact, agent_long, agent_giant = build_agent_corpus(sources, cutoff)
def classify(ts, project, text, n_img, into):
"""Route one candidate text to the right bucket. `into` is 'user' or 'queued'."""
label = _command_label(text)
if label is not None:
result.commands.append(Entry(ts, project, label))
return
norm = normalize_text(text)
if norm in boilerplate_norm:
result.injected.append(Entry(ts, project, text))
return
for block in boilerplate_raw:
idx = text.find(block)
if idx > 0:
text = text[:idx].rstrip()
break
text = IMG_TOKEN_RE.sub('', text).strip()
if not text:
return
norm = normalize_text(text)
if len(norm) >= PASTE_MIN_CHARS:
# ascii ratio uses normalized text (whitespace-stripped) so padding can't
# defeat it; line count uses raw text to see real paragraph structure.
ascii_ratio = sum(1 for ch in norm if ord(ch) < 128) / len(norm)
non_empty_lines = sum(1 for line in text.split('\n') if line.strip())
if ascii_ratio >= PASTE_ASCII_RATIO or non_empty_lines >= PASTE_MULTI_LINE_MIN:
result.pastes.append(Entry(ts, project, text))
return
if len(norm) >= AGENT_EXACT_MIN:
h = hashlib.md5(norm.encode()).digest()
earlier = agent_exact.get(h)
if earlier is not None and earlier < ts:
result.subtracted += 1
return
if len(norm) >= AGENT_CONTAINS_MIN and norm in agent_giant:
if any(ats < ts and norm in at for ats, at in agent_long):
result.subtracted += 1
return
probe = re.sub(r'[。！!？?～~．.\s]+$', '', text).lower()
result.entries.append(Entry(ts, project, text, images=n_img, filler=probe in FILLER))
for ts, project, text, n_img in raw_user:
classify(ts, project, text, n_img, 'user')
# Queued mid-work input: skip when the same text was delivered as a real record
# (the delivered copy owns the timeline slot).
delivered = {}
for e in result.entries + result.commands:
delivered.setdefault(normalize_text(e.text), []).append(e.ts)
for ts, project, text in raw_queued:
norm = normalize_text(text)
if any(abs((t2 - ts).total_seconds()) <= DELIVERED_WINDOW_SEC for t2 in delivered.get(norm, [])):
continue
before = len(result.entries) + len(result.commands) + len(result.injected) + len(result.pastes)
classify(ts, project, text, 0, 'queued')
after = len(result.entries) + len(result.commands) + len(result.injected) + len(result.pastes)
if after > before:
delivered.setdefault(norm, []).append(ts)
for bucket in (result.entries, result.commands, result.injected, result.pastes):
bucket.sort(key=lambda e: e.ts)
return result
# ---------------- rendering ----------------
def _proj_short(p: str) -> str:
parts = [x for x in p.lstrip('-').split('-') if x]
return parts[-1] if parts else p
def _esc(s: str) -> str:
return htmllib.escape(s)
def render_html(ext: Extraction, days: int, group_by: str) -> str:
entries = ext.entries
n_filler = sum(1 for e in entries if e.filler)
groups = {}
for e in entries:
key = e.project if group_by == 'project' else e.ts.astimezone(CST).strftime('%Y-%m-%d')
groups.setdefault(key, []).append(e)
if group_by == 'project':
order = sorted(groups, key=lambda k: max(e.ts for e in groups[k]), reverse=True)
else:
order = sorted(groups, reverse=True)
out = [f"""<!DOCTYPE html>
<html lang="zh-CN"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>我的原话 · 近{days}天</title>
<style>
:root {{ --ink:#1a1a1a; --meta:#9a9a9a; --hair:#e8e4de; --bg:#fbfaf8; }}
* {{ margin:0; padding:0; box-sizing:border-box; }}
body {{ background:var(--bg); color:var(--ink); font-family:-apple-system,"PingFang SC","Songti SC",serif;
display:flex; justify-content:center; padding:48px 20px 120px; }}
main {{ width:100%; max-width:700px; }}
header.page {{ font-size:13px; color:var(--meta); margin-bottom:56px;
display:flex; justify-content:space-between; align-items:baseline; }}
header.page label {{ cursor:pointer; user-select:none; }}
h2.grp {{ font-size:15px; font-weight:600; color:#4a4a4a;
margin:72px 0 26px; padding-bottom:10px; border-bottom:1px solid var(--hair);
display:flex; justify-content:space-between; align-items:baseline; gap:12px; }}
h2.grp:first-of-type {{ margin-top:0; }}
h2.grp .p {{ font-size:11px; font-weight:400; color:var(--meta);
font-family:ui-monospace,SFMono-Regular,Menlo,monospace;
overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }}
.msg {{ margin-bottom:34px; }}
.msg.filler {{ display:none; }}
body.show-filler .msg.filler {{ display:block; }}
body.show-filler .msg.filler .words {{ color:#8a8a8a; }}
.words {{ font-family:"Songti SC","Noto Serif CJK SC",serif; font-size:18.5px; line-height:1.9;
white-space:pre-wrap; word-break:break-word; }}
.meta {{ margin-top:7px; font-size:12px; color:var(--meta); }}
details {{ margin-top:96px; font-size:13px; color:var(--meta); }}
details summary {{ cursor:pointer; }}
details ol {{ margin-top:14px; padding-left:20px; line-height:2; }}
details .c {{ color:#6a6a6a; }}
</style></head><body><main>
<header class="page"><span>我的原话 · 近 {days} 天 · {len(entries)} 条</span>
<label><input type="checkbox" onchange="document.body.classList.toggle('show-filler',this.checked)"> 显示短应答（{n_filler} 条）</label></header>
"""]
for key in order:
rs = groups[key]
label = _proj_short(key) if group_by == 'project' else key
sub = key if group_by == 'project' else ''
out.append(f'<h2 class="grp"><span>{_esc(label)} · {len(rs)} 条</span><span class="p">{_esc(sub)}</span></h2>\n')
for e in rs:
t = e.ts.astimezone(CST).strftime('%m-%d %H:%M')
img = f' · 附图×{e.images}' if e.images else ''
cls = ' class="msg filler"' if e.filler else ' class="msg"'
out.append(f'<div{cls}><div class="words">{_esc(e.text)}</div><div class="meta">{t}{img}</div></div>\n')
for title, bucket, show_full in (
('斜杠命令', ext.commands, False),
('注入的固定文本', ext.injected, False),
('粘贴的长文', ext.pastes, True),
):
out.append(f'<details><summary>附录 · {title} {len(bucket)} 条（不计入原话）</summary><ol>\n')
for e in bucket:
t = e.ts.astimezone(CST).strftime('%m-%d %H:%M')
head = re.sub(r'\s+', ' ', e.text)[:80]
out.append(f'<li><span class="c">{t} · {_esc(_proj_short(e.project))}</span> {_esc(head)}'
f'{"…" if len(e.text) > 80 else ""}</li>\n')
out.append('</ol>')
if show_full:
for e in bucket:
t = e.ts.astimezone(CST).strftime('%m-%d %H:%M')
head = re.sub(r'\s+', ' ', e.text)[:80]
out.append(f'<details style="margin-top:14px"><summary>全文：{_esc(head)}…</summary>'
f'<div style="white-space:pre-wrap;font-size:12px;margin-top:8px">{_esc(e.text[:20000])}</div></details>\n')
out.append('</details>\n')
out.append('</main></body></html>\n')
return ''.join(out)
def render_markdown(ext: Extraction, days: int, group_by: str) -> str:
entries = ext.entries
groups = {}
for e in entries:
key = e.project if group_by == 'project' else e.ts.astimezone(CST).strftime('%Y-%m-%d')
groups.setdefault(key, []).append(e)
if group_by == 'project':
order = sorted(groups, key=lambda k: max(e.ts for e in groups[k]), reverse=True)
else:
order = sorted(groups, reverse=True)
md = [f'# 我的原话 · 近 {days} 天（{len(entries)} 条）\n']
for key in order:
rs = groups[key]
label = _proj_short(key) if group_by == 'project' else key
md.append(f'\n## {label}（{len(rs)} 条）\n')
for e in rs:
t = e.ts.astimezone(CST).strftime('%m-%d %H:%M')
img = f' · 附图×{e.images}' if e.images else ''
md.append(f'\n{e.text}\n\n<sub>{t}{img}</sub>\n')
for title, bucket in (('斜杠命令', ext.commands), ('注入的固定文本', ext.injected), ('粘贴的长文', ext.pastes)):
md.append(f'\n---\n\n## 附录 · {title}（{len(bucket)} 条，不计入原话）\n')
for e in bucket:
head = re.sub(r'\s+', ' ', e.text)[:80]
md.append(f'\n- {head}{"…" if len(e.text) > 80 else ""} <sub>{e.ts.astimezone(CST).strftime("%m-%d %H:%M")} · {_proj_short(e.project)}</sub>')
return ''.join(md)
def main() -> int:
ap = argparse.ArgumentParser(description='Extract verbatim user messages from Claude history.')
ap.add_argument('out_base', nargs='?', default='~/.claude-flow-viewer/user-words',
help='output path without extension (.html/.md appended); '
'default ~/.claude-flow-viewer/user-words — persistent, NOT /tmp which the OS purges')
ap.add_argument('--days', type=int, default=7)
ap.add_argument('--group-by', choices=('project', 'day'), default='project')
ap.add_argument('--min-dup', type=int, default=5,
help='identical long texts at/above this count are treated as injected boilerplate')
ap.add_argument('--home', action='append', default=None,
help='exact scope: only this config home (repeatable); bypasses archive registry')
args = ap.parse_args()
sources, warnings = discover_claude_sources(explicit_homes=args.home)
for w in warnings:
print(f'warning: {w}', file=sys.stderr)
cutoff = datetime.now(timezone.utc) - timedelta(days=args.days)
ext = extract(sources, cutoff, min_dup=args.min_dup)
out = Path(args.out_base).expanduser()
out.parent.mkdir(parents=True, exist_ok=True)
out.with_suffix('.html').write_text(render_html(ext, args.days, args.group_by), encoding='utf-8')
out.with_suffix('.md').write_text(render_markdown(ext, args.days, args.group_by), encoding='utf-8')
n_filler = sum(1 for e in ext.entries if e.filler)
print(f'entries={len(ext.entries)} (filler={n_filler}) commands={len(ext.commands)} '
f'injected={len(ext.injected)} pastes={len(ext.pastes)} agent-subtracted={ext.subtracted}')
print(f'out={out}.html / {out}.md')
return 0
if __name__ == '__main__':
raise SystemExit(main())
scripts/recover_content.py
#!/usr/bin/env python3
"""Recover files from Claude Code session history.
The highest-fidelity source is a ``file-history-snapshot`` record paired with
``<claude-home>/file-history/<session-id>/``. Those backups preserve exact
bytes captured after Write, Edit, and shell-driven changes. When no snapshot
metadata exists for a path, the script retains the older Write-tool recovery
mode and labels it as a lower-fidelity checkpoint.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import sys
import tempfile
from collections import deque
from datetime import datetime
from pathlib import Path, PurePosixPath, PureWindowsPath
from typing import Any, Dict, Iterable, List, Optional
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _core.sources import ( # noqa: E402
HistorySource,
HistorySourceConfigError,
discover_claude_sources,
)
BACKUP_VERSION_RE = re.compile(r"@v(\d+)$")
class RecoveryError(RuntimeError):
"""Raised when exact recovery cannot be completed safely."""
def _timestamp_rank(value: object) -> float:
if not isinstance(value, str) or not value:
return float("-inf")
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return float("-inf")
if parsed.tzinfo is None:
return float("-inf")
return parsed.timestamp()
def _backup_version(metadata: Dict[str, Any]) -> int:
value = metadata.get("version")
metadata_version: Optional[int] = None
if isinstance(value, int) and not isinstance(value, bool) and value >= 0:
metadata_version = value
elif isinstance(value, str) and value.isdigit():
metadata_version = int(value)
backup_name = metadata.get("backupFileName")
name_version: Optional[int] = None
if isinstance(backup_name, str):
match = BACKUP_VERSION_RE.search(backup_name)
if match:
name_version = int(match.group(1))
if (
metadata_version is not None
and name_version is not None
and metadata_version != name_version
):
raise ValueError(
f"version {metadata_version} conflicts with backup name {backup_name!r}"
)
version = metadata_version if metadata_version is not None else name_version
if version is None:
raise ValueError("snapshot entry has no valid non-negative version")
return version
def _entry_rank(entry: Dict[str, Any]) -> tuple[int, float, float]:
return (
entry["version"],
_timestamp_rank(entry.get("backup_time")),
_timestamp_rank(entry.get("snapshot_time")),
)
def _metadata_version_hint(metadata: Dict[str, Any]) -> Optional[int]:
"""Return a conservative version hint for malformed snapshot metadata."""
values: List[int] = []
raw_version = metadata.get("version")
if (
isinstance(raw_version, int)
and not isinstance(raw_version, bool)
and raw_version >= 0
):
values.append(raw_version)
elif isinstance(raw_version, str) and raw_version.isdigit():
values.append(int(raw_version))
backup_name = metadata.get("backupFileName")
if isinstance(backup_name, str):
match = BACKUP_VERSION_RE.search(backup_name)
if match:
values.append(int(match.group(1)))
return max(values) if values else None
def _inspect_file(path: Path) -> tuple[str, int, int]:
digest = hashlib.sha256()
size = 0
newlines = 0
with path.open("rb") as handle:
while chunk := handle.read(1024 * 1024):
digest.update(chunk)
size += len(chunk)
newlines += chunk.count(b"\n")
lines = newlines + (1 if size else 0)
return digest.hexdigest(), size, lines
def _atomic_write(path: Path, content: bytes) -> None:
"""Atomically replace ``path`` with ``content``."""
path.parent.mkdir(parents=True, exist_ok=True)
descriptor, temporary_name = tempfile.mkstemp(
dir=path.parent,
prefix=f".{path.name}.",
suffix=".tmp",
)
temporary_path = Path(temporary_name)
try:
with os.fdopen(descriptor, "wb") as handle:
handle.write(content)
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary_path, path)
finally:
if temporary_path.exists():
temporary_path.unlink()
def _atomic_copy(path: Path, source: Path, expected_sha256: str) -> None:
"""Copy one checkpoint in chunks and verify it before publishing it."""
path.parent.mkdir(parents=True, exist_ok=True)
descriptor, temporary_name = tempfile.mkstemp(
dir=path.parent,
prefix=f".{path.name}.",
suffix=".tmp",
)
temporary_path = Path(temporary_name)
digest = hashlib.sha256()
try:
with (
source.open("rb") as input_handle,
os.fdopen(descriptor, "wb") as output_handle,
):
while chunk := input_handle.read(1024 * 1024):
digest.update(chunk)
output_handle.write(chunk)
output_handle.flush()
os.fsync(output_handle.fileno())
if digest.hexdigest() != expected_sha256:
raise RecoveryError(
f"Checkpoint changed while it was being copied: {source}"
)
os.replace(temporary_path, path)
finally:
if temporary_path.exists():
temporary_path.unlink()
class SessionContentRecovery:
"""Extract and recover files from all known copies of one Claude session."""
CODEX_RECORD_TYPES = {
"session_meta",
"response_item",
"event_msg",
"turn_context",
"world_state",
}
CLAUDE_RECORD_TYPES = {
"assistant",
"user",
"queue-operation",
"attachment",
"file-history-snapshot",
}
def __init__(
self,
session_file: Path,
output_dir: Optional[Path] = None,
file_history_roots: Optional[Iterable[Path]] = None,
write_only: bool = False,
) -> None:
self.session_file = Path(session_file).expanduser()
self.output_dir = (
Path(output_dir).expanduser()
if output_dir
else Path.cwd() / "recovered_content"
)
self.explicit_file_history_roots = [
Path(root).expanduser() for root in (file_history_roots or [])
]
self.write_only = write_only
self.warnings: List[str] = []
self._scan_result: Optional[Dict[str, Any]] = None
self._source_cache: Optional[List[HistorySource]] = None
self._session_file_cache: Optional[List[Path]] = None
self.stats = {
"total_lines": 0,
"duplicate_records_skipped": 0,
"session_copies": 0,
"write_calls": 0,
"failed_write_calls_skipped": 0,
"edit_calls": 0,
"snapshot_records": 0,
"snapshot_paths": 0,
"tombstone_paths": 0,
"files_recovered": 0,
"exact_recoveries": 0,
"write_recoveries": 0,
}
def _history_sources(self) -> List[HistorySource]:
if self._source_cache is not None:
return self._source_cache
try:
sources, warnings = discover_claude_sources()
except HistorySourceConfigError as error:
raise RecoveryError(str(error)) from error
self.warnings.extend(warnings)
self._source_cache = sources
return sources
def _discover_session_files(self) -> List[Path]:
"""Find same-id JSONL copies in active homes and registered archives."""
if self._session_file_cache is not None:
return self._session_file_cache
files: List[Path] = []
seen: set[str] = set()
def add(candidate: Path) -> None:
if not candidate.is_file():
return
try:
key = str(candidate.resolve())
except OSError:
key = str(candidate.absolute())
if key not in seen:
seen.add(key)
files.append(candidate)
add(self.session_file)
filename = self.session_file.name
for source in self._history_sources():
projects_dir = source.home / "projects"
if not projects_dir.is_dir():
continue
for project_dir in sorted(projects_dir.iterdir()):
if project_dir.is_dir():
add(project_dir / filename)
if not files:
raise RecoveryError(f"Session file not found: {self.session_file}")
self._session_file_cache = files
self.stats["session_copies"] = len(files)
return files
@staticmethod
def _record_error(
errors: Dict[str, List[Dict[str, Any]]],
original_path: str,
message: str,
metadata: Optional[Dict[str, Any]],
snapshot_time: object,
source_file: Path,
line_num: int,
) -> None:
version = _metadata_version_hint(metadata or {})
rank = (
(
version,
_timestamp_rank((metadata or {}).get("backupTime")),
_timestamp_rank(snapshot_time),
)
if version is not None
else None
)
errors.setdefault(original_path, []).append(
{
"message": f"{source_file}:{line_num}: {message}",
"rank": rank,
}
)
def _consider_snapshot_entry(
self,
mapping: Dict[str, Dict[str, Any]],
candidate: Dict[str, Any],
errors: Dict[str, List[Dict[str, Any]]],
) -> None:
original_path = candidate["file_path"]
previous = mapping.get(original_path)
if previous is None or _entry_rank(candidate) > _entry_rank(previous):
mapping[original_path] = candidate
return
if _entry_rank(candidate) != _entry_rank(previous):
return
previous_name = previous.get("backup_file_name")
candidate_name = candidate.get("backup_file_name")
if previous_name != candidate_name:
self._record_error(
errors,
original_path,
"conflicting snapshot states have the same version and timestamp",
{
"version": candidate["version"],
"backupFileName": candidate_name,
"backupTime": candidate.get("backup_time"),
},
candidate.get("snapshot_time"),
candidate["source_file"],
candidate["line"],
)
def _consume_snapshot(
self,
data: Dict[str, Any],
source_file: Path,
line_num: int,
snapshots: Dict[str, Dict[str, Any]],
tombstones: Dict[str, Dict[str, Any]],
errors: Dict[str, List[Dict[str, Any]]],
) -> None:
self.stats["snapshot_records"] += 1
snapshot = data.get("snapshot")
if not isinstance(snapshot, dict):
self.warnings.append(
f"{source_file}:{line_num}: snapshot payload is not an object"
)
return
tracked = snapshot.get("trackedFileBackups")
if tracked is None:
return
if not isinstance(tracked, dict):
self.warnings.append(
f"{source_file}:{line_num}: trackedFileBackups is not an object"
)
return
snapshot_time = snapshot.get("timestamp", "")
for original_path, metadata in tracked.items():
if not isinstance(original_path, str) or not original_path:
self.warnings.append(
f"{source_file}:{line_num}: ignored snapshot entry with an invalid path"
)
continue
if not isinstance(metadata, dict):
self._record_error(
errors,
original_path,
"snapshot metadata is not an object",
None,
snapshot_time,
source_file,
line_num,
)
continue
if "backupFileName" not in metadata:
self._record_error(
errors,
original_path,
"snapshot metadata is missing backupFileName",
metadata,
snapshot_time,
source_file,
line_num,
)
continue
backup_name = metadata.get("backupFileName")
if backup_name is not None and (
not isinstance(backup_name, str) or not backup_name
):
self._record_error(
errors,
original_path,
"snapshot backupFileName must be a non-empty string or null",
metadata,
snapshot_time,
source_file,
line_num,
)
continue
try:
version = _backup_version(metadata)
except ValueError as error:
self._record_error(
errors,
original_path,
str(error),
metadata,
snapshot_time,
source_file,
line_num,
)
continue
candidate = {
"line": line_num,
"source_file": source_file,
"file_path": original_path,
"backup_file_name": backup_name,
"version": version,
"backup_time": metadata.get("backupTime", ""),
"snapshot_time": snapshot_time,
}
destination = tombstones if backup_name is None else snapshots
self._consider_snapshot_entry(destination, candidate, errors)
def _scan_session(self) -> Dict[str, Any]:
if self._scan_result is not None:
return self._scan_result
writes: List[Dict[str, Any]] = []
failed_tool_use_ids: set[str] = set()
edit_summaries: deque[Dict[str, Any]] = deque(maxlen=5)
snapshots: Dict[str, Dict[str, Any]] = {}
tombstones: Dict[str, Dict[str, Any]] = {}
snapshot_error_candidates: Dict[str, List[Dict[str, Any]]] = {}
session_ids: List[str] = []
record_hashes_from_prior_copies: set[str] = set()
saw_claude_signature = False
saw_codex_signature = False
session_files = self._discover_session_files()
for session_file in session_files:
copy_record_hashes: set[str] = set()
stem = session_file.stem
if stem and stem not in session_ids:
session_ids.append(stem)
try:
handle = session_file.open("r", encoding="utf-8", errors="replace")
except OSError as error:
raise RecoveryError(
f"Cannot read session copy {session_file}: {error}"
) from error
with handle:
for line_num, line in enumerate(handle, 1):
self.stats["total_lines"] += 1
try:
data = json.loads(line)
except json.JSONDecodeError:
continue
if not isinstance(data, dict):
continue
canonical = json.dumps(
data,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
record_hash = hashlib.sha256(canonical).hexdigest()
if record_hash in record_hashes_from_prior_copies:
self.stats["duplicate_records_skipped"] += 1
continue
copy_record_hashes.add(record_hash)
record_type = data.get("type")
if record_type in self.CODEX_RECORD_TYPES:
saw_codex_signature = True
if record_type in self.CLAUDE_RECORD_TYPES or isinstance(
data.get("sessionId"), str
):
saw_claude_signature = True
session_id = data.get("sessionId")
if (
isinstance(session_id, str)
and session_id
and session_id not in session_ids
):
session_ids.append(session_id)
if record_type == "file-history-snapshot":
self._consume_snapshot(
data,
session_file,
line_num,
snapshots,
tombstones,
snapshot_error_candidates,
)
message = data.get("message")
nested_role = (
message.get("role") if isinstance(message, dict) else None
)
role = data.get("role") or nested_role
content = data.get("content")
if content is None and isinstance(message, dict):
content = message.get("content", [])
if not isinstance(content, list):
continue
for item in content:
if (
isinstance(item, dict)
and item.get("type") == "tool_result"
and item.get("is_error") is True
):
tool_use_id = item.get("tool_use_id")
if isinstance(tool_use_id, str) and tool_use_id:
failed_tool_use_ids.add(tool_use_id)
if role != "assistant":
continue
for item in content:
if not isinstance(item, dict) or item.get("type") != "tool_use":
continue
tool_input = item.get("input")
if not isinstance(tool_input, dict):
continue
if item.get("name") == "Write":
file_path = tool_input.get("file_path", "")
file_content = tool_input.get("content", "")
if isinstance(file_path, str) and isinstance(
file_content, str
):
writes.append(
{
"line": line_num,
"source_file": session_file,
"file_path": file_path,
"content": file_content,
"timestamp": data.get("timestamp", ""),
"tool_use_id": (
item.get("id")
if isinstance(item.get("id"), str)
else None
),
}
)
self.stats["write_calls"] += 1
elif item.get("name") == "Edit":
edit_summaries.append(
{
"line": line_num,
"source_file": session_file,
"file_path": tool_input.get("file_path", ""),
"timestamp": data.get("timestamp", ""),
}
)
self.stats["edit_calls"] += 1
record_hashes_from_prior_copies.update(copy_record_hashes)
usable_writes: List[Dict[str, Any]] = []
for write in writes:
tool_use_id = write.get("tool_use_id")
if tool_use_id and tool_use_id in failed_tool_use_ids:
self.stats["failed_write_calls_skipped"] += 1
self.warnings.append(
"Skipped Write checkpoint with an explicit failed tool result: "
f"{write['file_path']} ({write['source_file']}:{write['line']})"
)
continue
usable_writes.append(write)
writes = usable_writes
if saw_codex_signature and not saw_claude_signature:
raise RecoveryError(
"This is a Codex rollout. Keyword search supports Codex with "
"analyze_sessions.py --codex, but file recovery currently supports "
"Claude Code JSONL sessions only."
)
all_snapshot_paths = set(snapshots) | set(tombstones)
blocking_errors: Dict[str, str] = {}
for original_path, error_entries in snapshot_error_candidates.items():
valid_ranks = [
_entry_rank(entry)
for entry in (
snapshots.get(original_path),
tombstones.get(original_path),
)
if entry is not None
]
best_valid_rank = max(valid_ranks) if valid_ranks else None
for error_entry in error_entries:
error_rank = error_entry["rank"]
if (
error_rank is None
or best_valid_rank is None
or error_rank >= best_valid_rank
):
blocking_errors[original_path] = error_entry["message"]
else:
self.warnings.append(
"Ignored older malformed snapshot metadata for "
f"{original_path}: {error_entry['message']}"
)
for original_path in set(snapshots) & set(tombstones):
if _entry_rank(snapshots[original_path]) == _entry_rank(
tombstones[original_path]
):
blocking_errors[original_path] = (
"the same file-history version is recorded as both a backup "
"and a deletion tombstone"
)
self.stats["snapshot_paths"] = len(all_snapshot_paths)
self.stats["tombstone_paths"] = len(tombstones)
self._scan_result = {
"writes": writes,
"edit_summaries": list(edit_summaries),
"snapshots": snapshots,
"tombstones": tombstones,
"snapshot_errors": blocking_errors,
"session_ids": session_ids,
"session_files": session_files,
}
return self._scan_result
def extract_write_calls(self) -> List[Dict[str, Any]]:
"""Return every valid Write tool call found in the session union."""
return list(self._scan_session()["writes"])
def extract_edit_calls(self) -> List[Dict[str, Any]]:
"""Return lightweight summaries for at most the five latest Edit calls."""
return list(self._scan_session()["edit_summaries"])
def extract_file_history_snapshots(self) -> Dict[str, Dict[str, Any]]:
"""Return the latest usable backup checkpoint for every path."""
return dict(self._scan_session()["snapshots"])
def _file_history_roots(self) -> List[Path]:
roots: List[Path] = []
seen: set[str] = set()
def add(root: Path) -> None:
expanded = root.expanduser()
try:
key = str(expanded.resolve())
except OSError:
key = str(expanded.absolute())
if key not in seen:
seen.add(key)
roots.append(expanded)
for session_file in self._discover_session_files():
for parent in session_file.resolve().parents:
if parent.name == "projects":
add(parent.parent / "file-history")
break
for source in self._history_sources():
add(source.home / "file-history")
for root in self.explicit_file_history_roots:
add(root)
return roots
@staticmethod
def _safe_identifier(value: str, label: str) -> str:
if not value or value in {".", ".."} or "/" in value or "\\" in value:
raise RecoveryError(f"Unsafe {label}: {value!r}")
return value
def _read_snapshot_backup(
self, entry: Dict[str, Any], roots: List[Path], session_ids: List[str]
) -> tuple[Path, str, int, int]:
backup_name = self._safe_identifier(
entry["backup_file_name"], "file-history backup name"
)
safe_session_ids = [
self._safe_identifier(session_id, "session id")
for session_id in session_ids
]
matches: List[tuple[Path, str, int, int]] = []
searched: List[Path] = []
for root in roots:
try:
resolved_root = root.resolve()
except OSError:
resolved_root = root.absolute()
for session_id in safe_session_ids:
session_dir = root / session_id
candidate = session_dir / backup_name
searched.append(candidate)
if session_dir.is_symlink():
raise RecoveryError(
f"Unsafe file-history session directory symlink: {session_dir}"
)
if not session_dir.exists():
continue
if not session_dir.is_dir():
raise RecoveryError(
"Unsafe file-history session object (expected a directory): "
f"{session_dir}"
)
try:
resolved_dir = session_dir.resolve()
resolved_dir.relative_to(resolved_root)
except (OSError, ValueError) as error:
raise RecoveryError(
f"file-history session directory escapes its root: {session_dir}"
) from error
if candidate.is_symlink():
raise RecoveryError(
f"Unsafe file-history backup symlink: {candidate}"
)
if not candidate.exists():
continue
if not candidate.is_file():
raise RecoveryError(
"Unsafe file-history backup object (expected a regular file): "
f"{candidate}"
)
try:
resolved_candidate = candidate.resolve()
resolved_candidate.relative_to(resolved_dir)
resolved_candidate.relative_to(resolved_root)
except (OSError, ValueError) as error:
raise RecoveryError(
f"file-history backup escapes its session directory: {candidate}"
) from error
try:
digest, size, lines = _inspect_file(candidate)
except OSError as error:
raise RecoveryError(
f"Cannot read file-history backup {candidate}: {error}"
) from error
matches.append((candidate, digest, size, lines))
if not matches:
searched_dirs = sorted({str(path.parent) for path in searched})
detail = (
", ".join(searched_dirs) if searched_dirs else "no roots discovered"
)
raise RecoveryError(
"Snapshot metadata exists but its exact backup is unavailable for "
f"{entry['file_path']}. Expected {backup_name} under: {detail}. "
"Provide the companion root with --file-history-root, or use "
"--write-only only if a lower-fidelity Write checkpoint is acceptable."
)
digests = {digest for _, digest, _, _ in matches}
if len(digests) != 1:
locations = ", ".join(str(path) for path, _, _, _ in matches)
raise RecoveryError(
"Conflicting file-history backups have the same metadata name but "
f"different bytes: {locations}"
)
return matches[0]
def _output_path(self, original_path: str) -> Path:
if re.match(r"^[A-Za-z]:[\\/]", original_path):
pure_path = PureWindowsPath(original_path)
parts = list(pure_path.parts)
start = 1
if len(parts) > 2 and parts[1].lower() == "users":
start = 3
else:
pure_path = PurePosixPath(original_path)
parts = list(pure_path.parts)
start = 0
if pure_path.is_absolute():
start = 1
if len(parts) > 2 and parts[1].lower() in {"users", "home"}:
start = 3
relative_parts = parts[start:]
if not relative_parts and pure_path.name:
relative_parts = [pure_path.name]
if not relative_parts or any(
part in {"", ".", ".."} for part in relative_parts
):
raise RecoveryError(f"Unsafe recovered file path: {original_path!r}")
output_path = self.output_dir.joinpath(*relative_parts)
try:
output_path.resolve().relative_to(self.output_dir.resolve())
except (OSError, ValueError) as error:
raise RecoveryError(
f"Recovered file path escapes the output directory: {original_path!r}"
) from error
return output_path
def _preflight_destinations(self, planned: List[Dict[str, Any]]) -> None:
"""Reject deterministic collisions before writing any recovered bytes."""
if self.output_dir.exists() and not self.output_dir.is_dir():
raise RecoveryError(
f"Recovery output path is not a directory: {self.output_dir}"
)
report_path = self.output_dir / "recovery_report.txt"
report_key = report_path.resolve()
destinations: Dict[Path, str] = {}
for item in planned:
output_path = self._output_path(item["original_path"])
key = output_path.resolve()
if key == report_key:
raise RecoveryError(
"A recovered artifact would overwrite the reserved recovery "
f"report: {item['original_path']!r} -> {report_path}"
)
collision = destinations.get(key)
if collision and collision != item["original_path"]:
raise RecoveryError(
"Two original paths map to the same recovery destination: "
f"{collision!r} and {item['original_path']!r} -> {output_path}"
)
destinations[key] = item["original_path"]
item["output_path"] = output_path
destination_keys = list(destinations)
for index, first in enumerate(destination_keys):
for second in destination_keys[index + 1 :]:
if first in second.parents or second in first.parents:
raise RecoveryError(
"Recovered destinations have a file/directory ancestor "
f"collision: {first} and {second}"
)
for destination in [
*(item["output_path"] for item in planned),
report_path,
]:
if destination.is_symlink():
raise RecoveryError(
f"Recovery destination is an existing symlink: {destination}"
)
if destination.exists() and destination.is_dir():
raise RecoveryError(
f"Recovery destination is an existing directory: {destination}"
)
parent = destination.parent
while parent != self.output_dir and self.output_dir in parent.parents:
if parent.is_symlink():
raise RecoveryError(
f"Recovery destination parent is a symlink: {parent}"
)
if parent.exists() and not parent.is_dir():
raise RecoveryError(
f"Recovery destination parent is not a directory: {parent}"
)
parent = parent.parent
@staticmethod
def _tombstone_note(tombstone: Dict[str, Any]) -> str:
timestamp = tombstone.get("backup_time") or tombstone.get("snapshot_time")
when = timestamp if isinstance(timestamp, str) and timestamp else "unknown time"
return f"recorded deleted at file-history v{tombstone['version']} ({when})"
@staticmethod
def _tombstone_follows_write(
tombstone: Dict[str, Any], write: Dict[str, Any]
) -> bool:
tombstone_time = _timestamp_rank(
tombstone.get("backup_time") or tombstone.get("snapshot_time")
)
write_time = _timestamp_rank(write.get("timestamp"))
return tombstone_time > write_time
@staticmethod
def _write_follows_tombstone(
write: Dict[str, Any], tombstone: Dict[str, Any]
) -> bool:
write_time = _timestamp_rank(write.get("timestamp"))
tombstone_time = _timestamp_rank(
tombstone.get("backup_time") or tombstone.get("snapshot_time")
)
return write_time > tombstone_time
@staticmethod
def _write_follows_snapshot(
write: Dict[str, Any], snapshot: Dict[str, Any]
) -> bool:
write_time = _timestamp_rank(write.get("timestamp"))
snapshot_time = max(
_timestamp_rank(snapshot.get("backup_time")),
_timestamp_rank(snapshot.get("snapshot_time")),
)
return (
write_time != float("-inf")
and snapshot_time != float("-inf")
and write_time > snapshot_time
)
def _select_writes(
self, writes: List[Dict[str, Any]]
) -> tuple[Dict[str, Dict[str, Any]], Dict[str, str]]:
calls_by_path: Dict[str, List[Dict[str, Any]]] = {}
for call in writes:
file_path = call["file_path"]
if file_path:
calls_by_path.setdefault(file_path, []).append(call)
selected: Dict[str, Dict[str, Any]] = {}
conflicts: Dict[str, str] = {}
for file_path, calls in calls_by_path.items():
latest_rank = max(_timestamp_rank(call.get("timestamp")) for call in calls)
latest = [
call
for call in calls
if _timestamp_rank(call.get("timestamp")) == latest_rank
]
latest.sort(key=lambda call: (str(call["source_file"]), call["line"]))
selected[file_path] = latest[0]
conflict = next(
(
call
for call in latest[1:]
if call["content"] != latest[0]["content"]
),
None,
)
if conflict is not None:
conflicts[file_path] = (
"Conflicting Write checkpoints have the same timestamp for "
f"{file_path}: {latest[0]['source_file']}:{latest[0]['line']} and "
f"{conflict['source_file']}:{conflict['line']}"
)
return selected, conflicts
def recover_files(
self, keywords: Optional[List[str]] = None
) -> List[Dict[str, Any]]:
"""Plan and preflight the whole recovery before writing selected files."""
scan = self._scan_session()
snapshots: Dict[str, Dict[str, Any]] = scan["snapshots"]
tombstones: Dict[str, Dict[str, Any]] = scan["tombstones"]
snapshot_errors: Dict[str, str] = scan["snapshot_errors"]
writes_by_path, write_conflicts = self._select_writes(scan["writes"])
failures: List[str] = []
all_paths = (
set(writes_by_path)
| set(write_conflicts)
| set(snapshots)
| set(tombstones)
| set(snapshot_errors)
)
if keywords:
lowered = [keyword.casefold() for keyword in keywords]
all_paths = {
path
for path in all_paths
if any(keyword in path.casefold() for keyword in lowered)
}
roots = self._file_history_roots()
planned: List[Dict[str, Any]] = []
for original_path in sorted(all_paths):
if not self.write_only and original_path in snapshot_errors:
failures.append(f"{original_path}: {snapshot_errors[original_path]}")
continue
snapshot = snapshots.get(original_path)
tombstone = tombstones.get(original_path)
write = writes_by_path.get(original_path)
later_state: Optional[str] = None
latest_tombstone = (
tombstone
if tombstone is not None
and (
snapshot is None
or _entry_rank(tombstone) > _entry_rank(snapshot)
)
else None
)
write_supersedes_snapshot = bool(
write is not None
and snapshot is not None
and (
self._write_follows_snapshot(write, snapshot)
or (
latest_tombstone is not None
and self._write_follows_tombstone(write, latest_tombstone)
)
)
)
if (
snapshot
and latest_tombstone
and not write_supersedes_snapshot
):
later_state = self._tombstone_note(latest_tombstone)
if (
not self.write_only
and snapshot is not None
and not write_supersedes_snapshot
):
try:
backup_path, digest, size, lines = self._read_snapshot_backup(
snapshot, roots, scan["session_ids"]
)
except RecoveryError as error:
failures.append(str(error))
continue
fidelity = "exact bytes from captured checkpoint"
if later_state:
fidelity += " before a later recorded deletion"
planned.append(
{
"original_path": original_path,
"content": None,
"source_file": backup_path,
"sha256": digest,
"size": size,
"lines": lines,
"source": "file-history",
"source_path": str(backup_path),
"version": snapshot["version"],
"timestamp": snapshot["backup_time"]
or snapshot["snapshot_time"],
"fidelity": fidelity,
"later_state": later_state,
}
)
continue
if write is None:
if self.write_only and (snapshot is not None or tombstone is not None):
self.warnings.append(
"Skipped snapshot-only path in --write-only mode: "
f"{original_path}"
)
elif tombstone is not None:
self.warnings.append(
"Skipped deleted path with no recoverable prior checkpoint: "
f"{original_path} ({self._tombstone_note(tombstone)})"
)
continue
if original_path in write_conflicts:
failures.append(write_conflicts[original_path])
continue
content = write["content"].encode("utf-8")
if latest_tombstone is not None and self._tombstone_follows_write(
latest_tombstone, write
):
later_state = self._tombstone_note(latest_tombstone)
planned.append(
{
"original_path": original_path,
"content": content,
"source_file": None,
"sha256": hashlib.sha256(content).hexdigest(),
"size": len(content),
"lines": content.count(b"\n") + (1 if content else 0),
"source": "Write",
"source_path": f"{write['source_file']}:{write['line']}",
"version": None,
"timestamp": write.get("timestamp", ""),
"fidelity": (
"Write checkpoint; later Edit or shell changes may be absent"
),
"later_state": later_state,
}
)
if failures:
raise RecoveryError(
"Recovery aborted before writing files:\n- " + "\n- ".join(failures)
)
self._preflight_destinations(planned)
saved: List[Dict[str, Any]] = []
for item in planned:
output_path = item["output_path"]
if item["source"] == "file-history":
_atomic_copy(output_path, item["source_file"], item["sha256"])
else:
_atomic_write(output_path, item["content"])
saved.append(
{
"file": output_path.name,
"original_path": item["original_path"],
"size": item["size"],
"lines": item["lines"],
"timestamp": item["timestamp"] or "unknown",
"output_path": str(output_path),
"source": item["source"],
"source_path": item["source_path"],
"version": item["version"],
"fidelity": item["fidelity"],
"later_state": item["later_state"],
"sha256": item["sha256"],
}
)
self.stats["files_recovered"] += 1
if item["source"] == "file-history":
self.stats["exact_recoveries"] += 1
else:
self.stats["write_recoveries"] += 1
return saved
def save_recovered_files(
self, write_calls: List[Dict[str, Any]], keywords: Optional[List[str]] = None
) -> List[Dict[str, Any]]:
"""Compatibility wrapper; recovery now selects the best source itself."""
del write_calls
return self.recover_files(keywords)
def generate_report(self, saved_files: List[Dict[str, Any]]) -> str:
"""Generate a provenance-rich recovery report."""
scan = self._scan_session()
report_lines = [
"=" * 60,
"Claude Code Session Content Recovery Report",
"=" * 60,
"",
f"Requested session file: {self.session_file}",
f"Output directory: {self.output_dir}",
"",
"Session copies scanned:",
*(f" - {path}" for path in scan["session_files"]),
"",
"Statistics:",
f" Session copies: {self.stats['session_copies']}",
f" Total lines processed: {self.stats['total_lines']:,}",
f" Duplicate records skipped: {self.stats['duplicate_records_skipped']}",
f" Write tool calls found: {self.stats['write_calls']}",
" Failed Write tool calls skipped: "
f"{self.stats['failed_write_calls_skipped']}",
f" Edit tool calls found: {self.stats['edit_calls']}",
f" File-history snapshot records: {self.stats['snapshot_records']}",
f" Paths with snapshot metadata: {self.stats['snapshot_paths']}",
f" Paths with deletion tombstones: {self.stats['tombstone_paths']}",
f" Files recovered: {self.stats['files_recovered']}",
f" Exact checkpoint recoveries: {self.stats['exact_recoveries']}",
f" Write checkpoint recoveries: {self.stats['write_recoveries']}",
"",
]
if self.warnings:
report_lines.append("Warnings:")
report_lines.extend(f" - {warning}" for warning in self.warnings)
report_lines.append("")
if saved_files:
report_lines.extend(["Recovered Files:", ""])
for item in saved_files:
version = (
f" v{item['version']}" if isinstance(item["version"], int) else ""
)
report_lines.extend(
[
f"OK {item['file']}",
f" Original: {item['original_path']}",
f" Source: {item['source']}{version} ({item['source_path']})",
f" Fidelity: {item['fidelity']}",
]
)
if item["later_state"]:
report_lines.append(f" Later state: {item['later_state']}")
report_lines.extend(
[
f" Captured: {item['timestamp']}",
f" Size: {item['size']:,} bytes",
f" Lines: {item['lines']:,}",
f" SHA-256: {item['sha256']}",
f" Saved to: {item['output_path']}",
"",
]
)
else:
report_lines.extend(["No files matched the requested recovery scope.", ""])
report_lines.extend(["=" * 60, ""])
return "\n".join(report_lines)
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Recover exact file-history checkpoints when available, otherwise "
"recover explicitly labeled Write-tool checkpoints"
)
)
parser.add_argument("session_file", type=Path, help="Claude Code session JSONL")
parser.add_argument(
"-o",
"--output",
type=Path,
help="Output directory (default: ./recovered_content)",
)
parser.add_argument(
"-k",
"--keywords",
nargs="+",
help="Recover paths matching any keyword",
)
parser.add_argument(
"--file-history-root",
action="append",
type=Path,
default=[],
metavar="DIR",
help=(
"Additional file-history root containing <session-id>/ directories "
"(repeatable)"
),
)
parser.add_argument(
"--write-only",
action="store_true",
help=(
"Ignore file-history metadata and explicitly recover lower-fidelity "
"Write checkpoints"
),
)
parser.add_argument(
"--show-edits",
action="store_true",
help="List the five latest Edit operations",
)
args = parser.parse_args()
if not args.session_file.is_file():
print(f"Error: Session file not found: {args.session_file}", file=sys.stderr)
return 1
recovery = SessionContentRecovery(
args.session_file,
args.output,
args.file_history_root,
args.write_only,
)
print(f"Analyzing session: {args.session_file}")
print(f"Output directory: {recovery.output_dir}\n")
try:
write_calls = recovery.extract_write_calls()
print(f"Write calls: {len(write_calls)}")
print(f"Paths with file-history metadata: {recovery.stats['snapshot_paths']}")
if args.write_only:
print("Recovery mode: explicit Write-only checkpoint mode")
else:
print("Recovery mode: exact file-history checkpoint preferred")
if args.keywords:
print(f"Path filters: {', '.join(args.keywords)}")
print()
saved = recovery.recover_files(args.keywords)
if args.show_edits:
edits = recovery.extract_edit_calls()
print(f"Edit calls: {recovery.stats['edit_calls']}")
for edit in edits:
print(f" - {Path(str(edit['file_path'])).name} (line {edit['line']})")
print()
report = recovery.generate_report(saved)
print(report)
report_file = recovery.output_dir / "recovery_report.txt"
_atomic_write(report_file, report.encode("utf-8"))
except (RecoveryError, OSError) as error:
print(f"Error: {error}", file=sys.stderr)
return 2
print(f"Report saved to: {report_file}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
SKILL.md
---
name: claude-code-history-files-finder
description: >-
Searches and recovers Claude Code JSONL history across all active config homes
and archives registered in ~/.claude/history-sources.json. Use --all-projects
when the project is unknown, --codex to include Codex rollout search, and
--kimi to include Kimi CLI (kimi-code) session search. Uses internal
timestamps and searches messages, thinking, tool inputs/results, queues,
attachments, summaries, titles, and file-history paths. Recovers exact
captured bytes from Claude file-history snapshots, including post-Write edits
and binary files; otherwise labels Write checkpoints as lower fidelity. Use
for keyword/date-bounded history search, prior-conversation forensics,
deleted-file recovery, vanished ~/.claude/jobs artifacts, tool/file-operation
analysis, or requests mentioning session history, find in history, previous
conversation, or .claude/projects. For a recent Claude+Codex+Kimi CLI
inventory, use local-conversation-history instead.
---
# Claude Code History Files Finder
Search and recover content from Claude Code session history stored in active
homes and explicitly registered long-term archives.
## Capabilities
- Recover exact captured bytes for deleted or lost files from file-history
snapshots, including files changed after their original Write call
- Search for specific code or content across conversation history
- Analyze file modifications across past sessions
- Track tool usage and file operations over time
- Find sessions containing specific keywords or topics
## Completeness invariant
A normal history search must cover both source classes:
1. auto-discovered active homes (`~/.claude`, profile homes, and the current
`CLAUDE_CONFIG_DIR`), and
2. every archive registered in `~/.claude/history-sources.json`.
Do not conclude that a session, topic, file, or action is absent unless the
command output confirms that the registered archives were searched. A required
archive that is unavailable is a hard configuration error. `--home` and
`--main-only` are exact diagnostic scopes that intentionally bypass the archive
registry; results from either flag cannot support a whole-history absence claim.
A complete source set is necessary but not sufficient — three more failure
modes produce a false "not found" even with every source covered, and each has
a dedicated widening (the script prints these automatically on zero matches):
1. **Wrong project guess.** You searched one project, the conversation lived
in another. Widening: `--all-projects` sweeps every project in one pass.
2. **Wrong tool.** The conversation happened in Codex or Kimi CLI, whose
rollouts/wire logs are separate stores the Claude registry never covers.
Widening: `--codex` and `--kimi` (both opt-in; pass both when the project
was developed with both tools).
3. **Wording drift.** A remembered quote differs from the real wording in
punctuation or a few words, so the exact phrase misses. Widening: retry
shorter distinctive substrings.
One trap pairs with all three: **the current session always matches the
phrase you just typed** (the skill args, your commands, and this reasoning all
land in its records). A top hit whose range starts a few minutes ago is
almost certainly this session — confirm with the internal range, then rerun
with `--exclude-session <id>` to see the real results.
## Session File Locations
Each Claude history root stores sessions at
`<history-root>/projects/<encoded-project-path>/<session-id>.jsonl`. Active roots
are discovered automatically. Durable archive roots are configured once in
`~/.claude/history-sources.json` and then included by default.
Claude may also keep checkpoint payloads at
`<history-root>/file-history/<session-id>/<opaque-backup-name>`. The JSONL's
`file-history-snapshot.snapshot.trackedFileBackups` map connects each original
path to its opaque backup name and version. This companion store is separate
from `projects/`: copying only a JSONL into a long-term archive does not prove
its checkpoint bytes were copied too. The format is an observed Claude Code
runtime detail rather than a documented stable API, so the bundled recovery
parser validates the selected mapping, version/name agreement, path containment,
and byte identity, then fails visibly when those facts disagree.
**The directory name is the project's ABSOLUTE working-directory path with every `/` replaced by `-` — never the basename.** For example `/Users/<name>/Desktop/my-app` becomes `-Users-<name>-Desktop-my-app`, so a bare `my-app` cannot match a directory directly.
**Before concluding that a project has no history, run the bundled command with
its default source set. Do not infer absence from a failed `ls`:**
```bash
python3 scripts/analyze_sessions.py list /path/to/project
python3 scripts/analyze_sessions.py search /path/to/project '<keyword>'
```
A `ls <basename>` that returns nothing means the lookup used the wrong name, NOT
that history is absent. The bundled `analyze_sessions.py` expands `~`, resolves
an absolute path, falls back to an unambiguous basename reverse lookup, and
searches every configured source. Prefer passing it the full absolute project
path; `~`, relative paths, and bare names are also accepted.
Note: sessions run from **Claude Desktop's cowork / built-in Claude Code mode** also land here (Desktop runs a bundled CLI); only Desktop's *native* chat lives elsewhere (a LevelDB store, not JSONL). So "it ran inside Desktop" does not mean it is missing from `~/.claude/projects/`.
### Active profiles and long-term archives — searched together by default
`~/.claude` is only the *default* home. Anyone who runs Claude Code against **third-party models through per-model profiles** (each profile is its own `CLAUDE_CONFIG_DIR`) accumulates **parallel history that never touches `~/.claude`**:
- `~/.claude-profiles/<name>/projects/…` — one per profile (e.g. a `kimi`, `deepseek`, `glm`, `step` profile)
- `~/.claude-<name>/projects/…` — occasional sibling homes
- whatever `CLAUDE_CONFIG_DIR` points at in the current shell
Long-term archives are a second independent source class. Active directories can
retain only recent sessions, while an archive keeps older JSONL files after they
disappear from the active tree. A search limited to active homes can therefore
produce the same false negative as a main-home-only search.
`analyze_sessions.py` handles both classes: **`list` and `search` auto-discover
every active home and load the archive registry**, de-duplicate sessions by ID,
union the internal range across copies, and retain every source label as
provenance. Keyword search streams every physical copy and de-duplicates
identical records, so an archive-only record cannot disappear merely because a
newer active copy has the same session ID. Scope it only for a deliberate
diagnostic:
```bash
# default: active homes + registered archives
scripts/analyze_sessions.py search /path/to/project keyword
# exact diagnostic scope; not a completeness check
scripts/analyze_sessions.py search /path/to/project keyword --main-only
# exact diagnostic scope (repeatable)
scripts/analyze_sessions.py search /path/to/project keyword --home ~/.claude-profiles/kimi
# test a non-default source registry
scripts/analyze_sessions.py search /path/to/project keyword \
--history-sources /path/to/history-sources.json
```
**Do not use an ad hoc raw grep to prove absence.** It must independently parse
the registry, cover every active root, search non-message event payloads, and
apply dates to internal record timestamps; the bundled script already does so.
For detailed JSONL structure and extraction patterns, see `references/session_file_format.md`.
## Core Operations
### 1. List Sessions for a Project
Find all session files for a specific project:
```bash
python3 scripts/analyze_sessions.py list /path/to/project
```
Shows sessions ordered by their maximum internal JSONL timestamp, with the full
internal range, size, path, and source provenance. File mtime is never used.
Optional: `--limit N` to show only N sessions (default: 10), and `--from-date`
or `--to-date` to keep sessions whose internal range overlaps the requested
window. `--all-projects` lists every project (grouped by encoded project
name); `--exclude-session <id>` (repeatable) skips sessions.
### 2. Search Sessions for Keywords
Locate sessions containing specific content:
```bash
python3 scripts/analyze_sessions.py search /path/to/project keyword1 keyword2
```
Returns sessions ranked by keyword frequency with:
- Total mention count
- Per-keyword breakdown
- Session and matching-record internal time ranges
- Matching field types, session provenance, and match provenance
- Primary matching path plus any other matching copies
Search covers messages, thinking text (not signatures), tool inputs/results,
queue-operation content, attachments, last prompts, system/summary content,
custom titles, and original paths in file-history snapshots. Optional:
`--case-sensitive` for exact casing; `--from-date` and
`--to-date` constrain matching records by their own internal timestamps, not by
session mtime. `--exclude-session <id>` (repeatable) drops sessions — pass the
current session's id whenever you search for a phrase you just typed, because
your own command makes this session match.
Date-only bounds cover the whole local calendar day. Datetime bounds must carry
`Z` or an explicit UTC offset. Records without a valid internal timestamp are
excluded with a visible note while a date filter is active; never substitute
file mtime after a migration or copy.
### 2a. Search when the project is unknown — `--all-projects`
The required project argument encodes a guess; when the guess is wrong, a
project-scoped search reports a false "not found". Drop the positional and
sweep every project instead (`list` accepts the same flag):
```bash
python3 scripts/analyze_sessions.py search --all-projects 'some phrase'
```
With `--all-projects`, every positional term is a keyword, so multi-keyword
search is valid: `search --all-projects keyword1 keyword2`. Without that flag,
the first positional is the project path and the remaining terms are keywords.
Expected output: one pass over every project's sessions across all sources,
with a `Project:` line naming the encoded project dir on each hit. This is a
full-history sweep — expect minutes, not seconds, on a large tree.
### 2b. Include Codex history — `--codex`
Claude Code is not the only tool with history. Codex keeps rollouts at
`<codex-home>/sessions/<YYYY>/<MM>/<DD>/rollout-*.jsonl` plus
`archived_sessions/` (codex home = `--codex-home`, `$CODEX_HOME`, or
`~/.codex`). Their schema differs from Claude's, so the default search skips
them entirely; `--codex` adds a rollout pass:
```bash
python3 scripts/analyze_sessions.py search /path/to/project 'some phrase' --codex
```
Codex hits print in their own section (📦) with session id, cwd, internal
ranges, mention counts, and match fields. A project positional filters
rollouts by their `session_meta` cwd (recursive match); with `--all-projects`
every rollout is searched. `event_msg` message mirrors are strict duplicates
of `response_item` message text and are counted once. Rollout record shapes
are documented in `references/session_file_format.md`.
`--codex` widens **search only**. Codex rollouts do not carry Claude's
file-history mapping, so `recover_content.py` rejects a Codex rollout with a
clear boundary error instead of returning an empty, apparently successful
recovery.
### 2c. Include Kimi CLI history — `--kimi`
Kimi CLI (kimi-code) keeps sessions at
`<kimi-home>/sessions/wd_<workspace>_<hash>/session_<uuid>/agents/<agent>/wire.jsonl`
with a per-session `state.json` holding id/cwd/title and millisecond
created/updated bounds (kimi home = `--kimi-home`, `$KIMI_HOME`, or
`~/.kimi-code`). The wire schema differs from both Claude's and Codex's, so
the default search skips it entirely; `--kimi` adds a wire pass:
```bash
python3 scripts/analyze_sessions.py search /path/to/project 'some phrase' --kimi
```
Kimi hits print in their own section (🌙) with session id, title, cwd,
internal ranges, mention counts, and match fields. A project positional
filters sessions by their `state.json` cwd (recursive match); with
`--all-projects` every session is searched. Subagent wires
(`agents/agent-N/`) are runs of the same conversation, so matches aggregate
at session level with the agent name prefixed into each match field (e.g.
`main:message`, `agent-0:tool_input`). Searchable coverage is the
conversation itself — prompts (`turn.prompt` / `turn.steer`, indexed for any
`origin` — search deliberately errs wide), appended messages, assistant
content parts, tool calls, and tool results. Static
boilerplate (config/profile system prompts, tool snapshots, usage/token
metrics) is deliberately not indexed: a keyword that only appears in a shared
system prompt would match every session, and "not found" is the answer this
tool is trusted to give about conversation content.
`--kimi` widens **search only**, exactly like `--codex`. Kimi wire logs do
not carry Claude's file-history mapping, so file recovery remains Claude Code
JSONL only; do not pass a Kimi wire path to `recover_content.py`. `--codex`
and `--kimi` compose — a project developed with both tools needs both flags
in the same command.
The zero-match hint printed by the script already suggests whichever of
`--all-projects` / `--codex` / `--kimi` / shorter substrings was not yet
applied — read stderr before concluding anything is absent.
### 3. Recover Deleted Content
Recover files from the selected session:
```bash
python3 scripts/recover_content.py <session-path-from-search>
```
For each path, the default mode unions every known copy of the same session ID,
selects the newest valid file-history checkpoint, and restores its exact bytes
from active or registered-archive companion stores. That captures later Edit or
shell-driven changes and can recover binary files or files without a Write tool
call. A later `backupFileName: null` is a deletion tombstone: the last available
checkpoint is recovered with the later deletion stated in the report. If a path
has no usable snapshot checkpoint, the script recovers the latest Write call and
labels it as lower fidelity in `recovery_report.txt`. A Write whose matching
`tool_result` explicitly has `is_error: true` is skipped; an attempted write is
not a checkpoint. Original directory structure is preserved under
`./recovered_content/`.
**Filtering by keywords**:
```bash
python3 scripts/recover_content.py <session-path-from-search> \
-k ModelLoading FRONTEND deleted
```
Recovers only files matching any keyword in their path.
**Custom output directory**:
```bash
python3 scripts/recover_content.py <session-path-from-search> -o ./my_recovery/
```
Registered archive roots and same-ID JSONL copies are included automatically.
If an unregistered companion checkpoint store lives elsewhere, add the root
that directly contains `<session-id>/` directories:
```bash
python3 scripts/recover_content.py <session-path-from-search> \
--file-history-root /path/to/file-history \
-o ./my_recovery/
```
Snapshot metadata without its referenced backup is a fidelity error: recovery
aborts before writing any selected files instead of silently substituting stale
Write content. Use `--write-only` only when the user explicitly accepts that
later Edit or shell changes may be absent.
### 4. Analyze Session Statistics
Get detailed session metrics:
```bash
python3 scripts/analyze_sessions.py stats /path/to/session.jsonl
```
Reports:
- Message counts (user/assistant)
- Tool usage breakdown
- File operation counts (Write/Edit/Read)
Optional: `--show-files` to list all file operations.
### 5. Extract Verbatim User Messages
Produce a reading page of what the user actually typed — every user message
across all homes and archives, with harness noise routed to appendices:
```bash
python3 scripts/extract_user_messages.py --days 7
```
Writes `~/.claude-flow-viewer/user-words.html` (primary) and `.md` by default —
a persistent path; avoid `/tmp` (the OS purges it). Pass an `OUT_BASE`
positional to write elsewhere. Useful flags: `--group-by project|day`
(default `project`), `--min-dup N` (boilerplate frequency threshold),
`--home <path>` (exact scope, repeatable).
The hard part of this task is not parsing, it is *authorship*: a `user` record
is not necessarily user-authored. The extractor implements the five
contamination classes in `references/session_file_format.md` — command
envelopes, hook/loop-injected boilerplate (frequency-detected, both standalone
and tail-appended shapes), `[Image #N]` placeholders, whole-document pastes,
and agent-voiced re-injection (content-matched against earlier assistant
texts) — and recovers mid-work input from `attachment.queued_command` records,
de-duplicated against later-delivered user records. Read that reference section
before modifying the filters.
### 6. Triage Session Endings (Crash Recovery / Backlog Audit)
Classify how sessions in a time window or project ended, with the full last
assistant message printed for each — the tool for "which sessions did a
reboot/crash cut off" or "which older sessions are still waiting on a reply,
not actually done":
```bash
python3 scripts/analyze_sessions.py triage --all-projects \
--from-date 2026-08-05T12:30:00+00:00 --to-date 2026-08-05T13:05:00+00:00
```
Reports each in-scope session's session ID (always full, never truncated —
copy it directly into any follow-up report), cwd, and one of five structural
`kind`s: `interrupted_explicit` (the session's last relevant record is an
explicit interruption marker), `net_error` (the last assistant turn died on
an API/transport error), `done` (the last assistant turn produced real text,
and it didn't start with an API error), `empty` (no assistant turns at all),
or `stuck_no_result` — the catch-all for every other shape, because they all
mean the same thing for triage purposes: **the final turn produced no
textual reply, so the model was still working when the file stopped.** That
covers a tool call still waiting on its result, a tool call whose result
already landed but no further assistant turn followed it (the harness likely
stopped between the result landing and the model's next turn being
captured), and a final turn that is thinking-only or otherwise empty.
**`done` is a structural label, not a claim that nothing is outstanding.** A
session can end in a clean `text` block precisely because the assistant
surfaced a finding, a decision, or a question and never got a reply — telling
"fully wrapped up" from "waiting on you" requires reading the printed
`last_assistant_text`, which is why this command prints up to 4000 characters
of it by default (`--tail-chars 0` for no cap) rather than a truncated title.
See "Detect Session
Interruption" in `references/session_file_format.md` for the full reasoning,
including why this is a genuinely different axis from the `kind` field.
Useful flags: `--kind KIND` (repeatable, restrict to specific kinds — e.g.
`--kind stuck_no_result --kind interrupted_explicit` for "only the ones a
crash could plausibly explain"); `--exclude-title-prefix TEXT` (repeatable —
exclude sessions whose opening prompt starts with TEXT, for a project's own
automation convention such as a code-review hook that always opens with the
same fixed prompt; these otherwise dominate a triage pass because they end
in a routine structured tool call, not an interruption); `--tail-chars N`
(cap the printed last-assistant text; default 4000, 0 = unlimited). Shares
`--from-date`/`--to-date`/`--home`/`--main-only`/`--history-sources` with
`list`, and `--all-projects`/project-path/`--exclude-session` with both
`list` and `search`.
## Workflow Examples
For detailed workflow examples including file recovery, tracking file evolution, and batch operations, see `references/workflow_examples.md`.
## Recovery Best Practices
### Deduplication
`recover_content.py` unions same-ID session copies, keeps the highest
file-history version for each original path, and uses checkpoint timestamps for
ties. A later deletion tombstone does not erase an earlier recoverable backup;
it changes the reported state. For paths with no usable checkpoint, recovery
keeps the latest internally timestamped Write call. Physical JSONL line order
across copies is not treated as sufficient time evidence, and an explicitly
failed Write tool result excludes that attempted Write from recovery.
### Keyword Selection
Choose distinctive keywords that appear in:
- File names or paths
- Function/class names
- Unique strings in code
- Error messages or comments
### Output Organization
Create descriptive output directories:
```bash
# Bad
python3 scripts/recover_content.py session.jsonl -o ./output/
# Good
python3 scripts/recover_content.py session.jsonl -o ./recovered_deleted_docs/
python3 scripts/recover_content.py session.jsonl -o ./feature_xy_history/
```
### Verification
After recovery, always verify content:
```bash
# Check directory structure (files preserved in subdirectories)
find ./recovered_content/ -type f
# Read recovery report (shows full output paths)
cat ./recovered_content/recovery_report.txt
# Spot-check content and compare the report's SHA-256 with the source backup
head -20 ./recovered_content/src/components/ImportantFile.jsx
```
Treat `Source: file-history` plus its SHA-256 as exact captured-checkpoint
evidence. Treat `Source: Write` as a recoverable checkpoint, not proof of the
file's final state.
## Limitations
### What Can Be Recovered
✅ Exact bytes referenced by available file-history snapshots
✅ Binary files present in the companion file-history store
✅ Files changed by Edit or shell commands once a later checkpoint captured them
✅ Files written using Write when no snapshot metadata exists (lower fidelity)
✅ Text explicitly present in messages or tool results (manual extraction)
### What Cannot Be Recovered
❌ Files never written to disk (only discussed)
❌ Files deleted before session start
❌ Snapshot payloads that were deleted or not copied with an archived JSONL
❌ External tool outputs not captured in session
Edit/Read records can reveal a path and Edit delta, but they are not themselves
a full-file recovery source.
### File Versions
- A file-history backup is an exact captured checkpoint, not a guarantee that
no uncheckpointed filesystem change happened afterward.
- Without a file-history entry, Write recovery cannot reconstruct later Edit or
shell changes; Edit records contain deltas rather than the full resulting file.
- The file-history JSONL/store contract is runtime-observed and may evolve;
malformed or conflicting metadata must fail visibly rather than be guessed.
## Troubleshooting
### No Sessions Found
```bash
# Re-run with the full absolute project path and the default source set.
python3 scripts/analyze_sessions.py list /absolute/path/to/project
# Inspect a custom registry only when diagnosing its configuration.
python3 scripts/analyze_sessions.py list /absolute/path/to/project \
--history-sources /path/to/history-sources.json
```
**"Not found" is often a wrong project identity or an incomplete source set.**
Confirm that output says `Searched N source(s)` and includes both the expected
`active:<label>` and `archive:<label>` entries. If `--main-only` or `--home` was
used, re-run without it. A missing required archive must be repaired or its
registry entry deliberately changed; do not silently ignore the error and claim
the session does not exist. If the source set is confirmed complete, work the
widening ladder from the Completeness invariant section: `--all-projects` →
`--codex` → `--kimi` → shorter substrings, and `--exclude-session` the current
session.
### Empty Recovery
Possible causes:
- Keywords don't match file paths in session
- Session predates file creation
- The path was never captured by either file-history or Write
Solutions:
- Try `--show-edits` flag to see Edit operations
- Broaden keyword search
- Search adjacent sessions
- If an exact-backup error names a missing companion store, locate it and pass
`--file-history-root`; do not claim the stale Write checkpoint is final
### Large Session Files
For sessions >100MB:
- Search streams JSONL line by line instead of loading whole sessions.
- Recovery copies exact backup bytes in chunks and retains only five lightweight
Edit summaries, never full Edit old/new payloads.
- Recovery still retains valid Write payloads and record fingerprints needed for
copy union, so memory is not constant. Use `-k` to limit recovery scope and
expect runtime to scale with every discovered physical copy.
## Security & Privacy
### Before Sharing Recovered Content
Session files may contain:
- Absolute paths with usernames
- API keys or credentials
- Company-specific information
Always sanitize before sharing:
```bash
# Read-only audit; review every hit before creating a separate redacted copy.
rg -n --hidden -S \
'(api[_-]?key|password|token|secret|/Users/[^/]+/|/home/[^/]+/)' \
recovered_content/
```
`recovery_report.txt` is sensitive too: it records requested session copies,
original absolute paths, checkpoint locations, and output paths. Audit and
redact the report together with the recovered files; do not share it by default.
### Safe Storage
Recovered content inherits sensitivity from original sessions. Store securely and follow organizational policies for handling session data.
## Next Step: Resume Interrupted Work
After finding relevant session history, suggest continuing the work:
```
Found [N] relevant sessions with recoverable context.
Options:
A) Resume work — run /daymade-claude-code:continue-claude-work to pick up where you left off (Recommended)
B) Just show me the content — I'll decide what to do with it
```
## Maintainer verification
In the source repository, `daymade-claude-code/_conversation_core/` is the code
SSOT shared by this skill, `local-conversation-history`,
`continue-claude-work`, and `continue-codex-work`. `sync_core.py` bundles that
package into each skill's `scripts/_core/`; never edit a bundled copy directly.
After shared-code changes, synchronize and verify all four bundles, then run the
finder's isolated fixtures:
```text
uv run python ../sync_core.py sync
uv run python ../sync_core.py check
python -m unittest discover -s tests -p "test_*.py"
```
tests/test_analyze_sessions.py
#!/usr/bin/env python3
"""Regression tests for archive-aware Claude session search."""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
SKILL_DIR = Path(__file__).resolve().parents[1]
SCRIPT = SKILL_DIR / "scripts" / "analyze_sessions.py"
sys.path.insert(0, str(SKILL_DIR / "scripts"))
from _core.text import keywords_are_raw_byte_safe # noqa: E402
def _load_analyze_module():
"""Import analyze_sessions.py in-process to unit-test its helpers.
The rest of this suite drives the CLI as a subprocess, which cannot reach
module-level functions.
"""
import importlib.util
spec = importlib.util.spec_from_file_location("analyze_sessions_under_test", SCRIPT)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def write_jsonl(path: Path, records: list[object]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8") as handle:
for record in records:
handle.write(json.dumps(record, ensure_ascii=False) + "\n")
def project_dir(home: Path, workspace: Path) -> Path:
encoded = str(workspace.resolve()).replace("/", "-")
result = home / "projects" / encoded
result.mkdir(parents=True, exist_ok=True)
return result
def user_record(session_id: str, workspace: Path, text: str, timestamp: str) -> dict:
return {
"type": "user",
"sessionId": session_id,
"cwd": str(workspace),
"timestamp": timestamp,
"message": {"role": "user", "content": text},
}
class SessionAnalyzerTests(unittest.TestCase):
def setUp(self) -> None:
self.temp_dir = tempfile.TemporaryDirectory()
self.root = Path(self.temp_dir.name)
self.user_home = self.root / "user-home"
self.active_home = self.user_home / ".claude"
self.archive_home = self.root / "conversation-archive"
self.workspace = self.root / "workspaces" / "demo-project"
self.workspace.mkdir(parents=True)
project_dir(self.active_home, self.workspace)
project_dir(self.archive_home, self.workspace)
self.manifest = self.active_home / "history-sources.json"
self.manifest.write_text(
json.dumps(
{
"version": 1,
"sources": [
{
"provider": "claude",
"kind": "archive",
"label": "full-backup",
"home": str(self.archive_home),
"required": True,
}
],
}
),
encoding="utf-8",
)
def tearDown(self) -> None:
self.temp_dir.cleanup()
def run_cli(
self, *arguments: str, check: bool = True
) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(SCRIPT), *arguments],
text=True,
encoding="utf-8",
capture_output=True,
check=check,
env={**os.environ, "HOME": str(self.user_home)},
)
def seed_structured_events(self) -> tuple[str, str]:
active_id = "11111111-1111-4111-8111-111111111111"
archive_id = "22222222-2222-4222-8222-222222222222"
active_file = project_dir(self.active_home, self.workspace) / f"{active_id}.jsonl"
archive_file = project_dir(self.archive_home, self.workspace) / f"{archive_id}.jsonl"
write_jsonl(
active_file,
[
user_record(
active_id,
self.workspace,
"Active March session",
"2026-03-20T10:00:00Z",
)
],
)
write_jsonl(
archive_file,
[
user_record(
archive_id,
self.workspace,
"queued-topic before the requested window",
"2026-01-10T08:00:00Z",
),
{
"type": "assistant",
"sessionId": archive_id,
"cwd": str(self.workspace),
"timestamp": "2026-04-17T14:30:05Z",
"message": {
"role": "assistant",
"content": [
{
"type": "thinking",
"thinking": "reasoning-marker",
"signature": "signature-only-secret",
},
{
"type": "tool_use",
"name": "Bash",
"input": {
"command": "uv run python build_slides.py",
"description": "Install python-pptx",
},
},
],
},
},
{
"type": "user",
"sessionId": archive_id,
"cwd": str(self.workspace),
"timestamp": "2026-04-18T09:00:00Z",
"message": {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "tool-1",
"content": [
{"type": "text", "text": "topic-from-tool-result"}
],
}
],
},
},
{
"type": "queue-operation",
"operation": "enqueue",
"sessionId": archive_id,
"timestamp": "2026-04-18T10:00:00Z",
"content": "queued-topic",
},
{
"type": "attachment",
"sessionId": archive_id,
"cwd": str(self.workspace),
"timestamp": "2026-04-18T12:36:48Z",
"attachment": {
"type": "file",
"path": "brief.txt",
"content": {"notes": "attachment-topic"},
},
},
],
)
# Make file mtimes contradict their internal timestamps.
os.utime(archive_file, (1, 1))
os.utime(active_file, (2_000_000_000, 2_000_000_000))
return active_id, archive_id
def test_list_sorts_by_internal_timestamp_and_labels_archive_source(self) -> None:
active_id, archive_id = self.seed_structured_events()
completed = self.run_cli(
"list",
str(self.workspace),
"--history-sources",
str(self.manifest),
)
self.assertLess(completed.stdout.index(archive_id), completed.stdout.index(active_id))
self.assertIn("Internal range:", completed.stdout)
self.assertIn("archive:full-backup", completed.stdout)
self.assertNotIn("Modified:", completed.stdout)
def test_search_covers_structured_event_fields_and_filters_matching_records(self) -> None:
_, archive_id = self.seed_structured_events()
completed = self.run_cli(
"search",
str(self.workspace),
"topic-from-tool-result",
"queued-topic",
"attachment-topic",
"python-pptx",
"reasoning-marker",
"--from-date",
"2026-04-01",
"--to-date",
"2026-04-30",
"--history-sources",
str(self.manifest),
)
self.assertIn(archive_id, completed.stdout)
self.assertIn("queued-topic(1)", completed.stdout)
self.assertIn("Match range:", completed.stdout)
self.assertIn("tool_result", completed.stdout)
self.assertIn("tool_input:Bash", completed.stdout)
self.assertIn("thinking", completed.stdout)
self.assertIn("attachment", completed.stdout)
self.assertIn("queue-operation", completed.stdout)
signature_only = self.run_cli(
"search",
str(self.workspace),
"signature-only-secret",
"--history-sources",
str(self.manifest),
)
self.assertIn("No matches found.", signature_only.stdout)
def test_prefilter_and_no_prefilter_agree_byte_for_byte(self) -> None:
"""The 2026-08 pre-filter (rg/grep file-level + in-process line-level)
must be a pure speedup: identical stdout with and without it, on a
mix of matching, non-matching, and mixed-case sessions."""
matching_id = "44444444-4444-4444-8444-444444444444"
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{matching_id}.jsonl",
[
user_record(matching_id, self.workspace, "irrelevant filler line", "2026-04-01T00:00:00Z"),
user_record(matching_id, self.workspace, "contains the RareToken here", "2026-04-01T00:00:01Z"),
user_record(matching_id, self.workspace, "more filler after the hit", "2026-04-01T00:00:02Z"),
],
)
no_match_id = "55555555-5555-4555-8555-555555555555"
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{no_match_id}.jsonl",
[user_record(no_match_id, self.workspace, "nothing of interest", "2026-04-02T00:00:00Z")],
)
default_run = self.run_cli(
"search", str(self.workspace), "raretoken",
"--history-sources", str(self.manifest),
)
no_prefilter_run = self.run_cli(
"search", str(self.workspace), "raretoken", "--no-prefilter",
"--history-sources", str(self.manifest),
)
self.assertEqual(default_run.stdout, no_prefilter_run.stdout)
self.assertIn(matching_id, default_run.stdout)
self.assertNotIn(no_match_id, default_run.stdout)
def test_prefilter_disabled_under_date_window_keeps_untimed_count_exact(self) -> None:
"""The pre-filter is gated off automatically whenever a date window is
active (see search_sessions's use_prefilter docstring) because
excluded_untimed_records counts every untimed record in a session
regardless of keyword match, and skipping non-matching records would
silently under-report it. Prove the count is identical either way,
not just that both runs happen to exit 0."""
session_id = "66666666-6666-4666-8666-666666666666"
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{session_id}.jsonl",
[
user_record(session_id, self.workspace, "has the target keyword", "2026-04-15T00:00:00Z"),
{ # untimed record, no keyword — must still be counted while a date window is active
"type": "user",
"sessionId": session_id,
"cwd": str(self.workspace),
"message": {"role": "user", "content": "untimed filler, no timestamp field at all"},
},
{ # a SECOND untimed record with no keyword — count must reach 2, not 1
"type": "user",
"sessionId": session_id,
"cwd": str(self.workspace),
"message": {"role": "user", "content": "another untimed filler"},
},
],
)
default_run = self.run_cli(
"search", str(self.workspace), "target keyword",
"--from-date", "2026-04-01", "--to-date", "2026-04-30",
"--history-sources", str(self.manifest),
)
no_prefilter_run = self.run_cli(
"search", str(self.workspace), "target keyword", "--no-prefilter",
"--from-date", "2026-04-01", "--to-date", "2026-04-30",
"--history-sources", str(self.manifest),
)
self.assertEqual(default_run.stdout, no_prefilter_run.stdout)
self.assertIn("excluded 2 record(s) without an internal timestamp", default_run.stdout)
def test_repeated_copy_sizes_flags_only_actual_duplicates(self) -> None:
"""Only sizes that repeat are worth hashing — a unique size cannot
have a byte-identical twin, so single-copy sessions must pay nothing
for the duplicate-copy skip."""
module = _load_analyze_module()
base = project_dir(self.active_home, self.workspace)
same_a, same_b, unique = base / "a.jsonl", base / "b.jsonl", base / "c.jsonl"
same_a.write_bytes(b"x" * 100)
same_b.write_bytes(b"y" * 100) # same size, different bytes
unique.write_bytes(b"z" * 55)
flagged = module.repeated_copy_sizes(
[{"path": same_a}, {"path": same_b}, {"path": unique}]
)
self.assertEqual(flagged, {100})
# Same size is not same content: the digest must tell them apart, or
# the skip would drop a genuinely different copy's records.
self.assertNotEqual(
module.file_content_digest(same_a), module.file_content_digest(same_b)
)
self.assertEqual(
module.file_content_digest(same_a), module.file_content_digest(same_a)
)
# Unreadable input degrades to "no information", never to "identical".
self.assertIsNone(module.file_content_digest(base / "missing.jsonl"))
def test_byte_identical_copies_keep_output_identical(self) -> None:
"""A registered archive re-snapshots the same .jsonl every run, so one
session routinely carries many byte-identical copies (measured on a
real project: 46 of 60 sampled multi-copy sessions had 13 copies and
exactly 1 distinct content, and that redundancy was 85% of the
corpus's bytes). Parsing each copy costs Nx for one session's worth of
records, which _record_identity then dedupes straight back down.
Reusing a byte-identical copy's conclusion must be invisible in the
output: the mention count must not double, and the skipped copy must
still be reported with its own source label — that label is the one
piece of state a reused conclusion does not already carry."""
session_id = "77777777-7777-4777-8777-777777777777"
records = [
user_record(
session_id, self.workspace, "has the target keyword", "2026-04-15T00:00:00Z"
),
]
active_copy = project_dir(self.active_home, self.workspace) / f"{session_id}.jsonl"
archive_copy = project_dir(self.archive_home, self.workspace) / f"{session_id}.jsonl"
write_jsonl(active_copy, records)
write_jsonl(archive_copy, records)
self.assertEqual(
active_copy.read_bytes(),
archive_copy.read_bytes(),
"fixture must produce byte-identical copies for this test to exercise the skip",
)
result = self.run_cli(
"search", str(self.workspace), "target keyword",
"--history-sources", str(self.manifest),
)
# Counted once, not once per copy.
self.assertIn("Total mentions: 1", result.stdout)
# The skipped copy must still contribute its own source label. Anchor
# on the whole line: a bare "full-backup" also appears in the
# "Searching N session(s) across ..." banner, so it would pass even
# with the label handling removed entirely.
self.assertIn(
"Match sources: active:main, archive:full-backup", result.stdout
)
# ...and still be listed as a copy.
self.assertIn("Other matching copies:", result.stdout)
self.assertIn("[archive:full-backup]", result.stdout)
def test_differing_copies_are_both_parsed(self) -> None:
"""The skip is keyed on a content hash, so copies that differ — an
archive snapshot taken before the session grew — must both be parsed
and their distinct records merged, not collapsed by size or name."""
session_id = "88888888-8888-4888-8888-888888888888"
shared = user_record(
session_id, self.workspace, "has the target keyword", "2026-04-15T00:00:00Z"
)
newer = user_record(
session_id, self.workspace, "another target keyword line", "2026-04-16T00:00:00Z"
)
write_jsonl(
project_dir(self.archive_home, self.workspace) / f"{session_id}.jsonl",
[shared],
)
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{session_id}.jsonl",
[shared, newer],
)
result = self.run_cli(
"search", str(self.workspace), "target keyword",
"--history-sources", str(self.manifest),
)
# Both records counted once each: the shared one is deduped across
# copies, the newer one only exists in the active copy.
self.assertIn("Total mentions: 2", result.stdout)
def test_signature_only_keyword_finds_no_match_with_prefilter_on(self) -> None:
"""The pre-filter's raw byte scan is a deliberate OVER-approximation —
it will say a file/line "could" match even when the keyword only
appears in a field the structured search excludes (signature/id/
tool_use_id — see _flatten_search_strings). Confirm that over-approval
never leaks into an actual false-positive match."""
session_id = "77777777-7777-4777-8777-777777777777"
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{session_id}.jsonl",
[
{
"type": "assistant",
"sessionId": session_id,
"cwd": str(self.workspace),
"timestamp": "2026-04-10T00:00:00Z",
"message": {
"role": "assistant",
"content": [
{
"type": "thinking",
"thinking": "ordinary reasoning text",
"signature": "excluded-only-secret-marker",
}
],
},
}
],
)
result = self.run_cli(
"search", str(self.workspace), "excluded-only-secret-marker",
"--history-sources", str(self.manifest),
)
self.assertIn("No matches found.", result.stdout)
def test_all_projects_hint_appears_for_bare_keyword_project_path(self) -> None:
"""Regression for a real trap: `search KEYWORD1 KEYWORD2` without
--all-projects treats KEYWORD1 as a project path (documented
argparse grammar), finds no project by that name, and used to print
only "No sessions found for project: KEYWORD1" — which reads as "your
keyword doesn't exist" rather than "you searched for a project by
that name". An agent hit exactly this during a real investigation."""
result = self.run_cli(
"search", "embedding", "classifier",
"--history-sources", str(self.manifest),
check=False,
)
self.assertEqual(result.returncode, 1)
self.assertIn("No sessions found for project: embedding", result.stdout)
self.assertIn("--all-projects", result.stderr)
def test_all_projects_hint_absent_for_a_real_existing_path(self) -> None:
"""The hint must not fire when project_path genuinely looks like (or
is) a path — false "did you mean --all-projects" noise on an honest
typo'd real path would be its own annoyance."""
result = self.run_cli(
"search", str(self.workspace), "no-such-keyword-at-all",
"--history-sources", str(self.manifest),
check=False,
)
self.assertNotIn("--all-projects", result.stderr)
def test_duplicate_session_unions_distinct_records_and_keeps_provenance(self) -> None:
session_id = "33333333-3333-4333-8333-333333333333"
timestamp = "2026-04-20T10:00:00Z"
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{session_id}.jsonl",
[user_record(session_id, self.workspace, "active-choice", timestamp)],
)
write_jsonl(
project_dir(self.archive_home, self.workspace) / f"{session_id}.jsonl",
[user_record(session_id, self.workspace, "archive-choice", timestamp)],
)
active_match = self.run_cli(
"search",
str(self.workspace),
"active-choice",
"--history-sources",
str(self.manifest),
)
self.assertIn(session_id, active_match.stdout)
self.assertIn("active:main, archive:full-backup", active_match.stdout)
archive_match = self.run_cli(
"search",
str(self.workspace),
"archive-choice",
"--history-sources",
str(self.manifest),
)
self.assertIn(session_id, archive_match.stdout)
self.assertIn("Match sources: archive:full-backup", archive_match.stdout)
self.assertIn(str(self.archive_home), archive_match.stdout)
combined = self.run_cli(
"search",
str(self.workspace),
"active-choice",
"archive-choice",
"--history-sources",
str(self.manifest),
)
self.assertIn("active-choice(1)", combined.stdout)
self.assertIn("archive-choice(1)", combined.stdout)
self.assertIn("Other matching copies:", combined.stdout)
def test_duplicate_identical_records_are_not_double_counted(self) -> None:
session_id = "55555555-5555-4555-8555-555555555555"
record = user_record(
session_id,
self.workspace,
"shared-marker",
"2026-04-20T10:00:00Z",
)
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{session_id}.jsonl",
[record, record],
)
write_jsonl(
project_dir(self.archive_home, self.workspace) / f"{session_id}.jsonl",
[record],
)
completed = self.run_cli(
"search",
str(self.workspace),
"shared-marker",
"--history-sources",
str(self.manifest),
)
# Identical records copied to another physical file count once, while
# two real occurrences inside one file remain two occurrences.
self.assertIn("shared-marker(2)", completed.stdout)
self.assertIn(
"Match sources: active:main, archive:full-backup",
completed.stdout,
)
def test_explicit_home_remains_an_exact_scope(self) -> None:
archive_id = "44444444-4444-4444-8444-444444444444"
write_jsonl(
project_dir(self.archive_home, self.workspace) / f"{archive_id}.jsonl",
[
user_record(
archive_id,
self.workspace,
"archive-only-marker",
"2026-04-20T10:00:00Z",
)
],
)
completed = self.run_cli(
"search",
str(self.workspace),
"archive-only-marker",
"--home",
str(self.active_home),
check=False,
)
self.assertNotEqual(completed.returncode, 0)
self.assertIn("No sessions found", completed.stdout)
def test_all_projects_aggregates_sessions_across_projects(self) -> None:
other_workspace = self.root / "workspaces" / "other-project"
other_workspace.mkdir(parents=True)
first_id = "66666666-6666-4666-8666-666666666666"
second_id = "77777777-7777-4777-8777-777777777777"
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{first_id}.jsonl",
[
user_record(
first_id,
self.workspace,
"cross-project-marker second-marker",
"2026-04-20T10:00:00Z",
)
],
)
write_jsonl(
project_dir(self.active_home, other_workspace) / f"{second_id}.jsonl",
[
user_record(
second_id,
other_workspace,
"cross-project-marker second-marker",
"2026-04-21T10:00:00Z",
)
],
)
completed = self.run_cli(
"search",
"--all-projects",
"cross-project-marker",
"second-marker",
"--home",
str(self.active_home),
"--home",
str(self.archive_home),
)
self.assertIn(first_id, completed.stdout)
self.assertIn(second_id, completed.stdout)
self.assertIn("Project:", completed.stdout)
self.assertIn("2 project(s)", completed.stdout)
self.assertIn("cross-project-marker(1), second-marker(1)", completed.stdout)
listed = self.run_cli(
"list",
"--all-projects",
"--home",
str(self.active_home),
"--home",
str(self.archive_home),
)
self.assertIn("across 2 project(s)", listed.stdout)
self.assertIn("== ", listed.stdout)
def test_project_scope_is_required_and_list_scope_is_exclusive(self) -> None:
neither = self.run_cli("search", "anything", check=False)
self.assertEqual(neither.returncode, 2)
both = self.run_cli(
"list",
str(self.workspace),
"--all-projects",
check=False,
)
self.assertEqual(both.returncode, 2)
def test_exclude_session_skips_the_current_session(self) -> None:
current_id = "88888888-8888-4888-8888-888888888888"
target_id = "99999999-9999-4999-8999-999999999999"
for session_id in (current_id, target_id):
write_jsonl(
project_dir(self.active_home, self.workspace)
/ f"{session_id}.jsonl",
[
user_record(
session_id,
self.workspace,
"self-match-marker",
"2026-04-20T10:00:00Z",
)
],
)
completed = self.run_cli(
"search",
str(self.workspace),
"self-match-marker",
"--exclude-session",
current_id,
"--history-sources",
str(self.manifest),
)
self.assertIn(target_id, completed.stdout)
self.assertNotIn(current_id, completed.stdout)
def test_zero_match_hint_points_at_unapplied_widenings(self) -> None:
write_jsonl(
project_dir(self.active_home, self.workspace)
/ "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa.jsonl",
[
user_record(
"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
self.workspace,
"unrelated content",
"2026-04-20T10:00:00Z",
)
],
)
completed = self.run_cli(
"search",
str(self.workspace),
"definitely-absent-marker",
"--history-sources",
str(self.manifest),
)
self.assertIn("No matches found.", completed.stdout)
self.assertIn("--all-projects", completed.stderr)
self.assertIn("--codex", completed.stderr)
self.assertIn("substrings", completed.stderr)
widened = self.run_cli(
"search",
"--all-projects",
"definitely-absent-marker",
"--codex",
"--codex-home",
str(self.root / "empty-codex"),
"--home",
str(self.active_home),
"--home",
str(self.archive_home),
)
self.assertNotIn("--all-projects", widened.stderr)
self.assertNotIn("--codex (", widened.stderr)
self.assertIn("substrings", widened.stderr)
def test_search_finds_snapshot_only_original_path_once_across_copies(self) -> None:
session_id = "abababab-abab-4bab-8bab-abababababab"
original_path = "/tmp/generated/snapshot-only.bin"
record = {
"type": "file-history-snapshot",
"snapshot": {
"timestamp": "2026-04-22T10:00:00Z",
"trackedFileBackups": {
original_path: {
"backupFileName": "snapshot@v2",
"version": 2,
"backupTime": "2026-04-22T10:00:00Z",
}
},
},
}
write_jsonl(
project_dir(self.active_home, self.workspace) / f"{session_id}.jsonl",
[record, record],
)
write_jsonl(
project_dir(self.archive_home, self.workspace) / f"{session_id}.jsonl",
[record],
)
completed = self.run_cli(
"search",
str(self.workspace),
"snapshot-only.bin",
"--from-date",
"2026-04-22",
"--to-date",
"2026-04-22",
"--history-sources",
str(self.manifest),
)
self.assertIn(session_id, completed.stdout)
self.assertIn("snapshot-only.bin(1)", completed.stdout)
self.assertIn("file_history_path", completed.stdout)
self.assertIn("active:main, archive:full-backup", completed.stdout)
def seed_codex_rollout(
self,
codex_home: Path,
session_id: str,
cwd: Path,
text: str,
*,
mirror: bool = True,
archived_copy: bool = False,
) -> Path:
timestamp = "2026-04-20T10:00:00Z"
records = [
{
"type": "session_meta",
"timestamp": timestamp,
"payload": {
"id": session_id,
"cwd": str(cwd),
"timestamp": timestamp,
},
},
{
"type": "response_item",
"timestamp": timestamp,
"payload": {
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": text}],
},
},
]
if mirror:
# event_msg mirrors of message text must not double-count.
records.append(
{
"type": "event_msg",
"timestamp": timestamp,
"payload": {"type": "user_message", "message": text},
}
)
rollout = (
codex_home
/ "sessions"
/ "2026"
/ "04"
/ "20"
/ f"rollout-2026-04-20T10-00-00-{session_id}.jsonl"
)
write_jsonl(rollout, records)
if archived_copy:
write_jsonl(
codex_home / "archived_sessions" / rollout.name, records
)
return rollout
def test_codex_search_finds_rollout_once_and_counts_mirror_once(self) -> None:
codex_home = self.root / "codex-home"
session_id = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"
self.seed_codex_rollout(
codex_home,
session_id,
self.workspace,
"codex-only-marker",
archived_copy=True,
)
completed = self.run_cli(
"search",
str(self.workspace),
"codex-only-marker",
"--codex",
"--codex-home",
str(codex_home),
"--history-sources",
str(self.manifest),
)
self.assertIn("Codex rollout matches", completed.stdout)
self.assertIn(session_id, completed.stdout)
# sessions/ + archived_sessions/ copies of one rollout dedupe to a
# single result, and the event_msg mirror adds no extra mention.
self.assertEqual(completed.stdout.count("📦"), 1)
self.assertIn("Total mentions: 1", completed.stdout)
self.assertIn("Match fields: message", completed.stdout)
def test_codex_search_filters_rollouts_by_project_cwd(self) -> None:
codex_home = self.root / "codex-home"
matching_id = "cccccccc-cccc-4ccc-8ccc-cccccccccccc"
other_id = "dddddddd-dddd-4ddd-8ddd-dddddddddddd"
self.seed_codex_rollout(
codex_home, matching_id, self.workspace, "codex-cwd-marker"
)
self.seed_codex_rollout(
codex_home,
other_id,
self.root / "elsewhere",
"codex-cwd-marker",
)
scoped = self.run_cli(
"search",
str(self.workspace),
"codex-cwd-marker",
"--codex",
"--codex-home",
str(codex_home),
"--history-sources",
str(self.manifest),
)
self.assertIn(matching_id, scoped.stdout)
self.assertNotIn(other_id, scoped.stdout)
swept = self.run_cli(
"search",
"--all-projects",
"codex-cwd-marker",
"--codex",
"--codex-home",
str(codex_home),
"--home",
str(self.active_home),
"--home",
str(self.archive_home),
)
self.assertIn(matching_id, swept.stdout)
self.assertIn(other_id, swept.stdout)
class RawBytePrefilterSafetyTests(unittest.TestCase):
"""The raw-byte pre-filter must never rule out a session a full parse
would have matched.
Every case here was a real divergence found by an A/B run of the CLI
against its own pre-pre-filter version — and every one of them slipped
past the 59 tests above, because those fixtures are written with
``ensure_ascii=False`` and searched with ASCII-only keywords. The shapes
below are exactly the ones that combination cannot reach.
"""
def test_non_ascii_keyword_is_not_raw_byte_safe(self) -> None:
# Two independent reasons, either one sufficient: json.dumps defaults
# to ensure_ascii=True (so "café" is stored as "caf\\u00e9" and the
# UTF-8 bytes are absent), and case folding disagrees between Python
# and the byte scanners.
for keyword in ("café", "你好", "straße", "ẞ"):
with self.subTest(keyword=keyword):
self.assertFalse(keywords_are_raw_byte_safe([keyword]))
def test_ascii_keyword_is_raw_byte_safe(self) -> None:
# The guard must not misfire on ordinary input — killing the speedup
# for everyone would be a worse outcome than the bug it prevents.
for keyword in ("hello", "TODO", "session_id", "a-b_c.d"):
with self.subTest(keyword=keyword):
self.assertTrue(keywords_are_raw_byte_safe([keyword]))
def test_guard_must_see_original_keyword_not_the_folded_one(self) -> None:
# "ß".casefold() == "ss", which is pure ASCII. A call site that folds
# before asking would get "safe" back for the exact input the guard
# exists to reject, silently restoring the bug.
self.assertFalse(keywords_are_raw_byte_safe(["ß"]))
self.assertTrue(keywords_are_raw_byte_safe(["ß".casefold()]))
def test_json_escaped_keywords_are_not_raw_byte_safe(self) -> None:
# JSON *must* escape control characters, '"' and '\\'; '/' is optional
# but some writers escape it. All of them mean the keyword's bytes are
# not in the file verbatim. The quote and backslash cases are the ones
# that matter most in practice — quoted error fragments, JSON config
# snippets and Windows paths are common history queries.
for keyword in ("foo\nbar", "a\tb", "src/main.py",
'say "hello world"', r"C:\Users", r"a\b"):
with self.subTest(keyword=keyword):
self.assertFalse(keywords_are_raw_byte_safe([keyword]))
def test_ascii_keyword_matches_fold_equivalent_content(self) -> None:
# The keyword-side guard cannot see the CONTENT. Python's casefold does
# full folding, so an all-ASCII keyword legitimately matches non-ASCII
# content — "financial" must find "financial" (U+FB01, what you get
# pasting from a PDF). A byte scanner does not fold that way, so the
# pre-filter has to treat such content as unscannable.
for content, keyword in (
("the financial model is attached", "financial"),
("DIE STRAẞE IST LANG", "strasse"),
("a ſtudy of state machines", "study"),
):
with self.subTest(content=content):
self._assert_search_finds(content, keyword)
def test_fold_equivalent_table_is_derived_not_hand_written(self) -> None:
# Guards against someone "tidying" the table into a hand-typed literal.
# A hand-written draft of this list had 17 entries, several of which do
# not fold to ASCII at all.
from _core.text import _FOLDS_TO_ASCII
self.assertIn("fi", _FOLDS_TO_ASCII)
self.assertIn("ß", _FOLDS_TO_ASCII)
for ch in _FOLDS_TO_ASCII:
self.assertFalse(ch.isascii(), f"{ch!r} is already ASCII")
self.assertTrue(ch.casefold().isascii(), f"{ch!r} does not fold to ASCII")
def _assert_search_finds(self, content: str, keyword: str,
ensure_ascii: bool = False) -> None:
with tempfile.TemporaryDirectory() as raw_root:
root = Path(raw_root)
home, workspace = root / "home", root / "ws"
workspace.mkdir()
target = project_dir(home, workspace) / "018f0000-0000-4000-8000-0000000000ab.jsonl"
target.parent.mkdir(parents=True, exist_ok=True)
record = {
"type": "user",
"timestamp": "2026-04-01T00:00:00Z",
"message": {"role": "user", "content": content},
}
target.write_text(
json.dumps(record, ensure_ascii=ensure_ascii) + "\n", encoding="utf-8"
)
completed = subprocess.run(
[sys.executable, str(SCRIPT), "search", str(workspace), keyword],
capture_output=True, text=True, check=False,
env=dict(os.environ, CLAUDE_CONFIG_DIR=str(home)),
)
self.assertNotIn("No matches found", completed.stdout,
f"lost a match: content={content!r} keyword={keyword!r}")
def test_one_unsafe_keyword_disables_the_filter_for_the_whole_query(self) -> None:
self.assertFalse(keywords_are_raw_byte_safe(["safe", "你好"]))
def test_escaped_unicode_session_is_still_found(self) -> None:
# End-to-end: a session file written with ensure_ascii=True (as any
# archive rewritten by default-parameter json.dumps would be) must
# still be found by a non-ASCII keyword.
with tempfile.TemporaryDirectory() as raw_root:
root = Path(raw_root)
home = root / "home"
workspace = root / "ws"
workspace.mkdir()
target = project_dir(home, workspace) / "018f0000-0000-4000-8000-0000000000ff.jsonl"
target.parent.mkdir(parents=True, exist_ok=True)
record = {
"type": "user",
"timestamp": "2026-04-01T00:00:00Z",
"message": {"role": "user", "content": "café 你好世界 discussion"},
}
# ensure_ascii=True is the point of this fixture — do not "fix" it.
target.write_text(json.dumps(record, ensure_ascii=True) + "\n", encoding="utf-8")
env = dict(os.environ, CLAUDE_CONFIG_DIR=str(home))
for keyword in ("café", "你好"):
with self.subTest(keyword=keyword):
completed = subprocess.run(
[sys.executable, str(SCRIPT), "search", str(workspace), keyword],
capture_output=True, text=True, env=env, check=False,
)
self.assertNotIn("No matches found", completed.stdout)
self.assertIn("018f0000", completed.stdout)
def test_codex_path_does_not_build_line_keywords(self) -> None:
# Line-level filtering is incompatible with Codex's session_range:
# observe() needs every record (including the session_meta first line)
# to report the conversation's span. With line filtering on, a measured
# 4-line rollout reported "01-10 .. 01-10" instead of "01-01 .. 01-20"
# — a wrong value in a field callers quote, on the default
# no-date-window search. This asserts nobody adds it back.
lines = SCRIPT.read_text(encoding="utf-8").splitlines()
starts = [i for i, l in enumerate(lines) if l.startswith("def search_codex_rollouts")]
self.assertEqual(len(starts), 1, "anchor is no longer unique")
start = starts[0]
# The next top-level construct is a `class`, not a `def` — searching
# only for "\ndef " runs past it and swallows SessionAnalyzer's own
# (legitimate) line_keywords, failing this test on correct code.
end = next(
(i for i, l in enumerate(lines)
if i > start and (l.startswith("def ") or l.startswith("class "))),
len(lines),
)
codex_body = "\n".join(lines[start:end])
self.assertIn(
"line_keywords = None", codex_body,
"sliced the wrong region — Codex body should contain the disabling line",
)
self.assertNotIn(
"[kw.casefold() for kw in keywords]",
codex_body,
"Codex must not build line_keywords — it collapses Internal range",
)
if __name__ == "__main__":
unittest.main()
tests/test_core_text.py
#!/usr/bin/env python3
"""Unit tests for the search pre-filter primitives in ``_core.text``.
These are the new, reusable building blocks a 2026-08 performance fix added
after a real search on this machine failed to complete in over three minutes
against a single 1.6GB/294-file project: brute-force ``json.loads()`` of
every line of every file, with no cheap way to rule out files/lines that
plainly cannot contain any of the search keywords. ``files_possibly_matching``
(file-level) and ``iter_jsonl``'s ``line_keywords`` (line-level) are both
deliberate OVER-approximations — see their docstrings for why that direction
is the only safe one for a completeness-first tool. These tests exist to hold
that safety property, not just the happy path.
"""
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from pathlib import Path
SKILL_DIR = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(SKILL_DIR / "scripts"))
from _core.text import files_possibly_matching, iter_jsonl # noqa: E402
class FilesPossiblyMatchingTests(unittest.TestCase):
def setUp(self) -> None:
self.temp_dir = tempfile.TemporaryDirectory()
self.root = Path(self.temp_dir.name)
def tearDown(self) -> None:
self.temp_dir.cleanup()
def _write(self, name: str, text: str) -> Path:
path = self.root / name
path.write_text(text, encoding="utf-8")
return path
def test_rules_out_a_file_with_no_keyword_anywhere(self) -> None:
hay = self._write("no-match.jsonl", '{"message": "totally unrelated content"}\n')
result = files_possibly_matching([hay], ["needle"])
self.assertIsNotNone(result)
self.assertNotIn(hay, result)
def test_keeps_a_file_that_contains_the_keyword(self) -> None:
hit = self._write("match.jsonl", '{"message": "found the needle here"}\n')
result = files_possibly_matching([hit], ["needle"])
self.assertIsNotNone(result)
self.assertIn(hit, result)
def test_case_insensitive_by_default(self) -> None:
hit = self._write("match.jsonl", '{"message": "Found the NEEDLE here"}\n')
result = files_possibly_matching([hit], ["needle"], case_sensitive=False)
self.assertIsNotNone(result)
self.assertIn(hit, result)
def test_case_sensitive_mode_rules_out_a_wrong_case_only_file(self) -> None:
wrong_case = self._write("wrong-case.jsonl", '{"message": "NEEDLE only in caps"}\n')
result = files_possibly_matching([wrong_case], ["needle"], case_sensitive=True)
self.assertIsNotNone(result)
self.assertNotIn(wrong_case, result)
def test_or_semantics_across_multiple_keywords(self) -> None:
only_second = self._write("second.jsonl", '{"message": "contains haystack only"}\n')
result = files_possibly_matching([only_second], ["needle", "haystack"])
self.assertIsNotNone(result)
self.assertIn(only_second, result, "OR semantics: matching ANY keyword must keep the file")
def test_batches_many_files_in_one_call(self) -> None:
paths = [
self._write(
f"f{i}.jsonl",
json.dumps({"message": "needle" if i % 3 == 0 else "noise"}) + "\n",
)
for i in range(10)
]
result = files_possibly_matching(paths, ["needle"])
self.assertIsNotNone(result)
expected = {paths[i] for i in range(10) if i % 3 == 0}
self.assertEqual(result, expected)
def test_degrades_to_none_on_empty_keywords(self) -> None:
hay = self._write("f.jsonl", '{"message": "anything"}\n')
self.assertIsNone(files_possibly_matching([hay], []))
def test_degrades_to_none_on_empty_paths(self) -> None:
self.assertIsNone(files_possibly_matching([], ["needle"]))
def test_degrades_to_none_on_control_character_keyword(self) -> None:
"""A keyword containing a raw newline can't be raw-byte-matched against
a JSONL file, because json encodes an embedded newline as the two
bytes ``\\n`` — the literal 0x0A byte never appears in the file. The
safe response is to skip filtering entirely for this keyword, not to
guess at the escaped form."""
hay = self._write("f.jsonl", '{"message": "line one\\nline two"}\n')
self.assertIsNone(files_possibly_matching([hay], ["line one\nline two"]))
def test_nonexistent_file_does_not_crash_and_is_excluded(self) -> None:
missing = self.root / "does-not-exist.jsonl"
result = files_possibly_matching([missing], ["needle"])
# rg/grep report a missing path as an error for that path but still
# succeed overall when other paths are given; alone, treat it as "no
# scanner result available" and degrade to not-filtering rather than
# crash the caller.
if result is not None:
self.assertNotIn(missing, result)
class IterJsonlLineKeywordsTests(unittest.TestCase):
def setUp(self) -> None:
self.temp_dir = tempfile.TemporaryDirectory()
self.root = Path(self.temp_dir.name)
def tearDown(self) -> None:
self.temp_dir.cleanup()
def _write_lines(self, records: list[dict]) -> Path:
path = self.root / "session.jsonl"
with path.open("w", encoding="utf-8") as handle:
for record in records:
handle.write(json.dumps(record, ensure_ascii=False) + "\n")
return path
def test_without_line_keywords_yields_every_record(self) -> None:
path = self._write_lines([{"a": 1}, {"a": 2}, {"a": 3}])
self.assertEqual(len(list(iter_jsonl(path))), 3)
def test_line_keywords_filters_out_non_matching_lines(self) -> None:
path = self._write_lines(
[
{"message": "irrelevant one"},
{"message": "contains the needle"},
{"message": "irrelevant two"},
]
)
results = list(iter_jsonl(path, line_keywords=["needle"]))
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["message"], "contains the needle")
def test_line_keywords_is_an_over_approximation_not_exact(self) -> None:
"""The pre-check matches the RAW line, including JSON structural
bytes / keys the real search deliberately excludes (see
_flatten_search_strings). A keyword appearing only in an excluded
key name still passes this cheap check — that is fine, it costs a
wasted json.loads() on one line, not a correctness bug, and the
real (structured) search downstream is what enforces the field
exclusion."""
path = self._write_lines([{"tool_use_id": "needle-shaped-id", "message": "hi"}])
results = list(iter_jsonl(path, line_keywords=["needle"]))
self.assertEqual(len(results), 1, "over-approximation must still parse the line")
def test_line_keywords_never_produces_a_false_negative(self) -> None:
"""Cross-check against the no-filter path on a larger synthetic file:
every record containing the keyword in an ORDINARY field must survive
filtering — this is the actual safety property, checked mechanically
rather than by eyeballing one example."""
records = [
{"message": f"record {i}" + (" needle" if i % 7 == 0 else "")}
for i in range(50)
]
path = self._write_lines(records)
unfiltered = {r["message"] for r in iter_jsonl(path)}
filtered = {r["message"] for r in iter_jsonl(path, line_keywords=["needle"])}
expected = {r["message"] for r in records if "needle" in r["message"]}
self.assertTrue(expected.issubset(filtered))
self.assertTrue(filtered.issubset(unfiltered))
def test_line_keywords_case_insensitive_via_casefold(self) -> None:
path = self._write_lines([{"message": "Found NEEDLE here"}])
results = list(iter_jsonl(path, line_keywords=["needle"]))
self.assertEqual(len(results), 1)
def test_bounded_mode_still_respects_line_keywords(self) -> None:
records = [{"message": "noise"} for _ in range(5)]
records.append({"message": "has needle"})
path = self._write_lines(records)
results = list(iter_jsonl(path, bounded=True, line_keywords=["needle"]))
self.assertEqual([r["message"] for r in results], ["has needle"])
if __name__ == "__main__":
unittest.main()
tests/test_extract_user_messages.py
#!/usr/bin/env python3
"""Tests for scripts/extract_user_messages.py — synthetic fixture homes only."""
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from datetime import datetime, timedelta, timezone
from pathlib import Path
SKILL_DIR = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(SKILL_DIR / "scripts"))
import extract_user_messages as ex # noqa: E402
from _core.sources import discover_claude_sources # noqa: E402
NOW = datetime.now(timezone.utc)
TS = (NOW - timedelta(hours=1)).strftime('%Y-%m-%dT%H:%M:%S.000Z')
TS_LATER = (NOW - timedelta(minutes=59)).strftime('%Y-%m-%dT%H:%M:%S.000Z')
TS_EARLIER = (NOW - timedelta(hours=2)).strftime('%Y-%m-%dT%H:%M:%S.000Z')
def write_jsonl(path: Path, records: list) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open('w', encoding='utf-8') as fh:
for r in records:
fh.write(json.dumps(r, ensure_ascii=False) + '\n')
def user(uuid, content, ts=TS, **extra):
rec = {
'type': 'user', 'uuid': uuid, 'sessionId': 's1', 'timestamp': ts,
'message': {'role': 'user', 'content': content},
}
rec.update(extra)
return rec
def assistant(uuid, text, ts=TS_EARLIER):
return {
'type': 'assistant', 'uuid': uuid, 'sessionId': 's1', 'timestamp': ts,
'message': {'role': 'assistant', 'content': [{'type': 'text', 'text': text}]},
}
def queued(uuid, prompt, kind='human', ts=TS):
att = {'type': 'queued_command', 'prompt': prompt, 'commandMode': 'prompt'}
if kind is not None:
att['origin'] = {'kind': kind}
return {'type': 'attachment', 'uuid': uuid, 'sessionId': 's1', 'timestamp': ts, 'attachment': att}
def run_extract(home: Path, records: list) -> ex.Extraction:
write_jsonl(home / 'projects' / '-tmp-proj' / 's1.jsonl', records)
sources, _ = discover_claude_sources(explicit_homes=[str(home)])
cutoff = NOW - timedelta(days=1)
return ex.extract(sources, cutoff)
class ExtractUserMessagesTest(unittest.TestCase):
def test_typed_message_is_kept(self):
ext = run_extract(self.home, [user('u1', '今天把这个修一下')])
self.assertEqual([e.text for e in ext.entries], ['今天把这个修一下'])
def test_structural_noise_is_dropped(self):
ext = run_extract(self.home, [
user('u1', 'real words'),
user('u2', '<task-notification> <task-id>x</task-id> </task-notification>', promptSource='system'),
user('u3', 'sdk template text', promptSource='sdk'),
user('u4', 'meta text', isMeta=True),
user('u5', [{'type': 'tool_result', 'tool_use_id': 't', 'content': 'res'}]),
user('u6', '[Request interrupted by user]'),
user('u7', 'This session is being continued from a previous conversation that ran out of context. Summary: ...'),
])
self.assertEqual([e.text for e in ext.entries], ['real words'])
def test_command_envelopes_go_to_appendix_with_args(self):
ext = run_extract(self.home, [
user('u1', '<command-message>foo</command-message>\n<command-name>/transcript-fixer</command-name>\n<command-args>修一下这个词</command-args>'),
user('u2', '/compact'),
user('u3', '/opt/data/x.md 这个文件看下'), # path text is NOT a command
])
self.assertEqual([c.text for c in ext.commands], ['/transcript-fixer 修一下这个词', '/compact'])
self.assertEqual([e.text for e in ext.entries], ['/opt/data/x.md 这个文件看下'])
def test_queued_command_human_is_recovered(self):
ext = run_extract(self.home, [
user('u1', 'normal'),
queued('q1', '打断一下，先别跑'),
])
texts = [e.text for e in ext.entries]
self.assertIn('打断一下，先别跑', texts)
def test_queued_list_variant_and_peer_origin(self):
ext = run_extract(self.home, [
queued('q1', [{'type': 'text', 'text': '第一句'}, {'type': 'text', 'text': '第二句'}]),
queued('q2', 'agent 投递', kind='peer'),
queued('q3', 'harness 通知', kind=None),
])
texts = [e.text for e in ext.entries]
self.assertEqual(texts, ['第一句\n第二句'])
def test_queued_dedupes_against_delivered_record(self):
ext = run_extract(self.home, [
queued('q1', '同一句话', ts=TS),
user('u1', '同一句话', ts=TS_LATER),
])
self.assertEqual(len(ext.entries), 1)
def test_boilerplate_frequency_detection_and_tail_strip(self):
boiler = '固定注入块 ' + '这是一个很长的注入提示词块。' * 40 # >400 normalized chars
records = [user(f'b{i}', boiler) for i in range(5)]
records.append(user('u1', '我自己的话。' + boiler))
ext = run_extract(self.home, records)
self.assertEqual(len(ext.injected), 5)
self.assertEqual([e.text for e in ext.entries], ['我自己的话。'])
def test_long_ascii_paste_goes_to_appendix(self):
paste = 'x' * 2100
ext = run_extract(self.home, [user('u1', paste), user('u2', '正常中文消息')])
self.assertEqual([e.text for e in ext.entries], ['正常中文消息'])
self.assertEqual(len(ext.pastes), 1)
def test_agent_voiced_reinjection_is_subtracted_only_when_earlier(self):
agent_text = '这是一段 agent 产出的分析结论，足够长，超过三十个字符的阈值限制。'
ext = run_extract(self.home, [
assistant('a1', agent_text, ts=TS_EARLIER),
user('u1', agent_text, ts=TS), # agent said it first -> dropped
])
self.assertEqual(ext.subtracted, 1)
self.assertEqual(len(ext.entries), 0)
ext2 = run_extract(self.home2, [
user('u1', agent_text, ts=TS_EARLIER), # user said it first -> kept
assistant('a1', agent_text, ts=TS),
])
self.assertEqual(ext2.subtracted, 0)
self.assertEqual(len(ext2.entries), 1)
def test_filler_flag(self):
ext = run_extract(self.home, [user('u1', '继续'), user('u2', '把这个方案展开讲讲')])
by_text = {e.text: e for e in ext.entries}
self.assertTrue(by_text['继续'].filler)
self.assertFalse(by_text['把这个方案展开讲讲'].filler)
def test_multi_line_transcript_paste_goes_to_appendix(self):
# Long CJK text spanning many non-blank lines = meeting transcript / dialog,
# even though ASCII ratio is far below 60%. Multi-line structure (NOT speaker
# labels — those misfire on user prose that quotes people) is what separates
# a transcript from coherent dictation.
transcript = '\n'.join(
f'张三：这是第{i}段会议发言内容，包含足够长的论述用于检测。' for i in range(70))
ext = run_extract(self.home, [user('u1', transcript), user('u2', '我的真实发言')])
self.assertEqual([e.text for e in ext.entries], ['我的真实发言'])
self.assertEqual(len(ext.pastes), 1)
def test_long_coherent_dictation_not_misfired_as_paste(self):
# Long CJK voice dictation in ONE paragraph (coherent, few lines) must stay
# in entries — the reason the multi-line branch requires >=10 lines, not just
# length. This is the case the original ASCII safe-harbor protected.
dictation = '我今天想跟你聊一下关于这个产品的一些想法和感受。' * 100 # ~2200 chars, 1 line
ext = run_extract(self.home, [user('u1', dictation)])
self.assertEqual(len(ext.entries), 1)
self.assertEqual(len(ext.pastes), 0)
def setUp(self):
self._tmp = tempfile.TemporaryDirectory()
self._tmp2 = tempfile.TemporaryDirectory()
self.home = Path(self._tmp.name)
self.home2 = Path(self._tmp2.name)
def tearDown(self):
self._tmp.cleanup()
self._tmp2.cleanup()
if __name__ == '__main__':
unittest.main()
tests/test_kimi_search.py
#!/usr/bin/env python3
"""Fixture tests for Kimi CLI wire search in analyze_sessions.py (--kimi).
All fixtures are synthetic tempfile trees shaped like ``~/.kimi-code``; no test
reads a real user store. Wire record timestamps are epoch MILLISECONDS (the
on-disk contract), the search API works in epoch seconds floats.
"""
from __future__ import annotations
import importlib.util
import json
import os
import subprocess
import sys
import tempfile
import unittest
from datetime import datetime, timezone
from pathlib import Path
SKILL_DIR = Path(__file__).resolve().parents[1]
SCRIPT = SKILL_DIR / "scripts" / "analyze_sessions.py"
def load_analyze_module():
"""Import analyze_sessions.py in-process to unit-test its helpers."""
spec = importlib.util.spec_from_file_location(
"analyze_sessions_kimi_under_test", SCRIPT
)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def epoch_ms(
year: int, month: int, day: int, hour: int = 0, minute: int = 0, second: int = 0
) -> int:
return int(
datetime(year, month, day, hour, minute, second, tzinfo=timezone.utc).timestamp()
* 1000
)
def epoch_s(
year: int, month: int, day: int, hour: int = 0, minute: int = 0, second: int = 0
) -> float:
return datetime(
year, month, day, hour, minute, second, tzinfo=timezone.utc
).timestamp()
def write_jsonl(path: Path, records: list[object]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8") as handle:
for record in records:
handle.write(json.dumps(record, ensure_ascii=False) + "\n")
def kimi_metadata_record(created_ms: int) -> dict[str, object]:
return {"type": "metadata", "protocol_version": "1.5", "created_at": created_ms}
def kimi_user_prompt(text: str, time_ms: int, *, kind: str = "user") -> dict[str, object]:
return {
"type": "turn.prompt",
"input": [{"type": "text", "text": text}],
"origin": {"kind": kind},
"time": time_ms,
}
def kimi_message(role: str, text: str, time_ms: int | None) -> dict[str, object]:
record: dict[str, object] = {
"type": "context.append_message",
"message": {"role": role, "content": [{"type": "text", "text": text}]},
}
if time_ms is not None:
record["time"] = time_ms
return record
def kimi_state(
session_id: str,
cwd: str,
title: str,
created_ms: int,
updated_ms: int,
) -> dict[str, object]:
return {
"id": session_id,
"cwd": cwd,
"title": title,
"createdAt": created_ms,
"updatedAt": updated_ms,
"archived": False,
"agents": {"main": {}},
}
def write_kimi_session(
home: Path,
session_id: str,
*,
bucket: str = "wd_demo-project_0a1b2c",
state: dict[str, object] | None = None,
wires: dict[str, list[object]] | None = None,
) -> Path:
session_dir = home / "sessions" / bucket / session_id
if state is not None:
session_dir.mkdir(parents=True, exist_ok=True)
(session_dir / "state.json").write_text(
json.dumps(state, ensure_ascii=False), encoding="utf-8"
)
for agent_name, records in (wires or {}).items():
write_jsonl(session_dir / "agents" / agent_name / "wire.jsonl", records)
return session_dir
def normalized_matches(matches: list[dict[str, object]]) -> list[dict[str, object]]:
"""Stringify Paths so two match dict lists compare equal byte-for-byte."""
return [
{
key: (str(value) if isinstance(value, Path) else value)
for key, value in match.items()
}
for match in matches
]
class KimiSearchableSegmentsTests(unittest.TestCase):
"""``kimi_searchable_segments`` covers conversation records, skips boilerplate."""
@classmethod
def setUpClass(cls) -> None:
cls.module = load_analyze_module()
def segment_pairs(self, record: dict[str, object]) -> set[tuple[str, str]]:
return {
(segment.source, segment.text)
for segment in self.module.kimi_searchable_segments(record)
}
def test_conversation_record_types_are_extracted(self) -> None:
prompt = kimi_user_prompt("hello world from the user", epoch_ms(2026, 4, 1))
self.assertIn(("prompt", "hello world from the user"), self.segment_pairs(prompt))
steer = {
"type": "turn.steer",
"input": [{"type": "text", "text": "steer note here"}],
"origin": {"kind": "user"},
"time": epoch_ms(2026, 4, 1),
}
self.assertIn(("prompt", "steer note here"), self.segment_pairs(steer))
message = {
"type": "context.append_message",
"message": {
"role": "user",
"content": [{"type": "text", "text": "user says hello"}],
"toolCalls": [{"name": "Bash", "input": {"command": "pytest -x"}}],
},
"time": epoch_ms(2026, 4, 1),
}
message_pairs = self.segment_pairs(message)
self.assertIn(("message", "user says hello"), message_pairs)
self.assertIn(("tool_input", "pytest -x"), message_pairs)
content_part = {
"type": "context.append_loop_event",
"event": {
"type": "content.part",
"part": {"type": "text", "text": "assistant reply text"},
},
"time": epoch_ms(2026, 4, 1),
}
self.assertIn(
("message", "assistant reply text"), self.segment_pairs(content_part)
)
tool_call = {
"type": "context.append_loop_event",
"event": {
"type": "tool.call",
"name": "Read",
"arguments": {"path": "/tmp/x"},
},
"time": epoch_ms(2026, 4, 1),
}
call_pairs = self.segment_pairs(tool_call)
self.assertIn(("tool_input", "Read"), call_pairs)
self.assertIn(("tool_input", "/tmp/x"), call_pairs)
tool_result = {
"type": "context.append_loop_event",
"event": {
"type": "tool.result",
"content": [{"type": "text", "text": "file contents here"}],
},
"time": epoch_ms(2026, 4, 1),
}
self.assertIn(
("tool_result", "file contents here"), self.segment_pairs(tool_result)
)
plugin = {
"type": "plugin.session_start",
"content": "plugin boot log token",
"time": epoch_ms(2026, 4, 1),
}
self.assertIn(("plugin", "plugin boot log token"), self.segment_pairs(plugin))
# step.begin / step.end carry no conversation text.
step = {
"type": "context.append_loop_event",
"event": {"type": "step.begin", "step": 1},
"time": epoch_ms(2026, 4, 1),
}
self.assertEqual(self.module.kimi_searchable_segments(step), [])
def test_structural_identifier_keys_are_not_indexed(self) -> None:
"""UUID-class fields must not become searchable text (review M1)."""
record = {
"type": "context.append_loop_event",
"event": {
"type": "tool.call",
"name": "Read",
"arguments": {"path": "/tmp/keep-me"},
"uuid": "uuid-needle-aaa",
"stepUuid": "uuid-needle-bbb",
"turnId": "uuid-needle-ccc",
"toolCallId": "tool_uuid-needle-ddd",
"parentUuid": "uuid-needle-eee",
},
"time": epoch_ms(2026, 4, 1),
}
segments = self.module.kimi_searchable_segments(record)
texts = [segment.text for segment in segments]
self.assertIn("/tmp/keep-me", texts)
self.assertIn("Read", texts)
for leaked in (
"uuid-needle-aaa",
"uuid-needle-bbb",
"uuid-needle-ccc",
"tool_uuid-needle-ddd",
"uuid-needle-eee",
):
self.assertNotIn(leaked, texts)
message = {
"type": "context.append_message",
"message": {
"role": "assistant",
"id": "msg_uuid-needle-fff",
"content": [{"type": "text", "text": "real answer"}],
"toolCalls": [
{"id": "call_uuid-needle-ggg", "name": "Bash", "input": {"command": "ls"}}
],
},
"time": epoch_ms(2026, 4, 1),
}
message_texts = [
segment.text for segment in self.module.kimi_searchable_segments(message)
]
self.assertIn("real answer", message_texts)
self.assertNotIn("msg_uuid-needle-fff", message_texts)
self.assertNotIn("call_uuid-needle-ggg", message_texts)
def test_boilerplate_records_have_no_searchable_segments(self) -> None:
boilerplate = [
{
"type": "config.update",
"config": {"systemPrompt": "You are Kimi with needle-token"},
"time": epoch_ms(2026, 4, 1),
},
{
"type": "profile.bind",
"profile": {"systemPrompt": "needle-token system prompt"},
"time": epoch_ms(2026, 4, 1),
},
{
"type": "llm.tools_snapshot",
"tools": [{"name": "needle-tool", "description": "needle-token"}],
"time": epoch_ms(2026, 4, 1),
},
{
"type": "usage.record",
"usage": {"note": "needle-token"},
"time": epoch_ms(2026, 4, 1),
},
{
"type": "token_counting.measured",
"detail": "needle-token",
"time": epoch_ms(2026, 4, 1),
},
]
for record in boilerplate:
with self.subTest(record_type=record["type"]):
self.assertEqual(self.module.kimi_searchable_segments(record), [])
class KimiWireSearchTests(unittest.TestCase):
"""``search_kimi_wires`` aggregation, filtering and range accounting."""
@classmethod
def setUpClass(cls) -> None:
cls.module = load_analyze_module()
def setUp(self) -> None:
self.temp_dir = tempfile.TemporaryDirectory()
self.root = Path(self.temp_dir.name)
self.workspace = self.root / "workspaces" / "demo-project"
self.workspace.mkdir(parents=True)
self.kimi_home = self.root / "kimi-home"
def tearDown(self) -> None:
self.temp_dir.cleanup()
def search(self, keywords: list[str], **overrides: object):
wires = self.module.discover_kimi_wires(self.kimi_home)
return self.module.search_kimi_wires(wires, keywords, **overrides)
def test_subagent_wires_aggregate_into_one_session_match(self) -> None:
session_id = "session_11111111-1111-4111-8111-111111111111"
base_ms = epoch_ms(2026, 4, 1, 10, 0, 0)
write_kimi_session(
self.kimi_home,
session_id,
state=kimi_state(
session_id,
str(self.workspace),
"Hunt session",
base_ms,
base_ms + 120_000,
),
wires={
"main": [
kimi_metadata_record(base_ms),
kimi_message("user", "shared hunt target alpha", base_ms + 60_000),
],
"agent-0": [
kimi_metadata_record(base_ms + 60_000),
kimi_message(
"assistant", "shared hunt target beta", base_ms + 120_000
),
],
},
)
matches = self.search(["hunt target"], use_prefilter=False)
self.assertEqual(len(matches), 1)
match = matches[0]
self.assertEqual(match["session_id"], session_id)
self.assertEqual(match["title"], "Hunt session")
self.assertEqual(match["cwd"], str(self.workspace))
# main + agent-0 hits merge into ONE session match; sources name the wire.
self.assertEqual(match["total_mentions"], 2)
self.assertEqual(match["keyword_counts"], {"hunt target": 2})
self.assertEqual(match["match_sources"], ["agent-0:message", "main:message"])
def test_project_path_filter_uses_state_cwd(self) -> None:
other_workspace = self.root / "workspaces" / "other-project"
other_workspace.mkdir(parents=True)
base_ms = epoch_ms(2026, 4, 2, 10, 0, 0)
in_id = "session_22222222-2222-4222-8222-222222222222"
out_id = "session_33333333-3333-4333-8333-333333333333"
stateless_id = "session_44444444-4444-4444-8444-444444444444"
for session_id, cwd in ((in_id, self.workspace), (out_id, other_workspace)):
write_kimi_session(
self.kimi_home,
session_id,
state=kimi_state(
session_id, str(cwd), "Scoped session", base_ms, base_ms + 60_000
),
wires={
"main": [
kimi_metadata_record(base_ms),
kimi_message("user", "scoped probe marker", base_ms + 60_000),
]
},
)
# A session without state.json has no cwd and must not match a scope.
write_kimi_session(
self.kimi_home,
stateless_id,
bucket="wd_unknown_123456",
wires={
"main": [
kimi_metadata_record(base_ms),
kimi_message("user", "scoped probe marker", base_ms + 60_000),
]
},
)
scoped = self.search(
["scoped probe"], project_path=str(self.workspace), use_prefilter=False
)
self.assertEqual([match["session_id"] for match in scoped], [in_id])
swept = self.search(["scoped probe"], use_prefilter=False)
self.assertEqual(
{match["session_id"] for match in swept}, {in_id, out_id, stateless_id}
)
def test_date_window_filters_records_and_counts_untimed(self) -> None:
session_id = "session_55555555-5555-4555-8555-555555555555"
meta_ms = epoch_ms(2026, 4, 1, 0, 0, 0)
in_ms = epoch_ms(2026, 4, 15, 10, 0, 0)
out_ms = epoch_ms(2026, 5, 10, 10, 0, 0)
write_kimi_session(
self.kimi_home,
session_id,
wires={
"main": [
kimi_metadata_record(meta_ms),
kimi_message("user", "date window probe", in_ms),
kimi_message("user", "date window probe again", out_ms),
kimi_message("user", "untimed filler, no time field", None),
]
},
)
windowed = self.search(
["date window probe"],
from_timestamp=epoch_s(2026, 4, 10),
to_timestamp=epoch_s(2026, 4, 30, 23, 59, 59),
use_prefilter=False,
)
self.assertEqual(len(windowed), 1)
match = windowed[0]
# Only the in-window occurrence counts.
self.assertEqual(match["keyword_counts"], {"date window probe": 1})
self.assertEqual(match["total_mentions"], 1)
self.assertEqual(match["excluded_untimed_records"], 1)
# The session range still spans every timed record; the match range
# covers only in-window hits.
self.assertAlmostEqual(match["created_at"], meta_ms / 1000, places=3)
self.assertAlmostEqual(match["updated_at"], out_ms / 1000, places=3)
self.assertAlmostEqual(match["match_created_at"], in_ms / 1000, places=3)
self.assertAlmostEqual(match["match_updated_at"], in_ms / 1000, places=3)
unwindowed = self.search(["date window probe"], use_prefilter=False)
self.assertEqual(unwindowed[0]["keyword_counts"], {"date window probe": 2})
self.assertEqual(unwindowed[0]["excluded_untimed_records"], 0)
def test_prefilter_is_neutral_for_ascii_keyword(self) -> None:
base_ms = epoch_ms(2026, 4, 3, 10, 0, 0)
matching_id = "session_66666666-6666-4666-8666-666666666666"
other_id = "session_77777777-7777-4777-8777-777777777777"
for session_id, text in (
(matching_id, "contains the NeutralProbe keyword here"),
(other_id, "nothing of interest in this wire"),
):
write_kimi_session(
self.kimi_home,
session_id,
bucket=f"wd_demo-project_{session_id[-6:]}",
state=kimi_state(
session_id,
str(self.workspace),
"Prefilter session",
base_ms,
base_ms + 60_000,
),
wires={
"main": [
kimi_metadata_record(base_ms),
kimi_message("user", text, base_ms + 60_000),
]
},
)
default_run = self.search(["NeutralProbe"], use_prefilter=True)
no_prefilter_run = self.search(["NeutralProbe"], use_prefilter=False)
self.assertEqual(
normalized_matches(default_run), normalized_matches(no_prefilter_run)
)
self.assertEqual([match["session_id"] for match in default_run], [matching_id])
def test_session_range_covers_prefiltered_wire_times(self) -> None:
session_id = "session_88888888-8888-4888-8888-888888888888"
main_start_ms = epoch_ms(2026, 4, 1, 0, 0, 0)
main_end_ms = epoch_ms(2026, 4, 20, 0, 0, 0)
agent_hit_ms = epoch_ms(2026, 4, 10, 12, 0, 0)
write_kimi_session(
self.kimi_home,
session_id,
wires={
# The keyword lives ONLY in agent-0. With the pre-filter on,
# the main wire is ruled out as a whole file — but the session
# range must still fold its time span back in.
"main": [
kimi_metadata_record(main_start_ms),
kimi_message("user", "ordinary chatter without the token", main_end_ms),
],
"agent-0": [
kimi_metadata_record(epoch_ms(2026, 4, 10, 11, 0, 0)),
kimi_message("assistant", "contains OnlyAgentProbe here", agent_hit_ms),
],
},
)
matches = self.search(["OnlyAgentProbe"], use_prefilter=True)
self.assertEqual(len(matches), 1)
match = matches[0]
self.assertEqual(match["session_id"], session_id)
self.assertEqual(match["match_sources"], ["agent-0:message"])
self.assertAlmostEqual(match["created_at"], main_start_ms / 1000, places=3)
self.assertAlmostEqual(match["updated_at"], main_end_ms / 1000, places=3)
self.assertAlmostEqual(match["match_created_at"], agent_hit_ms / 1000, places=3)
self.assertAlmostEqual(match["match_updated_at"], agent_hit_ms / 1000, places=3)
def test_keyword_only_in_system_prompt_boilerplate_finds_nothing(self) -> None:
session_id = "session_99999999-9999-4999-8999-999999999999"
base_ms = epoch_ms(2026, 4, 4, 10, 0, 0)
write_kimi_session(
self.kimi_home,
session_id,
wires={
"main": [
kimi_metadata_record(base_ms),
{
"type": "config.update",
"config": {
"systemPrompt": "You are Kimi; always mention BoilerProbe"
},
"time": base_ms + 1000,
},
kimi_message("user", "an ordinary user message", base_ms + 2000),
]
},
)
# The raw bytes are in the file (so the pre-filter keeps the file as a
# candidate), but the structured search must never index boilerplate.
for use_prefilter in (True, False):
with self.subTest(use_prefilter=use_prefilter):
self.assertEqual(
self.search(["BoilerProbe"], use_prefilter=use_prefilter), []
)
class KimiCliSearchTests(unittest.TestCase):
"""End-to-end wiring of the ``search --kimi --kimi-home`` CLI flags."""
def setUp(self) -> None:
self.temp_dir = tempfile.TemporaryDirectory()
self.root = Path(self.temp_dir.name)
self.user_home = self.root / "user-home"
self.active_home = self.user_home / ".claude"
self.archive_home = self.root / "conversation-archive"
self.workspace = self.root / "workspaces" / "demo-project"
self.workspace.mkdir(parents=True)
(self.active_home / "projects").mkdir(parents=True)
(self.archive_home / "projects").mkdir(parents=True)
self.manifest = self.active_home / "history-sources.json"
self.manifest.write_text(
json.dumps(
{
"version": 1,
"sources": [
{
"provider": "claude",
"kind": "archive",
"label": "full-backup",
"home": str(self.archive_home),
"required": True,
}
],
}
),
encoding="utf-8",
)
self.kimi_home = self.root / "kimi-home"
def tearDown(self) -> None:
self.temp_dir.cleanup()
def run_cli(self, *arguments: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(SCRIPT), *arguments],
text=True,
encoding="utf-8",
capture_output=True,
check=True,
env={**os.environ, "HOME": str(self.user_home)},
)
def test_cli_kimi_flag_searches_wires(self) -> None:
session_id = "session_aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"
base_ms = epoch_ms(2026, 4, 5, 10, 0, 0)
write_kimi_session(
self.kimi_home,
session_id,
state=kimi_state(
session_id,
str(self.workspace),
"CLI probe session",
base_ms,
base_ms + 60_000,
),
wires={
"main": [
kimi_metadata_record(base_ms),
kimi_user_prompt(
"please run CliKimiProbe check", base_ms + 60_000
),
]
},
)
completed = self.run_cli(
"search",
str(self.workspace),
"CliKimiProbe",
"--kimi",
"--kimi-home",
str(self.kimi_home),
"--history-sources",
str(self.manifest),
)
self.assertIn("Kimi CLI session matches", completed.stdout)
self.assertIn(session_id, completed.stdout)
self.assertIn("CLI probe session", completed.stdout)
self.assertIn("main:prompt", completed.stdout)
self.assertIn("Total mentions: 1", completed.stdout)
# Without --kimi the Kimi store is invisible: a Claude session with
# unrelated content lets the search run to completion and prove the
# keyword only lived in the unsearched Kimi wire.
claude_project = (
self.active_home
/ "projects"
/ str(self.workspace.resolve()).replace("/", "-")
)
write_jsonl(
claude_project / "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb.jsonl",
[
{
"type": "user",
"sessionId": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
"cwd": str(self.workspace),
"timestamp": "2026-04-05T10:00:00Z",
"message": {"role": "user", "content": "unrelated claude chat"},
}
],
)
without_flag = self.run_cli(
"search",
str(self.workspace),
"CliKimiProbe",
"--history-sources",
str(self.manifest),
)
self.assertNotIn("Kimi CLI session matches", without_flag.stdout)
self.assertNotIn(session_id, without_flag.stdout)
self.assertIn("No matches found.", without_flag.stdout)
if __name__ == "__main__":
unittest.main()
tests/test_recover_content.py
#!/usr/bin/env python3
"""Regression tests for exact Claude file-history recovery."""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path, PurePosixPath
from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
from recover_content import SessionContentRecovery # noqa: E402
SKILL_DIR = Path(__file__).resolve().parents[1]
SCRIPT = SKILL_DIR / "scripts" / "recover_content.py"
SESSION_ID = "11111111-1111-4111-8111-111111111111"
def write_jsonl(path: Path, records: list[object]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8") as handle:
for record in records:
handle.write(json.dumps(record, ensure_ascii=False) + "\n")
def write_record(
path: str,
content: str,
timestamp: str,
tool_use_id: str | None = None,
) -> dict:
tool_use = {
"type": "tool_use",
"name": "Write",
"input": {"file_path": path, "content": content},
}
if tool_use_id is not None:
tool_use["id"] = tool_use_id
return {
"type": "assistant",
"sessionId": SESSION_ID,
"timestamp": timestamp,
"message": {
"role": "assistant",
"content": [tool_use],
},
}
def tool_result_record(tool_use_id: str, is_error: bool, timestamp: str) -> dict:
return {
"type": "user",
"sessionId": SESSION_ID,
"timestamp": timestamp,
"message": {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use_id,
"is_error": is_error,
"content": "permission denied" if is_error else "write complete",
}
],
},
}
def snapshot_record(
path: str, backup_name: str | None, version: int, timestamp: str
) -> dict:
return {
"type": "file-history-snapshot",
"messageId": "22222222-2222-4222-8222-222222222222",
"snapshot": {
"messageId": "22222222-2222-4222-8222-222222222222",
"timestamp": timestamp,
"trackedFileBackups": {
path: {
"backupFileName": backup_name,
"version": version,
"backupTime": timestamp,
}
},
},
"isSnapshotUpdate": True,
}
def edit_record(path: str, marker: str, timestamp: str) -> dict:
return {
"type": "assistant",
"sessionId": SESSION_ID,
"timestamp": timestamp,
"message": {
"role": "assistant",
"content": [
{
"type": "tool_use",
"name": "Edit",
"input": {
"file_path": path,
"old_string": f"old-{marker}" * 1000,
"new_string": f"new-{marker}" * 1000,
},
}
],
},
}
class SessionRecoveryTests(unittest.TestCase):
def setUp(self) -> None:
self.temp_dir = tempfile.TemporaryDirectory()
self.root = Path(self.temp_dir.name)
self.user_home = self.root / "user-home"
self.claude_home = self.user_home / ".claude"
self.project_dir = self.claude_home / "projects" / "-tmp-demo"
self.project_dir.mkdir(parents=True)
self.session_file = self.project_dir / f"{SESSION_ID}.jsonl"
self.original = self.root / "jobs" / "task" / "artifact.bin"
self.output_dir = self.root / "recovered"
def tearDown(self) -> None:
self.temp_dir.cleanup()
def run_cli(
self, *arguments: str, check: bool = True
) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(SCRIPT), *arguments],
text=True,
encoding="utf-8",
capture_output=True,
check=check,
env={
**os.environ,
"HOME": str(self.user_home),
"CLAUDE_CONFIG_DIR": str(self.claude_home),
},
)
def expected_output(self, original: Path | None = None) -> Path:
source = original or self.original
return self.output_dir.joinpath(*PurePosixPath(str(source)).parts[1:])
def backup_path(self, name: str, root: Path | None = None) -> Path:
history_root = root or (self.claude_home / "file-history")
path = history_root / SESSION_ID / name
path.parent.mkdir(parents=True, exist_ok=True)
return path
def test_file_history_snapshot_wins_over_stale_write(self) -> None:
backup_name = "abc123@v2"
final_bytes = b"final after edit\n"
write_jsonl(
self.session_file,
[
write_record(
str(self.original), "initial write\n", "2026-07-01T10:00:00Z"
),
snapshot_record(
str(self.original), backup_name, 2, "2026-07-01T10:05:00Z"
),
],
)
self.backup_path(backup_name).write_bytes(final_bytes)
completed = self.run_cli(
str(self.session_file), "-k", "artifact.bin", "-o", str(self.output_dir)
)
self.assertEqual(self.expected_output().read_bytes(), final_bytes)
self.assertIn("Source: file-history v2", completed.stdout)
self.assertIn("exact bytes from captured checkpoint", completed.stdout)
def test_snapshot_recovers_binary_file_without_write_call(self) -> None:
backup_name = "binary123@v1"
binary = b"\x00\xff\x10binary\x00"
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 1, "2026-07-01T11:00:00Z"
)
],
)
self.backup_path(backup_name).write_bytes(binary)
self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), binary)
def test_latest_snapshot_version_is_selected(self) -> None:
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), "versioned@v1", 1, "2026-07-01T12:00:00Z"
),
snapshot_record(
str(self.original), "versioned@v2", 2, "2026-07-01T12:05:00Z"
),
],
)
self.backup_path("versioned@v1").write_bytes(b"old")
self.backup_path("versioned@v2").write_bytes(b"new")
self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), b"new")
def test_missing_exact_backup_aborts_without_stale_write_fallback(self) -> None:
write_jsonl(
self.session_file,
[
write_record(str(self.original), "stale", "2026-07-01T13:00:00Z"),
snapshot_record(
str(self.original), "missing@v2", 2, "2026-07-01T13:05:00Z"
),
],
)
completed = self.run_cli(
str(self.session_file), "-o", str(self.output_dir), check=False
)
self.assertEqual(completed.returncode, 2)
self.assertIn("exact backup is unavailable", completed.stderr)
self.assertFalse(self.expected_output().exists())
def test_write_only_is_an_explicit_lower_fidelity_mode(self) -> None:
write_jsonl(
self.session_file,
[
write_record(str(self.original), "stale", "2026-07-01T14:00:00Z"),
snapshot_record(
str(self.original), "missing@v2", 2, "2026-07-01T14:05:00Z"
),
],
)
completed = self.run_cli(
str(self.session_file),
"--write-only",
"-o",
str(self.output_dir),
)
self.assertEqual(self.expected_output().read_text(encoding="utf-8"), "stale")
self.assertIn(
"Write checkpoint; later Edit or shell changes may be absent",
completed.stdout,
)
def test_explicit_file_history_root_recovers_archive_copy(self) -> None:
archive = self.root / "archive"
self.session_file = archive / "projects" / "-tmp-demo" / f"{SESSION_ID}.jsonl"
backup_name = "archive123@v3"
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 3, "2026-07-01T15:00:00Z"
)
],
)
companion_root = self.root / "companion-file-history"
self.backup_path(backup_name, companion_root).write_bytes(b"archive-final")
self.run_cli(
str(self.session_file),
"--file-history-root",
str(companion_root),
"-o",
str(self.output_dir),
)
self.assertEqual(self.expected_output().read_bytes(), b"archive-final")
def test_unsafe_backup_name_is_rejected(self) -> None:
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), "../outside", 1, "2026-07-01T16:00:00Z"
)
],
)
completed = self.run_cli(
str(self.session_file), "-o", str(self.output_dir), check=False
)
self.assertEqual(completed.returncode, 2)
self.assertIn("Unsafe file-history backup name", completed.stderr)
self.assertFalse(self.expected_output().exists())
def test_snapshot_version_and_backup_name_must_agree(self) -> None:
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), "mismatch@v1", 2, "2026-07-01T16:30:00Z"
)
],
)
completed = self.run_cli(
str(self.session_file), "-o", str(self.output_dir), check=False
)
self.assertEqual(completed.returncode, 2)
self.assertIn("conflicts with backup name", completed.stderr)
self.assertFalse(self.output_dir.exists())
def test_later_tombstone_recovers_last_available_checkpoint(self) -> None:
backup_name = "before-delete@v2"
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 2, "2026-07-01T17:00:00Z"
),
snapshot_record(str(self.original), None, 3, "2026-07-01T17:05:00Z"),
],
)
self.backup_path(backup_name).write_bytes(b"last existing bytes")
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), b"last existing bytes")
self.assertIn("before a later recorded deletion", completed.stdout)
self.assertIn(
"Later state: recorded deleted at file-history v3", completed.stdout
)
def test_lower_version_tombstone_does_not_poison_newer_backup(self) -> None:
backup_name = "newer@v2"
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 2, "2026-07-01T18:00:00Z"
),
snapshot_record(str(self.original), None, 1, "2026-07-01T18:05:00Z"),
],
)
self.backup_path(backup_name).write_bytes(b"newer")
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), b"newer")
self.assertNotIn("Later state:", completed.stdout)
def test_unrelated_tombstone_does_not_abort_write_recovery(self) -> None:
deleted = self.root / "jobs" / "task" / "deleted.txt"
write_jsonl(
self.session_file,
[
write_record(
str(self.original), "write survives", "2026-07-01T19:00:00Z"
),
snapshot_record(str(deleted), None, 4, "2026-07-01T19:05:00Z"),
],
)
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(
self.expected_output().read_text(encoding="utf-8"), "write survives"
)
self.assertIn("Skipped deleted path", completed.stdout)
def test_write_after_tombstone_is_reported_as_recreated_not_later_deleted(
self,
) -> None:
write_jsonl(
self.session_file,
[
snapshot_record(str(self.original), None, 2, "2026-07-01T19:30:00Z"),
write_record(str(self.original), "recreated", "2026-07-01T19:35:00Z"),
],
)
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(
self.expected_output().read_text(encoding="utf-8"), "recreated"
)
self.assertNotIn("Later state:", completed.stdout)
def test_write_after_snapshot_and_tombstone_recovers_recreated_bytes(self) -> None:
backup_name = "before-recreate@v2"
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 2, "2026-07-01T19:20:00Z"
),
snapshot_record(str(self.original), None, 3, "2026-07-01T19:25:00Z"),
write_record(
str(self.original), "recreated-new", "2026-07-01T19:30:00Z"
),
],
)
self.backup_path(backup_name).write_bytes(b"pre-delete-old")
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(
self.expected_output().read_text(encoding="utf-8"), "recreated-new"
)
self.assertIn("Source: Write", completed.stdout)
self.assertNotIn("Later state:", completed.stdout)
def test_failed_later_write_does_not_override_exact_snapshot(self) -> None:
backup_name = "confirmed-before-failure@v2"
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 2, "2026-07-01T19:30:00Z"
),
write_record(
str(self.original),
"never-written",
"2026-07-01T19:35:00Z",
"toolu_failed_write",
),
tool_result_record(
"toolu_failed_write", True, "2026-07-01T19:35:01Z"
),
],
)
self.backup_path(backup_name).write_bytes(b"confirmed-exact")
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), b"confirmed-exact")
self.assertIn("Source: file-history v2", completed.stdout)
self.assertNotIn("never-written", completed.stdout)
def test_unrelated_write_conflict_does_not_abort_keyword_recovery(self) -> None:
unrelated = self.root / "jobs" / "task" / "unrelated.txt"
write_jsonl(
self.session_file,
[
write_record(
str(self.original), "wanted-content", "2026-07-01T19:31:00Z"
),
write_record(str(unrelated), "noise-a", "2026-07-01T19:32:00Z"),
write_record(str(unrelated), "noise-b", "2026-07-01T19:32:00Z"),
],
)
self.run_cli(
str(self.session_file),
"-k",
"artifact.bin",
"-o",
str(self.output_dir),
)
self.assertEqual(
self.expected_output().read_text(encoding="utf-8"), "wanted-content"
)
def test_newer_snapshot_makes_older_write_conflict_irrelevant(self) -> None:
backup_name = "after-conflict@v2"
write_jsonl(
self.session_file,
[
write_record(str(self.original), "draft-a", "2026-07-01T19:33:00Z"),
write_record(str(self.original), "draft-b", "2026-07-01T19:33:00Z"),
snapshot_record(
str(self.original), backup_name, 2, "2026-07-01T19:34:00Z"
),
],
)
self.backup_path(backup_name).write_bytes(b"exact-after-conflict")
self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), b"exact-after-conflict")
def test_older_write_conflict_is_superseded_by_unique_later_write(self) -> None:
write_jsonl(
self.session_file,
[
write_record(str(self.original), "draft-a", "2026-07-01T19:35:00Z"),
write_record(str(self.original), "draft-b", "2026-07-01T19:35:00Z"),
write_record(
str(self.original), "final-write", "2026-07-01T19:36:00Z"
),
],
)
self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(
self.expected_output().read_text(encoding="utf-8"), "final-write"
)
def test_write_before_tombstone_reports_the_later_deletion(self) -> None:
write_jsonl(
self.session_file,
[
write_record(str(self.original), "pre-delete", "2026-07-01T19:40:00Z"),
snapshot_record(str(self.original), None, 3, "2026-07-01T19:45:00Z"),
],
)
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(
self.expected_output().read_text(encoding="utf-8"), "pre-delete"
)
self.assertIn("Later state: recorded deleted", completed.stdout)
def test_registered_archive_file_history_root_is_used_automatically(self) -> None:
archive_home = self.root / "registered-archive"
(archive_home / "projects" / "-tmp-demo").mkdir(parents=True)
self.claude_home.joinpath("history-sources.json").write_text(
json.dumps(
{
"version": 1,
"sources": [
{
"provider": "claude",
"kind": "archive",
"label": "backup",
"home": str(archive_home),
"required": True,
}
],
}
),
encoding="utf-8",
)
backup_name = "archive-root@v4"
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 4, "2026-07-01T20:00:00Z"
)
],
)
self.backup_path(backup_name, archive_home / "file-history").write_bytes(
b"registered exact"
)
self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), b"registered exact")
def test_registered_archive_session_copy_supersedes_stale_active_write(
self,
) -> None:
archive_home = self.root / "registered-archive"
archive_session = (
archive_home / "projects" / "-moved-project" / self.session_file.name
)
self.claude_home.joinpath("history-sources.json").write_text(
json.dumps(
{
"version": 1,
"sources": [
{
"provider": "claude",
"kind": "archive",
"label": "backup",
"home": str(archive_home),
"required": True,
}
],
}
),
encoding="utf-8",
)
backup_name = "archive-copy@v5"
write_jsonl(
self.session_file,
[write_record(str(self.original), "stale", "2026-07-01T21:00:00Z")],
)
write_jsonl(
archive_session,
[
snapshot_record(
str(self.original), backup_name, 5, "2026-07-01T21:05:00Z"
)
],
)
self.backup_path(backup_name, archive_home / "file-history").write_bytes(
b"archive final"
)
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_bytes(), b"archive final")
self.assertIn("Session copies: 2", completed.stdout)
def test_report_destination_collision_aborts_before_writing(self) -> None:
write_jsonl(
self.session_file,
[write_record("recovery_report.txt", "artifact", "2026-07-01T22:00:00Z")],
)
completed = self.run_cli(
str(self.session_file), "-o", str(self.output_dir), check=False
)
self.assertEqual(completed.returncode, 2)
self.assertIn("reserved recovery report", completed.stderr)
self.assertFalse(self.output_dir.exists())
def test_parent_child_destination_collision_aborts_without_partial_output(
self,
) -> None:
write_jsonl(
self.session_file,
[
write_record("a", "parent", "2026-07-01T23:00:00Z"),
write_record("a/b", "child", "2026-07-01T23:01:00Z"),
],
)
completed = self.run_cli(
str(self.session_file), "-o", str(self.output_dir), check=False
)
self.assertEqual(completed.returncode, 2)
self.assertIn("ancestor collision", completed.stderr)
self.assertFalse(self.output_dir.exists())
def test_symlinked_session_directory_cannot_escape_file_history_root(self) -> None:
explicit_root = self.root / "declared-root"
outside = self.root / "outside" / SESSION_ID
outside.mkdir(parents=True)
explicit_root.mkdir()
backup_name = "escape@v1"
(outside / backup_name).write_bytes(b"outside")
(explicit_root / SESSION_ID).symlink_to(outside, target_is_directory=True)
write_jsonl(
self.session_file,
[
snapshot_record(
str(self.original), backup_name, 1, "2026-07-02T00:00:00Z"
)
],
)
completed = self.run_cli(
str(self.session_file),
"--file-history-root",
str(explicit_root),
"-o",
str(self.output_dir),
check=False,
)
self.assertEqual(completed.returncode, 2)
self.assertIn("session directory symlink", completed.stderr)
self.assertFalse(self.expected_output().exists())
def test_null_message_is_ignored_without_traceback(self) -> None:
write_jsonl(
self.session_file,
[
{"type": "assistant", "sessionId": SESSION_ID, "message": None},
write_record(str(self.original), "valid", "2026-07-02T01:00:00Z"),
],
)
completed = self.run_cli(str(self.session_file), "-o", str(self.output_dir))
self.assertEqual(self.expected_output().read_text(encoding="utf-8"), "valid")
self.assertNotIn("Traceback", completed.stderr)
def test_codex_rollout_fails_fast_with_supported_boundary(self) -> None:
write_jsonl(
self.session_file,
[
{
"type": "session_meta",
"timestamp": "2026-07-02T02:00:00Z",
"payload": {"id": SESSION_ID},
},
{
"type": "response_item",
"timestamp": "2026-07-02T02:00:01Z",
"payload": {
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "hello"}],
},
},
],
)
completed = self.run_cli(
str(self.session_file), "-o", str(self.output_dir), check=False
)
self.assertEqual(completed.returncode, 2)
self.assertIn("Codex rollout", completed.stderr)
self.assertIn("Claude Code JSONL sessions only", completed.stderr)
self.assertFalse(self.output_dir.exists())
def test_edit_summaries_drop_large_old_and_new_payloads(self) -> None:
records = [
edit_record(str(self.original), str(index), f"2026-07-02T03:00:0{index}Z")
for index in range(6)
]
write_jsonl(self.session_file, records)
with mock.patch.dict(
os.environ,
{
"HOME": str(self.user_home),
"CLAUDE_CONFIG_DIR": str(self.claude_home),
},
clear=False,
):
recovery = SessionContentRecovery(self.session_file, self.output_dir)
summaries = recovery.extract_edit_calls()
self.assertEqual(recovery.stats["edit_calls"], 6)
self.assertEqual(len(summaries), 5)
self.assertNotIn("old_string", summaries[0])
self.assertNotIn("new_string", summaries[0])
if __name__ == "__main__":
unittest.main()