SKILL.md
---
name: blackforge
description: >-
Answer crypto market-data questions with BlackForge. Use this skill whenever the user asks about
order-book or trade data for a coin on a spot venue — its latest stats on an exchange,
order-book depth or resting/pulled liquidity for a pair, taker buy-vs-sell volume, order-ladder
rungs, price-level lifetime, trade timing, outsized trades, or market-cap/attention enrichment;
when they want to pull, chart or compare a metric over a time range; or when they ask which pairs
a venue lists. Covers 9 spot exchanges (binance, bitget, bybit, coinbase, gate, kraken, kucoin,
mexc, okx) and ~11,800 spot pairs — 120 measurement columns per pair per closed
5-minute window. Trigger even when BlackForge is not named but the question is
about crypto market data on a venue. Drives the BlackForge MCP tools (blackforge_catalog /
blackforge_symbols / blackforge_latest / blackforge_series / blackforge_usage) or the `blackforge`
CLI; never reimplements the API. Every returned column is a measurement, not a trade call.
---
# BlackForge market-data
BlackForge is a **raw market-data** product: for every `(exchange, symbol)` it stores one wide row
per closed **5-minute window**, **120 measurement columns** across ~11,800 spot pairs —
order-book depth, resting-liquidity dynamics, trade-flow and trade-timing measurements, plus
market-cap and attention enrichment, and a per-row quality bitmask. This skill lets you answer a
plain-language market-data question by calling BlackForge's own tools and reading the rows back
**as measurements**.
You are a thin orchestration + interpretation layer. **Never** build HTTP requests, curl the API,
or hardcode an endpoint URL. Always go through the MCP tools or the `blackforge` CLI. Your job is to
know the vocabulary (which metric answers which question), run the right call, and explain the
numbers correctly.
## What BlackForge is — and is not
It is a measurement feed. Each column has a precise definition (e.g. *"resting sell liquidity from
the best ask up to +100%"*, *"quote notional that left the bid side of the book"*, *"median
lifetime of a price level created and removed inside the window"*). Present results in exactly that
register: **a measurement with a definition and a unit**.
It describes what happened in the book and on the tape — it does not tell the user what will happen
next or what to trade. Do not describe any column, or the data as a whole, using the words **signal,
pump, anomaly, probability-scored, alpha, prediction, detection, or alert**, and do not imply
the data forecasts or recommends anything. Say what was *measured* ("bid depth within 5% fell from X
to Y"), not what it *means for a trade*. This framing is the whole point of the skill.
**Where the line falls.** If the question contains a real market-data question wearing trading
clothes — "is there a big sell wall on DOGE, should I be worried?" — hold the framing and answer
with the measurement. But a request for a **recommendation with no data question inside it**
("should I buy ETH right now?") is **not** a BlackForge question: do not trigger on it, and do not
reach for market data to dress up an answer. Say plainly that you do not give trade advice, and
offer to show what the book and the tape actually measured if that would help.
("Flag" is the one exception, and only in its literal sense: `qualityFlags` is a real column and a
flagged *bucket* is a statement about data quality, never about the market.)
Also: never propose narrowing the venue or coin universe to save cost — the full universe is the
product.
## The playbook: discover → pick → call → interpret
### 1. Discover first — never guess identifiers
Before any keyed query, call **`blackforge_catalog`** (CLI: `blackforge catalog`). It is keyless and
returns the 9 venues (each with its `minPlan`) and all 120 metrics with `key`, `label`, `unit`,
`family`, `description`, `howToRead` and `minPlan`. Use it to resolve:
- the exact **`exchange`** identifier (lowercase: `binance`, `okx`, …), and
- the exact **`metric`** key the user's words map to (e.g. "resting depth"/"sell wall"/"pulled
liquidity" → the right `downDepth*` / `upDepth*` / `bidLiqRemoved` … key).
**Some words have no key.** There is no spread column — the catalog has `bestBid` and `bestAsk`,
and a spread is something YOU derive from two `blackforge_series` calls. When the catalog has no
key for what was asked, say so and offer what it does measure. Never answer a spread question
with a depth number: depth is resting size, not the distance between the two sides.
Never invent a metric key or a venue name. If you already hold a recent catalog in the conversation
you may reuse it, but when unsure, re-fetch — it is cheap and keyless. For a compact index of every
metric grouped by family with its one-line measurement definition, read
[`references/metrics-glossary.md`](references/metrics-glossary.md); the live catalog wording is
canonical when they differ.
To list the pairs a venue trades, call **`blackforge_symbols({exchange})`**
(CLI: `blackforge symbols --exchange <v>`). Symbol format is the venue's own
(`BTCUSDT` on binance, `BTC-USDT` on okx/coinbase) — confirm via symbols rather than assuming.
### 2. Pick the right tool for the shape of the question
| The user wants… | Call | Notes |
|---|---|---|
| a coin's **latest** stats on a venue (one snapshot) | `blackforge_latest({exchange, symbol, columns?})` | returns `{ ts, values }` for the last complete bucket **at the caller's plan granularity** — `5m` on max/ultra, **`1h` on pro, `1d` on free**. On the coarser tiers `ts` is the bucket start, so a free key's "latest" can be a day old. Nothing in the response says which granularity you got, so state the bucket length you are reading. Pass `columns` (metric keys) to keep the answer focused; omit for the full row. |
| how a metric **moved over a time range** | `blackforge_series({exchange, symbol, metric, from, to, interval})` | returns `{ points: [{ ts, value }] }`, `ts` in epoch ms. One metric per call. |
| **which pairs** a venue lists | `blackforge_symbols({exchange})` | |
| **usage / quota** left | `blackforge_usage()` | recent daily usage + rows remaining this month. |
CLI fallback maps 1:1: `blackforge latest …`, `blackforge series …`, `blackforge symbols …`,
`blackforge usage`. Prefer `--output json` when you will parse the result.
**Choosing `interval` for a series.** The only valid values are **`5m`, `1h`, `1d`** — anything
else 400s. The interval is **plan-gated as well as size-gated**: asking finer than your plan's floor
returns a **403**, not fewer points. **`5m` is max/ultra only; `pro` floors at `1h`; `free` floors at
`1d`.** Pick the coarsest interval that answers the question, and on a 403 step one rung coarser
(`5m` → `1h` → `1d`) rather than reporting no data. Guard the 50k-point cap — points ≈ span ÷ interval:
- hours to a few days → `5m` on max/ultra · `1h` on pro · `1d` on free
- about a week to a month → `1h` on max/ultra and pro · `1d` on free
- multiple months → `1d` (every plan)
`from`/`to` are ISO-8601 UTC. If the user says "last week", compute the range from today and state
the window you used. If a single call would exceed ~50k points, widen the interval or split the range.
### 3. Interpret the rows as measurements
When you present numbers, define each column with its catalog `description` / `howToRead` wording
(or the glossary). Convert quote-relative values to USD when helpful by multiplying by
`quoteUsdRate` (units are documented per metric). Anchor `ts` on the timeline. Compare windows in
plain measurement terms — "taker-buy volume was 2.3× taker-sell volume", "median resting-level
lifetime dropped from 4.1s to 0.6s" — and stop there. Do not translate a measurement into a buy/sell
call or label it with any banned word.
**Always read `qualityFlags`.** It is the one column that qualifies every other column on the row,
it is free on every plan, and it is deliberately queryable — request it alongside whatever else you
ask for. It is a **bitmask**: `0` means no known problem, and each set bit names one condition. The
full bit table ships on the catalog entry for `qualityFlags` as `bits`, and each bit carries a
`contaminates` list of the metric families it calls into question — so a broken order book leaves
the trade columns on the same row sound. Read the bit table from the catalog rather than hardcoding
bit numbers.
Nothing in a row is ever hidden, filtered or nulled. Every value is exactly as measured; the flags
tell you which of them to trust. Two companion columns are worth requesting with it:
- **`lastTradeAgeTime`** — how long before the window closed the pair last traded, `0` when the
window itself contained a trade. About half of all windows contain no trade, and their candle
carries the last traded price forward rather than inventing one. A large value means the price is
real but old.
- **`bookObservedAt`** — the instant the book was actually read, which is later than the window
close by a different amount on each venue. Use it, not `ts`, to line two venues up.
**The `QUALITY_UNKNOWN` flag (mask 32768) is not a defect.** It means the row predates the quality
rail and was never assessed — **unchecked, not unreliable**. It is the ClickHouse column default, so
the entire pre-migration-006 archive carries it. Say "not assessed", never "bad data".
Where a chart draws this, the convention is: **wherever the mark is fainter or hollow, that bucket
is flagged; solid means final.**
**Five columns that 400 the WHOLE request if you name them in `columns=`.** `quoteAsset`,
`baseAsset`, `enrichmentTs`, `bookSynced` and `missingTrades` are identity/state fields, not data
series. Naming any one of them fails the entire call — the columns you actually wanted included —
with `Unknown metric(s): …`. They arrive on their own in a full response; just never ask for them
by name. Use `qualityFlags` for the `bookSynced` / `missingTrades` concerns.
`bookAgeTime` and `seedDepth` are the opposite case: `internal: true`, they measure our collector
rather than the market, and the API **accepts-and-ignores** them, as it does the structural keys
`ts`, `exchange`, `symbol` and `ingestedAt`. Requesting those six is harmless.
### 4. Handle entitlements gracefully — omitted ≠ nonexistent
Entitlements (venues, columns, granularity, history depth) are enforced **server-side by plan**.
Three things to recognise and explain:
- A response header **`X-BlackForge-Columns-Omitted`** (or simply missing expected columns) means
those columns sit **above the caller's plan** and were dropped — the data exists, the key just
doesn't include it. Tell the user which tier includes them and point to **blackforge.so/pricing**.
Never report it as "there is no data for that".
- A **`403`** on a venue or interval means the same at the request level (e.g. a `pro`-only venue on
a free key, or a `5m` interval on a pro key, whose floor is `1h`). Explain the plan gap and the
upgrade path. There is **no `1m` interval** — do not go looking for a plan that unlocks one.
- **History depth is clamped SILENTLY — there is no header and no error.** If you ask for a `from`
earlier than the plan's window, the API quietly moves it forward to the plan's floor and returns
a shorter `points` array. Nothing in the response says it happened, so a short series is
ambiguous: it may be the plan's window, not the end of the data.
**Never narrate this as retention.** "BlackForge only has data going back two weeks" is wrong and
is the single easiest mistake to make here. **Retention is infinite — nothing is ever deleted.**
The window is an *entitlement*: how far back this key may read. Compare the first timestamp you
got against the `from` you asked for, and when it moved, say so — "your plan reads back 2 weeks,
so the series starts there; the archive itself goes back further" — then point at
**blackforge.so/pricing**.
`blackforge_usage` / `X-BlackForge-Rows-Remaining` tell you the monthly quota left; if a call fails
for quota, say so plainly.
### 5. Prefer MCP, fall back to CLI, else help them set up
1. If the **`blackforge_*` MCP tools** are available, use them — this is the primary path.
2. Otherwise, if the **`blackforge` CLI** is installed (or `npx -y @blackforge-so/cli` is usable), shell
out to it and parse `--output json`.
3. If neither exists, don't hand-roll API calls — tell the user how to set one up and point them to
[`references/setup.md`](references/setup.md) (MCP config block, CLI install, and where to get a
key at app.blackforge.so → API).
## Worked examples
**"What's the resting depth for ETH on Binance right now?"**
→ `blackforge_catalog` to confirm `binance` and the depth metric keys → `blackforge_symbols` if
unsure of the symbol (`ETHUSDT`) → `blackforge_latest({exchange:'binance', symbol:'ETHUSDT',
columns:['price','downDepth5','downDepth10','upDepth30','upDepth100','qualityFlags']})`. Report each as its
measurement: "bid depth within −5% of top-of-book: \$X; ask depth to +30%: \$Y", noting they are
resting-liquidity sums in the quote currency at the last complete bucket for that key's plan
(5 min on max/ultra, 1 h on pro, 1 d on free), and reading `qualityFlags` before trusting them.
**"Chart the bid-ask spread for BTC-USDT on OKX last week."**
→ catalog — **there is no spread key.** Say so, then derive it: two `blackforge_series` calls
(`bestAsk` and `bestBid`) over the same range and subtract point by point. Use `interval:'1h'` for a
week (it is inside every paid plan's floor and stays well under the point cap; `5m` needs max/ultra
and 7 days of it approaches the cap anyway; a free key gets `1d`). State the window and the interval
you actually used, and describe the line as the measured quantity over time, not as a trade cue.
**"Compare taker buy vs sell volume for SOL on Binance today."**
→ two `blackforge_series` calls (`buyTradeVol`, `sellTradeVol`) or one `blackforge_latest` with both
columns → present the ratio as measured aggressor balance.
## References
- [`references/metrics-glossary.md`](references/metrics-glossary.md) — all 120 metrics grouped by
family, each with its one-line measurement definition and `min plan`. Read it to map the user's
words to the right `metric` key and to explain a column.
- [`references/setup.md`](references/setup.md) — how to configure the MCP server or install the CLI,
and where to get an API key.
LICENSE
MIT License
Copyright (c) 2026 BlackForge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
references/setup.md
# Setting up access to BlackForge
This skill is a thin orchestration layer. It does not talk to the API directly — it drives one of
two clients that the user installs once. Read this when the user has neither the MCP tools nor the
`blackforge` CLI available, or asks how to get set up.
## Install the skill
The skill is distributed from GitHub and installed with the
[`skills`](https://github.com/vercel-labs/skills) CLI (GitHub is the registry):
```bash
npx skills add blackforge-so/skill # into the current project's skills dir
npx skills add blackforge-so/skill -g # or globally (user-level)
```
`skills` auto-detects the agent and installs into whichever skills directory it uses — `.claude/skills/`
or `.agents/skills/`. It works with any skills-compatible agent (Claude Code, Cursor, and others).
`npx skills list` shows what is installed. This is only the skill itself; you still need a data client
(MCP or CLI) and an API key, below.
## Get an API key
Every keyed call needs a **BlackForge API key**. Get one at **app.blackforge.so → API** (create a
key, copy the `bf_…` string). The key's plan decides which venues, columns and granularities are
returned, and the three gates behave **differently**:
- **Columns** degrade silently — unentitled ones are dropped and an `X-BlackForge-Columns-Omitted`
header names them. Not an error.
- **Venues and intervals** are a hard **403** before any data is read (e.g. `5m` on a pro key,
whose floor is `1h`).
- **History depth** is clamped **silently and unheadered** — a `from` earlier than the plan's
window is moved forward and you simply get fewer points, with nothing saying so.
See the pricing tiers at blackforge.so/pricing.
The public API base is `https://api.blackforge.so/v1`. **Note the split:** `/v1` is part of the
route, not of the configured origin — see `BLACKFORGE_BASE_URL` below.
---
## Option A — MCP server (preferred)
The MCP server exposes the five `blackforge_*` tools this skill calls directly. Once configured,
the tools appear automatically and no shelling out is needed. Add a `blackforge` server that runs
`npx -y @blackforge-so/mcp` with `BLACKFORGE_API_KEY` in its environment. The config lives wherever
your agent keeps MCP servers — for example:
**JSON config** (e.g. an MCP-enabled desktop app or a project `.mcp.json`):
```json
{
"mcpServers": {
"blackforge": {
"command": "npx",
"args": ["-y", "@blackforge-so/mcp"],
"env": { "BLACKFORGE_API_KEY": "bf_your_key_here" }
}
}
}
```
**Via a CLI helper** (e.g. an agent that exposes an `mcp add` command):
```bash
<your-agent> mcp add blackforge --env BLACKFORGE_API_KEY=bf_your_key_here -- npx -y @blackforge-so/mcp
```
Restart the agent. The tools become available as:
| tool | purpose | key params |
|---|---|---|
| `blackforge_catalog` | venues + 120 metric definitions | *(keyless — call first)* |
| `blackforge_symbols` | pairs a venue trades | `exchange` |
| `blackforge_latest` | latest closed 5-min bucket | `exchange`, `symbol`, `columns?` |
| `blackforge_series` | a metric over a time range | `exchange`, `symbol`, `metric`, `from`, `to`, `interval` |
| `blackforge_usage` | recent usage + rows remaining | *(none)* |
Optional env `BLACKFORGE_BASE_URL` overrides the **origin** for a local/dev server
(e.g. `http://localhost:3001/api`). It is the origin, **not** the `/v1` base above: both clients
append `/v1/...` themselves, so setting it to `https://api.blackforge.so/v1` makes every request
hit `/v1/v1/...` and 404. The default is `https://api.blackforge.so`.
---
## Option B — `blackforge` CLI (fallback)
Use this when the MCP tools are not configured but a shell is available. No install step is
required — `npx` fetches it on demand:
```bash
npx -y @blackforge-so/cli catalog
```
Or install once for the bare `blackforge` binary:
```bash
npm install -g @blackforge-so/cli
blackforge auth set-key bf_your_key_here # stored at ~/.blackforge/config.json (mode 0600)
```
The key is read from (in order) `--api-key`, `$BLACKFORGE_API_KEY`, then the stored config.
Commands mirror the MCP tools:
```bash
blackforge catalog # keyless: venues + metrics
blackforge symbols --exchange binance
blackforge latest --exchange binance --symbol BTCUSDT [--columns price,downDepth5,askLiqRemoved]
blackforge series --exchange binance --symbol BTCUSDT \
--metric downDepth5 --interval 1h \
--from 2026-07-01T00:00:00Z --to 2026-07-08T00:00:00Z --output json
blackforge usage
```
Global options: `--output table|json|csv` (default table on a TTY, json when piped), `--api-key`,
`--base-url`, `--verbose`. Add `--output json` when the result will be parsed rather than read.
---
## Response headers worth surfacing
Both clients pass through BlackForge's accounting headers. When present, use them to explain results:
- `X-BlackForge-Columns-Omitted` — columns dropped because they sit above the caller's plan. Tell
the user which tier includes them; do not report the data as missing.
- `X-BlackForge-Rows-Remaining` — rows left in the monthly quota for this key.
- `X-BlackForge-Rows-Served` / `X-BlackForge-Blocks-Billed` — what this call consumed.
A `403` on a venue/interval means the plan does not include it — point the user to blackforge.so/pricing.
README.md
# blackforge — agent skill
An [agent skill](https://github.com/vercel-labs/skills) that teaches any skills-compatible coding
agent (Claude Code, Cursor, and others) to answer crypto **market-data** questions by orchestrating
the [BlackForge](https://blackforge.so) MCP tools (preferred) or the `blackforge` CLI. It is a thin
orchestration + interpretation layer — it never reimplements the API.
BlackForge stores one wide row per `(exchange, symbol)` per closed 5-minute window — 120
measurement columns (order-book depth and depth walls, order-ladder rungs, resting-liquidity
add/withdraw, price-level lifetime, trade timing, outsized-trade counts, market-cap and attention
enrichment, and a per-row quality bitmask) across 9 spot exchanges and ~11,800 spot pairs. The skill knows
that vocabulary and the discover → pick → call → interpret playbook, and it frames every returned
column as a **measurement** with a definition, never as a trade call.
## Install
Install with the [`skills`](https://github.com/vercel-labs/skills) CLI — GitHub is the registry:
```bash
npx skills add blackforge-so/skill # into this project's skills dir
npx skills add blackforge-so/skill -g # or globally (user-level)
```
`skills` installs into whichever skills directory your agent uses (`.claude/skills/` or
`.agents/skills/`) and detects the agent automatically. Then configure access — either the
BlackForge MCP server (preferred) or the `blackforge` CLI — and get an API key at
**app.blackforge.so → API**. See [`references/setup.md`](references/setup.md).
## Contents
| Path | What |
|---|---|
| `SKILL.md` | The skill: frontmatter trigger `description` + the playbook |
| `references/metrics-glossary.md` | All 120 metrics grouped by family, each with its measurement definition and min plan |
| `references/setup.md` | How to install the skill, configure the MCP server or CLI, and get an API key |
| `scripts/latest-json.sh` | Optional CLI wrapper: dump the latest bucket for a pair as JSON |
| `scripts/check-catalog-sync.mjs` | Fails if these docs disagree with the live catalog — run it before publishing |
| `evals/trigger-eval.json` | Trigger eval set (should / should-not queries) for description tuning |
## Source of truth
This GitHub repo is the versioned source. `npx skills add blackforge-so/skill` installs a copy into
your agent's skills directory — regenerate it from here rather than editing the installed copy.
**Run this before you publish a change:**
```bash
node scripts/check-catalog-sync.mjs
```
It reads the public catalog — no key, no sibling checkout — and fails if the glossary has gained
or lost a metric, if any documented unit or min plan disagrees with the live one, if a family's
count is wrong, or if any prose here states a column count that is not real. It exits non-zero
when the catalog cannot be reached, rather than reporting success against a source it never read.
This exists because the count has drifted three separate times across the product, and because a
glossary that has silently lost one row looks exactly like a complete one. `baseAsset` went
undocumented here from the day migration 007 added it until 2026-07-28.
.claude-plugin/plugin.json
{
"name": "blackforge",
"displayName": "BlackForge",
"description": "Crypto spot market data across 9 venues: order book depth, resting-liquidity lifetimes, taker buy/sell flow and trade timing, per pair per closed 5-minute window.",
"version": "0.1.0",
"author": {
"name": "BlackForge",
"url": "https://blackforge.so"
},
"homepage": "https://blackforge.so",
"repository": "https://github.com/blackforge-so/skill",
"license": "MIT",
"keywords": [
"crypto",
"market-data",
"order-book",
"microstructure",
"finance",
"trading"
],
"mcpServers": {
"blackforge": {
"command": "npx",
"args": ["-y", "@blackforge-so/mcp"]
}
}
}
references/metrics-glossary.md
# BlackForge metrics glossary
Every column BlackForge returns is a **measurement of the order book or trade tape** over one
closed 5-minute window for one `(exchange, symbol)`. There are **120 catalog metrics**, and a full
row on the top tier carries all 120 — keys, quote/USD conversion fields and quality markers
included. (The ClickHouse table has 122 physical columns; `bookAgeTime` and `seedDepth` are
`internal: true`, measure our own collector rather than the market, and are never sold.) Describe
each column to the user using the measurement wording below — never as a score, a call, or an event
to act on.
**How to use this file.** Pick the `metric key` that matches what the user asked for and pass it
verbatim to `blackforge_series` (or `blackforge_latest`'s `columns`). Always reconcile against the
live `blackforge_catalog` output — plans, units and wording there are canonical; this table is a
fast index. `min plan` tells you the lowest tier that includes the column: if the caller's key is
below it the column comes back empty with an `X-BlackForge-Columns-Omitted` note (see SKILL.md).
**Units.** `quote` = the pair's quote currency (multiply by `quoteUsdRate` for USD); `base` = the
base coin; `price` = quote per base; `ms` = milliseconds; `seconds` = seconds (not ms — check the
unit before converting); `percent` = percent, already ×100, not a 0–1 ratio;
`count`/`index`/`ratio`/`bool`/`usd` as named. Those eleven are the whole set the API returns.
Columns marked quote-relative are raw in the quote currency.
**Families at a glance.** keys (4) · candle (4, price OHLC) · tradeFlow (10, taker buy/sell
aggression) · bookWalls (14, cumulative resting depth within a % band of top-of-book, plus how far
the book reaches) · orderLadders (14, resting depth in fixed 3%-wide slices) · bookMicro (11,
liquidity added and removed, level counts, level flicker, level lifetime) · tradeTiming (8,
silences, repeating intervals, same-size / same-instant groupings) · strong (19, counts and value
of outsized trades at size multiples) · enrichment (18, market-cap, rank, per-pair
CoinMarketCap liquidity/volume, plus the attention and developer-activity block) · context (9, BTC/ETH reference
prices, fear-and-greed, exchange app-store ranks, market-wide news rate) · quality (11 physical,
**9 sold** — the two internal ones are excluded).
> **The depth-band names are PERCENT, not basis points.** `upDepth30` is +30%, `upDepth400` is
> +400%, `downDepth5` is −5%. An earlier audit read them as bps and built a wrong conclusion on it.
> Asks use wide bands (+30…+400%) because alt asks are sparse; bids use tight ones (−5…−20%)
> because support sits near price.
> **bookWalls vs orderLadders.** A *wall* is cumulative depth from top-of-book out to a band edge
> (e.g. `upDepth100` = all resting asks up to +100%). A *ladder* rung is the depth inside one
> discrete 3% slice (e.g. `buyOrderVol6` = bids 3–6% below top). Walls measure total thickness;
> ladder rungs measure how that depth is distributed across price.
> **Read `qualityFlags` on every row.** It is free on every plan, deliberately queryable, and it is
> what qualifies every other column. `0` means no known problem; each set bit names one condition
> and carries a `contaminates` list. The `QUALITY_UNKNOWN` flag (mask 32768) means the row predates
> the quality rail — **unchecked, not unreliable**. Where a chart draws this: wherever the mark is fainter or hollow,
> that bucket is flagged; solid means final.
> **Five columns you must not request.** `bookSynced`, `missingTrades`, `quoteAsset`, `baseAsset`
> and `enrichmentTs` are **non-queryable**: naming any one in `columns=` returns a 400 for the whole
> request, the columns you actually wanted included. Nothing in the catalog marks them — all five
> are served at `minPlan: free`, and `plottable: false` is not the tell (`qualityFlags` and
> `bookObservedAt` are also `plottable: false` and both ARE queryable). They arrive on their own in
> a full row; just never ask for them by name. `qualityFlags` replaces the `bookSynced` and
> `missingTrades` concerns.
>
> `bookAgeTime` and `seedDepth` are the opposite case: internal, and deliberately
> **accepted-and-ignored** in `columns=` — harmless there, though they do 400 a `series` call.
## Keys & timestamps
_family key: `keys` · 4 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `exchange` | Exchange | index | free | The exchange the snapshot came from. |
| `symbol` | Symbol | index | free | The trading pair the snapshot describes. |
| `ts` | Snapshot time | ms | free | The time the 5-minute window closed. |
| `ingestedAt` | Ingested time | ms | free | The time the snapshot was written to storage. |
## Candle (price)
_family key: `candle` · 4 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `priceOpen` | Open price | price | free | The price at the start of the window. |
| `priceHigh` | High price | price | free | The highest price reached during the window. |
| `priceLow` | Low price | price | free | The lowest price reached during the window. |
| `price` | Close price | price | free | The last price of the window. |
## Trade flow (taker aggression)
_family key: `tradeFlow` · 10 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `buyTradeVol` | Buy trade volume | quote | free | Total value of taker-buy trades in the window. |
| `sellTradeVol` | Sell trade volume | quote | free | Total value of taker-sell trades in the window. |
| `buyTradeCount` | Buy trade count | count | free | Number of taker-buy trades in the window. |
| `sellTradeCount` | Sell trade count | count | free | Number of taker-sell trades in the window. |
| `buyTradePriceAvg` | Buy trade average price | price | free | Plain average price of taker-buy trades in the window. |
| `sellTradePriceAvg` | Sell trade average price | price | free | Plain average price of taker-sell trades in the window. |
| `buyTradeSizeAvg` | Buy trade average size | base | free | Average size of taker-buy trades in base coin units. |
| `sellTradeSizeAvg` | Sell trade average size | base | free | Average size of taker-sell trades in base coin units. |
| `buyTradeMax` | Largest buy trade | quote | free | Value of the single largest taker-buy trade in the window. |
| `sellTradeMax` | Largest sell trade | quote | free | Value of the single largest taker-sell trade in the window. |
## Book walls — cumulative depth bands
_family key: `bookWalls` · 14 metrics · band names are PERCENT_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `upDepth30` | Ask depth to +30% | quote | free | Resting sell liquidity from the best ask up to 30% above it. |
| `upDepth60` | Ask depth to +60% | quote | pro | Resting sell liquidity from the best ask up to 60% above it. |
| `upDepth100` | Ask depth to +100% | quote | pro | Resting sell liquidity from the best ask up to 100% above it. |
| `upDepth200` | Ask depth to +200% | quote | pro | Resting sell liquidity from the best ask up to 200% above it. |
| `upDepth300` | Ask depth to +300% | quote | max | Resting sell liquidity from the best ask up to 300% above it. |
| `upDepth400` | Ask depth to +400% | quote | max | Resting sell liquidity from the best ask up to 400% above it. |
| `upDepthFull` | Total ask depth | quote | max | All resting sell liquidity across the entire order book. |
| `downDepth5` | Bid depth to -5% | quote | free | Resting buy liquidity from the best bid down to 5% below it. |
| `downDepth10` | Bid depth to -10% | quote | pro | Resting buy liquidity from the best bid down to 10% below it. |
| `downDepth15` | Bid depth to -15% | quote | pro | Resting buy liquidity from the best bid down to 15% below it. |
| `downDepth20` | Bid depth to -20% | quote | pro | Resting buy liquidity from the best bid down to 20% below it. |
| `downDepthFull` | Total bid depth | quote | max | All resting buy liquidity across the entire order book. |
| `askDepthReachPct` | Ask book reach | percent | pro | How far above the best ask, **in percent**, the book we maintain actually reaches this window. It is the ceiling on every ask-side depth column: a band wider than this reach reports only the part of the book we hold. Deliberately "reach", not "coverage" — it states how far the book extends, it does not assert nothing is missing. Not comparable to the bid figure: the ask span is unbounded, so it is routinely enormous. |
| `bidDepthReachPct` | Bid book reach | percent | pro | How far below the best bid, **in percent**, the book we maintain actually reaches. Hard-bounded at 100% by the price floor and usually at or near it; well under 100 means the bid ladder ran out before the band you asked for. Not comparable to the ask figure. |
> ⚠️ **Known catalog bug — unit.** Both reach columns are PERCENT but the catalog declares
> `unit: 'ratio'`. Read them as percent (0–100+, not 0–1). Do not "fix" a value by multiplying by
> 100.
## Order ladders — fixed 3% slices
_family key: `orderLadders` · 14 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `buyOrderVol3` | Bid volume 0 to 3% | quote | free | Resting buy liquidity in the slice from 0 to 3% below top of book. |
| `buyOrderVol6` | Bid volume 3 to 6% | quote | free | Resting buy liquidity in the slice from 3 to 6% below top of book. |
| `buyOrderVol9` | Bid volume 6 to 9% | quote | pro | Resting buy liquidity in the slice from 6 to 9% below top of book. |
| `buyOrderVol12` | Bid volume 9 to 12% | quote | pro | Resting buy liquidity in the slice from 9 to 12% below top of book. |
| `buyOrderVol15` | Bid volume 12 to 15% | quote | pro | Resting buy liquidity in the slice from 12 to 15% below top of book. |
| `buyOrderVol18` | Bid volume 15 to 18% | quote | max | Resting buy liquidity in the slice from 15 to 18% below top of book. |
| `buyOrderVol21` | Bid volume 18 to 21% | quote | max | Resting buy liquidity in the slice from 18 to 21% below top of book. |
| `sellOrderVol3` | Ask volume 0 to 3% | quote | free | Resting sell liquidity in the slice from 0 to 3% above top of book. |
| `sellOrderVol6` | Ask volume 3 to 6% | quote | free | Resting sell liquidity in the slice from 3 to 6% above top of book. |
| `sellOrderVol9` | Ask volume 6 to 9% | quote | pro | Resting sell liquidity in the slice from 6 to 9% above top of book. |
| `sellOrderVol12` | Ask volume 9 to 12% | quote | pro | Resting sell liquidity in the slice from 9 to 12% above top of book. |
| `sellOrderVol15` | Ask volume 12 to 15% | quote | pro | Resting sell liquidity in the slice from 12 to 15% above top of book. |
| `sellOrderVol18` | Ask volume 15 to 18% | quote | max | Resting sell liquidity in the slice from 15 to 18% above top of book. |
| `sellOrderVol21` | Ask volume 18 to 21% | quote | max | Resting sell liquidity in the slice from 18 to 21% above top of book. |
## Book microstructure — liquidity add / remove / flicker / lifetime
_family key: `bookMicro` · 11 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `bestBid` | Best bid | price | free | The highest bid price at the moment the window closed. |
| `bestAsk` | Best ask | price | free | The lowest ask price at the moment the window closed. |
| `bookLevelChangeCount` | Book level change count | count | pro | Price-level changes recorded in the window: one per level whose size actually moved, additions and removals alike, both sides. A level re-broadcast at its existing size is not counted. Counting changes rather than messages makes it largely independent of how a venue batches its pushes, but a level born and removed between two pushes never arrives — read it as a **lower bound** on churn. The damping is partial: a residual of roughly 4.5× remains across venues. (This **replaces** `bookUpdateCount`, which counted applied WebSocket frames — a different measurement, not a rename.) |
| `bidLevelCount` | Bid level count | count | pro | Number of price levels on the bid side at window close. |
| `askLevelCount` | Ask level count | count | pro | Number of price levels on the ask side at window close. |
| `bidLiqAdded` | Bid liquidity added | quote | max | Buy-side resting liquidity placed into the book during the window, in quote units. |
| `bidLiqRemoved` | Bid liquidity removed | quote | max | **Gross** decrease in buy-side resting size across the window, with trades at the same price within 500 ms netted out (measured to remove under **0.3%** of the total). Read it as gross bid-side book decrease, not as cancellations alone: the trade netting is small enough that this is essentially all level-decrease notional. |
| `askLiqAdded` | Ask liquidity added | quote | max | Sell-side resting liquidity placed into the book during the window, in quote units. |
| `askLiqRemoved` | Ask liquidity removed | quote | max | **Gross** decrease in sell-side resting size across the window, with trades at the same price within 500 ms netted out (measured to remove under **0.3%** of the total). Read it as gross ask-side book decrease, not as cancellations alone: the trade netting is small enough that this is essentially all level-decrease notional. |
> **The liquidity family is not comparable across venues.** `maintainedDepth` differs 12.5× between
> venues (okx 400 · bitget 500 · bybit/kraken 1,000 · binance/coinbase/gate/mexc/kucoin 5,000), so these are per-window flows
> accumulated against books of very different size. Normalise per row before comparing venues —
> `liqAdded / bookLevelChangeCount` is the closed-form figure, and dividing by a book-size column on
> the same row (`upDepth30 + downDepth5`) also collapses most of the spread.
| `levelFlickerCount` | Level flicker count | count | max | Count of price levels that appeared, vanished, then reappeared within the window (a placed-removed-placed cycle). |
| `levelLifetimeMedianTime` | Median level lifetime | ms | max | Median lifetime, in ms, of price levels that were both created and removed inside the window. |
## Trade timing & cadence
_family key: `tradeTiming` · 8 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `tradeSilenceMaxTime` | Longest trade silence | ms | max | The longest gap between trades during the window. |
| `tradeGapModeTime` | Common trade interval | ms | max | The most frequent gap between consecutive trades. |
| `tradeGapModeCount` | Common interval count | count | max | How many trades followed the most common interval. |
| `sameQtyTradeCount` | Same-size trade count | count | max | Trades sharing an exact quantity in groups of three or more. |
| `sameQtyMaxCount` | Largest same-size group | count | max | The biggest group of trades sharing an exact quantity. |
| `atc` | Same-instant trade count | count | max | Trades sharing the same millisecond timestamp in clusters of three or more. |
| `atcMaxCluster` | Largest same-instant cluster | count | max | The biggest cluster of trades sharing one millisecond timestamp. |
| `ltc` | Loser round-trip count | count | max | Same-quantity buy then sell round-trips closed at a loss within 30 minutes. |
## Strong (outsized) trades
_family key: `strong` · 19 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `stc50` | Strong trade count 1.5x | count | pro | Trades at least 1.5 times the average trade size in the window. |
| `stc100` | Strong trade count 2x | count | pro | Trades at least 2 times the average trade size in the window. |
| `stc200` | Strong trade count 3x | count | max | Trades at least 3 times the average trade size in the window. |
| `sbc50` | Strong buy count 1.5x | count | pro | Taker-buy trades at least 1.5 times the average trade size. |
| `sbc100` | Strong buy count 2x | count | pro | Taker-buy trades at least 2 times the average trade size. |
| `sbc200` | Strong buy count 3x | count | max | Taker-buy trades at least 3 times the average trade size. |
| `sbc500` | Strong buy count 6x | count | max | Taker-buy trades at least 6 times the average trade size. |
| `ssc50` | Strong sell count 1.5x | count | pro | Taker-sell trades at least 1.5 times the average trade size. |
| `ssc100` | Strong sell count 2x | count | pro | Taker-sell trades at least 2 times the average trade size. |
| `ssc200` | Strong sell count 3x | count | max | Taker-sell trades at least 3 times the average trade size. |
| `ssc500` | Strong sell count 6x | count | max | Taker-sell trades at least 6 times the average trade size. |
| `sbcVol50` | Strong buy volume 1.5x | quote | pro | Total value of taker-buy trades at least 1.5 times the average size. |
| `sbcVol100` | Strong buy volume 2x | quote | pro | Total value of taker-buy trades at least 2 times the average size. |
| `sbcVol200` | Strong buy volume 3x | quote | max | Total value of taker-buy trades at least 3 times the average size. |
| `sbcVol500` | Strong buy volume 6x | quote | max | Total value of taker-buy trades at least 6 times the average size. |
| `ssVol50` | Strong sell volume 1.5x | quote | pro | Total value of taker-sell trades at least 1.5 times the average size. |
| `ssVol100` | Strong sell volume 2x | quote | pro | Total value of taker-sell trades at least 2 times the average size. |
| `ssVol200` | Strong sell volume 3x | quote | max | Total value of taker-sell trades at least 3 times the average size. |
| `ssVol500` | Strong sell volume 6x | quote | max | Total value of taker-sell trades at least 6 times the average size. |
## Enrichment (market-cap / attention / developer activity)
_family key: `enrichment` · 18 metrics_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `cgMarketCap` | CoinGecko market cap | usd | free | The coin market capitalisation reported by CoinGecko. |
| `cgRank` | CoinGecko rank | index | free | The coin market-cap rank on CoinGecko. |
| `cgAprox` | CoinGecko match confidence | usd | pro | How closely the pair price matched a CoinGecko candidate. |
| `cmcMarketCap` | CoinMarketCap market cap | usd | pro | The coin market capitalisation reported by CoinMarketCap. |
| `cmcDilutedMc` | CoinMarketCap diluted cap | usd | pro | The fully diluted market cap reported by CoinMarketCap. |
| `cmcSelfMc` | CoinMarketCap self-reported cap | usd | pro | The self-reported market cap from CoinMarketCap. |
| `cmcRank` | CoinMarketCap rank | index | pro | The coin market-cap rank on CoinMarketCap. |
| `cmcLiquidity` | CoinMarketCap liquidity | usd | pro | The effective liquidity for the pair from CoinMarketCap. |
| `cmcVolume` | CoinMarketCap volume | usd | pro | The 24-hour trading volume for the pair from CoinMarketCap. |
| `cgTrendingRank` | CoinGecko trending rank | index | max | The coin's position in CoinGecko's current trending list. |
| `watchlistUsers` | Watchlist users | count | max | How many CoinGecko users hold the coin in a watchlist portfolio. |
| `sentimentUpPct` | Community up-vote share | ratio | max | The share of CoinGecko community up/down votes that are 'up', as a percentage. |
| `githubCommits4w` | GitHub commits (4 weeks) | count | max | Commits to the project's linked GitHub repositories in the last four weeks. |
| `githubStars` | GitHub stars | count | max | Stars on the project's linked GitHub repositories. |
| `githubContributors` | GitHub contributors | count | max | Distinct pull-request contributors to the project's linked GitHub repositories. |
| `newsCount24h` | News articles (24h) | count | max | News articles about the coin in the last 24 hours. |
| `videoCount24h` | Videos (24h) | count | max | Videos about the coin published in the last 24 hours. |
| `aiVisibility` | AI visibility (experimental) | count | max | The size of an AI assistant's generated answer about the coin (experimental). |
## Market context
_family key: `context` · 9 metrics_
These describe the market, not the pair — the same value is stamped on every row in the window.
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `btcPriceUsd` | Bitcoin price | price | free | The reference Bitcoin price in USD at the snapshot time. |
| `ethPriceUsd` | Ethereum price | price | free | The reference Ethereum price in USD at the snapshot time. |
| `fearGreed` | Fear and greed index | index | max | The market-wide crypto fear and greed reading. |
| `fearGreedCmc` | Fear and greed index (CMC) | index | max | The CoinMarketCap version of the fear and greed reading. |
| `coinbaseAppRank` | Coinbase app-store rank | index | max | Coinbase's position in the Finance chart of its app store. |
| `coinbaseAppRankRegion` | Coinbase app-rank region | index | max | The app-store storefront region the Coinbase rank was measured in (a code such as `us`). |
| `binanceAppRank` | Binance app-store rank | index | max | Binance's position in the Finance chart of its app store. |
| `binanceAppRankRegion` | Binance app-rank region | index | max | The app-store storefront region the Binance rank was measured in (a code such as `tr`). |
| `cryptoNewsPerHour` | Crypto news arrival rate | count | max | How fast crypto news articles are being published market-wide, in articles per hour. |
> `searchInterest` was **retired in migration 005** — it was never populated. It does not exist; do
> not request it.
## Quality & units (data-integrity fields)
_family key: `quality` · 11 physical columns, **9 sold**_
| metric key | label | unit | min plan | measurement |
|---|---|---|---|---|
| `qualityFlags` | Row quality flags | count (bitmask) | free | A bitmask of everything known to be wrong with this row; each bit is one named condition, `0` = no known problem. The catalog entry ships the full flag table as `bits`, each with the metric families it `contaminates` — so a broken order book leaves the trade columns on the same row sound. Nothing in the row is hidden, filtered or nulled — this column tells you which values to trust. Name the flag, never a bit number: `PRICE_FROM_LAST_TRADE` marks a window whose price is carried forward from an earlier trade; `QUALITY_UNKNOWN` (mask 32768) means the row predates the quality rail and was never assessed — **unchecked, not unreliable** — and is the ClickHouse column default, so the whole pre-migration-006 archive carries it. |
| `lastTradeAgeTime` | Last trade age | seconds | free | How long before the window closed the pair last traded, in seconds; `0` when the window contained a trade. A large value means the price is real but old. Read it with the `PRICE_FROM_LAST_TRADE` quality flag. |
| `bookObservedAt` | Book observation time | ms | free | The instant the order book was actually read for this row — later than the window close, by a different amount on each venue. This, not `ts`, is when the depth and best bid/ask were true. Use it to line two venues up. |
| `baseAsset` | Base asset | index | free | **Non-queryable — naming it in `columns=` 400s the whole request.** The base asset of the pair — the coin being priced — as the venue itself spells it (kraken, for one, writes bitcoin `XBT` and dogecoin `XDG`). Use it to group or match a coin across venues without splitting the symbol string, which every venue formats differently. Read it with `quoteAsset` to name the full instrument. |
| `quoteAsset` | Quote asset | index | free | **Non-queryable — naming it in `columns=` 400s the whole request.** The currency the pair is quoted in. |
| `quoteUsdRate` | Quote to USD rate | ratio | free | The rate to convert the quote asset into USD. |
| `enrichmentTs` | Enrichment time | ms | free | **Non-queryable — naming it in `columns=` 400s the whole request.** The time the enrichment data was captured. |
| `bookSynced` | Book synced | bool | free | **Non-queryable — do not name it in `columns=`; it 400s the whole request.** When false, read the depth and wall figures on that row as not final. Use `qualityFlags`. |
| `missingTrades` | Missing trades | bool | free | **Non-queryable — naming it in `columns=` 400s the whole request.** Whether some trades may have been missed in the window. Use `qualityFlags`. |
**Not sold — `internal: true`, catalogued but never served:**
| metric key | why |
|---|---|
| `bookAgeTime` | Time since the book was last **re-seeded**, and **larger is healthier** — it is preserved across a warm handover, so a big value means a long uninterrupted run. Any reading of it as a data-staleness or freshness indicator is backwards. It measures our collector, not the market. |
| `seedDepth` | The number of levels the book was seeded with — again a property of our collector. |
Both are accepted-and-ignored if you name them, so a request does not fail, but no value comes
back. Use `qualityFlags` for anything you would have asked them.
.gitignore
# transient eval/benchmark artifacts live in the sibling workspace, not here
/skill-workspace/
*.log
.DS_Store
evals/trigger-eval.json
[
{"query": "what's the resting bid depth for ETH on binance right now, like within 5% of the top of book?", "should_trigger": true},
{"query": "pull the 5-minute spread and order book depth for BTC-USDT on okx for the last week and chart it", "should_trigger": true},
{"query": "compare taker buy vs sell volume for SOLUSDT on binance over the last 24h", "should_trigger": true},
{"query": "which pairs does kraken list on blackforge? i need everything quoted in usdt", "should_trigger": true},
{"query": "how much sell-side liquidity got pulled from the DOGE book on bybit yesterday vs what actually traded", "should_trigger": true},
{"query": "give me the latest market-cap and rank plus book depth for PEPE on okx", "should_trigger": true},
{"query": "i want to compare the median resting price-level lifetime for AVAX across binance and coinbase last month", "should_trigger": true},
{"query": "how many rows do i have left this month on my blackforge key and what did i use recently", "should_trigger": true},
{"query": "grab the order ladder rungs (0-3%, 3-6% etc) for the LINKUSDT bid side on gate right now", "should_trigger": true},
{"query": "show me trade timing stats - longest gap between trades and the strong outsized trade counts for XRP on kucoin today", "should_trigger": true},
{"query": "what's the current price of bitcoin in dollars?", "should_trigger": false},
{"query": "explain how a limit order book works in general, i'm new to trading", "should_trigger": false},
{"query": "write me a python script that connects to the binance websocket and prints trades", "should_trigger": false},
{"query": "should i buy ETH right now? give me a trade recommendation", "should_trigger": false},
{"query": "what's the market cap of the whole crypto market today according to coingecko?", "should_trigger": false},
{"query": "set up a postgres database to store my own crypto candles that i scrape", "should_trigger": false},
{"query": "deploy the blackforge collector to azure and run the blue green swap", "should_trigger": false},
{"query": "translate this solidity smart contract to explain what it does", "should_trigger": false},
{"query": "what were the SEC's latest rulings on crypto exchanges this year?", "should_trigger": false},
{"query": "help me pick a hardware wallet for storing my bitcoin safely", "should_trigger": false}
]
scripts/latest-json.sh
#!/usr/bin/env bash
# latest-json.sh — dump the latest closed 5-minute bucket for one pair as JSON.
#
# This is an OPTIONAL convenience wrapper around the `blackforge` CLI for the common
# "give me the latest row for a pair" task. It shells out to the CLI on purpose — the
# skill never talks to the API directly. Prefer the blackforge_latest MCP tool when the
# MCP server is configured; use this only on the CLI fallback path.
#
# Usage:
# scripts/latest-json.sh <exchange> <symbol> [col1,col2,...]
# Examples:
# scripts/latest-json.sh binance BTCUSDT
# scripts/latest-json.sh okx BTC-USDT price,downDepth5,askLiqRemoved
#
# Auth comes from the CLI's own resolution (--api-key option > $BLACKFORGE_API_KEY >
# ~/.blackforge/config.json). Set BLACKFORGE_API_KEY or run `blackforge auth set-key` first.
set -euo pipefail
exchange="${1:?usage: latest-json.sh <exchange> <symbol> [columns]}"
symbol="${2:?usage: latest-json.sh <exchange> <symbol> [columns]}"
columns="${3:-}"
# Resolve the CLI: installed binary if present, else npx.
if command -v blackforge >/dev/null 2>&1; then
bf=(blackforge)
else
bf=(npx -y @blackforge-so/cli)
fi
args=(latest --exchange "$exchange" --symbol "$symbol" --output json)
[ -n "$columns" ] && args+=(--columns "$columns")
exec "${bf[@]}" "${args[@]}"
scripts/check-catalog-sync.mjs
#!/usr/bin/env node
/**
* Fail if this skill's documentation disagrees with the live BlackForge catalog.
*
* WHY THIS EXISTS
* ---------------
* The sold-column count has now drifted THREE separate times across this product — the Stripe
* products said 69/117, the landing site said 38/72/119, and this skill said 119 — every time
* because the number was typed by hand into prose and nothing could tell it had gone stale.
* Worse, `references/metrics-glossary.md` claims to document every catalog metric and had
* silently lost `baseAsset` when migration 007 added it: a reader could not tell, because a
* glossary missing one row looks exactly like a glossary that is complete.
*
* Prose cannot be generated — the measurement wording is written by hand and that is the point
* of the file. So it is CHECKED instead. This script is the artefact that makes the drift
* visible, and it fails loudly rather than skipping.
*
* node scripts/check-catalog-sync.mjs
*
* It reads the PUBLIC catalog (no key, no auth, no sibling checkout), so it works anywhere with
* a network connection and cannot quietly pass by finding nothing to compare against. If the
* catalog cannot be reached it EXITS NON-ZERO rather than reporting success — an unreachable
* source is an unknown answer, not a passing one.
*
* WHAT IT CANNOT CHECK, AND WHY
* -----------------------------
* Which columns are QUERYABLE. Five of them — quoteAsset, baseAsset, enrichmentTs, bookSynced,
* missingTrades — 400 the entire `/v1/latest` request if named in `columns=`, and the catalog
* does not say so: it exposes `plottable`, which is a different idea (bookObservedAt is
* queryable but not plottable). That set lives only in `api/src/metrics/metrics.service.ts`
* (NON_QUERYABLE_KEYS), so no client can discover it without reading the API's source or
* hitting the 400. The glossary's "Non-queryable" flags were reconciled against that file by
* hand on 2026-07-28 and will silently rot if it changes. Exposing `queryable` on the catalog
* would let this script check them like everything else.
*/
const CATALOG_URL = process.env.BF_CATALOG_URL ?? 'https://api.blackforge.so/v1/catalog';
import { readFileSync } from 'node:fs';
import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
const root = resolve(dirname(fileURLToPath(import.meta.url)), '..');
const read = (rel) => readFileSync(resolve(root, rel), 'utf8');
/** `ultra` shares `max`'s set — nothing gates above `max`. */
const LADDER = ['free', 'pro', 'max'];
const problems = [];
const fail = (file, msg) => problems.push(`${file}: ${msg}`);
// ---------------------------------------------------------------------------
// 0. The live truth.
// ---------------------------------------------------------------------------
let metrics;
try {
const res = await fetch(CATALOG_URL);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
({ metrics } = await res.json());
if (!Array.isArray(metrics) || metrics.length === 0) throw new Error('no metrics in the response');
} catch (error) {
console.error(`catalog-sync: could not read ${CATALOG_URL} — ${error.message}`);
console.error('Refusing to report success against a source that could not be read.');
process.exit(1);
}
const TOTAL = metrics.length;
const columnsFor = (plan) => {
const ceiling = LADDER.indexOf(plan);
return metrics.filter((m) => LADDER.indexOf(m.minPlan ?? 'free') <= ceiling).length;
};
const LADDER_COUNTS = { free: columnsFor('free'), pro: columnsFor('pro'), max: TOTAL, ultra: TOTAL };
const liveFamilies = new Map();
for (const m of metrics) liveFamilies.set(m.family, (liveFamilies.get(m.family) ?? 0) + 1);
// The two internal columns are catalogued but never served, so they are absent from this
// public response. They both live in `quality`, which is why quality's physical count is two
// higher than its sold count and no other family's is.
const INTERNAL = ['bookAgeTime', 'seedDepth'];
const PHYSICAL = TOTAL + INTERNAL.length;
// ---------------------------------------------------------------------------
// 1. The glossary table must cover exactly the catalog — in BOTH directions.
// ---------------------------------------------------------------------------
const glossaryFile = 'references/metrics-glossary.md';
const glossary = read(glossaryFile);
const rows = [];
for (const line of glossary.split('\n')) {
const m = line.match(/^\|\s*`([A-Za-z0-9_]+)`\s*\|([^|]*)\|([^|]*)\|([^|]*)\|/);
if (m) rows.push({ key: m[1], label: m[2].trim(), unit: m[3].trim(), minPlan: m[4].trim() });
}
const documented = new Map(rows.map((r) => [r.key, r]));
const live = new Map(metrics.map((m) => [m.key, m]));
for (const key of [...documented.keys()].filter((k, i, a) => a.indexOf(k) !== i)) {
fail(glossaryFile, `\`${key}\` is documented more than once`);
}
for (const m of metrics) {
if (!documented.has(m.key)) {
fail(glossaryFile, `MISSING \`${m.key}\` (${m.family}/${m.unit}/${m.minPlan}) — the catalog serves it and this file claims to list every metric`);
}
}
for (const r of rows) {
if (!live.has(r.key) && !INTERNAL.includes(r.key)) {
fail(glossaryFile, `documents \`${r.key}\`, which the catalog does not serve`);
}
}
// A trailing parenthetical is editorial ("count (bitmask)"), so compare the leading token only.
const bare = (s) => s.replace(/\s*\(.*\)\s*$/, '').trim();
for (const r of rows) {
const c = live.get(r.key);
if (!c) continue;
if (bare(r.unit) !== c.unit) fail(glossaryFile, `\`${r.key}\` unit is "${r.unit}", live says "${c.unit}"`);
if (r.minPlan !== (c.minPlan ?? 'free')) fail(glossaryFile, `\`${r.key}\` min plan is "${r.minPlan}", live says "${c.minPlan}"`);
}
// ---------------------------------------------------------------------------
// 2. The "Families at a glance" paragraph must match the live grouping.
// ---------------------------------------------------------------------------
for (const [family, count] of liveFamilies) {
const isQuality = family === 'quality';
// quality is written as "quality (N physical, **M sold** ...)" because the two internal
// columns sit in it; every other family is written as "family (N ...)".
const re = isQuality
? new RegExp(`${family}\\s*\\((\\d+)\\s+physical[^)]*?\\*\\*(\\d+)\\s+sold\\*\\*`, 'i')
: new RegExp(`\\b${family}\\s*\\((\\d+)`, 'i');
const m = glossary.match(re);
if (!m) {
fail(glossaryFile, `"Families at a glance" does not state a count for \`${family}\` in the expected form`);
continue;
}
if (isQuality) {
if (Number(m[1]) !== count + INTERNAL.length) fail(glossaryFile, `quality physical count is ${m[1]}, should be ${count + INTERNAL.length}`);
if (Number(m[2]) !== count) fail(glossaryFile, `quality sold count is ${m[2]}, should be ${count}`);
} else if (Number(m[1]) !== count) {
fail(glossaryFile, `family \`${family}\` is written as ${m[1]}, live has ${count}`);
}
}
// ---------------------------------------------------------------------------
// 3. Each family section's subtitle states that family's own count. Check it against
// that family, not against a global allowlist — otherwise "10 physical columns"
// under quality would sail through simply because tradeFlow happens to have 10.
// ---------------------------------------------------------------------------
const SUBTITLE_RE = /^_family key: `([A-Za-z]+)`[^\n]*$/gm;
const seenSubtitles = new Set();
for (const m of glossary.matchAll(SUBTITLE_RE)) {
const [subtitle, family] = m;
const line = glossary.slice(0, m.index).split('\n').length;
seenSubtitles.add(line);
const count = liveFamilies.get(family);
if (count === undefined) {
fail(`${glossaryFile}:${line}`, `section claims family \`${family}\`, which the catalog does not have`);
continue;
}
if (family === 'quality') {
const q = subtitle.match(/(\d+)\s+physical columns,\s*\*\*(\d+)\s+sold\*\*/);
if (!q) fail(`${glossaryFile}:${line}`, 'quality subtitle no longer states "N physical columns, **M sold**"');
else {
if (Number(q[1]) !== count + INTERNAL.length) fail(`${glossaryFile}:${line}`, `quality subtitle says ${q[1]} physical, should be ${count + INTERNAL.length}`);
if (Number(q[2]) !== count) fail(`${glossaryFile}:${line}`, `quality subtitle says ${q[2]} sold, should be ${count}`);
}
continue;
}
const n = subtitle.match(/·\s*(\d+)\s+metrics/);
if (!n) fail(`${glossaryFile}:${line}`, `\`${family}\` subtitle no longer states "· N metrics"`);
else if (Number(n[1]) !== count) fail(`${glossaryFile}:${line}`, `\`${family}\` subtitle says ${n[1]} metrics, live has ${count}`);
}
// ---------------------------------------------------------------------------
// 4. No prose anywhere may state a metric/column count that is not one of the real ones.
// ---------------------------------------------------------------------------
// Only counts that QUALIFY metrics/columns are checked, and the per-family subtitles above are
// skipped because step 3 already checked them against their own family. A units note reading
// "3%-wide slices" never matches, which is what keeps this specific enough to be worth having.
const PROSE_FILES = ['SKILL.md', 'README.md', 'references/setup.md', 'references/metrics-glossary.md'];
const COUNT_RE = /\b(\d+)\s+(?:catalog\s+|measurement\s+|metric\s+|physical\s+|sold\s+)?(?:metrics|columns|metric definitions)\b/g;
const ALLOWED = new Set([...Object.values(LADDER_COUNTS), PHYSICAL]);
for (const file of PROSE_FILES) {
const text = read(file);
for (const m of text.matchAll(COUNT_RE)) {
const n = Number(m[1]);
if (ALLOWED.has(n)) continue;
const line = text.slice(0, m.index).split('\n').length;
if (file === glossaryFile && seenSubtitles.has(line)) continue;
fail(`${file}:${line}`, `"${m[0].replace(/\s+/g, ' ')}" — not a real count. Live: ${TOTAL} sold, ${PHYSICAL} physical, ladder ${LADDER_COUNTS.free}/${LADDER_COUNTS.pro}/${LADDER_COUNTS.max}`);
}
}
// ---------------------------------------------------------------------------
if (problems.length > 0) {
console.error(`catalog-sync: ${problems.length} problem(s) — the skill's docs disagree with ${CATALOG_URL}\n`);
for (const p of problems) console.error(` ${p}`);
console.error('\nFix the docs (this file is the source of truth about what is wrong, the catalog is the source of truth about what is right).');
process.exit(1);
}
console.log(
`catalog-sync: docs match the live catalog — ${TOTAL} sold metrics ` +
`(${PHYSICAL} physical), ladder ${LADDER_COUNTS.free} / ${LADDER_COUNTS.pro} / ${LADDER_COUNTS.max}, ` +
`${documented.size} glossary rows, ${liveFamilies.size} families.`,
);