evals/evals.json
{
"skill_name": "customer-health-score",
"evals": [
{
"id": 1,
"prompt": "I run CS ops at Kelfast, B2B workflow automation, 620 accounts, about $14M ARR. My VP wants a customer health score live before the QBR in six weeks. I already sketched the model in a workshop with the CSM leads: logins 40%, NPS 20%, ticket count 20% (more tickets = worse), CSM gut rating 20%, and five bands A through E. We have three years of CRM and product analytics data, and 47 accounts churned in the last 24 months. Can you turn this into something we can actually ship? Mostly I want to know whether the weights look right.",
"expected_output": "A rebuilt model: weights re-derived from churned-vs-retained separation rather than the workshop, three bands, per-seat rate normalization, ticket trend instead of ticket count, decay toward neutral, a capped category structure with a stakeholder-continuity input that can pull the band down alone, a structured replacement for the gut rating, and validation framed on precision/recall against a defined churn event.",
"files": [],
"expectations": [
"Rejects weights set in a workshop or by intuition and instructs deriving them from the separation between the churned and retained cohorts over the trailing 12 to 24 months.",
"Requires defining the churn event, the entity level, and the prediction window before any further signal or weight discussion.",
"Reduces the band count from five to three.",
"Replaces raw login counts with a per-seat or per-active-user rate, stating that raw counts make large accounts look healthy and small accounts look sick by construction.",
"Flags that scoring tickets as 'more tickets = worse' punishes engaged accounts, and replaces it with a rate-normalized ticket trend or severity and resolution-time trend.",
"States that zero tickets can indicate disengagement rather than health.",
"Adds a decay rule so engagement-type signals drift toward neutral as they age, and specifies drift toward neutral rather than toward red.",
"Requires at least one stakeholder-continuity input with enough downside weight to pull an account's band down on its own.",
"Caps every category's share so no single input can move an account across a band by itself.",
"Replaces the unstructured CSM gut rating with structured rating definitions that state what each rating requires, or with gated overrides carrying reason codes.",
"Specifies validation on precision and recall rather than raw accuracy, naming class imbalance as the reason."
]
},
{
"id": 2,
"prompt": "Hoyt Systems, 1,100 accounts, we have had a health score running for about two years. Problems, all at once: the CSMs basically ignore it and call it 'the weather', our big enterprise logos are permanently red because they open tons of tickets, accounts bounce green to yellow and back week to week, and 88% of the book sits green so the number never tells us anything. We have roughly 60 churns over the past two years to learn from. Where do we start? I have one analyst for about a day a week.",
"expected_output": "A diagnosis of all four symptoms with an explicit fix ordering by churn prevented per hour of analyst time, leading with wiring bands to plays and normalization, and deferring cohort re-derivation on effort grounds while naming it high value.",
"files": [],
"expectations": [
"Diagnoses every reported symptom rather than only the first or the most severe.",
"States the fix ordering explicitly and ranks it by churn actually prevented per hour of analyst effort.",
"Places wiring each band to a play with an owner and an SLA at or near the top of the fix order.",
"Attributes the CSMs' distrust to unwired plays, absent gated overrides, and unpublished weight evidence.",
"Attributes the permanently red enterprise accounts to raw counts used instead of rates, and prescribes rate normalization per seat or against similar-size accounts.",
"Attributes weekly band flapping to single-point noise treated as signal, and requires two consecutive readings or a trailing average before a band change fires.",
"Attributes the 88% green book to missing decay, and names more than roughly 80% green as a broken-model red flag independent of backtest results.",
"Prescribes rate-normalizing tickets and reading them as a trend rather than as counts.",
"Places re-deriving weights from the churned-versus-retained cohort late in the fix order on effort grounds while acknowledging it is among the highest-value fixes.",
"Does not order the fixes by severity of fault or in the order the symptoms were reported.",
"Gives each fix an order-of-magnitude effort (near-zero, an hour, a week, a standing job) rather than a currency amount or a precise hour count."
]
},
{
"id": 3,
"prompt": "Brightsill, Series A, 140 customers all on annual contracts, 18 months since our first paying customer. We have lost 5 accounts total. The board wants a churn-risk score by end of quarter. Our data scientist wants to train a logistic regression on what we have and output a 0-100 probability per account. Does that work, and what would you build instead if it doesn't?",
"expected_output": "A rules-based proxy score labeled as hypothesis-weighted and explicitly non-predictive, with the outcome definition, score logging, and monthly CSM interview validation set up now, and statistically fitted weights deferred until tens of churn events per segment exist.",
"files": [],
"expectations": [
"Rejects fitting a statistical model on 5 churn events and names the insufficient number of churn events as the reason.",
"Recommends a rules-based proxy score instead, built from inputs such as onboarding-milestone completion, activation or adoption thresholds, recency tripwires, and champion-contact recency.",
"Labels every weight in the rules-based score as a hypothesis rather than an evidenced weight.",
"Refuses to present the initial score as predictive, or states plainly that it carries no predictive claim.",
"Still requires defining the churn event, the entity level, and the prediction window now, so validation can begin once events accumulate.",
"Instructs logging the score at fixed intervals during the first 2 to 4 quarters so later backtests can retro-read it honestly.",
"Prescribes monthly interview validation with experienced CSMs on which accounts the score got wrong.",
"Names tens of churn events per segment as the threshold that promotes statistically fitted weights.",
"States that the fitted score is the higher-value option and is demoted on effort ratio rather than on quality.",
"Warns against tuning the model to individual churn anecdotes while event counts are small.",
"Does not propose treating a backtest against the 5 existing churn events as validation of the model."
]
},
{
"id": 4,
"prompt": "Vantail, B2B, around 900 accounts, $60M ARR. Our CRO wants an 'expansion readiness score' separate from the churn health score we already run, so CS can hand AEs a ranked list of accounts ready to upsell. He also asked me for a number he can put in the board deck on how much expansion lift a green account gives us. We do not track whitespace anywhere today: product usage sits in our analytics tool, everything else in the CRM. What should the expansion model look like?",
"expected_output": "A recommendation to keep one composite score with the top band gating expansion plays, a refusal to quote an expansion lift figure, and a qualification layer on whitespace, fit, plan-limit pressure, and budget or contract timing.",
"files": [],
"expectations": [
"Recommends one composite score whose top band gates expansion plays, instead of building a second dedicated expansion model.",
"Names the systematically tracked expansion inputs that would justify a second model, such as whitespace mapping, multi-team adoption spread, and plan-limit trajectories.",
"States that a second model built before those inputs are tracked is a second unvalidated score.",
"Refuses to provide a numeric expansion lift figure for a top-band or green account.",
"States that no rigorous published study shows high health causally predicting expansion, and that the claim rests on vendor case studies.",
"Frames health as necessary but not sufficient for expansion.",
"Requires qualifying top-band accounts on non-health conditions including whitespace, fit, plan-limit pressure, and budget or contract timing.",
"Describes the deliverable as a better-qualified expansion queue rather than a promised expansion lift.",
"Names the cost of a second model as roughly a month to build plus a standing second validation loop to maintain.",
"Notes that legitimate score splits are more commonly by lifecycle stage or segment than by risk versus expansion outcome.",
"Wires the top band to a named expansion or advocacy play with an owner."
]
},
{
"id": 5,
"prompt": "Two of our biggest accounts churned last quarter at Nerrin Cloud, and both were sitting at 82 and 79 in our health score the week they gave notice. In both cases the exec who originally bought us left months earlier, but usage never dropped because their ops team kept running the same weekly reports. Our current model is usage 60 / support 20 / NPS 20, aggregated as total events per account. We have about 400 accounts averaging 45 seats. How do we stop this from happening again?",
"expected_output": "A model change adding a weighted stakeholder-continuity component with band-down power, a usage cap, per-user normalized rollup with role weighting, and an engagement-breadth check.",
"files": [],
"expectations": [
"Names the pattern as silent churn: flat aggregate usage hiding a departed or disengaged champion.",
"Adds a relationship or stakeholder-continuity category as its own weighted component rather than folding it into aggregate usage.",
"Requires that at least one stakeholder-continuity input be able to pull the band down on its own.",
"Caps the usage category so that no combination of usage greens can hold an account green after a champion departs.",
"States the deliberate asymmetry: a continuity input may force the band down alone, while no single positive signal may force green.",
"Replaces total per-account event aggregation with per-user normalized rates rolled up to the account level.",
"Applies stakeholder-role weighting in the rollup so champion and decision-maker signals count more than end-user signals.",
"Adds an engagement-breadth check, flagging engagement concentrated in one person as fragile even when the aggregate is high.",
"States that usage describes behavior and never intent.",
"Names concrete continuity inputs such as executive-sponsor contact recency, responsiveness trend, or stakeholder breadth across roles.",
"Notes that continuity inputs depend on humans logging meetings and sponsor changes, making the category a standing job rather than a one-time wire-up."
]
},
{
"id": 6,
"prompt": "I run growth at Tumbledry, a $12/month self-serve design tool, about 40,000 paying subscribers, month to month, and we have no CSMs at all. Monthly churn is around 4.5% and I want a health score so we can catch people before they cancel. I was going to reuse the model from my last company, enterprise SaaS: sponsor engagement 30%, QBR attendance 20%, adoption 30%, support 20%, reviewed by the CSM team every quarter. What changes?",
"expected_output": "A continuous, system-driven subscriber score: sponsor and QBR inputs removed, an involuntary-churn billing layer added, growth/product ownership, monthly recalibration, no rollup for single-user subscribers, automated band plays, and explicit confirmation of which B2B mechanics carry over unchanged.",
"files": [],
"expectations": [
"Removes the sponsor-engagement and QBR-attendance inputs as inapplicable without a CSM layer or a renewal cliff.",
"Replaces renewal-anchored periodic evaluation with continuous, system-driven scoring.",
"Adds an involuntary-churn layer covering failed payments and dunning outcomes into the score itself rather than handling it outside the score.",
"Justifies the billing layer by payment failure correlating with engagement decline and being an intervention trigger in its own right.",
"Assigns ownership of the scoring logic to growth or product rather than to CS ops.",
"Sets recalibration to monthly rather than quarterly, on the grounds of higher volume and faster drift.",
"Skips user-to-account rollup for single-user subscribers, treating the subscriber as the scored entity.",
"States explicitly that trend-over-level scoring, decay, band-to-play wiring, and the backtest loop carry over from the B2B model unchanged.",
"Defines the churn event for this business as cancellation or no reactivation within a stated number of days.",
"Uses a recency and frequency read of usage rather than a cumulative usage total.",
"Wires each band to an automated play such as a win-back sequence, dunning retry, lifecycle nudge, or upgrade offer, rather than to a human motion."
]
},
{
"id": 7,
"prompt": "We just finished backtesting the new health score at Orcastead, two years of data. Results: overall model accuracy 91%, the red band churns at 2.1x the portfolio base rate, and 44% of the accounts that actually churned were still green 60 days before they left. 78% of the book currently sits green. We run one model across everyone, enterprise accounts averaging 300 seats with a dedicated CSM, and a self-serve SMB tier averaging 6 seats with no CSM where about half are still in onboarding. My VP says 91% accuracy means we are ready to launch. Thoughts?",
"expected_output": "A refusal to launch: accuracy discarded on class-imbalance grounds, both the 3x red-band floor and the two-thirds pre-churn recall floor shown as failed, the 78% green distribution flagged as near the red-flag line, and segment-specific weights and thresholds required before relaunch.",
"files": [],
"expectations": [
"Rejects the 91% accuracy figure as evidence, naming class imbalance and noting that a model calling everything green scores high accuracy while predicting nothing.",
"States that the red band fails the pass floor of at least 3x the portfolio base rate, citing the 2.1x result.",
"Converts the 44% still-green figure into 56% of churned accounts having sat below green, and states that this fails the floor of at least two-thirds.",
"Refuses to approve the launch, or refuses to present the score as predictive in its current state.",
"Notes that 78% green sits just under the roughly 80% distribution red flag and must be monitored rather than treated as a clean result.",
"Requires segment-specific weights and thresholds for the enterprise and self-serve SMB tiers instead of one global model.",
"States that a score healthy for a mature enterprise account can be a red flag for an SMB still in onboarding.",
"Requires segmenting at minimum by size or touch model and by lifecycle stage.",
"Warns that each segment needs its own validation cohort, so the segment count must stay as small as the differences justify.",
"Prescribes identifying which signal category would have caught the accounts that read green until the end, and reweighting accordingly.",
"Notes that low-touch segments run naturally lower engagement, shifting weight there toward sentiment or support responsiveness."
]
},
{
"id": 8,
"prompt": "The model is finally agreed at Pellhurst Software: usage 35, relationship 30, support 15, commercial 10, sentiment 10, backtest done and it held up. Now I have to hand it over to three groups. Our CS ops lead builds it, the CSM team uses it every day, and analytics engineering owns the pipeline. Last time we rolled out a scoring thing it drifted within six months because every team quietly added their own field to it. What do I actually hand them? Also, the CSMs keep asking to be able to mark an account healthy when they think the score is wrong, and I would rather not let them do that.",
"expected_output": "A complete written health score spec with evidence attached to each weight, bands wired to plays with owners and SLAs, named single owners for logic and data, versioned change logging, plus a reversal of the user's position on overrides in favor of reason-coded overrides feeding recalibration.",
"files": [],
"expectations": [
"Delivers a single written health score spec artifact rather than loose prose guidance.",
"The spec states the outcome definition: churn event, entity level, and prediction window.",
"The spec lists segments together with their per-segment weight and threshold overrides.",
"The spec attaches outcome-correlation evidence to each weight rather than stating the percentage alone.",
"Labels any weight lacking outcome evidence as a provisional hypothesis carrying a validation date.",
"The spec maps each band to a play, an owner, and an SLA.",
"The spec includes decay rules, the user-to-account rollup method, validation results, band-change triggers, and a governance block.",
"Reverses the user's position on overrides and recommends allowing CSM overrides rather than blocking them.",
"Requires a mandatory reason code on every override.",
"Treats override clusters as labeled disagreement data reviewed at recalibration, and a repeated reason code as a model bug filed by the field.",
"Names one owner for the scoring logic, one data owner, and the CSMs as acting owners, against the diffuse ownership that caused the previous drift.",
"Requires versioning every change with date, author, reason, and expected effect, announced to the CS team before it lands."
]
},
{
"id": 9,
"prompt": "Kessler Rowe, B2B compliance software, 260 accounts, renewals clustered heavily in Q1. Our product already streams every click into the warehouse, billing is in Stripe, support in Zendesk. Nobody logs meetings: the entire CS team is two people and they live in their inbox. Board review is in seven weeks and they want an at-risk list out of this. Realistically those two CSMs can work maybe 12 accounts a month on a save motion. Which signals do we wire first, and where do we set the red cutoff?",
"expected_output": "A build sequence starting from usage and billing signals, with continuity promoted despite poor ratio because renewals are date-anchored, sentiment surveys deferred, renewal proximity weighted inside the score, and the red band sized to the stated 12-accounts-per-month capacity.",
"files": [],
"expectations": [
"Sizes the red band to the stated intervention capacity of roughly 12 accounts per month before tuning it on statistics.",
"States that the score is then improved until the same capacity catches more of the actual churn, rather than widening the red band.",
"Starts from product usage and adoption plus commercial or billing signals, citing near-zero effort because the product and billing systems already emit those events.",
"Defers the sentiment survey category, citing roughly a day or two to stand up plus a standing job to run, and unstable results from low response rates.",
"Explicitly re-ranks the default build order against the user's stated constraints and names which constraint moved which signal category.",
"Promotes stakeholder continuity despite its poor effort ratio because this is a B2B book with contract renewal dates.",
"Resolves the conflict between nobody logging meetings and the continuity requirement by falling back to whatever the CRM captures on its own, such as email or calendar-derived contact recency.",
"Weights renewal proximity inside the score rather than applying it as a filter after scoring.",
"Rejects integrating every available signal category at once.",
"Drops any candidate signal that could not trigger a play, naming it a vanity signal."
]
},
{
"id": 10,
"prompt": "We are about to validate the score at Aldermist. The plan: pull the current warehouse tables, compute the score for every account that churned in the last 18 months using their full history, compare against the accounts that renewed, then report to the exec team what percentage of accounts have a score at all plus the overall hit rate. After that we would review the model once a year at annual planning. Anything missing?",
"expected_output": "A corrected backtest that retro-scores at a fixed past point using only data available then, adds the 60-90 day pre-churn retro check and hand spot-checks, replaces coverage reporting with save and expansion outcomes, and replaces the annual review with monthly-then-quarterly recalibration plus off-cycle triggers.",
"files": [],
"expectations": [
"Rejects scoring churned accounts on their full history and requires retro-scoring at a fixed point in the past, such as 90 days before each churn or renewal event.",
"Requires using only data that existed at that past point, and names signals backfilled after the outcome as the classic leak.",
"Rejects score coverage, the percentage of accounts carrying a score, as a success metric reported upward.",
"Replaces coverage with save and expansion outcomes: red-band plays that retained the account, and top-band plays that expanded.",
"Adds the retro check on what churned accounts scored at 60 and 90 days before churning, with a post-mortem for each account that read green until the end.",
"Adds hand spot-checks of known accounts covering a recent churn, a recent expansion, a known-shaky renewal, and a champion-departure case that must not read green.",
"Replaces the annual review with monthly review for the first 90 days after launch, then a standing quarterly recalibration.",
"Names off-cycle recalibration triggers such as a product or pricing change, an ICP or segment shift, green creep in the distribution, a cluster of green-account churns, or a rising override rate.",
"Reports a KPI set covering the red-band churn multiple, missed-churn rate, score distribution by segment, play completion within SLA, and override rate clustered by reason code."
]
}
],
"trigger_queries": [
{ "query": "design a customer health score for our B2B SaaS", "should_trigger": true },
{ "query": "our green accounts keep churning and nobody can explain it", "should_trigger": true },
{ "query": "how should I weight the inputs in our account health model?", "should_trigger": true },
{ "query": "we need a way to flag at-risk accounts before the renewal conversation", "should_trigger": true },
{ "query": "what should the bands be on a health score?", "should_trigger": true },
{ "query": "the CSMs completely ignore the health score we built last year", "should_trigger": true },
{ "query": "help me build one composite number that tells us which customers are drifting away", "should_trigger": true },
{ "query": "our score says 90% of the book is healthy, that cannot be right", "should_trigger": true },
{ "query": "how do I validate a customer health score?", "should_trigger": true },
{ "query": "should we have a separate expansion score or just one health score?", "should_trigger": true },
{ "query": "we score customers 0 to 100 but nobody trusts the number", "should_trigger": true },
{ "query": "customer health scoring model design", "should_trigger": true },
{ "query": "how many health tiers should we use, three or five?", "should_trigger": true },
{ "query": "our biggest accounts always look red, something is off with the scoring", "should_trigger": true },
{ "query": "recalibrate our churn risk score, it has drifted", "should_trigger": true },
{ "query": "we want one number per account that tells the CSM what to do next", "should_trigger": true },
{ "query": "backtest our account health model against last year's churn", "should_trigger": true },
{ "query": "how do I stop the score flapping between green and yellow every week?", "should_trigger": true },
{ "query": "what is a sensible split between product usage and relationship signals in a health score?", "should_trigger": true },
{ "query": "we need to know which customers are ready for an upsell and which are about to leave", "should_trigger": true },
{ "query": "the champion left and our score stayed green for two months", "should_trigger": true },
{ "query": "health scoring for a PLG product with no CSMs", "should_trigger": true },
{ "query": "should health thresholds be different for enterprise and SMB accounts?", "should_trigger": true },
{ "query": "our score has no decay so everything just accumulates green forever", "should_trigger": true },
{ "query": "who should own the health scoring logic in a RevOps team?", "should_trigger": true },
{ "query": "should CSMs be allowed to override the health score?", "should_trigger": true },
{ "query": "write the spec for our customer health scoring model", "should_trigger": true },
{ "query": "how do I roll user-level product usage up into one account health number?", "should_trigger": true },
{ "query": "we only have five churns ever, can we still build a risk score?", "should_trigger": true },
{ "query": "how do I prove to the exec team that our health score actually predicts churn?", "should_trigger": true },
{ "query": "the account health model only ever tells us things we already knew", "should_trigger": true },
{ "query": "I want a red yellow green system across our customer base", "should_trigger": true },
{ "query": "what should happen automatically when an account drops a band?", "should_trigger": true },
{ "query": "customer health index design for a subscription business", "should_trigger": true },
{ "query": "the CS team wants a health score and I have no idea where to start", "should_trigger": true },
{ "query": "is NPS a good input into a health score or not?", "should_trigger": true },
{ "query": "we are building post-sale account scoring, not lead scoring", "should_trigger": true },
{ "query": "how often should we re-derive the weights in our health model?", "should_trigger": true },
{ "query": "I need a systematic way to grade every account in the book from healthiest to most at risk", "should_trigger": true },
{ "query": "the number next to each account means something different to every team here", "should_trigger": true },
{ "query": "we want to catch subscribers before they cancel based on how they use the app", "should_trigger": true },
{ "query": "how do we decide how many accounts should land in the red band?", "should_trigger": true },
{ "query": "score said green, the renewal came back no, what is broken?", "should_trigger": true },
{ "query": "design the scoring model CS uses to prioritise its week", "should_trigger": true },
{ "query": "tune our at-risk threshold, there are way too many false alarms", "should_trigger": true },
{ "query": "our health score predicts nothing, I want to rebuild it from scratch", "should_trigger": true },
{ "query": "add payment failures and dunning outcomes into how we score subscriber health", "should_trigger": true },
{ "query": "account health scoring for enterprise accounts anchored on renewal dates", "should_trigger": true },
{ "query": "which product events actually predict churn for us? rank them by lift over base rate", "should_trigger": false },
{ "query": "build an early warning system of churn indicators with thresholds, windows and lead times", "should_trigger": false },
{ "query": "score inbound leads on fit and engagement so marketing knows what counts as an MQL", "should_trigger": false },
{ "query": "our MQL threshold is letting junk through, recalibrate the lead model", "should_trigger": false },
{ "query": "route new demo requests to the right rep by territory and capacity", "should_trigger": false },
{ "query": "check the health of our Kubernetes pods and tell me what is degraded", "should_trigger": false },
{ "query": "add a /healthz endpoint to this service", "should_trigger": false },
{ "query": "diagnose why this cloud resource is reporting degraded health", "should_trigger": false },
{ "query": "score our codebase health and flag the worst modules for refactoring", "should_trigger": false },
{ "query": "build a metric tree from board-level ARR down to what an individual SDR owns", "should_trigger": false },
{ "query": "what goes in the sales to CS handoff packet at closed won?", "should_trigger": false },
{ "query": "design our funnel stages from first touch to closed won", "should_trigger": false },
{ "query": "find where we are losing revenue between MQL and closed won and size it in dollars", "should_trigger": false },
{ "query": "audit our pipeline stage exit criteria against buyer-verifiable milestones", "should_trigger": false },
{ "query": "our sales forecast is always wrong, figure out why", "should_trigger": false },
{ "query": "flag stale deals in the pipeline and give me a disposition for each one", "should_trigger": false },
{ "query": "who owns the industry field in our CRM and how often must it be refreshed?", "should_trigger": false },
{ "query": "which system should be source of truth for subscription objects across our stack?", "should_trigger": false },
{ "query": "we run 40 GTM tools, help me decide which ones to cut at renewal", "should_trigger": false },
{ "query": "set up the discount approval matrix for non-standard deals", "should_trigger": false },
{ "query": "what metrics belong in the board revenue deck and in what order?", "should_trigger": false },
{ "query": "am I ready for a senior RevOps role? review my resume", "should_trigger": false },
{ "query": "write the job description and interview loop for a RevOps manager", "should_trigger": false },
{ "query": "which RevOps newsletters and podcasts should I be following?", "should_trigger": false },
{ "query": "write a win-back email sequence for customers who already cancelled", "should_trigger": false },
{ "query": "design a cancellation flow with a save offer in it", "should_trigger": false },
{ "query": "improve activation in our product onboarding so more trials convert", "should_trigger": false },
{ "query": "how do I run my first customer QBR next week?", "should_trigger": false },
{ "query": "draft the renewal negotiation email for this account", "should_trigger": false },
{ "query": "build an NPS survey and help me word the follow-up question", "should_trigger": false },
{ "query": "what is a good monthly logo churn rate for a seed-stage B2B SaaS?", "should_trigger": false },
{ "query": "calculate our net revenue retention for last quarter", "should_trigger": false },
{ "query": "set up a BI dashboard showing product usage by account", "should_trigger": false },
{ "query": "write a customer success career ladder from CSM to director", "should_trigger": false },
{ "query": "our support queue is backed up, how do we triage tickets faster?", "should_trigger": false },
{ "query": "segment our customer base by industry for a marketing campaign", "should_trigger": false },
{ "query": "predict which free trial users will convert to paid", "should_trigger": false },
{ "query": "price our new enterprise tier", "should_trigger": false },
{ "query": "how do we work out whether our CS team is understaffed?", "should_trigger": false },
{ "query": "instrument the product with analytics events for the new feature", "should_trigger": false },
{ "query": "our warehouse sync is failing and the account table is stale", "should_trigger": false },
{ "query": "build a scorecard for evaluating the vendors we are buying from", "should_trigger": false },
{ "query": "score our channel partners by tier and performance", "should_trigger": false },
{ "query": "run a post-mortem on the outage that hit 30 customers yesterday", "should_trigger": false },
{ "query": "write the CS playbook for onboarding a new enterprise customer in the first 30 days", "should_trigger": false },
{ "query": "forecast next quarter's renewal revenue by account", "should_trigger": false },
{ "query": "how do I get our CSMs to actually log their meeting notes in the CRM?", "should_trigger": false },
{ "query": "compare the main customer success platforms and tell me which to buy", "should_trigger": false }
]
}
SKILL.md
---
name: customer-health-score
description: Design, validate, and govern a composite customer health score - product usage and engagement signals combined into one weighted, normalized, decayed, banded account score that flags both churn risk and expansion readiness, backtested against real churn outcomes and recalibrated. Use whenever the user mentions a customer health score, health scoring model, account health, signal weighting, score bands, at-risk account flagging, churn risk score, expansion risk, or says "green accounts keep churning" - even if they never say "health score". Covers B2B account-level rollup and B2C/PLG continuous scoring. Do NOT use for discovering which indicators actually predict churn - use mbfinotti/revops-skills@customer-churn-signals instead.
license: MIT
metadata:
author: Maya-Beth Finotti
version: "1.0.2"
---
# Customer Health Score
Design, validate, and maintain one composite score that tells a post-sale team which accounts are drifting toward churn and which are ready for an expansion conversation. In ChurnZero's 2025 Customer Revenue Leadership Study (~800 CS and post-sales leaders surveyed), 73% said their current health score does not reliably predict churn.
- **Performative score.** Someone picks ten metrics, weights them by intuition; CS ignores the result because it predicts nothing.
- **Predictive score.** Derives its weights from what churned accounts actually did before churning, then proves itself in backtest.
This skill owns the composition and validation that turns one into the other:
- Normalization
- Weighting
- Decay
- Banding
- Segment calibration
- Override governance
- Wiring every band to a play
Which indicators to feed it, and how predictive each one is, is discovery work owned by `mbfinotti/revops-skills@customer-churn-signals`. Treat signals here at the category level (product usage, relationship/engagement, support, commercial, sentiment) and take the discovered indicator list as input.
Draw on the field's named canon where it genuinely helps:
- **Gainsight's DEAR framework** (Deployment, Engagement, Adoption, ROI): a balanced-scorecard model of 4-6 weighted metric groups summed into bands. Gainsight cautions that weighting "is highly unique to each company".
- **Lincoln Murphy's Success Potential**: ties expansion to achieved health ("No Customer Success... No Expansion Revenue").
- **RFM-style reads**: recency, frequency, and depth of usage.
- **Leading-vs-lagging split**: adoption trend, sponsor engagement, and response-time trend lead; churn, NRR, and renewal rate only confirm afterward. Score the leading ones.
One composite score, or a separate expansion model:
- **One composite, top band gates expansion (default).** Already the score being built and validated; a second model costs a month to build and two validation loops to keep. Gating expansion plays on the top band is the dominant practitioner pattern: low bands trigger retention, the top band triggers expansion/advocacy motions.
- **A separate expansion model.** Justified only when expansion-specific inputs are systematically tracked; otherwise it is a second unvalidated score.
Only vendor case studies support the claim that high health causally predicts expansion; no rigorous published study does. Treat health as necessary but not sufficient for expansion: the top band earns an account the conversation, and fit, whitespace, and budget decide it.
B2B and B2C/PLG share most of the mechanics: per-seat normalization, trend-over-level, an action wired to every band, and the backtest loop are identical in both. Say so when asked. What genuinely differs:
- **B2B.** Rolls user-level signals up to account level with extra weight on champion/sponsor continuity: renewals hinge on the buying organization, not one user, and flat aggregate usage can hide a disengaged champion (the "silent churn" pattern). Risk concentrates around a renewal date, so weight renewal proximity inside the score.
- **B2C/PLG.** No CSM and no renewal cliff: scoring is continuous and system-driven, ownership sits with growth/product rather than CS ops, and an involuntary-churn layer (failed payments, dunning) belongs in the score because payment failures correlate with engagement decline.
## Interview
Ask before designing. One question per message; offer multiple-choice answers when possible. Skip anything already answered.
- Which motion? (a) CSM-covered B2B with contract renewal dates, (b) PLG/self-serve continuous subscription, (c) B2C consumer subscription.
- How many accounts, and roughly how many churn events in the last 12-24 months? (Decides statistical weighting vs the rules-based cold-start path.)
- Which data sources are actually connected today - product usage data, engagement/meeting records, support system, billing? Rough coverage of each?
- Has signal discovery been done - is there an evidenced list of indicators that preceded past churn (see `mbfinotti/revops-skills@customer-churn-signals`), or only a gut list?
- Does a health score already exist? If yes: diagnose first or rebuild? (Diagnosis path: see Diagnosing a Broken Score.)
- What segments exist - enterprise vs SMB, onboarding vs mature? (A 70 that is healthy for enterprise is a red flag for an onboarding SMB; segment calibration is mandatory, not optional.)
- How many at-risk accounts can the team actually work per month? (The red band is sized to this, not to statistics.)
- Who owns the scoring logic, who acts on the score day-to-day, and who may override it?
- By what date must the score be live and acted on? (A hard date - a renewal cycle, a board review - promotes the rules-based proxy and the top two signal rungs; a distant date makes statistically fitted weights and a survey category affordable.)
- One-off win or compounding asset - this quarter's at-risk list, or a model that keeps getting better? (A one-off list drops decay, segment calibration, and the standing recalibration cadence; a compounding mandate promotes all three plus fitted weights.)
- What is the effort ceiling - analyst hours, whose data-engineering time, and how much manual logging CS will genuinely do? (Manual logging is what makes stakeholder-continuity signals real; if nobody will log, that category drops to whatever the CRM captures on its own.)
## Build order
Build the score rung by rung, ranked by churn actually prevented per hour of analyst time - not by what is cheapest to wire, and not by what scores best in backtest. Effort here is analyst hours, data the product must already emit, ongoing re-fitting, and how legible the score stays to the people who must act on it: a score CSMs cannot reconstruct is expensive no matter how accurate it is.
- efficiency: usage/adoption > commercial/billing > support > stakeholder continuity > sentiment survey
- value (churn caught): stakeholder continuity > usage/adoption > support == commercial/billing > sentiment survey
- effort:
- usage/adoption and commercial/billing: near-zero - the product and billing system already emit the events.
- support: an hour to map tickets to rates and trends.
- stakeholder continuity: a standing job - humans must log meetings and sponsor changes for the input to stay true.
- sentiment survey: a day or two to stand up, plus a standing job to run - low response rates keep it unstable.
Support and commercial tie on value because each catches a blind spot the other cannot - escalation clusters, invoice disputes - and neither substitutes for the other.
Start from the top two rungs plus whatever discovery already evidenced, validate that, then add rungs.
**What this order starves:**
- **Stakeholder continuity.** The highest-value input, the one that catches silent churn, and the worst ratio because it depends on humans logging contact. Promote it to rung one in any B2B book with renewal dates, or the moment one green account churns after a champion left - the continuity rule in step 5 makes it mandatory regardless of its ratio.
- **Statistically fitted weights and a second, dedicated expansion model.** Both lose every round on ratio and are still right eventually. Both need an account base large enough to fit and validate on its own outcomes (tens of churn events per segment), and both buy better signal once it exists.
This ordering is a default, not a law. It shifts with context and with who executes it - re-rank it against everything already known about the user before proposing it:
- A product already emitting rich usage telemetry locks usage at rung one for near-zero effort.
- A customer-success team of one demotes everything that needs manual logging and promotes automated usage and billing signals.
- Fewer than a hundred accounts takes fitted weights off the table entirely (see the cold-start path) and makes monthly CSM interview-validation the strongest check available.
## Workflow
1. Run the Interview; collect every answer before designing.
2. Define the outcome before any signal talk. A score that predicts an undefined event cannot be validated.
- Churn event: non-renewal, cancellation, or no reactivation within N days for B2C.
- Entity level: account for B2B, subscriber for B2C.
- Prediction window: what the score claims to predict, e.g. churn within 90 days.
3. Take the indicator list as input - from the customer-churn-signals discovery work or, failing that, the user's best-evidenced candidates - and group it into the categories in [references/signal-categories.md](references/signal-categories.md). Audit data coverage per category, then build in the Build order ranking above, re-ranked against the user's answers. Never integrate everything at once, and drop any signal that could never trigger a play - if it can't trigger a play, it's a vanity signal.
4. Normalize per [references/signal-categories.md](references/signal-categories.md):
- Rate-normalize by seats/users (500 logins from 50 users is not 500 logins from 1,000 users).
- Read support and sentiment as trends, not counts.
- In B2B, roll user-level signals up to account level with champion/sponsor weighting.
5. Weight per [references/weighting-and-bands.md](references/weighting-and-bands.md): derive weights from correlation with the defined outcome in the churned-vs-retained cohort, never from intuition. Cap every category so no single input can move an account across a band alone, and require at least one stakeholder-continuity input so no combination of usage signals can hold an account green after a champion departs.
6. Add decay: every engagement-type signal drifts toward neutral as it ages, so silence reads as uncertainty instead of accumulated green. Score the trend and the level - a falling 75 is a worse account than a stable 60.
7. Band: three bands, no more - more bands sound precise and add confusion. Size the red band to intervention capacity first, then improve the score until the same capacity catches more of the churn. Set segment-specific thresholds and weight overrides.
8. Validate per [references/validation-and-kpis.md](references/validation-and-kpis.md): retro-score the historical cohort, use precision/recall (never raw accuracy - churn is class-imbalanced), and run the retro check on what churned accounts scored 60-90 days pre-churn. Iterate weights, decay, and thresholds until the pass floor in the spec section clears. Too few churn events → take the cold-start path in the same reference.
9. Wire every band to a play with an owner and an SLA - a band change that triggers nothing is a report, not a health score. Define the override policy: any CSM may override with a mandatory reason code; overrides feed recalibration as labeled disagreements.
10. Emit the spec (next section), pilot with a small group before full rollout, review score distribution and misses monthly for the first 90 days, then move to the standing recalibration cadence.
11. If your harness has persistent memory, memorize the approved spec - categories, weights, bands, segment overrides, validation results, review date - so later recalibration and diagnosis runs start from it instead of re-interviewing.
## The Health Score Spec
Deliver every engagement as this artifact - a document the user can hand to CS ops, RevOps, and the CSM team. Fully worked versions live in [references/worked-examples.md](references/worked-examples.md).
```
HEALTH SCORE SPEC - <company/product>, <date>, v<n>
Outcome : churn event definition | entity level | prediction window
Segments : list + per-segment weight/threshold overrides
Signals : category -> weight | inputs (from discovery) | normalization | evidence (outcome correlation)
Decay : per-category drift-to-neutral rule and window
Rollup (B2B) : user->account aggregation method, champion/sponsor weighting
Bands : 3 bands -> score ranges -> wired play, owner, SLA per band
Expansion : top-band gate + the non-health conditions (fit, whitespace, contract timing)
Overrides : who may override, reason codes, how overrides feed recalibration
Validation : backtest window | red-band churn multiple vs base rate | % churned accounts
below green at 60-90 days pre-churn | precision/recall at the red threshold
Triggers : band-change alert -> task/workflow; renewal-proximity weighting (B2B)
Governance : logic owner | acting owner | data owner | review cadence | change log location
```
Three rules about the spec itself:
- **Every weight carries its evidence.** "Adoption-depth trend 35%: churned accounts declined here in 7 of 9 pre-churn quarters analyzed" is reviewable. A bare percentage is folklore. If a weight has no outcome evidence yet, label it a provisional hypothesis with a validation date.
- **The spec has a pass threshold.** No standards body publishes a numeric ship-it bar for health scores; these floors are this skill's convention, set because a model that clears less will not survive live noise or earn CSM trust.
- In backtest, the red band must churn at **at least 3x the portfolio base rate**.
- **At least two-thirds** of accounts that actually churned must have sat below green 60-90 days before churning.
- No more than **~80%** of the book may sit green (a documented practitioner red flag if crossed), independent of backtest results.
Iterate until all three hold. Refuse to ship a score that reads as decoration.
- **The score is versioned and owned.** Diffuse ownership produces a score that changes meaning per reader: CS wants adoption, RevOps wants CRM activity, product wants feature usage, and everyone gets a field.
- One owner for the logic (CS ops/RevOps).
- CSMs act and override.
- One data owner.
- Every change logged and announced.
## Diagnosing a Broken Score
When the user arrives with a running score ("green accounts keep churning", "the team ignores it"), diagnose every symptom, then fix in the order below - not in the order the symptoms were reported, and not worst-fault-first. The ratio is churn actually prevented per hour, and the deadliest fault is rarely the one that pays back first.
- efficiency: wire band->play > normalize per seat > add decay > cap categories + continuity input > noise guard == ticket-read fix > segment calibration > re-derive weights from the cohort > qualify the expansion gate
- value: re-derive weights == wire band->play > cap categories + continuity input > normalize per seat == add decay > segment calibration > noise guard == ticket-read fix > qualify the expansion gate
- effort:
- decay and the noise guard: near-zero.
- normalization, caps, the ticket read, and the expansion gate: an hour each.
- segment calibration: a day or two of analysis.
- the cross-team coordination that wires plays to owners and SLAs: a week.
- cohort re-derivation: a standing job, which also needs churned-account history the book may not have.
Ties justified:
- An unacted score and a wrongly-weighted one both prevent zero churn - they tie on value and split on effort.
- Normalization and decay tie because each removes one systematic misread of the same book.
- The noise guard and the ticket read are the same trailing-trend machinery applied twice, each buying back a different flavor of CSM distrust.
The table is ordered to match.
| Symptom | Likely fault | Fix |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| CSMs ignore or silently distrust the score | No wired plays; no gated override; unexplainable weights | Wire band->play; allow overrides with reason codes; publish the weight evidence |
| Large accounts always look sick (or always healthy) | Raw counts instead of rates | Normalize per seat/user; benchmark against similar-size accounts |
| Score rarely moves; most of book green | No decay - old positive signals accumulate forever | Decay engagement signals toward neutral; recheck distribution monthly (>80% green = broken) |
| Champion left; account stayed green for weeks | Single-category dominance - usage alone can hold green | Cap category shares; require a stakeholder-continuity input that can pull the band down alone |
| Score flaps band-to-band weekly | Single-point noise treated as signal | Score trailing-window trends; require two consecutive readings before a band change fires |
| High-touch accounts flagged red on ticket volume | Ticket count read as risk | Rate-normalize tickets, read the trend; treat zero tickets as possible disengagement, not health |
| Same score means different things across accounts | One global model over unlike segments | Segment-specific weights and thresholds (enterprise vs SMB, onboarding vs mature) |
| Green accounts keep churning | Performative weights - intuition, not outcome correlation; missing relationship/sentiment inputs | Re-derive weights from the churned-vs-retained cohort; add a stakeholder-continuity input |
| Top-band accounts never expand | Health treated as sufficient for expansion | Gate expansion plays on top band, but qualify on fit, whitespace, and contract timing separately |
Re-rank this too: a book with no churned-account history has no cohort to re-derive from, so that rung leaves the list rather than sitting last (see the cold-start path).
## Reference
- See [references/signal-categories.md](references/signal-categories.md) for how each signal category behaves inside a composite: normalization, trend windows, rollup, and per-category traps. Category mechanics only; indicator discovery lives in `mbfinotti/revops-skills@customer-churn-signals`.
- See [references/weighting-and-bands.md](references/weighting-and-bands.md) for deriving weights from outcomes, caps, decay mechanics, banding, segment overrides, and the one-score-vs-two decision in full.
- See [references/validation-and-kpis.md](references/validation-and-kpis.md) for the backtest procedure, precision/recall framing, pass floors, the cold-start path, KPIs, recalibration cadence, and override governance.
- See [references/worked-examples.md](references/worked-examples.md) for a CSM-covered B2B spec and a PLG/B2C spec worked in full, plus a plausible-looking broken score decomposed.
- See `mbfinotti/revops-skills@lead-scoring` for pre-sale scoring - lead scoring predicts who will buy; this skill predicts who will stay and grow. Different outcome, different model.
- See `mbfinotti/revops-skills@sales-to-cs-handoff` for the sales-to-CS transition where the first health baseline gets set.
- See `mbfinotti/revops-skills@revenue-kpi-framework` for the org-wide metric tree this score reports into - the composite can serve as a leading input under the retention branch, and that skill sets which level owns it and what guardrail rides alongside it.