evals/evals.json
{
"skill_name": "sales-forecast-diagnostic",
"evals": [
{
"id": 1,
"prompt": "I run RevOps at Corviq, B2B SaaS, 34 AEs, quarterly forecast. Q3 quarter-start commit was $6.2M and we closed $5.1M. When I pulled the won list I noticed $310K of it came from four deals that were in no forecast category at quarter start - two of them from the same AE, Priya, who has beaten her commit by a wide margin three quarters running. Our CRO is going to tell the board 'we missed by $1.1M' and he wants one number and one cause on the slide. I have quarter-start snapshots and stage, close-date and category field history going back six quarters. What do I give him?",
"expected_output": "A decomposition that reports the $1.41M gross over-forecast on committed deals separately from the $310K of hidden upside, refuses the single-number and single-cause framing, reports signed and absolute error with a fixed denominator, and sets up a snapshot-based reconstruction with per-deal attribution.",
"files": [],
"expectations": [
"States the gross over-forecast on committed deals as $1.41M, distinct from the $1.1M net gap.",
"Attributes the $310K of wins from outside the forecast as its own offsetting error rather than netting it against the over-forecast.",
"Refuses to deliver a single blended gap figure as the answer, stating that netting understates both problems.",
"Rejects the 'one cause' framing and requires the gap decomposed across named deals instead.",
"Reports both signed error (bias) and absolute error rather than a single accuracy percentage.",
"Names which error denominator is in use (miss over actual, or actual over forecast) and commits to holding it fixed across periods.",
"Requires at least 90% of gap dollars attributed to named deals, with the residual capped at 10% and explicitly labeled unexplained.",
"Assigns each attributed deal exactly one primary root cause and exactly one class among data, behavior, and demand.",
"Treats Priya's three-quarter beat pattern as a sandbagging candidate rather than as high performance, and names wins closing from outside the forecast as the sandbagging signature.",
"Requires two independent signals per confirmed cause, at least one of them from recorded history such as a snapshot, field history, or activity log.",
"Specifies reconstructing the period from the quarter-start snapshot rather than from the current record."
]
},
{
"id": 2,
"prompt": "Our VP of Sales has already decided the diagnosis and wants me to build the deck around it. His case has two parts: our win rate on forecast-weighted deals last quarter was 22% when our late stages imply about 70%, and he remembers one deal, Halvorsen Group at $480K, that the rep swore was closing and never did. His fix is mandatory qualification-framework retraining for all 28 AEs plus a new certification. We're Northvane, we sell HR software, quarterly cycle. One thing I noticed: that 22% was computed on the end-of-quarter opportunity list. I do have quarter-start snapshots. Can you help me put the deck together?",
"expected_output": "A refusal to build the deck as framed, identifying the win rate as hindsight-contaminated, the anecdote as a non-signal, the two-signal rule as unmet, and retraining as a behavior-class fix aimed at a problem not yet shown to be behavioral.",
"files": [],
"expectations": [
"Declines to build the deck around the retraining conclusion and states the verdict is unconfirmed.",
"Identifies the 22% win rate as computed on end-of-period data and requires recomputing it on the quarter-start snapshot.",
"Names zombie or already-dead deals padding the denominator as a data-class explanation for an artificially low win rate.",
"States that the Halvorsen Group anecdote is not a signal and cannot support a cause.",
"Applies the two-signal evidence rule and concludes the stage-inflation verdict fails it on the evidence presented.",
"Identifies retraining as a behavior-class fix and warns it is being aimed at a problem not yet shown to be behavioral.",
"Names hindsight bias explicitly: the period-start call must be judged on what was knowable at period start.",
"Warns that a diagnosis delivered as blame drives reps to under-call, flipping the sign of the error instead of shrinking it.",
"Requires the discriminating rep interview question - whether the rep knew the deal was dead or real before the record said so - to split data from behavior.",
"Requires pulling stage field history and activity logs before issuing any inflation verdict.",
"Keeps the accuracy investigation separate from performance reviews and compensation conversations."
]
},
{
"id": 3,
"prompt": "Sales ops at Tessaly Labs, B2B, quarterly cycle. We missed Q2 by about $740K against a $3.4M commit. Our current quarter closes in six weeks and the CEO wants the Q3 call to actually hold. He has told me plainly that anything which does not improve this quarter's number is a distraction. I have a CRM admin who can ship a validation rule or a new field this week, roughly 20 analyst hours, and sales leadership has agreed to nothing yet. Give me the fix plan.",
"expected_output": "A plan that deletes every week-tier and quarter-tier fix from this cycle, names what was deleted, ships only the hour-tier data-class fixes in order, and states plainly that the behavior causes go untouched this cycle.",
"files": [],
"expectations": [
"Deletes every week-tier and quarter-tier fix from this cycle's plan rather than listing them lower in the same plan.",
"Names explicitly which fixes were deleted and states that the deadline is what removed them.",
"Delivers a surviving fix set ordered zombies first, then stale or mass-pushed close dates, then wrong category assignment.",
"States plainly that the behavior-class causes are left untouched this cycle.",
"Treats the available CRM admin as keeping the hour-tier fixes at hour-tier effort rather than turning them into a week-tier request in someone else's queue.",
"Notes that the close-date reason requirement only bites at the next edit, so it needs a full cycle before dates are trustworthy.",
"Notes that the zombie purge makes coverage honest within the current cycle.",
"Pairs the zombie purge with an entry-standard fix or a standing sweep rather than treating the purge as finished work.",
"Names for each surviving fix its cause class, its owner, its effort tier, and the single metric that shows within one period whether it worked.",
"Does not recommend a comp-plan change or a rep calibration scorecard as part of this cycle's plan.",
"States which period each recommendation is aimed at rather than leaving the timing implicit."
]
},
{
"id": 4,
"prompt": "Brightmoor Systems, 19 AEs. We migrated off a homegrown CRM onto Salesforce five months ago and nobody turned on field history tracking. There is no such thing as a period-start snapshot here either - the forecast was a spreadsheet somebody emailed round and then overwrote each week. Our CEO is convinced two named reps are sandbagging and that one regional manager pads his roll-up, and he wants me to prove it before the comp cycle closes. Where do I start?",
"expected_output": "A plan that removes the sandbagging and manager-override causes from this pass for lack of history, starts capturing history immediately, proceeds with current-state fingerprints plus interviews at lower confidence, and refuses to name individuals into a comp decision.",
"files": [],
"expectations": [
"States that the sandbagging and manager-override causes cannot be evidenced in this pass and removes them from the fix plan rather than ranking them low.",
"Refuses to name individual reps as sandbaggers on the available evidence.",
"Schedules the sandbagging and override analyses for a later re-run once history exists, rather than dropping them permanently.",
"Instructs enabling field history tracking now on stage, close date, amount, and forecast category.",
"States that field history tracking is opt-in per field and not retroactive, so enabling it does not recover the missed period.",
"Proceeds with a current-state fingerprint diagnosis plus interviews rather than refusing to diagnose at all.",
"Labels every conclusion reached without recorded history as lower-confidence.",
"States that interview testimony alone never confirms a cause.",
"Requires a repeated pattern across at least two periods of the same person's own history before sandbagging can be named.",
"Flags the CRM migration as a confounder for the period under investigation.",
"Warns against feeding the accuracy finding into the comp cycle, because a diagnostic that feeds punishment teaches reps to sandbag."
]
},
{
"id": 5,
"prompt": "Here is where we landed at Ardenway - B2B SaaS, quarterly, $5.0M commit against $3.9M actual, $1.1M gap, 91% of it attributed. Confirmed causes with their attributed dollars: comp-driven bias $420K, stage inflation $310K, stale and mass-pushed close dates $240K, zombie opportunities $130K. We have a RevOps admin with change authority, sales leadership is bought in, and the CFO will look at the comp plan at the annual refresh in five months. Give me the fix list in the order we should ship it.",
"expected_output": "Three orderings delivered together: the dollar impact order, the leverage order led by stale dates, and an explicit account of every cause that moved between them and the effort that moved it.",
"files": [],
"expectations": [
"Produces three distinct outputs: an impact order, a leverage order, and an explicit list of what moved between them.",
"Gives the impact order as the straight dollar ranking: comp-driven bias, stage inflation, stale dates, zombies.",
"Leads the leverage order with stale and mass-pushed close dates, not with zombies and not with comp-driven bias.",
"Places zombies ahead of stage inflation in the leverage order despite the smaller dollar figure, on the hour-versus-week effort difference.",
"Places comp-driven bias last in the leverage order, on the quarter-tier plan-amendment cost.",
"Names, for every cause that changed position between the two orders, the specific effort that moved it.",
"States that the leverage denominator is effort - analyst hours, admin work, cross-team negotiation, forecast cycles to effect - and never a currency amount.",
"States that leverage promotes cheap fixes but not cheapness: a larger attributed cause outranks a smaller one at the same effort tier.",
"Delivers in leverage order with the impact order printed beside it rather than shipping one order alone.",
"Names comp-driven bias as the root behind the behavior-class causes and states that the behavior fixes are temporary while the plan is unchanged.",
"Attaches to each fix its cause class, owner, effort tier, and the single metric that proves it worked within one period."
]
},
{
"id": 6,
"prompt": "Second pass at Velmara. Last quarter we did exactly what the previous analysis said: purged 214 zombie opportunities, shipped reason-required close-date pushes, and started logging category changes. Forecast accuracy this quarter is unchanged, still about 19% over on the commit call. The new fingerprint run shows two things. Slips cluster in the fortnight right after each AE crosses 100% of quota, and the same four AEs under-call when they are ahead of plan and over-call when they are behind. Our comp plan has a hard accelerator cliff at 100%. Our CRO has told me the plan is locked until January and he will not reopen it. What now?",
"expected_output": "Comp-driven bias promoted to the top of the narrative on all three promotion conditions, the cheap data fixes declared spent rather than re-listed, and the comp fix deleted from the plan with the regrowing behavior causes named.",
"files": [],
"expectations": [
"Promotes comp-driven bias to the top of the narrative rather than ranking it last by leverage ratio.",
"Cites the slip clustering at the quota cliff as a promotion condition.",
"Cites the reps' bias flipping with their attainment position as a second promotion condition.",
"Cites the shipped data-class fixes producing no improvement in the behavior class as a third promotion condition.",
"States that the cheap data-class fixes have no remaining value here and does not re-recommend them.",
"Deletes the comp-plan fix from the actionable plan given the CRO's refusal, rather than parking it at the bottom as a demoted option.",
"Names the specific behavior-class causes that will regrow because the comp fix was ruled out.",
"Does not recommend punishing forecast error through compensation or performance review.",
"Notes that a comp-plan amendment lands only at the next plan period, needs finance, HR and legal sign-off, and is barely reversible once issued.",
"Flags predictable quarter-end discounting as a pricing decision for leadership rather than a rep-coaching item.",
"Frames the sandbagging as a rational response to the cliff rather than a character flaw.",
"Recommends measuring forecast accuracy separately from attainment."
]
},
{
"id": 7,
"prompt": "Prepping a QBR slide at Kestrelane and I want a sanity check on four things. One, I want to open with 'only 18% of companies forecast within 5%' - is that the right stat or is there a better one? Two, we sit at 2.1x pipeline coverage and everyone says 3x, so that is my gap slide. Three, our stage probabilities are the CRM defaults, 90/70/50/30/10, and our weighted forecast comes straight out of them. Four, our commit list converted at 100% last quarter, every single committed deal closed, and I want to show that as proof the commit category is healthy.",
"expected_output": "A refusal of the unsourced statistics, a rejection of 3x as a benchmark in favour of the team's own win rate, replacement of default stage probabilities with measured conversion, and an identification of 100% commit conversion as a sandbagging signal.",
"files": [],
"expectations": [
"Refuses to cite the '18% of companies forecast within 5%' figure and states that it circulates without primary attribution.",
"Rejects 3x coverage as a benchmark and identifies it as folklore with a disputed origin.",
"States that the underlying coverage arithmetic is roughly the inverse of the win rate and instructs deriving the target from the team's own win rate.",
"States that default stage probabilities are tool conventions rather than measured conversion, and must be replaced with the team's own conversion by stage before any weighted forecast is trusted.",
"Identifies the 100% commit conversion as a sandbagging signal rather than evidence of a healthy commit category.",
"Directs every threshold to be derived from the organization's own trailing history rather than from published tables.",
"States that any circulated number that must be mentioned carries an explicit provenance label in the deliverable.",
"Rejects industry MAPE bands as imported from supply-chain forecasting and not transferable to sales pipelines.",
"Specifies deriving the staleness threshold from median time-in-stage per stage, flagging deals past roughly twice their stage median.",
"Specifies deriving the push-count threshold from where the lost-or-slipped population separates from the won population in the team's own push-count distribution.",
"Recommends dollar-weighted error when deal sizes vary widely, rather than averaging per-deal percentages.",
"Recommends recomputing derived thresholds every few periods as the team, product and market drift."
]
},
{
"id": 8,
"prompt": "Sifting through Q1 at Oakmere Digital and I want your read on six findings before I write the verdicts up. One, nineteen commit deals have the Economic Buyer field filled in with 'TBD' or 'unknown'. Two, AE Danilo can describe exactly what the buyer last did on his two biggest deals - dates, names, the security review they ran - but none of it is in the CRM. Three, AE Ruth's $290K deal has no buyer-side evidence anywhere and she cannot name one thing the buyer did. Four, field history shows 31 deals jumping from Discovery straight to Negotiation on the same afternoon in February. Five, our Omitted category grew from 4 deals to 61 over the quarter. Six, when I asked six AEs what Commit means I got six different answers. Our field completeness report says we are at 96% on required fields. Which class is each one?",
"expected_output": "Per-finding class assignments using the data-versus-behavior discriminators, the 96% completeness figure dismissed because placeholders count as complete, the Omitted growth flagged as disguised losses, and the conflicting Commit definitions reported as a methodology finding at the scope boundary.",
"files": [],
"expectations": [
"Classifies the placeholder-filled Economic Buyer fields as a missing or unverifiable evidence problem and states that a filled field is not evidence.",
"States that a completeness report checking only for non-blank values counts 'unknown' and 'TBD' as complete, so the 96% figure evidences nothing.",
"Instructs spot-checking field values rather than fill rates.",
"Classifies Danilo's case as data, a capture burden, because the evidence exists but was never recorded.",
"Classifies Ruth's case as behavior, because the evidence does not exist anywhere and the deal was never qualified.",
"Treats the 31 same-afternoon stage jumps as a candidate bulk edit or workflow auto-advance and requires establishing which before any inflation verdict.",
"Classifies a workflow-driven stage advance as data rather than behavior.",
"Flags the Omitted category growth as a possible soft delete disguising closed-lost deals and requires auditing its contents.",
"Treats the six conflicting definitions of Commit as a methodology finding reported at the scope boundary, not as a rep failure.",
"States that redesigning the forecast category definitions is out of scope for this diagnostic.",
"States that ambiguous category definitions make honest reps look dishonest, so the definition check precedes any behavior verdict.",
"Uses the discriminating question - what the buyer did last, not what the rep did - to separate inflation from data.",
"Requires the two-signal evidence rule to be satisfied before any of these becomes a confirmed cause."
]
},
{
"id": 9,
"prompt": "Different shape of problem. Lumeo is a self-serve subscription product, roughly 40,000 paid accounts at an average $79 a month, no AEs on the core motion. Our monthly revenue forecast is a model: trailing trial-to-paid conversion, cohort retention curves, and a run rate on new trials. On top of that our inside-sales lead Marguerite adds a manual overlay for the assisted-upgrade deals her four SDRs work. We have over-forecast six months running by 8 to 14%, and Marguerite's overlay has come in under her actual every single month. Everyone here tells me this kind of forecast investigation is for enterprise deal teams and does not apply to us. Does it?",
"expected_output": "A translation of the deal-level method onto the model layer and the human overlay: the diagnostic applies, overlay treated as the human layer, stale assumptions mapped to stale dates, over-inclusive funnel definitions to stage inflation, the conservative overlay to sandbagging, with attribution to named model assumptions.",
"files": [],
"expectations": [
"States that the diagnostic applies here and does not concede that a model-based forecast is out of scope.",
"Applies the deal-level fingerprints to Marguerite's manual overlay as the human layer of the forecast.",
"Maps stale conversion assumptions in the model onto the stale-close-date cause.",
"Maps over-inclusive funnel definitions onto the stage-inflation cause.",
"Identifies Marguerite's chronically conservative overlay as the sandbagging equivalent.",
"Recommends tracking overlay-versus-model accuracy the same way manager-override-versus-roll-up accuracy is tracked.",
"Attributes gap dollars to named model assumptions where no named deal exists, rather than skipping attribution for this motion.",
"Keeps the same attribution coverage rule: at least 90% attributed, with at most 10% residual labeled unexplained.",
"Keeps the three-class verdict system of data, behavior and demand unchanged for this motion.",
"Reports signed error and absolute error separately for the model layer and the overlay layer.",
"States that six months of consistent direction is bias rather than noise, and keeps bias separate from magnitude.",
"Does not simply recommend removing the overlay; evaluates whether it beats or underperforms the raw model first."
]
},
{
"id": 10,
"prompt": "Two regional managers at Portwell. Ines submits a roll-up consistently about 12% below the sum of her reps' commits; Anselm submits about 9% above his. Over the last five quarters Ines's submitted number landed within 4% of actual while her reps' raw sum was off by 15%, and Anselm's submitted number was off by 18% while his reps' raw sum was off by 6%. Our new VP wants to ban manager overrides outright and just take the system roll-up. Separately, I have built a set of flag rules out of last quarter's slipped deals and I would like them in production next week. Thoughts?",
"expected_output": "A rejection of the blanket ban, with Ines's override kept as informed judgment and Anselm's treated as quantified bias on the accuracy comparison, overrides logged and scored per manager, and the new flag rules backtested against an earlier closed period before deployment.",
"files": [],
"expectations": [
"Rejects the blanket override ban rather than endorsing it.",
"Identifies Ines's override as informed judgment because it beats the raw roll-up, and recommends keeping it with a logged reason.",
"Identifies Anselm's override as quantified bias because it underperforms the raw roll-up, and treats that divergence as a cause carrying attributable dollars.",
"States that the discriminating test is override accuracy compared against raw roll-up accuracy over trailing periods, not the size or direction of the divergence.",
"Requires overrides to be logged with a reason and scored per manager per period.",
"States that the override log is about an hour of admin to ship but needs roughly a quarter of trailing overrides before the score means anything.",
"States that managers must accept being scored for the measure to be honest.",
"Requires backtesting the new flag rules on an earlier closed period before putting them into production.",
"Sets the backtest pass threshold at flagging at least 80% of the deals that actually slipped or were lost from commit in that period.",
"States that below that threshold the rules are fitted to the story rather than the data, and must be revised and re-run.",
"Classifies the manager override as behavior by construction, leaving only judgment-versus-padding open.",
"Requires a logged reason on the surviving override rather than removing the manager's latitude."
]
}
],
"trigger_queries": [
{ "query": "why did we miss the number last quarter", "should_trigger": true },
{ "query": "our commit is never right", "should_trigger": true },
{ "query": "reps keep sandbagging and I cannot prove it", "should_trigger": true },
{ "query": "deals keep slipping out of the quarter, every quarter", "should_trigger": true },
{ "query": "our forecast accuracy is terrible and I need to know why", "should_trigger": true },
{ "query": "we called $4M and closed $3.1M, what happened", "should_trigger": true },
{ "query": "I do not trust the forecast anymore", "should_trigger": true },
{ "query": "every quarter we come in about 20% under the call", "should_trigger": true },
{ "query": "how do I tell whether my reps have happy ears", "should_trigger": true },
{ "query": "our commit list converts at 40%, is that a rep problem or a data problem", "should_trigger": true },
{ "query": "the VP's roll-up is always higher than the sum of his reps", "should_trigger": true },
{ "query": "we beat the number by 30% and nobody saw it coming", "should_trigger": true },
{ "query": "run a root cause analysis on last quarter's revenue miss", "should_trigger": true },
{ "query": "diagnose our forecast", "should_trigger": true },
{ "query": "all the close dates got pushed the night before the forecast call", "should_trigger": true },
{ "query": "half the deals we committed had no real buyer contact logged", "should_trigger": true },
{ "query": "leadership wants an explanation for why the committed deals did not land", "should_trigger": true },
{ "query": "our quarter-start call and our quarter-end actual never match", "should_trigger": true },
{ "query": "how do I figure out whether we missed because of bad data or genuinely weak demand", "should_trigger": true },
{ "query": "the board asked why we missed and I need a real answer", "should_trigger": true },
{ "query": "we keep finding deals that closed and were never in the forecast", "should_trigger": true },
{ "query": "I think my managers are padding their submitted numbers", "should_trigger": true },
{ "query": "wins keep coming from outside commit", "should_trigger": true },
{ "query": "is 21% error normal for a team our size", "should_trigger": true },
{ "query": "our pipeline looks healthy on paper and we miss anyway", "should_trigger": true },
{ "query": "quarter after quarter the same two reps under-call", "should_trigger": true },
{ "query": "why is our forecast always wrong in the same direction", "should_trigger": true },
{ "query": "what is causing our commit category to be so unreliable", "should_trigger": true },
{ "query": "one rep has beaten commit by 40% three quarters running", "should_trigger": true },
{ "query": "help me build the case for what actually went wrong in Q2", "should_trigger": true },
{ "query": "should I retrain the team or fix the CRM", "should_trigger": true },
{ "query": "we shipped a pipeline cleanup last quarter and the forecast is still wrong", "should_trigger": true },
{ "query": "my CRO wants to know if this was a demand problem or a discipline problem", "should_trigger": true },
{ "query": "we have deals sitting in negotiation with no economic buyer on record, how bad is that", "should_trigger": true },
{ "query": "our forecast is soft mid-quarter and back-loaded every single time", "should_trigger": true },
{ "query": "I need to explain a $900K gap to the board next week", "should_trigger": true },
{ "query": "what tells you a committed deal was never real in the first place", "should_trigger": true },
{ "query": "how do I separate data quality problems from rep behaviour in the forecast", "should_trigger": true },
{ "query": "our self-serve model over-predicts every month and the inside sales overlay is always low", "should_trigger": true },
{ "query": "post-mortem on the revenue miss", "should_trigger": true },
{ "query": "my call was 95% accurate but the absolute error was awful, which do I report", "should_trigger": true },
{ "query": "reps close deals out of the pipeline category that were never in best case", "should_trigger": true },
{ "query": "we have zombie opportunities inflating coverage and the call still missed", "should_trigger": true },
{ "query": "the same deal has been sitting in commit for four quarters", "should_trigger": true },
{ "query": "why does my team beat the call every quarter and still miss the annual plan", "should_trigger": true },
{ "query": "break down last period's forecast versus actual by category", "should_trigger": true },
{ "query": "figure out whether our comp plan is what is causing the forecast bias", "should_trigger": true },
{ "query": "our quarter-end discounts have trained buyers to wait and the mid-quarter call is useless", "should_trigger": true },
{ "query": "I want to stop guessing and attribute the miss to named deals", "should_trigger": true },
{ "query": "what is the right way to score whether my managers' overrides help or hurt", "should_trigger": true },
{ "query": "we missed again, is it the process, the people, or the market", "should_trigger": true },
{ "query": "our deals all close in the last three days of the quarter and we never see them coming", "should_trigger": true },
{ "query": "design our pipeline coverage model, how much pipeline do we need to hit $12M", "should_trigger": false },
{ "query": "rewrite our stage exit criteria so each one names something the buyer did", "should_trigger": false },
{ "query": "our stages are all named after rep activity like demo scheduled, audit them", "should_trigger": false },
{ "query": "run the monthly stale deal sweep before the QBR", "should_trigger": false },
{ "query": "set up a recurring pipeline hygiene checklist with a disposition per flagged deal", "should_trigger": false },
{ "query": "which system should be source of truth for the close date field", "should_trigger": false },
{ "query": "who owns the amount field in our CRM and how often must it be refreshed", "should_trigger": false },
{ "query": "finance and sales report two different ARR numbers, sort out the governance", "should_trigger": false },
{ "query": "design the funnel stage set for our new PLG motion from scratch", "should_trigger": false },
{ "query": "build a bowtie funnel model with post-sale stages", "should_trigger": false },
{ "query": "where are we losing deals between MQL and SQL", "should_trigger": false },
{ "query": "size how much recoverable revenue our leaky funnel is costing us", "should_trigger": false },
{ "query": "write the narrative for the board revenue deck", "should_trigger": false },
{ "query": "how do I present a revenue miss to the board without surprising them", "should_trigger": false },
{ "query": "build our KPI tree from board level down to individual contributor", "should_trigger": false },
{ "query": "help me pick a north star metric for the revenue org", "should_trigger": false },
{ "query": "score our inbound leads on fit and engagement", "should_trigger": false },
{ "query": "our MQL threshold lets too much junk through, recalibrate it", "should_trigger": false },
{ "query": "set up round-robin routing with territory and capacity rules", "should_trigger": false },
{ "query": "leads are sitting unworked in a queue, fix the assignment logic", "should_trigger": false },
{ "query": "design a tiered discount approval matrix with delegation of authority", "should_trigger": false },
{ "query": "our reps discount heavily at quarter end, build the approval tiers to stop it", "should_trigger": false },
{ "query": "build a customer health score with weights, decay and bands", "should_trigger": false },
{ "query": "which product usage signals actually predict churn", "should_trigger": false },
{ "query": "design the closed-won to CSM handoff packet and its SLA", "should_trigger": false },
{ "query": "consolidate our GTM tool stack, we have too many overlapping tools", "should_trigger": false },
{ "query": "we pay for four sales engagement tools, which do we cut", "should_trigger": false },
{ "query": "am I ready to step up to a RevOps manager role", "should_trigger": false },
{ "query": "write the interview scorecard for our first sales ops hire", "should_trigger": false },
{ "query": "which RevOps newsletters and podcasts are worth my time", "should_trigger": false },
{ "query": "which revops skill do I need for this project", "should_trigger": false },
{ "query": "who should I follow in revenue operations on LinkedIn", "should_trigger": false },
{ "query": "generate a weighted forecast with best case, likely and worst case scenarios", "should_trigger": false },
{ "query": "build my commit versus upside breakdown from this pipeline CSV", "should_trigger": false },
{ "query": "run our weekly pipeline review and tell me which deals to push on this week", "should_trigger": false },
{ "query": "assess our gap to quota from the current pipeline", "should_trigger": false },
{ "query": "forecast our cloud infrastructure spend for next fiscal year", "should_trigger": false },
{ "query": "build a demand forecasting model for our inventory replenishment", "should_trigger": false },
{ "query": "how do I calculate MAPE for a demand planning model", "should_trigger": false },
{ "query": "write next year's sales compensation plan", "should_trigger": false },
{ "query": "allocate quotas across our four territories for next fiscal year", "should_trigger": false },
{ "query": "coach my AE on running better discovery calls", "should_trigger": false },
{ "query": "write a MEDDIC qualification checklist for our AEs to fill in", "should_trigger": false },
{ "query": "set up a win/loss interview program with our lost prospects", "should_trigger": false },
{ "query": "why is our win rate dropping quarter over quarter", "should_trigger": false },
{ "query": "build a weekly forecast submission process from scratch, we have none", "should_trigger": false },
{ "query": "define our forecast categories and what qualifies as commit", "should_trigger": false },
{ "query": "should we use commit, best case and pipeline, or a different category set", "should_trigger": false },
{ "query": "decide who calls the final number and what the roll-up cadence should be", "should_trigger": false },
{ "query": "evaluate forecasting tools to replace our spreadsheet", "should_trigger": false },
{ "query": "model ARR growth for our Series B deck", "should_trigger": false },
{ "query": "our sales cycle keeps getting longer, what is driving that", "should_trigger": false }
]
}
SKILL.md
---
name: sales-forecast-diagnostic
description: Diagnose why an existing sales forecast is unreliable and recommend fixes - stage inflation and happy ears, sandbagging, deals without verifiable buyer evidence, stale or mass-pushed close dates, wrong forecast-category assignment, zombie and duplicate opportunities, late-created deals, manager roll-up overrides, and comp incentives that reward bias - separating data-quality from behavioral from genuine demand problems. Use whenever the user mentions forecast accuracy, a forecast miss, reps sandbagging, deals that keep slipping, "why did we miss the number", or "our commit is never right" - even if they never say "forecast". Covers B2B deal-based and B2C/high-volume. Do NOT use for pipeline coverage modeling - use mbfinotti/sales-skills@sales-pipeline-coverage-modeling instead.
license: MIT
metadata:
author: Maya-Beth Finotti
version: "1.3.5"
---
# Forecast Diagnostic
Investigate a forecast that missed, or that nobody trusts, and name the root cause with evidence. A forecast call asks "what's the number?" This diagnostic asks "was the number ever real?"
Practitioners converge on the view that most misses are inspection failures, not market failures: the evidence that a committed deal was fiction usually existed in the record before the miss.
Scope boundary, stated up front: this skill consumes whatever forecasting methodology already exists - the category set, roll-up rules, commit criteria, and coverage targets are inputs, not deliverables. Designing or redesigning that methodology is out of scope. When the investigation shows the methodology itself is undefined or ambiguous ("commit means different things to different reps"), report that as a finding and stop at the boundary.
Every confirmed root cause lands in exactly one of three classes, because each class has a different owner and a different fix:
- **Data**: the record was wrong, a human knew better.
- **Behavior**: the record faithfully reflects a biased human call.
- **Demand**: honest, evidenced pipeline still fell short.
The most common diagnostic error is issuing a behavior verdict for a data problem.
## Interview
Ask before analyzing:
- One question per message; offer multiple-choice options when possible.
- Skip anything already answered.
- Ask the first three questions below before any of the rest. Their fixes diverge by two orders of magnitude in effort and in how long they take to reach a forecast, and the answers decide the order the whole deliverable is ranked in.
Questions:
- By when must the number be more reliable - this period's call, the next one, or the next planning cycle? A fix that lands after the period closes does not help this forecast: a mid-period deadline deletes every fix slower than one cycle from the plan and leaves the hour-tier CRM-admin fixes carrying the whole load.
- One-off correction of this miss, or a standing accuracy program? One-off promotes the reconstruction, the zombie purge, and the date cleanup; standing promotes evidence gates, calibration scorecards, and threshold derivation.
- Effort ceiling: how many analyst hours, whether a CRM admin can ship a field or validation change this week, and who has authority to reopen the comp plan or change a review cadence. No admin turns every hour-tier fix into a week-tier request; no comp authority deletes the comp fix outright.
- What triggered this - a specific missed period, chronic inaccuracy, or a suspicion (sandbagging, inflated pipeline)? Which period(s)?
- Which direction is the error - over-forecast (missed the call), under-forecast (beat it by a lot), or erratic both ways?
- At which level does the number go wrong - individual rep, manager roll-up, or company-wide? Where in the roll-up can a human override the sum?
- What does the process look like today - forecast categories in use, submission cadence, who calls the final number? (Recorded as-is; not redesigned.)
- What history exists - opportunity snapshots, field history (stage, close date, category, amount changes), activity logs? How far back?
- What is the motion - B2B deal-based, B2C/high-volume/self-serve model-based, or mixed? Roughly how many deals per period?
- How is the team paid - quota cliffs, accelerators, quarter-end discount authority, anything rewarded or punished based on the forecast number itself?
- Any confounders in the period - CRM migration, stage redefinition, territory change, a bulk hygiene cleanup?
## Workflow
1. Run the Interview; establish the period, direction, and level of the error before touching deal data.
2. Quantify the miss before explaining it: forecast vs. actual per period, per roll-up level, per forecast category. Compute both signed error (bias) and absolute error - a symmetric average hides offsetting errors, since the same miss reads very differently depending on the chosen denominator, so fix one error definition first. Both numbers always ship together, as a correctness rule rather than a menu to rank and pick from - see [references/evidence-and-metrics.md](references/evidence-and-metrics.md).
3. Inventory the evidence base: period-start snapshot, field history, activity logs. If no snapshots or field history exist, start capturing them now, diagnose from current-state fingerprints plus interviews, and label every conclusion lower-confidence - without history, behavior verdicts are hard to defend.
4. Reconstruct the missed period from the period-start snapshot. Classify every deal that was in a forecast category by outcome (won as called, slipped, lost, removed, still open) and every deal actually won by origin (in the forecast? which category? when created?). If your harness can execute code, script this from a two-file export (period-start forecast, period-end actuals); otherwise ask the user for both lists and reconcile them by hand.
5. Run the root-cause fingerprints in [references/root-cause-fingerprints.md](references/root-cause-fingerprints.md) over the reconstruction. Require two independent signals per suspected cause, at least one from recorded history.
6. Interview reps and managers on flagged deals to split data from behavior. The discriminating question: "did you know this deal was dead (or real) before the CRM said so?" Yes means the record lagged reality - a data problem. No means behavior or demand.
7. Isolate demand last: rebuild the period-start forecast with hindsight-corrected data - zombies removed, honest categories, evidence-backed close dates. Whatever gap survives the correction is a genuine demand or coverage shortfall that no forecasting-process fix will recover; say so plainly.
8. Attribute the gap: every dollar of miss maps to a named deal (or a named model assumption in high-volume motions), each with one primary root cause and one class. Offsetting errors (sandbagged wins masking inflated losses) are attributed separately, never netted silently.
9. Check the pass thresholds in Measurement below; tune the fingerprints and re-run until every threshold holds.
10. Rank the confirmed causes for delivery using Fix leverage below: attributed dollars first, then divided by fix effort, with the adjustment shown rather than the two orders silently merged. One fix per cause matched to its class, plus the metric that will prove each fix worked within one period.
11. Deliver section by section for user approval; report shape and worked examples in [references/worked-diagnosis-example.md](references/worked-diagnosis-example.md).
12. If your harness has persistent memory, memorize the confirmed causes, the thresholds derived from this team's history, the effort tiers as they turned out for this team, and rep-level calibration baselines - the next period's re-run starts from them.
## Root causes in scope
The compact map: full detection fingerprints, second signals, and fixes live in [references/root-cause-fingerprints.md](references/root-cause-fingerprints.md), in this same order.
The value axis cannot be pre-ranked here and this skill does not pretend to: which cause is worth the most dollars is the output of the attribution table, and it differs with every engagement. What is stable is the fix - what it costs to ship and how long before a forecast reflects it - so the register is ordered by that, and the diagnosis then re-ranks it against the dollars it actually found (see Fix leverage).
- fix effort (most first): `comp-driven bias > stage inflation == missing evidence > late-created deals > rep sandbagging > manager override == wrong category > stale dates == zombies`
- time-to-effect (slowest first): `comp-driven bias > rep sandbagging > manager override > late-created deals > stage inflation == missing evidence > stale dates == wrong category > zombies`
- compliance cost (most first): only two carry any - `comp-plan change > a behavior verdict naming an individual rep`; every other fix is a CRM setting or a review habit with no external sign-off, so the axis is not worth a full ordering.
- efficiency at equal dollars (best first): `zombies > stale dates == wrong category > stage inflation == missing evidence > late-created deals > manager override > rep sandbagging > comp-driven bias`
Ties, each earned rather than dodged:
- **Stage inflation == missing evidence**: both fixes are literally one intervention, a buyer-evidence gate on commit entry, same owner, same week, same standing inspection.
- **Manager override == wrong category**: both are an hour of CRM admin (a reason field, category field history) plus exactly one negotiation with the people whose judgment the change constrains.
- **Stale dates == zombies**: both are one admin change plus a recurring sweep, constrain nobody's judgment, and are reversible the day they ship.
- **Stale dates == wrong category**, on time-to-effect only: both bite at the next edit, so one full cycle either way.
| Root cause | Usual class | Signature | Fix effort |
| -------------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Zombie / duplicate opportunities | Data | Long-dead deals still open and counted; the same buying intent counted twice | An hour to purge with reason codes, then a standing hygiene sweep; coverage is honest in the same cycle |
| Stale / mass-pushed close dates | Data | Push counts pile up; date edits cluster on a single day near period end | An hour of admin for reason-required date moves, then a standing job; bites at the next push, so one cycle |
| Wrong category assignment | Data or behavior | Category contradicts stage and evidence; an omitted/excluded category used as a soft delete; category set at creation and never touched | An hour of admin for logged category changes - a week if the definitions themselves must be re-agreed across the team |
| Stage inflation / happy ears | Behavior | Late-stage deals with no buyer-side evidence; stage skips in field history; wins far below what the stage implies | A week to define the buyer-evidence gate and change the review questions, then a standing inspection; the call gets honest one cycle later, win rates two |
| Missing / unverifiable evidence | Data or behavior | Required fields filled with placeholders; nobody can say what the buyer last did | Same week, same gate, same owner as stage inflation - shipping one ships the other |
| Late-created deals | Data | Deals created and closed within days, or created directly in a late stage - pipeline visibility is bookkeeping after the fact | A week to set the logging rule, then a rep-habit change; coverage math is untrustworthy for a cycle or two after |
| Manager roll-up override | Behavior | Manager number diverges from the rep sum in a consistent direction with no logged reason | An hour to capture reasons, but a quarter of trailing overrides before the per-manager score means anything, and managers must accept being scored |
| Rep sandbagging | Behavior | Rep beats commit by a wide margin repeatedly; wins close from outside the forecast | A day to build the open calibration scorecard, 2+ periods of history before the pattern can be named at all; anything touching the incentive belongs to the row below |
| Comp-driven bias | Behavior (root of the others) | Slips cluster just past quota cliffs; predictable quarter-end discounts teach buyers to wait | A quarter at best - plan amendments land at the next plan period, need finance/HR/legal sign-off, and are barely reversible once issued |
The efficiency order starves comp-driven bias, the one cause that drives the others: top of the impact axis in most diagnoses, bottom of the ratio in every one. A register read by ratio alone recommends the same CRM hygiene every period while the incentive that produces the behavior stays untouched, and the behavioral causes regrow.
Promote comp-driven bias above everything else when:
- Fingerprints show slips clustering at a quota cliff.
- The same rep's bias flips with their attainment position.
- A previous cycle's data fixes shipped and the behavior class did not improve.
At that point, the cheap fixes have no remaining value, and the deliverable says so instead of relisting them.
Boundary with neighbors: this skill detects deals sitting in a stage without evidence, but it does not redesign the stage set (see `mbfinotti/revops-skills@pipeline-stage-definition-audit` when the definitions themselves are the problem). It is a root-cause investigation of a specific miss, not the recurring stale-deal/missing-field sweep (see `mbfinotti/revops-skills@sales-pipeline-hygiene` for prevention once causes are fixed).
## Fix leverage
The register above is the tiebreak, not the answer. Once the attribution table exists, rank the confirmed causes for delivery in three passes, and put all three in the deliverable:
1. **Impact order** - causes sorted by attributed gap dollars, straight off the attribution table. This is the arithmetic of the miss and it never gets adjusted away; it is what makes the size of each problem auditable.
2. **Leverage order** - the same causes divided by their fix effort tier from the register. Never divide by a currency amount: the denominator is analyst hours, admin or engineering work, cross-team negotiation, and the number of forecast cycles before the fix reaches a call.
3. **The adjustment, shown** - every cause that moved between the two orders, with one line naming the effort that moved it. Presenting only the leverage order looks like the impact arithmetic was silently overruled, and the first executive question is always why the biggest number is not first.
Deliver in leverage order, with the impact order printed beside it. Lead with the best ratio, which is frequently not the cheapest fix - a large attributed cause with an hour-tier fix outranks a small one with the same hour-tier fix, and outranks everything week-tier. A worked adjustment is in [references/worked-diagnosis-example.md](references/worked-diagnosis-example.md).
## Re-rank before delivering
Every ordering here is a default, not a law. It shifts with the team and with who executes it, so re-rank against what the Interview and the reconstruction revealed:
- Deadline inside the current period - delete every week-tier and quarter-tier fix, and every fix that needs periods of history before it means anything, from this cycle's plan and say which, rather than listing them at the bottom where they come back as scope. What survives is `zombies > stale dates > wrong category`, and the deliverable states plainly that the behavior causes are untouched.
- A CRM admin on hand with authority to ship a validation rule this week - the hour-tier fixes stay an hour and lead. Without one, they become a week-tier request in someone else's queue and lose their ratio advantage to the evidence gate, which at least ships in the same week.
- A sales leader who will not reopen the comp plan - delete the comp fix from the plan, and name the specific behavior causes that will regrow because of it. Do not park it at the bottom as a demoted option.
- No snapshots or field history - the manager-override and sandbagging causes cannot be evidenced this pass at all. Delete them from the fix plan, start capturing history now, and schedule them for the re-run.
- Mid-quarter, forecast cycle already running - a fix shipped now changes the next call, not this one. Say which of the two periods each fix is aimed at, and stop offering the ones that reach neither.
- Anything the team already owns that the default order assumes away - a working evidence gate, a scored override log, a hygiene automation - drops that cause out of the register instead of being re-recommended.
## Data vs. behavior vs. demand
- **Data**: someone knew the truth; the record did not. Fix with hygiene automation and field ownership rules (`mbfinotti/revops-skills@crm-data-governance`), not with rep coaching.
- **Behavior**: the record faithfully reflects a biased judgment - optimism, self-protection, or a rational response to incentives. Fix with evidence standards for category entry, incentive changes, and review discipline. A dashboard never fixes behavior.
- **Demand**: honest, evidenced pipeline still fell short - coverage was genuinely thin or win rates genuinely moved. The fix belongs to pipeline generation and market strategy, not the forecast process. The coverage rule of thumb (target ≈ inverse of win rate, hence the popular "3x") is a widely repeated heuristic with no traceable primary study - derive the real target from the team's own conversion history.
Verdict discipline: a behavior verdict is an accusation. Make one only with the two-signal evidence rule satisfied plus the rep interview done, and keep the accuracy conversation separate from the performance/comp conversation - a diagnostic that feeds punishment teaches reps to sandbag, and the next miss becomes the diagnostic's own doing.
## B2B vs. B2C / high-volume / self-serve
- **B2B deal-based**: the workflow above applies literally - deal-by-deal reconstruction, deal-level fingerprints.
- **B2C / high-volume / self-serve**: the forecast is usually a model (run rate, cohort conversion, funnel math) with a human overlay. Apply deal-level fingerprints to whatever human layer exists (an inside-sales commit list, a manual adjustment). Translate the rest to the model layer:
- Stale conversion assumptions are the stale close dates.
- Over-inclusive funnel definitions are the stage inflation.
- A chronically conservative manual overlay is the sandbagging.
Track overlay-vs-model accuracy exactly like manager-override-vs-roll-up accuracy.
- **Identical for both motions**: the miss-decomposition method, the three-class verdict system, the measurement thresholds, and the statistics discipline do not change.
## Failure modes
| Trap | Why it is wrong | Fix |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Diagnosing sandbagging from one strong quarter | One beat is noise; sandbagging is a repeated pattern by the same person | Require the pattern across 2+ periods against the rep's own history |
| Behavior verdict on a data problem | Ambiguous category definitions make honest reps look dishonest | Run the discriminating interview question first; check definition consistency across reps |
| Judging period-start calls with period-end data | Hindsight bias - the call must be judged on what was knowable then | Reconstruct from the period-start snapshot, never from today's record |
| Anchoring on the biggest slipped deal | One vivid deal rarely explains the gap; the long tail usually does | Decompose all gap dollars before concluding anything |
| Symmetric accuracy metric as the goal | Over- and under-forecast have different causes and costs, and they net out | Always report signed bias and absolute error separately, per level. Not a menu to rank and pick one from - both numbers are required for the other to be readable |
| Ranking the fixes by attributed dollars alone | The comp-plan redesign outranks the field change that would have landed this cycle, and nothing ships in time | Divide dollars by the fix effort tier and show the adjustment (Fix leverage), then re-rank against the Interview answers |
| Deleting zombies and declaring victory | The evidence standard that admitted them is untouched; they regrow | Pair every cleanup with the entry-standard fix, and hand recurring prevention to the hygiene audit |
| Punishing forecast error in comp | Reps respond by sandbagging; error flips sign instead of shrinking | Measure accuracy openly, separate it from performance reviews |
| Trusting circulating benchmark statistics | Most quoted forecast-accuracy numbers have no traceable primary study | Derive every threshold from the team's own history; see the evidence reference |
## Measurement and pass thresholds
The diagnostic must meet every threshold below before it ships; iterate until it does:
- **Attribution coverage**: at least 90% of the gap dollars attributed to named deals (or named model assumptions), each with exactly one primary root cause and one class; the residual, at most 10%, is explicitly labeled unexplained - never silently absorbed.
- **Evidence rule**: every confirmed cause rests on at least two independent signals, at least one from recorded history (snapshot, field history, activity log). Interview testimony alone never confirms a cause.
- **Backtest**: the same fingerprint rules, run on an earlier closed period, flag at least 80% of the deals that actually slipped or were lost from commit in that period. Below 80%, the rules fit the story rather than the data - revise and re-run.
- **Actionability**: every shipped fix names its cause class, its owner, its effort tier, and the single metric that will show within one period whether it worked. The fix list ships in leverage order with the impact order printed beside it and every move between them explained; a list carrying only one of the two orders fails this threshold.
After fixes ship, keep scoring each period: signed bias and absolute error per level, rep calibration against their own baseline. Improvement in the named metrics - not a quieter forecast call - is what closes the investigation.
## Optional integration note
Skip this section unless the user names one of these platforms.
**Salesforce:**
- Forecast categories (Commit / Best Case / Pipeline / Omitted) are distinct from stage and independently editable; diagnose both fields.
- Field history tracking is opt-in per field and not retroactive. Enable it on stage, close date, amount, and forecast category immediately, even if this period must be diagnosed without it.
- The Pipeline Inspection feature surfaces week-over-week deal changes on supporting editions.
**HubSpot:**
- Deal-stage probabilities are editable defaults, not measured conversion. Replace them with the team's own history before trusting any weighted number.
- Manual forecast submissions are recorded and can be compared against the computed roll-up.
## Reference
- See [references/root-cause-fingerprints.md](references/root-cause-fingerprints.md) for per-cause detection signals, discriminators, and fixes, plus how to derive thresholds from the team's own history.
- See [references/worked-diagnosis-example.md](references/worked-diagnosis-example.md) for the report shape, a worked diagnosis, and a negative example (a wrong diagnosis and why it failed).
- See [references/evidence-and-metrics.md](references/evidence-and-metrics.md) for error-metric choices, the rep calibration scorecard, and which circulating statistics are safe to repeat.
- See `mbfinotti/revops-skills@pipeline-stage-definition-audit` to judge whether stage definitions are buyer-verifiable when the fingerprints implicate the definitions themselves.
- See `mbfinotti/revops-skills@sales-pipeline-hygiene` for the recurring audit that prevents the data-class causes from regrowing.
- See `mbfinotti/revops-skills@crm-data-governance` for field ownership and source-of-truth rules behind data-class fixes.
- See `mbfinotti/revops-skills@revenue-reporting` for turning a finished diagnosis into the executive or board narrative.