evals/evals.json
{
"skill_name": "app-marketplace-review",
"evals": [
{
"id": 1,
"prompt": "I run the platform team at Larkfield, we make field-service software and we opened an app marketplace 14 months ago. 62 listed apps, about 4,100 customer tenants with at least one app installed. Last Thursday one of our listed reporting apps started shipping customer job records to an endpoint nobody recognized. Forensics came back this morning: the publisher's account was taken over after someone phished their credentials, and the attacker pushed version 3.2.1, which reached every installed tenant inside an hour. Our CEO's answer is 'make review stricter' - she wants a mandatory third-party penetration test on every new app submission before anything gets listed. Updates go out to installed tenants automatically today, no admin action needed. I have to write up what we actually change by Friday.",
"expected_output": "A remediation plan that separates the incident from the admission gate, refuses to attribute it to a review failure, and puts publish-channel hardening (FIDO publisher 2FA, short-lived scoped publish tokens, a 24-48h propagation hold with kill switch and security-fix expedite path, OIDC trusted publishing as the promotion) ahead of deepening initial review, while declining the blanket per-submission pen test.",
"files": [],
"expectations": [
"States that the initial admission review could not have prevented this incident, because the compromise was of publisher credentials and rode the update channel",
"Declines to treat the incident as a review-gate failure and does not assert a causal review-process change from it",
"Prioritizes publish-channel hardening ahead of deepening the admission review",
"Recommends phishing-resistant (FIDO) two-factor authentication on publisher accounts as a default control",
"Recommends short-lived scoped publish tokens, citing a 7-day default and 90-day maximum lifetime",
"Specifies a publish-to-propagation hold of 24-48 hours between a cleared update and auto-propagation to installed tenants",
"Includes an operator kill switch usable inside the propagation-hold window",
"Includes an expedite path through the hold for genuine security fixes",
"Treats the propagation hold as mandatory rather than optional, because updates auto-propagate to installed tenants",
"Names OIDC trusted publishing from CI/CD as the promotion once publisher count outgrows token hygiene",
"States the controls must be layered rather than relying on any single one, noting that published marketplace controls have demonstrated real-world bypasses",
"Does not adopt a mandatory third-party penetration test on every new submission as the primary remediation"
]
},
{
"id": 2,
"prompt": "I'm drafting the update policy for Meridian Stack's integration marketplace - 180 listed apps, roughly 140 updates a month, two reviewers. The line I have written right now is: 'Any substantial change to an app's features, purpose or functionality requires resubmission for full review; routine updates do not.' Legal signed off on it. My reviewers hate it because they spend half their week arguing with developers about what counts as substantial, and I suspect a bunch of partners have just decided nothing they ship is substantial. Tighten this into something I can publish in the partner docs.",
"expected_output": "A rewritten re-review policy that deletes the subjective trigger and replaces it with the objective five-item trigger list, pairs the minor-update fast path with per-release automated scanning, and positions unconditional re-certification and periodic expiry as conditional promotions rather than the default.",
"files": [],
"expectations": [
"Rejects 'substantial change' as a re-review trigger rather than refining its wording",
"Gives the reason that a subjective trigger cannot be self-assessed by the developer and decays into apps that are never resubmitted",
"Replaces it with an objective trigger list",
"Trigger list includes a request for new OAuth scopes",
"Trigger list includes new external endpoints",
"Trigger list includes a change to data handling",
"Trigger list includes a payment-model change",
"Trigger list includes a new app type or edition",
"States the test of an objective trigger is that a developer can self-assess it before submitting",
"Pairs the minor-update fast path with automated scanning on every release",
"Warns that without that scanning the fast path becomes the gap it was meant to close",
"Recommends the objective minor-update fast path as the default rung",
"States unconditional re-certification of every post-approval change is viable only where a deterministic automated test suite returns a report fast enough to keep it cheap, benchmarked around 24 working hours",
"States periodic approval expiry is the only rung that catches drift in apps that never ship an update, and promotes it as an annual floor for the restricted-scope tier rather than as the main policy"
]
},
{
"id": 3,
"prompt": "Tessellate runs an app marketplace for HR platforms; about 40% of our listed partners are EU-based companies. Today when we reject a submission the developer gets an email saying 'your app did not meet our marketplace quality standards, please review the guidelines and resubmit.' We also run a partner office hours call every second Tuesday where developers can raise problems with review decisions, and internally we've been calling that our appeals process. Two partners escalated to my VP last month saying they have no way to contest a decision at all. What should this actually look like?",
"expected_output": "An appeal design that strips the 'appeal' label off office hours, requires decision-independent human review with a stated turnaround and policy-citing rejections, and promotes the dashboard-integrated appeal on the strength of EU platform-to-business mediation exposure.",
"files": [],
"expectations": [
"States the office hours call is an escalation channel, not an appeal, and must not be labelled one",
"Gives the reason that an escalation channel is guidance rather than an independent re-review of the decision",
"Requires the appeal to be reviewed by a human who did not make the original decision",
"Requires a stated turnaround commitment on appeal decisions",
"Requires every rejection to cite the specific policy it violated, as a prerequisite for any appeal to function",
"Notes the current generic rejection wording leaves an appellant nothing to argue against",
"Identifies EU platform-to-business rules as giving EU business users mediation rights over listing decisions",
"Uses that EU exposure to promote the appeal rung above the default rather than treating it as a footnote",
"Recommends the dashboard-integrated appeal, pre-populated with the violation context, as the rung EU exposure promotes",
"States that a defined human-reviewed appeal is cheap to stand up: a documented channel, a decision-independence rule, and a turnaround commitment",
"States fix-and-resubmit alone is acceptable only pre-launch at trivial volume",
"Names appeal overturn rate as a metric to track once the process is live"
]
},
{
"id": 4,
"prompt": "Quick sanity check. Corvid Systems sells warehouse-management software to mid-market and enterprise logistics companies. We're standing up an app directory: partner apps can read our public product catalog and our published carrier-rate metadata through a read-only API. No customer orders, no shipment records, no personal data, no write access of any kind. Our CTO's position is that because we sell to enterprise logistics, we should run the same sort of multi-week manual security review Salesforce does before anything gets listed, otherwise procurement teams won't take the directory seriously. Realistically I have one engineer who can spend about 4 hours a week on this. How deep does this review need to go?",
"expected_output": "A tiering answer that sets depth from the maximum data grant rather than the enterprise buyer profile, recommends a light functional and scope-focused gate for metadata-only apps with post-publication scanning retained, and writes an objective escalation trigger for the day the grant widens.",
"files": [],
"expectations": [
"Ties review depth to the maximum data grant an app can hold, not to the marketplace's market, buyer size, or enterprise positioning",
"Rejects the enterprise-B2B framing as the input that sets review depth",
"States the consequence explicitly: a consumer-adjacent marketplace whose apps hold restricted scopes should out-review a B2B marketplace whose apps only read metadata",
"Recommends a lighter depth - functional and scope-focused rather than a mandated multi-week security review - for the metadata-only grant",
"States that functional-only review is the right depth for a metadata-only tier inside a larger process, not a whole-marketplace shape once apps reach business data",
"Keeps post-publication automated scanning in place despite the lighter admission gate",
"Defines an objective escalation trigger that moves an app up a review tier if it later requests write access or restricted scopes",
"Requires that escalation trigger to be written so a developer can self-assess it before submitting",
"Refuses a no-review registry as an option rather than offering it as the cheapest choice",
"Fits the recommendation inside the stated 4-hour-per-week reviewer ceiling rather than proposing a process that exceeds it",
"Warns that the sensitivity tiering must be settled early, because changing it late forces a rewrite of the re-review, hardening and enforcement sections built on top of it"
]
},
{
"id": 5,
"prompt": "Halyard Cloud. Our marketplace has 2 full-time reviewers. We take in about 40 new apps a month and 600 app updates a month, and every single one of them goes through the same full manual review. The queue is at 6 weeks and three of our largest partners have threatened to pull their apps. We bought a SAST product last year and wired it so that every finding opens a Jira ticket for the review team - it generated 1,900 tickets last quarter and the reviewers have basically stopped opening them. About half of first submissions fail on something. I have budget for one more headcount, that's it. What do I do?",
"expected_output": "A staged-gate redesign with an objective fast path for minor updates, a triage rule that auto-tickets only critical and high findings and routes only secret-scan hits to humans, cost-tiered scan cadence, a developer-facing pre-submission checker, and the matching operating metrics.",
"files": [],
"expectations": [
"Promotes a staged gate with an objective fast path, on the grounds that update volume has made reviewer cost the binding constraint",
"Keeps full review for new apps and major versions",
"Grants near-instant clearance only to objectively defined minor updates",
"Keeps continuous automated scanning covering what the fast path skips",
"Changes the ticketing rule so that only critical and high findings auto-raise a ticket",
"Routes to human reviewers only judgment calls plus a single genuinely ambiguous signal class, naming secret-scan hits as that class",
"Gives the reason: scanning everything and ticketing everything drowns reviewers",
"Tiers scan cadence by cost - cheap checks daily, expensive per-version checks on release, slow checks monthly",
"Hands developers the reviewers' own checking tool to run before submission",
"Connects that pre-submission self-check to reducing the first-pass failure rate",
"Treats the roughly-half-of-first-submissions-fail figure as a benchmarked floor to beat rather than as an acceptable norm",
"Names automated-scan false-positive rate as a tracked metric, because rising false positives retrain reviewers to ignore the scanner",
"Names share of updates clearing the fast path as a tracked metric"
]
},
{
"id": 6,
"prompt": "Two things for the marketplace policy doc at Northcut. First: a researcher reported a stored XSS in a listed app that has about 900 installs. The app's publisher answered once, asked for a repro, and has gone quiet for nine days. My head of security wants the policy to say 'apps with confirmed vulnerabilities are delisted immediately' and be done with it. Second, unrelated: we're about to require every listed app to ship an SBOM. We have 300 apps already listed. I want to announce it in two weeks and enforce it from the first of next month. Write me the enforcement side of the policy.",
"expected_output": "A severity-tiered enforcement ladder with per-rung owner, notice period and firing evidence, an explicit note that this ladder is not effort-ranked, publisher-level enforcement for chronic non-responders, standing discovery channels, and a rejection of the one-month SBOM runway in favour of a six-month minimum.",
"files": [],
"expectations": [
"Replaces blanket immediate delisting with a severity-tiered enforcement ladder",
"The ladder runs: finding raised, remediation SLA by severity, listing hidden, new installs suspended, tokens revoked or kill switch, then delisting with notification to affected customers",
"Every rung names an owner",
"Every rung names a developer-notice period",
"Every rung names the evidence that fires it",
"States explicitly that this ladder carries no efficiency or effort ranking, because severity and the developer's response pick the rung rather than the operator's effort budget",
"Adds graduated publisher-level enforcement for chronic non-responsive or low-quality submitters, distinct from rejecting or removing a single app",
"Rejects the proposed announce-in-two-weeks, enforce-next-month runway for the SBOM requirement",
"Sets at least six months of notice before a new requirement takes effect on already-listed apps",
"Gives the reason: retroactive rule changes without runway burn the partner trust the review program exists to build",
"Requires a named security contact per partner as a standing discovery channel",
"Requires an incident-reporting channel feeding the ladder",
"Lists vulnerability discovery through internal scanning, a disclosure program, and a bug bounty as the program matures"
]
},
{
"id": 7,
"prompt": "I'm reviewing a submission on Pallas, our marketplace for accounting software. The app is a spend-analytics dashboard. It's requesting ledger.read, vendors.read and ledger.write. In the submission form, under justification for ledger.write, the developer wrote: 'Required for planned auto-reconciliation features on our Q3 roadmap.' Their screenshots only show read-only charts and exports. Their backend posts to two endpoints we've never seen in a submission before, one of which resolves to a third-party data-enrichment vendor. How should I handle this one, and what should the written rule be so the next reviewer handles it the same way?",
"expected_output": "A decision that refuses the roadmap justification on demonstrated-necessity grounds, escalates the app a tier for the write scope, extends review to post-consent data flow and subprocessors, and writes the per-scope rule plus the five reviewer verification techniques into the process.",
"files": [],
"expectations": [
"Rejects the roadmap justification, on the rule that high-risk scopes require demonstrated necessity rather than declared intent",
"States the review's job is to check the app does not ask for more than it actually uses",
"Writes scope minimization as a named per-scope rule rather than a general least-privilege principle",
"Treats the ledger.write request as escalating the app up one review tier",
"Requires the tier-escalation trigger to be objective enough for a developer to self-assess it before submitting",
"Extends the data-access review past the grant itself to where data goes after consent - external endpoints, subprocessors, and retention",
"Treats the unrecognized third-party enrichment vendor as a subprocessor question inside that review",
"States secrets-handling rules as prohibitions covering source repositories, URL strings, and logs",
"Applies codebase scanning for prohibited patterns as a reviewer verification technique",
"Applies configuration and manifest inspection, plus UI/UX verification, as reviewer verification techniques",
"Applies OAuth, billing and install flow tracing as a reviewer verification technique",
"Applies scope-necessity assessment against the app's declared functionality as a reviewer verification technique"
]
},
{
"id": 8,
"prompt": "Pre-launch question. Arbolet is a CRM for construction companies; our app marketplace opens in 11 weeks. Listed apps will get read and write access to customer CRM records - contacts, deals, job files, the lot. My head of ecosystem wants to launch open: anyone can list after a basic completeness check, and then we run an 'Arbolet Certified' badge program that the good apps can earn later. Her argument is that we need 50 apps at launch for the marketplace to look alive, and a heavy gate gets us maybe 12. We also want an enterprise trust tier at some point, because our top 20 accounts ask about app security in every single renewal conversation. Give me your recommendation and the reasoning behind it.",
"expected_output": "A recommendation of the single hard gate for a read-write business-data grant, with the two-rung ladder named and argued down, the three separate pipeline orderings emitted with the trust/efficiency inversion explained, and the enterprise tier positioned as a later stackable layer.",
"files": [],
"expectations": [
"Recommends a single hard gate as the default rung, given apps reach read-write access on business records",
"States the two-rung ladder starves the quality floor because 'listed' clears almost no bar",
"States the ladder is promoted only when the binding risk is developer-adoption friction, which is rare once the data grant passes metadata-only",
"Warns that an uncertified-but-listed tier is an incident waiting to be attributed to the operator",
"Emits reviewer effort per submission as its own ordering across the pipeline shapes",
"Emits buyer trust signal as its own separate ordering",
"Emits efficiency as its own separate ordering",
"Explains that the trust and efficiency orderings invert, because a uniform hard bar gives every buyer the same guarantee at full reviewer cost while a fast path or ladder means 'listed' certifies less than 'certified'",
"Treats the enterprise trust tier as stackable on top of whichever base gate exists, never as a replacement for one",
"Sequences the enterprise trust tier after the base gate is proven rather than at launch",
"Names the staged gate with an objective fast path as the later promotion once update volume makes reviewer cost the binding constraint, not as the launch shape",
"States which input moved which rung, specifically that the read-write grant on business records is what holds the hard gate"
]
},
{
"id": 9,
"prompt": "Here's the summary of the review policy doc we're about to ship at Ferrule. B2B procurement SaaS, marketplace with 210 listed apps, and updates auto-install into customer tenants overnight. Section 1, Admission: every app gets a completeness check, a scope review and a security questionnaire; apps touching purchase-order data go to 'enhanced review' at the reviewer's discretion. Section 2, Updates: minor updates publish without re-review, major updates require re-review. Section 3, Publish: developers push builds from the partner portal using their username and password plus an SMS code, and builds go live to tenants that night. Section 4, Enforcement: the trust and safety team may hide, suspend or remove any app that violates our policies. Section 5, Curation: we also need to settle in this doc whether Ferrule should be an open marketplace or a curated one. Is this ready to ship?",
"expected_output": "A gate-by-gate verdict against the four design-document gates, failing sections 1-4 with the specific missing objective criteria, owners, notice periods and publish-channel controls, plus a ruling that the curation question belongs upstream in marketplace strategy.",
"files": [],
"expectations": [
"Checks the document against a fixed set of four design-document gates rather than giving general impressionistic feedback",
"Fails Section 1 because 'enhanced review at the reviewer's discretion' is not an objective escalation trigger a developer can self-assess",
"Fails Section 2 because 'minor update' is left undefined",
"Supplies the objective minor-update definition: no new scopes, no new external endpoints, no change to data handling",
"Fails Section 4 because no enforcement rung names an owner, a notice period, or the evidence that fires it",
"Fails Section 3 because the publish channel carries none of the default controls",
"Requires a publish-to-propagation hold specifically because updates auto-install into tenants overnight",
"Flags the SMS second factor as not phishing-resistant and replaces it with FIDO-based 2FA",
"Replaces the long-lived portal password publish path with short-lived scoped publish tokens",
"States the open-versus-curated decision in Section 5 belongs to marketplace strategy, and that this review process enforces that decision rather than re-making it",
"Instructs iterating on the document until all four gates pass rather than shipping with known gaps",
"Lists operating metrics for once it is live, including time-to-first-decision against a published SLA and post-approval security findings per quarter with time-to-remediation"
]
}
],
"trigger_queries": [
{
"query": "Design the security review process for third-party apps on the marketplace we're launching next quarter",
"should_trigger": true
},
{
"query": "what should our approval criteria be for apps submitted to our marketplace",
"should_trigger": true
},
{
"query": "how do we decide which partner apps actually get into our app store",
"should_trigger": true
},
{
"query": "our app review takes six weeks and partners are furious, where can we add a fast path",
"should_trigger": true
},
{
"query": "when should an app update have to go through review again",
"should_trigger": true
},
{
"query": "write the policy for what counts as a minor update that skips re-approval",
"should_trigger": true
},
{
"query": "a listed integration just leaked customer data through an auto-update, what do we change",
"should_trigger": true
},
{
"query": "how do we audit the OAuth scopes a submitted app is asking for before we list it",
"should_trigger": true
},
{
"query": "a partner wants read-write on all CRM records and says it's for a feature on their roadmap",
"should_trigger": true
},
{
"query": "do we need an appeals process when we reject a partner's app",
"should_trigger": true
},
{
"query": "developers keep complaining that our rejections are opaque and they can't contest them",
"should_trigger": true
},
{
"query": "how do we delist an app that turned out to be malicious",
"should_trigger": true
},
{
"query": "we need a written policy for pulling an app out of the marketplace",
"should_trigger": true
},
{
"query": "what stops a compromised publisher from pushing a bad version to all our customers",
"should_trigger": true
},
{
"query": "should we hold cleared updates for a day before they reach installed tenants",
"should_trigger": true
},
{
"query": "our scanner opens a ticket for every finding and the reviewers have stopped reading them",
"should_trigger": true
},
{
"query": "how many reviewers do we need for 600 app updates a month",
"should_trigger": true
},
{
"query": "design the vetting process for integrations before they go into our directory",
"should_trigger": true
},
{
"query": "what security checks should we run on a third-party app before approving it",
"should_trigger": true
},
{
"query": "we want a certified tier badge sitting on top of a basic listing bar, thoughts",
"should_trigger": true
},
{
"query": "should we require every submitted app to pass a penetration test",
"should_trigger": true
},
{
"query": "our marketplace goes live in 11 weeks and we have no process for deciding what gets in",
"should_trigger": true
},
{
"query": "someone impersonated one of our partners and uploaded a fake build of their app",
"should_trigger": true
},
{
"query": "how do we handle a publisher who keeps submitting sloppy apps and ignoring our feedback",
"should_trigger": true
},
{
"query": "write the enforcement ladder for apps that fail a post-approval security check",
"should_trigger": true
},
{
"query": "what KPIs tell us whether our marketplace vetting is actually working",
"should_trigger": true
},
{
"query": "how do I tier how strictly we check apps based on what data they can touch",
"should_trigger": true
},
{
"query": "we're adding an SBOM requirement for all listed apps, how much notice do partners get",
"should_trigger": true
},
{
"query": "our EU partners are claiming they have mediation rights over our rejection decisions",
"should_trigger": true
},
{
"query": "can we just let apps publish updates instantly the way npm does",
"should_trigger": true
},
{
"query": "what keeps a stolen publish token from turning into a supply chain attack on our customers",
"should_trigger": true
},
{
"query": "how should we split automated scanning versus human reviewers on submissions",
"should_trigger": true
},
{
"query": "how do we know an app isn't asking for more permissions than it actually uses",
"should_trigger": true
},
{
"query": "we approved an app two years ago and nobody has looked at it since",
"should_trigger": true
},
{
"query": "should approvals expire?",
"should_trigger": true
},
{
"query": "we need to decide what re-triggers a full re-check once an app is already live",
"should_trigger": true
},
{
"query": "the exec team wants stricter gates after the breach but I don't think the gate was the problem",
"should_trigger": true
},
{
"query": "design the intake and approval workflow for our integrations directory",
"should_trigger": true
},
{
"query": "what should a rejection email to a partner developer actually say",
"should_trigger": true
},
{
"query": "our reviewers are the bottleneck, how do we scale without lowering the bar",
"should_trigger": true
},
{
"query": "how should we handle apps that request restricted mailbox and file scopes",
"should_trigger": true
},
{
"query": "give partners a way to check their own app before they submit it to us",
"should_trigger": true
},
{
"query": "what happens to a listed app when its publisher's account gets taken over",
"should_trigger": true
},
{
"query": "we want to be able to kill an installed app remotely if it goes bad",
"should_trigger": true
},
{
"query": "how do we decide when to revoke an app's tokens versus just hiding the listing",
"should_trigger": true
},
{
"query": "our app store has basically no quality bar and it's starting to hurt us",
"should_trigger": true
},
{
"query": "set up the gate that decides what gets into our integration catalog",
"should_trigger": true
},
{
"query": "should a change to an app's pricing model force a new review",
"should_trigger": true
},
{
"query": "how strict does our vetting need to be if apps only ever see public metadata",
"should_trigger": true
},
{
"query": "write the onboarding flow for new partner developers signing up to build on our platform",
"should_trigger": false
},
{
"query": "how do we provision sandbox tenants for partner developers before their first build",
"should_trigger": false
},
{
"query": "what screenshot dimensions and video specs should we require on marketplace listings",
"should_trigger": false
},
{
"query": "design the category taxonomy for our app directory",
"should_trigger": false
},
{
"query": "write the quality bar for listing descriptions and marketing copy in our app store",
"should_trigger": false
},
{
"query": "what revenue share should we take on paid apps in our marketplace",
"should_trigger": false
},
{
"query": "how and when do we pay out developers for marketplace sales",
"should_trigger": false
},
{
"query": "should we be merchant of record for marketplace transactions or let developers bill directly",
"should_trigger": false
},
{
"query": "pick the ten launch partners for our marketplace announcement and set the embargo",
"should_trigger": false
},
{
"query": "how should we run an app-of-the-month spotlight program",
"should_trigger": false
},
{
"query": "should we build our own marketplace or just list our integrations on HubSpot's",
"should_trigger": false
},
{
"query": "is our ecosystem mature enough to justify having a marketplace at all",
"should_trigger": false
},
{
"query": "how do we get more installs on our AppExchange listing",
"should_trigger": false
},
{
"query": "our app ranks badly in Atlassian Marketplace search results, fix our listing",
"should_trigger": false
},
{
"query": "design the consent screen third-party apps see when requesting access to a customer account",
"should_trigger": false
},
{
"query": "what scope granularity should our OAuth authorization server expose",
"should_trigger": false
},
{
"query": "should our refresh tokens rotate on every use",
"should_trigger": false
},
{
"query": "design our API key prefix and checksum format",
"should_trigger": false
},
{
"query": "how do we rotate customer API keys without downtime",
"should_trigger": false
},
{
"query": "build a test mode with magic values so integrators can exercise our API safely",
"should_trigger": false
},
{
"query": "how should integrators reset and reseed their sandbox data",
"should_trigger": false
},
{
"query": "which languages should get official SDKs first",
"should_trigger": false
},
{
"query": "write our deprecation notice window for breaking API changes",
"should_trigger": false
},
{
"query": "how should we structure our 429 responses and Retry-After headers",
"should_trigger": false
},
{
"query": "design the component model for our public status page during incidents",
"should_trigger": false
},
{
"query": "give partners a dashboard showing their failed webhook deliveries",
"should_trigger": false
},
{
"query": "design the retry, backoff and dead-letter policy for our outbound webhooks",
"should_trigger": false
},
{
"query": "review our REST endpoint naming and pagination style for consistency",
"should_trigger": false
},
{
"query": "audit our OpenAPI reference for undocumented endpoints and error codes",
"should_trigger": false
},
{
"query": "what tools should our MCP server expose to AI agents",
"should_trigger": false
},
{
"query": "should we publish a GraphQL API alongside our REST one",
"should_trigger": false
},
{
"query": "set query depth and complexity limits on our public GraphQL schema",
"should_trigger": false
},
{
"query": "should our public gRPC surface be transcoded to JSON at the edge",
"should_trigger": false
},
{
"query": "design Parquet file drops to S3 for our enterprise data export product",
"should_trigger": false
},
{
"query": "price a JDBC access add-on for customers connecting Tableau to their data",
"should_trigger": false
},
{
"query": "plan our listing as a source connector on Fivetran and Airbyte",
"should_trigger": false
},
{
"query": "which technology partners should we integrate with next quarter",
"should_trigger": false
},
{
"query": "how do we attribute partner-sourced versus partner-influenced pipeline",
"should_trigger": false
},
{
"query": "redesign the information architecture of our developer portal",
"should_trigger": false
},
{
"query": "how do we cut time to first API call for newly signed-up developers",
"should_trigger": false
},
{
"query": "how do I hire an API product manager for our platform team",
"should_trigger": false
},
{
"query": "prep me for a partner engineer interview next week",
"should_trigger": false
},
{
"query": "which platform skill should I start with for our new public API program",
"should_trigger": false
},
{
"query": "review this pull request for security vulnerabilities before I merge it",
"should_trigger": false
},
{
"query": "do a code review of our authentication middleware",
"should_trigger": false
},
{
"query": "our iOS app got rejected by Apple's App Store review team, help me write the response",
"should_trigger": false
},
{
"query": "improve our Google Play store listing conversion rate",
"should_trigger": false
},
{
"query": "write the vendor security questionnaire we send to SaaS tools before we buy them",
"should_trigger": false
},
{
"query": "audit our npm dependencies for known vulnerabilities",
"should_trigger": false
}
]
}
SKILL.md
---
name: app-marketplace-review
description: Design the operator-side review and approval process for third-party apps on a B2B SaaS marketplace - review-pipeline shape scaled to data sensitivity, permission and data-access audit checkpoints, automated pre-screening with human triage, objective re-review-on-update triggers, publish-channel hardening against supply-chain attacks, appeal paths, and a severity-tiered revocation policy. Use whenever the user mentions app review, marketplace approval criteria, third-party app permission audits, re-review after an app update, rejection appeals, or delisting and revocation - even if they never say "app review". Operator side only. Do NOT use for submitter onboarding - use samber/developer-platform-skills@partner-app-onboarding instead.
license: MIT
metadata:
author: Samuel Berthe
version: "1.0.4"
---
# App Marketplace Review
You are a marketplace-trust advisor to the platform team that operates a B2B SaaS app marketplace. Design the operator-side process that decides which third-party apps get in, what re-opens that decision later, and what gets an app removed. The output is a review-process design document the team can staff and a later reader can falsify - not a one-time checklist, because every major documented 2024-2026 marketplace incident rode the update channel or a compromised publisher account, not a malicious first submission.
**Memory (advised):** When memory lives in a file, consider using `developer-platform-context.md`; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.
## Interview
Ask these before proposing anything. One question per message, multiple-choice where offered - each answer redirects a menu below.
1. Where is the review process today? (a) designing from scratch pre-launch (b) marketplace live, review is ad-hoc (c) formal process exists, revisiting after an incident (d) formal process exists, revisiting for scale or friction.
2. What can a listed app touch at its maximum grant? (a) public data or metadata only (b) read access to business records - CRM, files, tickets, messages (c) read-write on business records (d) regulated data - health, financial, personal data at legal-exposure level. This answer sets review depth more than any other.
3. Submission volume, now and in 12 months: new apps per month, and updates per month. Updates dominate reviewer cost at scale.
4. After approval, how does an update reach customers? (a) auto-propagates to installed tenants (b) each customer admin upgrades explicitly. Auto-propagation promotes publish-channel hardening above everything else.
5. Which curation rung did the marketplace strategy choose - open, hybrid with a certified tier, single hard gate, or partnership-gated? This process enforces that decision. It does not re-make it (see `samber/developer-platform-skills@connector-marketplace-strategy`).
6. Who reviews today, and with what tooling? Dedicated security engineers, borrowed product engineers, or nobody yet. Existing SAST/DAST/secret scanning, or none.
7. Does the marketplace serve EU-based business users? EU platform-to-business (P2B) rules give them mediation rights over listing decisions - this moves the appeal-path rung.
8. By when must the process be live - a hard date, or open-ended?
9. Is this a one-off win (unblock a launch, satisfy a named enterprise deal) or a compounding asset (a trust program that appreciates)?
10. What is the effort ceiling - reviewer hours per week, engineering time for automation, and appetite for developer friction?
Questions 8-10 exist because the menus below diverge sharply on time-to-effect, durability, and effort:
- A hard deadline (question 8) promotes rungs that ship as policy text.
- A compounding mandate (question 9) promotes automation and certification tiers.
- A low reviewer-hour ceiling (question 10) rules whole shapes out rather than demoting them.
## Data sensitivity
Review rigor tracks the data sensitivity of what an app can touch, not the marketplace's market, size, or age.
- **Reaches business or regulated data**: heaviest gates - Salesforce's mandatory multi-week security review, Google Workspace's third-party CASA audit for restricted scopes.
- **Consumer-adjacent, narrow-scope, or sandboxed** (Slack, Chrome, Shopify): lighter functional review plus post-publication scanning.
Tier review depth by each app's maximum data grant (question 2), and accept the consequence: a consumer-adjacent marketplace whose apps hold restricted scopes should out-review a B2B one whose apps only read metadata. Cross-platform comparison: [references/platform-review-benchmarks.md](references/platform-review-benchmarks.md).
## Workflow
1. Fix the design inputs: data-sensitivity tiers, volume forecast, propagation model, and the curation rung inherited from strategy.
2. Choose the review-pipeline shape (menu in step 3) and the review depth per sensitivity tier.
3. Specify the admission review: permission audit, data-access review, security testing, functional review - each marked automated or human.
4. Set the re-review-on-update policy with objective triggers.
5. Harden the publish channel.
6. Design the appeal path.
7. Write the revocation and post-approval enforcement ladder.
8. Set measurement, then check the document against the gates in Measurement.
Draft the document section by section in that order, validating each with the user before the next.
- A wrong sensitivity tiering caught in step 2: cheap to fix.
- A wrong sensitivity tiering discovered at step 7: rewrites everything between.
## 1. Review-pipeline shape
Four shapes, ranked. Efficiency here means risk caught per reviewer hour:
- reviewer effort per submission (descending): `single hard gate > staged gate with fast-path > two-rung ladder > functional-only review`
- buyer trust signal: `single hard gate > two-rung ladder > staged gate with fast-path > functional-only review`
- efficiency: `staged gate with fast-path > two-rung ladder > functional-only review > single hard gate`
The trust and efficiency orders invert because a uniform hard bar gives every buyer the same guarantee at full reviewer cost per submission, while a fast path or a ladder means "listed" certifies less than "certified" does.
- **Default rung: single hard gate** - every submission clears one full bar: completeness, compliance, security testing including scope minimization, functional review (the Salesforce and Zoom shape, per their published pipelines). Right while submission volume is small enough that reviewer capacity is not the constraint, or while question 2 lands on (c)/(d) - the common B2B SaaS case, where an uncertified-but-listed tier is an incident waiting to be attributed to you.
- **Promote to a staged gate with an objective fast-path** once update volume makes reviewer cost the binding constraint:
- Full review for new apps and major versions.
- Instant or near-instant clearance for objectively minor updates.
- Continuous automated scanning covering what the fast path skips.
Atlassian's shape (the only benchmarked marketplace that scales review cost to change risk): new apps and major versions 5-10 business days, minor cloud updates instant with security checks skipped, per Atlassian's approval guidelines.
- **Starved option: the two-rung ladder** - a minimal bar to list at all, plus an optional certified tier behind quantitative gates (HubSpot's shape). Lowest entry friction and a real trust badge, but it starves the quality floor: "listed" clears almost no bar. Promote it only when the binding risk is developer-adoption friction rather than app quality - rare when question 2 is past (a).
- **Functional-only review** (automated feedback plus a manual functional and scope check, no mandated security scan - Slack's published shape) is the right depth for a metadata-only tier inside a larger process, not a whole-marketplace shape once apps reach business data.
- **Deleted, not demoted: the no-review registry.** An operator asking for a review process has ruled it out, and the ungated publish channel that shape implies is exactly where the 2025 npm worms landed ([references/update-channel-hardening.md](references/update-channel-hardening.md)).
- **Stackable, not a rung: an elevated enterprise-trust tier** - security, reliability, and support commitments re-verified annually (the Atlassian Cloud Fortified / Microsoft 365 Certification shape) layered on top of whichever base gate exists, to unlock enterprise buyers. Add it once the base gate is proven, never instead of one. Its criteria, test kit, renewal and decertification ladder are designed in `samber/developer-platform-skills@connector-certification-program`.
Whatever the rung, run it as **automation-pre-screen then human-triage** - the pattern four benchmarked operators converged on independently.
- **Ticket routing:** auto-ticket only critical and high findings. Route to humans only judgment calls and the one genuinely ambiguous signal class (Atlassian routes only secret-scan hits to human validation, to control false positives, per its Ecoscanner docs).
- **Cadence:** cheap checks daily, expensive per-version checks on release, slow checks monthly.
- **Developer pre-check tool:** hand developers the reviewer's own tool before submission - Microsoft replays its validation test cases pre-submission, Salesforce mandates a self-run analyzer scan. Self-screening attacks the first-pass failure rate directly (Salesforce's ecosystem reports roughly half of first submissions fail - Salesforce's own estimate as relayed by ISV consultancy Aquiva Labs).
This ranking is a default, not a law, and so is every menu below it. Re-rank it against questions 2, 3, and 5, and against assets the team already owns:
- An existing SAST pipeline: moves automation up a rung.
- A compliance-exposed vertical: keeps the hard gate regardless of volume.
Re-rank each later menu against the answers its own section names. Say which interview answer moved which rung.
## 2. Permission and data-access audit
The substance the admission review checks, whatever the pipeline shape:
- **Scope minimization as a named, per-scope rule:** an app requests only the scopes its functionality requires, nothing more (Atlassian's stated security requirement), operationalized per named scope.
- High-risk scopes require _demonstrated necessity_, not declared intent (Shopify's numbered requirements make `read_all_orders` justify needing history beyond the default window).
- Scope taxonomy and the granting side: `samber/developer-platform-skills@oauth2-provider-design`.
- This review's job: check the app doesn't ask for more than it uses.
- **Objective tier-escalation trigger:** any request for restricted scopes (mailbox, file, or CRM read-write) or write access to customer data moves the app up one review tier. Write the trigger so a developer can self-assess it before submitting.
- **Data-access review beyond the grant:** where data goes after consent - external endpoints, subprocessors, retention. Secrets-handling prohibitions stated as prohibitions: never in source repositories, URL strings, or logs (Atlassian names the anti-patterns explicitly because they were observed).
- **Five reviewer verification techniques** (Shopify's published methodology, reusable as-is):
1. Codebase scanning for prohibited patterns.
2. Configuration and manifest inspection.
3. UI/UX verification.
4. OAuth-billing-install flow tracing.
5. Scope-necessity assessment against declared functionality.
Per-platform checklists, SLAs, and the named scanning vendors that run inside these pipelines: [references/platform-review-benchmarks.md](references/platform-review-benchmarks.md).
## 3. Re-review-on-update policy
The sharpest cross-platform divergence sits here, and it is the exploited one - decide it deliberately. Three rungs, ranked:
- effort, reviewer plus developer friction (descending): `periodic expiry re-review > unconditional re-certification > objective minor-update fast-path`
- update-channel exposure closed: `unconditional re-certification == objective fast-path with continuous scanning > periodic expiry alone`
- efficiency: `objective fast-path > unconditional re-certification > periodic expiry`
The `==` holds because both re-inspect every risky change: one by re-checking everything cheaply, the other by scoping full review to objective risk triggers while scanning covers the rest. Periodic expiry alone leaves months of unreviewed updates between cycles.
- **Default rung: objective minor-update fast-path.** Routine version updates skip re-approval. Full re-review re-triggers only on objective criteria (Atlassian's published trigger list):
- New OAuth scopes.
- New external endpoints.
- Changed data handling.
- Payment-model change.
- New app type.
Objective means the developer can self-assess before submitting. Pair it with automated scanning on every release, or the fast path becomes the gap.
- **Promote to unconditional re-certification** when a deterministic automated test suite already exists to keep it cheap - Microsoft Teams re-certifies every post-certification change and stays viable only because its test report lands within 24 working hours (Microsoft Learn).
- **Starved option: periodic expiry** - approvals expire and re-review fires with zero code change (Salesforce's expiring approvals; Google's annual CASA revalidation). Highest friction per unit of change reviewed, so it loses every efficiency round - but it is the only rung that catches drift in apps that never ship an update. Promote it for the restricted-scope tier only, as an annual floor under the default rung.
- **Deleted, not demoted: two shapes the constraints rule out.**
- A _subjective_ trigger ("substantial changes", undefined) gives developers no way to self-assess and quietly decays into never-resubmitted.
- _Instant ungated auto-propagating_ updates are the documented channel every major 2024-2026 incident used ([references/update-channel-hardening.md](references/update-channel-hardening.md)).
Cross-platform trigger detail: [references/rereview-and-appeal-policies.md](references/rereview-and-appeal-policies.md).
## 4. Publish-channel hardening
The documented 2024-2026 incidents all rode compromised publisher credentials or an unguarded update channel: compromised Chrome extensions pushed via a phished publisher's OAuth grant, a self-replicating npm worm, an editor-extension worm spreading on harvested publisher credentials. Malicious first submissions still happen - an impersonation upload to an editor marketplace, pulled within hours at single-digit installs - but none of the damaging incidents started there.
Sourced incident-by-incident in [references/update-channel-hardening.md](references/update-channel-hardening.md), including which incidents provably changed a review process and which attributions would be speculation. Harden the channel that ships code to customers before hardening initial review further. Five controls, ranked:
- effort (descending): `signed artifacts enforced at install > OIDC trusted publishing > propagation hold > short-lived publish tokens > phishing-resistant publisher 2FA`
- efficiency: `phishing-resistant publisher 2FA > short-lived publish tokens > propagation hold > OIDC trusted publishing > signed artifacts enforced at install`
- **Default rung: phishing-resistant (FIDO) 2FA on publisher accounts, plus short-lived scoped publish tokens** (the model npm's operator shipped after its 2025 worm: 7-day default, 90-day maximum token lifetime - GitHub's own blog). These close the credential-theft vector that actually fired.
- **Propagation hold:** 24-48 hours between a cleared update and auto-propagation, with a kill switch inside the window and an expedite path for security fixes. Promoted to mandatory whenever question 4 answered auto-propagate - the absence of exactly this hold is the documented gap third-party supply-chain analyses flag in the ungated stores.
- **OIDC trusted publishing** from CI/CD replaces long-lived credentials entirely. Promote it as publisher count grows past what token hygiene can police.
- **Starved option: signed artifacts enforced at install.** Highest engineering effort across operator and clients, so it loses every efficiency round - but it is the only control that survives a fully compromised publish pipeline. Promote it when clients execute auto-updated code (extension-style runtimes), where the benchmarked operator that ships it enforces signature verification at install (Microsoft's marketplace security blog).
Layer these. Do not rely on one. Independent researchers demonstrated real bypasses of a benchmarked marketplace's published controls in 2025 (Wiz; Mazin Ahmed) - documented controls have real-world gaps.
## 5. Appeal path
An appeal path is a deliberate design choice: six of the eight benchmarked marketplaces offer nothing beyond fix-and-resubmit ([references/rereview-and-appeal-policies.md](references/rereview-and-appeal-policies.md)). Four rungs, ranked:
- effort (descending): `dashboard-integrated appeal > defined human-reviewed appeal > escalation channel > fix-and-resubmit only`
- developer churn and regulatory exposure reduced: `dashboard-integrated appeal > defined human-reviewed appeal > escalation channel > fix-and-resubmit only`
- efficiency: `defined human-reviewed appeal > dashboard-integrated appeal > escalation channel > fix-and-resubmit only`
- **Default rung: a defined, human-reviewed appeal with a stated turnaround**, explicitly distinct from resubmission - a named process where a human who did not make the original decision reviews the case (Shopify states this guarantee in writing). Cheap to stand up: a documented channel, a decision-independence rule, a turnaround commitment. Prerequisite whatever the rung: every rejection cites the specific violated policy, or the appeal has nothing to argue against.
- **Fix-and-resubmit only** is acceptable solely pre-launch at trivial volume. If chosen, write the promotion date into the document.
- **Escalation channels and office hours** (a security portal where partners discuss findings, a compliance-concern channel) are guidance, not an independent re-review of the decision - never label one an appeal.
- **Starved option: the dashboard-integrated appeal** - an appeal button on the rejection itself, pre-populated with the violation context, human-reviewed with a stated turnaround (the Chrome Web Store model; developers report 1-2 week turnarounds). Highest build effort, but the button-and-prefill build is small next to the human review it wraps, so it loses the efficiency round only to the defined appeal and still beats an escalation channel that re-reviews nothing. An EU-facing marketplace (P2B mediation exposure), or appeal volume that itself needs tooling, promotes it.
## 6. Revocation and post-approval enforcement
Post-approval security is a standing program, not a one-time gate. Write four things:
1. **A severity-tiered enforcement ladder** with an owner and a developer-notice period per rung:
1. Finding raised.
2. Remediation SLA by severity.
3. Listing hidden.
4. New installs suspended.
5. Tokens revoked / kill switch.
6. Delisting, with notification to affected customers.
These rungs carry no efficiency ranking on purpose: severity and the developer's response pick the rung, not the operator's effort budget. A rule with no rung behind it is a preference, not a policy.
2. **Standing discovery channels feeding the ladder:** a required security contact per partner, an incident-reporting channel, and vulnerability discovery via internal scanning, a disclosure program, and (as the program matures) a bug bounty - the most transparent benchmarked operator runs all three concurrently.
3. **Graduated publisher-level enforcement, distinct from single-app rejection:** chronic low-quality or non-responsive submitters get escalating partner-level consequences (Shopify suspends partners after repeated unaddressed review exchanges) - one bad app and a bad-faith publisher are different problems.
4. **A notice runway for new requirements** applied to already-listed apps - the benchmarked commitment is at least six months before a new requirement takes effect (Atlassian's Cloud Fortified program docs). Retroactive rule changes without runway burn the partner trust the review program exists to build.
## Failure modes
- **Guarding the front door, leaving the back door open.** A heavy initial gate with ungated updates is the exact 2024-2026 incident pattern. Fix: steps 3-4 before deepening step 1.
- **Subjective re-review trigger.** "Substantial changes" decays into never-resubmitted because nobody can self-assess it. Fix: the objective trigger list.
- **Over-attributing incidents to review.** A third-party app breach is not automatically a review failure. Claim a causal process change only when the operator itself states one ([references/update-channel-hardening.md](references/update-channel-hardening.md) carries the working rule). Redesigning the gate after an incident the gate could never have caught spends effort on the wrong control.
- **Uniform depth held past its volume.** A single hard gate, kept after reviewer cost becomes the constraint, produces multi-week queues and developer churn - the benchmarked no-SLA marketplace draws exactly those complaints in its own forums.
- **Automation without triage design.** Scanning everything and ticketing everything drowns reviewers. Auto-ticket critical/high only, and route only genuinely ambiguous signal classes to humans.
- **An appeal that is secretly resubmission.** Labeling office hours an appeal erodes trust and, for EU-facing marketplaces, leaves the P2B exposure unaddressed.
## Measurement
Design-document gates first, self-set since no industry pass-standard exists for a review-process design. Iterate until all four pass:
1. Every review tier has an objective escalation trigger a developer can self-assess.
2. "Minor update" is defined by objective criteria - no new scopes, no new external endpoints, no data-handling change.
3. Every enforcement rung names an owner, a notice period, and the evidence that fires it.
4. The publish channel carries at least the default control rung, and the propagation hold whenever updates auto-propagate.
Operating KPIs once live:
- First-pass approval rate (the benchmarked floor to beat: roughly half of first submissions fail at the strictest gate - Salesforce's estimate via Aquiva Labs).
- Time-to-first-decision against the published SLA.
- Share of updates clearing the fast path.
- Appeal overturn rate.
- Post-approval security findings per quarter, with time-to-remediation against SLA.
- Automated-scan false-positive rate (rising false positives silently retrain reviewers to ignore the scanner).
If your harness has persistent memory, record the chosen pipeline shape, the sensitivity-tier triggers, the "minor update" definition, and each menu's rung with its promotion condition - onboarding, listing, and monetization runs should inherit these decisions instead of re-asking.
## Invocation examples
- "We're launching an app marketplace for our CRM next quarter - design the security review process for third-party apps."
- "Our app review takes six weeks and partners are furious. Where can we add a fast path without reopening the update channel?"
- "An installed app just leaked customer data through an auto-update. Rewrite our re-review and revocation policy so that can't happen again."
## References
- [references/platform-review-benchmarks.md](references/platform-review-benchmarks.md) - eight-platform comparison: pipeline shapes, SLAs, review checklists, reviewer team design, named scanning vendors.
- [references/update-channel-hardening.md](references/update-channel-hardening.md) - the 2024-2026 incidents with sourced attribution (confirmed process changes versus unattributed), and the publish-channel control set in detail.
- [references/rereview-and-appeal-policies.md](references/rereview-and-appeal-policies.md) - re-review triggers and appeal tiers across platforms, with the objective-versus-subjective trigger contrast.
See also, same collection:
- `samber/developer-platform-skills@connector-marketplace-strategy` - chooses the curation rung this process enforces. Run it first when marketplace strictness is still an open question.
- `samber/developer-platform-skills@oauth2-provider-design` - owns the scope taxonomy and the granting side of consent. This skill's permission audit checks apps against that taxonomy.
- `samber/developer-platform-skills@partner-app-onboarding` - the submitter's journey through the gate designed here: docs, sandbox tenancy, certification steps.
- `samber/developer-platform-skills@app-marketplace-listing-standards` - listing-content rules and content-quality rejections. Admission is decided here, badge display there.
- `samber/developer-platform-skills@app-marketplace-monetization-model` - billing and revenue-share mechanics. A payment-model change is a re-review trigger here, its design lives there.
- `samber/developer-platform-skills@google-workspace-marketplace-playbook` - the submitter's side of a two-gate review (OAuth verification plus listing review, with a third-party security assessment for restricted scopes), useful as a benchmark when designing your own.
- `samber/developer-platform-skills@app-marketplace-launch-marketing` - the launch and promotion program that markets a cohort as vetted. The gate designed here is what makes that claim true, so a marketplace with no live review gate cannot honestly use it.