references/ai-search.md
# AI search and crawler controls
## Contents
- Eligibility and evidence
- Useful content without filler
- Crawlers and policy
- robots.ts example
- llms.txt
- Measurement and sources
## Eligibility and evidence
Google says AI Overviews and AI Mode need no special schema or AI text file.
The page must be indexed and eligible for a Search snippet. This is eligibility,
not a guarantee of selection or citation. Apply ordinary technical SEO first.
Check the effective Search generative AI inclusion setting (see below) when
diagnosing absence from Google's AI features.
Do not describe FAQPage, a TL;DR, llms.txt, an author bio or a particular word
count as a demonstrated AI ranking factor. Separate vendor-documented behavior
from hypotheses.
## Useful content without filler
Answer the actual question directly. Use descriptive headings and original,
verifiable information. Add a Q&A section only for useful questions that the
page has not already answered. Do not repeat the hero in a summary, FAQ and
keyword paragraph. There is no required opening word count.
A visible update date helps when freshness matters, but must track meaningful
changes. Keep the primary answer available in server-rendered HTML where
practical; crawler JavaScript support varies. Clear HTML is more portable than
relying on client-only fetching.
## Crawlers and policy
Check current vendor docs before changing robots rules. Training, search and
user-initiated fetching are different purposes; a vendor may use separate bots
or a product-control token. Preserve the owner's policy instead of copying a
blanket allow/block list.
| Vendor | Training | Search | User-initiated retrieval |
|--------|----------|--------|--------------------------|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None (PerplexityBot is not used for training) | PerplexityBot | Perplexity-User |
Google Search and its AI features use Googlebot controls; `noindex`, `nosnippet`,
`data-nosnippet` and `max-snippet` limit what they can show. `Google-Extended`
has no separate HTTP user agent: it is a robots product token controlling both
Gemini training and certain Gemini/Vertex AI grounding uses. It does not control
Google Search inclusion or ranking. Do not call it a training-only switch or
promise that blocking it has no effect on all AI referral channels.
Search Console's [Search generative AI control](https://support.google.com/webmasters/answer/16908024) (Settings > Search generative AI)
offers include/exclude/inherit for AI Overviews, AI Mode and Discover AI features; top-level properties default to include, child properties inherit.
It controls applicable AI-feature inclusion, not AI training or inclusion/ranking elsewhere in Search.
Blocking a search crawler can prevent direct content retrieval; it does not
prove that a URL or brand can never be mentioned through other sources.
User-initiated fetchers differ: OpenAI says robots.txt "may not apply" to
ChatGPT-User, Perplexity-User generally ignores it, and Anthropic says all three
of its bots honor it. Each vendor's settings are independent, so allowing
OAI-SearchBot while disallowing GPTBot is a valid policy.
Inspect hosting firewall rules separately when diagnosing denied public crawls. A
User-Agent string can be spoofed; use vendor-documented verification methods
for bot identity, and do not weaken preview protection to make an audit pass.
## robots.ts example
Start with the existing wildcard policy. Add named groups only when a deliberate
exception is needed. Specific groups do not inherit wildcard restrictions.
```typescript
import type { MetadataRoute } from 'next';
export default function robots(): MetadataRoute.Robots {
const disallow = ['/api/', '/admin/'];
return {
rules: [
{ userAgent: '*', allow: '/', disallow },
// Only if the owner chooses to opt out of OpenAI training:
// { userAgent: 'GPTBot', disallow: '/' },
// If a named search group is needed, repeat the restrictions:
// { userAgent: 'OAI-SearchBot', allow: '/', disallow },
],
sitemap: 'https://your-site.com/sitemap.xml',
};
}
```
## llms.txt
Treat llms.txt as an optional documentation-discovery convention, not a proven
citation boost. Google Search does not use it, so it neither helps nor hurts
there. It can be useful when developers or
agents explicitly consume documentation through it. Verify the intended consumer
before adding it. Prefer a static `public/llms.txt` for a fixed document; use a
Route Handler only when generation is needed, with caching supported by the
installed Next.js configuration. Do not copy `dynamic = 'force-static'` into a
Cache Components app.
## Measurement and sources
Use referral analytics and concrete citation examples to observe changes.
Missing referrals do not prove missing citations; crawler hits do not prove
citations. Google's Search Console includes AI-feature traffic in overall Web
search reporting, so do not label the whole export as AI traffic.
Search Console's Generative AI performance report shows
AI Overviews and AI Mode link impressions by page, country, date and device.
If it is unavailable, check impression volume and the effective inclusion
setting; insufficient impressions or exclusion are possible causes.
Sources to re-check for changing policies:
- [Google AI features](https://developers.google.com/search/docs/appearance/ai-features)
- [Google AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
- [Generative AI performance report (Search)](https://support.google.com/webmasters/answer/16984139)
- [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended)
- [OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [llms.txt proposal](https://llmstxt.org/)
references/checklist.md
# Scoped SEO audit
Select representative route types, languages and deployment environments.
Record production URLs and the relevant Next.js version/configuration.
Do not add every optional feature to make a checklist pass.
1. **Page usefulness:** Does each indexable page answer a distinct task with
accurate information? Check misleading availability/dates, duplicate pages
and repetitive UI copy.
2. **Access and discovery:** Check actual status/redirect chains, internal
links, robots rules, sitemap entries and index directives. Verify public
versus preview/private intent.
3. **Canonical/localization:** Inspect absolute canonical URLs, query handling
and reciprocal hreflang where applicable. Compare Google-selected canonicals
through URL Inspection when available.
4. **Served metadata:** Inspect title/description, social images and icons for
the route. Check complete responses and streaming behavior for relevant
bots. Test image URLs; missing optional tags are not automatically defects.
5. **Structured data:** Add only relevant accurate types. Check current
eligibility, visible facts and safe serialization; validate the production
output.
6. **Rendering and usability:** Inspect critical public content in the raw
response ([diagnosis](troubleshooting.md#content-missing-from-the-served-html)),
then directly open the route and exercise hydration/navigation. Check mobile
equivalence, media dimensions and the changed interaction.
7. **Performance:** Use field CWV where available and lab tools for diagnosis.
Distinguish LCP, INP and CLS evidence; lab Lighthouse is not field INP.
8. **Publication:** Change representative content and verify page, metadata
and sitemap freshness through the real invalidation/rebuild path.
9. **Crawler policy:** Preserve intentional training/search distinctions and
access protection. Check firewall/access logs when public crawls fail.
For Google AI eligibility, check the effective Search Console inclusion
setting (including inheritance); see [AI search policy](ai-search.md#crawlers-and-policy).
10. **Search evidence:** Use indexing/URL Inspection for indexing and canonical
questions, performance for query/traffic questions, and analytics for
referrals. A performance export cannot establish all three.
Rank findings by impact and evidence, link affected routes/files and record
checks that could not run. Separate confirmed bugs from hypotheses and
optional improvements. See [troubleshooting.md](troubleshooting.md) and
[Google Search Essentials](https://developers.google.com/search/docs/essentials).
references/json-ld.md
# Structured data
Start from the page's real content and intended consumer. Schema.org validity
and Google rich-result eligibility are different checks. Consult the current
[Google search gallery](https://developers.google.com/search/docs/appearance/structured-data/search-gallery)
and the selected feature's documentation before choosing a type; supported
features and restrictions change.
Retired Google rich results that older examples still add (as of 2026-10):
FAQ (gone for all sites from 2026-05-07), HowTo (2023), Course Info, Claim
Review, Estimated Salary, Learning Video, Special Announcement and Vehicle
Listing (2025), Practice Problems (2026-01). Dataset markup serves Dataset
Search only. Such markup can remain for other consumers but yields no Google
rich result ([Search updates](https://developers.google.com/search/updates)).
Mark up only facts users can verify on the page or through its legitimate
associated content. Do not invent reviews, ratings, prices, stock, authors,
credentials or organization details. An optional FAQ section does not justify
schema solely for search visibility.
Use stable absolute URLs and identifiers for the relevant entities. Share
organization/page data where useful; `@graph` is an organizational option,
not a requirement. Avoid conflicting duplicates from manual markup and a
library. In App Router, a plain `application/ld+json` script is appropriate;
it does not require executable-script loading strategies.
Serialize data safely. JSON text inside an HTML script can contain
`</script>`; escape `<` after JSON serialization or use an established safe
serializer, for example `JSON.stringify(data).replace(/</g, '\\u003c')`.
Do not concatenate untrusted strings into markup. See the
[Next.js JSON-LD guide](https://nextjs.org/docs/app/guides/json-ld).
Validate the served production markup with the relevant consumer's validator.
Google's Rich Results Test assesses supported features; a Schema.org
validator answers a broader vocabulary question. Fix invalid or misleading
markup, then inspect Search Console reporting where available. Valid markup
does not guarantee a rich result or ranking improvement.
references/metadata-api.md
# Metadata contracts
Use the installed Next.js docs and
[Metadata API](https://nextjs.org/docs/app/api-reference/functions/generate-metadata)
for signatures. Metadata exports run in Server Components; choose
`metadata` or `generateMetadata` in the same route segment, not both.
Export `viewport` / `generateViewport` for viewport-related fields;
`themeColor`, `colorScheme` or `viewport` inside `metadata` emit no tag, only
a build warning.
Google ignores the `keywords` meta tag; do not tune it for Google visibility.
See [supported meta tags](https://developers.google.com/search/docs/crawling-indexing/special-tags).
## URL resolution and inheritance
Set `metadataBase` to the intended public origin when using relative metadata
URLs. It applies to canonical/language alternates as well as social images.
An absolute field URL ignores `metadataBase`. An unset child canonical inherits
the parent's, so a root homepage canonical needs deliberate per-page overrides.
Missing-base behavior can differ between versions, fields and hosts; current
docs require a base for relative URL-based fields. Inspect emitted production
URLs rather than treating a development fallback or warning as the contract.
Metadata merges shallowly from root to leaf. Replacing a nested object such as
`openGraph` can discard inherited fields; explicitly carry forward needed
values or use a shared helper. File-based metadata can override config-based
metadata. Verify title templates, canonical queries and per-route image
selection in the served response.
In localized pages, provide absolute reciprocal hreflang URLs, including the
page itself. Preserve the distinction between language versions and unrelated
regional pages; avoid generating alternates for nonexistent routes.
## Async and cached metadata
Follow the installed async contracts for `params`, `searchParams` and generated
metadata IDs. Do not copy v15/16 examples into an older application without
checking types.
Metadata can be static, cached or request-dependent. With Cache Components,
request-dependent metadata and prerendered page content must meet that mode's
requirements; don't force every page into `use cache`. A cached
`generateMetadata` return must be serializable under the documented contract:
use URL strings rather than `URL` instances where required.
Next.js can stream metadata to capable clients and block for HTML-limited
bots. Check the installed `htmlLimitedBots` behavior and the complete response
for relevant agents before diagnosing “missing head tags”. The streamed tags
can appear in `<body>`; the default HTML-limited list includes Twitterbot,
Bingbot and Slackbot. Configuring `htmlLimitedBots` replaces that list rather
than extending it. Do not broaden overrides without evidence.
## Social images, icons and manifest
Use [metadata file conventions](https://nextjs.org/docs/app/api-reference/file-conventions/metadata)
when they fit. Static and generated OG/Twitter images, favicon/icon/apple-icon
files and manifests have different formats and limits; read the exact
convention before adding an export or file.
For generated images, follow
[ImageResponse](https://nextjs.org/docs/app/api-reference/functions/image-response)
rendering, font and bundle limitations: Satori supports flexbox but not
`display: grid`, the bundle (JSX, fonts, images) is capped at 500 KB, and
fonts must be ttf/otf/woff. Static `opengraph-image` files over 8 MB and
`twitter-image` files over 5 MB fail the build. Test the actual generated URL
and image, including fonts and long titles.
A web manifest is useful for an installable web app; it is not a generic SEO
requirement. Avoid duplicating equivalent icon/social-image definitions merely
to increase metadata volume.
references/sitemap-robots.md
# Sitemaps and crawl policy
Use [sitemap](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap)
and [robots](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)
file conventions from the installed Next.js version. Static files can be
sufficient for a fixed site; dynamic generation needs an actual update policy.
## Sitemaps
Include absolute canonical URLs intended to be indexed. Check real statuses
and exclude redirects, unavailable pages and deliberate noindex routes.
Sitemap presence does not override crawl restrictions, canonical selection or
indexing decisions.
Use `lastModified` only for meaningful content changes from reliable data.
Do not stamp every URL with the current build/request time. Google ignores
`priority` and `changeFrequency`; they do not need tuning.
Each sitemap is limited to 50,000 URLs and 50 MB uncompressed. Split large
sets, verify the generated endpoint paths and expose all children through a
sitemap index or their submission/discovery mechanism. Next.js
`generateSitemaps` does not itself create a sitemap index. In 15+, children are
served at `<segment>/sitemap/<id>.xml` in dev and production; in 16 the sitemap
function receives `id: Promise<string>`. Expose the children through the
`robots` sitemap array or a separately served index, and verify those URLs.
Localized entries need absolute reciprocal language alternates including
themselves. Image/video extensions should describe actual media and meet the
consumer's requirements.
`sitemap.ts`/`robots.ts` are cached by default unless request-time APIs or
supported dynamic configuration change that. A database-backed sitemap may
therefore run only at build time. Verify that a
publication update reaches the served sitemap using the configured cache,
invalidation or rebuild path. See
[Google sitemap guidance](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap).
## Robots and previews
[Robots rules](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
control cooperative crawling, not access or index removal. Keep a noindex
page crawlable when the crawler must see its directive. Authentication and
deployment protection serve a different purpose.
Specific user-agent groups do not inherit the wildcard group's restrictions.
Repeat intended shared rules in named groups or avoid unnecessary groups.
Do not block render-critical assets when Google needs them. Google's parser
does not use `host` or `crawl-delay`; use redirects/canonicals for the preferred
host.
`NODE_ENV=production` also describes optimized preview builds. Use the
deployment's explicit environment policy (`VERCEL_ENV` on Vercel where
appropriate), and verify preview protection and index directives independently.
Preserve the owner's AI crawler policy; see [ai-search.md](ai-search.md).
Verify production robots status/body, affected paths and crawl diagnostics
before narrowing a rule.
references/troubleshooting.md
# SEO diagnosis
Start with an affected production URL and the exact observation. Compare the
complete served response, browser behavior and Search Console evidence;
these describe different layers.
## Indexing and crawl status
“Discovered – currently not indexed” says Google found the URL but has not
crawled it yet, typically because the crawl was rescheduled to avoid overloading
the site. “Crawled – currently not indexed” says a crawl did not result in
indexing. Neither status identifies a single cause or establishes a
content-length problem.
Use URL Inspection and the relevant indexing report to check fetch success,
crawl permission, noindex, declared and Google-selected canonicals. Inspect
redirects, status codes, duplicate intent and whether the page has a distinct
useful purpose. Check server/firewall logs when access is uncertain.
Change robots rules only after confirming the route should be public, and
preserve unrelated restrictions. A robots block can stop discovery of a
noindex directive. Indexing requests are optional follow-up after meaningful
fixes; repeated submissions do not guarantee indexing.
## Content missing from the served HTML
The browser hides this fault, because hydration fills the section in. Fetch the
production URL without JavaScript and search the response for a visible phrase
and a data value from each important section. Count fallback skeletons too.
With Cache Components (Next.js 16), check two causes:
- **The value is in the RSC payload (`self.__next_f`) but not in the markup.**
The server had the data and the prerender left the Client Component out.
Reading the clock or randomness during its render (`Date.now()`, `new Date()`,
`Math.random()`, `crypto.randomUUID()`) stops the prerender at that point, and
the nearest Suspense fallback is what the HTML holds. Libraries do this too:
TanStack Query v5 reads `Date.now()` in `useQuery` for a numeric `staleTime`,
and `dehydrate()` reads it as well. A `<Suspense>` boundary silences the
build's prerender error by deferring the content, so the error going away does
not mean the content is in the HTML. Keep the read out of server render, for
example `staleTime: isServer ? "static" : ms` with seeded `initialData` (tested
on Next.js 16.3 with TanStack Query 5.103), or move it to an effect.
- **The value is in neither.** Check the section's `cacheLife`: `revalidate: 0`,
`expire` under 5 minutes or `stale` under 30 seconds exclude it from
prerenders, and it resolves at request time. `stale` under 5 minutes also keeps
it out of the App Shell. For content a crawler must read, keep `stale` and
`expire` at 5 minutes or more and refresh live values on the client.
## Missing or wrong metadata
Compare source and the complete production response for the installed
metadata-streaming mode and relevant bot. Check inheritance, file precedence,
relative URL resolution and the correct deployment origin.
See [metadata-api.md](metadata-api.md).
## Content is stale
Trace the actual publication, cache and invalidation path for the page,
metadata and sitemap. A database read or changed source file is not proof the
deployed route refreshed. Test the real update rather than substituting a
generic cache lifetime.
## HTML is correct but interaction fails
Indexable HTML does not prove hydration. Directly open the route in a
production browser and test buttons, forms and Suspense content. Inspect
console errors, streamed RSC responses, failed assets, client/server initial
state and the deployed proxy.
Test both a direct production load and client navigation: a partially
prerendered route (for example one reading `searchParams`) can serve correct
SEO HTML yet leave its Suspense content unhydrated on a direct load while
working after client navigation. An inactive browser tab can also delay streamed reveal;
rule that out before reporting a failure. Preserve a reproducible case and
investigate the installed framework/configuration.
Use [URL Inspection documentation](https://support.google.com/webmasters/answer/9012289)
for report semantics. State missing credentials, reports or verified bot logs
as limitations instead of inferring them from unrelated exports.
SKILL.md
---
name: nextjs-seo
description: Next.js App Router SEO implementation and audits. Use for metadata and social images, sitemap/robots, canonicals and localization, structured data, Core Web Vitals evidence, AI crawler policy or search-indexing diagnosis. Not for general Next.js feature work unrelated to SEO.
---
# Next.js SEO
Start from the page's task, production response and observed search behavior.
Read the installed Next.js version and relevant `node_modules/next/dist/docs/`
pages (bundled from 16.2); otherwise use matching official documentation.
`Metadata`, `Viewport` and `MetadataRoute` types are also in
`node_modules/next/dist/lib/metadata/types/metadata-interface.d.ts`. Keep
framework APIs, caching mode and deployment policy consistent with the project.
Useful, distinct content matters more than a longer page. Give titles,
headings and descriptions separate jobs; add a FAQ, summary, badges or extra
pages only when they help the reader. Publication dates must reflect meaningful
changes. Verify authorship, credentials and factual claims.
## Implementation and audit scope
- Verify public URLs, redirects, status codes, crawl rules, index directives
and canonicals before adding markup. A sitemap supports discovery; it does
not make a page indexable or guarantee indexing.
- Keep important public content available in the served HTML when practical.
A Client Component can be server-prerendered; `use client` alone does not
remove content from HTML. Check the raw response for a visible phrase and a
data value from each important section: hydration hides a missing section in
the browser, and the build can still pass. See
[Troubleshooting](references/troubleshooting.md#content-missing-from-the-served-html).
- Use the App Router Metadata API and file conventions. Resolve relative
metadata URLs with the correct production `metadataBase` and verify emitted
absolute URLs. Do not rely on missing-base behavior across versions/hosts.
- Configure refresh/invalidation for data-backed pages, metadata and sitemaps.
A database query alone does not guarantee publication updates reach them.
With Cache Components enabled, follow the installed Next.js caching docs
(`node_modules/next/dist/docs/`). No cache setting is an SEO directive, but
a short `cacheLife` also decides whether content is in the prerendered HTML.
- Structured data must describe real visible content and a supported consumer.
Check current rich-result eligibility instead of adding every schema type.
- Preserve the owner's crawler policy. Training, search and user-triggered
retrieval have different controls; robots rules are not access control.
## Read for the affected area
- [Metadata](references/metadata-api.md): inheritance, social images, icons and streaming.
- [Sitemaps/robots](references/sitemap-robots.md): discovery, localization and preview policy.
- [Structured data](references/json-ld.md): current eligibility and safe serialization.
- [AI search](references/ai-search.md): crawler purposes, policy and evidence.
- [Audit checklist](references/checklist.md): scoped production and Search Console checks.
- [Troubleshooting](references/troubleshooting.md): indexing, missing HTML content and hydration diagnosis.
Separate evidence in the report: a performance export shows queries/traffic;
URL Inspection and indexing reports address crawl/index/canonical status;
field measurements address Core Web Vitals. Lighthouse is a lab diagnostic
and cannot establish field INP. Good field thresholds at the 75th percentile
are LCP ≤ 2.5 s, INP ≤ 200 ms and CLS ≤ 0.1.
Verify status, headers and complete production HTML, including relevant bot
responses, then open the route directly and test its interaction. A spoofed
User-Agent tests response handling, not actual verified bot access. Report
unavailable checks explicitly and avoid ranking, indexing or citation promises.