Skip to main content
Narrative crawls your site with its own crawler rather than a third-party index. The goal is to see your pages the way search and answer engines see them: fetched over plain HTTP, read as served, with no browser and no JavaScript, and reached by following links from your homepage.

​
Identifying the crawler

Every request carries this user agent:
It requests HTML only (Accept: text/html, application/xhtml+xml, ...) and discards anything else. It does not fetch images, scripts, stylesheets or fonts, and it does not execute JavaScript. What it reads is the raw response your server sends, which is what most AI crawlers read too.
The crawler fetches every page it keeps. HTML is stored so the audit can judge it; nothing is stored from pages your robots.txt disallows.

​
How a crawl runs

A crawl is started from Site Health, and also runs once when a brand is onboarded. One crawl runs per brand at a time; a second request while one is running waits for it rather than doubling the load on your server.
1

robots.txt and sitemaps

The crawler fetches robots.txt once per host and parses it with a parser that follows Google’s specification, including * wildcards, $ anchors and longest-match Allow precedence. It then fetches the sitemaps the file names, following sitemap indexes up to 50 files per host, and seeds the crawl with the URLs they declare. Both files are kept so the audit can judge them.
2

Breadth-first from the homepage

Starting at your homepage, it follows internal <a href> links level by level, up to 10 clicks deep and up to 10,000 pages. Subdomains of your domain are included. Tracking parameters are stripped and URLs are normalized so one page is counted once.
3

Record everything, render nothing

For every page: the final URL, every redirect hop on the way to it, the HTTP status, the response headers, the HTML, and every link on the page with where it points. Nothing is rendered or executed.
4

Build the link graph

When the crawl finishes, a second pass reads the whole graph and computes, per page: inbound and outbound link counts, broken outbound links, click depth from the homepage, a link score, and near-duplicate groups. Every crawl’s graph is kept, so you can see what changed between crawls.
5

Audit and measure

The audit library runs over the pages and the graph, Core Web Vitals are fetched from Chrome’s field data, and any open Fixes are checked against what the new crawl found.

​
Politeness

The crawler is designed to be a good guest on your server.

​
Allowing or restricting NarrativeBot

Narrative only crawls sites you have connected as your brand, so there is rarely a reason to restrict it. If your robots.txt blocks all crawlers by default, add an allowance for NarrativeBot so Narrative can see your pages:
To keep a section out of Narrative’s crawl, disallow it for NarrativeBot the same way you would for any crawler:
A robots.txt that blocks NarrativeBot also blocks the audit from seeing your pages, and the pages it cannot see will show as unfetched in Site Health. It does not change what the real engines can see, so a block here only hides problems from you.

​
Where the crawler reads from, and what it stores

  • Fetched: robots.txt, sitemaps, and HTML pages on your domain and its subdomains that robots.txt allows.
  • Stored: the HTML of each page, its headers, status, redirect chain, the links it carries, and the graph facts above. All of it scoped to your brand and to the crawl session that produced it.
  • Not fetched: images, scripts, stylesheets, fonts, media, or anything off your domain. Outbound links are recorded as URLs but not followed.