> ## Documentation Index
> Fetch the complete documentation index at: https://docs.yournarrative.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# NarrativeBot: how we crawl

> What Narrative's crawler does on your site, how it behaves, and how to allow or restrict it in robots.txt

Narrative crawls your site with its own crawler rather than a third-party
index. The goal is to see your pages the way search and answer engines see
them: fetched over plain HTTP, read as served, with no browser and no
JavaScript, and reached by following links from your homepage.

## Identifying the crawler

Every request carries this user agent:

```text theme={null}
NarrativeBot/1.0 (+https://yournarrative.ai/bot)
```

It requests HTML only (`Accept: text/html, application/xhtml+xml, ...`) and
discards anything else. It does not fetch images, scripts, stylesheets or
fonts, and it does not execute JavaScript. What it reads is the raw response
your server sends, which is what most AI crawlers read too.

> [!NOTE]
>
> The crawler fetches every page it keeps. HTML is stored so the audit can judge
> it; nothing is stored from pages your robots.txt disallows.

## How a crawl runs

A crawl is started from **Site Health**, and also runs once when a brand is
onboarded. One crawl runs per brand at a time; a second request while one is
running waits for it rather than doubling the load on your server.

**1. robots.txt and sitemaps**

The crawler fetches `robots.txt` once per host and parses it with a parser
that follows Google's specification, including `*` wildcards, `$` anchors
and longest-match `Allow` precedence. It then fetches the sitemaps the
file names, following sitemap indexes up to 50 files per host, and seeds
the crawl with the URLs they declare. Both files are kept so the audit can
judge them.

**2. Breadth-first from the homepage**

Starting at your homepage, it follows internal `<a href>` links level by
level, up to 10 clicks deep and up to 10,000 pages. Subdomains of your
domain are included. Tracking parameters are stripped and URLs are
normalized so one page is counted once.

**3. Record everything, render nothing**

For every page: the final URL, every redirect hop on the way to it, the
HTTP status, the response headers, the HTML, and every link on the page
with where it points. Nothing is rendered or executed.

**4. Build the link graph**

When the crawl finishes, a second pass reads the whole graph and computes,
per page: inbound and outbound link counts, broken outbound links, click
depth from the homepage, a link score, and near-duplicate groups. Every
crawl's graph is kept, so you can see what changed between crawls.

**5. Audit and measure**

The [audit library](https://docs.yournarrative.ai/technical-seo/audit-library.md) runs over the pages and
the graph, [Core Web Vitals](https://docs.yournarrative.ai/technical-seo/core-web-vitals.md) are fetched
from Chrome's field data, and any open [Fixes](https://docs.yournarrative.ai/technical-seo/repairs-and-fixes.md)
are checked against what the new crawl found.

## Politeness

The crawler is designed to be a good guest on your server.

| Behaviour | Detail |
| - | - |
| **Respects robots.txt** | A disallowed URL is never fetched. `Crawl-delay` is honoured per host, up to 5 seconds. |
| **Per-host concurrency** | Starts at 4 requests in flight per host and adapts: any refusal halves it, and it only climbs back after a run of clean responses. It never goes below one request at a time. |
| **Backs off on refusal** | A `429` or `5xx` response pauses that host and retries with exponential backoff. `Retry-After` is honoured up to 3 minutes. A host that keeps refusing is given up on rather than hammered. |
| **Bounded** | At most 10,000 pages, 10 clicks deep, 5 redirects per URL, 20 seconds per request. |
| **One crawl at a time** | Per brand. Re-running a crawl replaces the previous graph pass rather than stacking on it. |

## Allowing or restricting NarrativeBot

Narrative only crawls sites you have connected as your brand, so there is
rarely a reason to restrict it. If your robots.txt blocks all crawlers by
default, add an allowance for NarrativeBot so Narrative can see your pages:

```text theme={null}
User-agent: NarrativeBot
Allow: /
```

To keep a section out of Narrative's crawl, disallow it for NarrativeBot the
same way you would for any crawler:

```text theme={null}
User-agent: NarrativeBot
Disallow: /internal/
Crawl-delay: 2
```

> [!WARNING]
>
> A robots.txt that blocks NarrativeBot also blocks the audit from seeing your
> pages, and the pages it cannot see will show as unfetched in Site Health. It
> does not change what the real engines can see, so a block here only hides
> problems from you.

## What the link graph computes

| Measure | What it means |
| - | - |
| **Click depth** | Shortest number of clicks from the homepage. The homepage is 0. Pages at depth 4 are flagged, past 5 fail: deep pages are crawled less and receive less authority. |
| **Link score** | Internal authority, 0 to 100, from a PageRank-style pass over your internal links. Your best-linked page scores 100; others are proportional. A page under 5% of the best is flagged as buried. |
| **Inbound links** | How many of your own pages link to it. Zero (and not the homepage) is an orphan. |
| **Broken outbound links** | Internal links on the page that point to a URL that did not load. Reported against the page holding the link, which is the one you can edit. |
| **Redirect chain** | Every hop the crawler walked to reach the final URL. Two hops is flagged, three or more fails. |
| **Near-duplicate group** | Pages whose visible text is at least 90% the same, by a MinHash comparison of word pairs. Pages in a group compete to answer the same question. |

## Where the crawler reads from, and what it stores

* **Fetched**: `robots.txt`, sitemaps, and HTML pages on your domain and its
  subdomains that robots.txt allows.
* **Stored**: the HTML of each page, its headers, status, redirect chain, the
  links it carries, and the graph facts above. All of it scoped to your brand
  and to the crawl session that produced it.
* **Not fetched**: images, scripts, stylesheets, fonts, media, or anything
  off your domain. Outbound links are recorded as URLs but not followed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.