# FeedmindBot

FeedmindBot is the crawler behind the Feedmind AI Readiness Check. When someone submits a shop URL, it reads a small sample of that store’s pages the way an AI assistant does — the plain, no-JavaScript HTML — to score how readable the catalog is to tools like ChatGPT, Perplexity and Gemini.

## What it is, and why it fetched your page

FeedmindBot runs on demand: a human ran an AI-readiness scan for your domain at getfeedmind.com. It is not a bulk web crawler and does not spider the whole internet — one scan reads your homepage plus a representative sample of product pages (about a dozen), then stops.

It fetches the plain, no-JavaScript HTML — exactly the surface an AI assistant sees. It never renders scripts, fills forms, signs in, or adds to cart.

## What you’ll see in your logs

Requests carry this User-Agent:

```
FeedmindBot/1.0 (+https://getfeedmind.com/bot)
```

A scan makes a handful of GET requests over a short window, paced to be polite (below), then goes quiet.

## How it behaves

- Respects robots.txt, including Crawl-delay — it never fetches faster than you ask, and reduces how many pages it samples rather than ignoring a long delay.
- Reads only the no-JavaScript HTML: no rendering, no scripts, no cookies, no logins.
- Fetches your homepage and about a dozen product pages per scan — never your whole catalog.
- Supports Web Bot Auth: when request signing is enabled, each request is signed with an Ed25519 key per RFC 9421, verifiable against our key directory.

## How to allow it

If you gate crawlers, allow the FeedmindBot token in robots.txt:

```
User-agent: FeedmindBot
Allow: /
```

For firewalls that verify signed bots (Web Bot Auth), FeedmindBot signs its requests with HTTP Message Signatures (RFC 9421). When signing is enabled, the public key is published as a JWKS at https://getfeedmind.com/.well-known/http-message-signatures-directory.

## How to block or opt out

To stop FeedmindBot crawling, disallow it in robots.txt — we honor it:

```
User-agent: FeedmindBot
Disallow: /
```

To remove an already-published scorecard for your store, email privacy@getfeedmind.com. We mark the store opted-out, which removes its scorecard from search indexing and from our sitemap.