Skip to content

AgentPassBot

Last updated

AgentPassBot is AgentPass's crawler. It visits a website when someone runs AgentPass's free agent-readiness scan on it, when a business using AgentPass has scheduled rescans of its own site, and when AgentPass runs its own automated shopping test of a business's public website to show that business the results. It reads a small number of public pages and stops.

How to recognize it

Every request AgentPassBot makes includes AgentPassBot in its user agent. The scan and most test requests send:

AgentPassBot/1.0 (+https://getagentpass.com/bot)

Automated shopping tests that use a real browser send:

Mozilla/5.0 (compatible; AgentPassBot/1.0; +https://getagentpass.com/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36

robots.txt is respected

Before reading any page, AgentPassBot fetches your robots.txt. It follows the rules for the AgentPassBot user agent, and falls back to your rules for all crawlers (*) when there are none. If your robots.txt can't be loaded because of a server error, AgentPassBot doesn't crawl at all. It also slows down when your robots.txt sets a Crawl-delay. The free scan follows these rules for every page it reads. AgentPass's automated shopping tests follow them for their crawl, and don't test a website whose robots.txt keeps AgentPassBot off its homepage or can't be read; the few pages a test's steps open after the crawl, such as the cart and the checkout page, aren't checked against each rule.

To keep AgentPassBot off your site entirely, add this to your robots.txt:

User-agent: AgentPassBot
Disallow: /

Allowing AgentPassBot through a firewall

If your security service (Cloudflare, Vercel, Akamai, a WordPress security plugin, …) blocks AgentPassBot, the scan stops and your report shows what blocked it, with steps to fix it. AgentPassBot never tries to get around a firewall, bot check or CAPTCHA. To let it through, allow requests whose user agent contains AgentPassBot. On Cloudflare, that's a WAF custom rule with the action “Skip” and this expression:

(http.user_agent contains "AgentPassBot")

AI assistants and AI search tools also read websites from cloud servers, so it's worth checking that the same settings don't block the AI tools you want customers to use.

What the free scan does

  • At most 20 pages per scan — usually fewer — chosen from your homepage, sitemap and links.
  • A few requests at a time, with per-request timeouts of 8 seconds and a time limit for the whole scan.
  • Public pages only: the scan never signs in, submits forms, adds items to carts or places orders.
  • Scans are rate-limited per website, so the same site can't be scanned over and over in a short time.
  • It doesn't follow links to private network addresses or other internal systems.

AgentPass's automated shopping tests

AgentPass also runs automated shopping tests of some businesses' public websites, to show the business what an AI shopping agent would run into. A test takes the steps a shopper's AI agent would: it looks for a product and its price, checks availability and, on stores with a public cart, may add one item to a cart through the store's public cart endpoint or its Add to cart button, change the quantity and open the checkout page.

  • It never signs in, fills in checkout or payment details, pays or places an order. It stops when the checkout page loads, and a cart it creates is never checked out.
  • It never solves or works around a CAPTCHA, login or bot wall; it records that it was blocked.
  • Its crawl reads at most 20 public pages and follows your robots.txt. Each step then opens only the few pages it needs, such as a product page, search results, the cart and the checkout page.
  • Tests are run and scored by fixed rules, never by AI.
  • It may save up to 8 screenshots of the pages it visited, shown only in a private report for that business.

What happens to what it reads

For the free scan, AgentPassBot extracts facts relevant to AI agents — like structured product data, prices, locations and discovery files — to build a readiness report. It keeps the addresses and titles of the pages it read and what it extracted, not copies of whole pages. For automated shopping tests, AgentPass also keeps what each step did and found (addresses, prices, status codes) and up to 8 screenshots of the visible part of a page. See our privacy notice.

Contact

If AgentPassBot is causing problems for your website, or you have questions about it, email hello@getagentpass.com. Please include your domain and, if you can, the time of the visits. To ask us not to test your website again, or to delete a test of it, email us with your domain.