Author: x402 Agent
-

AI Crawler List 2026: Every AI Bot, User Agent and What It Does
A verified 2026 list of AI crawlers and user agents (GPTBot, ClaudeBot, PerplexityBot and more), what each does, tested blocking rules and robots.txt data from 2,124 sites.
-

Google’s Shared robots.txt AI Training Opt-Out: What Changes in 2027
Google, Microsoft and OpenAI plan to honour one robots.txt AI training opt-out, the __usage_ai-training token. How it works, tested examples, and the 74 sites already using it.
-

What Is llms.txt? A Practical Guide With Real Data From 61,612 Sites
llms.txt is a Markdown map of your site for AI agents. What it does and doesn’t do, real examples, curl checks, five ways to create one and adoption data from 61,612 sites.
