News
Prompt injection in robots.txt files is very rare according to a Signals Corps study of Common Crawl.
Out of 3,881,598 real files, only one contained an attempt, and it was a security firm's test.
375,582 sites, or 9.7%, have a robots.txt with text addressed to AI.
61,682 Shopify stores ask agents to have their user install the Shop skill.
The reference MCP fetch server, when it refuses a page, passes the whole robots.txt file to the model.