Training bots and retrieval bots deserve different policies. How to read your logs, write a per-bot robots.txt, and avoid disappearing from AI…