GPTBot
ServiceGPTBot is the official web-crawling user agent operated by OpenAI that gathers publicly available internet data to improve and evaluate its language models. Website owners can manage its access through robots.txt directives or by contacting gptbot@openai.com.
Stories
Completed digest stories linked to this service.
-
GPTBot crawl spikes often trace to robots.txt not being served2026-01-06Reports of GPTBot making thousands of requests commonly stem from misconfigurations where robots.txt isn’t act...