This thread was split off from Getting my website's content into ChatGPT
Watch out for what you’re asking for!
I just got a message from my hosting provider that my server is at 87% of its allocated bandwidth. The culprit? 3 IPs (74.7.241.55, 74.7.242.50, and 74.7.243.200) from ChatGPT bots that have been hammering my websites (about 50) during March. Luckily, we’re the 30th but last year I had a similar issue with the same bots, but it was the 17th.
Update.
It looks like I was partly guilty !
The main culprit is OpenAI’s GPTBot, accounting for 98.8% of all traffic. A single IP (74.7.241.55) has been running since midnight at a relentless ~2 requests/second, scraping 59,233 unique pages and pulling 18.9 GB of content — just in the first 9.5 hours covered by this log. A second OpenAI IP (74.7.243.200) joined the party around 09:23.
In the 2n part of the log, the situation has escalated. The secondary IP 74.7.243.200 has now fully ramped up and is running in parallel with 74.7.241.55 — both crawling at ~6,800 req/hour simultaneously.
Why it can’t stop: Your site appears to generate unique URL tokens for each page view (those long base64-like strings after the path, e.g. /Curso/.../yuBz_2g-vL0P...). From GPTBot’s perspective, every link it finds is a brand new page — it’s fallen into a crawler trap with an effectively infinite number of URLs to visit.