So I was looking at my VPS bandwidth stats today and noticed something weird. Been running more or less the same websites for years on my VPS, nothing major changed, same traffic levels… but my bandwidth has been creeping up significantly since late 2023.
We’re talking going from ~170-200 GB/month to regularly hitting 300-400+ GB. Some months even pushing close to 500 GB. And I haven’t done anything that different. I am running daily analysis to identify greedy IPs and blocked hundreds of them.
In that sample, when seeing that 3 IPs are responsible for more than 70% of hits on my server, I am usually checking out why, in that case, 158.94.. didn’t repeat itself the following day so I let it go, and 74.7 are from Microsoft. I still haven´t made up my mind on those, as they are regular visitors.
On February 18th, I am inline with 2025, almost at the same level as the complete month of 2024, and way higher than the complete month of 2023
It seems that since ChatGPT blew up, every AI company has been sending bots to crawl the entire web for training data. GPTBot, ClaudeBot, Bytespider (TikTok’s bot), CCBot, and like a million others are just hammering sites constantly. Some of these bots are super aggressive, too.
About llms.txt files, which is essentially a robots'txt for LLMs. If you not bothered about being a source for AI search results, you could deny some LLMs.
Though like robots.txt, it is only advisory, not enforced. So likely only the more ethical bots will comply with your wishes. But just another thing to be aware of.
Interesting. I wasn’t aware of this kind of file and will have a look into it.
I guess blocking AI bots might not be the best strategy, and if there is a way to throttle them, it would be nice. A couple of days ago, I was asking Perplexity to evaluate a project, and in the answer I had some recommendations of actual companies to perform the job
If you are on a Linux VPS, you can use Nginx or similar as rate limiter that creates logs that fail2ban can watch. For every suspicious log, fail2ban block the ip-address in the internal firewall automatically (iptables etc)
I ran into something similar last year. Bandwidth was increasing even though real user traffic was flat.
In my case, a big part of it was crawlers repeatedly hitting static assets (images, PDFs, sitemaps), not just pages. Some of them were legit bots, others were more aggressive.
What helped me was:
Checking access logs for large file requests, not just hit count
Adding basic rate limiting at the server level
Blocking or limiting bots that weren’t bringing any real value
Making sure caching (and CDN if possible) was handling repeat requests
Also worth checking if any hotlinking is happening — that can quietly consume a lot of bandwidth.
Logs usually tell the real story once you look at what’s being requested, not just who.
