I had a sudden increase in Googlebot accessing my website. The last week of May this bot was burning about 25Gb per day of bandwith whereas “normal” users were using around 2Gb
Eventually this caused an overuse of bandwith costing me an hefty US$476 ! Pretty high for a no-commercial website !
Anybody else around beeing hit by something similar ?
How often do you really need the Google bot to crawl? In my case, it only shows up once every two weeks (and I don’t update my personal site much if any, so that’s fine). You may want to consider once every x hours instead milisecs
Also, you may consider to create a txt file for the bots, to tell them where they should go so they don’t try to crawl areas of no interest but which do reflect in your bandwidth