Starting a few days ago, we've run into some major issues with our forum and wiki. I'll start at the beginning to give the necessary background.
The site is a fansite for a tv show (Game of thrones) and it's rising popularity has been straining the server for some time. However, last weekend (when the show premiered its second season) it ran reasonably well after some tweaks and upgrades. We had brief outages, but they'd be intermittent slowness and not complete choking.
Starting ... Friday, I think, things started going very badly. We were hit by a large number of requests from msnbot that took the site down for quite a while, forcing us to add an .htaccess block on the msnbot. Our host also did some other tweaks.
Since then, the site has been performing extremely poorly in what we estimate to be similar traffic to the previous weekend, where it did alright. Most alarming is that at times the server load will just spiral completely out of control, going from an average of 5-10 to as high as 200. Today, we had the forum running just fine with almost 1700 users, plus I don't know how many users on the wiki, but then all of a sudden the load spiraled out of control.
When this happens, it just keeps going up. 256 web requests are left waiting, 300 MySql threads lock, and load climbs and climbs and climbs. Not having root access, we can't stop and restart things, so our expedient is to rename the index files for the wiki and forum to kill hits to them... but load still climbs for a long while.
Given that the load keeps climbing after we've renamed those index file, we're thinking it has to be a server configuration issue rather than an issue with the MediaWiki or the IPB Forum, but we have no clue where to start looking for the culprit. What information do we need to collect to be able to figure out what's happening? As I said, we don't have root access, but right now we don't even know what questions to ask our host or what sort of information to ask our host for.