You should consider testing loading time of your site using some testing tools like pingdom tools (tools.pingdom.com), and also check in on various browsers - if you're using FF 4.0 beta version you may face ridiculous issues!
And, yeah even Chrome & Opera have their own share of miseries, and same goes with IE..!!!
So, just ensure that it's NOT browser specific. Here's how you can use Pingdom tools
It'll show you which element is taking more time to load, and what the root cause is!