Well, the base library itself (jquery's main lib for example) should be okay being loaded, but yeah once you start mixing libraries you have the potential of interference.
Someone on another forum I'm on asked something like, "wouldn't it be better if browsers came with jQuery already loaded?"
I didn't think so (why jQuery, for one thing), but it's an interesting question. Reducing accidental errors and interference would mean strong limits on developer freedom.
With WP, the problems I hear the most about involve widgets. The trouble with widgets is, they're all written by other people.
Like, I know the jQuery guys are pretty aware of interference and they work hard to write their code tight as far as namespacing and whatnot, but any Joe or Jane can write a widget plugin thingie. And they offer it on the web. And it does something useful or neat so lots of others use it. But it wasn't written to the same high standard as the library it uses. It's not careful with its scope or whatever.
Also, re the question of loading times: often widgets are connecting to other servers. Suddenly it doesn't matter how super-fast your server is... some widget is stopping the loading of your page because it's waiting for acknowledgement from, say, Facebook, so it can load an iframe with fb stuff in it.
Go to deviantart.com and watch all the requests to servers that aren't deviantart.com or .net. That's what you're waiting for, all those server requests. Your page is then only as fast as the slowest server, if you're measuring full page load.