Yeah. I personally use Wordpress as my basic CMS for most things instead of Drupal, but it's basically the same deal.
There are lots of performance tweaks you can do on the server end that help:
- OpCode Caching (APC is my favorite)
- MemCaching (this gets very tricky, use with caution)
- Optimizing queries (at least the ones you write)
- Using functions that don't run the same query again
- Add caching on your database (lots of options here)
You'll inevitably have to scale your servers as well (you aren't going to handle 1,000,000 users / month on a social media site on one server). Clustering your database is relatively easy to set up, but can be costly (MySQL for example requires a minimum of 4 boxes to properly cluster). Clustering your web server can be a lot trickier, depending on if you have local caching and how you handle it. Personally I prefer to offload any caching to it's own server (cluster) and keep the static files in another place as much as possible (especially images).
If you want to support 1,000,000 users, you have a lot of research ahead. There is no way around getting into server management when you are those numbers.
Back to my point though, the better your code, the less hardware you'll need to run it.