If Joomla is doing everything you want and you know it and are comfortable with it maybe it needs to be optimized for speed. Perhaps the server needs to be tuned up for better load optimization.
All of what I call "industrial strength" CMS packages will load up the server because they have a lot more to do that just serve content. Often times there are many things that go on in the background and various areas are exposed to the developer to exploit and customize. As a result, you use more memory and bandwidth.
Actually the more I searched "Joomla Drupal Comparison", the more instances I found where people had run both systems and found that Joomla had a speed disadvantage. Here are a couple more opinions I found: