I believe so when done properly, but it can just as easily be done poorly. Good content semantically marked up SHOULD be good enough for all devices assuming you don't shoe-horn too much crap on the page. IMHO if it's too slow for mobile, it's too slow for desktop too.
That's why my target for the average page on a website is 72k in 16 or less separate files; which makes it fast EVERYWHERE. Media queries and shifting the content around doesn't change that. You have a megabyte sized train wreck built from dozens or even hundreds of separate files, it's going to suck EVERYWHERE, just more so on mobile... and unless it's something inherently heavy like a image gallery, I refuse to deploy any page larger than 144k in 32 files.
Setting meaningful size limits helps EVERYBODY.
Which has NOTHING to do with if it's responsive or not, and everything to do with how good your markup is, how minimalist your content is, and if you're packing it full of presentation for nothing -- which again without seeing the site in question means we can't tell you anything useful in that department.
Though responsive CAN help you reduce the bandwidth use -- while CSS is often loaded used or not, background images for example if you strip them off the layout with the media query aren't even loaded -- poof, faster page.
Responsive done right is great on modern phones; done wrong it's rubbish... and a fat bloated slow poorly written site is a fat bloated slow poorly written site REGARDLESS of if you are using media queries or not.
Because while people like us may be sitting on 20mbps or faster broadband connections, my neighbor at 768kbps or the folks 50 miles north of me for whom 33.6kbps dialup is a good day, fat bloated sites still suck on the desktop -- without even getting mobile in the equation... which is why I still try to keep them in mind.
It's one of the few places I agree with the "mobile first" crowd -- making a fat bloated slow page for desktop users is better how exactly? But again that goes into design philosophy differences -- there's a bitter pill that really stings most 'designers' need to have shoved ... "I can't swallow that!" <farnsworth>Good news everyone!</farnsworth>
... and that pill says: "People visit websites for the content!" -- NOT the goofy layout elements, stupid distracting animations and images we hang around it... cut down to the bone what people are actually visiting for, give it a wee bit of decoration, and move on.
You may want to look at some of the questions I posed.... what's your code to content ratio on the markup? that's HTML... if you deleted all the tags and just had the plaintext, and compared it to the size of the HTML output, what's the ratio? Anything in excess of 3:1 code to content is usually fat bloated poorly written rubbish; which means slow to load and slow to render... how many DOM elements are there? More DOM elements the longer it takes to render (part of why I think HTML 5 is rubbish, encouraging the use of all these pointless extra structural tags)... How much CSS do you have total? Honestly if you have more than 48k for desktop/screen (one media target) for AN ENTIRE SITE, your CSS is fat bloated trash.... Did you take the sleazy shortcuts of 'frameworks' like YUI, Mootools, blueprint, jQuery? Welcome to extra pointless slow overhead.
Which is why responsive layout is a non-issue when it comes to site speed.
Honestly, I'll take responsive over custom for each any day, but you have to plan that way from the start, or start out with a minimalist website from the start. If you have good clear content, logically divided into separate pages, don't play any of the goofy content/ajax tricks to replicate what we've been told for a decade NOT to do with framesets, the so called "responsive" layouts just make sense -- even more so if you've just been practicing fluid or semi-fluid layouts all along; you should already HAVE a layout that adapts to screen width, this is just a simple extra step.
Though if you've been sleazing out crappy fixed width layouts with crappy fixed metric fonts using outdated presentational markup techniques, not only will you fit in well with the HTML 5 crowd, "responsive layout" may as well be an alien language.