No, you don't need to worry about increasing page load time with the use of @media queries, generally speaking.
One thing you do need to consider is making the site mobile friendly, which is a big part of the responsive design piece.
One of the most important things in making it mobile friendly is making it lean with a small download footprint - but that should always be the aim for any site.
I'm asking this because I don't know how to measure page load time and would like your opinion too.
There are good tools for this, I often look at the Net panel in Firebug which shows how long all of the HTTP requests are taking.
YSlow is another great tool.
The YUI guides are the best resource on web performance, study those.