Only browsers that understand that rule will be hit by performance issues which means that IE9 and under are only affected (note that css gradients can also slow pages down a little when scrolling anyway). Therefore there is no benefit to removing them to an IE only file apart from cleaning up the css.
Note that IE filters can cause lots of other strange behaviours (border-radius won’t work in IE9 when some filters are used) so I tend to avoid using them at all if possible and just let IE9 and under fallback to a solid colour (or an image).
I’ve recently come across this very issue and ended up using .svg image gradients that work both in modern browsers and ie9. Additionally the CSS markup I ended up with (base64) was smaller in bytes than the CSS-gradient rules that included a copy to address each specific vendor. The advantage with a .svg over a bitmap image is that is scales along with the content and you keep the freedom for radial/angular gradients without the need for huge bitmaps. When you absolutely need to support <ie9 just use a png fallback for those browsers, otherwise degrade to a solid colour.
Yes those gradient rules can be very large when you include all the versions. SVG is a good alternative for IE9 and there’s an svg gradient generator here which might be useful.
As far as I know linear gradients don’t suffer from quality issues on retina devices and there doesn’t seem to be any complaints about it around the web that I can find.