I'm still a little confused because if it looks bad in IE8 at smaller screen widths then it will look bad in all desktop browsers also.
Maybe I'm not understanding this bit.
engage via respond.js minus the fluid grid and navbar
Why is IE8 only not getting the fluid grid and navbar?
The beauty of responsive design is that you scale the page for all browsers nicely so I don't know why you are picking on IE8 (assuming you have enabled the media queries properly). I've just created 150 odd responsive pages in bootstrap for a client and IE8 was no problem at all.
I feel I may be missing a vital bit of information
Do you have a link to the site so I can see the problem?
In answer to your question there should be no problem in over-riding the bootstrap files because that's usually how you code anyway and leave the original intact. Just set up a conditional stylesheet for IE8 and call it last in sequence. Of course you will have to over-ride any media queries and any other cascading styles you have applied.
As far as quirks mode goes then you should never code IE in quirks mode as you revert all versions of ie9 and below into ie5 mode and they forget everything they've learned this century
I feel we may need to see the site to help specifically.