Bootstrap isn't really that heavy. Minified CSS for v3 is 22kb after gzip and the js is 11kb for a total of 33kb. For comparison, the jQuery dependency is 34kb. So if you want the full suite, you're looking at 67kb, which is less than the size of a small jpeg.
With Bootstrap you also get tons of themes that can make your site look pretty good and get away from the Bootstrap feel, with very little effort spent. Bootswatch is good resource, but there are a lot of others.
In my experience, frameworks that are lightweight end up causing me far more issues due to the lack of community support, complication from trying to be lightweight, and just not being as well put together at Bootstrap.
I've been using v4 for the last few months and love it.
degrading SEO results
It's not going to degrade SEO results. Anything you read that says that is probably filled with all kind of other voodoo hocus pocus SEO garbage "tips". Google and Bing are pretty smart. I promise, that using the number 1 repository on Github is not going to hurt your SEO.
If it does anything, it will force a more unstandable standardized semantic structure onto your markup, that is mobile friendly, and help your SEO.