For people using the mobile search page (google.com/m), Google does prioritise sites that it perceives as mobile-friendly. That doesn't necessarily mean sites that have a mobile version, just those that would be better to use on a mobile device. So it would increase the significance of a fast pageload and simple navigation, as a couple of examples.
To me, that makes eminent sense. Some sites might be great to use on a PC, but absolutely suck on a mobile device. Some sites have a mobile version specifically geared to dealing with the sort of issues people are likely to have when they're on a mobile. If Google can figure out what kind of device I'm using and send me to the most appropriate page, I'm delighted!
You're forgetting something – Google isn't here to be fair to webmasters, it's here to give people searching for stuff the best page(s) it can for them, right there and then. It really doesn't care less whether this means that big sites have an advantage over two-bit Mom & Pop sites – if the big sites are giving better results for people searching then it's big sites that will appear at the top of the list.
It doesn't often make much of a difference. I've just done some test searches on both the main and mobile sites, and the top half a dozen results are almost always the same in both, it's only in the second half of the page where you occasionally see a few sites swap the order they're listed in. No, it isn't a big deal, and it almost certainly isn't worth creating a mobile version of your site just for SEO reasons.