As with anything to do with SEO, it isn't as simple as that.
At one time, Google would give you credit for pretty much any link you were able to get, as long as it wasn't on a link farm or in any way cloaked or hidden. A huge network of "I'll link to you if you link to me" type of links sprang up, and Google has been whittling away at these ever since.
It's wrong to say that linking to a site that also links to you will harm you. If the links look "natural" then Google will treat them as the typical 'vote of confidence' that it wants links to be. But if they look too much like "I'll link to you if you link to me" then they are unlikely to be counted, and if you have too many of those and it really skews your link profile, it is entirely possible that Google will decide that you are breaking its webmaster guidelines and downgrade your site as a result.
One thing to remember is that often when people talk about a site being "penalised", they don't actually mean that ... what is happening is that a site previously had plus points for something dodgy (such as exchanged links), and so when those plus points are removed, the site can drop down the rankings. That isn't because it has been actively penalised, it's just that the site has taken up a new, lower position in line with the revised ranking criteria.