Sorry, but that’s just not true.
Google will only penalise your site for “unnatural” links it perceives to be part of a link scheme. Odd links from unrelated sites will do you no harm. (They may not do you much good, either, but they won’t incur any penalty.)