Yea, I actually went that route and whipped up a basic CRM in an afternoon using Organic Groups and simple node relations (Basically Creating a relation between Client and User or Company, along with Relation between Invoice and Client). It really wasn't that tough ... but working with Drupal does kind of require you to get your hands dirty, initially, if you're not already familiar with the idea of CCK and Views.
Another option might be to take something like VTiger, and simply get rid of the options you don't need. Which isn't incredibly hard, but you do have to do a lot of searching around to find the right documentation -- and it would certainly be a lot quicker than learning drupal cck, views and node relations. But the end result wouldn't be quite as clean and tidy.