I think you may be right, and that is, in fact, what I had in mind. There's also the problem of having my own pages competing with each other.
My website is about 'hand drum repair'. When I first launched my site, each of my pages targeted that particular phrase (among others). I now have my Home page dedicated to that keyword, another page to 'hand drums', yet other pages targeting specific hand drums, and so on.
My problem is that I'm addressing a very finite niche, so maximizing my traffic is crucial because, at best, traffic will be very limited. When I see the number of impressions drop by 50% I see a big red flag.