Over [SEO] optimization is exactly as it sounds -- when you focus so much on optimizing for SEO that you're attempting to game the system rather than develop something useful that's reasonably optimized. Engines know you need to do certain things to appear -- like make accessible links, name your pages well, use header tags -- they want and need this. What they don't want is sites that never link out or flag every link as nofollow -- links are useful. They don't want sites that harvest, trade and build up links just for seo "juice". They don't want pages written to keywords and length only.
When you reach a point where everything is about an SEO trick you stop being useful and that's when you get flagged.