It's difficult to understand the reason without you giving us more detail about your site. Did you check Webmaster tools? For all you know, it could be that your server has been going down because of which you are seeing less traffic. Otherwise, do you have a lot of duplicate content? Keyword stuffed articles? Did you build backlinks recently? It could be due to any of this. Just make the change, and you should see results immediately - in a span of a week or so.
If you have Google Analytics, you can get to the root of the problem. Go into the “All traffic” report, then o check the box for Google, and then click “Plot Rows”. It should show you when you start declining and if you find the particular date the rot began you can back track and find out what Google update hit that day. This will help you narrow down the problem. If it is a Panda refresh the it is spammy content, if it is Penguin refresh then it is a links issue.
There are following reasons behind this situation which may be occur:
1. Your website content isn't update
2. Your links building isn't in flow
3. Your On-Page SEO needs fresh keyword to rank
4. Your website popularity may be lose
5. Minimization of Social Signals
Have you been consistently posting content? I have seen Google penalize sites who do not update their site in favor of sites with regularly updated blogs.
Google has also been issuing more penalties for onsite web-spam techniques. If you are trying to hide text/links on the page this could be throwing up a red flag to Google. The amount of links you have on your page is also more important now than in previous years. If you have a lot of links on your page, it may have dropped during a recent update.
Obviously if your competitors earn some high profile links and you do not have a solid backlink profile, this could also be a reason.
Start off by looking into the onpage optimization and backlink profile of the webpages that are now outranking your site. Sometimes you drop because someone else has a good SEO strategy.
It could be the competition. Try to check your google analytics, from the date with 4000 hits to 400 hits. Compare top 100 keywords and see if you can find the problem there. Maybe some of your keywords do not perform well like before.