As for the Panda update, I don't know. I will tell you that I ranked #1 for the most competitive keyword in my niche for a site I'm part of back in 2008 before Google killed the ranking. The site ranked #1 for maybe two weeks or so, I don't recall. I was checking the ranking on different Google data centers when I saw the site fall from #1 to well below 600, far beneath hundreds of irrelevant crap sites. The site never recovered. In fact, I get less traffic from Google now than I did one year ago. I get about 25 visitors a day from Google now and between 300 to 400 or more per day from Yahoo/Bing, depending on the day. I don't know where the site ranks now for the most competitive keywords, but it isn't top 100. In fact, the site dropped for every possible search terms imaginable. Google killed it, sent me an email suggesting I buy some Adwords advertising, and never allowed it to recover.
So it is quite possible you may never recover. All I have to say is "welcome to the club". Thousands of webmasters have experienced the exact same thing. So you are not alone.
My Yahoo/Bing visitor counts vary day by day. But the count of the visitors from Google holds pretty constant to around 25 visitors. I sometimes wonder if Google is deliberately "throttling" my site in the search results to prevent it from getting any traffic. Google Webmaster Tools shows hundreds of page impressions for certain keywords but a click-through rate of only a few percent. I don't know how accurate that is.