There are still the same number of sites being shown, there are still the same number of people using Google. If your site is moved down the rankings then someone else’s site is moving up – which probably means that Google thinks their site is better than yours. You need to give Google good reasons to think that your site is better.