What happens is that when you're collapsing the extra links, the height of the document changes, and it looks like this is causing Firefox to repaint the screen (several times in quick succession, appearing as a flicker).
To test this, head in to Firebug and hide/remove all content until you effectively only have your "siteLinksWrapper group" div on the page, if you hit the more/less link without a scrollbar present, the page will not flicker.
I did just give this a quick try with the jQuery UI Slide Effect and the flickering is a lot less noticeable. (It's also a bit less noticeable when you change the speed of the effect to, say 100ms).
There are of course other drawbacks to using the jQuery UI effect, least of which is loading a whole extra library just for 1 effect.
Note that if you use toggle without a duration there would only be 1 repaint of the page.
Has anyone else come across this problem? More importantly, know any other solutions to this?