Does anyone know what forum software do they use?
and how did they build the forum home? the forum home looks very clean and very well organized. http://www.sitepoint.com/forums/
There used to be a “powered by” type of thing in the footer, but this forum is vBulletin
*Note, the SitePoint forums will be moving to Discourse
Let me add to to that: the forum uses vBulletin which has been heavily modified but soon we will moving to Discourse 
The home page is a plugin (vB plugins are written in PHP) that we wrote ourselves.
To be honest with you though, I wouldn’t recommend vBulletin if you are looking at setting up a forum of your own. There are plenty of other options around these days that are better. If you’d like some advice, let me know what your situation is and I’d be happy to give you more information.