Apple has published an excellent guide to installing, configuring and running X11 on OS X at the Apple Developers Connection. The post includes excellent detail on getting the environment setup to your preference and also on working with remote servers.
There are some trade-offs considering the core Terminal in OS X has very nice integration with the GUI such as drag and drop. However, when you need an environment to run X11 apps – you cannot beat having the core BSD-Unix kernel of Darwin to run X11 on and still having immediate access to OS X.
One nice feature of X11 on the Mac is Finder integration – meaning if you double-click on an X11 app in OS X it will launch in X11.