The state of play on your production server

[ServerStorm]

I liked Mint-Debian (Debian with a bunch of non-freedom-fries junk on top out of the box like Firefox...) but for some reason I've not been able to install it on my work Dell or Thinkpad (no idea why, these are generally Linux-friendly machines and they both run Ubuntu fine... both had trouble with graphics cards which surprised me). The windowing system was xfce which is buggy (there are buttons who don't work and config options without options or buttons!) but fine for development, and I could choose Debian Testing.

Yes, in the past I've had to install the proprietary drivers for the chip-set on Dells via the command line and manually create (on modern Mint distros) configure the xorg.conf. After this Mint worked well, but issues when upgrading the Kernel so not too worth it (unless you have no reason to change your kernel and can handle the wrath of less secure over time )

The original Unicorn server was kinda meant for small quick clients, while we're running a fat e-commerce thing on it. gUnicorn is different though. SphinxSearch is a smaller cousin to ElasticSearch, which might be the best full-text search out there, and apparently both of those are without the hair-pulling of Apache SOLR/Lucene (or so I've heard). REDIS is just one of the many trendy no-SQL thingies out there; we could have used MongoDB or something instead (and it seems Python folks are doing that a lot, like Django + some combination of PostgreSQL and Mongo).
I get the impression that Python isn't getting more popular in the States, but over here I'm noticing more and more webshops using it (especially new ones though, without any legacy junk to worry about).

Thanks for the background

[ServerStorm]

On proxies vs firewalls -- in all honesty it wasn't necessarily planned this way, it grew up this way but it worked out so well that we kept it.

We started using HTTP reverse proxies for some co-location efforts -- like needing a wordpress wing of a .NET site. We expanded the proxies a bit for management flexibility -- a dumb proxy need just be repointed to deploy the app to a new server; no DNS downtime scenarios to work through. A bit later we had some live events (Oprah!) and we needed to be ready. Turned out the proxy did a great job of handling this. Finally we got DDOSed and that is where it really shone; we got our hands on the attack script and used the url rewriting capabilites to black hole the attack effectively. After said attack we went shopping for http application firewalls and we pretty rapidly reached the conclusion that 95% of the vendors were running Apache with mod_security and some management / reporting tools. It also clicked for us that having a very HTTP-aware server acting as a firewall made total sense. Being stateful enough to really understand what is going on opened all kinds of options to solve a number of different problems.

There are still some headaches -- IIS is not horribly reverse proxy friendly in some ways. Many apps do something really dumb like think their own address is the actual public address. Windows integrated auth -- a pretty handy quick and dirty authentication method for apps not really needing real user management -- does not work through proxies. But overall it has been an astounding success.

Tech-wise we've been using IIS principally, but we'll probably start shifting to NGINX as it isn't worth the licensing and we are comfortable enough to do that.



I'd say NoSql deserves it's own thread -- lots of different styles out there. Lucene, which was NoSql before the marketing types thought up the term, is a complex beast but astounding once you get your head around it. I haven't had much seat time with SOLR but in this day of VMs it is pretty easy to stand up a turnkey appliance.

Lucene, MongoDb as well as things like CouchDb (much better than mongo) or RavenDb (also much better than mongo) are document databases. Really great for lots of modern web scenarios, especially if you have gone object / document oriented farther up the stack. REDIS is slick but it is a very different beast. Really closer to memcachd than document DBs I've mentioned.

I think python is getting alot more play in the states these days. PHP has basically descended into Wordpress / CodeIgniter / ExpressionEngine / Drupal configuration and plugins. As well as some cases of advanced SSI. Anyone building applications has moved past it. The folks in the *nixy end of the world who like dynamic languages have flocked to RoR and Python. The rise of frameworks like Flask also helps -- thin servers are making more and more sense. Or at least I'm learning me some pyhton.

Quality information here! Thank you

[Stomme poes]

I'd say NoSql deserves it's own thread -- lots of different styles out there. Lucene, which was NoSql before the marketing types thought up the term, is a complex beast but astounding once you get your head around it. I haven't had much seat time with SOLR but in this day of VMs it is pretty easy to stand up a turnkey appliance.

As I understand it, SOLR is a sort of library +(lots of) extras on top of Lucene, which some people hated trying to set up. And then that still wasn't good enough for the ElasticSearch guy :D. I kinda have a soft spot for ES because a Perl guy (and my husband's old boss) wrote the Perl modules for ES and gives lots of great talks on ES. The easy sharding and real-time search is popular.

We use Sphinx because it's fast, works well with SQL but also with an index, and lets us serve many languages, deal with word-stem changes with intelligence, and keep a record of search terms that don't get any hits (a sort of error log and great to have).

The problem with NoSQL is that it got "hip", which means it has to be the solution to every freaking problem and relational DBs are wrong and old. Arg. It's a solution to a type of problem, which might be why I like the idea of combining the two for larger projects.

REDIS is slick but it is a very different beast. Really closer to memcachd than document DBs I've mentioned.

That's pretty much how we're using it: a glorified B-tree on top of a large and bit slower DB, except we can store lists, dictionaries and integers in it too (and we don't care if it doesn't write everything or loses stuff; that's what the base DB is for). But the Python/Mongo combo seems popular, as is an ORM like SQAlchemy on top of some *sql server (maybe because Django, the first big *popular* Python framework, has its own special internal ORM so you never have to touch an SQL query while have a "normal" DB?), or some similar ORM. A sort of replacement for Zope's ZODB, the original big Python framework (which we used at my former workplace), who not only stores things as objects and uses object inheritance, but also uses "contextual inheritance" which I really had trouble figuring out how to use (prototype chains? I learned those in JavaScript. But add in context chains too and I'm lost). My Python n00bness didn't help : ) Zope finds things in mysterious ways, but it's really great at finding things.

Or at least I'm learning me some python.

For great power? : D

I wonder if the mini-frameworks are popular in part because while developers want to Get Stuff Done, they also want to write code rather than only plugging tools together and churning out big stuff with lots of magic. Which you do get with the WordPresses and stuff (I mean, that's the whole point, isn't it? Get something up and running that's been around and well-tested and all that).


Also, I didn't mention, but our gUnicorn runs on top of Nginx.