Open source enthusiasts were reeling this week when Sun Microsystems announced that it will be purchasing MySQL AB, the company behind the development of the popular MySQL database software, for US $1 billion.

The spin from both camps was positive, but what does the deal mean for web developers? And what impact is it likely to have on other open source projects?

To attempt to answer these questions, we need to look at a number of factors, including Sun’s history with open source technologies, its competitors, and the success of other open source technologies that have been commercialized.

While it’s not without its detractors, there’s no denying that Sun has a pretty solid history when it comes to involvement in the open source world. The company claims to be the biggest contributor to the open source community, a claim that, based upon its release of the Solaris source code in 2005, and the Java programming language in 2006, is difficult to refute.

But there is other evidence of Sun understanding the value of keeping software open source and free: projects like OpenOffice.org and NetBeans have healthy communities and promising road maps. In fact, if you examine Sun’s current offerings to the corporate world, the only key component missing from the portfolio (and one that competitors such as Oracle, IBM, and Microsoft already offer) is a database. MySQL, with over 10 million installations worldwide (including high-traffic sites such as Google and Facebook) is a natural fit.

One concern that web developers may have is that the licensing terms will change — what if Sun decides to start charging a licensing fee for each of the installations of MySQL?

In my opinion, this just won’t happen, simply because Sun realizes that it hasn’t bought a database — it’s bought a community. A community of employees, developers, and clients.

Related to this is the question of whether development of the cross-platform releases of MySQL will stagnate, with Sun’s energy being focused on compatibility with the Solaris platform. Here’s what MySQL’s VP of Community Relations, Kaj Arnö, had to say on the matter:

“I don’t expect (the development of MySQL on Solaris) in any way to be at the cost of other popular operating systems (Linux, Windows, Mac OS/X, other Unixes etc.) or development environments (PHP, Ruby on Rails, Perl, Python, ODBC, C++, C#, VB etc.). MySQL grew with LAMP, and MySQL without LAMP at its core is simply unimaginable. It was MySQL’s part of LAMP that interested Sun in the first place. Hence I don’t see Sun having a platform migration strategy, but to continue to be an integral part of the dot in .com.”

In short, Sun understands the business model around open source software, and it wouldn’t have invested $1 billion into a company only to screw over its users for a short-term profit when the long-term prospects — driving MySQL into the corporate arena with paid support — look so bright. There are plenty of other open source databases (PostgreSQL, for example) with comparable features and performance that customers would quickly migrate to if they were unhappy.

And as for the figure paid by Sun? While some would suggest the number was low given the size of the MySQL user base, $1 billion is a lot of money in anyone’s eyes. Let’s take a look at some other recent open source acquisitions [source]:

• US$210 million for SUSE Linux (Novell)
• US$400 million for JBoss (Red Hat)
• US$500 million for XenSource (Citrix)
• US$350 million for Zimbra (Yahoo!)

I’d say the MySQL team would be feeling pretty happy with themselves right now. Who said there was no money in open source software?

In this talk, Adam presented a number of methods for effective advocacy and for getting along with other developers. In other words, interacting with people is as important as interacting with hardware or software. Problems in this sphere have no technical solution — they require a social one. Thus, the idea of ‘people hacking.’

People hacking is not rooted in nefarious black-hat hacking, but simply in using tools of social jujutsu (sometimes on yourself, even) to gain traction for ideas you’re trying to advocate. Paying more attention to the people side of things can help you build the kind of integrated, smoothly functioning team capable of executing (to borrow a basketball metaphor) the “no-look pass.”

A few sample ideas included:

• Simply smiling — makes yourself and people around you feel more positive
• Avoid negativity — try the ABBA Method (see the slides for details)
• Avoid criticism
• Compliment before criticizing

Some ideas followed about how to deal with jerks in your organization, including the simple-but-effective “No Asshole Rule” (from the book of the same name) that encourages a zero-tolerance toward asshole behavior in your group — even from so-called superstar programmers.

Adam also encouraged a little self-evaluation to determine if your yourself may be a recovering jerk, and that by focusing on improving your ability to working well with those around you, you can also reform yourself.

Adam has his slides available online here.

Andy’s talk gave practical advice for catching up on all the tasks you put off until some later date (that inevitably never comes) — stuff like postponed docs, fixing broken tests (or just writing tests at all), backup regimes, TODOs in your code, etc.

He had a very simple and straightforward formula:

• Identify your debts
• Determine the costs
• Pay the most profitable
• Stop incurring new debt
• Repeat as necessary

The most important takeaway for me was the idea that you don’t try to fix all your problems at once. You figure out what fix will give you the most bang for your buck — not the easiest thing to fix, not the biggest problem. You fix whatever it’s the most profitable to fix.

There was also a lot of good discussion during this talk on how to convince a manager to allow you to begin reducing technical debt with things like refactoring, increasing test coverage, adding documentation, or actually testing your disaster-recovery plan. The best advice was the idea of putting it in terms that the manager can understand (often with a manager either in money or time).

For example, if managers know that investing two days testing disaster recovery would save weeks of rebuilding critical infrastructure, or that spending a week refactoring crufty old code would make it 20% faster to implement new features, it’s easier for them to justify the cost.

Andy has his slides available online here.

The days of one guy putting together all the hardware and software together by himself in his garage are long gone — working with computers increasingly means doing something very specialized in one of the layers between the hardware and the end-user.

Adam made a good case for the idea that despite this specialization people working with computers should develop at least a passing familiarity with the layers other than their own — e.g., a guy working in the database learning typography, and a Web designer learning about compilers.

Many developers may not have the interest (or the discipline) to spend time learning about subjects so far removed from their little piece of the pie, but a couple of the benefits to doing so that Adam described seemed pretty compelling:

• The ability to converse with the people you work with and understand better how your work interfaces with theirs.
• Gaining a new perspective that allows you to find unexpected or creative solutions to problems in your own domain.

Adam spent the end of his talk going over the two example layers just mentioned — CSS/layout/typography, and compilers. That might not have been something every developer was super-interested in learning about, but for a guy like myself who’s kind of all over the place from the database layer all the way up to end-user-facing design, it was really interesting stuff.

Adam has his slides available online here.

Photo Archiving: The Golden Rule

Store metadata in the image.

Actually should be more precise about the word metadata here - there are essentially three significant types of metadata when in comes to digital images, when it comes to who created it and where it’s stored.

First; the stuff your digital camera attaches to the image when you take a picture (e.g. camera make, time the picture was taken, exposure etc.), second; meta data you manually add to an image, typically at the time you download images from the camera to your PC (e.g. where the picture was taken, a description, keywords, name of the photographer etc.) and third; “grouping metadata” - how a collection of images relate to each other (e.g. they were all part of a single photo shoot or they are all family pictures). For the purposes of this discussion, the first two types of metadata can be regarded as a the same thing and it’s this information that you want stored in the image.

The third type - the “grouping” information is “relative” and, practically, can only work well if it’s stored externally from images. This may be as “lightweight” as a tree of directories on your filesystem with timestamp-based names, under which you store your images, but image archive tools may offer additional facilities in this area. Important is if you want to group a collection of images under a heading like “family”, you also want to have a keyword “family” stored in the image. In such case it may be the image archiving tool can build the groups automatically from your keywords.

But you want the first two types of metadata in the image, so that if you switch archiving tools, transfer your images to another system or forget to backup the archive database but remember the images, it won’t be lost. A decent archiving tool will probably generate it’s own database from this inline metadata, to make searches etc. efficient, but the final authority should be the image itself.

Inline Image Metadata Standards

The next question is how is data stored in an image. File formats like JPEG, RAW and TIFF, commonly used in digital photography / processing, allow additional metadata to be stored in the image file.

So here’s a “avoid too much detail” view: there are three significant inline metadata formats here - Exif, IPTC and XMP. Exif is used by most digital cameras to attach information like exposure at the point you take a picture and tools are available for adding information manually (or automatically) such as geocoding. IPTC “extends” Exif, providing a whole load more named “tags” for metadata. Both are fragile from a technical point of view (easy to destroy / corrupt) and problematic from the point of view of extending (if there’s no named “tag” for the metadata you want to store, you’re out of luck). Meanwhile, XMP, the work of Adobe, is the “holy grail”, providing XML / RDF / Dublin core goodness for storing metadata. It’s just not well supported yet, but starting to gain significant acceptance. And of course our friend IE has it’s own opinion about XMP…

In practice, you need to be able to work with all three formats for effective photo archiving.

Image Archiving Apps

There are a bunch of image archiving tools out there, for any and all operating systems. A good one will allow you to manually supply additional meta data about an image (e.g. keywords like “family”, “dog” etc.) but, from a straw poll, the majority store this data separately from the image, in some kind of database. Where Ubuntu is concerned GThumb does just this - stores any user supplied meta data externally from the image (if I remember right, it’s under ~/.gnome2/gthumb).

There’s a bunch of reasons why I consider this fundamentally wrong, which become obvious the moment an image leaves your filesystem (or perhaps even changes directory / filename) plus the fact that there’s no “clear winner” in the image archiving space - there’s a good chance you’ll switch tool more than once. Bottom line: storing metadata separately from the image it refers to is just not the way to do it.

In the Open Source / Linux space, there are a couple of contenders which are getting it “right”, namely F-spot (C# / Mono) and jbrout (python / wxpython). Both are capable of storing metadata in the image but there are limitations, e.g. jbrout has no awareness of XMP. Some worthwhile blogs (and comments) to read are How to manage EXIF, XMP metadata on linux gnome kde, Managing Photographs and Photo organizing in Linux.

Image Archiving Done Right

By contrast, the model application, IMO, for how to do it right (sadly Win32 only) but free to use is PixVue.

For starters, rather than being an monolithic application, it adds a bunch of Windows Shell Namespace Extensions meaning you can work with your images in Windows Explorer - e.g. right click and “annotate” with your metadata.

As I see it “Digital workflow”, in general, is all about procedures - you need “process oriented” applications that fit into your procedure, rather than monolithic windowed views of your images, which dictate how you work. PixVue gets this right by getting out of the way and hence my desire to extend Nautilus.

PixVue also stores almost all metadata in the image, understanding EXIF, IPTC and XMP. The exception is the “grouping” metadata, which it provides something called “galleries”, which are a virtual directory hierarchy - in Explorer you can drag and drop images into a gallery (or galleries) to organise them but these are only “symbolic links” to the original files on your filesystem. You can also export your gallery structures as XML, to allow at least the chance of portability.

Also the “templates” functionality it provides helps make attaching metadata to a large collection of images as fast and painless as possible.

Finally it has excellent search facilities (via the gallery view) plus it “integrates” with (populates the indexes of) Windows search and Google desktop.

The only downside of PixVue is the source isn’t available and there doesn’t seem to be an API (it does register some COM components but, from a quick scan, these don’t allow you to drive it with, say, a Python script), so if you want to extend it to integrate with some geocoding tool, you’re out of luck.

Open Source Image Metadata Tools

So, given a less-than-ideal situation for Open Source image archiving applications, what’s available in the form of libraries, so you can hack your own?

The long and short of it is more bad news. There’s a whole bunch of libraries that do a little bit, perhaps only (partially) understanding one format (typically Exif), read only - not write or lacking support for manufacturer Exif extensions.

The situation is particularly acute when it comes to Python, which is badly lacking - there’s a few “readers” out there, and the occasional writer (no XMP writer though) but nothing that’s really complete. That’s bad news for jbrout and even more bad news for Python on Series 60 phones, where being able to tag your pictures, while you sit on a train, would be a killr app.

There are exceptions though, the #1 being Phil Harvey’s exiftool (Perl - available via CPAN). ExifTool isn’t just good - it’s excellent. It’s basically got it all - EXIF, IPTC and XMP support (read/write), support for manufacturer extensions and a common dictionary of tag names that are, to an extent, metadata-format-independent. The only downside is there doesn’t seem to be (correct me if wrong) a GUI front-end for Linux that exposes all of Exiftool’s functionality - you’re either talking command line or writing your own Perl scripts.

Another noteworthy exception Evan Hunter’s PHP JPEG Metadata Toolkit, which has EXIF, IPTC and XMP support and awareness of some manufacturer quirks. The API is also pretty simple.

The is of course plenty of other stuff around, which offer part of the picture e.g. jempbox (Java) but YMMV.

Anyway - hopefully that saves someone some time in research.

Elein showed how to use timestamps to keep an audit trail of all changes in your DB. With this technique, you never actually delete records, you just give them an end date. Multiple copies of a record (again, with timestamps for each one) also allow you to track edits. (This is a pretty standard approach, especially in ‘validated’ environments, or in government-regulated industries.)

The value of using Postgres here is in the advanced features like triggers that you can use with your deletes and updates to offload the work needed to maintain this ‘time travel’ system — rather than forcing your app logic to keep up with all of it.

You can also use PostgreSQL’s views to query the database for only the current data, or write procedural functions to query the state of the database at a particular point in time in the past.

Elein has her slides available online here.

This talk was on converting a really large (over 3 terabytes, largest table is 1.8 billion rows) data warehouse database from Oracle 8i to PostgreSQL. The reason for the conversion was to save in licensing costs. They wanted to move their Oracle licenses over from the data warehouse DB to use them on the online transaction processing (OLTP) system.

The reason for choosing Postgres over MySQL was that Postgres has a much longer history with the kind of advanced features he needed. They needed (and were able to hack PostgreSQL to get) the following features:

1. Data partitioning (spreading tables over multiple drives)
2. Large selects (50-million-row return sets, over 100GB of data)
2. Incremental COMMITs for really, really long queries
3. Replication

The bottom line was that with some patience and creating thinking, they were able to migrate a really humongous database to PostgreSQL from Oracle, and ended up saving themselves $500,000 USD in licensing costs. A pretty good chunk of change.

Theo has put his slides up here.

Quite by coincidence, three times in the past week I have had to hack the code of some open source software that went into the site I was working on. First I had to modify phpBB to include an embedded calendar on the home page of a private forum I administer. Next I made some custom tweaks to the code of the K2 theme for Wordpress. Finally, I had to hack phpAdsNew to produce XHTML Strict output.

In each case, the hack required me to actually modify the code of the software. Obviously I prefer not to do this, because when the next release comes along the updated files will overwrite my hacks, and I’ll need to implement them all over again.

Normally I’d just document my hacks someplace and grumble about the lack of customization features in the software, but three times in a week was too much. Let me show you how I solved the problem using a common development tool in an unconventional way!

CVS (Concurrent Versions System) is a system for tracking changes made to files in a project over time, potentially by multiple developers, each working on his or her own copy of the project files at the same time. As it turns out, it’s extremely useful for managing the custom hacks you make to open source software.

In the past few years, Subversion has sprung up as an alternative to CVS that eliminates some of the headaches in CVS to do with things like moving or renaming files. Since such changes don’t usually happen when you’re hacking an existing script, and since SitePoint already has a decent introduction to CVS, I’ll stick with CVS for this discussion. If you know Subversion, you can use it instead.

Setting Up

You first need to create a CVS repository for yourself (if you don’t already have one). Because I use Windows, I set up CVSNT to do this. If you’re on Linux, you can use the original CVS software. You’ll also want to get an easy-to-use client program (I recommend SmartCVS), unless you particularly like working from the command prompt, in which case I’ve included all the commands below.

When your CVS server is set up, store a “clean” copy of the software version that you have hacked for use on your site as a new module (or project) in the repository (e.g. cvs import phpBB2 phpBB2 init_ver). Tag this “clean” version in the repository to indicate the version number of the software it represents (e.g. cvs tag release-2-0-17 .).

Immediately create a branch in the repository (e.g. cvs tag -b custom-mods-branch .) from this initial version, and then check out a copy of the files from the branch into a convenient working directory (e.g. cvs checkout -r custom-mods-branch phpBB2). This copy is where you’ll keep track of your hacks.

Copy your site’s (hacked) copy of the software’s files on top of the “clean” copy you just checked out from the branch, and then perform a CVS update to identify the files that have been modified with hacks (e.g. cvs update .). Review these changes to make sure they are all wanted.

Fig 1. See hacks as changes in the working copy

You can now track changes to your hacks as you make them in this branch. Simply hack the files in this working copy of the branch to your liking and commit your changes to the repository. To update your site with these hacks, delete everything in the destination directory and then export the latest version of your branch files to your site (e.g. cvs export -D /home/www/htdocs phpBB2).

Merging with New Releases

When a new version of the software comes out, extract its files into the “clean” copy you made at the beginning, commit all the changes to CVS, and then tag the updated files for the release (e.g. cvs tag release-2-0-19 .). These updates will be stored into the trunk of your repository, so they won’t affect your hacked version (which is tracked in the branch).

Now, here’s the payoff: to update your hacked version of the software with the changes in the latest official release, just go to your working copy of the branch and merge in all the changes from the trunk (e.g.
cvs update -j release-2-0-17 -j release-2-0-19). Your hacked files should be updated seamlessly with changes that were made in the official update(s).

Files where your hacks occurred close to or on the same line as a change in an official release will report conflicts when you perform the merge. You’ll have to open these files and resolve the conflicts yourself (CVS will helpfully include both versions of the code at the point of conflict) before committing the corrected versions to the branch. You should then set a tag on the branch to indicate where you did the merge (e.g. cvs tag merge-2-0-19 .).

In effect, CVS will perform all the updates that it can for you, and then will pick out those updates that appear to interfere with your hacks so that you can deal with them. If that isn’t useful, I don’t know what is!

Fig 2. The CVS tree showing merged releases.

The next time an official update comes out, you can perform another merge, but remember to update the starting tag for the merge so you only get the changes since the last time you did a merge (e.g. cvs update -j release-2-0-19 -j release-2-0-21).

A Note on the Vendor Branch

The CVS gurus in the audience may be up in arms at this point. In fact, CVS automatically creates a vendor branch for every module you check into it. You can import each new version of the software into this special branch, and then merge the changes from the branch into your (hacked) trunk.

So why didn’t I use this? The truth is, it’s just as easy to do all this using a normal branch as I described above, and it doesn’t require you to learn all about vendor branches and their pitfalls. Also, you can do it in simpler CVS clients that don’t support vendor branches (like the free version of SmartCVS).

It’s probably unfair to compare it to the Web Essentials conference that I attended earlier this year, because OSDC is - in the organizers’ words - “created by developers, for developers” whereas Web Essentials has less of a grass-roots base.

But I’m going to make the comparison anyway, as I think it does provide a useful measuring stick. As far as I could tell, OSDC differed from Web Essentials in five aspects:

1. speakers were not paid. This meant that the presentations varied wildly in terms of quality and relevance. There were a number of excellent presentations that appealed to the entire audience, but there were also quite a few talks that were amateurish and about completely obscure topics that would only interest a handful of people.
2. presentations were much shorter. Apart from a 60-minute keynote presentation at the beginning and end of each day, all talks were only half an hour long. The more experienced speakers made good use of this limited time, but for some speakers it resulted in their 73 slides being whizzed through far too quickly for anyone to digest.
3. the variety of topics and industry representation was mind-boggling. The broad umbrella of ‘open source’ was the only common thread linking all presentations, and this resulted in topics ranging from doctorate research papers to entrepeneurial advice to snapshots of large-scale government projects to corporate sales pitches.
4. three streams, instead of two. Unfortunately this meant that occasionally there were three topics that interested me and only one that I could attend.
5. less innovation, more unashamed geekiness. WE05 was exciting to be a part of because of all the new technologies and new ways of thinking that were presented. Microformats, AJAX, user-centred design, innovative ways to make web pages accessible - all these topics were introduced by the people who either invented them or were the most proficient in the world at applying them. We were hearing the latest and the greatest from the forward thinkers responsible for advancing the medium of choice for everyone in the room - the web. OSDC presenters on the other hand, were passionate about their area of expertise, but there wasn’t a common medium for everyone to get excited about other than “being a geek” and “free software”.

That said, it was a worthwhile three days with plenty of topics relevant for web professionals. Here were some of the highlights for me.

• Jon Oxer’s opening presentation about how easy it is to connect random bits of hardware to your PC and talk to them using C, PHP or whatever language you fancy. I haven’t decided whether the coolest part of his talk was his demonstration of how his letter box sends him email (and a text message to his phone) when the postman delivers his mail, or the fact that his cat has been injected with a microchip that opens his cat flap.
• Paul Fenwick’s entertaining tips for giving a good presentation, which included using a Perl module to echo pre-prepared characters to the screen regardless of what keys you press, so that you don’t make any mistakes when giving a live demo.
• Scott Penrose’s talk about the ambitious project he calls Zaltana, where he has added HTML hooks to the templates of common web applications such as SquirrelMail, MediaWiki and other custom objects and methods in order to provide a common look and feel across the whole framework for his clients that is reusable.
• Gus Gollings’ discussion of the Semantic Web. This was a different (less technical) take on the concept than I had heard in the past. Gus provided a rich historical context and posed some thought-provoking questions about what might be one day achievable, and anything that gets me thinking about the future always captures my interest.
• SitePoint’s own Kevin Yank’s talk about Mozilla XForms (watch Kev’s blog for more on this soon).
• Luce Chandon’s description of how open source software and the concept of “agents” was used at a Ford manufacturing plant to ensure motor vehicle parts were delivered to the assembly line in a just-in-time fashion.
• The “Lightning Talk” sessions (nothing to do with the miserable weather outside) where interested attendees gave a 5 minute presentation about whatever they liked. The preparation that went into some of these snappy talks was incredible and it was a good, encouraging environment for those new to public speaking to give it a go.

In summary, OSDC is a professionally-run conference that represents good value for money if you are a web professional, but even more so if you are an enthusiastic Perl/PHP programmer who wears a red hat everywhere, has named his first son Linus and has a Slashdot ID of three digits or less.

The networking aspect was also good and I met some terrific folks, many of whom are involved in the various user groups active around Melbourne and Australia, and some who travelled all the way from Japan, Taiwan and elsewhere for the conference.

However, it was a conference that required you to pick the sessions you attended very carefully, as both the speaker’s presentation skills and the relevance of the content varied (sometimes significantly from the talk’s title).

Credit to the organizers for running things so smoothly. I’ll be back next year!

Many of you no doubt have tried out the Beta and Release Candidate builds. Here’s a list of things to do and see with Firefox 1.5.

• Use the back and forward buttons to see the dramatic increase in rendering speed resulting from Firefox’s new “fastback” cache.
• If you installed one of the Beta or Release Candidate versions, you could wait until Firefox updates itself using its new incremental update system.
• When tabbed browsing, grab the tabs with your mouse and drag them to the order you want.
• Press Ctrl+Shift+Delete to bring up the “Clear Private Data” window to clear private data such as history, cache and cookies.
• Go to websitedoesnt-exist.com to check out the new, non-obtrusive error messages.
• Check out “Help > Report Broken Website” to see how easy it is to report a Website that doesn’t work in Firefox.
• Install Firefox on your Mac and see how easy it is to migrate your profile information from Safari.

If you’re still itching to get involved with Firefox, you could consider the following:

• Get your Webcam ready and head on down to SpreadFirefox to take part in some sort of celebration.
• Catch up with Firefox related news at MozillaZine, Asa Dotzler’s blog and Mozilla.org.
• Get involved with Firefox 2.0 development by signing up for a MozillaZine and/or Bugzilla account and testing post-1.5 builds.
• Download the free chapters from SitePoint’s book Firefox Secrets.
• Start making your Websites Firefox-friendly!

Firefox 1.5 is available now for your language and operating system.