Apparently database normalization isn’t something that web developers find all that interesting. (But thanks to malikyte and xhtmlcoder for keeping the question from being a complete ghost town!)

That’s a shame, though — there are all sorts of pragmatic reasons behind good data design. To name just a few: properly designed tables often perform better than their de-normalized brethren, normalized data is much easier to aggregate successfully, and (most importantly) properly designed tables are much easier to understand.

That last one’s really the crux behind normalizing tables. Remember — computers don’t care if we write good code; when we write good code, it’s so that future developers won’t curse our names. Data normalization falls into the same future-proofing category.

Anyway, though: on to the answers. I’ll be brief, I promise.

1. The FDA’s nutritional content database is — to my utter surprise — actually 3NF (everything has a primary key, and every piece of data appears to be atomic). I have some quibbles with a couple of the design choices, but they’re actually pretty minor. It’s pretty remarkable when you come across data this clean out of the box.
2. Although the population demographic data is fairly well designed (and easy to munge into better forms), it doesn’t even achieve 1NF: records lack primary keys. This is usually the case with public data, and it stinks; it makes tracking changes from version to version extremely difficult.

3. The SEC filings were a trick question. They’re in a XML dialect, so normal forms don’t apply.

   I think it’s important to notice how different formats change the way we can produce and consume data; the SEC data is a pretty good example of well done XML, but it would be pretty difficult trying to push this data into a database in any sort of structured way.

   If I wanted to build a site around this stuff, I’d likely use something like Berkeley DB for XML instead of a relational database.

4. The gas price data, though crammed into a Excel sheet used more for presentation than data management, is actually in 3NF (if only because it’s pre-aggregated data). The data of the measurement is the primary key, and all columns are singularly dependent on the primary key (i.e. price is a function of date, and nothing more).
5. The Juvenile Arrest Rate data is, like the gas prices, nominally 3NF data crammed into Excel.

Coming up…

Tune in tomorrow for a special super-difficult (I hope) challenge to keep us all occupied over the holidays.

As always, if you’ve got a question, puzzle, or challenge that you think would make a good question for this quiz, email me at jacob -at- jacobian.org.

Anyway, let’s get right to this week’s question. A few weeks ago, I posted a scavenger hunt for public data (answers); today we’ll return to dealing with that data.

Of course, finding a viable source of data is only the first step; once you’ve figured out what to use, you have to figure out how to use it. Since I’m a certified database geek, the first thing I do once I’ve got some sweet data in hand is start thinking about database design.

When we talk database design, we’re usually talking about formal database normalization, and specifically first, second, and third normal forms. Although I’ll be the first to admit that often formal normalization needs to take a back seat to pragmatic design or performance requirements, we’ll ignore that big caveat this week and plunge ahead.

Here, again, are the five data sources we located in the scavenger hunt:

1. Nutritional content of food from the USDA.
2. (Links to) population demographics of every major city in the US, courtesy of the US Census Bureau.
3. The latest SEC filings (in RSS, no less) straight from the horse’s mouth.
4. Historical gas prices, from the Energy Information Administration (which I had never heard of until writing this quiz).
5. Juvenile arrest rates from the Office of Juvenile Justice and Delinquency Prevention (part of the Department of Justice).

So, which normal form are each of these sources in (and why)?

We’ll discuss the answers and a bit more about the implications of database normalization this weekend.

Bonus challenge!

For an extra challenge, pick one of the sources and define a fully normalized (i.e. 3NF) schema for it. There’s not in any way a “right” answer here, but if anyone’s brave enough to post their schemas, I’ll critique ‘em when we go over the answers.

Got a question of your own?

As always, if you’ve got a question, puzzle, or challenge that you think would make a good question for this quiz, email me at jacob -at- jacobian.org.

Nobody who attempted to answer got all five right. However, I may have left just a little to much ambiguity in some of the questions for the “right” answer to be findable… Either way, I’ll dive in and explain which answers I was looking for. Let me know in comments if I’m totally out of my mind :)

1. Specifications

Of the four specifications given (WSDL, APP, RDF, and WS-Policy), only RDF specifies a non-XML-based data representation. Yes, RDF can be represented in XML, but it also has alternative formats; the other three don’t.

2. HTTP methods

kasimir got it: “POST is not idempotent”.

Idempotence is actually an incredibly important concept in web development, but rather than shoot my mouth off about it, let me point you to Wikipedia’s words on the subject, which are quite good.

3. MD5 hashes

Mindaugas got it: B is an MD5 hash of the empty string.

This one was a little bit of a “gotcha” question; I usually try to avoid trick questions, but I just couldn’t resist here.

4. Programming langauges

Also a bit of a trick question: the answer is based on some knowledge of the history of the languages, not on features languages themselves.

The answer? Python is the only language not named after a (real) person.

5. HTML 4 elements

boomsb got it first: <U> is deprecated.

There’s a little bit of a joke in this one, though: even though <U> is officially deprecated, it’s nevertheless supported by every browser under the (virtual) sun, while <Q> — officially part of HTML — is not supported by Explorer.

Isn’t this a fun world to code for?

Got a question of your own?

If you’ve got a question, puzzle, or challenge that you think would make a good question for this quiz, email me at jacob -at- jacobian.org. If I use your question in a future quiz, I’€™ll even send you a nice little present…

Tomorrow

Tune in tomorrow for the next question. I think I’m going to try something a bit more open-ended this time: a question about data modeling. Be sure to check it out.

In “one of these things is not like the other” style, for each group below tell me which item doesn’t belong, and (more importantly) why:

1. Specifications:

1. WSDL
2. Atom Publishing Protocol
3. RDF
4. WS-Policy

2. HTTP methods:

1. GET
2. PUT
3. POST
4. HEAD

3. MD5 hashes:

1. f97c5d29941bfb1b2fdab0874906ab82
2. d41d8cd98f00b204e9800998ecf8427e
3. 35d6d33467aae9a2e3dccb4b6b027878
4. 0495651fa03c897470784990f33d86cd

4. Programming languages:

1. Erlang
2. Ada
3. Haskell
4. Python

5. HTML 4 elements:

1. <Q>
2. <U>
3. <I>
4. <A>

As usual, tune in this weekend for the answers.

Got a question of your own?

If you’ve got a question, puzzle, or challenge that you think would make a good question for this quiz, email me at jacob -at- jacobian.org. If I use your question in a future quiz, I’ll even send you a nice little present…

If you missed it, last week’s challenge dealt with deciphering regular expressions and finding subtle bugs within ‘em.

As with last week, before getting to the actual answers please indulge while I pontificate a bit:

Hopefully it’s pretty obvious that regular expressions are a double-edged sword. Sure, deciphering them makes a fun quiz, but imagine running across these monsters in code and trying to figure out what they do… not fun.

Fortunately, nearly every regex implementation has a “verbose” mode that allows you to embed comments inside regular expressions (n most languages this is the x flag). For the sake of those who must read your code, please use the verbose mode!

OK, on to the answers:

1. [A-PR-Y0-9]{3}-[A-PR-Y0-9]{3}-[A-PR-Y0-9]{4}

This is a US phone number, including ones that use letters (i.e. 831-555-CODE). Rewritten in verbose mode, it makes a lot more sense:

  [A-PR-Y0-9]{3}  # Area code prefix
  -
  [A-PR-Y0-9]{3}  # 3-digit exchange
  -
  [A-PR-Y0-9]{4}  # 4-digit suffix

birman had a nice roundup of the problems with this pattern:

[It] doesn’t account for a preceding 1, if the area code is in parenthesis, if the digit groups are separated by a dot or space instead of a dash, or the fact that cell phones have Q and Z on them. It also doesn’t make sure the group is isolated, and not part of something like 1234888-234-123456123.

That last point — the isolation error — is a very common error when writing regular expressions.

2. &(?!(\w+|#\d+);)

This is not, as most people thought, a mistaken attempt to match HTML entities. It’s actually a pattern that will match ampersands in HTML that are not part of entities (it’s taken from Django’s fix_ampersands template filter).

Here’s the verbose mode:

  &     # Match an ampersand...
  (?!       # ... that is *not* followed by...
    (
      \w+   # ... word characters...
      |     # ... or...
      \#\d+ # ... numeric entity symbols...
    )
    ;       # ... and a semi-colon.
  )

The “problem” with this pattern is pretty subtle: it matches HTML entities that are well-formed by still invalid (e.g. &#ggxy;). So as a way of finding unencoded ampersands it’s just fine, but if you wanted to use it as part of an HTML validator, it would be unacceptable.

3. (-?(?:0|[1-9]\d*))(\.\d+)?([eE][-+]?\d+)?

Most readers got this one; it’s a IEEE floating point number, with optional exponent. In verbose mode:

  (             # The non-fractional part of the base
    -?            # could be a leading negative sign
    (?:           # Non-matching group...
      0|[1-9]\d*  # 0, or multiple digits
    )
  )
  (\.\d+)?      # Decimal point and fractional part of the base
  (             # Exponent
    [eE]          # \
    [-+]?         #  > "e", plus or minus, exponent.
    \d+           # /
  )?

Some readers thought the \d in the base part was a bug; it’s not, actually — that expression matches either 0, or a number that starts with 1-9 and then contains any digits.

The actual bug is that this pattern matches non-normalized numbers (i.e. 123.45e3, which should more properly be written 1.2345e5).

4. ([\da-f]{2}:){5}([\da-f]{2})

Nearly everyone got this one: it’s a MAC address:

  ([\da-f]{2}:){5}  # Two hex digits followed by a colon, x5
  ([\da-f]{2})      # Two hex digits to end.

As birman noted, this pattern fails to match a few other forms allowed for MAC addresses; they can be written with hyphens (12-34-56-78-9A-BC), or as dotted quads (1234.5678.9ABC).

5. <[^>]*?>

This one also seemed to be easy for most readers; it matches any SGML tag. In verbose syntax:

  <        # Atart the tag
  [^>]*?   # Any non-gt character
  >        # End the tag

The “bug” in this one is a little more abstract: malformed SGML/HTML will severely muck it up. I’ll leave finding such code an exercise for the reader, though.

Next time

Tune in tomorrow for the next installment of the quiz. This week’s question will be a “things that every web developer should know” quiz; I think it’s a lot of fun.

See you tomorrow!

Of course, we all also know how dangerous they can be. As always, with great power comes great responsibility.

Still, if you know how — and when — and why — to use regular expressions, they’re indispensable. So this week, regular expressions will be our theme.

Below are five regular expressions. Each one of them matches a real-world string; that is, a semi-structured piece of text you might want to pull out of a greater document. Here’s an example question to give you an idea what I mean:

0. [0-9]{5}

This, of course, is a US ZIP code.

So, what “things” do these regular expressions match? We’ll assume for this quiz that the regex engine is running in case-insensitive mode:

1. [A-PR-Y0-9]{3}-[A-PR-Y0-9]{3}-[A-PR-Y0-9]{4}
2. &(?!(\w+|#\d+);)
3. (-?(?:0|[1-9]\d*))(\.\d+)?([eE][-+]?\d+)?
4. ([\da-f]{2}:){5}([\da-f]{2})
5. <[^>]*?>

Of course, since we’re dealing with regular expressions here, I’d be amiss if I didn’t give you two problems for the price of one.

In each case, the regular expression has something wrong with it. For example, the ZIP code regex above doesn’t correctly match the ZIP+4 format (i.e. 66044-0034) that’s used for many addresses these days.

So, for part two, what’s wrong with the rest of ‘em?

Enjoy your Thanksgiving belly-stuffing, and tune in over the weekend for the answers.

If you missed it, this week’s challenge deals with finding computer-readable public data resources. Before getting to the answers, though, let’s talk a little about technique.

Finding public data

By law (in the US), much of the data produced by government agencies must be made available publicly. As you might expect, however, this is often the last thing an acronym’d agency wants to think about. Thus, even when data is made available on the web, it’s often only provided in formats that are difficult to parse on websites that are difficult to find.

Google does a pretty good job of penetrating this maze of government websites. Most of those who commented on the original question were able to find at least a few sources using Google. For me, at least, a good deal of poking around and trying searches with different keywords was required.

Once at the right place, most people had no trouble finding data in a form at least nominally parseable. That’s a good sign; in this age of Microsoft Office, I often have to fight IT departments to get access to data in a format suitable for parsing into a database. I glad to see that the people responding to my question have a good grasp of what constitutes a friendly format.

A few readers had some nice tips for finding government data:

• malikyte pointed out out that “advanced filters can help quite a bit when you know what form of information you’re looking for, especially if a government organization is most likely involved. With [G]oogle, for instance, you can specify in the search terms: site:.gov “sec filings” or site:.org “sec filings” — limiting your search results goes a long way in removing unimportant data.” I hadn’t realized that Google’s site: operator could be used on TLDs; thanks!
• WindUpDoll easily found demographics for Wisconsin through her girlfriend who works for the city. I don’t in any way consider this cheating; nearly all of the cool work that we do at work is mad possible by an inside connection. If you’re in the business of dealing with public data, friends on the inside are key.
• dmbfansim mentioned the redundantly-named-yet-useful FirstGov.gov, “The U.S. Government’s Official Web Portal”. More specifically, the reference center is an invaluable resource.

Finally, a wonderful clearinghouse for government data is FedStats.gov; I found the questions for this quiz starting at that site.

The answers

Right, enough dallying; here are the answers. In some cases there were multiple sources found (by readers or by me); I’ve only provided one below:

1. Nutritional content of food from the USDA.
2. (Links to) population demographics of every major city in the US, courtesy of the US Census Bureau.
3. The latest SEC filings (in RSS, no less) straight from the horse’s mouth.
4. Historical gas prices, from the Energy Information Administration (which I had never heard of until writing this quiz).
5. Juvenile arrest rates from the Office of Juvenile Justice and Delinquency Prevention (part of the Department of Justice).

Was it good for you, too?

Next time…

Come Tuesday, we’ll tackle a tool that’s perhaps the most powerful text-processing engine known to man: regular expressions. Now you have two problems.

See you then.

At some point, nearly every web geek gets a chance to hack on some open data, usually from a government source. The buzzword here is “mashup,” but knowing how to find and consume openly available data will remain a valuable skill long after its faddishness ends.

Unfortunately, governments, and especially the US government, are often incredibly awful at providing this data. Sure, it’s available — but you’ve got to find it first.

So this question is all about finding that data. Since I’m most familiar with the USA, this question is USA-specific (but I’d love to see answers to any questions that apply to other nations).

In each case, the answer should be a URL where you can either download the data in question, or at least find a direct link to the data. There may be multiple sources for each, including ones that could be screen-scraped for the data. I’m not looking for those sources, however — just the ones with easily downloadable data in a format that can be easily parsed by a computer (i.e. CSV, XML, plain text). “Friendly” formats, in other words.

So, where can I download data to:

1. Analyze the nutritional content of foods?
2. Find the population (and other basic demographics) of my city?
3. Analyze the latest SEC filings by public companies?
4. Look at historical gas prices?
5. Look for trends in juvenile arrest rates?

Post your answers into the comments. For extra brownie points, tell us how you located each piece of data — did The Google serve you well, or were you forced to turn elsewhere?

If you really want to stretch your brain, try to write a tool to import each chunk of data into your favorite relational database. There will be a related question in a couple of weeks involving modeling one of these pieces of data, so you overachievers can start thinking about it now…

Good luck, and check back this weekend for the answers.

So what’s this all about?

There’s no question that being a web developer is incredibly complicated. Most of us know the in’s and out’s of enough acronyms to choke a government agency: HTML, CSS, AJAX, SQL, XML, JSON, HTTP… Being a jack-of-all-trades is in the job description.

Piecing all these disparate technologies together often feels like doing a complicated jigsaw puzzle. For many of us, the best part of being a web geek is that “aha!” moment when the puzzle is finally solved.

But the brain’s a muscle: stop exercising it and it’ll atrophy. This quiz is designed to keep your web development brain strong; each week, I’ll post a question that only a web developer should be able to solve.

Since we all have such a disparate set of skills, I’ll aim to make these questions accessible to any web developer by focusing on the tools and technologies that we all know. I’ll usually post questions on Tuesday, and then post a follow-up with the answer over the weekend.

Meet your quizmaster

My name is Jacob Kaplan-Moss; I’m the lead developer at the Lawrence Journal-World, a family-owned newspaper in Lawrence, KS. If my name looks familiar, it’s probably because I’m also one of the lead developers of Django, an open-source web framework (which was just featured in a SitePoint tutorial written by a coworker of mine).

This is my first gig hosting a quiz show (yeah, a virtual one, but still…), but that doesn’t mean I’ll go easy; expect these questions to be as hard as I can make ‘em.

Back after these messages…

So tune in tomorrow for the first question — a bit of a warm up to set the tone for questions to come.

See you there!