Now that I know what you're talking about:
Jakob Nielsen on tabs (His actual research is published in books and PDF form, but they are large, expensive studies, often done for clients, so you will pay for detailed study results... not a bad thing, I have one and it was excellent.)
search to Tabbed Dialogs:
Tabbed Dialog Layout (as part of a school course)
Excerpt from Aaron Gustavson's book @ A List Apart (not a whole lot here)
Old examples of BAD tabbed dialogue interfaces (famous old MS mistakes lawlz... mostly, tabs were more than one row. Avoid.)
Go ahead and read this one, but I suspect 20 years has made a heck of a difference in whether users know how to use tabs
Steve Krug mentions how to make sure they are usable in his book Don't Make Me Think: They need to be obvious who's selected, all tabs must be visible at once (no adding new choices dynamically) and that one of them is selected by default. Of course they need to work with keyboard like any selectable thing. A good book to purchase ANYWAY.
This is kind of a good idea: Make paper prototypes of the tabs you plan on building, and do real user testing with them. Just ask people what happens when they click on X, and/or ask them what they would do (or, it you must tell them, where they would click) to find information X. Nothing wrong with local, real testing. It gives you a kind of tangible data that online studies can supplement.
Search terms: tabbed interfaces, tabbed dialogues, + usability