Theory Development as Freak Management

This is the second in an intermittent series of ruminations (the first being here). This post is what makes it a series, so hooray.

As you may have guessed, this post was inspired by my reading of the book Freakonomics. First off, it's an excellent book, but that wasn't my first reaction to it. I'd learned about it pretty much as soon as it came out last year, and the few pages I'd read off a friend's coffee table were definitely interesting. However I'd waited till last month to pick it up (Singapore airport bookstore impulse!), and the reason was simply -- the title. "Economics", as "oikos" + "nomos", translates roughly to "household management". The quest for the sexy title had led this book to be called "freak management", and that gave me pause. Was this a boo-boo, or was someone having a quiet laugh in the corner?

I was right, in a way. Much as this book is about the hard science of interesting questions well answered, it is also, in almost-equal parts, a hagiographic rumba around the Missing Beatle of Academia. Every chapter interlude contains italicized rhapsodies about the Great Levitt, including such wonders as how he notices headphones on a hobo (no kidding). At first, this was more than a little irritating. Then an extract from a speech I'd read came to mind: this is from Frank Kardes' presidential address to the Society for Consumer Psychology.

Many people believe ... [experimental] psychology is not useful. Laypeople want simple, concrete, and easy-to-use information, and this is not the type of information that is provided by experimental psychologists. However, many practising psychologists are more than happy to provide this type of information on television talk shows, self-help books, and audiotapes on psychokinesis, telepathy, clairvoyance, astral projection, ESP, biorhythms, holistic medicine, psychic 'surgery', pyramid power, and subliminal persuasion. Unfortunately, the public pays much more attention to the charlatans than to the scientists because the charlatans are much more entertaining.

I'm a marketing person. If this kind of freak management is what it takes to sell some real research to the paying public, how can I be against it?

The great strength of the book, naturally, is the great work of Steven Levitt. The man has a knack for identifying interesting questions as well as really intelligent ways of trying to answer them. These have been well documented in the myriad reviews the book has spawned -- the advent of abortion in the 70s as an explanation for the crime drop of the 90s; catching schoolteachers cheating for their students' benefit by developing an algorithm that checks for systematic repetitions of strings on multiple choice answer sheets; why drug dealers live with their mothers; forms of racism on open fora such as The Weakest Link; myths of parenting. As long as one is agnostic about what the data reveal, there isn't very much to quibble with here. The questions he chooses are very important, and his analysis strategies are unique. And usually brilliant.

But this book is about more than the freak that is Levitt. At one level, it is also about the freak questions that he asks. What, at first blush, does Roe vs. Wade have to do with Giuliani's seeming success story? Yet, once it's put that way, the immediate response is, Hmm, couldn't it?

Levitt's brilliance is in managing to ask a question as freaky as that.

I'm going to take this argument further, now, possibly beyond the scope of this book. The skill to ask such freaky questions is enviable, yet, I'd argue it isn't unique. Indeed, at some level, every good doctoral program drills the motto Ask interesting questions into everyone that passes through it. Years 2, 3, and 4 of most PhD students' lives are spent wrestling with the question What exactly *is* interesting? (often in more communicative language). Yet every successful dissertation eventually has an advisor and a committee behind it, admitting - sometimes grudgingly, sometimes reservedly - that yes, the question was interesting. Every published paper needs to pass that test. But even Levitt, for all his freaky interesting questions, for all his "Rogue Economist Explores the Hidden Side of Everything" blurbs, doesn't really give us very much of a theory. Of anything.

I realize that's a controversial statement to make, the word "theory" itself being so nebulous. All Levitt's hypotheses could well be cast as theories in the sense that they explain why certain things happened the way they happened (and might well happen again, if things were to stay the same). However, to my non-economist mind, a theory has to have predictive power as well as descriptive. What Levitt demonstrates is an astonishing power to uncover covariations between variables. And here I find my third spin on Theory Development as Freak Management -- the management of freaky data.

The positivist research process in social science, much as we hate to admit it, is appallingly linear. Blame this if you will on the human tendency to interpret all information in terms that support their prior beliefs. Social scientists develop hypotheses, collect and analyze data, and check to see whether their hypotheses are supported or not. Data where the initial hypotheses are only partially (I mean tantalizingly partially) supported, are sifted, sorted, coaxed, and battered till the green light of statistical significance is seen. Data that do not support the hypotheses are usually junked without ceremony. Our eyes seek desperately for particular patterns in the data, specific numbers and trends that would confirm our beliefs. Often as not, we miss the freaks -- the numbers that stand out, do not conform, that tell us things we hadn't thought to think. And it is of vital importance that we not miss the freaks in our data.

I say this from personal experience. Twice in my research life so far, I have caught a freak. Both times unexpected, unpredicted, and with hugely beneficial consequences. One freak got me a published paper plus five more in the pipeline, the other one a large chunk of my dissertation. It scares me to think about the amount of data I have rejected as "does not work", and the number of freaks that may have gotten away. Yet - as with many of Levitt's contentions - this one too is evident in hindsight: our reasoning powers are limited. We cannot predict everything. If we are reasonably competent, our investigations will lead us close to a "truth". However, we can never be confident of being anything except in the vicinity, and it is the unexpected variation in the data that often leads us to what we were seeking, but did not know.

That, to me, is the hidden beauty of the research process. That is theory development as freak management.