Nomological Net

Stray thoughts from here and there. The occasional concern for construct validity. No more logic. Fish.


faults in the clouds of delusion

Saturday, May 13, 2006

Theory Development as Freak Management

This is the second in an intermittent series of ruminations (the first being here). This post is what makes it a series, so hooray.

As you may have guessed, this post was inspired by my reading of the book Freakonomics. First off, it's an excellent book, but that wasn't my first reaction to it. I'd learned about it pretty much as soon as it came out last year, and the few pages I'd read off a friend's coffee table were definitely interesting. However I'd waited till last month to pick it up (Singapore airport bookstore impulse!), and the reason was simply -- the title. "Economics", as "oikos" + "nomos", translates roughly to "household management". The quest for the sexy title had led this book to be called "freak management", and that gave me pause. Was this a boo-boo, or was someone having a quiet laugh in the corner?

I was right, in a way. Much as this book is about the hard science of interesting questions well answered, it is also, in almost-equal parts, a hagiographic rumba around the Missing Beatle of Academia. Every chapter interlude contains italicized rhapsodies about the Great Levitt, including such wonders as how he notices headphones on a hobo (no kidding). At first, this was more than a little irritating. Then an extract from a speech I'd read came to mind: this is from Frank Kardes' presidential address to the Society for Consumer Psychology.

Many people believe ... [experimental] psychology is not useful. Laypeople want simple, concrete, and easy-to-use information, and this is not the type of information that is provided by experimental psychologists. However, many practising psychologists are more than happy to provide this type of information on television talk shows, self-help books, and audiotapes on psychokinesis, telepathy, clairvoyance, astral projection, ESP, biorhythms, holistic medicine, psychic 'surgery', pyramid power, and subliminal persuasion. Unfortunately, the public pays much more attention to the charlatans than to the scientists because the charlatans are much more entertaining.

I'm a marketing person. If this kind of freak management is what it takes to sell some real research to the paying public, how can I be against it?

The great strength of the book, naturally, is the great work of Steven Levitt. The man has a knack for identifying interesting questions as well as really intelligent ways of trying to answer them. These have been well documented in the myriad reviews the book has spawned -- the advent of abortion in the 70s as an explanation for the crime drop of the 90s; catching schoolteachers cheating for their students' benefit by developing an algorithm that checks for systematic repetitions of strings on multiple choice answer sheets; why drug dealers live with their mothers; forms of racism on open fora such as The Weakest Link; myths of parenting. As long as one is agnostic about what the data reveal, there isn't very much to quibble with here. The questions he chooses are very important, and his analysis strategies are unique. And usually brilliant.

But this book is about more than the freak that is Levitt. At one level, it is also about the freak questions that he asks. What, at first blush, does Roe vs. Wade have to do with Giuliani's seeming success story? Yet, once it's put that way, the immediate response is, Hmm, couldn't it?

Levitt's brilliance is in managing to ask a question as freaky as that.

I'm going to take this argument further, now, possibly beyond the scope of this book. The skill to ask such freaky questions is enviable, yet, I'd argue it isn't unique. Indeed, at some level, every good doctoral program drills the motto Ask interesting questions into everyone that passes through it. Years 2, 3, and 4 of most PhD students' lives are spent wrestling with the question What exactly *is* interesting? (often in more communicative language). Yet every successful dissertation eventually has an advisor and a committee behind it, admitting - sometimes grudgingly, sometimes reservedly - that yes, the question was interesting. Every published paper needs to pass that test. But even Levitt, for all his freaky interesting questions, for all his "Rogue Economist Explores the Hidden Side of Everything" blurbs, doesn't really give us very much of a theory. Of anything.

I realize that's a controversial statement to make, the word "theory" itself being so nebulous. All Levitt's hypotheses could well be cast as theories in the sense that they explain why certain things happened the way they happened (and might well happen again, if things were to stay the same). However, to my non-economist mind, a theory has to have predictive power as well as descriptive. What Levitt demonstrates is an astonishing power to uncover covariations between variables. And here I find my third spin on Theory Development as Freak Management -- the management of freaky data.

The positivist research process in social science, much as we hate to admit it, is appallingly linear. Blame this if you will on the human tendency to interpret all information in terms that support their prior beliefs. Social scientists develop hypotheses, collect and analyze data, and check to see whether their hypotheses are supported or not. Data where the initial hypotheses are only partially (I mean tantalizingly partially) supported, are sifted, sorted, coaxed, and battered till the green light of statistical significance is seen. Data that do not support the hypotheses are usually junked without ceremony. Our eyes seek desperately for particular patterns in the data, specific numbers and trends that would confirm our beliefs. Often as not, we miss the freaks -- the numbers that stand out, do not conform, that tell us things we hadn't thought to think. And it is of vital importance that we not miss the freaks in our data.

I say this from personal experience. Twice in my research life so far, I have caught a freak. Both times unexpected, unpredicted, and with hugely beneficial consequences. One freak got me a published paper plus five more in the pipeline, the other one a large chunk of my dissertation. It scares me to think about the amount of data I have rejected as "does not work", and the number of freaks that may have gotten away. Yet - as with many of Levitt's contentions - this one too is evident in hindsight: our reasoning powers are limited. We cannot predict everything. If we are reasonably competent, our investigations will lead us close to a "truth". However, we can never be confident of being anything except in the vicinity, and it is the unexpected variation in the data that often leads us to what we were seeking, but did not know.

That, to me, is the hidden beauty of the research process. That is theory development as freak management.


Blogger MockTurtle said...

" Blame this if you will on the human tendency to interpret all information in terms that support their prior beliefs. " - Couldn't agree more. Doesn't statistics train us to discard all freaky data?

5/15/2006 4:59 AM  
Blogger Tabula Rasa said...

well, yes. the "confirmation bias" is one of the most robust psychological effects known (check it out on wikipedia, along with the hostile media effect and the disconfirmation bias).

you're right about what statistics rightfully says, but selective dropping of outliers (across data sets) is one of the easiest legal ways to get the most out of ones data. (doing it within the same data set is definitely not kosher.)

5/15/2006 10:03 AM  
Blogger GhostOfTomJoad said...

Exactly what I would've liked to read on my return to the blogworld... a treatise on freakin' management...oops, freak management :-)

Anyway, wonder how someone arrived at the conclusion that all drug pushers live with their mothers!

5/15/2006 12:31 PM  
Blogger Tabula Rasa said...

sheesh, read some of the other posts, willya?! (wiggle-waggle, cow tipping...)

the drug pushers thing is really interesting -- read the book for the bits about how a young part-hippy sociology grad student called sudhir venkatesh spent three years living amongst the denizens of the chicago inner city.

and yeah -- welcome back... again!

5/15/2006 1:12 PM  
Blogger J. Alfred Prufrock said...

(in a small voice) I liked the book.


5/15/2006 10:48 PM  
Blogger Tabula Rasa said...

(smaller voice... i liked your post...)


5/15/2006 11:24 PM  
Blogger km said...

Wonderful post, TR.

So what you are saying is, we see what we believe (and now I remember that *awesome* Groucho quote: "who are you going to believe, me or your own eyes?") and we believe what we see. Don't all systems, hierarchies and institutions want us to look past freak data?

A hagiographic rumba around the Missing Beatle of Academia

I am gonna go Kaavya on ya and use that line in a post someday. Naturally, it will *not* be credited to you and I will end up sounding cool and all. Ha!

5/17/2006 4:26 AM  
Blogger Tabula Rasa said...

thanks ;-) you're right -- i hadn't thought of it that way, but yes, very large parts of society are based on the principle that one should studiously ignore freak data. that's some food for thought, isn't it? i'm curently in the middle of "collapse" by jared diamond, and i can see some links rightaway. hmmm. great insight.

and rumba on, mate.

5/17/2006 6:14 PM  
Blogger Brown Magic said...

You might have already read this but there is a fantastic profile on Levitt in The New York Times Magazine back from two/three years ago.

Made me a total fangirl.

5/17/2006 11:48 PM  
Blogger Tabula Rasa said...

hi brown magic,
thanks for the link! yes, i remember reading that story when it came out. nicely hagiographic, eh? :-D the world needs more people like him.
thanks for stopping by, and hope to see you again.

5/18/2006 8:54 PM  
Blogger GhostOfTomJoad said...

Hi, weren't you supposed to go and watch the cricket in the West Indies? Ad, to hold up a baner for us to recognise you? Are you there yet?

5/22/2006 9:33 PM  
Blogger Tabula Rasa said...

Indeed, I will! We leave on Wednesday, and should be at the games Friday and Sunday. Still taking suggestions for signs :-)

5/22/2006 11:20 PM  
Anonymous Astha said...

Very interesting post. I'm a psychologist myself and have come to the eventual love-hate relationship with stats. Well isn't that what they say about statistics- The average depth of the river is 6 feet, was 50 feet where you stepped. Oops...b'bye!

I think the true value of quantitative analysis is not in discovering a question or contextualizing it. That's the sphere of qualitative analysis. When you come to a question, explore it, live it- then perhaps it is time to look at the numbers and support it.

5/28/2006 2:45 AM  
Blogger Tabula Rasa said...

thanks for stopping by. yes, i agree that stats come second to the theory. however, as a down-home stat-lover, i have to say that there's no theory without the support of stats -- it's opinion!

6/02/2006 4:31 AM  

Post a Comment

<< Home