Monday, May 10, 2010

The Data-Driven Life -

Humans make errors. We make errors of fact and errors of judgment. We have blind spots in our field of vision and gaps in our stream of attention. Sometimes we can’t even answer the simplest questions. Where was I last week at this time? How long have I had this pain in my knee? How much money do I typically spend in a day? These weaknesses put us at a disadvantage. We make decisions with partial information. We are forced to steer by guesswork. We go with our gut.

That is, some of us do. Others use data. A timer running on Robin Barooah’s computer tells him that he has been living in the United States for 8 years, 2 months and 10 days. At various times in his life, Barooah — a 38-year-old self-employed software designer from England who now lives in Oakland, Calif. — has also made careful records of his work, his sleep and his diet.

A few months ago, Barooah began to wean himself from coffee. His method was precise. He made a large cup of coffee and removed 20 milliliters weekly. This went on for more than four months, until barely a sip remained in the cup. He drank it and called himself cured. Unlike his previous attempts to quit, this time there were no headaches, no extreme cravings. Still, he was tempted, and on Oct. 12 last year, while distracted at his desk, he told himself that he could probably concentrate better if he had a cup. Coffee may have been bad for his health, he thought, but perhaps it was good for his concentration.

Barooah wasn’t about to try to answer a question like this with guesswork. He had a good data set that showed how many minutes he spent each day in focused work. With this, he could do an objective analysis. Barooah made a chart with dates on the bottom and his work time along the side. Running down the middle was a big black line labeled “Stopped drinking coffee.” On the left side of the line, low spikes and narrow columns. On the right side, high spikes and thick columns. The data had delivered their verdict, and coffee lost.

He was sad but also thrilled. Instead of a stimulating cup of coffee, he got a bracing dose of truth. “People have such very poor sense of time,” Barooah says, and without good time calibration, it is much harder to see the consequences of your actions. If you want to replace the vagaries of intuition with something more reliable, you first need to gather data. Once you know the facts, you can live by them.

Five years ago, Ben Lipkowitz, who is now 28, was living with some friends in Bloomington, Ind., and he found himself wondering how much time he spent doing one of his roommates’ dishes. Lipkowitz had a handheld electronic datebook that he purchased on a trip to Tokyo, and on May 11, 2005, at 2:20 p.m., he started using it to keep a record of his actions. Instead of entering his future appointments, he entered his past activities, creating a remarkably complete account of his life. In one sense this was just a normal personal journal, albeit in a digital format and unusually detailed. But the format and detail made all the difference. Lipkowitz eventually transferred the data to his computer, and now, using a few keyboard commands, he can call up his history. He knows how much he has eaten and how much he has spent. He knows what books he has read and what objects he has purchased. And of course, he knows the answer to his original question. “I was thinking I was spending an hour a day cleaning up after this person,” Lipkowitz says. He shrugs. “It turned out it was more like 20 minutes.”

Another person I’m friendly with, Mark Carranza — he also makes his living with computers — has been keeping a detailed, searchable archive of all the ideas he has had since he was 21. That was in 1984. I realize that this seems impossible. But I have seen his archive, with its million plus entries, and observed him using it. He navigates smoothly between an interaction with somebody in the present moment and his digital record, bringing in associations to conversations that took place years earlier. Most thoughts are tagged with date, time and location. What for other people is an inchoate flow of mental life is broken up into elements and cross-referenced.

These men all know that their behavior is abnormal. They are outliers. Geeks. But why does what they are doing seem so strange? In other contexts, it is normal to seek data. A fetish for numbers is the defining trait of the modern manager. Corporate executives facing down hostile shareholders load their pockets full of numbers. So do politicians on the hustings, doctors counseling patients and fans abusing their local sports franchise on talk radio. Charles Dickens was already making fun of this obsession in 1854, with his sketch of the fact-mad schoolmaster Gradgrind, who blasted his students with memorized trivia. But Dickens’s great caricature only proved the durability of the type. For another century and a half, it got worse.

Or, by another standard, you could say it got better. We tolerate the pathologies of quantification — a dry, abstract, mechanical type of knowledge — because the results are so powerful. Numbering things allows tests, comparisons, experiments. Numbers make problems less resonant emotionally but more tractable intellectually. In science, in business and in the more reasonable sectors of government, numbers have won fair and square.

For a long time, only one area of human activity appeared to be immune. In the cozy confines of personal life, we rarely used the power of numbers. The techniques of analysis that had proved so effective were left behind at the office at the end of the day and picked up again the next morning. The imposition, on oneself or one’s family, of a regime of objective record keeping seemed ridiculous. A journal was respectable. A spreadsheet was creepy.

More ...