By David J. Epstein, Riverhead Books, May 28, 2019, 0593084497
I think of myself as being a generalist so this book affirmed my believe that generalists are better problem solvers than specialists. More specifically, I think generalists make better programmers, but Epstein doesn’t really talk much about programming or programmers. He’s more of a sports and science guy.
There were many “aha!” moments in the book, e.g. Carter Racing biz school case study. The book is full of insights into how people think and how they get stuck. The standard list of experts on decision making (Tetlock, Kahneman, Levitt, etc.) are quoted, but also people like Darwin and Kepler, who analyze their own thinking process.
Epstein writes extremely well. I highly recommend the book. I also listened to the audio book The Sports Gene by David Epstein, which I enjoyed.
[k119] Tiger was not merely playing golf. He was engaging in “deliberate practice,” the only kind that counts in the now-ubiquitous ten-thousand-hours rule to expertise.
[k155] The professed necessity of hyperspecialization forms the core of a vast, successful, and sometimes well-meaning marketing machine, in sports and beyond. In reality, the Roger path to sports stardom is far more prevalent than the Tiger path, but those athletes’ stories are much more quietly told, if they are told at all.
[k196] I was slightly bemused to find that a former Navy SEAL with an undergraduate degree in history and geophysics pursuing graduate degrees in business and public administration from Dartmouth and Harvard could feel behind. But like the others in the room, he had been told, implicitly or explicitly, that changing directions was dangerous.
[k208] Researchers at Northwestern, MIT, and the U.S. Census Bureau studied new tech companies and showed that among the fastest-growing start-ups, the average age of a founder was forty-five when the company was launched.
[k224] A recent study found that cardiac patients were actually less likely to die if they were admitted during a national cardiology meeting, when thousands of cardiologists were away; the researchers suggested it could be because common treatments of dubious effect were less likely to be performed.
[k242] The challenge we all face is how to maintain the benefits of breadth, diverse experience, interdisciplinary thinking, and delayed concentration in a world that increasingly incentivizes, even demands,
[k321] Every time, they learned they had done barely better than blind guessing. Every time, they gained experience and gave confident judgments. And every time, they did not improve. Kahneman marveled at the “complete lack of connection between the statistical information and the compelling experience of insight.”
[k329] In 2009, Kahneman and Klein took the unusual step of coauthoring a paper in which they laid out their views and sought common ground. And they found it. Whether or not experience inevitably led to expertise, they agreed, depended entirely on the domain in question. Narrow experience made for better chess and poker players and firefighters, but not for better predictors of financial or political trends, or of how employees or patients would perform. The domains Klein studied, in which instinctive pattern recognition worked powerfully, are what psychologist Robin Hogarth termed “kind” learning environments. Patterns repeat over and over, and feedback is extremely accurate and usually very rapid.
[k339] Kahneman was focused on the flip side of kind learning environments; Hogarth called them “wicked.”
In wicked domains, the rules of the game are often unclear or incomplete, there may or may not be repetitive patterns and they may not be obvious, and feedback is often delayed, inaccurate, or both.
[k371] A few years later, the first “freestyle chess” tournament was held. Teams could be made up of multiple humans and computers. The lifetime-of-specialized-practice advantage that had been diluted in advanced chess was obliterated in freestyle.
[k396] The first took place in the 1940s, when Dutch chess master and psychologist Adriaan de Groot flashed midgame chessboards in front of players of different ability levels, and then asked them to re-create the boards as well as they could. A grandmaster repeatedly re-created the entire board after seeing it for only three seconds. A master-level player managed that half as often as the grandmaster. A lesser, city champion player and an average club player were never able to re-create the board accurately. Just like Susan Polgar, grandmasters seemed to have photographic memories.
[k404] This time, the chess players were also given boards with the pieces in an arrangement that would never actually occur in a game. Suddenly, the experts performed just like the lesser players.
[k402] That test reenacted an experiment from 1973, in which two Carnegie Mellon University psychologists, William G. Chase and soon-to-be Nobel laureate Herbert A. Simon, repeated the De Groot exercise, but added a wrinkle. This time, the chess players were also given boards with the pieces in an arrangement that would never actually occur in a game. Suddenly, the experts performed just like the lesser players.
[k443] It took Treffert decades to realize he had been wrong, and that savants have more in common with prodigies like the Polgar sisters than he thought. They do not merely regurgitate. Their brilliance, just like the Polgar brilliance, relies on repetitive structures, which is precisely what made the Polgars’ skill so easy to automate.
[k465] According to Gary Marcus, a psychology and neural science professor who sold his machine learning company to Uber, “In narrow enough worlds, humans may not have much to contribute much longer. In more open-ended games, I think they certainly will. Not just games, in open ended real-world problems we’re still crushing the machines.” The progress of AI in the closed and orderly world of chess, with instant feedback and bottomless data, has been exponential.
[k472] “The difference between winning at Jeopardy! and curing all cancer is that we know the answer to Jeopardy! questions.” With cancer, we’re still working on posing the right questions in the first place.
[k495] The subtitle of Schwartz’s paper: “How Not to Teach People to Discover Rules”–that is, by providing rewards for repetitive short-term success with a narrow range of solutions.
[k502] As Robin Hogarth put it, much of the world is “Martian tennis.” You can see the players on a court with balls and rackets, but nobody has shared the rules. It is up to you to derive them, and they are subject to change without notice.
[k596] The gains around the world on Raven’s Progressive Matrices–where change was least expected–were the biggest of all. “The huge Raven’s gains show that today’s children are far better at solving problems on the spot without a previously learned method for doing so,”
[k656] To use a common metaphor, premodern people miss the forest for the trees; modern people miss the trees for the forest. Since Luria’s voyage to the interior, scientists have replicated his work in other cultures.
[k659] Given similarities tests, teenagers who were engaged with modern institutions grouped items by abstract categories (“All of these things can keep us warm”), while the traditional teens generated groups that were comparatively arbitrary, and changed frequently even when they were asked to repeat the exact same task. Because the touched-by-modernity teens had constructed meaningful thematic groups, they also had far superior recall when asked later to recount the items.
[k662] The more they had moved toward modernity, the more powerful their abstract thinking, and the less they had to rely on their concrete experience of the world as a reference point.
[k669] Words that represent concepts that were previously the domain of scholars became widely understood in a few generations. The word “percent” was almost absent from books in 1900. By 2000 it appeared about once every five thousand words. (This chapter is 5,500 words long.) Computer programmers pile layers of abstraction. (They do very well on Raven’s.) In the progress bar on your computer screen that fills up to indicate a download, abstractions are legion, from the fundamental–the programming language that created it is a representation of binary code, the raw 1s and 0s the computer uses–to the psychological: the bar is a visual projection of time that provides peace of mind by estimating the progress of an immense number of underlying activities.
[k682] As Flynn makes sure to point out, this does not mean that brains now have more inherent potential than a generation ago, but rather that utilitarian spectacles have been swapped for spectacles through which the world is classified by concepts.
[k719] Flynn was bemused to find that the correlation between the test of broad conceptual thinking and GPA was about zero. In Flynn’s words, “the traits that earn good grades at [the university] do not include critical ability of any broad significance.”*
[k728] Econ majors did the best overall. Economics is a broad field by nature, and econ professors have been shown to apply the reasoning principles they’ve learned to problems outside their area. Chemists, on the other hand, are extraordinarily bright, but in several studies struggled to apply scientific reasoning to nonchemistry problems.
Students Flynn tested often mistook subtle value judgments for scientific conclusions, and in a question that presented a tricky scenario and required students not to mistake a correlation for evidence of causation, they performed worse than random.
[k762] “Computational thinking is using abstraction and decomposition when attacking a large complex task,” she wrote. “It is choosing an appropriate representation for a problem.”
[k777] As statistician Doug Altman put it, “Everyone is so busy doing research they don’t have time to stop and think about the way they’re doing it.”
[k807] The more constrained and repetitive a challenge, the more likely it will be automated, while great rewards will accrue to those who can take conceptual knowledge from one problem or domain and apply it in an entirely new one.
[k1165] In offering advice to parents, psychologist Adam Grant noted that creativity may be difficult to nurture, but it is easy to thwart. He pointed to a study that found an average of six household rules for typical children, compared to one in households with extremely creative children.
[k1240] Lindsey Richland, a University of Chicago professor who studies learning, watched that video with me, and told me that when the students were playing multiple choice with the teacher, “what they’re actually doing is seeking rules.” They were trying to turn a conceptual problem they didn’t understand into a procedural one they could just execute. “We’re very good, humans are, at trying to do the least amount of work that we have to in order to accomplish a task,” Richland told me.
[k1248] Teachers in every country fell into the same trap at times, but in the higher-performing countries plenty of making-connections problems remained that way as the class struggled to figure them out. In Japan, a little more than half of all problems were making-connections problems, and half of those stayed that way through the solving.
[k1255] (There is a specific Japanese word to describe chalkboard writing that tracks conceptual connections over the course of collective problem solving: bansho.)
[k1274] But for learning that is both durable (it sticks) and flexible (it can be applied broadly), fast and easy is precisely the problem.
[k1293] Tolerating big mistakes can create the best learning opportunities.
[k1343] As with excessive hint-giving, it will, as a group of psychologists put it, “produce misleadingly high levels of immediate mastery that will not survive the passage of substantial periods of time.” For a given amount of material, learning is most efficient in the long run when it is really inefficient in the short run. If you are doing too well when you test yourself, the simple antidote is to wait longer before practicing the same material again, so that the test will be more difficult when you do. Frustration is not a sign you are not learning, but ease is.
[k1349] In 2007, the U.S. Department of Education published a report by six scientists and an accomplished teacher who were asked to identify learning strategies that truly have scientific backing. Spacing, testing, and using making-connections questions were on the extremely short list. All three impair performance in the short term.
[k1397] Focusing on “using procedures” problems worked well forty years ago when the world was flush with jobs that paid middle-class salaries for procedural tasks, like typing, filing, and working on an assembly line. “Increasingly,” according to Duncan, “jobs that pay well require employees to be able to solve unexpected problems, often while working in groups. . . . These shifts in labor force demands have in turn put new and increasingly stringent demands on schools.”
[k1422] Interleaving has been shown to improve inductive reasoning. When presented with different examples mixed together, students learn to create abstract generalizations that allow them to apply what they learned to material they have never encountered before.
[k1436] In one of Kornell and Bjork’s interleaving studies, 80 percent of students were sure they had learned better with blocked than mixed practice, whereas 80 percent performed in a manner that proved the opposite. The feeling of learning, it turns out, is based on before-your-eyes progress, while deep learning is not.
[k1464] The research team recommended that if programs want to impart lasting academic benefits they should focus instead on “open” skills that scaffold later knowledge.
[k1478] When a knowledge structure is so flexible that it can be applied effectively even in new domains or extremely novel situations, it is called “far transfer.”
[k1533] His fastidious documentation of every meandering path his brain blazed is one of the great records of a mind undergoing creative transformation. It is a truism to say that Kepler thought outside the box.
[k1544] “In my opinion,” Gentner told me, “our ability to think relationally is one of the reasons we’re running the planet. Relations are really hard for other species.” Analogical thinking takes the new and makes it familiar, or takes the familiar and puts it in a new light, and allows humans to reason through problems they have never seen in unfamiliar contexts.
[k1561] Like kind learning environments, a kind world is based on repeating patterns. “It’s perfectly fine,” she said, “if you stay in the same village or the same savannah all your life.” The current world is not so kind; it requires thinking that cannot fall back on previous experience. Like math students, we need to be able to pick a strategy for problems we have never seen before. “In the life we lead today,” Gentner told me, “we need to be reminded of things that are only abstractly or relationally similar. And the more creative you want to be, the more important that is.”
[k1634] The outside view is deeply counterintuitive because it requires a decision maker to ignore unique surface features of the current project, on which they are the expert, and instead look outside for structurally similar analogies. It requires a mindset switch from narrow to broad.
[k1737] In one of the most cited studies of expert problem solving ever conducted, an interdisciplinary team of scientists came to a pretty simple conclusion: successful problem solvers are more able to determine the deep structure of a problem before they proceed to match a strategy to it. Less successful problem solvers are more like most students in the Ambiguous Sorting Task: they mentally classify problems only by superficial, overtly stated features, like the domain context.
[k1757] Kepler could have assumed his model was correct and those two observations were slightly off, or he could dispense with five years of work. He chose to trash his model. “If I had believed we could ignore these eight minutes,” he wrote, “I would have patched my hypothesis accordingly.”
[k1969] On the “Freakonomics Experiments” home page, he invited readers who were considering life changes to flip a digital coin. Heads meant they should go ahead and make the change, tails that they should not. Twenty thousand volunteers responded, agonizing over everything from whether they should get a tattoo, try online dating, or have a child, to the 2,186 people who were pondering a job change. But could they really trust a momentous decision to chance? The answer for the potential job changers who flipped heads was: only if they wanted to be happier. Six months later, those who flipped heads and switched jobs were substantially happier than the stayers.* According to Levitt, the study suggested that “admonitions such as ‘winners never quit and quitters never win,’ while well-meaning, may actually be extremely poor advice.” Levitt identified one of his own most important skills as “the willingness to jettison” a project or an entire area of study for a better fit.
[k2337] Our work preferences and our life preferences do not stay the same, because we do not stay the same.
[k2348] The precise person you are now is fleeting, just like all the other people you’ve been. That feels like the most unexpected result, but it is also the most well documented.
[k2395] Instead of asking whether someone is gritty, we should ask when they are. “If you get someone into a context that suits them,” Ogas said, “they’ll more likely work hard and it will look like grit from the outside.”
[k2411] Ibarra concluded that we maximize match quality throughout life by sampling activities, social groups, contexts, jobs, careers, and then reflecting and adjusting our personal narratives. And repeat. If that sounds facile, consider that it is precisely the opposite of a vast marketing crusade that assures customers they can alight on their perfect matches via introspection alone.
[k2420] Instead, she told me, in a clever inversion of a hallowed axiom, “First act and then think.” Ibarra marshaled social psychology to argue persuasively that we are each made up of numerous possibilities. As she put it, “We discover the possibilities by doing, by trying new activities, building new networks, finding new role models.” We learn who we are in practice, not in theory.
[k2443] Be a flirt with your possible selves.* Rather than a grand plan, find experiments that can be undertaken quickly. “Test-and-learn,” Ibarra told me, “not plan-and-implement.”
[k2468] Michelangelo might have fit well in Silicon Valley; he was a relentless iterator.
[k2522] Of course, there is nothing wrong with coming through the formal talent development system, but if that’s the only pipeline that exists, some of the brightest talents get missed.
[k2642] Pegau was basically describing the Einstellung effect, a psychology term for the tendency of problem solvers to employ only familiar methods even if better ones are available.
[k2648] InnoCentive works in part because, as specialists become more narrowly focused, “the box” is more like Russian nesting dolls. Specialists divide into subspecialties, which soon divide into sub-subspecialties. Even if they get outside the small doll, they may get stuck inside the next, slightly larger one.
[k2655] Karim Lakhani, codirector of the Laboratory for Innovation Science at Harvard, had InnoCentive solvers rate problems on how relevant they were to their own field of specialization, and found that “the further the problem was from the solver’s expertise, the more likely they were to solve it.”
[k2662] For the most intractable problems, “our research shows that a domain-based solution is often inferior,” according to Lakhani. “Big innovation most often happens when an outsider who may be far away from the surface of the problem reframes the problem in a way that unlocks the solution.”
[k2967] Yokoi was the first to admit it. “I don’t have any particular specialist skills,” he once said. “I have a sort of vague knowledge of everything.” He advised young employees not just to play with technology for its own sake, but to play with ideas.
[k3074] His hypothesis is that organizations simply don’t need as many specialists. “As information becomes more broadly available, the need for somebody to just advance a field isn’t as critical because in effect they are available to everybody,” he said.
[k3087] Specialization is obvious: keep going straight. Breadth is trickier to grow.
[k3148] They titled their study Superman or the Fantastic Four? “When seeking innovation in knowledge-based industries,” they wrote, “it is best to find one ‘super’ individual. If no individual with the necessary combination of diverse knowledge is available, one should form a ‘fantastic’ team.” Diverse experience was impactful when created by platoon in teams, and even more impactful when contained within an individual.
[k3159] In kind environments, where the goal is to re-create prior performance with as little deviation as possible, teams of specialists work superbly. Surgical teams work faster and make fewer mistakes as they repeat specific procedures, and specialized surgeons get better outcomes even independent of repetitions.
[k3165] When the National Transportation Safety Board analyzed its database of major flight accidents, it found that 73 percent occurred on a flight crew’s first day working together.
[k3168] “Some tools work fantastically in certain situations, advancing technology in smaller but important ways, and those tools are well known and well practiced,” Andy Ouderkirk told me. “Those same tools will also pull you away from a breakthrough innovation. In fact, they’ll turn a breakthrough innovation into an incremental one.”
[k3207] Facing kind problems, narrow specialization can be remarkably efficient. The problem is that we often expect the hyperspecialist, because of their expertise in a narrow area, to magically be able to extend their skill to wicked problems. The results can be disastrous.
[k3208] The problem is that we often expect the hyperspecialist, because of their expertise in a narrow area, to magically be able to extend their skill to wicked problems. The results can be disastrous. CHAPTER 10 Fooled by Expertise THE BET WAS ON, and it was over the fate of humanity.
[k3232] Simon saw that innovation was altering the equation. More people would actually be the solution, because it meant more good ideas and more technological breakthroughs. So Simon proposed a bet. Ehrlich could choose five metals that he expected to become more expensive as resources were depleted and chaos ensued over the next decade.
[k3247] Despite one erroneous prediction after another, Ehrlich amassed an enormous following and continued to receive prestigious awards. Simon became a standard-bearer for scholars who felt that Ehrlich had ignored economic principles, and for anyone angry at an incessant flow of dire predictions that did not manifest.
[k3261] “The opposite happened with Paul Ehrlich and Julian Simon.” As each man amassed more information for his own view, each became more dogmatic, and the inadequacies in their models of the world more stark.
There is a particular kind of thinker, one who becomes more entrenched in their single big idea about how the world works even in the face of contrary facts, whose predictions become worse, not better, as they amass information for their mental representation of the world. They are on television and in the news every day, making worse and worse predictions while claiming victory, and they have been rigorously studied.
[k3288] “There is often a curiously inverse relationship,” Tetlock concluded, “between how well forecasters thought they were doing and how well they did.”
[k3319] Tetlock would start in one direction, then interrogate himself and make an about-face.
[k3390] In separate work, from 2000 to 2010 German psychologist Gerd Gigerenzer compiled annual dollar-euro exchange rate predictions made by twenty-two of the most prestigious international banks–Barclays, Citigroup, JPMorgan Chase, Bank of America Merrill Lynch, and others. Each year, every bank predicted the end-of-year exchange rate. Gigerenzer’s simple conclusion about those projections, from some of the world’s most prominent specialists: “Forecasts of dollar-euro exchange rates are worthless.” In six of the ten years, the true exchange rate fell outside the entire range of all twenty-two bank forecasts. Where a superforecaster quickly highlighted a change in exchange rate direction that confused him, and adjusted, major bank forecasts missed every single change of direction in the decade Gigerenzer analyzed.
[k3423] Charles Darwin must have been one of the most curious and actively open-minded human beings in history. His first four models of evolution were forms of creationism or intelligent design. (The fifth model treated creation as a separate question.)
[k3442] Foxes see complexity in what others mistake for simple cause and effect.
[k3548] “How many times did I say yesterday if you want additional information let me know?” Muffled gasps spread across the room. “Four times,” the professor answers himself. “Four times I said if you want additional information let me know.” Not one student asked for the missing data.
[k3552] Every single race below 65 degrees had an engine failure.
[k3554] He informs the students that there is a 99.4 percent probability of engine failure at 40 degrees. “Do we have any remaining fans of racing?” he asks. And now he has another surprise.
The temperature and engine failure data are taken exactly from NASA’s tragic decision to launch the space shuttle Challenger, with the details placed in the context of racing rather than space exploration. Jake’s face goes blank. Rather than a broken gasket, Challenger had failed O-rings–the rubber strips that sealed joints along the outer wall of the missile-like rocket boosters that propelled the shuttle.
[k3559] The characters in the case study are loosely based on managers and engineers at NASA and its rocket-booster contractor, Morton Thiokol, on an emergency conference call the night before the Challenger launch.
[k3566] “Like all of you, nobody [at NASA or Thiokol] asked for the seventeen data points for which there had been no problems,” he explains.
[k3579] Business professors around the world have been teaching Carter Racing for thirty years because it provides a stark lesson in the danger of reaching conclusions from incomplete data, and the folly of relying only on what is in front of you.
And now for one last surprise. They all got it wrong. The Challenger decision was not a failure of quantitative analysis. NASA’s real mistake was to rely on quantitative analysis too much.
Before ignition, Challenger’s O-rings sat squashed in the joints that connected vertical sections of the booster.
[k3606] There was other important information the Thiokol engineers presented that could have helped NASA avert disaster. But it was not quantitative, so NASA managers did not accept it. The Carter Racing study teaches that the answer was available, if only engineers looked at the right numbers. In reality, the right numbers did not contain an answer at all.
[k3642] The very tool that had helped make NASA so consistently successful, what Diane Vaughan called “the original technical culture” in the agency’s DNA, suddenly worked perversely in a situation where the familiar brand of data did not exist. Reason
[k3644] Reason without numbers was not accepted. In the face of an unfamiliar challenge, NASA managers failed to drop their familiar tools.
[k3809] The researcher who led that work went on to study thousands of businesses. She found that the most effective leaders and organizations had range; they were, in effect, paradoxical.
[k3879] The management and culture aspects of the Challenger and Columbia disasters were so eerily similar that the investigation board decreed that NASA was not functioning as “a learning organization.”
[k3917] “You almost couldn’t go into a meeting without someone saying, ‘Let’s take that offline,’” he recalled, just as Morton Thiokol had done for the infamous offline caucus.
[k3988] Most people with a torn meniscus, it turns out, don’t have any symptoms at all and will never even know. And for those who do have a torn meniscus and knee pain, the tear may have nothing to do with the pain.
[k4323] Everyone progresses at a different rate, so don’t let anyone else make you feel behind. You probably don’t even know where exactly you’re going, so feeling behind doesn’t help.