Click on the Link to Download the PDF Precis of The Wisdom of Crowds
.
‘The Wisdom of Crowds’
James Surowiecki
A précis of the book by Pete Laburn
The powerful truth at the heart of James Surowiecki’s book ‘The Wisdom of Crowds’ states that under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them. Groups do not have to be dominated by exceptionally intelligent people in order to be smart. Even if most of the people within a group are not especially well‐informed or rational, it can still reach a collectively wise decision. Human beings are not perfect decision makers, due to limited information and foresight into the future, a lack of ability to make sophisticated cost‐benefit decisions and that we let emotion affect our judgement. Despite all these limitations, when our imperfect judgements are aggregated in the right way, our collective intelligence – referred to in the book as ‘the wisdom of crowds’ – is often excellent.
This wisdom is at work in the world around us, but is easy to miss, and even when it’s seen, it is hard to accept. Most of us believe that valuable knowledge is concentrated in a very few hands. We assume that the key to solving problems or making good decisions is finding that one right person who will have the answer. Even when we see a large crowd of people, many of them not well‐informed, do something amazing, we are more likely to attribute that success to a few smart people in the crowd than to the crowd itself. As sociologists Jack B Soll and Richard Larrick put it, we feel the need to ‘chase the expert’. However, chasing the expert can be a very costly mistake. Rather companies should stop hunting for the expert and ask the crowd because chances are it knows the answer. This knowledge has the potential to make a profound difference in the way companies do business. Although different groups of people are very different, they have in common the ability to act collectively to make decisions and to solve problems – even if the people in the groups are not always aware that is what they are doing. Groups are smart and good at solving problems. If you put together a big enough and diverse enough group of people and ask them to make decisions affecting matters of general interest, the groups decisions will, over time, be intellectually superior to the isolated individual, no matter how smart or well informed he is.
The book is about the way the world works and highlights that the wisdom of crowds has a far more important and beneficial impact on our everyday lives than we recognise, and its implications for the future are immense. But in the present, many groups struggle to make even mediocre decisions, while others wreak havoc with their bad judgement. Groups work well under certain circumstances and less well under others. Groups generally need rules to maintain order and coherence, and when these are missing or malfunctioning, the result is trouble. Groups benefit from members talking to, and learning from each other, but too much communication can actually make the group less intelligent. While big groups are often good for solving certain kinds of problems, they can become unmanageable and inefficient. Conversely, small groups are easier to run, but run the risk of having too little diversity of thought and too much consensus. In addition, there are times when aggregating individual decisions produces a collective decision that is utterly irrational.
The conditions necessary for a crowd to be wise
There are four conditions that characterise wise crowds: diversity of opinion, independence, decentralisation and aggregation. If a group satisfies these conditions, its judgement is likely to be accurate. The reason is a mathematical truism. If you ask a large enough group of diverse, independent people to make a prediction or estimate a probability, and then average those estimates, the errors each of them makes in coming up with an answer will cancel themselves out. Even if the errors cancelled out, it is possible for the group’s judgement to be bad. For the group to be smart, there has to be at least some private information that each person has, even if it is just an eccentric interpretation of the known facts.
Diversity
Why is group diversity important in determining the wisdom of crowds? Group diversity is important in two ways. Diversity adds perspectives that would otherwise be absent, and it weakens some of the destructive characteristics of group decision making. Fostering group diversity is more important in small groups and in formal organisations than in larger unconnected groups because the sheer size of most markets means that a certain level of diversity is guaranteed. On teams or in organisations, cognitive diversity needs to be actively selected because in small groups it is easy for a few biased individuals to exert undue influence and skew the groups collective decision.
Research by political scientist Scott Page has shown that diversity is valuable and by simply making a group diverse makes it better at problem solving. On the group level, intelligence alone is not enough, as intelligence cannot guarantee you different perspectives on a problem. In fact, grouping only smart people together doesn’t work as smart people tend to resemble each other in what they can do. Adding in a few people who know less, but have different skills, actually improves the group’s performance.
This does not mean that if you assemble a group of diverse but uninformed people, their collective wisdom will be smarter than an experts. But if you can assemble a diverse group of people who possess varying degrees of knowledge and insight, you are better off entrusting it with major decisions rather than leaving them in the hands of one or two people. This suggests that the organisation with the smartest people may not be the best organisation. Most companies today are governed by the assumption that a few superstars can make the difference between an excellent and a mediocre company, but in reality, the value of expertise is in many contexts, overrated.
If this is the case, why do we cling to the idea that the right expert will save us? And why do we ignore the fact that simply averaging a groups estimates will produce a very good result? The reason is that we assume averaging means dumbing down or compromising. We also assume that true intelligence resides only in individuals, and that finding the right person, consultant or CEO will make all the difference. In a sense, the crowd is blind to its own wisdom. With most other things, the average is mediocre, but in decision making, the average is often excellence – you could says it is as if we have been programmed to be collectively smart.
Independence
Independence is another important factor in determining if a group is smart. It is important in decision making for two reasons. First, it keeps the mistakes that people make from becoming correlated. Errors in individual judgement won’t wreck the group’s collective judgement as long as the individuals are not dependent on each other for information.
Second, independent individuals are more likely to have new information rather than the same old data everyone is already familiar with. The smartest groups are then made up of people with diverse perspectives who are able to stay independent of each other. Independence is a Western liberal idea that underpins much of economics where it is given that people are self-interested. Independence is important in decision making because the more influence a group’s members exert on each other and the more personal contact they have with each other, the less likely it is that the group’s decisions will be wise ones. The more influence we exert on each other, the more likely it is that we will believe the same things and make the same mistakes. This raises the question: can people make collectively intelligent decisions even when they are in constant interaction with each other?
The idea of Social Proof is the tendency of most people to assume that if lots of people are doing something or believe something, there must be a good reason why. This is different from conformity where people follow others due to peer pressure or fear of being reprimanded. Social proof is when the crowd becomes more influential as it becomes bigger as every additional person is proof that something important is happening. And when things are uncertain, the best thing to do is just to follow along. However, if too many people adopt the strategy of just following the crowd, it stops being sensible and the group stops being smart.
Herding is another concept that hinders independence in groups. It refers to the phenomenon where people will go along with a less risky strategy and fail, rather than take a big risk and follow a strategy they feel is right. Here, sticking with the crowd and failing small is better than trying to innovate and running the risk of failing big. An information cascade is another hindrance to independence in groups. It starts when people’s decisions are not made all at once, but in sequence, so that people are able to see which decision others made before them and can follow. A cascade is not the result of mindless trend-following, or conformity, but that people fall in line because they believe they are learning something important from the example of others. The functional problem with an information cascade is that after a certain point people do stop paying attention to their own knowledge and private information and to start looking instead at imitating the actions of others. Once this happens, the cascade stops becoming informative and instead of making decisions based on what they know, people are making decisions based on what they think the people who came before them know.
In cascades, none of the prerequisites for a good collective decision are true. In effect, a few influential people – who either happened to go first, or who have particular skills or are well connected – determine the course of the cascade. In a cascade peoples decisions are not made independently, but are influenced or determined by those around them. Certain products or problems are more susceptible to cascades than others. The more important a decision, the less likely a cascade is to take hold. The more important a decision, the more likely people are to make the decision on their own, rather than mimicking those who went before them. This is a good thing, since it means that the more important the decision, the more likely it is that the group’s collective verdict will be right. All of the problems associated with cascades are as a result of some people making their decisions before others. If you want to improve an organisations decision making, make sure that decisions are made simultaneously rather than one after the other.
Decentralisation
Decentralisation is the other prerequisite necessary for good group decision making. If a crowd of self-interested, independent people work in a decentralised way on the same problem, instead of trying to direct their efforts from the top down, their collective solution is likely to be better than any other solution you could come up with. Decentralisations great strength is that it encourages independence and specialisation on the one hand, while still allowing people to coordinate their activities and solve difficult problems on the other. Decentralisations great weakness is that there is no guarantee that valuable information which is uncovered in one part of the system will find its way through the rest of the system, making it less useful than it otherwise would be. What you would like is a way for individuals to specialise and to acquire local knowledge – which increases the total amount of information available in the system – while also being able to aggregate that local knowledge and private information into a collective whole.
A decentralised system can only produce genuinely intelligent results if there is a means of aggregating the information of everyone in the system. Without such a means there is no reason to think that decentralisation will produce a smart result. In the case of the free market, that aggregating mechanism is price. The price of a good reflects the actions of buyers and sellers everywhere.
Aggregation
If a group of autonomous individuals tries to solve a problem without any means of putting their judgements together, then the best solution they can hope for is the solution that the smartest person in the group produces. But if that group has a means of aggregating all those different opinions, the group’s collective solution may well be smarter than even the smartest person’s solution. Aggregation is therefore important to the success of decentralisation. There are three kinds of problems – cognition, coordination and cooperation problems – that groups of people are faced with in daily life, and that collective intelligence can help solve.
1. Coordination problems
Coordination problems are ubiquitous in everyday life. They are defined as problems that in order to be solved, a person has to think not only about what he believes the right answer is but about what other people think the right answer is. This is because what each person does effects and depends on what everyone else will do and vice versa. Examples include pedestrians walking on a busy pavement, people meeting in a crowded public place, peoplestanding in a long queue and navigating through a traffic jam.
Coordination problems are very hard to solve and coming up with a good answer to them is a real triumph. When what people want to do depends on what everyone else wants to do, every decision affects every other decision, and there is no outside reference point that can stop the self-reflexive spiral. One way to coordinate people’s actions is via authority or coercion. But in a liberal society, authority has only limited reach over the dealings of private citizens. As a result, many coordination problems require bottom-up solutions, but how can people voluntarily make their actions fit together in an efficient and orderly way?
Research has shown that people can find their way to collectively beneficial results not only without centralised direction, but also without even talking to each other. This is a good thing as conversation is not always possible in large groups. Second, because people’s experiences of the world are often surprisingly similar, this helps make successful coordination easier. The similar reality we all share is cultural in nature and is something in common that we share with one another. Culture also enables coordination in a different way, by establishing norms and conventions that regulate behaviour. Conventions maintain order and stability, but also reduce the amount of cognitive work you have to put in to get through the day. Conventions allow us to deal with certain situations without thinking much about them, and they allow groups of disparate, unconnected people to organise themselves with relative ease and an absence of conflict. Convention has a profound effect on economic life and on the way companies do business.
A flock of birds is a wonderful example of a social organisation that accomplishes its goals and solves problems in a bottom-up fashion without leaders and without having to follow complex algorithms or complicated rules. It seems to have spontaneous order and is biologically programmed spontaneity.
Without anyone leading them or directing them, people, most of them not especially rational or farsighted, are also able to coordinate their economic activities. The free market is a mechanism designed to solve the complex coordination problem of getting resources to the right place at the right cost. If the market is working well, products and services go from the people who can produce them most cheaply to the people who want them most fervently.
What is amazing is that the market demonstrates that people who can be naive, unsophisticated agents can coordinate themselves to achieve complex, mutually beneficial ends, even if they are not really sure at the start what those ends are or what it will take to accomplish them. As individuals, they don’t know where they are going, but as part of the market, they are suddenly able to get there fast.
We can see the promise and perils of different ways of coordinating a business in three business models. The first is the top-down hierarchy of the traditional corporation. This allows the man at the top to make decisions quickly and to have them carried out decisively. It allows for long-term investment and planning. But the downside to this corporate structure is that it is difficult for the man on top to get the information he needs to make wise decisions and it becomes harder to stay on top of everything. The CEO becomes more and more isolated from points of view other than his own.
A very different model is the small, tight-knit professional coherent group that has the advantage of trust, specialisation and mutual awareness of each member’s abilities. In small groups people are less likely to slack off than in large organisations, and since the rewards for their efforts are often directly connected to their effort, there is a powerful incentive for each member to contribute. But being a small group limits the organisations possibilities due to their limited resources, and because their rewards depend on their own effort, there is no room for error in what they do.
The third model is a group of individuals that comes together for a single job and then disperses. This model allows people to be handpicked for their diverse abilities so that the group can have exactly what it needs for the job, and the once-off nature of the project ensures that everyone has an incentive to perform well. This model however, has high transaction costs as it takes a lot of work to put the team together, keep them on track, and check up on them etc.
This suggests that no business model is an ideal solution. When trying to consciously organise individuals toward a common goal, you face inevitable trade-offs. That is why today’s companies are trying to blend the three models of business into one. Companies want to retain the structure and institutional coherence of the traditional corporation, they want tightly knit groups to do much of the work at the day-to-day level, and they want to be able to have access to workers and thinkers from outside the corporation as well.
The only reason to organise thousands of people to work in a company is that together they can be more productive and more intelligent than they would be apart. But in order to do this, individuals need to work as hard to get and act upon good information as they would if they were a small businessman competing in the marketplace. In too many organisations today the incentive system is skewed against dissent and independent analysis. Bosses don’t want to be told the cold hard truth about what is going on. Research has shown that the more anxious a young executive is about moving up the job ladder, the less accurately they communicate problem-related information. There is not much evidence to suggest that the flow of information up the traditional corporate hierarchy has improved much over time either. One of the things that gets in the way of the exchange of real information in traditional organisations is a deep-rooted hostility on the part of bosses to opposition from subordinates. This is the real cost of top-down approach to decision making and encourages everyone to just play along with what the top-man decides.
Compounding this problem is the fact that managerial pay is often based not on how one performs, but on how one performs relative to expectations. Companies pay people based on whether they do what they are expected to do. In a market, people are paid based simply on what they do. In this way, top-down corporations give people an incentive to hide information and dissemble. In a market on the other hand, businesses have an incentive to uncover valuable information and act on it – making the information public. This is what corporations should be looking for if they want to cultivate collective wisdom – ways to provide their employees with the incentive to uncover and act on private information.
So what would the wider distribution of real decision-making power look like? Decisions about local problems should be made as much as possible, by people close to the problem. Instead of filtering all problems up the hierarchy and every solution back down again, companies should start with the assumption that people with local knowledge are often best positioned to come up with a workable and efficient solution. The more responsibility people have for their own environments, the more engaged they will be. Decentralisation also makes coordination easier. Instead of having to constantly resort to orders and threats, companies can rely on workers to find new, more efficient ways of getting things done. This reduces the need for supervision, cuts transaction costs, and allows managers to concentrate on other things.
2. Cooperation problems
Cooperation problems often look like coordination problems because in both cases a good solution requires people to take what everyone else is doing into account. But if the mechanism is right, coordination problems can be solved even if each individual is singlemindedly pursuing his self interest, but to solve cooperation problems, the members of the group or society have to do more. They need to adopt a broader definition of self-interest and they need to be able to trust those around them in order to achieve a common good.
Examples include paying taxes, tipping waiters, financial transactions with strangers across the globe and collaboration between scientific researchers all over the world to find a cure for disease. Societies and organisations work only if people cooperate. It is impossible for a society to rely on law alone to make sure citizens act honestly and responsibly. And it’s impossible for any organisation to rely on contracts alone to make sure that its managers and workers live up to
their obligations. So cooperation typically makes everyone better off. A sense of trust in the market system could not exist without the institutional and legal framework that underpins every modern capitalist economy. All you trust is that the other person will recognise his self interest and over time that reliance on his attention to his self-interest becomes a general sense of reliability, a willingness to cooperate because cooperation is the best way to get things done.
The social benefits of trust and cooperation are relatively unquestionable. But they do create a problem – the more people trust, the easier they are for others to exploit. And if trust is the most valuable social product of market interactions, corruption is the most damaging. Taxpaying is a prime example of cooperation in society. Paying taxes is individually costly, but collectively beneficial due to the services the government provides with the tax money. But the collective benefits only materialise if everyone takes part. The goods that government provides are non-excludable – meaning that it’s not possible to allow some people to enjoy them while excluding others. Since you get the goods whether or not you personally pay for them, it is rational for you to free ride, but when many people free ride, the public good disappears.
So why do people pay their taxes if there is a big incentive to free ride? Most people will participate as long as they believe that everyone else is participating too. When people start to feel that most people are not paying their taxes and getting away with it, they start to feel like they are being taken advantage of. People want to do the right thing, but they don’t want to be the sucker. The key to cooperation in tax paying is to convince the conditional consenters to contribute by making sure the free riders are being punished.
3. Cognition Problems
Even as companies acknowledge the potential benefits of decentralisation, they are not using bottom-up methods to solve cognition problems. These are problems that define corporate strategy and tactics. They include everything from deciding among potential new products; to building new factories; to forecasting demand and setting prices. Today, most corporations have these big decisions made by one man: the CEO. What is perplexing about this is how little evidence there is that single individuals can consistently make superior forecasts or strategic decisions in the face of genuine uncertainty.
No decision making system is going to guarantee corporate success. The strategic decisions that corporations have to make are very complex. But we know that the more power you give a single individual in the face of complexity and uncertainty, the more likely it is that bad decisions will be made. As a result, there are good reasons for companies to try to think past hierarchy as a solution to cognition problems. In practice, this would mean the flow of information within the organisation shouldn’t be dictated by management charts. Specifically, companies can use methods of aggregating collective wisdom – like using internal decision markets – when trying to come up with reasonable forecasts of the future and when trying to evaluate the probability of possible strategies.
A decision market is an elegant and well-designed method for capturing collective wisdom. The specific method does not matter very much, so long as it satisfies the conditions of diversity, independence and decentralisation, the group decision will be smart. So if we know that the aggregate answer of a diverse, independent and decentralised group is so smart, why don’t we use this knowledge? Decision markets could be a relatively simple and quick means of transforming many diverse opinions into a single collective judgement and have the potential to improve dramatically the way organisations make decisions and think about the future. Corporate strategy today is all about collecting information from many different sources, evaluating the probabilities of potential outcomes and making decisions in the face of an uncertain future. These are tasks for which decision markets are tailor-made, and yet companies have remained indifferent to this source of potentially excellent information, and have been unwilling to improve their decision making by tapping into the collective wisdom of their employees.
Decision markets are well suited to companies because they circumvent the problems that obstruct the flow of information at many firms. Major corporate decisions should be informed by decision markets, not made by them. But when the decisions are made, it makes little sense, given everything we know about the virtues of collective decision making and the importance of diversity, to concentrate power in the hands of one person. In fact, the more important the decision, the more important it is that it not be left in the hands of a single person.
Small Groups
Small groups have their own problems in everyday life. They are ubiquitous and their decisions are consequential. Small groups are different in important ways from large, diverse, unconnected groups which are statistical realities. Even if a small group is formed for the sake of a single project, it has an identity of its own and the influence of the people in the group on each other’s judgement is inescapable. On the one hand this means small groups can make very bad decisions because influence is more direct and immediate and small-group judgements tend to be more volatile and extreme. On the other hand, it also means that small groups have the opportunity to be more than just the sum of their parts. A successful face-to-face group is more than just collectively intelligent. It makes everyone work harder, think smarter and reach better conclusions than they would have on their own. One of the real dangers that small groups face is emphasising consensus over dissent. In extreme cases the members of the group become so identified with the group that the possibility of dissent seems practically unthinkable. But in a more subtle way small groups can exacerbate our tendency to prefer the illusion of certainty to the reality of doubt. One of the consistent findings from decades of small-group research is that group deliberations are more successful when they have a clear agenda and when leaders take an active role in making sure that everyone gets a chance to speak.
Another recurring problem with small groups is that they start with a conclusion and fit new information into the conclusion they have come up with. New messages are often modified so that they fit old messages, or they are modified to suit a pre-existing picture of the situation. This is especially dangerous since unusual messages often add the most value to group discussions.
In small groups, diversity of opinion is the single best guarantee that the group will reap benefits from face-to-face discussion. Research has shown that the presence of a minority viewpoint makes a group decision more nuanced and the decision-making process more rigorous – even when the minority viewpoint turns out to be ill conceived. The confrontation with a dissenting view forces the majority to interrogate their own positions more seriously. The order in which people speak in a small group also has a profound effect on the course of a discussion. Earlier comments are more influential and tend to provide a framework within which the discussion occurs. Once that framework is in place, it is difficult for a dissenter to break it down.
In small groups ideas often do not succeed simply on their own merits. Even when its virtues might seem self-evident, an idea needs a champion in order to be adopted by the group as a whole. That’s why a popular position tends to become more popular in the course of deliberations as it usually has more potential champions to begin with. In a small group, having a strong advocate for an idea, no matter how good it is, is essential. When advocates are chosen on the basis of status or talkativeness rather than perceptiveness or keenness of insight, the groups chance of making a smart decision shrinks.
Talkativeness may seem like a curious think to worry about, but in fact talkativeness has a major impact on the kinds of decisions small groups reach. If you talk a lot in a group, people will tend to think of you as influential by default. This might be okay if people only spoke when they had expertise in a particular matter, but in reality there is no clear correlation between talkativeness and expertise.
Given all these challenges with small group decision making, the temptation is to do away with or minimise the role that small groups play in shaping policy or making decisions. But it would be a mistake to do this. Evidence does demonstrate that non-polarised groups consistently make better decisions and come up with better answers than most of their members and often outperform their best member. In fact, group decision making can be both faster and more accurate than the best individual member.
What is important to remember in a small group setting is that deliberations can be valuable when done well. Secondly, there is no point in making small groups part of a leadership structure if there is no method of aggregating the opinions of its members. If small groups are included in the decision-making process, then they should be allowed to make decisions. If an organisation sets up teams and then uses them for purely advisory purposes, it loses the true advantage that a team has which is its collective wisdom.
The solutions to cooperation, coordination and cognition problems by collective wisdom are at work in real situations in society every day. These solutions are not imposed from above, but emerge from the crowd and on the whole are better solutions than any group of platonic guardians could come up with. The question now is: is your organisations tapping into collective wisdom and aggregating the opinions of diverse, independent, decentralised groups within the organisation in order to enhance decision making?
Join the Conversation