The Power of Experiments

1. Experiments Replace Intuition with Evidence-Based Decisions

Experiments help to complement intuition and guesswork with evidence-based decision making.

From Gut Feelings to Data. In a world often driven by hunches and assumptions, experiments offer a structured way to test ideas and validate their impact. By replacing reliance on intuition with concrete data, organizations can make more informed decisions, reducing the risk of costly mistakes and increasing the likelihood of success.

• The UK tax authority's A/B testing of reminder letters, which led to millions in recovered revenue, exemplifies this shift.
• eBay's $50 million advertising mistake, uncovered through experimentation, further underscores the importance of data-driven decision-making.

The Scientific Method in Action. Experiments bring the rigor of the scientific method to real-world problems. By formulating hypotheses, designing controlled trials, and analyzing results, organizations can gain a deeper understanding of cause-and-effect relationships. This approach allows for continuous improvement and adaptation, ensuring that strategies are based on evidence rather than guesswork.

Humility in the Face of Data. One of the most valuable lessons from the rise of experiments is the importance of humility. Even the most experienced leaders can be wrong about what works. Experiments provide a humbling reminder that our intuition is often flawed and that data should guide our decisions.

2. Experiments Thrive Where Data and Randomization Meet

Historically, experimental methods may have been alien to the managerial toolkit.

Data as the Foundation. Experiments require data to measure their impact. The digital age has made it easier to track outcomes, but choosing the right metrics remains a challenge. Organizations need to identify relevant outcome data and develop a clear map between the data they have and the data they care most about.

• A vacuum manufacturer's use of Amazon reviews to evaluate customer satisfaction demonstrates the value of leveraging available data.
• The ability to track user behavior on online platforms like Google and Facebook has fueled the rise of experimentation in the tech sector.

Randomization: The Key to Causality. Random assignment is crucial for establishing causality. Without diligent randomization, it's hard to know whether a factor of interest is actually causing the observed effect. Online platforms make it easier to vary what individual customers see, simplifying the randomization process.

• The ability to randomize which users see ads on Google, as opposed to a billboard on Route 66, highlights the importance of randomization in advertising experiments.
• Medical research relies on random assignment to ensure that treatment effects are not confounded by selection bias.

Overcoming Barriers. The availability of large subject pools and the ease of randomization have contributed to the adoption of experimental methods in tech and government. Organizations that can overcome these barriers are well-positioned to benefit from the power of experiments.

3. Ethical Considerations are Paramount in Experiment Design

Some corporate experiments may risk being good for companies but bad for their customers.

Balancing Profit and Ethics. Experiments can raise ethical concerns, especially when they risk being good for companies but bad for their customers. It's crucial to consider the potential stress and harm that an experiment could impose on participants.

• StubHub's experiment on shrouded fees, which led to increased revenue but potentially exploited customers, highlights the ethical dilemmas that can arise.
• Facebook's emotion experiment, which manipulated users' News Feeds, sparked public outrage and raised questions about informed consent.

Transparency and Consent. Organizations need to be transparent about their experimentation practices and obtain informed consent from participants whenever possible. This builds trust and ensures that experiments are conducted in a responsible manner.

• Facebook's updated terms of service, which acknowledge the company's right to use personal information for testing and research, represent a step toward greater transparency.
• The creation of Institutional Review Boards (IRBs) in the 1970s reflects a broader effort to protect human test subjects from harm.

Long-Term Consequences. It's important to consider the long-term consequences of experiments, not just the short-term gains. Experiments that prioritize short-term profit at the expense of customer satisfaction or brand reputation can backfire in the long run.

4. Experiments Test Theories, Magnitudes, and Policies

Roth noted that similar experiments in different field settings have generated effects in opposite directions.

Testing Theories and Mechanisms. Experiments can help to confirm or disconfirm theories and shed light on the mechanisms driving observed patterns. By understanding the underlying causes of behavior, organizations can develop more effective interventions.

• The Airbnb experiment, which revealed that discrimination was concentrated among hosts who had never had a black guest, challenged the theory that discrimination was based on prior experience.
• Experiments in psychology, such as those conducted by Kahneman and Tversky, have identified cognitive biases that influence decision-making.

Understanding Magnitudes and Tradeoffs. Experiments can help to quantify the magnitude of effects and identify tradeoffs. This allows organizations to make informed decisions about which interventions to implement.

• The eBay advertising experiment, which showed that brand advertising was largely ineffective, quantified the wasted ad spending and allowed the company to reallocate resources.
• The teacher incentive program experiment, which found that loss-framed contracts were more effective than gain-framed contracts, quantified the impact of framing on teacher performance.

Evaluating Policies. Experiments can be used to evaluate the effectiveness of policies and programs. This allows organizations to make evidence-based decisions about which policies to adopt or modify.

• The organ donation experiment, which showed that active choice was ineffective, challenged the conventional wisdom and informed policy debates.
• The UK tax letter experiment, which tested different messaging strategies, led to the collection of millions of pounds in additional tax revenue.

5. Behavioral Insights Nudge Towards Better Choices

The pair urge policymakers to be thoughtful about their role as choice architect and to try to “nudge” people in a direction that would leave society better off.

Choice Architecture and Nudges. Behavioral insights can be used to design choice architectures that "nudge" people toward better decisions. By understanding cognitive biases and decision-making heuristics, organizations can create environments that promote positive behaviors.

• Thaler and Sunstein's concept of choice architecture highlights the importance of defaults, framing, and other design elements in shaping decisions.
• The use of opt-out defaults for organ donation registries, which capitalizes on the status quo bias, is a classic example of a nudge.

Dual-Systems Model. The dual-systems model of the mind, which distinguishes between intuitive (System 1) and deliberative (System 2) thinking, provides a framework for understanding how nudges work. By designing choice architectures that engage System 2 thinking, organizations can help people make more rational decisions.

Context and Design Matter. The effectiveness of nudges depends on the context and design. Small changes in how a nudge is implemented can have a big impact.

• The finding that the impact of a nudge telling people that they use less energy than their neighbors depended on whether or not there was a smiley face next to the message highlights the importance of design choices.
• The unintended consequence of defaulting people into 401k plans, which led to increased debt, underscores the need to consider both intended and unintended consequences.

6. Context Matters: Tailor Experiments to Specific Settings

The impact of an intervention on people’s decisions will vary depending on the context.

Generalizability Challenges. The magnitude of an effect in any given field setting is likely to be different from its magnitude in other field settings. This implies that it can be hard to figure out how to translate findings between settings.

• Emphasizing a company's mission, rather than pay, in hiring advertisements will likely affect the number of applications—but might increase or decrease applications, depending on myriad factors.
• The impact of a nudge on people's decisions will vary depending on the context, such as cultural differences.

The Importance of Tailoring. Organizations need to experiment to refine ideas from research to put them into practice in the area they are looking to change, taking into account their specific goals. Academic research and other existing evidence can help to build general frameworks, but running their own experiments can help organizations gain an advantage by expanding these frameworks.

No One-Size-Fits-All Solution. Asking whether a nudge works is akin to asking whether advertising works. The answer for advertising is yes, for some businesses, in some situations—but in other situations, it can be ineffective or even annoying to customers. The impact of nudges is similarly nuanced.

7. Transparency Builds Trust and Improves Outcomes

Transparency would be an important next step for Airbnb.

Demystifying Experimentation. Experiments can feel complex, invasive, and Big Brotherish, especially when we realize we've been totally oblivious to our regular participation in them. Transparency can help to demystify the experimental method and alleviate these concerns.

Building Trust. By being open about their experimentation practices, organizations can build trust with customers, employees, and other stakeholders. This can lead to greater acceptance of experiments and a willingness to participate.

• Facebook's decision to stop publicizing the results of its experiments after the emotion study backlash highlights the dangers of secrecy.
• Airbnb's lack of transparency about the results of its experiments on discrimination has fueled criticism and suspicion.

Improving Outcomes. Transparency can also improve the outcomes of experiments. By soliciting feedback from participants and sharing results with the public, organizations can gain valuable insights and refine their interventions.

8. Experiments Drive Innovation and Competitive Advantage

When testing and experimentation are at the heart of a culture, a team or company can be more responsive and nimble, driving a better product and more growth alongside a better employee experience.

Rapid Iteration and Adaptation. Experiments allow organizations to rapidly iterate and adapt their products, services, and processes. By testing new ideas in close to real time, they can quickly identify what works and what doesn't.

• Google's ability to run thousands of experiments per year has fueled its innovation and competitive advantage.
• Booking.com's experimental infrastructure allows product managers to easily test new features before rolling them out to all customers.

Data-Driven Innovation. Experiments provide a data-driven approach to innovation. By relying on evidence rather than intuition, organizations can make more informed decisions about which ideas to pursue.

• Microsoft's Bing search engine, which ran an experiment to vary the physical size of ads on the screen, generated an additional $50 million per year in profits.
• Amazon found that moving credit card offers from the homepage to the shopping cart page increased profits by millions of dollars.

Competitive Advantage. Organizations that embrace experimentation are better positioned to compete in today's rapidly changing world. By continuously testing and refining their strategies, they can stay ahead of the curve and gain a competitive edge.

9. Market-Level Experiments Capture System-Wide Effects

It’s too important to leave to politics, even academic politics.

Spillover Effects. When evaluating the impact of a new product or policy, it's important to consider spillover effects. Changes in one area can affect other areas, potentially biasing the results of the experiment.

• Uber's market-level experiments, which rolled out Uber Express Pool in a randomly selected subset of markets, allowed the company to account for the effect of the new product on existing Uber products.
• The Campbell's Soup example, in which the company was worried about whether its new semi-condensed Soup for One would crowd out its other soups, highlights the importance of considering spillover effects.

System-Wide Perspective. Market-level experiments provide a system-wide perspective. By looking at the entire market, organizations can gain a more complete understanding of the impact of their interventions.

• Uber's experiments allowed the company to understand how the rollout of Express Pool was influencing Uber usage across its entire product portfolio.
• The ability to capture system-wide effects makes market-level experiments particularly valuable for large organizations with complex ecosystems.

Challenges and Limitations. Market-level experiments can be challenging to run and may limit the total number of experiments that an organization can conduct. However, they provide the best idea of how markets will evolve in the wake of a product change or launch.

10. Experiments are a Leadership Imperative

Leaders who create an environment for taking calculated risks get better results.

Setting the Tone. Leaders play a crucial role in creating a culture of experimentation. By embracing experimentation and encouraging others to do the same, they can foster a mindset of continuous learning and improvement.

• David Halpern's leadership in creating the Behavioural Insights Team and Hal Varian's leadership in pushing for experimentation within Google demonstrate the importance of leadership in driving a culture of experimentation.
• Leaders who are barriers to experimentation are also barriers to finding more effective ways for their organization to operate.

Embracing Uncertainty. Effective leaders have the humility and confidence to know what they don't know and to use experiments as part of their toolkit for answering tough questions. They are willing to take calculated risks and learn from both successes and failures.

Promoting Transparency. Leaders can promote transparency by publishing clear guidelines about when and how users will be informed about any experiments they are taking part in. This builds trust and ensures that experiments are conducted in an ethical manner.

Last updated: March 16, 2025