Want Your Company to Get Better at Experimentation?

Learn fast by democratizing testing. by Iavor Bojinov, David Holtz, Ramesh Johari, Sven Schmit and Martin Tingley

From the Magazine (January–February 2025)

Jamie Chung/Trunk Archive

Summary.

For years, online experimentation has fueled the innovations of leading tech companies, enabling them to rapidly test and refine new ideas, optimize product features, personalize user experiences, and maintain a competitive edge. The widespread availability and lower cost of experimentation tools today mean that most organizations—even outside the technology sector—conduct tests. After initial adoption, however, many of them restrict experimentation to just a handful of carefully selected projects. That’s because their data scientists are the only ones who can design, run, and analyze tests. Vastly increasing the capacity to conduct online experiments is becoming more critical as the expanding capabilities and applications of artificial intelligence—particularly generative AI—reshape innovation. Scaling up experimentation entails moving away from a data-scientist-centric approach to one that empowers everyone else on the product, marketing, and sales teams to run experiments. The authors suggest how to do that.

Leer en español Ler em português

For years online experimentation has fueled the innovations of leading tech companies such as Amazon, Alphabet, Meta, Microsoft, and Netflix, enabling them to rapidly test and refine new ideas, optimize product features, personalize user experiences, and maintain a competitive edge. Owing to the widespread availability and lower cost of experimentation tools today, most organizations—even those outside the technology sector—conduct online experiments.

Read more on Experimentation or related topics Open innovation, Technology and analytics, Information management, Analytics and data science, Organizational change, Performance indicators and AI and machine learning

A version of this article appeared in the January–February 2025 issue of Harvard Business Review.

Iavor Bojinov is an assistant professor of business administration and the Richard Hodgson Fellow at Harvard Business School. He is also a faculty affiliate of Harvard’s statistics department and the Harvard Data Science Initiative.

David Holtz is an assistant professor in the management of organizations and entrepreneurship and innovation groups at the University of California, Berkeley’s Haas School of Business and a research affiliate at the MIT Initiative on the Digital Economy.

Ramesh Johari is a professor of management science and engineering at Stanford University and an associate director at Stanford Data Science.

Sven Schmit is the head of statistics engineering at Eppo, an experimentation platform vendor.

Martin Tingley is the head of the analysis team on the Netflix experimentation platform.

Want Your Company to Get Better at Experimentation?

Summary.

Partner Center