Beyond The Experiment

Beyond The Experiment#

The Role for Causal Inference with Observational Data in Industry

I don’t think there’s anyone who would question the importance of A/B testing in the tech sector today. It is used constantly to help companies refine their products, better target their ads, and incrementally innovate.

But while A/B testing is definitely a skill a good data scientist should have in their toolbox, it’s important to understand that it is not the be-all and end-all of causal inference in industry. In this reading, we’ll discuss why it’s important to learn how to think critically about causality when working with observational data (“observational data” is data that we gather by passively observing the world – or that somebody else collected by passively observing the world – rather than data that we get by directly manipulating subjects in an experiment).

A/B Testing Isn’t Always Feasible#

The first reason it’s important to know about tools that go beyond A/B testing is that A/B testing isn’t always feasible. A/B testing only works when you can efficiently randomize people into different treatment arms, and where you can do it at low cost (if running a test is expensive, then even if you want to run the test, you should be doing some analysis first to make sure it’s worth while!). But this isn’t possible for every decision, even in big tech companies. Here’s just a handful of situations where you can’t run an A/B test:

  • You Don’t Control The Relevant User Behavior: Suppose you want to know the effect of, say, following a specific person on facebook or twitter on user satisfaction, or choosing to use a certain function in your product. The “ideal experiment” would be to randomly assign some users to follow the person in question / use your new function, and block others from doing so. But of course you can’t do that! So you have to find other ways to compare followers and non-followers that accounts for ways they may be different in terms of user satisfaction or behavior.

  • It’s a One-Shot Deal: Suppose you’re trying to decide whether to run a Super Bowl ad, or open a second store. You can’t A/B test these small-N events. When dealing with these types of “big strategic” decisions, the best you can do is look to your companies past experiences / the experiences of other companies that you think are comparable using the best available tools for observational causal inference.

  • Randomization is Bad Business: It’s one thing to randomize someone’s landing page color scheme; it’s another thing entirely to randomly show people different prices for your products – customers wouldn’t stand for it! Similarly, sometimes you want a big rollout – like the launch of a new title on Netflix, or the premier of a new movie – and an A/B test would interfere with that. (Not an abstract example: here’s Netflix talking about this constraint).

  • Time Horizons are Too Long: A/B testing is popular in tech because you get so much data from users so quickly that they’re easy to run on short time horizons. But if you’re thinking about opening more stores across the country, you can’t put your business on hold, pick a few locations, setup new shops, and wait for sales data; you need to get a sense of what’s going to happen now. As we read in Kohavi, Tang and Xu, A/B tests work best when we have an Overall Evaluation Criterion (OEC) that is “measurable in the short term (the duration of an experiment) yet believed to causally drive long-term strategic objectives.” But when that is not available / the shortest experiment available that would allow you to measure a meaningful OEC would take too long, you need other options!

Going Beyond Testing for the Future#

Another limitation of A/B tests is that we usually use them to do small scale tests to motivate larger, future rollouts. But sometimes we want to just measure the effect of a single, large intervention.

In many areas of business, it’s increasingly common to hire outside companies to do very complicated projects, like administering government programs or providing a service to employees. In these situations, companies have increasingly found that paying vendors based on inputs (hours worked, money spent) doesn’t actually encourage vendors to do a very good job, or to try to be efficient – if you’re being paid on the basis of what you spend, you will obviously you will not have an incentive to cut costs!

In these situations, companies are increasingly turning to something called “at-risk contracting.” In these arrangements, the company doing the hiring offers to pay the vendor if they achieve a specific outcome. For example, a hospital might hire a company to reduce in-hospital infections and pay them based on reductions in infections, or a company may hire a vendor to minimize the downtime of their internal network, and pay them based on downtime. By tying compensation to the outcome that the hiring party actually cares about, the vendors well incentivized to do precisely what the hiring company wants them to do!

But in these arrangements, it’s critical to both parties that there’s a good way to measure the effect of the vendor’s efforts or someone will get over or under paid. That’s not an average effect, that’s a specific effect, and something for which experiments are not well-suited.

In these circumstances, it’s unfortunately the case that people often just measure outcomes before the vendor starts and compare them to outcomes when the vendor finishes. But if the outcome in question is subject to natural variation – such as seasonality – then we expect the outcome to vary whether the vendor does anything or not, and so you need a better causal design (e.g. difference-in-difference) to take those types of confounders into account.

Don’t want to be limited#

Here’s another important thought: professionally, if you only know how to do A/B testing, you risk limiting your opportunities for upward advancement. Big companies are getting better and better at providing analysts with extremely user friendly A/B testing tools, which means doing A/B tests is getting easier and easier. Twitter, for example, has an internal tool called Duck, Duck, Goose that makes running an A/B test a relatively low-difficulty task. And while running a good A/B test is never easy (as we have discussed at length, even with A/B testing, the internal and external validity of our causal estimates will always be dependent on a wide range of assumptions), it is something that more and more people are learning to do well and which companies are learning to make easier. Thus, if all you know how to do are A/B tests, you may find it increasingly hard to differentiate yourself in the data science market.

Again, this isn’t to say you shouldn’t learn to do A/B testing, just don’t only learn A/B testing.

Not enough to be able to do it; you must be able to explain it#

One final reason it’s important to understand how to apply the logic of causal inference to observational data is that even if you only work with A/B data, it is almost inevitable that you will interact with other people in your company who want to use observational data. In these situations, it’s not enough that you understand the considerations that go into making causal inferences from observational data; you also need to be able to explain those to colleagues.

Contents