What Solving the Wrong Problem Looks Like#
Recruitment#
Outliers#
Pizza Ads#
Our discussion up to this point has been a little abstract, so to illustrate what it means to “mis-specifying a problem.” The details of this example are fictitious, but the underlying logic of this example is not; indeed, the insight illustrated by this example is central to one of the biggest pivots in how people think about online advertising.
You have been hired by the advertising division of a fictitious national pizza chain—let’s call it Little Papa Dominos (LPD). LPD spends a lot on online advertising, but their resources aren’t being deployed as effectively as they could be. They spend more than most of their competitors, and yet their online sales are lagging.
After consulting industry groups and online advertising experts, they discovered that the rate at which people click their ads (their ads’ click-through rate, or CTR) is well below the industry average.
To address the problem, they’ve hired you—a newly minted Data Scientist—to improve the CTR of their ads. They give you a large budget, access to all the cloud computing resources you need, and even a small staff.
“Well,” you reason, “maybe the problem is that our ads aren’t being shown to the right people. After all, it seems unlikely that any ad for pizza—no matter how appealing—is likely to draw a click if it’s shown to a 75-year-old at 7 am.” So you set out to build a statistical model to answer the question, “given a user’s demographics and online behavior, how likely are they to click on one of LPDs ads?” If you can answer that, you figure, LPD can prioritize buying the ad spots for the types of users most likely to click on their ads.
To develop that model, you use your budget to run your ads on different sites and at different times. You then use that data (and those glorious cloud computing resources) to train a machine learning model that predicts whether someone will click on one of your ad based on the user’s demographics and ad placement. You try out a few different models, tune the model parameters, and eventually settle on a neural network model with extremely high precision and recall. Hooray!
LPD uses the model to target users likely to click their ads, and almost immediately the CTR of their ads increases 5-fold! Not only that, but the share of people who click on ads that go on to buy a pizza has also increased. Everyone congratulates you, and you move on to the next project feeling very smug.
A few months later, though, you are called into a meeting with the LPD advertising team and the company’s Chief Financial Officer. They’ve been looking over the numbers, and despite the huge rise in CTR, they seem to be getting fewer online orders than before you arrived. CTR rates are up, but somehow it isn’t generating greater profits.
Can you figure out what went wrong?
OK, this is the place in most books where the authors ask you that question, and you look up at the ceiling for a minute, shrug, and then read on.
But I’m really, really serious about this: close your laptop, stand up, set a 5-minute timer on your phone, and go for a walk. Ponder this example. See if you can figure out what’s going on. This is precisely the kind of problem you will soon face as a professional data scientist, so why not practice trying to think through the problem?
Solving The Wrong Problem#
So what happened?
The reason an increased click rate wasn’t making LPD richer is that LPD’s problem was never the fact they had a low CTR; LPD’s real problem was that they weren’t getting a lot of orders online. And because Little Papa Domino’s problem wasn’t a low CTR, being able to answer the question “How likely is a given user to click on an ad” didn’t actually solve their real problem.
What question, if answered, would have helped solve their problem? “Given a user’s demographics and online behavior, how much more likely are they to buy a pizza from LPD if we show them an ad?”
Or, expressed more succinctly, LPD thought their problem was that their ads weren’t getting clicks, but really their problem was that their ads weren’t driving increased sales.
The difference is subtle, but critically important: someone clicking an ad doesn’t make Little Papa Dominos any money. Someone clicking an ad and ordering a pizza doesn’t necessarily make LPD any money. Why? Because they may be someone who would have bought a pizza from LPD anyway, whether you showed them an ad or not. The person who was already thinking of ordering a pizza from LPD is precisely the type of person your algorithm may have targeted, and who may have clicked the ad to save themselves a Google search!
But the person LPD wants to show an ad to isn’t the person who was already thinking of ordering a pizza from LPD, it’s the person who was thinking of a pizza but wasn’t sure who to order it from, or the person who wanted dinner but didn’t know what to get. They may be less likely to click the ad than the person who was about to Google “Little Papa Dominos,” but their precisely the type of user who is more likely to buy a pizza from LPD as a result of seeing an ad than they would have been otherwise.
Counter-Factual Advertising#
Lest you think this example is contrived, it’s not. The realization that the goal of ads isn’t to maximize clicks but rather to induce the largest possible change in purchasing behavior is one of the most important ideas in online advertising. It has had a huge impact on how online advertising works, and how people evaluate the success of ad campaigns.
Indeed, this is why companies like Meta and Google are so eager to track user behavior across apps and websites. When Meta and Google can “follow” users after they’ve clicked an ad, they can evaluate ad performance based not on clicks but on customer behavior. When paired with their ability to show ads to some users and not to others and track both groups as they move around the web, Meta and Google can see whether users who see the ads are more likely to make purchases than those who don’t. This allows them to estimate the true effect of ads on sales, data they use to improve ad targeting and justify higher prices to advertisers.
Next Up: Types of Questions#
Having established the importance of first articulating the problem one seeks to solve, we will shortly turn to developing our understanding of the three types of questions introduced in the first chapter of this book.
First, though, a quick digression into understanding the historical context of data science. This may feel like an odd topic to talk through in a technical data science text, but as we’ll see understanding how we got to where we are today is key to successfully navigating modern data science.