Bridging Data Science and Business Intuition

Target corporation’s massively profitable data science project threw them into the news spotlight a few years back. Their story makes for a valuable case study in the interplay between business intuition and data science.

After painstakingly developing a ‘golden-goose’ analytic marketing model, Target began mailing brochures for baby clothing to certain women who had done nothing to directly indicate to Target that they were pregnant.  Target knew that their retail spending patterns were about to change big time, and they wanted in on the action. The amazing accuracy of these mailings soon came to light when a man famously intercepted Target’s mailing to his teenaged daughter.

Target had realized the teenager was pregnant before her own father had.

Target got a lot of bad press for that little trick, but we in the analytics world were sufficiently impressed.  Target was also writing a chapter in the playbook for how to bridge data science research and business intuition.

It started with intuitive insight

It’s crucial to realize what kicked off the data science effort in this case.  It started with intuitive insight: The power of habit. Target realized that the key to winning customers from competitors such as Walmart (the market leader) was to reach out to them at key habit-forming moments.

Starting from this piece of business intuition, they made a goal of identifying customers who were approaching life-changing, habit-forming moments. Target could then focus marketing efforts with the goal of securing these people as loyal customers for years to come.  The most important such habit-forming moment was the birth of a child. But how could they identify the expecting mothers and market to them at precisely the right times during the pregnancy?

Enter the data scientists

Target’s data scientists were then given a clear business goal: use your data and models to tell us which shoppers are pregnant. Do this, and we’ll secure loyal customers who will spend a lot of money over the next few years. This was a few years before Big Data systems appeared, so Target analysts had only traditional data sources at their disposal: point of sale, birth registry, demographic repositories, etc.  What the analysts did have, however, was this clear purpose and well-defined goals; goals that were rooted in business intuition.

It was this business intuition that started the team of analysts working for several months on analytic models that would eventually be able to accurately single out the pregnant shoppers, even identifying their due dates to within a small window

(Aside: this was eventually accomplished by focusing on purchases of 25 specific but otherwise innocuous items which had high signaling potential when purchased at certain intervals)

Target made a huge profit from their analytics. The executives had an intuition for strategic growth, and the analysts made it real by collecting data and building accurate models, an effort spanning several months.  The result, although questionable in terms of privacy, resulted in significant financial gains that extended over several years.

Data Beats Opinion

But business intuition is sometimes completely and utterly wrong.

Recall Tesco’s seasoned chairman, Ian MacLaurin, who famously told his data consultants,

‘You know more about my customers in three months than I know in 30 years’
and whose management would later say that data “has guided almost all of the key business decisions the management team have made in recent times, reducing the risk of taking bold new initiatives… what we have to do becomes as plain as the nose on your face.” 2

Or think of the Chicago theatre executives whose misconceptions of theatre guest personas were shattered when an external consultant used customer data to show that guests personas were completely different from what the executives had assumed, despite their many years of industry experience.3

A Cycle of Continuous Improvement

Business intuition and data science should together walk through a series of insight-analysis-value cycles.

To illustrate, some time back I was working with a company that had been thrown into crisis by …. you guessed it…  a sudden drop in revenue.  Sitting in a war room with the company’s senior leaders, one after the other attributed the revenue drop to various intuitive, albeit speculative causes.  We only identified the true cause of the drop after several days of digging into the data. Once the analytics put the company on the right scent, one of the executives most familiar with business trends realized that the same situation would re-occur within 6-9 months.  We then refocused our analysis on that future period to forecast damage and to identify potential mitigating steps.

It was another case of business intuition and data insights sitting at the same table and together producing maximum value for the company.

Putting the pieces together

In my previous posting, 150 data scientists and still no business value?, I mentioned how critical it is for data scientists to live and breath business intuition. In the examples above, we briefly illustrate the back-and-forth: how intuition directs analytics while analytics, in turn, guides intuition.

Gartner has developed a very useful four stage value progression, the Gartner Analytics Maturity Model. In terms of this progression, Gartner’s analytic stages one and two (description and diagnosis) serve to validate and inspire business intuition, while stages three and four (prediction and prescription) provide the means to make business intuition operational. It’s a back-and-forth all the way down the value chain.

To quote Simon Uwins:

“We don’t forget our intuition, but better data lead to better thinking, and our data give us the confidence to ask the right questions”2
Let’s keep asking the right questions, looking to both data as well as to intuition for the answers.

References:

  1. http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html
  2. Humby, Clive, Terry Hunt, and Tim Phillips.Scoring Points: How Tesco Continues to Win Customer Loyalty Ed. 2. N.p.: Kogan Page, 2008. Print.
  3. Kelly Leonard and Tom Yorton. Yes, And: How Improvisation Reverses “No, But” Thinking and Improves Creativity and Collaboration–Lessons from The Second City. HarperBusiness, February 3, 2015