Advanced Computing in the Age of AI | Tuesday, November 29, 2022

The ‘Insight-Driven Business’: How to Become a Master of the Data Universe 

Masters of the Data Universe: Uber, Netflix, Facebook, Amazon, Google. We know who they are. They hire genius data scientists who wrestle data into submission, building elite analytics superstructures that light up their data and reveal insights about their markets while nurturing interactive customer relationships at scale that leave everyone else shaking their heads, wondering how they do it.

It can seem like magic. As Forrester Research’s Brian Hopkins, vice president and principal analyst, observes, there’s the vague notion (encouraged by some vendors) “that…somehow data goes into a box with the elephant and good things come out of the box, magic happens.”

hadoop-logo-2But now, several years into the Big Data analytics revolution, patterns and common characteristics are emerging among the Masters of the Data Universe. According to Hopkins, they have landed on the critical issue: they view analytics through a strategic prism that distinguishes data from action.

Put another way: there’s a wide gulf between Big Data and actual analytics - much more of the former is going on than the latter. Most companies have built the initial Big Data framework that could be used in an analytics implementation. “We aren’t lacking for investment in new big data technologies,” Hopkins said at last week’s Strata + Hadoop conference in New York. “In fact, this year at Strata I assume every company has built a data lake. You probably have Hadoop.”

forrester-gapSo while the enterprise landscape is a data lakes district, the lakes themselves are dead pools for generating the kind of market and customer insights that companies crave. Forrester ran a survey last year in which nearly three-quarters of data architects said they aspire to be data driven. “In the same survey, I asked how good are you at actually taking the results of analytics, the insights, and creating actions that matter to your firm? Far fewer people say they can do that well,” Hopkins said. The old saw that we’re drowning in data and thirsting for insight still holds true.

Hopkins outlined a strategic framework for moving toward transformative analytics, one based on his study of the Masters of the Data Universe and their shared traits for becoming an “Insight-Driven Business.” Attaining this status, he said, is both achievable and mandatory: failure to figure out Big Data analytics will mean significant competitive disadvantage and eventual decline.

The first step is to embrace a concept what Hopkins calls “digital insight,” the ability “to systematically harness and apply digital insights to create a sustainable competitive advantage.”

“By that I mean new actionable knowledge. It’s data, but it’s data that leads to action in the context of a process or a decision,” he said. The transformative aspect of “digital insight” is that new knowledge becomes embedded in software, getting it into the guts of the analytics system and serving as an insight engine that knows no rest.

This “refocuses our conversation from ‘How do I get insight (that’s) in the data?’ to ‘How do I implement insight – no matter where it comes from – in software?’ Hopkins said. “So it’s a marriage of the insight execution in application development.”

The second step is to understand how Insight Driven businesses operate. Hopkins said he has interviewed hundreds of “digital disrupters, ” ranging from the monsters (Netflix, et. al) to start-ups “trying to figure out what it is they’re doing differently.”

“It’s not the technology or what they did with data that’s so interesting,” he said, “it’s the fact that they understand how to apply insights in software to drive competitive advantage in ways that many of us don’t.”

Common among them is the circular and continual use/re-use of data within a closed loop system. “The pattern appears over and over again. They’re operating in this closed loop and they’re operating faster in a closed loop than their competition.” This “system of insight,” he said, is going to the right data, combining the right data and creating effective actions using a process Hopkins calls Insight to Execution.

The idea of “circular insights” is not new. What’s different about Insight to Execution, according to Hopkins, is “continuous experimentation, testing and learning with your insights. So (is it) having an insight, deploying a predictive model, updating that predictive model.” Updates aren’t happening every six months, either, they happening continually. Insight-Driven businesses “are always hypothesizing.”

Hopkins summarizes the Insights to Execution loop as:


  1. Experiment and learn continuously – question every process and decision.
  2. Identify outcomes and interim metrics – develop metrics for every outcome; instrument and measure processes, decisions and outcomes.
  3. Gather more data – Start with data you have, then add new sources and kinds of data as you learn.
  4. Develop insights – apply analytics and AI methods to develop potential insights.
  5. Test and implement insights in software – run insights, experiments in software, processes and decisions
  6. Measure results and refine insights – courageously assess and share the results. Did what you happen?

Hopkins cited Stitch Fix, a custom tailoring online clothing company. One way Stitch Fix is an Insight-Driven Business is that it continually enriches its data by running experiments to gain greater customer understanding – to the point of sending clothing to people whom they believe are likely to return items “just to learn stuff about the people who will keep the clothing, so they can make their models better. They’re running experiments. They’re experimenting and learning continuously as they go around the loop.”

A key element to this experimentation, he said, is “understanding the outcomes you want to change and getting granular about the level of detail so you can measure an instance.”

Brian Hopkins of Forrester Research

Brian Hopkins of Forrester Research

Hopkins said Stitch Fix followed the wisdom of starting a 1000-mile journey with a single step: they began with the data that they had, and worked their way up from there. This runs counter to many companies that begin by amassing enormous amounts of (unwieldy) data. Stitch Fix doesn’t “have a whole lot of really exotic data, they’re merely optimizing the data in that ‘System of Insight’ in which they transact with people.

“What a lot of these companies do is start that way, and as you go around the loop, then you add those secondary data sets,” Hopkins said. “But start with the data that you have, add more d over time as you find and drive those insights.

Hopkins said the testing and implementing insights in software “makes application developers as important as data scientists in this process.” At Stitch F ix, they’re the same person, he said, while at others, like Tesla, they’re different but they work together as a team. Hopkins said he’s talking to Insight-Driven CEOs who tell him they put as much emphasis on hiring good software developers, who can embed insight into code, as they do good data scientists.

Hopkins said Stich Fix’s chief algorithm officer (CAO) told him the company runs algorithms on Amazon S3, deploying them into their business applications use by employees to send clothing to customers, “feeding all that data back to the system in real time, back into S3, and round and round they go. It’s a pretty common pattern. You go to Uber, Netflix, they learned how to do this.”

stitch-fix-systemHe said they use Apache Yarn to stand up Spark clusters. “They’re…standing up instances of data science, updating those algorithms, deploying them back into their applications, and it’s not just that they’re doing this, it’s how quickly they can do it, they can update their algorithms very fast.”

The Stitch Fix CAO told Hopkins the company their analytics strategy has helped them understand “not only what their customers want to buy but how much to make and how much inventory to carry. So this way of working is not just a matter of engaging with the customer, it applies to the whole company front and back. They’ve dramatically reduced their inventory carrying costs as well as knowing their customers better than their competition. That’s the secret to success in today’s BI world.”

teslaAt Tesla, the advanced car manufacturer, the company has built what Hopkins calls an “insights fabric server” that serves as an analytics platform and gathering place for “all the different places that they keep data.”

“They say, ‘Look, there’s too much data to keep it all in Hadoop, we have to put it all over the place and bring into a platform.’” Inside the platform is a web-based UI that serves up insights on-demand to engineers that are “data-massaged with knowledge.” Using a set of data pipelines that feed the platform, data scientists and design engineers work together to update cars’ firmware.

“They’ve added horsepower, they’ve changed the elevation, they’ve changed the experience in real time, and what that does is create more data and goes then back through their pipelines into all their sources and that data then becomes available to their engineers to change the experience again.”

A final step: after implementing and testing insights in software, “you’ve got to courageously measure and share the results. And that’s not easy specifically because a lot of times those results will be ugly, and you won’t want to share them across the organization.” But at Uber, for example, “half the employees access their data warehouse every day. They share data,” Hopkins said. Sharing is critical to hypothesizing, learning and refining insights on an ongoing basis.

“This is what these companies do,” Hopkins said. “They go around this loop in hours and days, not weeks, months and years. That’s how they’re outpacing their competition.”


Add a Comment