Insight to Action: 8 Analytics Trends for the New Algorithmic Enterprise
(Aha-Soft/Shutterstock)
“I spend millions on analytics, machine learning and big data. I have tons of insights. Too many insights. What I need is more action.” A CMO recently said this to me during an executive summit, and his feelings are shared by many: with the hype around cognitive computing, most business leaders feel they have more insights to ponder than actions to execute.
Frost & Sullivan recently released a report that points to the problem: the convergence of the IoT, analytics, and mobile e-commerce. The report describes a future vision of “enterprises (using) automated responses based on data trigger points to personalize and individualize their customer conversation(s).”
Why aren’t businesses doing this today? Even companies with advanced machine learning and analytics tools have too wide a gap between insight and algorithmic action. At the same time, there are great pockets of algorithmic innovation in the intelligence community, Wall Street and high-tech manufacturing that we can learn from. Here are eight seismic shifts that will bring algorithmic automation into the mainstream.
#1: Streaming Analytics Will Become Ubiquitious
Business intelligence, predictive analytics, and machine learning are essential for the future, but they have a fundamental flaw: they are designed to look at data at rest, not data in motion. Looking at data in the rear-view mirror view is the root cause of inaction. Unless you can automate, you can’t act.
Streaming analytics turns the physics of analytics on its head. Streaming technology evaluates predictive models and algorithms on the fly while data is moving. This allows a business to ask and answer millions of questions in real-time about what’s happening now, and to continuously act based on live events.
For example, many companies today can use big data to discover IoT algorithms that predict the failure of a device based temperature and voltage sensor readings. But in too often it takes minutes, hours or days to digest streaming data, and that’s too late to act on the most important signs of failure. Streaming analytics allows the continuous computation of the moving average of sensor readings over a 15-minute time window, in memory, against each IoT sensor reading. When conditions trigger the algorithm, it can trigger an action based on algorithms that consider the severity of the condition. For a mild signal: alert an operator; for a severe signal: shut down the device; for a complex signal: open a maintenance case for human investigation, or send an alert to an IFTTT channel.
Today, streaming analytics are popular in heavily automated markets, such as security, fraud detection and algorithmic trading. Tomorrow, it will be ubiquitous in all markets, including loyalty in retail, drone-aware transportation and logistics, and high-tech manufacturing.
#2 The Insight Platform Category Will Replace the BI Category
Imagine you’re using a BI tool to explore customer data and you notice an undiscovered upsell trend. What do you do? You might call a meeting with the CMO, plan a new email campaign, or maybe even automate a coupon offer. This “action” can take weeks or months to set execute.
In 2016, Forrester published a Wave Report the defines a market of tools that they call “Insight Platform” a combination of BI and streaming that helps you take action on insights. Once you discover that upsell model, an Insight Platform allows you, with proper security and governance, to inject that model into a streaming analytics component with the click of a button. This injection of a new algorithm takes minutes, and streaming analytics dashboards allow you to track how well the offer is doing and modify the algorithm as it runs, in real-time. In a sense, it turns the test-and-learn cycle for analytics into an interactive game – you go from insight to action to refinement in real-time, as the digital business runs.
How do Insight Platforms work? It starts with traditional visual BI, machine learning, and artificial intelligence to find insights. Then, the modern BI tool can create a real-time algorithm based on a model you create in the tool. For example, some Insight Platforms can deploy an R statistical model into streaming analytics with a click of a button.
In the future, BI tools without the ability to deploy automation models will become extinct.
#3: Drone-Scale Analytics
Today’s business requires consideration of hundreds of millions of customer connection points; tomorrow’s business must consider the opportunity in billions of connected drones and devices for delivery, customer interaction, and mobile conversations with customers. Every retailer must be able to utilize analytics to support drone-scale delivery of products to compete with Amazon. Dealing with several orders or magnitude greater volumes of streaming data and analytics to adapt to this real-time capability will be essential. As scalable streaming analytics infrastructure becomes available to the mainstream, smart drone-scale enterprise conversations will become a reality.
Drone-scale analytics will feature massive data lakes, deep learning, AI, faster in-memory model evaluation (see prediction #4), and second-generation streaming analytics (see prediction #6).
#4: Analytics Break the Automation Speed Barrier
IoT is graduating from hype to mainstream, and along with it will come a transition to data-stream analytics. This will require increasingly sophisticated tooling designed for sensor data, and the resulting predictive execution models will need to be optimized to run on streams at machine-to-machine speeds, as opposed to request-response (human-to-machine) speed.
For example, R statistical models are designed to respond to human requests to evaluate an algorithm - typically just a few times a second.
Streaming systems often need to check models 10, 100 or even 10,000 times or more per second than R is designed to handle. They do this by injecting queries into the stream and incrementally evaluating them as events flow by, and computing answers on the fly. A single streaming analytics computing node can check models 50,000 times a second, and so a pool of a few hundred nodes would check 5 million possible actions a second. Without model execution at this rate there’s no way to optimize a continuous digital conversation with customers and automated devices.
#5: The Man Versus Machine Debate Will Be Settled (we already know the answer)
The rise of algorithmic trading began about 15 years ago. At the time, the industry was buzzing with the debate over man versus machine. Since then, algorithmic trading has risen from less than 5 percent of the world’s trading to over 80 percent of volume.
Who won, man or machine? They both did. As any head of electronic trading will tell you, automation is superior for high-volume, statistically predictable decisions. But at the same time, the role of humans in trading has only become more important – humans are always superior to computers when something statistically unique happens and judgement and intuition is required.
Any head of HR on Wall Street will tell you the jobs have changed. Fifteen years ago, traders were salespeople with business backgrounds; today, traders are more geeky. In aggregate, there are fewer traders than before. But there are also more high paying jobs for algorithmic research and analytics. Customer service on Wall Street has turned from reactive to proactive – using AI techniques for years to predict opportunities and present them quickly to customers via mobile channels. And executives are now just as likely to have a statistics or math background instead of an MBA – or both.
So while fear over job loss is real, there’s also a healthy future for new skills and new, exciting jobs, most of them built on mathematics, statistics and computer science. So if you have school-age kids, don’t let them skimp in math class!
#6: 2nd Generation Streaming Analytics Platforms
Streaming analytics will be a hot topic in 2017, but it’s not exactly new. It was born from research at MIT, Stanford and Cambridge University around the year 2000. However, the rise of IoT and public cloud platforms is driving the need for a new generation of streaming technology.
Beginning in 2017, we’ll see some of the “big boys” of commercial streaming analytics pivot to the cloud, with capabilities such as:
- Public and private deployment of streaming analytics in AWS, Azure, Cloud Foundry, and within such IoT platforms as GE’s Predix, Syniverse, British Telecom and more.
- Elastic streaming analytics capabilities baked into enterprise platforms.
- Frictionless integration between cloud BI and cloud streaming via remote injection of streaming analytics into the public and private cloud infrastructure.
- New thing-based security protocols. As more consumer business flows through automated streaming analytics, advanced security will be applied to machine-to-machine interaction. This, in turn, will be deployed in the new IoT cloud infrastructure as well as in streaming analytics.
#7: The Consumerization of Human Action
When you walk into your home, do your lights turn on? If so, you understand the amazing world of home automation. If not, you will soon. Consumer technology that enables mobile control of home automation is coming to your enterprise. Firms with sophisticated automation are creating new consumer channels of communication. For example, I recently met a top bank executive who uses IFTTT to get client and trading alerts with the same application he uses to turn his lights on and monitor his security system at home.
The more you automate, the more you need better tools for humans. First generation automated analytics mostly happened behind the firewall on private networks. But automation in the future will extend to the consumer via social platforms and event-driven mobile tools like IFTTT. These tools will be increasingly integrated with enterprise Insight Platforms, and we will continue to solve the security, governance and reliability challenges they naturally carry.
#8: Spark Trumps Hadoop for the Algorithmic Enterprise
The future of computing is big data, but it’s also being shaped by the IoT and the need for streaming. Spark works well for streaming apps; Hadoop does not. As a result, we will continue to see Spark supplant Hadoop as the go-to platform for big data, with a focus on important extensions that make it more easily connect to streaming platforms. This will require extension points and integration with enterprise streaming platforms, with native support for AVRO, H20, PMML, SparkML, and Kafka.
Moving Forward
For analytics, the mainstream must become more entrepreneurial than ever before – it must apply lessons and technologies born in Silicon Valley, Wall Street, government and disruptive digital businesses to help create a new algorithmic enterprise using new tools like Insight Platforms, streaming analytics, Spark-style big data, and streaming IoT data and analytics. The very role of the human workforce in this future will change. Skill sets will change. And for those who find change exciting, the future of algorithmic digital business will be a fantastic innovation playground.
Mark Palmer is senior vice president and general manager of analytics at TIBCO Software.
