See it Now: Streaming Analytics Visualization on Kafka 

 In traditional IT, data’s gathered up, stored and analyzed in batch. But in our new, digitally transformed world, we want everything converted to data and analyzed – including reality.

If you’re in retail, you want to know which products are moving now, not yesterday, last week or last month, but right now. If you’re in energy discovery, you want predictive insight helping you decide when to do preventive maintenance on your oil rig – you don’t want to take it offline before it’s necessary, nor after it’s broken down, but at the optimal time.

Hence the growth in IoT to collect data from everything all the time, edge computing to processing out where the data is located and streaming data analytics on Apache Kafka, the open source stream processing platform, to deliver immediate insight. But there’s always been a snag with real time analytics: it requires significant development work from software engineers that’s beyond the skill level of business analysts.

Addressing this need, Arcadia Data, builder of a visual analytics and BI platform native to big data, this week announced Arcadia Instant for KSQL, which the company said is the first native visual analytics capability for Apache Kafka. The open source offering combines two existing technologies: streaming analytics and data visualization, with the result that you don’t have to be a software jock to make the software produce the bar graphs and pie charts you want, in real time.

“The promise of streaming data is a game changer and is generating plenty of excitement, but today that information is in the purview of engineers,” said Priyank Patel, co-founder and chief product officer, Arcadia Data. “The software we released today makes creating visualizations on streaming data a simple drag-and-drop task for business users. In doing so it makes real-time insight actionable to anyone in the organization.”

The software can be installed on any desktop and enables visualization of data streaming through Apache Kafka topics. The company developed the product in partnership with Confluent; it connects to Confluent Platform 4.1 and is available as a complimentary download on the Arcadia Data website.

“Event stream processing is becoming a major part of business analytics,” stated industry analyst firm Gartner Inc., in a recent report. “To get information from sensor data, web clickstreams, geolocation data, weather reports, market data, social media and other event streams, data and analytics leaders must employ three specific application styles."

According to Arcadia’s VP of marketing Steve Wooledge, businesses have been limited in their adoption of streaming technologies by the coding-heavy environments that dominate industry offerings, which he said often take time to deliver results, diminishing data agility. Arcadia Instant for KSQL integrates Confluent’s streaming SQL architecture for Kafka with Arcadia Instant, its visualization engine, leveraging KSQL’s native architecture to support a push-based query model. Arcadia said this integration is not possible with non-native BI software that is limited by JDBC- and ODBC-based connectivity.

“This is all about making that experience easier, especially for non-technical folks,” Wooledge told EnterpriseTech. “It’s a freely downloadable package that allows Apache Kafka users to get immediate exposure to exploring visualization on Kafka. Historically, when dealing with streaming data, especially on a Kafka cluster, it required a lot of development effort, you needed engineers to write code to analyze that streaming data.”

Arcadia said the new product supports complex data types, allowing users to read a variety of data stored in Apache Kafka directly from the stream, including MAP and ARRAY data common in JSON and Avro file formats. Its “Time Warp” capability provides a view of data in the stream based on time, allowing users to see what happened in the recent past that they might have missed, to compare past and current trends, and replay or pause at various points in time.

Designed to run on commodity X86 servers, Arcadia Instant for KSQL is available now.

Arcadia, formed in 2012, has approximately 70 employees and grew more than 300 percent last year, according to Wooledge. He said the company ranks Fortune 500 companies among its customers, including Kaiser Permanente and Nokia.