Advanced Computing in the Age of AI | Thursday, April 18, 2024

Bullish Hadoop Forecast Despite Spark Hype 

Despite major market inroads being made by Apache Spark, a new forecast estimates the global market for the Hadoop big data framework will continue to grow at a healthy clip through 2021, fueled in part by growing enterprise demand for Hadoop services.

According to a market forecast released this week by Allied Market Research, the Hadoop market is expected to grow at a 63.4 percent compound annual growth rate over the next five years, reaching $84.6 billion by 2021. The sustained growth is attributed in part to accelerated Hadoop adoption in Europe, where annual growth rates are expected to top 65 percent.

The sustained growth of the Hadoop market stems largely from higher rates of adoption in North America, especially in the IT, banking and government sectors as enterprise big data strategies have been rolled out.

The forecast includes Hadoop software, hardware and services. The market watcher found that Hadoop services accounted for nearly half (49 percent) of global demand. Hadoop services include consulting, "integration and deployment" along with middleware and support. Consulting and related services alone accounted for an estimated $1.6 billion in revenue last year, the market researcher said.

Meanwhile, increasing investments in big data analytics and "real-time operations" are expected to drive integration and deployment services, making it the fastest growing Hadoop services segment (64.8 percent) over the next five years. The "trade and transportation" sectors are expected to fuel adoption of Hadoop services during the forecast period, with an annual growth rate pegged at 76.3 percent through 2021.

Along with trade and transportation, other key end-users of Hadoop products and services include a banking, financial services and insurance category along with manufacturing, retail, telecommunications, healthcare and media and entertainment. "Factors such as aggrandized generation of structured and unstructured data and efficient and affordable data processing services offered by Hadoop technology are the major drivers of the market," the analyst concluded.

Along with the rise of Spark, other potential challenges to the continuing growth of the Hadoop adoption include distributed computing and security issues, researchers added.

Among the Hadoop market leaders identified in the forecast are Hortonworks Inc., Cloudera Inc. and Mark Logic that "have focused on development of advanced, Hadoop-based data storage, management, and analytics solutions to cater to the customized requirements of business enterprises."

The Hadoop market also may face headwinds as the much-hyped Spark platform matures. Billed as the next-generation data processing engine, Spark 2.0 is expected to offer a new structured streaming approach that could help unify development of big data batch and streaming applications.

Launched as a batch processing system for search data, another knock on Hadoop is that it was not designed for real-time, interactive analytics and reporting. Development work on new processing engines to supplement or replace Hadoop's core MapReduce processing engine have met with mixed results, critics contend.

About the author: George Leopold

George Leopold has written about science and technology for more than 30 years, focusing on electronics and aerospace technology. He previously served as executive editor of Electronic Engineering Times. Leopold is the author of "Calculated Risk: The Supersonic Life and Times of Gus Grissom" (Purdue University Press, 2016).