IBM BigInsights for Apache Hadoop Introduced 

IBM shifts data science into high gear today with the announcement of new In-Hadoop analytics technologies to accelerate the conversion of data into valuable insight for the business. IBM is delivering machine learning, R, and many new features that can run over large-scale data in the new IBM BigInsights for Apache Hadoop.

A growing number of organizations today recognize Apache Hadoop as a powerful technology for collecting and economically storing a very large set of highly variable data, and yet they struggle to realize its full potential in every part of their business. For example, a business analyst needs to quickly find relevant information, the data scientist needs to make sense of the data with statistical modeling, and the whole environment needs to be easy for IT to manage and deploy for everyone to use.

As the size and complexity of Hadoop applications continues to grow, data science has become a critical function for unlocking meaningful insights and identifying actions that could optimize outcomes across the entire business. IBM BigInsights for Apache Hadoop includes a broad data science toolset to query data, visualize, explore and conduct distributed machine learning at scale.

“There’s so much data out there that it’s often difficult for companies to find the information that really matters,” said Anand Mahurkar, CEO of Findability Sciences. “IBM BigInsights helps connect data elements that often go undiscovered to bring context to the customer relationship. Some of our clients have improved customer retention by up to 25 percent with this enhanced understanding. They can now focus efforts and resources on where they’ll have the greatest effect, helping win and retain more business and drive more profitable operations.”

IBM BigInsights for Apache Hadoop will introduce three new modules:

• IBM BigInsights Analyst will include IBM’s SQL engine and IBM’s intuitive spreadsheet and visualizations to find data quickly and easily. On average, millions of SQL queries are run each year. With BigInsights Analyst, the efficiency of these queries has been shown in some cases to improve by approximately 2x to 4x on Apache Hadoop depending on the shuffle size. The ANSI compliant SQL means queries can run unchanged against Hive, HBase and relational databases.
• IBM BigInsights Data Scientist will deliver a new machine-learning engine that automatically tunes its performance over large-scale data to find interesting patterns– plus over a dozen industry-specific algorithms such as Decision Trees, PageRank and Clustering to help tackle complex problems out of the box. It will also provide native support for open source R statistical computing helping clients leverage their existing R algorithms, or gain from the more than 4,500 freely available statistics packages from the R community.
• IBM BigInsights Enterprise Management will introduce new management tools for clients to realize faster time to results. Designed to help allocate resources and optimize workflows, these tools will allow deployments that can scale to large numbers of users and clusters, and will help satisfy high workload demand. These tools will provide multi-tenancy and multi-instance support in a cluster.