The long and windy road from ingest to insight

In an ever-changing well-connected world, enterprises large and small strive to react faster and faster to market conditions. Many wish they could better leverage the data they already have but have been struggling with how best to achieve this.

The traditional approach had been with any combination of ETL, queues, databases, data warehouses, and BI. However, there were limits to the types and volumes of data that could be handled. Depending on the number of sources and the required transformations, batch processing meant that results would take hours and perhaps days.

The Hadoop Era

Solutions based on Hadoop were supposed to fix many of the limitations of traditional architectures.  The ability to ingest any type of data, whether structured, semi-structured, or unstructured using a whole zoo of components that could provide any type of data processing, meant that Hadoop could be the central repository for all of the enterprises’ data.  In theory, this approach looked very promising.  The reality was much different.  For an IT department with no previous Hadoop experience, deploying it was non-trivial because of the complexity of the system.  Training courses just to learn how to operate and maintain the environment takes four full days.  Going into production using Professional Services could easily take 6 months to a year with no guarantee of a successful outcome.

The Cloud Native version

With the advent of cheap, reliable, and highly available cloud storage platforms, highly optimized point solutions running on the cloud have been introduced into the market.  They still replicate on-premise and Hadoop functionality (ETL, queues, database, data warehouses), but, being cloud services, they have the big benefit of being able to hide much of the complexity of operating and maintaining them.  Customers, however, still need to string together these point solutions to achieve the outcomes they desire and still need significant time and effort.  As the data still needs to go through multiple stages and most of the processing, at the core, is still batch mode, reducing the time from ingest to insight is still a big struggle.

The age of bi(OS)

Isima’s bi(OS)^® is the next stage of evolution. Rather than optimizing a particular stage as data takes a long and windy road from ingestion to insight, bi(OS)^® takes a radically different approach and consolidates all the multiple stages into one platform.  By doing so, it can achieve dramatic improvements not only in the time it takes from ingestion to insight but also in the time it takes to go into production.  It’s equally versatile running ad-hoc queries as it is experimenting with AI/ML models and running these models in production — all on the same unified platform.

To hear perspectives from my fellow partners in crime, see here, here, and here.  The press release announcing our launch is here.

To get started on your bi(OS)^® journey, click here. And you can reach us @ imagine@isima.io or +1(833)-API-AIBI.