Algorithms: The Silent Game Changer in Big Data

Hype is synonymous with Big Data. But data is nothing new.

What’s new is that manufacturers are finally learning how to draw critical insights from that data for competitive advantage. The key to that, of course, is having the knowledge and understanding to spend wisely on knowledge discovery and getting the data to talk to us.

According to the results of a recent Gartner study, enterprise data is on pace to grow 650% over the next few years, with 80% of that data in unstructured form. This is more than any data scientist -- or person -- can manage.

People’s ability to process data hasn’t changed much over the years, but the technology we employ to help us process it changes constantly. Therein lays the promise of Big Data. Specifically, Big Data analytics that rely on algorithms are our only hope of keeping up with the data explosion.

Algorithms and Big Data

Every time we have faced technology challenges that are computationally intensive, algorithms have come to our rescue, helped us to overcome the barriers and we have pushed the frontier of innovation. A few good examples are:

• Operating systems are algorithms that enable computers to interact with humans. Without operating systems there would no PCs, laptops or smart phones.
• An algorithm that defines how content is stored and transmitted enables the Internet, or the World Wide Web. These were world-changing algorithms and today we have more than 200 Billion web sites at our disposal.
• Google’s ranked search is an algorithm that lets us quickly search for highly relevant content on the web site. 

And yes, there are many more. What they all share in common in the ability to break through barriers that limit our ability to reach across get to the next frontier of innovation. Big data poses a ton of such barriers -- and smart algorithms will help us to overcome these barrier.

Taking the Human Element Out of the Big Data Equation

The biggest problem with a traditional approach to data is that it’s typically an extension of old thinking and hence time-consuming because it may fail to consider the barriers facing us today -- the volume of data and the cost and time to sift it.

It’s time for an entirely new approach that addresses these growing needs.

This new approach demands a paradigm shift that focuses on the following:

• A fundamental change in the role played by analysts from data-miners to insight-evaluators.

• Fast and efficient methods that automatically convert data to insight for analysts to evaluate and operationalize.

• Continual improvement of these automation methods to keep up with the speed of data and critical need for timely insights.

The next wave of challenges is upon us. How can we scale these efforts to automate getting the gold from the data?

The data-scientist approach is labor intensive, time and cost prohibitive -- and most importantly cannot scale to the speed of data. The new approaches need to automate the process just like processes before them such as automated mining of gold and other precious metals from dirt. The analogy is so very appropriate.

Manufacturing companies are finally looking to harness the power of Big Databecause it drives enormous opportunity for business improvement; however, this is still just the first inning of the game, and I would caution against investing too heavily in approaches that rely in hardware and staffing increases.

If a method you’re considering is intrusive, watch out! It’s not sustainable.

Ask yourself how much training the new tool is going to require, and whether or not it demands hiring a lot of experts to manage. Do you really need a team of data scientists to surface insight from data? How smoothly will it fit into your current business to help drive intelligent decisions with insight from data. The best approaches are ones that are sustainable.

And let me leave you with one more question to ponder as you go about your busy day -- Why is it that we have so much data, yet less than 1% of data is analyzed? It is because we have been so consumed, due to our hoarding mentality, with saving all this data while thinking not enough of how to get to the gold in them thar hills!

Emcien CEO Radhika Subramanian is a seasoned entrepreneur with decades of experience helping organizations utilize the insight buried within their data. Tweet to @RadhikaAtEmcien & join her Big Data Apps LinkedIn group.