I’m sure you’ve heard the term big data before and can likely use it in the proper context within a sentence. But can you define big data? What is big data? Is your data big enough, and is the size of your data important? These are questions many have been asking for years. But rather than taking my word for it, let’s head to the largest repository of data on the planet – the Internet. Wikipedia states, “Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications.” Terms such as “large,” “complex,” and “difficult” are relative. The definition of big data, therefore, has different meanings for different organizations, and there is no one right answer. Countless articles and commentaries on the Internet will describe how companies such as Google!, Amazon, and Facebook are using incredible amounts of data to mine intelligence for the benefit of their customers and assuredly their profit margins as well. But most organizations do not have access to that amount of data, and even if they did, they do not own the analytical horsepower to derive business intelligence. Therefore, what is considered “big data” varies depending on the capabilities of the organization and the capabilities of the applications used to process and analyze the data set within that specific domain. Let’s assume you’ve found a way to capture, analyze, store, search, and even visualize data. You now need to be concerned with privacy violations. No one wants to be the next Wall Street Journal headline for a data breach and the loss of customers’ private information.
But enough of the challenges and potential issues. The worth of big data is really in the insights that businesses can obtain from it, rather than the amount of information itself. If you can harness the data available to you for your benefit, the options are only limited by your imagination. Benefits include better operational visibility, actionable results, and strategic improvement. For decades, companies have been using big data to run large linear programming algorithms as part of supply chain network optimization modeling. Tools as mundane as trailer load and route planning are improved significantly by analyzing large data sets. While past performance is no guarantee of future results, it certainly is a better indication than the absence of any information at all. Moving beyond those examples and achieving the next level of business intelligence requires a mindset shift for almost all of us, as truly big data throws away the old scientific approach that we’ve all learned. The scientific method of establishing a hypothesis, gathering data, deleting “outlier” data, running an analysis, and drawing a conclusion about whether the experiment proves or disproves the hypothesis is now archaic. In truly big data, there is no such thing as “bad data.” All data is processed, and the analysis looks for correlations as opposed to causations. We start talking in probabilities instead of absolutes. And this makes sense. When was the last time you made a difficult business decision with all of the information you needed? Never. That’s just not reality. Artificial intelligence and machine learning are simply the analysis, manipulation, and resulting action derived from enormous data sets – more information than a human mind can process – and therefore, decisions with a higher probability of success.
If the daunting task of creating actionable business intelligence from your data seems overwhelming, do not be discouraged. Just get started. Solicit help making a plan and then go. Keep in mind that the amount of analysis you can do in Excel on your laptop was unthinkable hundreds of years ago. And remember, the journey of a thousand miles begins with the first step.
I thank you for listening to this mind dump!
— Matt Kulp, St. Onge Company
