Few business segments have had their processes, strategies, and results affected by new technologies, especially those related to analytical applications and big data, as much as retail. According to estimates by the McKinsey Global Institute (MGI), in 2009, almost all sectors of the US economy had, on average, 200 terabytes of data stored per company with more than a thousand employees. This volume of data is equivalent to 2 times the size of the Wal-Mart data warehouse, the largest American retail chain, in 1999. This study makes it clear that, in the first decade of the 21st century, companies and people produced data at a much faster pace compared to any other period in human history. This pace led many mid-sized companies to accumulate a volume of data greater than the amount of information existing (10 years earlier) in the centralized repositories of large companies.
Certainly, while its smaller competitors increased their data and application portfolio, Wal-Mart did not settle or reduce its strategy of transforming data into relevant information and knowledge for decision-making. This attitude was fundamental to increasing its productivity, profitability, and efficiency in its commercial processes, which led this company to become a retail icon worldwide. At the end of 2008, the company hired Hewlett Packard to build a data warehouse capable of storing 4 petabytes (or 4,000 terabytes), a volume of data approximately 40 times the amount stored in its first infrastructure. Its ambitious project aimed not only to store but also to identify consumption patterns and profiles based on more than 267 million daily transactions carried out in each of its more than 6,000 points of sale. For this, Wal-Mart also hired data analysis and mining specialists to implement models and applications based on machine learning algorithms that also aimed to evaluate the effectiveness of its pricing strategies and marketing campaigns, as well as contribute to better management of its product inventory and supply chain.
The characteristics of the project and the success of Wal-Mart's big data strategy (even before the term big data became widely known in society) ended up influencing other companies (not just retail) to follow the same path. Confirmation of this can be seen in a recent study conducted by IDG Enterprise, also related to the American market, indicating that 80% of large companies and 63% of small and medium-sized companies are implementing or intend to implement big data solutions in 2015.
From the analysis of surveys like IDG's, it is possible to infer that, despite the increase in volume, variety, and velocity with which data is produced, more and more companies are investing in implementing big data solutions, even with much more modest budgets than Wal-Mart. There are some explanations for this apparent contradiction, since for more complex scenarios with larger data volumes, a greater investment by companies in data storage, management, and analysis platforms would be expected.
The first explanation for a larger number of companies, especially in retail, investing in implementing analytical solutions is related to what was known as Kryder's Law, which estimates an increasingly favorable relationship for the consumer market between the price of digital storage devices and their respective data storage capacity. The technological advances that have allowed companies to accumulate and make more data accessible to their executives and analysts can also be observed in computer architecture that currently enables the use of processing capacity several orders of magnitude higher than what was possible to use in the recent past for the same price.
The second explanation is related to points we have already mentioned in other articles in this space. The advent of data discovery analytical platforms, with significantly lower investment demands than traditional BI platforms, which allow the use of advanced analytical solutions directly by managers and business analysts, has been a very relevant factor in expanding the reach of big data projects in all business segments.
In retail, as in any other type of business, once technology makes it easy, complete, and effective to obtain answers to questions relevant to the management of each process, asking the right questions becomes the great challenge for organizations. The most frequent questions in the management of this type of business involve: (1) defining commercial policy based on price analysis and demand elasticity, (2) selecting appropriate products and supply plans for each type of channel, considering social media insights, market reports, internal sales data, and customer consumption patterns, and (3) estimating adequate stock levels at points of sale and distribution centers based on customer sentiment analysis and the expected effect of promotions that can serve as a warning to anticipate future demand.
In Brazil, in addition to the relevant questions that guide the actions and strategies of practically all retail companies, the year 2015 will be particularly challenging due to macroeconomic adjustments that may have some negative effect on demand, especially in the first half of the year. In this sense, managing and, much more importantly, taking advantage of the large volume of data (public and internal) has become even more relevant for retail companies in terms of competitiveness and the ability to calmly overcome possible reductions in demand caused by external factors. For these companies, there is no time to lose: the time for big data has arrived. What is still a competitive differential will soon become a survival factor for retail.