Big Data Driving New Approaches in Econometrics

Data is finance’s new currency, healthcare’s latest wonder drug, and the energy sector’s new oil.

Another day, another Big Data analogy.

All of the hype doesn’t change the fact that businesses across nearly every industry are gaining competitive advantage by extracting value from large datasets.

Econometrics is an area that has been cautious about Big Data. The field is built on a strong foundation of theory and methodology, and relies on a variety of approaches that differ significantly from those of Big Data analytics. For example, econometrics typically starts with a theory and then uses data analysis to prove or disprove it, while Big Data and machine learning work in reverse. Econometricians have also expressed concerns regarding the context, reliability and representativeness of such vast datasets.

However, it’s becoming clear that Big Data has the potential to be disruptive to traditional econometrics. Data collection over social sources has produced unprecedentedly large and complex datasets about human behavior and interaction, and this unstructured data has proven itself to be a goldmine of economic information.

Econometricians are certainly not strangers to data analysis; however the growing volume of economic data from diverse sources is driving the need to adopt new computational approaches and develop better data manipulation tools.

Econometricians entering the field today also face a bit of a learning curve, and find they require a combination of skills in both economics and computer science to deal with the increasing volume, variety, and velocity of data. Hal Varian, Chief Economist at Google offers this word of advice to current students of econometrics: “Go to the computer science department and take a class in machine learning.”

While econometricians might still be working out the “kinks” in their Big Data approaches, the analysis of large datasets is already driving a number of advancements across the field:

• Over a two-year period, researchers analyzed millions of transactions among nearly 128,000 consumers of a packaged goods chain to determine whether characteristics of the first purchasers of a new product had any impact on that product’s long-term success. Customer characteristics and purchasing behavior were processed to reveal a small subset of customers they referred to as “harbingers of failure,” who had a propensity to buy new products that were likely to flop. The data also helped researchers quickly reveal hard-to-find patterns among consumer groups and challenge traditional early indicators of product success.

• Researchers have used Big Data to analyze investor behavior and its eventual effect on stock market performance. By collecting internet usage data, researchers could pinpoint investor’s attempts to gather information online before executing a trading decision. The resulting data allowed researchers to trace the consequences of specific investor behavior, as well as offered predictions regarding stock market performance and new insight into the early information-gathering stages of decision-making.

• MIT’s Billion Prices Project (BPP) aggregates daily price fluctuations of approximately five million items sold by 300 online retailers in more than 70 countries to provide real-time predictions on inflation. While traditionally econometricians have been forced to rely on historical data to generate future predictions, data sources are now available in real-time to help identify economic trends as they are occurring.

Machine learning by its very definition has the potential to rapidly alter the field of econometrics. The ability of computers to develop pattern recognition, and then learn from and make predictions based on data is a familiar task for econometricians, who on a daily basis analyze tremendously large volumes of economic data in order to form theories.

As Big Data continues to penetrate the methods of econometrics, the field will need to adopt new computational tools and approaches in order to extract insight from these increasingly large and complex economic datasets. The granularity offered by Big Data will enable econometricians to adopt new data-driven styles of analysis and investigation to help them resolve their biggest economic questions.