MMOOZZGG.com

Glossary

What is exploratory data analysis?

In short

Exploratory data analysis (EDA) is the first look at a dataset: charts, summary numbers and checks that show what the data contains, what is missing and where the errors are.

Machine learning · EDA

MMOOZZGG.com editorsChecked: 2 min read

For a child

Before you put together a big puzzle, you pour the pieces on the table and look at them. You separate the edges, sort the pieces by color, and check that nothing is lost and no pieces came from another box. Only then do you start. Scientists and programmers also look at their records and numbers first, and only then teach a computer on them.

In plain words

Before you build a model, you need to look at the data. An analyst counts the rows and columns, finds empty cells, sees which values occur most often and checks for absurd records, such as a customer who is two hundred years old. They build histograms, box plots and charts over time, and see which indicators change together. Duplicates, mixed-up units of measurement and outliers show up here too. The result of this review: it is clear whether the data fits the task, what needs cleaning and which hypotheses are worth testing. It is like inspecting an apartment before a renovation: first you walk through the rooms and notice the cracks, then you make an estimate.

For business

This stage is cheaper than the others and often decides the fate of the project. In a few days an analyst shows whether there is enough history for a forecast, whether the exports from the accounting system and the CRM match, and whether whole months are missing. If the data is scarce or dirty, you find out before money goes into development. You also see how much work cleaning will take, and it often becomes the main cost item. The risk is that teams skip the review for speed and learn about the problems only after the model goes live. Ask to see a data report with charts and a list of the problems found. A similar check comes before scenarios are rated in an AI solution map.

For a specialist

Exploratory data analysis (EDA) is an approach in which data is studied mostly with graphical methods, without a model set in advance, to reveal the structure, important variables, outliers and anomalies, to check assumptions and to sketch simple models. The founding work is John Tukey’s 1977 book Exploratory Data Analysis; the techniques are described in the NIST/SEMATECH e-Handbook. Typical tools: histograms, box plots, lag plots, scatter plots, correlation matrices and summary statistics by group. In machine learning, the list adds checks for gaps and duplicates, the distribution of the target variable and class imbalance, a search for data leakage, when a feature indirectly contains the answer, and a comparison of the training and test distributions. In practice this is done in Jupyter with pandas and matplotlib, or with report generators. Unlike confirmatory data analysis, hypotheses here are born, not tested.

Example

Before a demand forecast, an analyst plots sales by week and sees a gap of a whole month. It turns out the company was changing its accounting system then, and some sales were never exported. Without this finding, the model would have decided that demand fell at that time.

Related terms

Questions and answers

Why do you need exploratory data analysis? It shows whether the data fits the task before you pay for a model. At this stage you find gaps, errors, outliers and fields that leak the answer, and you choose the modeling approach.
How is EDA different from regular reporting? A report answers a question known in advance, for example about revenue for the quarter. Exploration looks for which questions are worth asking at all and which parts of the data you can trust.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us