In statistics, exploratory data analysis is an approach to analyzing data sets to
summarize their main characteristics, often with visual methods. A statistical
model can be used or not, but primarily EDA is for seeing what the data can tell
us beyond the formal modeling or hypothesis testing task. Exploratory
data analysis was promoted by John Tukey to encourage statisticians to explore
the data, and possibly formulate hypotheses that could lead to new data
collection and experiments. EDA is different from initial data analysis,
which focuses more narrowly on checking assumptions required for model fitting
and hypothesis testing, and handling missing values and making
transformations of variables as needed. EDA encompasses IDA.
Overview Tukey defined data analysis in 1961 as:
"[P]rocedures for analyzing data, techniques for interpreting the results
of such procedures, ways of planning the gathering of data to make its analysis
easier, more precise or more accurate, and all the machinery and results of
statistics which apply to analyzing data."
Tukey's championing of EDA encouraged the development of statistical computing
packages, especially S at Bell Labs. The S programming language inspired the
systems 'S'-PLUS and R. This family of statistical-computing environments
featured vastly improved dynamic visualization capabilities, which
allowed statisticians to identify outliers, trends and patterns in data
that merited further study. Tukey's EDA was related to two other
developments in statistical theory: Robust statistics and nonparametric
statistics, both of which tried to reduce the sensitivity of statistical
inferences to errors in formulating statistical models. Tukey promoted the
use of five number summary of numerical data—the two extremes, the median, and
the quartiles—because these median and quartiles, being functions of the
empirical distribution are defined for all distributions, unlike the mean and
standard deviation; moreover, the quartiles and median are more robust to
skewed or heavy-tailed distributions than traditional summaries. The packages
S, S-PLUS, and R included routines using resampling statistics, such as
Quenouille and Tukey's jackknife and Efron '​s bootstrap, which are
nonparametric and robust. Exploratory data analysis, robust
statistics, nonparametric statistics, and the development of statistical
programming languages facilitated statisticians' work on scientific and
engineering problems. Such problems included the fabrication of
semiconductors and the understanding of communications networks, which concerned
Bell Labs. These statistical developments, all championed by Tukey,
were designed to complement the analytic theory of testing statistical
hypotheses, particularly the Laplacian tradition's emphasis on exponential
families. EDA development
John W. Tukey wrote the book "Exploratory Data Analysis" in 1977.
Tukey held that too much emphasis in statistics was placed on statistical
hypothesis testing; more emphasis needed to be placed on using data to suggest
hypotheses to test. In particular, he held that confusing the two types of
analyses and employing them on the same set of data can lead to systematic bias
owing to the issues inherent in testing hypotheses suggested by the data.
The objectives of EDA are to: Suggest hypotheses about the causes of
observed phenomena Assess assumptions on which statistical
inference will be based Support the selection of appropriate
statistical tools and techniques Provide a basis for further data
collection through surveys or experiments
Many EDA techniques have been adopted into data mining, as well as into big
data analytics. They are also being taught to young students as a way to
introduce them to statistical thinking. Techniques
There are a number of tools that are useful for EDA, but EDA is characterized
more by the attitude taken than by particular techniques.
Typical graphical techniques used in EDA are:
Box plot Histogram
Multi-vari chart Run chart
Pareto chart Scatter plot
Stem-and-leaf plot Parallel coordinates
Odds ratio Multidimensional scaling
Targeted projection pursuit Principal component analysis
Multilinear PCA Projection methods such as grand tour,
guided tour and manual tour Interactive versions of these plots
Typical quantitative techniques are: Median polish
Trimean Ordination
History Many EDA ideas can be traced back to
earlier authors, for example: Francis Galton emphasized order
statistics and quantiles. Arthur Lyon Bowley used precursors of
the stemplot and five-number summary, p. 62 – he defines "the maximum and
minimum, median, quartiles and two deciles" as the "seven positions").
Andrew Ehrenberg articulated a philosophy of data reduction.
The Open University course Statistics in Society, took the above ideas and merged
them with Gottfried Noether's work, which introduced statistical inference
via coin-tossing and the median test. Example
Findings from EDA are often orthogonal to the primary analysis task. This is an
example, described in more detail in. The analysis task is to find the
variables which best predict the tip that a dining party will give to the
waiter. The variables available are tip, total bill, gender, smoking status, time
of day, day of the week and size of the party. The analysis task requires that a
regression model be fit with either tip or tip rate as the response variable.
The fitted model is tip rate = 0.18 - 0.01×size
which says that as the size of the dining party increase by one person tip
will decrease by 1%. Making plots of the data reveals other interesting features
not described by this model. What is learned from the graphics is
different from what could be learned by the modeling. You can say that these
pictures help the data tell us a story, that we have discovered some features of
tipping that perhaps we didn't anticipate in advance.
Software GGobi is a free software for interactive
data visualization data visualization CMU-DAP.
Data Applied, a comprehensive web-based data visualization and data mining
environment. High-D for multivariate analysis using
parallel coordinates. JMP, an EDA package from SAS Institute.
KNIME Konstanz Information Miner – Open-Source data exploration platform
based on Eclipse. Orange, an open-source data mining
software suite. SOCR provides a large number of free
Internet-accessible. TinkerPlots.
Weka an open source data mining package that includes visualisation and EDA
tools such as targeted projection pursuit
See also Anscombe's quartet, on importance of
exploration Predictive analytics
Structured data analysis Configural frequency analysis
References Bibliography
Andrienko, N & Andrienko, G Exploratory Analysis of Spatial and Temporal Data. A
Systematic Approach. Springer. ISBN 3-540-25994-5
Cook, D. and Swayne, D.F.. Interactive and Dynamic Graphics for Data Analysis:
With R and GGobi. Springer. ISBN 9780387717616.
Hoaglin, D C; Mosteller, F & Tukey, John Wilder. Exploring Data Tables, Trends
and Shapes. ISBN 0-471-09776-4. Hoaglin, D C; Mosteller, F & Tukey, John
Wilder. Understanding Robust and Exploratory Data Analysis. ISBN
0-471-09777-2. Inselberg, Alfred. Parallel
Coordinates:Visual Multidimensional Geometry and its Applications. London
New York: Springer. ISBN 978-0-387-68628-8.
Leinhardt, G., Leinhardt, S., Exploratory Data Analysis: New Tools for
the Analysis of Empirical Data, Review of Research in Education, Vol. 8, 1980,
pp. 85–157. Martinez, W. L., Martinez, A. R., and
Solka, J.. Exploratory Data Analysis with MATLAB, second edition. Chapman &
Hall/CRC. ISBN 9781439812204. Theus, M., Urbanek, S., Interactive
Graphics for Data Analysis: Principles and Examples, CRC Press, Boca Raton, FL,
ISBN 978-1-58488-594-8 Tucker, L; MacCallum, R.. Exploratory
Factor Analysis. [1]. Tukey, John Wilder. Exploratory Data
Analysis. Addison-Wesley. ISBN 0-201-07616-0.
Velleman, P. F.; Hoaglin, D. C.. Applications, Basics and Computing of
Exploratory Data Analysis. ISBN 0-87150-409-X.
Young, F. W. Valero-Mora, P. and Friendly M. Visual Statistics: Seeing
your data with Dynamic Interactive Graphics. Wiley ISBN 978-0-471-68160-1
External links Carnegie Mellon University – free online
course on EDA