Exploratory Data Analysis with Python

The Dataset Looked Clean Until I Asked Better Questions

The nutritional dataset I used in the UT Austin AI/ML program arrived in a familiar shape: rows of foods, columns of nutrients, and enough numeric values to make modeling feel like the obvious next step. It loaded into a pandas DataFrame without complaint. Nothing about it looked broken.

That was precisely the risk.

A dataset does not need malformed CSV or missing headers to mislead an analysis. A zero may mean “none,” “not measured,” or “unknown.” Two food names may describe the same product at different serving sizes. A nutrient measured in milligrams can dominate a feature expressed in grams once the values reach a distance-based model. The file can be syntactically valid while the assumptions underneath it remain unsettled.

Exploratory data analysis became useful here because it slowed down the rush toward modeling. The goal was not to produce every available chart. It was to identify which decisions I would otherwise make silently.

Start With the Shape, Then Challenge the Meaning

I worked in Google Colab with pandas, Matplotlib, and seaborn. The dataset included food descriptions and nutritional features such as protein, fat, vitamin C, and fiber. The full notebook and supporting files are available in my Google Drive folder.

My first pass was deliberately plain:

df.shape
df.head()
df.info()
df.describe(include="all").transpose()
df.isna().sum().sort_values(ascending=False)
df.duplicated().sum()

These commands answer different questions. shape and head confirm that I loaded what I expected. info exposes types and nullability. describe makes improbable ranges visible. Missing-value and duplicate counts identify where absence or repetition might alter the analysis.

The commands are simple; interpreting them is not. A duplicate row could be accidental, or it could represent two legitimate products with identical measured nutrients. A zero vitamin value could be real, or it could be the dataset's substitute for an unavailable measurement. Removing or imputing either one without checking the data dictionary would turn uncertainty into a fact.

Missing Values Are Domain Decisions

The usual menu for missing data is short: remove the row, fill the value, or preserve the absence as a signal. None of those choices is universally correct.

In nutritional data, replacing a missing nutrient with zero is especially tempting and especially dangerous. Zero states that the food contains none of that nutrient. Missing states that the dataset does not tell me. A model cannot recover that distinction after I erase it.

I started by making the missingness visible alongside the affected records:

missing_counts = df.isna().sum()
missing_columns = missing_counts[missing_counts > 0].index

for column in missing_columns:
    display(df.loc[df[column].isna(), ["food_name", column]].head())

That inspection changed the question from “How do I fill nulls?” to “Why is this feature missing for these foods?” Sometimes the answer supports imputation. Sometimes it supports dropping a feature. Sometimes the absence itself deserves a separate indicator. The important part is that the choice becomes explicit before it influences every later result.

Distributions Expose What Averages Hide

Summary statistics gave me a useful inventory, but nutrient features were heavily skewed. A small number of foods carried values far beyond the median, and those values pulled means away from what a typical row looked like.

I used the same compact routine across numerical columns to compare the distribution, center, and potential outliers:

def inspect_numeric_features(df, features):
    for feature in features:
        skewness = df[feature].skew()
        minimum = df[feature].min()
        maximum = df[feature].max()
        mean = df[feature].mean()
        mode = df[feature].mode().values[0]
        unique_count = df[feature].nunique()
        variance = df[feature].var()
        std_dev = df[feature].std()
        percentile_25 = df[feature].quantile(0.25)
        median = df[feature].median()
        percentile_75 = df[feature].quantile(0.75)
        data_range = maximum - minimum

        print({
            "feature": feature,
            "skewness": round(skewness, 4),
            "min": minimum,
            "max": maximum,
            "mean": round(mean, 4),
            "mode": mode,
            "unique_count": unique_count,
            "variance": round(variance, 4),
            "std_dev": round(std_dev, 4),
            "p25": percentile_25,
            "median": median,
            "p75": percentile_75,
            "range": data_range,
        })

        plt.figure(figsize=(18, 6))
        plt.subplot(1, 3, 1)
        sns.kdeplot(df[feature], fill=True)
        plt.title(f"KDE of {feature}")
        plt.subplot(1, 3, 2)
        sns.boxplot(df[feature])
        plt.title(f"Box Plot of {feature}")
        plt.subplot(1, 3, 3)
        sns.histplot(df[feature], bins=10, kde=True)
        plt.title(f"Histogram of {feature}")
        plt.tight_layout()
        plt.show()

The three views serve different purposes. The histogram shows where most values sit. The KDE makes the distribution's shape easier to compare across features. The box plot makes extreme observations difficult to ignore.

An outlier is not automatically bad data. In this dataset, a high nutrient value might be a unit error, a fortified product, or a genuinely unusual food. I treated the chart as a queue for investigation rather than permission to delete points.

Relationships Need a Question

It is easy to generate a correlation heatmap across every numeric feature. It is harder to decide which relationships matter.

For the nutritional analysis, I was preparing for a clustering experiment. That made scale, redundancy, and feature dependence more important than a generic search for strong correlations. If two columns carried nearly the same signal, including both could give that concept extra influence. If one feature had a much larger numeric range, it could dominate Euclidean distance even when it was not more important.

numeric = df.select_dtypes(include="number")
correlations = numeric.corr()

plt.figure(figsize=(12, 9))
sns.heatmap(correlations, cmap="coolwarm", center=0)
plt.title("Nutrient Feature Correlations")
plt.tight_layout()
plt.show()

The heatmap did not tell me which features to keep. It showed where I needed to make and document a choice. That distinction matters: visualization supports judgment; it does not replace it.

Automation Should Preserve Curiosity

Reusable inspection functions made the notebook faster and more consistent, but there is a trap in automating EDA. A function that emits thirty charts can create the appearance of thoroughness while making it easier to skim past the one chart that challenges the model design.

I found the better balance was to automate repetitive calculation and layout, then pause at each suspicious distribution or relationship. The notebook remained reproducible without turning exploration into a report generator.

This also changed how I thought about the next phase. Before EDA, K-means looked like a straightforward algorithm choice. After EDA, the important modeling questions were more concrete: Which features should be scaled? Which missing values could be defended? Were the apparent clusters driven by nutrition or by measurement units?

Those questions led directly to Exploring Nutritional Data with K-means Clustering. The clustering article begins where this one ends—not with a perfectly clean dataset, but with a documented set of decisions and uncertainties.

EDA is often described as a preliminary step. In practice, I keep returning to it whenever a model produces a surprising result. The charts and statistics are useful, but the real product is a better account of what I believe about the data and why. A model built after that work may still be wrong. At least its assumptions are visible enough to challenge.

Explore More

• When the First Visualization Answers the Wrong Question -- Why choosing a chart is part of the analysis
• Python: The Language of Data Science -- Understanding Python's Impact on Data Science
• Data Science for .NET Developers -- Why .NET Developers Should Consider Data Science
• Exploring Nutritional Data Using K-means Clustering -- Unveiling Patterns in Nutritional Data
• Understanding Neural Networks -- A Beginner's Guide to Neural Networks