6.3 Cleaning and Missing Data | Code Scholars Tutoring

How it works

Real data has gaps, duplicates, and wrong types. Every choice about them changes the result, so each must be deliberate and recorded.

Inspect the shape, the types, and the count of missing values before any analysis.

Common error

Dropping rows with missing values silently, which can remove exactly the group you were studying.

Worked example

These examples require NumPy or pandas, as shown by the imports.

Python

import pandas as pd
scores = pd.Series([5.0, None, 9.0])
print(int(scores.isna().sum()))
print(scores.dropna().mean())

Output

1
7.0

Missing values should not automatically become zero. Choose and document a treatment based on what missingness means and how the analysis will be used.

Exercises

6.3.1 · Explain

How would replacing the missing value with zero change the mean of 5, missing, 9?

Show solution

The observed-only mean is 7. Replacing missing with zero gives 14/3, about 4.667, asserting an unobserved zero and changing the interpretation.

6.3.2 · Write code

Count missing entries in [1.0, None, None, 4.0] using pandas.

Show solution

Use an explicit missingness check.

One valid solution

import pandas as pd
values = pd.Series([1.0, None, None, 4.0])
print(int(values.isna().sum()))

Expected output

Open Python compiler