Handle Missing Values in pandas Without Inventing Data

A missing quantity and a quantity of zero have different meanings. Replacing every blank with zero may make a chart look cleaner while changing the result. Before choosing fillna or dropna, decide what the missing value means and whether the analysis can remain valid without it.

Use a nullable type deliberately

pandas supports nullable integer types such as Int64, with a capital I. They allow a column to contain integers and missing values without requiring an ordinary floating-point representation. The type alone does not solve data quality; it makes the distinction representable.

import pandas as pd

quantities = pd.Series([2, None, 0], dtype="Int64")
assert quantities.isna().sum() == 1
assert quantities.notna().sum() == 2
assert quantities.sum() == 2

all_missing = pd.Series([None, None], dtype="Int64")
assert pd.isna(all_missing.sum(min_count=1))

Understand aggregation defaults

Many aggregations skip missing values. That can be useful, but the result should not imply all records were observed. The min_count argument lets the example preserve missingness when there are no valid inputs, instead of presenting an all-missing collection as a meaningful zero total.

Measure completeness alongside the result

Publish a non-missing count or completeness rate with important totals. If one region has complete data and another is missing half its records, comparing totals without that context may be misleading. Completeness should be measured at the grain relevant to the question, not merely across the entire file.

Choose a documented treatment

Fill a missing value only when a defensible rule supports the replacement. For example, a default label for an unknown category may be reasonable, while a guessed revenue value changes a financial claim. Dropping rows also changes the population under analysis; record how many rows were removed and why.

Check missing markers at ingestion

An empty field, the string NA, a dash and a literal zero are not always interchangeable. Configure import rules for your source and inspect their effect. Use isna rather than equality comparisons with missing values. Keep raw input available so cleaning decisions can be revisited.

Practice and check your understanding

Build a four-row dataset containing a real zero, a positive number, a missing value and a string marker such as unknown. Define the conversion policy, then report total, accepted count and rejected count without silently inventing a quantity.

Continue learning

Explore the related Code Hacker course syllabus for structured learning. Then read Python CSV Validation: Build a Row-by-Row Error Report to connect this exercise to the next skill.

Reference

pandas missing data guide provides the underlying API or technical guidance. Check the documentation for the version you use; examples here are small learning exercises rather than complete production implementations.