How it works
A system trained on or built around historical data reproduces the patterns in that data, including its inequities. The bias is in the data and the framing, not in the arithmetic.
Trace a disparate outcome back to the data or the objective that produced it.
Common error
Attributing bias to the algorithm as though it were a coding error. It is usually a data and design consequence.
Worked example
Example
System accuracy: 90% overall
Group A: 98 correct of 100
Group B: 82 correct of 100
Same decision affects both groupsThe overall 90% value hides a substantial difference between the two reported groups. Investigate data quality, coverage, objectives, and consequences rather than assuming aggregate accuracy proves equitable performance.
Exercises
9.1.1 · Explain
Why can one overall error rate conceal an important distribution problem?
Show solution
Groups may experience different error rates or consequences, especially when large groups dominate the average. Review relevant breakdowns while protecting privacy.
9.1.2 · Explain
Name a useful evaluation change before broad deployment of this system.
Show solution
Evaluate representative groups and failure types, define acceptable behavior for the context, and provide human review or correction for harmful errors.