Unit 4 · Most assessed

Model Evaluation

Whether a result can be believed. This is the unit that separates a student who can run a model from one who can report an honest finding.

What a strong answer looks like

A strong Unit 4 answer states which data the score came from, which metric it is, and what the baseline was.

Practice this unit Course overview

Topics in this unit

Splits and Validation

Know

A model must be judged on data it has not seen. A separate validation set absorbs tuning decisions so the test set stays uncontaminated.

Apply

Tune on validation, report on test, and touch the test set once.

Watch out

Tuning against the test set, which turns an unbiased estimate into an optimistic one.

Study move

Explain what each of the three splits is for in one sentence apiece.

Cross-Validation

Know

k-fold rotates the held-out portion so every example is tested once, giving a more stable estimate at k times the training cost.

Apply

Use cross-validation when the dataset is small enough that one split would be noisy.

Watch out

Believing cross-validation prevents overfitting. It estimates performance; it does not change the model.

Study move

State the cost and the benefit of 5-fold cross-validation.

The Confusion Matrix

Know

Precision asks how trustworthy a positive prediction is; recall asks how much of the positive class was found. The two move against each other as the threshold shifts.

Apply

Choose the metric from the cost of each error type in the actual application.

Watch out

Reporting accuracy on imbalanced data, where it can equal the do-nothing baseline.

Study move

For a screening test and a spam filter, say which metric matters more and why.

Bias and Variance

Know

High bias is underfitting and shows as poor performance everywhere. High variance is overfitting and shows as a gap between training and test.

Apply

Diagnose from the pair of scores rather than from either alone.

Watch out

Adding model complexity to fix underfitting when the real problem was insufficient or poor features.

Study move

Given a training and a test score, diagnose the problem and name a fix.

Emphasized in this unit

Connections and techniques that receive extra attention in this unit.

• Touching the test set once
• Choosing a metric from error costs
• Diagnosing from both scores together

Varies by course

Related topics some schools attach to this unit and others leave out. Covered on request rather than assumed.

ROC and AUC. Introduced in some sections.
Statistical significance. Usually left to a statistics course.

Mastery checklist

• Say what training, validation, and test sets are each for.
• State the cost and benefit of cross-validation.
• Choose precision or recall from an application.
• Diagnose bias versus variance from two scores.

Check yourself

• Why does tuning on the test set invalidate it?
• Does cross-validation prevent overfitting?
• When does accuracy tell you nothing?

Modeling drill

A model reports 99 percent accuracy on data that is 99 percent negative. State what you would ask for next and why.

Train/validation/testCross-validationPrecisionRecallConfusion matrixOverfittingBias-variance

Unit 3: Supervised Learning

Unit 5: Unsupervised Learning and Responsible AI