An insurance team has a queue of unlabeled claims and limited reviewer time. The team wants to choose which claims reviewers should label next for a classification model.
In his active learning literature survey, published in 2009 as University of Wisconsin-Madison Computer Sciences Technical Report 1648, Burr Settles describes a learner selecting data for an oracle, such as a human annotator, to label. The survey presents uncertainty sampling as a query strategy and explains that selecting examples can improve accuracy with fewer training labels.
For an enterprise task, a reviewer might examine claims the current model finds difficult to classify. The team can add approved labels to training data, retrain the model and check performance on a separate evaluation set.
A labeling workflow needs decisions beyond the selection rule:
- Define the task, label meanings and examples reviewers can use.
- Select unlabeled records using a documented query strategy.
- Check reviewer agreement and investigate uncertain or disputed labels.
- Track selection and labeling decisions so the training data can be reviewed later.
How is Active Learning different from random labeling?
- Active learning uses a selection strategy informed by the learner or task to request labels.
- Random labeling selects examples without using the model’s assessment of their value.
- Semi-supervised learning uses labeled and unlabeled data during learning; active learning specifically requests labels for selected examples. The approaches can be combined.
Where does Active Learning break in AI work?
A selection strategy focused on difficult cases can produce a labeled set that differs from the population the model will encounter. Maintain a separate evaluation set selected for the intended use. Actively selected training records do not make a fair test.
Rushed review can also introduce label noise. Review instructions, disagreement checks and a route for correcting labels help the team assess whether accepted labels are suitable as ground truth.