Weighting Areas Under the Margin in crowdsourced datasets

Tanguy Lefort (1, 2, 3, 4, 5) , Benjamin Charlier (2, 3, 4) , Alexis Joly (5) , Joseph Salmon (3, 2, 6, 4)

1 IMAG
2 UM - Université de Montpellier
3 CNRS - Centre National de la Recherche Scientifique
4 IMAG - Institut Montpelliérain Alexander Grothendieck
5 ZENITH - Scientific Data Management
6 IUF - Institut universitaire de France

In supervised learning — for instance in image classification — modern massive datasets are commonly labeled by a crowd of workers. Labeling errors can happen because of the workers abilities or tasks identification difficulty. Some intrinsically ambiguous tasks might fool expert workers, which could eventually be harmful to the learning step. In a standard supervised learning setting — with one label per task — the Area Under the Margin (AUM) is tailored to identify mislabeled data. We adapt the AUM to identify ambiguous tasks in crowdsourced learning scenarios, introducing the Weighted AUM (WAUM). The WAUM is an average of AUMs weighted by task-dependent scores. We show that the WAUM can help discard ambiguous tasks from the training set, leading to better generalization or calibration performance.

Mots clés: Crowdsourcing; Label ambiguity

Domaines: Interface homme-machine [cs.HC]

Tanguy Lefort, Benjamin Charlier, Alexis Joly, Joseph Salmon. Weighting Areas Under the Margin in crowdsourced datasets. JDS 2023 - 54es Journées de Statistique, Société Française de Statistique (SFdS), Jul 2023, Bruxelles, Belgium. ⟨hal-04562515⟩