Score Normalization and Aggregation for Active Learning in Multi-label Classification

Active learning is useful in situations where labeled data is scarce, unlabeled data is available, and labeling a large number of examples is costly or impractical. These techniques help by identifying a minimal set of examples to label that will support the training of an effective classifier. Thus active learning is particularly relevant for the automation of annotation tasks in multimedia. In this paper we consider the problem of employing active learning for the assignment of multiple annotations or “tags” to images in personal image collections. This form of multi-label classification has received a lot of attention in recent years, however active multi-label classification is still a new research area. The main challenge in active multilabel classification is the selection of unlabeled examples that will be informative for all tags under consideration. This selection task proves surprisingly difficult primarily because of the paucity of labeled data available. In this paper we present some solutions to this problem based on aggregated rankings from classifiers for individual tags.