Experimental design under the Bradley-Terry model

Yuan Guo, Peng Tian, Jayashree Kalpathy-Cramer, Susan Ostmo, J. Peter Campbell, Michael F. Chiang, Deniz Erdogmu ş, Jennifer Dy, Stratis Ioannidis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

Labels generated by human experts via comparisons exhibit smaller variance compared to traditional sample labels. Collecting comparison labels is challenging over large datasets, as the number of comparisons grows quadratically with the dataset size. We study the following experimental design problem: given a budget of expert comparisons, and a set of existing sample labels, we determine the comparison labels to collect that lead to the highest classification improvement. We study several experimental design objectives motivated by the Bradley-Terry model. The resulting optimization problems amount to maximizing submodular functions. We experimentally evaluate the performance of these methods over synthetic and real-life datasets.

Original languageEnglish (US)
Title of host publicationProceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018
EditorsJerome Lang
PublisherInternational Joint Conferences on Artificial Intelligence
Pages2198-2204
Number of pages7
ISBN (Electronic)9780999241127
DOIs
StatePublished - Jan 1 2018
Event27th International Joint Conference on Artificial Intelligence, IJCAI 2018 - Stockholm, Sweden
Duration: Jul 13 2018Jul 19 2018

Publication series

NameIJCAI International Joint Conference on Artificial Intelligence
Volume2018-July
ISSN (Print)1045-0823

Other

Other27th International Joint Conference on Artificial Intelligence, IJCAI 2018
CountrySweden
CityStockholm
Period7/13/187/19/18

    Fingerprint

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Guo, Y., Tian, P., Kalpathy-Cramer, J., Ostmo, S., Peter Campbell, J., Chiang, M. F., Erdogmu ş, D., Dy, J., & Ioannidis, S. (2018). Experimental design under the Bradley-Terry model. In J. Lang (Ed.), Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018 (pp. 2198-2204). (IJCAI International Joint Conference on Artificial Intelligence; Vol. 2018-July). International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2018/304