Evaluating Topic Modeling Interpretability Using Topic Labeled Gold-standard Sets

Publication Title

Communications of the Association for Information Systems

ISSN

15293181

Document Type

Article

Abstract

The paucity of rigorous evaluation measures undermines topic modeling results’ validity and trustworthiness. Accordingly, we propose a method that researchers can use to select models when they assess topics’ human interpretability. We show how they can evaluate different topic models using gold-standard sets that humans label. Our approach ensures that the topics extracted algorithmically from an entire corpus concur with the themes humans would have identified in the same documents. By doing so, we combine human coding’s advantages for topic interpretability with algorithmic topic Modeling’s analytical efficiency and scalability. We demonstrate that one can rigorously identify optimal model parametrizations for maximum interpretability and to rigorously justify model selection. We also contribute three open access gold-standard sets in the hospitality context and make them available so other researchers can use them to benchmark their models or validate their results. Finally, we showcase a methodology for designing and developing gold-standard sets for validating topic models, which researchers interested in developing gold-standard sets in domains and contexts appropriate for their research can use.

First Page

433

Last Page

451

Publication Date

1-1-2020

DOI

10.17705/1CAIS.04720

Keywords

Gold-standard Set, Human Interpretable Topics, Text Mining, Topic Evaluation, Topic Interpretability Measure, Topic Modeling

Department

Department of Operations Management and Information Systems (OMIS)

Link to Full Text

Share

COinS