Chapter 8: Crowdsourcing in Single-document Summary Evaluation: The Argo Way

In this chapter, we present the Argo crowdsourcing system and related functionalities for crowd-based summary evaluation. Then, we evaluate the pros and cons of three different approaches to single-document, multilingual summary evaluation: i) traditional, expert-based; ii) crowdsourcing, majority-based; and iii) crowdsourcing, Argo-based. The evaluation is performed over two languages of the MultiLing-2015 single document summarization (MSS) dataset, examining the result under different aspects.