No Access

Summary Augmenter: A Text Augmentation Framework to Improve Summarization Quality

Ala Alam Falaki

https://orcid.org/0000-0003-0131-2543

School of Computer Science, University of Windsor, Windsor, Ontario N9B3P4, Canada

E-mail Address: alamfal@uwindsor.ca

Corresponding author.

Search for more papers by this author

and

Robin Gras

School of Computer Science, University of Windsor, Windsor, Ontario N9B3P4, Canada

E-mail Address: rgras@uwindsor.ca

Search for more papers by this author

https://doi.org/10.1142/S0218213024500052Cited by:0 (Source: Crossref)

Abstract

Data augmentation in Natural Language Processing (NLP) faces various challenges that hinder its widespread adoption, unlike its ever-present usage in the field of vision. It is even more the case for the text summarization task where one should focus on both article and summary. In this paper, we review the effect of back translation augmentation, present the diverse beam search decoding strategy, and masking as a method to generate synthetic data for text summarization. The approaches will be evaluated by ROUGE score, novelty, summary length, and GPT-4 to analyze their effectiveness. Our proposed framework presents multiple combinations of back translation and masking for articles, along with diverse augmentation for summaries. Although applicable to networks of any size, we decided to use BART-large, a relatively smaller model, in order to conduct a larger number of experiments. The experiments demonstrated superior performance across all specified metrics when compared to fine-tuning BART-large on the CNN/Dailymail dataset. Specifically, we showed a significant improvement in novelty; 158% and 56% increase rate for bigrams and unigrams, respectively. It could eliminate some copyright concerns around generating content similar to human writing. Additionally, the GPT-4 assessment indicates that models trained using the augmentation technique tend to capture important information more effectively than the baseline model.

Keywords:

Remember to check out the Most Cited Articles!
Check out Notable Titles in Artificial Intelligence.

References

1. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner , Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11) (1998) 2278–2324. Crossref, Web of Science, Google Scholar
2. R. Sennrich, B. Haddow and A. Birch , Improving neural machine translation models with monolingual data, in Proc. of the 54th Annual Meeting of the Association for Computational Linguistics, Vol. 1 (Association for Computational Linguistics, 2016), pp. 86–96. Crossref, Google Scholar
3. A. Fabbri, S. Han, H. Li, H. Li, M. Ghazvininejad, S. Joty, D. Radev and Y. Mehdad , Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation, in Proc. of the 2021 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Association for Computational Linguistics, 2021), pp. 704–717. Crossref, Google Scholar
4. A. W. Yu, D. Dohan, Q. Le, T. Luong, R. Zhao and K. Chen , Fast and accurate reading comprehension by combining self-attention and convolution, in Int. Conf. on Learning Representations, Vol. 2, May 2018. Google Scholar
5. K. Kafle, M. Yousefhussien and C. Kanan , Data augmentation for visual question answering, in Proc. of the 10th Int. Conf. on Natural Language Generation, September 2017, pp. 198–202. Google Scholar
6. S. Hochreiter and J. Schmidhuber , Long short-term memory, Neural Computation 9(8) (1997) 1735–1780. Crossref, Web of Science, Google Scholar
7. X. Wu, S. Lv, L. Zang, J. Han and S. Hu , Conditional BERT contextual augmentation, in 19th Int. Conf. on Computational Science (CCS 2019), Proc., Part IV, Faro, Portugal (Springer International Publishing, 2019), pp. 84–95. Crossref, Google Scholar
8. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever , Language models are unsupervised multitask learners, OpenAI Blog 1(8) (2019) 9. Google Scholar
9. J. D. M. W. C. Kenton and L. K. Toutanova , BERT: Pre-training of deep bidirectional transformers for language understanding, in Proc. of NAACL-HLT, Vol. 1, June 2019, p. 2. Google Scholar
10. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov and L. Zettlemoyer , BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, in Proc. of the 58th Annual Meeting of the Association for Computational Linguistics (Association for Computational Linguistics, 2020), pp. 7871–7880. Crossref, Google Scholar
11. S. A. Somayajula, L. Song and P. Xie , A multi-level optimization framework for end-to-end text augmentation, Transactions of the Association for Computational Linguistics 10 (2022) 343–358. Crossref, Google Scholar
12. Y. Gal and Z. Ghahramani , A theoretically grounded application of dropout in recurrent neural networks, in Advances in Neural Information Processing Systems, Vol. 29 (2016). Google Scholar
13. W. Y. Wang and D. Yang , That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets, in Proc. of the 2015 Conf. on Empirical Methods in Natural Language Processing, September 2015, pp. 2557–2563. Google Scholar
14. K. Dhole et al., NL-Augmenter: A framework for task-sensitive natural language augmentation, arXiv e-prints, arXiv:2112.02721 (2021). Google Scholar
15. M. Loem, S. Takase, M. Kaneko and N. Okazaki , ExtraPhrase: Efficient data augmentation for abstractive summarization, in Proc. of the 2022 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop (Association for Computational Linguistics, 2022), pp. 16–24. Crossref, Google Scholar
16. S. Parida and P. Motlicek , Abstract text summarization: A low resource challenge, in Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Int. Joint Conf. on Natural Language Processing (EMNLP-IJCNLP), November 2019, pp. 5994–5998. Google Scholar
17. A. Laifa, C. Cruz and L. Gautier , Data augmentation impact on domain-specific text summarization, in French Regional Conf. on Complex Systems, Vol. 2021 (2021). Google Scholar
18. L. Martin, B. Muller, P. Ortiz Suarez, Y. Dupont, L. Romary, E. Clergerie, D. Seddah and B. Sagot , CamemBERT: A tasty french language model, in Proc. of the 58th Annual Meeting of the Association for Computational Linguistics (Association for Computational Linguistics, 2020), pp. 7203–7219. Crossref, Google Scholar
19. A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. J. Crandall and D. Batra, Diverse beam search: Decoding diverse solutions from neural sequence models, arXiv:abs/1610.02424 (2016). Google Scholar
20. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez and I. Polosukhin , Attention is all you need, in Advances in Neural Information Processing Systems, Vol. 30 (2017). Google Scholar
21. J. Tiedemann , The Tatoeba translation challenge — realistic data sets for low resource and multilingual MT, in Proc. of the Fifth Conf. on Machine Translation (Association for Computational Linguistics, 2020), pp. 1174–1182. Google Scholar
22. A. Graves, Sequence transduction with recurrent neural networks, arXiv:abs/1211.3711 (2012). Google Scholar
23. Y. Liu, P. Liu, D. Radev and G. Neubig , BRIO: Bringing order to abstractive summarization, in Proc. of the 60th Annual Meeting of the Association for Computational Linguistics, Vol. 1 (Association for Computational Linguistics, 2022), pp. 2890–2903. Crossref, Google Scholar
24. C. Y. Lin , ROUGE: A package for automatic evaluation of summaries, in Text Summarization Branches Out (Association for Computational Linguistics, 2004), pp. 74–81. Google Scholar
25. M. Gao, J. Ruan, R. Sun, X. Yin, S. Yang and X. Wan, Human-like summarization evaluation with chatgpt, arXiv preprint, arXiv:2304.02554 (2023). Google Scholar
26. K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman and P. Blunsom , Teaching machines to read and comprehend, in Advances in Neural Information Processing Systems, Vol. 28 (2015). Google Scholar
27. A. See, P. J. Liu and C. D. Manning , ‘Get to the Point: Summarization with pointer-generator networks, in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 1 (Association for Computational Linguistics, Vancouver, Canada, 2017), pp. 1073–1083, https://doi.org/10.18653/v1/P17-1099. Google Scholar
28. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu and C. Zhu , G-Eval: NLG evaluation using Gpt-4 with better human alignment, in Proc. of the 2023 Conf. on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, 2023), pp. 2511–2522. Crossref, Google Scholar
29. J. Fu, S. K. Ng, Z. Jiang and P. Liu, Gptscore: Evaluate as you desire, arXiv preprint, arXiv:2302.04166 (2023). Google Scholar
30. Z. H. A. O. Jiachen and Z. Yao , SELF-EXPLAIN: Teaching large language models to reason complex questions by themselves, in Workshop on R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, December 2023. Google Scholar