No Access

Vocabulary-Enhanced Named Entity Recognition and its Application on Distribution Network Maintenance

Yu Wang

https://orcid.org/0009-0005-7275-7090

Nanning Power Supply Bureau of Guangxi Power Grid, Nanning 530031, Guangxi, P. R. China

E-mail Address: wangyugx@hotmail.com

Corresponding author.

Search for more papers by this author

Pei Li

https://orcid.org/0009-0002-4548-4435

Nanning Power Supply Bureau of Guangxi Power Grid, Nanning 530031, Guangxi, P. R. China

Search for more papers by this author

Xu-Chun Yang

https://orcid.org/0009-0001-0264-6070

Nanning Power Supply Bureau of Guangxi Power Grid, Nanning 530031, Guangxi, P. R. China

Search for more papers by this author

Hui Luo

https://orcid.org/0009-0009-1357-8593

Nanning Power Supply Bureau of Guangxi Power Grid, Nanning 530031, Guangxi, P. R. China

Search for more papers by this author

Xiong-Yong Jiang

https://orcid.org/0009-0007-5220-1911

Nanning Power Supply Bureau of Guangxi Power Grid, Nanning 530031, Guangxi, P. R. China

Search for more papers by this author

Kai-Yang Luo

https://orcid.org/0009-0000-5950-9950

Nanning Power Supply Bureau of Guangxi Power Grid, Nanning 530031, Guangxi, P. R. China

Search for more papers by this author

, and

Yu-Ting Tan

https://orcid.org/0009-0002-0390-1480

Nanning Power Supply Bureau of Guangxi Power Grid, Nanning 530031, Guangxi, P. R. China

Search for more papers by this author

https://doi.org/10.1142/S0218126625501002Cited by:0 (Source: Crossref)

Abstract

Named entity recognition is a crucial task in natural language processing, serving downstream tasks such as information extraction and question-answering systems. With the success of Transformer in the field of natural language processing, named entity recognition has made significant strides. However, despite these advancements, the recognition capability in specialized domains remains challenging. The distribution network system contains a vast amount of equipment maintenance text, characterized by inherent complexity and specialization, lacking effective recognition and extraction methods. To address this issue, we propose a vocabulary-enhanced named entity recognition method. Unlike the original named entity recognition methods, we employ a vocabulary adapter to enhance the understanding of text semantic information by integrating matching information between the dictionary and the text. This method strengthens the ability to extract specialized nouns. In conjunction with the TaCL-BERT pre-trained model, we conducted training on our self-constructed distribution network maintenance dataset, GNP-M, to better align the model with the context and specific entities of the electrical grid maintenance domain. Simultaneously, we compared our method with four other named entity recognition models, revealing satisfactory performance results.

This paper was recommended by Regional Editor Tongquan Wei.

Keywords:

References

1. N. Jofche, K. Mishev, R. Stojanov, M. Jovanovik, E. Zdravevski and D. Trajanov , Named entity recognition and knowledge extraction from pharmaceutical texts using transfer learning, Procedia Comput. Sci. 203 (2022) 721–726. Crossref, Google Scholar
2. Y. Wang, W. Zhao, Y. Wan, Z. Deng and P. Yu, Named entity recognition via machine reading comprehension: A multi-task learning approach, preprint (2023), arXiv:abs/2309.11027. Google Scholar
3. A. Berger and J. Lafferty , Information retrieval as statistical translation, SIGIR Forum 51 (1999) 219–226. Crossref, Google Scholar
4. Y. Li, G. Du, Y. Xiang, S. Li, L. Ma, D. Shao, X. Wang and H. Chen , Towards Chinese clinical named entity recognition by dynamic embedding using domain-specific knowledge, J. Biomed. Inform. 106 (2020) 103435. Crossref, Web of Science, Google Scholar
5. J. Zhou, L. Li, A. Vajdi, X. Zhou and Z. Wu , Temperature-constrained reliability optimization of industrial cyber-physical systems using machine learning and feedback control, IEEE Trans. Autom. Sci. Eng. 20 (2023) 20–31. Crossref, Web of Science, Google Scholar
6. J. Zhou, T. Wang, W. Jiang, H. Chai and Z. Wu , Decomposed task scheduling for security-critical mobile cyber–physical systems, IEEE Internet Things J. 9 (2022) 22280–22290. Crossref, Web of Science, Google Scholar
7. Q. Yang, J. Jiang, Y. Feng, J. He, B. Chen and Z. Zhang , Named entity recognition of power substation knowledge based on transformer-BiLSTM-CRF network, Int. Conf. Smart Grids and Energy Systems ( IEEE, 2020), pp. 952–956. Crossref, Google Scholar
8. Z. Kong, C. Yue, Y. Shi, J. Yu, C. Xie and L. Xie , Entity extraction of electrical equipment malfunction text by a hybrid natural language processing algorithm, IEEE Access 9 (2021) 40216–40226. Crossref, Web of Science, Google Scholar
9. J. Devlin, M. Chang, K. Lee and K. Toutanova , Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding ( North American Chapter of the Association for Computational Linguistics, 2019). Google Scholar
10. H. Yang, Bert meets Chinese word segmentation, preprint (2019), arXiv:abs/1909.09292. Google Scholar
11. W. Wu, Y. Meng, F. Wang, Q. Han, M. Li, X. Li, J. P. Nie, X. Sun and J. Li , Glyce: Glyph-vectors for Chinese character representations, Neural Information Processing Systems ( Curran Associates, 2019), pp. 2746–2757. Google Scholar
12. Y. Hu and S. Verberne , Named entity recognition for Chinese biomedical patents, Proc. Int. Conf. Computational Linguistics ( International Committee on Computational Linguistics, Barcelona, Spain, 2020), pp. 627–637. Crossref, Google Scholar
13. Y. Su, F. Liu, Z. Meng, L. Shu, E. Shareghi and N. Collier, Tacl: Improving BERT pre-training with token-aware contrastive learning, preprint (2021), arXiv:abs/2111.04198. Google Scholar
14. J. Li, A. Sun, J. Han and C. Li , A survey on deep learning for named entity recognition, IEEE Trans. Knowl. Data Eng. 34 (2018) 50–70. Crossref, Google Scholar
15. Y. Zhu and G. Wang , CAN-NER: Convolutional Attention Network for Chinese Named Entity Recognition, Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( Association for Computational Linguistics, Minneapolis, Minnesota, 2019), pp. 3384–3393. Google Scholar
16. A. Abacha and P. Zweigenbaum , Automatic extraction of semantic relations between medical entities: A rule based approach, J. Biomed. Semant. 2 (2011) S4. Crossref, Google Scholar
17. S. Morwal, N. Jahan, D. Chopra and M. Tech , Named entity recognition using hidden markov model (HMM), Int. J. Nat. Lang. Comput. 1 (2012) 15–23. Crossref, Google Scholar
18. G. Zhou and J. Su , Named entity recognition using an HMM-based chunk tagger, Proc. Annual Meeting of the Association for Computational Linguistics ( Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002), pp. 473–480. Google Scholar
19. J. Hammerton , Named entity recognition with long short-term memory, Proc. Conf. Natural Language Learning ( Association for Computational Linguistics, 2003), pp. 172–175. Crossref, Google Scholar
20. Y. Zhang and J. Yang , Chinese NER using lattice LSTM, Proc. Annual Meeting of the Association for Computational Linguistics ( Association for Computational Linguistics, Melbourne, Australia, 2018), pp. 1554–1564. Crossref, Google Scholar
21. F. Saad , Named entity recognition for biomedical patent text using Bi-LSTM variants, Proc. Int. Conf. Information Integration and Web-based Applications & Services ( ACM, 2019), pp. 617–621. Crossref, Google Scholar
22. E. Strubell, P. Verga, D. Belanger and A. McCallum , Fast and accurate entity recognition with iterated dilated convolutions, Proc. Conf. Empirical Methods in Natural Language Processing ( Association for Computational Linguistics, Copenhagen, Denmark, 2017), pp. 2670–2680. Crossref, Google Scholar
23. W. Liu, X. Fu, Y. Zhang and W. Xiao , Lexicon enhanced Chinese sequence labeling using BERT adapter, Proc. Annual Meeting of the Association for Computational Linguistics ( Association for Computational Linguistics), pp. 5847–5858. Google Scholar
24. A. Akbik, D. Blythe and R. Vollgraf , Contextual string embeddings for sequence labeling, Proc. Int. Conf. Computational Linguistics ( Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2018), pp. 1638–1649. Google Scholar
25. W. Hu, L. He, H. Ma, K. Wang and J. Xiao , KGNER: Improving Chinese named entity recognition by BERT infused with the knowledge graph, Appl. Sci. 12 (2022) 7702. Crossref, Google Scholar
26. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. Laroussilhe, A. Gesmundo, M. Attariyan and S. Gelly , Parameter-efficient transfer learning for NLP, Int. Conf. Machine Learning (PMLR, 2019), pp. 2790–2799. Google Scholar
27. A. Bapna and O. Firat , Simple, scalable adaptation for neural machine translation, Proc. Conf. Empirical Methods in Natural Language Processing ( Association for Computational Linguistics, Hong Kong, China, 2019), pp. 1538–1548. Crossref, Google Scholar
28. J. Pfeiffer, I. Vulic, I. Gurevych and S. Ruder , Mad-x: An adapter-based framework for multi-task cross-lingual transfer, Conf. Empirical Methods in Natural Language Processing ( Association for Computational Linguistics, 2020), pp. 7654–7673. Crossref, Google Scholar
29. R. Wang, D. Tang, N. Duan, Z. Wei, X. Huang, J. Ji, G. Cao, D. Jiang and M. Zhou , K-Adapter: Infusing knowledge into pre-trained models with adapters, Findings of the Association for Computational Linguistics, ( Association for Computational Linguistics, 2021), pp. 1405–1418. Crossref, Google Scholar
30. J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick and G. Neubig, Towards a unified view of parameter-efficient transfer learning, preprint (2021) arXiv:abs/2110.04366. Google Scholar
31. H. Ng and J. Low , Chinese part-of-speech tagging: One-at-a-time or all-at-once? Word-based or character-based? Proc. Conf. Empirical Methods in Natural Language Processing ( Association for Computational Linguistics, Barcelona, Spain, 2004), pp. 277–284. Google Scholar
32. Y. Tian, Y. Song, F. Xia, T. Zhang and Y. Wang , Improving Chinese word segmentation with wordhood memory networks, Proc. Annual Meeting of the Association for Computational Linguistics ( Association for Computational Linguistics, 2020), pp. 8274–8285. Crossref, Google Scholar
33. A. Ghaddar and P. Langlais , Robust lexical features for improved neural network named-entity recognition, Int. Conf. Computational Linguistics ( Association for Computational Linguistics, 2018), pp. 1896–1907. Google Scholar
34. W. Liu, T. Xu, Q. Xu, J. Song and Y. Zu , An encoding strategy based word-character LSTM for Chinese NER, Proc. Conf. North American Chapter of the Association for Computational Linguistics ( Association for Computational Linguistics, Minneapolis, Minnesota, 2019), pp. 2379–2389. Google Scholar
35. F. Liu, I. Vulić, A. Korhonen and N. Collier , Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders, Proc. Conf. Empirical Methods in Natural Language Processing ( Association for Computational Linguistics, Punta Cana, Dominican Republic, 2021), pp. 1442–1459. Crossref, Google Scholar
36. I. Vulić, E. Ponti, A. Korhonen and G. Glavaš , LexFit: Lexical fine-tuning of pretrained language models, Proc. Annual Meeting of the Association for Computational Linguistics ( Association for Computational Linguistics, 2021), pp. 5269–5283. Google Scholar
37. N. Peng and M. Dredze , Named entity recognition for Chinese social media with jointly trained embeddings, Proc. Conf. Empirical Methods in Natural Language Processing ( Association for Computational Linguistics, Lisbon, Portugal, 2015), pp. 548–554. Crossref, Google Scholar
38. N. Peng and M. Dredze , Improving named entity recognition for Chinese social media with word segmentation representation learning, Proc. Annual Meeting of the Association for Computational Linguistics ( Association for Computational Linguistics, Berlin, Germany, 2016), pp. 149–155. Crossref, Google Scholar
39. E. Tjong Kim Sang and F. De Meulder , Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition, Proc. Seventh Conf. Natural Language Learning ( Association for Computational Linguistics, 2003), pp. 142–147. Crossref, Google Scholar
40. S. Zhao , Named entity recognition in biomedical texts using an HMM model, Proc. Int. Joint Workshop on Natural Language Processing in Biomedicine and its Applications ( COLING, Geneva, Switzerland, 2004), pp. 87–90. Crossref, Google Scholar
41. W. Chen, Y. Zhang and H. Isahara , Chinese named entity recognition with conditional random fields, Proc. Fifth SIGHAN Workshop on Chinese Language Processing ( Association for Computational Linguistics, Sydney, Australia, 2006), pp. 118–121. Google Scholar
42. Z. Huang, W. Xu and K. Yu, Bidirectional LSTM-CRF models for sequence tagging, preprint (2015) arXiv:abs/1508.01991. Google Scholar
43. L. Xu, S. Li, Y. Wang and L. Xu , Named entity recognition of BERT-BiLSTM-CRF combined with self-attention, Web Information System and Application Conf. ( Springer, 2021), pp. 556–564. Crossref, Google Scholar
44. H. He and X. Sun , F-score driven max margin neural network for named entity recognition in Chinese social media, Proc. Conf. European Chapter of the Association for Computational Linguistics ( Association for Computational Linguistics, Valencia, Spain, 2017), pp. 713–718. Crossref, Google Scholar