KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Korean Machine Reading Comprehension for Patent Consultation Using BERT

BERT를 이용한 한국어 특허상담 기계독해

  • 민재옥 (한국특허정보원 R&D센터 연구개발파트) ;
  • 박진우 (한국특허정보원 R&D센터) ;
  • 조유정 (한국특허정보원 R&D센터) ;
  • 이봉건 (한국특허정보원 특허넷응용팀 특허넷응용팀)
  • Received : 2019.12.30
  • Accepted : 2020.02.18
  • Published : 2020.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

MRC (Machine reading comprehension) is the AI NLP task that predict the answer for user's query by understanding of the relevant document and which can be used in automated consult services such as chatbots. Recently, the BERT (Pre-training of Deep Bidirectional Transformers for Language Understanding) model, which shows high performance in various fields of natural language processing, have two phases. First phase is Pre-training the big data of each domain. And second phase is fine-tuning the model for solving each NLP tasks as a prediction. In this paper, we have made the Patent MRC dataset and shown that how to build the patent consultation training data for MRC task. And we propose the method to improve the performance of the MRC task using the Pre-trained Patent-BERT model by the patent consultation corpus and the language processing algorithm suitable for the machine learning of the patent counseling data. As a result of experiment, we show that the performance of the method proposed in this paper is improved to answer the patent counseling query.

기계독해는(Machine reading comprehension) 사용자 질의와 관련된 문서를 기계가 이해한 후 정답을 추론하는 인공지능 자연어처리 태스크를 말하며, 이러한 기계독해는 챗봇과 같은 자동상담 서비스에 활용될 수 있다. 최근 자연어처리 분야에서 가장 높은 성능을 보이고 있는 BERT 언어모델은 대용량의 데이터를 pre-training 한 후에 각 자연어처리 태스크에 대해 fine-tuning하여 학습된 모델로 추론함으로써 문제를 해결하는 방식이다. 본 논문에서는 BERT기반 특허상담 기계독해 태스크를 위해 특허상담 데이터 셋을 구축하고 그 구축 방법을 소개하며, patent 코퍼스를 pre-training한 Patent-BERT 모델과 특허상담 모델학습에 적합한 언어처리 알고리즘을 추가함으로써 특허상담 기계독해 태스크의 성능을 향상시킬 수 있는 방안을 제안한다. 본 논문에서 제안한 방법을 사용하여 특허상담 질의에 대한 정답 결정에서 성능이 향상됨을 보였다.

Keywords

References

  1. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "Squad: 100,000+ questions for machine comprehension of text," arXiv preprint arXiv:1606.05250, 2016.
  2. S. Lim, M. Kim, and J. Lee, "KorQuAD: Korean QA Dataset for Machine Comprehension," in Proceedings of the Korea Software Congress 2018, pp.539-541, 2018.
  3. D. Jacob, C. Ming-Wei, L. Kenton, and T. Kristina, "Bert: pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018.
  4. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, "Glue: A multi-task benchmark and analysis platform for natural language understanding," arXiv preprint arXiv:1804.07461, 2018.
  5. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, and L. Kaiser, "Attention is all you need," Advances in Neural Information Processing Systems. 2017.
  6. K. H. Park, S. H. Na, Y.S. Choi, and D. S. Chang, "BERT and Multi-level Co-Attention Fusion for Machine Reading Comprehension," in Proceedings of the Korea Software Congress 2019, pp.643-645, 2019.
  7. D. Lee, C. Park, C. Lee, S. Park, S. Lim, M. Kim, and J. Lee, "Korean Machine Reading Comprehension using BERT," in Proceedings of the Korea Computer Congress 2019, pp.557-559, 2019.
  8. T. Lei, Y. Zhang, S.I. Wang, H. Dai, and Y. Artzi. "Simple Recurrent Units for highly Parallelizable Recurrence," arXiv:1709.02755v5, 2018.
  9. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, "XLNet: Generalized Autoregressive Pretraining for Language Understanding," arXiv preprint arXiv: 1906.08237, 2019.
  10. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "Albert: A lite bert for self-supervised learning of language representations," arXiv preprint arXiv:1909.11942, 2019.
  11. Y. Wu, M. Schuster, Z. Chen, Q, V. Le, and M. Norouzi, "Google's neural machine translation system: Bridging the gap between human and machine translation," arXiv preprint arXiv:1609.08144, 2016.
  12. D. P. Kingma and J. L. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.