Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Aspect-Based Sentiment Analysis Using BERT: Developing Aspect Category Sentiment Classification Models

BERT를 활용한 속성기반 감성분석: 속성카테고리 감성분류 모델 개발

  • 박현정 (고려대학교 Human-inspired 복합지능연구센터) ;
  • 신경식 (이화여자대학교 경영대학)
  • Received : 2020.09.23
  • Accepted : 2020.11.23
  • Published : 2020.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Sentiment Analysis (SA) is a Natural Language Processing (NLP) task that analyzes the sentiments consumers or the public feel about an arbitrary object from written texts. Furthermore, Aspect-Based Sentiment Analysis (ABSA) is a fine-grained analysis of the sentiments towards each aspect of an object. Since having a more practical value in terms of business, ABSA is drawing attention from both academic and industrial organizations. When there is a review that says "The restaurant is expensive but the food is really fantastic", for example, the general SA evaluates the overall sentiment towards the 'restaurant' as 'positive', while ABSA identifies the restaurant's aspect 'price' as 'negative' and 'food' aspect as 'positive'. Thus, ABSA enables a more specific and effective marketing strategy. In order to perform ABSA, it is necessary to identify what are the aspect terms or aspect categories included in the text, and judge the sentiments towards them. Accordingly, there exist four main areas in ABSA; aspect term extraction, aspect category detection, Aspect Term Sentiment Classification (ATSC), and Aspect Category Sentiment Classification (ACSC). It is usually conducted by extracting aspect terms and then performing ATSC to analyze sentiments for the given aspect terms, or by extracting aspect categories and then performing ACSC to analyze sentiments for the given aspect category. Here, an aspect category is expressed in one or more aspect terms, or indirectly inferred by other words. In the preceding example sentence, 'price' and 'food' are both aspect categories, and the aspect category 'food' is expressed by the aspect term 'food' included in the review. If the review sentence includes 'pasta', 'steak', or 'grilled chicken special', these can all be aspect terms for the aspect category 'food'. As such, an aspect category referred to by one or more specific aspect terms is called an explicit aspect. On the other hand, the aspect category like 'price', which does not have any specific aspect terms but can be indirectly guessed with an emotional word 'expensive,' is called an implicit aspect. So far, the 'aspect category' has been used to avoid confusion about 'aspect term'. From now on, we will consider 'aspect category' and 'aspect' as the same concept and use the word 'aspect' more for convenience. And one thing to note is that ATSC analyzes the sentiment towards given aspect terms, so it deals only with explicit aspects, and ACSC treats not only explicit aspects but also implicit aspects. This study seeks to find answers to the following issues ignored in the previous studies when applying the BERT pre-trained language model to ACSC and derives superior ACSC models. First, is it more effective to reflect the output vector of tokens for aspect categories than to use only the final output vector of [CLS] token as a classification vector? Second, is there any performance difference between QA (Question Answering) and NLI (Natural Language Inference) types in the sentence-pair configuration of input data? Third, is there any performance difference according to the order of sentence including aspect category in the QA or NLI type sentence-pair configuration of input data? To achieve these research objectives, we implemented 12 ACSC models and conducted experiments on 4 English benchmark datasets. As a result, ACSC models that provide performance beyond the existing studies without expanding the training dataset were derived. In addition, it was found that it is more effective to reflect the output vector of the aspect category token than to use only the output vector for the [CLS] token as a classification vector. It was also found that QA type input generally provides better performance than NLI, and the order of the sentence with the aspect category in QA type is irrelevant with performance. There may be some differences depending on the characteristics of the dataset, but when using NLI type sentence-pair input, placing the sentence containing the aspect category second seems to provide better performance. The new methodology for designing the ACSC model used in this study could be similarly applied to other studies such as ATSC.

대규모 텍스트에서 관심 대상이 가지고 있는 속성들에 대한 감성을 세부적으로 분석하는 속성기반 감성분석(Aspect-Based Sentiment Analysis)은 상당한 비즈니스 가치를 제공한다. 특히, 텍스트에 속성어가 존재하는 명시적 속성뿐만 아니라 속성어가 없는 암시적 속성까지 분석 대상으로 하는 속성카테고리 감성분류(ACSC, Aspect Category Sentiment Classification)는 속성기반 감성분석에서 중요한 의미를 지니고 있다. 본 연구는 속성카테고리 감성분류에 BERT 사전훈련 언어 모델을 적용할 때 기존 연구에서 다루지 않은 다음과 같은 주요 이슈들에 대한 답을 찾고, 이를 통해 우수한 ACSC 모델 구조를 도출하고자 한다. 첫째, [CLS] 토큰의 출력 벡터만 분류벡터로 사용하기보다는 속성카테고리에 대한 토큰들의 출력 벡터를 분류벡터에 반영하면 더 나은 성능을 달성할 수 있지 않을까? 둘째, 입력 데이터의 문장-쌍(sentence-pair) 구성에서 QA(Question Answering)와 NLI(Natural Language Inference) 타입 간 성능 차이가 존재할까? 셋째, 입력 데이터의 QA 또는 NLI 타입 문장-쌍 구성에서 속성카테고리를 포함한 문장의 순서에 따른 성능 차이가 존재할까? 이러한 연구 목적을 달성하기 위해 입력 및 출력 옵션들의 조합에 따라 12가지 ACSC 모델들을 구현하고 4종 영어 벤치마크 데이터셋에 대한 실험을 통해 기존 모델 이상의 성능을 제공하는 ACSC 모델들을 도출하였다. 그리고 [CLS] 토큰에 대한 출력 벡터를 분류벡터로 사용하기 보다는 속성카테고리 토큰의 출력 벡터를 사용하거나 두 가지를 함께 사용하는 것이 더욱 효과적이고, NLI 보다는 QA 타입의 입력이 대체적으로 더 나은 성능을 제공하며, QA 타입 안에서 속성이 포함된 문장의 순서는 성능과 무관한 점 등의 유용한 시사점들을 발견하였다. 본 연구에서 사용한 ACSC 모델 디자인을 위한 방법론은 다른 연구에도 비슷하게 응용될 수 있을 것으로 기대된다.

Keywords

Acknowledgement

이 논문 또는 저서는 2017년도 대한민국 교육부와 한국연구재단의 지원을 받아 수행된 연구임(NRF-2017S1A5B5A02024287).

References

  1. Araque, O., G. Zhu, and C.A. Iglesias, "A Semantic Similarity-based Perspective of Affect Lexicons for Sentiment Analysis," Knowledge-Based Systems, Vol.165, (2019), 346-359.
  2. Chen, P., Z. Sun, L. Bing, and W. Yang, "Recurrent Attention Network on Memory for Aspect Sentiment Analysis," Proceedings of Empirical Methods on Natural Language Processing, (2017), 463-472.
  3. Davidov, D., O. Tsur, and A. Rappoport, "Enhanced Sentiment Learning Using Twitter Hashtags and Smileys," Proceedings of the 23rd International Conference on Computational Linguistics, (2010), 241-249.
  4. Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," (2018), arXiv:1810.04805.
  5. Do, H.H., PWC. Prasad, A. Maag, and A. Alsadoon, "Deep Learning for Aspect-Based Sentiment Analysis: A Comparative Review," Expert Systems with Applications, Vol.118, (2019), 272-299. https://doi.org/10.1016/j.eswa.2018.10.003
  6. Dosoula, N., R. Griep, Rick den Ridder, R. Slangen, Ruud van Luijk, K. Schouten, and F. Frasincar, "Sentiment Analysis of Multiple Implicit Features per Sentence in Consumer Review Data," Proceedings of the 12th International Baltic Conference on Databases and Information Systems, (2016), 241-254.
  7. Dragoni, M., M. Federici, and A. Rexha, "An Unsupervised Aspect Extraction Strategy for Monitoring Real-time Reviews Stream," Information Processing and Management, (2018).
  8. Gao, Z., A. Feng, X. Song, and X. Wu, "Target-Dependent Sentiment Classification with BERT," IEEE Access, Vol.7, (2019), 154290-154299. https://doi.org/10.1109/access.2019.2946594
  9. Hai, Z., K. Chang, and J.-j. Kim, "Implicit Feature Identification via Co-occurrence Association Rule Mining," Proceedings of the 12th International Conference on Computational Linguistics and Intelligent Text Processing, (2011), 393-404.
  10. Hannach, H.E. and M. Benkhalifa, "WordNet based Implicit Aspect Sentiment Analysis for Crime Identification from Twitter," International Journal of Advanced Computer Science and Applications, Vol.9, No.12(2018), 150-159.
  11. Hoang, M., Oskar Alija Bihorac, and Jacobo Rouces. "Aspect-Based Sentiment AnalysisUusing BERT," Proceedings of the 22nd Nordic Conference on Computional Linguistics, (2019), 187-196.
  12. Howard, J. and S. Ruder, "Universal Language Model Fine-tuning for Text Classification," ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), Vol.1, (2018), 328-339.
  13. Khalil, T. and S. R. El-Beltagy, "NileTMRG at SemEval-2016 Task 5: Deep Convolutional Neural Networks for Aspect Category and Sentiment Extraction," Proceedings of the 10th International Workshop on Semantic Evaluation, (2016), 271-276.
  14. Lee, S., B. Seo, and D. Park, "Development of Music Recommendation System based on Customer Sentiment Analysis," Journal of Intelligence and Information Systems, Vol. 24, No. 4 (2018a), 197-217. https://doi.org/10.13088/JIIS.2018.24.4.197
  15. Lee, S. W., C. W. Choi, D. S. Kim, W. Y. Yeo, and J. W. Kim, "Multi-Category Sentiment Analysis for Social Opinion Related to Artificial Intelligence on Social Media," Journal of Intelligence and Information Systems, Vol. 24, No. 4 (2018b), 51-66. https://doi.org/10.13088/JIIS.2018.24.4.051
  16. Li, X., L. Bing, W. Zhang and W. Lam, "Exploiting BERT for End-to-End Aspect-based Sentiment Analysis," Proceedings of the 2019 EMNLP Workshop W-NUT: The 5th Workshop on Noisy User-generated Text, (2019), 34-41.
  17. Liu, B., Sentiment Analysis and Opinion Mining, Springer, Berlin, 2012.
  18. Liu, Q., H. Zhang, Y. Zeng, Z. Huang, and Z. Wu, "Content Attention Model for Aspect Based Sentiment Analysis," Proceedings of the 2018 World Wide Web Conference, (2018), 1023-1032.
  19. Ma, D., S. Li, X. Zhang, H. Wang, "Interactive Attention Networks for Aspect-Level Sentiment Classification," Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, (2017), 4068-4074.
  20. Park, H. and K. Kim, "Sentiment Analysis of Movie Review Using Integrated CNN-LSTM Model," Journal of Intelligence and Information Systems, Vol. 25, No. 4 (2019), 141-154.
  21. Park, H., M. Song, and K. Shin, "Sentiment Analysis of Korean Reviews Using CNN: Focusing on Morpheme Embedding," Journal of Intelligence and Information Systems, Vol. 24, No. 2 (2018), 59-83. https://doi.org/10.13088/JIIS.2018.24.2.059
  22. Park, H., M. Song, and K. Shin, "Deep Learning Models and Datasets for Aspect Term Sentiment Classification: Implementing Holistic Recurrent Attention on Target-dependent Memories," Knowledge-Based Systems, Vol.187, (2020), 104825. https://doi.org/10.1016/j.knosys.2019.06.033
  23. Peng, H., Y. Ma, Y. Li, and E. Cambria, "Learning Multi-grained Aspect Target Sequence for Chinese Sentiment Analysis," Knowledge-Based Systems, Vol.148, (2018), 167-176. https://doi.org/10.1016/j.knosys.2018.02.034
  24. Peters, M., M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, "Deep Contextualized Word Representations," Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol.1 (Long Papers), (2018), 2227-2237.
  25. Pontiki, M., D. Galanis, H. Papageorgiou, I. Androutsopoulos, S. Manandhar, M. AlSmadi, M. Al-Ayyoub, Y. Zhao, B. Qin, O.D. Clercq, V. Hoste, M. Apidianaki, X. Tannier, N.V. Loukachevitch, E.V. Kotelnikov, N. Bel, S. Maria J. Zafra, and G. Eryigit, "SemEval-2016 task 5: Aspect Based Sentiment Analysis," International Workshop on Semantic Evaluation, (2016). 19-30.
  26. Pontiki, M., D. Galanis, H. Papageorgiou, S. Manandhar, and I. Androutsopoulos, "SemEval 2015 task 12: Aspect Based Sentiment Analysis," International Workshop on Semantic Evaluation, (2015), 486-495.
  27. Pontiki, M., D. Galanis, J. Pavlopoulos, H. Papageorgiou, I. Androutsopoulos, and S. Manandhar, "SemEval-2014 Task 4: Aspect Based Sentiment Analysis," International Workshop on Semantic Evaluation, (2014), 27-35.
  28. Quan, C. and F. Ren, "Unsupervised Product Feature Extraction for Feature-oriented Opinion Determination," Information Sciences, Vol.272, (2014), 16-28. https://doi.org/10.1016/j.ins.2014.02.063
  29. Radford, A. and T. Salimans, "Improving Language Understanding by Generative Pre-Training," (2018).
  30. Rajpurkar, P., J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000C Questions for Machine Comprehension of Text," (2016), arXiv:1606.05250.
  31. Rietzler, A., S. Stabinger, P. Opitz, and S. Engl, "Adapt or Get Left Behind: Domain Adaptation through Bert Language Model Finetuning for Aspect-target Sentiment Classification," (2019), arXiv:1908.11860 [cs.CL].
  32. Ruder, S., P. Ghaffari, and J.G. Breslin, "INSIGHT-1 at SemEval-2016 Task 5: Deep Learning for Multilingual Aspect-based Sentiment Analysis," Proceedings of the 10th International Workshop on Semantic Evaluation, (2016).
  33. Schouten, K. and F. Frasincar, "Survey on Aspect-Level Sentiment Analysis," IEEE Transactions on Knowledge and Data Engineering, Vol. 28, No. 3(2016), 813-830. https://doi.org/10.1109/TKDE.2015.2485209
  34. Song, M., H. Park, and K. Shin, "Attention-based long short-term memory network using sentiment lexicon embedding for aspect-level sentiment analysis in Korean," Information Processing & Management, Vol.56, No.3(2019), 637-653. https://doi.org/10.1016/j.ipm.2018.12.005
  35. Song, Y., J. Wang, Z. Liang, Z. Liu, T. Jiang, "Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference," (2020), arXiv:2002.04815v1 [cs.CL].
  36. Sun, C., L. Huang, and X. Qiu, "Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence," (2019), arXiv:1903.09588.
  37. Tang, D., B. Qin, and T. Liu, "Aspect Level Sentiment Classification with Deep Memory Network," Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, (2016), 214-224.
  38. Tang, D., B. Qin, X. Feng, and T. Liu, "Effective LSTMs for Target-dependent Sentiment Classification," International Conference on Computational Linguistics, (2016), 3298-3307.
  39. Tubishat, M., N. Idris, and M.A.M. Abushariah, "Implicit Aspect Extraction in Sentiment Analysis: Review, Taxonomy, Opportunities, and Open Challenges," Information Processing and Management, Vol.54, No.4(2018), 545-563. https://doi.org/10.1016/j.ipm.2018.03.008
  40. Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N Gomez, L. Kaiser, and I. Polosukhin, "Attention is All You Need," Advances in Neural Information Processing Systems, Vol.2017-Decem, (2017), 5999-6009.
  41. Wang, Y., M. Huang, L. Zhao and X. Zhu, "Attention-based LSTM for Aspect-level Sentiment Classification," Proceedings of the Conference on Empirical Methods in Natural Language Processing, (2016), 606-615.
  42. Xiaomei, Z., Y. Jing, Z. Jianpei, and H. Hongyu, "Microblog Sentiment Analysis with Weak Dependency Connections," Knowledge-Based Systems, Vol.142, (2018), 170-180. https://doi.org/10.1016/j.knosys.2017.11.035
  43. Xu, H., B. Liu, L. Shu, and P. S. Yu, "Bert Post-training for Review Reading Comprehension and Aspect-Based Sentiment Analysis," (2019), arXiv:1904.02232.
  44. Zeng, B., Heng Yang, Heng Yang, Ruyang Xu, Wu Zhou, Xuli Han, "LCF: A Local Context Focus Mechanism for Aspect-Based Sentiment Classification," Applied Sciences, Vol.9, No.16(2019), 3389. https://doi.org/10.3390/app9163389
  45. Zhao, W., Z. Guan, L. Chen, X. He, D. Cai, B. Wang, and Q. Wang, "Weakly-Supervised Deep Embedding for Product Review Sentiment Analysis," IEEE Transactions on Knowledge and Data Engineering, Vol.30, No.1(2018), 185-197. https://doi.org/10.1109/TKDE.2017.2756658
  46. Zhu, J., H. Wang, M. Zhu, B.K. Tsou, and M. Ma, "Aspect-based Opinion Polling from Customer Reviews," IEEE Transactions on Affective Computing, Vol.2, (2011), 37-49. https://doi.org/10.1109/T-AFFC.2011.2
  47. Zhu, P., Z. Chen, H. Zheng, T. Qian, "Aspect Aware Learning for Aspect Category Sentiment Analysis," ACM Transactions on Knowledge Discovery from Data, Vol.13, No.6(2019).

Cited by

  1. 딥러닝 기법을 활용한 산업/직업 자동코딩 시스템 vol.12, pp.4, 2020, https://doi.org/10.15207/jkcs.2021.12.4.023