The Journal of Society for e-Business Studies (한국전자거래학회지)

Society for e-Business Studies (한국전자거래학회)
권호 표지
DOI QR코드

DOI QR Code

An Integrated Perspective of User Evaluating Personalized Recommender Systems : Performance-Driven or User-Centric

개인화 추천시스템의 사용자 평가에 대한 통합적 접근 : 시스템 성과와 사용자 태도를 기반으로

  • Received : 2012.08.16
  • Accepted : 2012.08.21
  • Published : 2012.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

This study focused on user evaluation for personalized recommender systems with the integrated view of performance of the system and user attitude of recommender systems. Since users' evaluations of recommender systems can be affected by recommendation outcomes and presentation methods, both system performances based on outcomes and user attitudes formed by the presentation methods should be considered when explaining users' evaluations. However, an integrated view of system performance and user attitudes has not been applied to explain users' evaluation of recommender systems. Thus, the goal of this study is to explain users' evaluations of recommender systems under the integrated view of predictive features and explanation features at the same time. Our findings suggest that social presence, both accuracy and noveltyhave impacts onuser satisfaction for recommender systems. Especially, predictive features including accuracy and novelty affected user satisfaction. Novelty as well as accuracy is one of the significant factors for user satisfaction while recommender systems provided usual items users have experienced when systems provide serendipitous items. Likewise, explanation features with social presence and self-reference were important for user evaluation of personalized recommender systems. For explanation features, while social presence appears as one of important factors to user satisfaction of evaluating personalized recommendations, self-reference has no significant effect on user's satisfaction for recommender systems when compared to the result of social presence. Self-referencing messages did not affect user satisfaction but the levels of self-referencing are different between low and high groups in the experiment.

온라인에서 추천시스템은 사용자들의 구매 이력 또는 선호도를 바탕으로 적절한 콘텐츠 또는 서비스를 제공하는 IT기술이다. 추천시스템에 대한 사용자의 평가에는 추천 결과에 기반한 시스템 성과와 추천 방식에 의해 형성되는 사용자의 태도에 대한 두 측면 모두 고려되어야 한다. 그러나 시스템 성과와 사용자 태도에 대한 통합적 관점의 추천시스템 평가에 대한 연구는 많지 않았다. 본 연구의 목적은 추천시스템에 대한 사용자 평가의 통합적 관점을 제시하는 것에 있다. 그에 따라 사용자 태도 형성과 관련하여 자기 참조(Self-reference)와 사회적 실재감(Social Presence)의 정도를 구분하여 웹 기반 실험을 수행하였으며 추천시스템의 성과 측정을 위하여 추천 알고리즘 평가에 널리 활용되어 온 정확성(Accuracy)과 새로움(Novelty)을 활용하였다. 연구의 결과로 추천시스템의 사용자 만족에 미치는 변수로 정확성과 새로움이 시스템 특성 요소로 제시되었으며 사용자 태도 관점에서 사회적 실재감이 사용자의 만족에 영향을 주었다.

Keywords

References

  1. 서길수, "과업의 특성과 매체 경험이 인지된 매체 풍요도와 사회적 존재성에 미치는 영향", Asia Pacific Journal of Information Systems, 제8권, 제3호, pp. 119-134, 1998. https://doi.org/10.1046/j.1365-2575.1998.00030.x
  2. 손재봉, 서용무, "협업필터링 시스템에서 Degree of March를 이용한 성능 향상", Information System Review, 제8권, 제2호, pp. 139-154, 2006.
  3. 엄태영, 김우주, 박상언, "태그네트워크를 이용한 개인화 북마크 추천시스템", 한국전자거래학회지, 제15권, 제4호, pp. 181-195, 2010.
  4. 이석준, 이희춘, "협업필터링 추천에서 대응 평균 알고리즘의 예측성능에 관한 연구", Information System Review, 제9권, 제1호, pp. 85-103, 2007.
  5. 이재식, 명훈식, "사례기반 추론을 이용한 인터넷 서점의 서적 추천시스템 개발", 한국전자거래학회지, 제13권, 제4호, pp. 173-191, 2008.
  6. Adomavicius, G. and Tuzhilin, A., "Personalization Technologies : A Processoriented Perspective," Communications of the ACM, Vol. 48, No. 10, pp. 83-90, 2005. https://doi.org/10.1145/1089107.1089109
  7. Al-Natour, S., Benbasat, I., and Cenfetelli, R. T., "The Effects of Process and Outcome Similarity on Users' Evaluations of Decision Aids," Decision Sciences, Vol. 39, No. 2, pp. 175-211, 2008. https://doi.org/10.1111/j.1540-5915.2008.00189.x
  8. Arazy, O., Kumar, N., and Shapira, B., "A Theory-Driven Design Framework for Social Recommender Systems," Journal of the Association for Information Systems, Vol. 11, No. 9, pp. 455-490, 2010. https://doi.org/10.17705/1jais.00237
  9. Berlyne, D. E. (Ed.), Novelty, Complexity, and Interestingness, New York : Wiley, 2004.
  10. Burke, R., "Hybrid Recommender Systems : Survey and Experiments," User Modeling and User-Adapted Interaction, Vol. 12, No. 4, pp. 331-370, 2002. https://doi.org/10.1023/A:1021240730564
  11. Burnkrant, R. E. and Unnava, H. R., "Effects of Self-referencing on Persuasion," Journal of Consumer Research, Vol. 22, No. 1, pp. 17-26, 1995. https://doi.org/10.1086/209432
  12. Chin, W. W. (Ed.), "The partial least squares approach to structural equation modeling : Modern Methods for Business Research, Mahwah," NJ, US : Lawrence Erlbaum Associates Publishers, 1998.
  13. Choeh, J. Y. and Lee, H. J., "Mobile push personalization and user experience," AI Communications, Vol. 21, No. 2-2, pp. 183-193, 2008.
  14. Choi, J., Lee, H. J., and Kim, Y. C., "The Influence of Social Presence on Evaluating Personalized Recommender Systems," Paper presented at the PACIS 2009 India, http://aisel.aisnet.org/pacis2009/49, 2009.
  15. Choi, J., Lee, H. J., and Kim, Y. C., "The Influence of Social Presence on Costomer Intention to Reuse Online Recommender Systems : The Roles of Personalization and Product Type," International Journal of Electronic Commerce, Vol. 16, No. 1, pp. 129-153, 2011. https://doi.org/10.2753/JEC1086-4415160105
  16. Cyr, D., Hassanein, K., Head, M., and Ivanov, A., "The Role of Social Presence in Establishing Loyalty in e-Service Environments," Interacting with Computers, Vol. 19, No. 1, pp. 43-56, 2007. https://doi.org/10.1016/j.intcom.2006.07.010
  17. Davis, F. D., "Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information Technology," MIS Quarterly, Vol. 13, No. 3, pp. 319-340, 1989. https://doi.org/10.2307/249008
  18. Escalas, J. E., "Self-Referencing and Persuasion : Narrative Transportation versus Analytical Elaboration," Journal of Consumer Research, Vol. 33, No. 4, pp. 421-429, 2007. https://doi.org/10.1086/510216
  19. Fornell, C. and Larcker, D. F., "Evaluating Structural Equation Models with Unobservable Variables and Measurement Error," Journal of Marketing Research, Vol. 18, No. 3, pp. 39-50, 1981. https://doi.org/10.2307/3151312
  20. Fouss, F. and Saerens, M., "Evaluating performance of recommender systems : An experimental comparison," Paper presented at the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 2008.
  21. Gefen, D., Karahanna, E., and Straub, D. W., "Trust and TAM in Online Shopping : An Integrated Model," MIS Quarterly, Vol. 27, No. 1, pp. 51-90, 2003. https://doi.org/10.2307/30036519
  22. Gefen, D. and Straub, D. W., "Consumer Trust in B2C e-Commerce and the Importance of Social Presence : Experiments in e-Products and e-Services," Omega, Vol. 32, No. 6, pp. 407-424, 2004. https://doi.org/10.1016/j.omega.2004.01.006
  23. Haubl, G. and Murray, K. B., "Double Agent : Assessing the Role of Electronic Product Recommendation System," Sloan Management Review, Vol. 47, No. 3, pp. 8-12, 2006.
  24. Haubl, G. and Trifts, V., "Consumer Decision Making in Online Shopping Environments : The Effects of Interactive Decision Aids," Marketing Science, Vol. 19, No. 1, pp. 4-21, 2000. https://doi.org/10.1287/mksc.19.1.4.15178
  25. Hassanein, K. and Head, M., "The Impact of Infusing Social Presence in the Web Interface : An Investigation Across Different Products," International Journal of Electronic Commerce, Vol. 10, No. 2, pp. 31-55, 2006. https://doi.org/10.2753/JEC1086-4415100202
  26. Herlocker, J. L., Konstan, J. A., Terveen, L. G., and Riedl, J. T., "Evaluating Collaborative Filtering Recommender Systems," ACM Transactions on Information Systems, Vol. 22, No. 1, pp. 5-53, 2004. https://doi.org/10.1145/963770.963772
  27. Ho, S. Y. and Kwok, S. H., "The Attraction of Personalized Service for Users in m-Commerce," ACM SIGecom Exchanges, Vol. 3, No. 4, pp. 10-18, 2003.
  28. Holbrook, M. B. (Ed.), "Introduction : The Esthetic Imperative in Consumer Research," MI : Association for Consumer Research, 1981.
  29. Karahanna, E. and Limayem, M., "E-Mail and V-Mail Usage : Generalizing Across Technologies," Journal of Organizational Computing and Electronic Commerce, Vol. 10, No. 1, pp. 49-66, 2000. https://doi.org/10.1207/S15327744JOCE100103
  30. Komiak, S. Y. X. and Benbasat, I., "The Effects of Personalization and Familiarity on Trust and Adoption of Recommendation Agents," MIS Quarterly, Vol. 30, No. 4, pp. 941-960, 2006. https://doi.org/10.2307/25148760
  31. Kramer, T., "The effect of measurement task transparency on preference construction and evaluations of personalized recommendations," Journal of Marketing Research, Vol. 44, No. 2, pp. 224-233, 2007. https://doi.org/10.1509/jmkr.44.2.224
  32. Kumar, N. and Benbasat, I., "The Influence of Recommendations and Consumer Reviews on Evaluations of Websites," Information Systems Research, Vol. 17, No. 4, pp. 425-429, 2006. https://doi.org/10.1287/isre.1060.0107
  33. Lee, H. J., Kim, J. W., and Park, S. J., "Understanding Collaborative Filtering Parameters for Personalized Recommendations in e-Commerce," Electronic Commerce Research, Vol. 7, No. 3-4, pp. 45-70, 2007. https://doi.org/10.1007/s10660-006-0062-z
  34. Liang, T., Lai, H., and Ku, Y., "Personalized Content Recommendation and User Satisfaction : Theoretical Synthesis and Empirical Findings," Journal of Management Information Systems, Vol. 23, No. 3, pp. 45-70, 2007. https://doi.org/10.2753/MIS0742-1222230303
  35. Linden, G., Smith, B., and York, J., "Amazon.com Recommendations : Itemto- item Collaborative Filtering," IEEE Internet Computing, Vol. 7, No. 3, pp. 76-80, 2003. https://doi.org/10.1109/MIC.2003.1167344
  36. Martin, R. and Hewstone, M., "Social-influence Processes of Control and Change : Conformity, Obedience to Authority and Innovation," London : Sage, 2003.
  37. Meyers-Levy, J. and Peracchio, L. A., "Moderators of the Impact of Self-Reference on Persuasion," Journal of Consumer Research, Vol. 22, No. 4, pp. 408-423, 1996. https://doi.org/10.1086/209458
  38. Moorman, C., "Organizational Market Information Processes : Cultural Antecedents and New Product Outcomes," Journal of Marketing Research, Vol. 32, No. 3, pp. 318-335, 1995. https://doi.org/10.2307/3151984
  39. Oliver, R. L., Robertson, T. S., and Mitchell, D. J., "Imagine and Analyzing in Response to New Product Advertising," Journal of Advertising, Vol. 22, No. 4, pp. 35-50, 1993. https://doi.org/10.1080/00913367.1993.10673417
  40. Oliver, R. L. and Swan, J. E., "Equity and disconfirmation perceptions as influences on merchant and product satisfaction," Journal of Consumer Research, Vol. 16, No. 3, pp. 372-383, 1989. https://doi.org/10.1086/209223
  41. Osgood, C. E., Succi, G. J., and Tannenbaum, P. H., "The Measurement of Meaning," IL : University of Illinois, 1957.
  42. Pu, P. and Chen, L., "A User-Centric Evaluation Framework of Recommender Systems," Paper presented at the In proceedings of the ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and their Interfaces (UCERSTI), 2010.
  43. Resnick, P., Iacovou, N., Suchak, N., and Bergstrom, P., "GroupLens : An Open Architecture for Collaborative Filtering of Netnews," Paper presented at the the 1994 ACM Conference on computersupported cooperative work, 1994.
  44. Resnick, P. and Varian, H. R., "Recommender Systems," Communications of the ACM, Vol. 40, No. 3, pp. 56-58, 1997. https://doi.org/10.1145/245108.245121
  45. Ricci, F., "Mobile Recommender Systems," International Journal of Information Technology and Tourism, Vol. 12, No. 3, pp. 1-24, 2011.
  46. Rogers, T. B., Kuiper, N. A., and Kirker, W., "Self-reference and the encoding of personal information," Journal of Personality and Social Psychology, Vol. 35, No. 9, pp. 677-688, 1977. https://doi.org/10.1037/0022-3514.35.9.677
  47. Sarwar, B., Karypis, G., Konstan, J., and Riedl, J., "Item-based Collaborative Filtering Recommendation Algorithms," Paper presented at the Proceedings of the 10th international conference on World Wide Web, Hong Kong, 2001.
  48. Shani, G. and Gunawardana, A., "Recommender Systems Handbook : Evaluating Recommendation Systems : Springer," 2011.
  49. Sinha, R. and Swearingen, K., "Comparing Recommendations made by Online Systems and Friends," Paper presented at the DELOS-NSF Workshop on Personalization and Recommender Systems in Digital Libraries, 2001.
  50. Swearingen, K. and Sinha, R., "ACM SIGIR Workshop on Recommender Systems Beyond Algorithms : An HCI Perspective on Recommender Systems," 2001.
  51. Tam, K. Y. and Ho, S. Y., "Web Personalization As A Persuasion Strategy : An Elaboration Likelihood Model Perspective," Information Systems Research, Vol. 16, No. 3, pp. 271-291, 2005. https://doi.org/10.1287/isre.1050.0058
  52. Tam, K. Y. and Ho, S. Y., "Understanding the Impact of Web Personalization on User Information Processing and Decision Outcome," MIS Quarterly, Vol. 30, No. 4, pp. 865-890, 2006. https://doi.org/10.2307/25148757
  53. Thirumalai, S. and Sinha, K. K., "Customization Strategies in Electronic Retailing : Implications of Customer Purchase Behavior," Decision Sciences, Vol. 40, No. 1, pp. 5-36, 2009. https://doi.org/10.1111/j.1540-5915.2008.00222.x
  54. Unger, L., "Consumer Marketing Trends in the 1980s : When Growth Slows," Journal of Consumer Research, Vol. 9, No. 2, pp. 69-73, 1981.
  55. Wang, W. and Benbasat, I., "Recommendation Agents for Electronic Commerce : Effects of Explanation Facilities on Trusting Beliefs," Journal of Management Information Systems, Vol. 23, No. 4, pp. 217-246, 2007. https://doi.org/10.2753/MIS0742-1222230410
  56. Xiao, B. and Benbasat, I., "E-commerce Product Recommendation Agents : Use, Characteristics, and Impact," MIS Quarterly, Vol. 31, No. 1, pp. 137-209, 2007. https://doi.org/10.2307/25148784
  57. Xu, D. J., "The Influence of Personalization in Affecting Consumer Attitudes Toward Mobile Advertising in China," Journal of Computer Information Systems, Vol. 47, No. 2, pp. 9-19, 2006.

Cited by

  1. 온라인 상품추천 서비스에 대한 소비자 사용 의도 -신뢰-몰입의 매개역할을 중심으로- vol.42, pp.5, 2018, https://doi.org/10.5850/jksct.2018.42.5.871
  2. Consumer-Driven Usability Test of Mobile Application for Tea Recommendation Service vol.9, pp.19, 2012, https://doi.org/10.3390/app9193961
  3. A Study on the Continuance Intention of Short Clip Application: Focusing on Chinese TikTok Users vol.22, pp.1, 2012, https://doi.org/10.9728/dcs.2021.22.1.125
  4. 온라인 패션쇼핑몰의 개인 상품 추천서비스가 인지적 태도와 감정적 애착을 통해 서비스 사용행동에 미치는 영향 vol.23, pp.5, 2012, https://doi.org/10.5805/sfti.2021.23.5.586