The Journal of Korean Institute of Information Technology (한국정보기술학회논문지)

Korean Institute of Information Technology (한국정보기술학회)
권호 표지
DOI QR코드

DOI QR Code

Proposal and Analysis of Various Link Architectures in Multilayer Neural Network

다층신경망의 다양한 연결구조 제안 및 분석

  • 김미숙 (강릉원주대학교 컴퓨터공학과) ;
  • 강태원 (강릉원주대학교 컴퓨터공학과)
  • Received : 2017.11.12
  • Accepted : 2018.03.28
  • Published : 2018.04.30
⟨ Previous Next ⟩

Abstract

Neural networks are computational models that simulate biological brain structures and behaviors. The most commonly used neural network is a multilayer forward propagation neural network composed of several layers, ie layers, and learning uses error propagation algorithms. In the case of existing multilayer neural networks, the learning performance deteriorates due to the change in the weight modification amount as the layer becomes deeper. In this paper, we propose and analyze a neural network connection structure in which non adjacent layers neurons are allowed to connect to each other so that the inputs can be transmitted to the entire neural network. As a result of the analysis, it was confirmed that the deep neural network of the proposed structure shows better learning performance even if the connection is added so as not to affect the learning speed.

신경망은 생물학적 뇌 구조와 동작을 모사한 계산모델이다. 가장 흔하게 사용하는 신경망은 여러 개의 레이어 즉, 층으로 구성된 다층 전진전파 신경망이고 학습은 오류역전파 알고리즘을 사용한다. 기존 다층신경망의 경우 레이어가 깊어질수록 가중치 수정량의 변화에 의해 학습성능이 나빠진다. 본 논문에서는 입력신호가 신경망 전체에 전달될 수 있도록 인접하지 않은 레이어의 뉴런들끼리 연결이 허용되는 신경망 연결구조를 제안하고 분석한다. 분석결과, 학습속도에 영향을 주지 않을 정도로 연결을 추가하여도 제안한 구조의 깊은 신경망이 그렇지 않은 경우보다 우수한 학습성능을 보임을 확인하였다.

Keywords

References

  1. Sung-Kwun Oh, "Computational Intelligence by Programming focused on Fuzzy, Neural Networks, and Genetic Algorithms", Naeha Press, pp. 225, Aug. 2002.
  2. J. T. Wall, J. Xu, and X. Wang, "Human brain plasticity: an emerging view of the multiple substrates and mechanisms that cause cortical changes and related sensory dysfunctions after injuries of sensory inputs from the body", Brain Research Reviews, Vol. 39, No. 2-3, pp. 181-215, Sep. 2002. https://doi.org/10.1016/S0165-0173(02)00192-3
  3. R. G. Coss, J. G. Brandon, and Albert. Globus, "Changes in morphology of dendritic spines on honeybee calycal interneurons associated with cumulative nursing and foraging experiences", Brain Research, Vol. 192, No. 1, pp. 49-59, Jun. 1980. https://doi.org/10.1016/0006-8993(80)91007-0
  4. F. H. Gage, "Structural plasticity: cause, result, or correlate of depression", Biol Psychiatry, Vol. 48, No. 8, pp. 713-714, Nov. 2000. https://doi.org/10.1016/S0006-3223(00)01019-2
  5. Robert G. Bartle and Donald R. Sherbert, "Introduction to REAL ANALYSIS", John Wiley & Sons, Inc., p. 208, 2006.
  6. E. K. Blum, "Approximation of Boolen Functions By Sigmoidal Networks: Part I: XOR and other two-variable functions", Neural Computation, Vol. 1, No. 4, pp. 532-540, Winter 1989. https://doi.org/10.1162/neco.1989.1.4.532
  7. M. K. Weir, "A Method for Self-Determination of Adaptive Learning Rates in Back Propagation", Neural Networks, Pergamon Press, Vol, 4, No. 3, pp. 371-379, Dec. 1991. https://doi.org/10.1016/0893-6080(91)90073-E
  8. Mohandes, M., Codrington, CW., and Gelfand S.B., "Two Adaptive Stepsize Rules for Gradient Descent and their Application to the Training of Feedforward Artificial Neural Networks", Proceedings of IEEE International Conference on Neural Network, pp. 555-560, Jul. 1994.
  9. Tae-Won Kang, "Fault Tolerance Analysis of the Neural Networks of Scale-Free Network Architecture", JKIIT, Vol. 11, No. 7, pp. 151-157, Jul. 2013.
  10. M. Faloutsos, P. Faloutsos, and C. Faloutsos, "On power-law relationships of the internet topology", Computer Communications Review 29, pp. 251-262, Sep. 1999. https://doi.org/10.1145/316194.316229
  11. Sangwoong Yoon and Byung-Tak Zhang, "Analyzing Performance and Dynamics of Echo State Networks Given Various Structures of Hidden Neuron Connections", KIISE Transactions on Computing Practices, Vol. 21, No. 4, pp. 338-342, Apr. 2015. https://doi.org/10.5626/KTCP.2015.21.4.338
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition", arXiv preprint arXiv:1512.03385, 2015b.
  13. Mark Buchanan, "Ubiquity: the science of history. or why the world is simpler than we think", Amazon, 2001.
  14. D. J. Watts, "Small world: the dynamics of networks between order and randomness", Princeton, 1999.
  15. M. E. J. Newman, "Power laws, Pareto distributions and Zipf's law", Contemporary Physics, Vol. 46, No. 5, pp. 323-351, Sep. 2005. https://doi.org/10.1080/00107510500052444
  16. UCI Machine Learning Repository, http://archive.ics.uci.edu/ml/index.php, [Accessed: Jan. 10. 2018]
  17. Semeion Handwritten Digit Dataset, https://archive.ics.uci.edu/ml/datasets/semeion+handwritten+digit, [Accessed: Jan. 10. 2018]
  18. Gisette Dataset, https://archive.ics.uci.edu/ml/datasets/Gisette, [Accessed: Jan. 10. 2018]
  19. X. Zhang and J. Zhou, "Support vector machines speed pattern recognition - Vision Systems Design", Vision Systems Design, Aug. 17. 2013.
  20. S. Gangaputra, "Handwritten digit database", Retrieved 17, Aug. 2013.
  21. Q. Yu, "THE MNIST DATABASE of handwritten digits", Retrieved 18, Aug. 2013.
  22. MNIST database Error Rates, https://en.wikipedia.org/wiki/MNIST_database, [Accessed: Oct. 5. 2017]

Cited by

  1. A Study on the Gender and Age Classification of Speech Data Using CNN vol.16, pp.11, 2018, https://doi.org/10.14801/jkiit.2018.16.11.11
  2. Rain Removal Via Deep Convolutional Neural Networks Considering Orientation and Strength of Rain Streaks vol.17, pp.1, 2019, https://doi.org/10.14801/jkiit.2019.17.1.85