Journal of the Institute of Electronics and Information Engineers (전자공학회논문지)

The Institute of Electronics and Information Engineers (대한전자공학회)
권호 표지
DOI QR코드

DOI QR Code

Design and Implementation of an Alternate System Interconnect based on PCI Express

PCI Express 기반 시스템 인터커넥트의 설계 및 구현

  • 김영우 (한국전자통신연구원 클라우드컴퓨팅연구부) ;
  • 런예 (한국전자통신연구원 클라우드컴퓨팅연구부) ;
  • 최원혁 (한국전자통신연구원 클라우드컴퓨팅연구부)
  • Received : 2015.06.01
  • Accepted : 2015.07.27
  • Published : 2015.08.25
Download PDF
⟨ Previous Next ⟩

Abstract

PCI Express is a well-known and widely used de-facto system bus standard for connecting among a processor and IO devices. PCI Express is originated from old PCI standard, and its most of applications are limited to be used within a PC or server system. But, because of its fast speed, low power consumption, and good protocol efficiency, it is considered as one of a good candidate for an alternate system interconnect for many years. In this paper, we present design, implementation and early evaluation of an alternate system interconnect by utilizing PCI Express. The developed alternate system interconnect using PCI Express (named PCIeLINK) utilizes non-transparent bridging (NTB) technic which generally used in fail-over system in PCI and PCI Express. By using NTB technic, PCI Express device can be extended to outside of a system without electrical and logical problems arising during system boot and enumeration. To build up an alternate system interconnect, we designed and implemented a network interface card having multiple PCI Express ${\times}4$ connections (theoretically 20 Gbps) and tested, The early test results revealed that an ${\times}4$ port in the card showed 8.6 Gbps peak performance for bulk transmission and 5.1 Gbps peak for normal TCP/IP transfer.

PCI Express는 프로세서와 시스템의 IO 장치들을 연결하기 위하여 널리 사용되는 업계 표준이다. PCI Express 는 이전 PCI 표준에서 유래하며, 전통적으로 하나의 PC 혹은 서버 내에서 사용되어져 왔다. PCI Express의 고속, 저전력, 고효율 특성은 기존 시스템 연결망과는 다른 형태의 대안 연결망으로써 고려되고 있다. 본 논문에서는 이와 같은 PCI Express를 이용한 시스템 연결망(PCIeLINK)을 설계, 구현하고 초기 시험 결과를 제시한다. 본 논문에서는 PCI Express를 이용한 fail-over 시스템에 자주 사용되는 non-transparent bridging(NTB)기법을 이용하여 PCI Express 기반 시스템 연결망을 설계, 구현 하였다. NTB는 PCI Express 장치를 단순 연결할 경우 발생되는 전기적, 논리적 충돌을 방지하는 기법으로써, PCI Express Gen2 규격에 기반한 20 Gbps급의 ${\times}4$ 연결을 하나의 카드에 복수개 구현하고 이를 시험하였다. 개발된 PCI Express기반 시스템 인터커넥트 장치는 최대 8.6 Gbps의 단방향 성능을 보였으며, Linux 기반의 TCP/IP 환경에서 최대 5.1 Gbps의 성능을 나타내는 것으로 측정 되었다.

Keywords

References

  1. Young Woo Kim, et. al., "Technology and Trends of High Performance Processors," Electronics and telecommunications trends, vol.25, no.5, pp.123-136, 2010.
  2. Interconnect Family Statistics, TOP500. org accessed May 14, 2015, http://www.top500.org/statistics/list/
  3. "The 2015 Ethernet Roadmap," white paper from Ethernet alliance accessed May 13, 2015, http://www.ethernetalliance.org/roadmap/
  4. "InfiniBand Roadmap," InfiniBand Trade Association accessed May13, 2015, http://www.infinibandta.org/content/pages.php?pg=technology_overview
  5. "Annual Meeting of Members Presentation," PCI-SIG accessed May13, 2015, https://www.pcisig.com/members/downloads/PCISIG_Annual_Meeting_of_Members_2013_Final.pdf
  6. "Why FPU Generator?" Floating-Point Unit Generator accessed May15, 2015, https://sites.google.com/a/stanford.edu/fpgen/why
  7. Wonok Kwon, et. al., "PCI Express Gen3 System Design using High-speed Signal Integrity Analysis," Journal of the Institute of Electronics and Information Engineers, vol.52, no.4, pp.125-132, 2015. https://doi.org/10.5573/ieie.2015.52.4.125
  8. Wonok Kwon, et. al., "ALTERA Embedded Gigabit Transceiver Measurement for PCI Express Protocol," Journal of the Institute of Electronics and Information Engineers, vol.41, no.4, pp.359-367, 2004.
  9. Vijay Medury, "PCI Express in Clustering," High Speed Interconnects Seminar, Linley Group, Nov., 2010.
  10. JohnB yrne, et. al., "Power-Efficient Networking for Balanced System Designs: Early Experiences with PCIe," HotPower '11 Proceedings of the 4th Workshop on Power-Aware Computing and Systems, Article No. 3, 2011.
  11. Ahmed Bu-Khamsin, "Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation," MS Thesis, Simon Fraser University, 2012.
  12. T. Hanawa,T. Boku, S. Miura, T. Okamoto, M. Sato, et al., "Low-Power and High-Performance Communication Mechanism for Dependable Embedded Systems,"International Workshop on Innovative Architecture for Future Generation High-Performance Processors and Systems, pp. 67-73, 2008.
  13. T. Hanawa, Y. Kodama, T. Boku, M. Sato, "Interconnection Network for Tightly Coupled Accelerators Architecture," IEEE 21st Annual Symposium on High-Performance Interconnects, San Jose, USA, pp. 79-82, Aug. 2013.
  14. Young Woo Kim, et. al., "Design of MAHA Supercomputing System for Human Genome Analysis," KIPS transactions on software and dataengineering, vol.2, no.2, pp.81-90, 2013. https://doi.org/10.3745/KTSDE.2013.2.2.081
  15. Young Woo Kim, "MAHA-Manycore HPC System for Bio-Applications," The Second KIISE-KOCSEA HPC SIG Joint Workshop, Denver, USA, Nov., 2013.
  16. Young Woo Kim, et. al., "Implementation of System Interconnection Device Using PCI Express," Proceedings on 2014 IEIE Summer Conference, pp.515-518, Jeju Island, Korea, 2014.
  17. Ye Ren, et. al., "A Preliminary Implementation of PCIeLINK and Its Performance Evaluation," Proceedings on 2014 IEIE Summer Conference, pp.519-522, Jeju Island, Korea, 2014.
  18. Ye Ren, et. al., "Implementation of System Interconnection Devices Using PCI Express," IEEE International Conference on Consumer Electronics, Las Vegas, USA, pp.300-301, Jan. 2015.
  19. D. Percival, "PCI Express Clustering," PCI-SIG Developers Conference, Israel, 2011.
  20. IDT's PCIe Gen2 Switch family Non-Trans patent Operation, Application Note, IDT, 2009
  21. "SFF-8436 Specification for QSFP+ 10 Gbs 4X PLUGGABLE TRANSCEIVER" SFF Committee accessed Apr. 13, 2012, ftp://ftp.seagate.com/sff/SFF-8436.PDF
  22. $IDT^{(R)}$ 89HPES24NT6AG2 PCI $Express^{(R)}$ Switch User Manual, IDT accessed May 25, 2012.
  23. WonHyuk Choi, et.al., "A Design of Dedicated Communication module for PCI Express network device," KIISE Proceedings on Korea Computer Congress 2015, pp 95-97, Jeju Island, Korea, 2015.
  24. NetPIPE, http://bitspjoule.org/netpipe/
  25. Netperf, http://www.netperf.org/netperf/

Cited by

  1. Implementation of High Speed Image Data Transfer using XDMA vol.25, pp.7, 2015, https://doi.org/10.9708/jksci.2020.25.07.001