DOI QR코드

DOI QR Code

Study of Optimization through Performance Analysis of Parallel Distributed Filesystem

병렬 분산파일시스템의 성능 분석을 통한 최적화 연구

  • Yoon, JunWeon (Dept. of Supercomputing Center, KISTI) ;
  • Song, Ui-Sung (Dept. of Computer Education, Busan National University of Education)
  • Received : 2016.10.10
  • Accepted : 2016.10.31
  • Published : 2016.10.31

Abstract

Recently, Big Data issue has become a buzzword and universities, industries and research institutes have been efforts to collect, analyze various data enabled. These things includes accumulated data from the past, even if it is not possible to analysis at this present immediately a which has the potential means. And we are obtained a valuable result from the collected a large amount of data via the semantic analysis. The demand for high-performance storage system that can handle large amounts of data required is increasing around the world. In addition, it must provide a distributed parallel file system that stability to multiple users too perform a variety of analyzes at the same time by connecting a large amount of the accumulated data In this study, we identify the I/O bandwidth of the storage system to be considered, and performance of the metadata in order to provide a file system in stability and propose a method for configuring the optimal environment.

최근 빅데이터 이슈가 화두가 됨에 따라 대학, 산업체, 연구소 등에서는 다양한 데이터들을 수집, 분석 하려는 노력이 활성화 되고 있다. 여기에는 과거부터 축적된 데이터, 현재에 바로 분석이 불가능하더라도 잠재적인 의미를 가지고 있는 데이터 등 대량의 데이터들이 수집되어 의미론적인 분석을 통해 가치 있는 분석결과를 얻게 된다. 이를 위해 전 세계적으로 대용량의 데이터 요구를 처리 할 수 있는 고성능 스토리지 시스템의 수요가 증가하고 있다. 또한, 여러 사용자들에게 축적된 대량의 데이터에 동시에 접속하여 다양한 분석을 수행할 수 있도록 안정성 있는 병렬 분산파일시스템을 제공해야 한다. 본 연구에서는 위와 같이 안정성 있는 파일시스템을 제공하기 위해 반드시 고려되어야 할 스토리지 시스템의 I/O 대역폭, 메타데이터의 성능 등을 파악하고 최적의 환경을 구성하기 위한 방법을 제시하고자 한다.

Keywords

References

  1. Zhao, D., Shou, C., Zhang, Z., Sadooghi, I., Zhou, X., Li, T., & Raicu, I." FusionFS: a distributed file system for large scale data-intensive computing", In Greater Chicago Area System Research Workshop pp. 10-11, 2013
  2. CEO & President George Teixeira"Why Parallel I/O Software and Moore's Law Enable Virtualization and Software-Defined Data Centers to Achieve their Potential", DataCore Opinion Paper, Aug, 2015
  3. Sun Microsystems, Inc., "$LUSTRE^{TM}$ FILE SYSTEM", Oct. 2008.
  4. Oral, Sarp, et al. "OLCF's 1 TB/s, next-generation lustre file system." Proceedings of Cray User Group Conference (CUG 2013), 2013.
  5. LOEWE, William; MCLARTY, T.; MORRONE, C., "Ior benchmar", 2012, https://sourceforge.net/projec ts/ior-sio.
  6. MPI-2: Extensions to the Message Passing Interface. http://www.mpi-forum.org/docs/mpi-20-html/m pi2-report.html
  7. Shan, Hongzhang, and John Shalf. "Analysis of Paral lel I/O for NERSC HPC Platforms: Application Requirements, Benchmarks, and Delivered System Performance."
  8. Mdtest benchmark. sourceforge.net/projects/mdtest.
  9. Eshel, Marc, et al. "Panache: A Parallel File System Cache for Global File Access." FAST. pp.155-168. 2010.
  10. SHAN, Hongzhang; SHALF, John. Using IOR to Analyze the I/O performance for HPC Platforms. Lawrence Berkeley National Laboratory,2007.
  11. University of Colorado Research Computing, "Parallel IO on Janus Lustre"
  12. DEVENDRAN, Dharshi, et al." Collective I/O Optimizations for Adaptive Mesh Refinement Data Writes on Lustre File System", 2016.
  13. Oertel, Rene. "Benchmarking the Chemnitz High Performance Linux Cluster (CHiC).", Jan, 2008.
  14. Alexander Oltu UniBCCS, "Performance Analysis- Synthetic benchmarks: IOR, bonnie++, mdtest", 2010.

Cited by

  1. Optimization of the computing environment to improve the speed of the modeling (WRF and CMAQ) calculation of the National Air Quality Forecast System vol.27, pp.8, 2018, https://doi.org/10.5322/JESI.2018.27.8.723
  2. A Content-based Audio Retrieval System Supporting Efficient Expansion of Audio Database vol.18, pp.5, 2016, https://doi.org/10.9728/dcs.2017.18.5.811
  3. Enhancing the User Experience: A Research on China Mobile E-book App vol.18, pp.8, 2017, https://doi.org/10.9728/dcs.2017.18.8.1475
  4. A Study on Factors Influencing the Intention to Use NFC Payment System for Public Transport - Focused on Ho Chi Minh Citizens in Vietnam vol.19, pp.3, 2016, https://doi.org/10.9728/dcs.2018.19.3.569
  5. Parallel File System Characteristics and Performance Analysis with NVMe SSD based Cache vol.21, pp.1, 2016, https://doi.org/10.9728/dcs.2020.21.1.157