World Library  

Add to Book Shelf
Flag as Inappropriate
Email this Book

An Approach to Enhance Pnetcdf Performance in Environmental Modeling Applications : Volume 8, Issue 4 (08/04/2015)

By Wong, D. C.

Click here to view

Book Id: WPLBN0003982224
Format Type: PDF Article :
File Size: Pages 14
Reproduction Date: 2015

Title: An Approach to Enhance Pnetcdf Performance in Environmental Modeling Applications : Volume 8, Issue 4 (08/04/2015)  
Author: Wong, D. C.
Volume: Vol. 8, Issue 4
Language: English
Subject: Science, Geoscientific, Model
Collections: Periodicals: Journal and Magazine Collection (Contemporary), Copernicus GmbH
Publication Date:
Publisher: Copernicus Gmbh, Göttingen, Germany
Member Page: Copernicus Publications


APA MLA Chicago

Yang, C. E., Fu, J. S., Wong, K., Wong, D. C., & Gao, Y. (2015). An Approach to Enhance Pnetcdf Performance in Environmental Modeling Applications : Volume 8, Issue 4 (08/04/2015). Retrieved from

Description: U.S. Environmental Protection Agency, Research Triangle Park, NC, USA. Data intensive simulations are often limited by their I/O (input/output) performance, and novel techniques need to be developed in order to overcome this limitation. The software package pnetCDF (parallel network Common Data Form), which works with parallel file systems, was developed to address this issue by providing parallel I/O capability. This study examines the performance of an application-level data aggregation approach which performs data aggregation along either row or column dimension of MPI (Message Passing Interface) processes on a spatially decomposed domain, and then applies the pnetCDF parallel I/O paradigm. The test was done with three different domain sizes which represent small, moderately large, and large data domains, using a small-scale Community Multiscale Air Quality model (CMAQ) mock-up code. The examination includes comparing I/O performance with traditional serial I/O technique, straight application of pnetCDF, and the data aggregation along row and column dimension before applying pnetCDF. After the comparison, optimal I/O configurations of this application-level data aggregation approach were quantified. Data aggregation along the row dimension (pnetCDFcr) works better than along the column dimension (pnetCDFcc) although it may perform slightly worse than the straight pnetCDF method with a small number of processors. When the number of processors becomes larger, pnetCDFcr outperforms pnetCDF significantly. If the number of processors keeps increasing, pnetCDF reaches a point where the performance is even worse than the serial I/O technique. This new technique has also been tested for a real application where it performs two times better than the straight pnetCDF paradigm.

An approach to enhance pnetCDF performance in environmental modeling applications

Behzad, B., Luu, H. V. T., Huchette, J., Byna, S., Prabhat, Aydt, R. A., Koziol, Q., and Snir, M.: Taming parallel I/O complexity with auto-tuning, SC 2013, ACM, Denver, CO, USA, 17–22 November 2013, p. 68, 2013.; Byun, D. W. and Schere, K. L.: Review of the governing equations, computational algorithms, and other components of the Models-3 Community Multiscale Air Quality (CMAQ) Modeling System, Appl. Mech. Rev., 59, 51–77, 2006.; Chen, Y., Sun, X.-H., Thakur, R., Song, H., and Jin, H.: Improving Parallel I/O Performance with Data Layout Awareness, Cluster '10: Proceedings of the IEEE International Conference on Cluster Computing 2010, Heraklion, Crete, Greece, 20–24 September 2010: IEEE Computer Society, 2010.; Cheng, A. and Folk, M.: HDF5: High performance science data solution for the new millennium, Proceedings of SC2000: High Performance Networking and Computing, Dallas, TX, ACM Press and IEEE Computer Society Press, 4–10 November 2000.; del Rosario, J., Brodawekar, R., and Choudhary, A.: Improved Parallel I/O via a Two-Phase Run-time Access Strategy, Workshop on I/O in Parallel Computer Systems at IPPS '93, Apr. 1993, 56–70, 1993.; Fu, J. S., Dong, X., Gao, Y., Wong, D., and Lam Y. F.: Sensitivity and linearity analysis of ozone in East Asia: The effects of domestic emission and intercontinental transport, J. Air Waste Manage., 62, 1102–1114, 2012.; Gao, Y., Fu, J. S., Drake, J. B., Lamarque, J.-F., and Liu, Y.: The impact of emission and climate change on ozone in the United States under representative concentration pathways (RCPs), Atmos. Chem. Phys., 13, 9607–9621, doi:10.5194/acp-13-9607-2013, 2013.; Huang, X. M., Wang, W. C., Fu, H. H., Yang, G. W., Wang, B., and Zhang, C.: A fast input/output library for high-resolution climate models, Geosci. Model Dev., 7, 93–103, doi:10.5194/gmd-7-93-2014, 2014.; Kordenbrock, T. H. and Oldfield, R. A.: Parallel I/O Advancements in Air Quality Modeling Systems, Poster on 5th annual CMAS conference, Chapel Hill, NC, USA, 16–18 October 2006.; Li, J., Liao, W.-K., Choudhary, A., Ross, R., Thakur, R., Gropp, W., Latham, R., Siegel, A., Gallagher, B., and Zingale, M.: Parallel netCDF: A High-Performance Scientific I/O Interface, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, Phoenix, AZ, USA, 15–21 November 2003, p. 39, doi:10.1145/1048935.1050189, 2003.; Nisar, A., Liao, W.-K., and Choudhary, A.: Delegation-Based I/O Mechanism for High Performance Computing Systems, IEEE Transactions on Parallel and Distributed Systems, 23, 271–279, doi:10.1109/TPDS.2011.166, 2012.; Nitzberg, B. and Lo, V. M.: Collective Buffering: Improving Parallel I/O Performance, HPDC, 148–157, 1997.; Otte, T. L., Pouliot, G., Pleim, J. E., Young, J. O., Schere, K. L., Wong, D. C., Lee, P. C. S., Tsidulko, M., McQueen, J. T., Davidson, P., Mathur, R., Chuang, H.-Y., DiMego, G., and Seaman, N. L.: Linking the Eta Model with the Community Multiscale Air Quality (CMAQ) modeling system to build a national air quality forecasting system, Weather Forecast., 20, 367–384, 2005.; Palmera, B., Koontza, A., Schuchardta, K., Heikesb, R., and Randallb, D.: Efficient data IO for a Parallel Global Cloud Resolving Model, Environmental Modelling and Software, 26, 1725–1735, 2011.; Thakur, R., Gropp, W., and Lusk, E.: Data sieving and collective I/O in ROMIO. Proceedings of the Seventh Symposium on the Frontiers of Massively Parallel Computation, IEEE Computer Society Press, 182–189, 1999.; Wong, D. C., Pleim, J., Mathur, R., Binkowski, F., Otte, T., Gilliam, R., Pouliot, G., Xiu, A., Young, J. O., and Kang, D.: WRF-CMAQ two-way coupled system with aerosol feedback: software development and preliminary results, Geosci. Model Dev., 5, 299–312,


Click To View

Additional Books

  • Use of Agricultural Statistics to Verify... (by )
  • Δ18O Water Isotope in the ILoveclim Mode... (by )
  • Impact of Climate, Vegetation, Soil and ... (by )
  • Suitability of Modelled and Remotely Sen... (by )
  • Modelling Turbulent Vertical Mixing Sens... (by )
  • Optimising the Famous Climate Model: Inc... (by )
  • Regional Scale Ozone Data Assimilation U... (by )
  • Evaluation of a Near-global Eddy-resolvi... (by )
  • Assessing the Cam5 Physics Suite in the ... (by )
  • Aircraft Routing with Minimal Climate Im... (by )
  • Simulating the Mid-pliocene Warm Period ... (by )
  • Gappard: a Computationally Efficient Met... (by )
Scroll Left
Scroll Right


Copyright © World Library Foundation. All rights reserved. eBooks from World eBook Library are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.