Use of High Performance Computing in Meteorology

PREFACE.

CONTENTS.

This paper describes the early experiences with the large IBM p690+ systems at ECMWF. Results are presented for the IFS (Integrated Forecasting System) which has been parallelized with MPI message passing and OpenMP to obtain good scalability on large numbers of processors. A new profiler called Dr.Hook, which has been developed to give detailed performance information for IFS, is described. Finally the code optimizations for IFS on the IBM p690+ are briefly outlined.

Application scientists have been frustrated by a trend of stagnating application performance, despite dramatic increases in claimed peak performance of high-performance computing systems. This trend is sometimes referred to as the “divergence problem” and often has been assumed that the ever-increasing gap between theoretical peak and sustained performance was unavoidable. However, recent results from the Earth Simulator (ES) in Japan clearly demonstrate that a close collaboration with a vendor to develop a science-driven architectural solution can produce a system that achieves a significant fraction of peak performance for critical scientific applications. This paper discusses the issues contributing to divergence problem, suggests a new approach to address the problem and documents some early successes for this approach.

High performance computing (HPC) provides the superior computational capability required for dramatic advances in key areas of science and engineering such as DNA analysis, drug design, or structural engineering. Over the past decade, progress in this area has been threatened by technology problems that pose serious challenges for continued advances in this field. One of the most important problems has been the lack of adequate language and tool support for programming HPC architectures. In today’s dominating programming paradigm users are forced to adopt a low level programming style similar to assembly language if they want to fully exploit the capabilities of parallel machines. This leads to high cost for software production and error-prone programs that are difficult to write, reuse, and maintain. This paper discusses the option of providing a high-level programming interface for HPC architectures. We summarize the state of the art, describe new challenges posed by emerging peta-scale systems, and outline features of the Chapel language developed in the DARPA-funded Cascade project.

As many large-scale distributed parallel programs are run efficiently in the Earth Simulator (ES), I/O throughput between the tape library system and the work disk of the PNs became the bottleneck for simulations. So, in 2003, we introduced NQSII and Mass Data Processing System (MDPS) to address this problem and to improve system utilization and maintainability. In this paper, the new system (a scheduling algorithm and MDPS) and the recent program execution status for the ES are described.

A global/regional non-hydrostatic atmosphere simulation code is introduced as a component of the global/regional non-hydrostatic coupled atmospheric-ocean-land simulation code that has been developed in the Earth Simulator Center. In this paper, outline of the non-hydrostatic atmospheric GCM design is described and preliminary result to validate physical performance are presented. Futhermore, we show its computational performance on the Earth Simulator.

Data assimilation with advanced algorithms based on the Kalman filter and large-scale numerical models is computationally extremely demanding. This motivates the parallelization of the assimilation problem. As another issue, the implementation of a data assimilation system on the basis of existing numerical models is complicated by the fact that these models are typically not prepared to be used with data assimilation algorithms. To facilitate the implementation of parallel data assimilation systems, the parallel data assimilation framework PDAF has been developed. PDAF allows to combine an existing numerical model with data assimilation algorithms, like statistical filters, with minimal changes to the model code. Furthermore, PDAF supports the efficient use of parallel computers by creating a parallel data assimilation system. Here the structure and abilities of PDAF are discussed. In addition, the application of filter algorithms based on the Kalman filter is discussed. Data assimilation experiments show an excellent parallel performance of PDAF.

The adoption of the Extended Kalman Filter (EKF) for data assimilation in operational weather forecasting would provide estimates of prediction error covariance and make it possible to take model error into account in the assimilation process. Unfortunately, full-blown Kalman filtering is not feasible even on the fastest parallel supercomputers. We propose a novel approximation to EKF, called the Variational Kalman Filter, or VKF. In VKF, a low rank approximation to the prediction error covariance matrix is generated by a very short, temporally local four dimensional variational assimilation cycle, at a low computational cost. VKF provides a locally optimal approximation to the error covariance matrix and also allows model error to be incorporated in the assimilation process. Initial numerical tests with VKF applied to an advection-reaction-dispersion equation are reported.

An increasing number of systems and solutions in the High-Performance Computing (HPC) market segment are moving from traditional proprietary (Vector-) Supercomputing and RISC/UNIX machines towards utilizing industry standards building block technologies to leverage volume economics such as the Intel® Architecture. This trend continues to enjoy increased performance at decreasing costs with an ongoing injection of new hardware and software technologies. At the same time the application areas of HPC are widening and spanning from science, research and academic over engineering to commercial business computing deployments. Hence, the customer focus is evolving from a pure technological point of view to a more solution oriented approach to increase the business value and higher productivity. This move takes place in both the scientific and industry market segments where HPC is being used as a mean to archive and implement better solutions. Intel® Architecture based HPC systems and solutions are playing a vital role in accelerating and improving science and research capabilities and capacity, business benefits and competitive advantage..

Earth System Modeling is both compute and data intensive. HPC Systems for Earth System Modeling consist of Compute- and Data-Servers as well as additional servers for visualization and other purposes, that are linked together to provide an efficient, easy to use workflow environment Huge amounts of data are exchanged between the Data-Server and its clients. Coupling the different servers of an HPC System via Shared File Systems is an effective solution for the data transfer problem, which has been successfully implemented at DKRZ. All Data and Scalar Compute Services are IA64, Linux-based computer systems which use the Shared File System for data exchange. Lessons learnt from this implementation will be reported.

The Australian Bureau of Meteorology has recently made a major upgrade to its supercomputer and its Central Computer Facilities (CCF), located in the Bureau’s new headquarters in Melbourne. This paper describes the overall CCF with particular emphasis on the new NEC SX-6 supercomputing facilities. The porting experiences in migrating from the previously installed SX-5 facilities to the multi-node SX-6 environment supported by NEC IA-64 TX7 file servers and associated global file system are described. The system usage and performance of several major Bureau operational applications on the SX-6 are presented along with a discussion of planned upgrades of these applications.

The Met Office 4D-Var scheme went operational in October 2004. In order to produce a timely analysis, 6 nodes are used compared to the 1 node that was previously required for 3D-Var. Even with this 6 fold increase in resources, 4D-Var takes 4 times longer to complete: Optimising 4D-Var for the SX-6 has been, and continues to be, a high priority. Porting the code from the Cray T3Es is described, along with optimisations introduced to October 2004, such as techniques for increasing vector lengths and reducing communication overheads. Impacts of optimisations are presented, along with investigations into the scalability of the code on a per routine basis.

The first non-beta release of the Weather Research and Forecast (WRF) modeling system in May, 2004 represented a key milestone in the effort to design and implement a fully-functioning, next-generation modeling system for the atmospheric research and operational NWP user communities. With efficiency, portability, maintainability, and extensibility as bedrock requirements, the WRF software framework has allowed incremental and reasonably rapid development while maintaining overall consistency and adherence to the architecture and its interfaces. The WRF 2.0 release supports the full-range of functionality envisioned for the model including efficient scalable performance on a range of high-performance computing platforms, multiple dynamic cores and physics options, low-overhead two-way interactive nesting, moving nests, model coupling, and interoperability with other common model infrastructure efforts such as ESMF.

With the application of NWP developed these years in China, the scale of the operational suite has increased. An efficient managing system is important for stability of the operational suite. It is necessary for us to do business process re-engineering in NWP. Given the distribution and diversification of NWP applications the challenge for CMA is how best to manage the operational system within the available facilities. Some methods and projects are being put in practice. In this paper, our approach on standardizing the running interface; visualizing the running processes by SMS are shown: and a ongoing project “National Meteorological data Access and Retrieval System”, a general way for NWP data access in China is described.

For operational numerical weather prediction, the Japan Meteorological Agency (JMA) has been running analysis and forecast models on the HITACHI SR8000 model E1 since March 2001. On the next-generation supercomputer procurement announced in April 2004, the HITACHI SR11000 was judged to be best. The JMA plans to improve its analysis and forecast models on the new computer.

This paper highlights the need to build a grid-based infrastructure to meet the challenges facing NOAA in the 21^st century. Given the enormous expected demands for data, and increased size and density of observational systems, current systems will not be scalable for future needs without incurring enormous costs. NOAA’s IT infrastructure is currently a set of independent systems that have been built up over time to support its programs and requirements. NOAA needs integrated systems capable of handling a huge increase in data volumes from expected launches of GOES-R, NPOESS, new observing systems being proposed or developed, and to meet requirements of the Integrated Earth Observation System. Further, NOAA must continue moving toward integrated compute resources to reduce costs, to improve systems utilization, to support new scientific challenges and to run and verify increasingly complex models using next generation high-density data streams. Finally, NOAA needs a fast, well-managed network capable of meeting the needs of the organization: to efficiently distribute data to users, to provide secure access to IT resources, and be sufficiently adaptable and scalable to meet unanticipated needs in the future.

An essential pattern of Grid computing is the virtualisation of resources. The NERC DataGrid (NDG) is a UK e-Science project that will provide discovery of, and virtualised access to, a wide variety of climate and earth-system science data. We present an overview of key elements of the NDG architecture beginning with the NDG metadata taxonomy for various information categories. Architecture components include: federation of discovery metadata exploiting the harvesting protocols of the Open Archives Initiative; a domain ontology to support navigation of data relationships during search and discovery; data storage transparency achieved through a wrapper mechanism and a data model based on emerging ISO standards for spatial data; and finally, a role-based authorisation framework which interfaces with existing access control infrastructures and supports mappings between credentials to establish networks of trust.

State-of-the-art atmospheric models comprise multiple components that interact synchronously. Because some components need to expend more computational effort than others, it is often the case that these components are computationally imbalanced. Although parallelism within a component can reduce the imbalance, there is still a need to coordinate component interaction via a coupler in real time. To address this issue, NCAR identified requirements for Task Geometry — a construct that specifies for the purpose of coordination the run-time topology within and between components. Although it has recently become generically available in Platform LSF HPC, the current focus is commodity architectures based on the Linux operating environment. Crafted originally for CCSM, Task Geometry appears applicable to other systems-coupled atmospheric models such as 4D-Var. Scale-coupled atmospheric models also show promise for application in areas such as subgrid-scale parameterizations for GCM models, and the class of interactions demanded by use cases emerging in the area of Homeland Security. In all cases it is possible to demonstrate via speedup and efficiency that Task Geometry enhances performance, however both metrics are problematical to quantify except in simple cases. With applicability from isolated Linux clusters to grids, Task Geometry remains useful in contexts that span organizational and/or geographic boundaries.

The Community Climate System Model (CCSM3) is the primary model for global climate research in the United States and is supported on a variety of computer systems. We present some of our porting experiences and describe the current performance of the CCSM3 on the Cray X1. We include the status of work in progress on other systems in the Cray product line.

Most modern architectures employ a hybrid memory model where memory is shared across some processors and distributed among others (clusters of SMPs). We describe a syntax for expressing memory dependencies in grid codes that takes the same form on distributed or shared memory, and can be optimally implemented on either, and on any hybrid layering thereof. This syntax can be applied to scalars, arrays or any other distributed data object. The syntax may be implemented on any of several current parallel programming standards, and also provides a way forward to future higher-level libraries.

No abstract received.

LIST OF PARTICIPANTS.