Reconstruction of gridded model data received via noaaport

Abstract

Our on-going work focuses on systems for and applications of operational mesoscale numerical weather prediction. In particular, our goal is to provide weather forecasts at a level of precision and fast enough to address specific weather-sensitive operations. Hence, we are addressing problems of high-performance computing, visualization, and automation while designing, evaluating and optimizing an integrated system that includes receiving and processing data, modelling, and post-processing analysis and dissemination. One aspect of the integration and automation is the creation of appropriate initial and boundary conditions for the nested domain for each geographic area for which we produce an operational, model-based forecast. In that regard, we are using the results from the Eta synoptic-scale model operated by the National Centers for Environmental Prediction, which covers all of North America and surrounding oceans at 12 km resolution with 60 vertical levels. These data are made available via the National Weather Service NOAAport satellite-based data transmission system after sampling to 40 km resolution on the AWIPS 212 grid and interpolated to 27 isobaric levels for the continental United States in a Lambert-Conformal projection. A subset of the surface fields are available at 20 km resolution on the AWIPS 215 grid, which we also utilize. The NOAAport system provides a number of different data sources as disseminated by the National Weather Service. These include in situ and remotely sensed observations, and model data such as the aforementioned Eta data. We currently operate a four-channel facility manufactured by Planetary Data, Incorporated. This NOAAport receiver system, based upon Red Hat Enterprise Linux, has a very flexible design, enabling the type of customization and integration necessary to satisfy our project goals. The various files transmitted via NOAAport are converted into conventional files in Unix filesystems in their native format, accessible via NFS mounting on both Linux and AIX systems via a private gigabit ethernet. We have decomposed our processing of the Eta data into two parts. The first is essentially a parsing of the data received via NOAAport into usable formats to be used by the second part -- analysis and visualization. For the aforementioned Eta-212 and Eta-215 grids, the data are received in the compressed Gridded Binary or GriB (GriB-1) format. The data are uncompressed (deGriBbed) via an automated process, which is run as a periodic Unix cron job for each of the four Eta runs per day (0Z, 6Z, 12Z and 18Z). It provides a set of flat binary files (one per each three-hour time step) as input to several other processes, and a set of summary statistics. These processes include isentropic analysis, which leads to appropriate initial and boundary conditions for mesoscale model execution, forecast verification and comparison, and interactive and production, web-based three-dimensional visualization. These consumers of the Eta data are relatively sensitive to degradation of the content of the GriB files that may occur due to problems at transmission or reception. For example, information about the former is not readily available a priori, although issues do arise on an irregular basis, which lead to retransmissions of the data at a later time. Problems at reception are typically due to aperiodic adjustments in the spacecraft signal strength, severe local weather or nearby flyovers of US Air Force AWACS planes. Since these factors are outside of our control, we determined that a simple format translation and related transformations (i.e., units, projections) is inadequate. Instead, we designed and implemented an approach that treats the input arrays of volumetric and surface fields from the GriB files as potentially incomplete samplings of a model atmosphere, and the output as a reconstructed representation of the intended Eta results. Depending on the extent of the degradation, different algorithms are introduced to compensate for data dropouts, intentional incomplete population of specific arrays and data outside of a reasonable dynamic range. This approach is implemented in Java to enable both portability and flexibility for different platforms and data sources. It has replaced an older, simplistic processing code in forecast operations, and provided more reliable and robust results, especially when the quality of the received GriB files has been compromised. The implementation is currently being extended to accommodate full-resolution Eta results at 12 km. These data are being transmitted via the NOAAport Direct Video Broadcast channel using the new GriB-2 format on the AWIPS 218 grid, which leverages JPEG2000-based lossless compression. This data stream will eventually replace the GriB-1-based 212 and 215 grids. Hence, there is a clear need to migrate away from processing of the older data format, and an opportunity to investigate the scalability of this approach to data reconstruction. In addition, it is expected that these ideas will be applied to other NOAAport-transmitted gridded data sets.