Observational Data Processing

Observational Data Processing at NCEP

NOAA/NWS/NCEP/EMC
(Last Revised 7/8/2020)

Please take a moment to read the Disclaimer for this non-operational web page.

For any questions about NCEP's observational data processing, please contact the NCEP Observation Processing Team.

Most of the observational data at NCEP are stored in WMO BUFR format. This format is an international standard and provides an efficient means for transferring data. In addition it allows for great flexibility for adding new observation elements.

There are a series of tables associated with BUFR. " Table A " defines the data category associated with a particular BUFR message containing report data. " Table B" classifies and defines data elements, or descriptors, according to scale, reference value, number of bits and units. " Table C " defines data description operators. " Table D" defines the list of common sequences. In addition, there are BUFR code and flag tables as well as code tables common to BUFR and other binary and alphanumeric codes . The need for external tables can make the process for BUFR data encoding and decoding quite cumbersome for a typical user.

As such, a special application has been designed at NCEP which provides user-friendly access to the BUFR files through a series of FORTRAN and C subroutines in a machine independent BUFR library (called BUFRLIB). These routines allow one to encode or decode data into BUFR using mnemonics to represent the data. The mnemonics are associated with BUFR descriptors in a special version of the Tables A, B, C and D. When a BUFR file is created, the mnemonic table is read in from an external location and is itself encoded into BUFR messages at the top of the output file. These messages have Table A data category (message type) 11 (BUFR tables). This allows each BUFR file to be "self defined". No external tables are needed to decode data out of the file.

The NCEP BUFRLIB team has written a BUFRLIB software user guide which provides a detailed explanation of the NCEP BUFRLIB subroutines along with other useful information on BUFR as it is used at NCEP. Click here to view a sample Fortran code which uses BUFRLIB routines to encode reports into a BUFR file.

Next is a brief outline on the current method for processing observations that arrive at NCEP. ts main function is to provide links to web pages which discuss each item in detail. Click here to see a flow chart depicting the data processing system at NCEP.

1. NCEP receives the majority of its data from the Global Telecommunications System (GTS) and the National Environmental Satellite, Data, and Information Service (NESDIS) .

a. The GTS and aviation circuit bulletins are transferred from the NWS Telecommunication Operations Center (TOC/NWSTG) to NCEP's Central Operations (NCO) and networked to one of two interactive nodes on the NCEP Weather & Climate Operational Supercomputing System (WCOSS) (one production, one development) using LDM (Local Data Manager) and DBNet (Distributive Brokered Networking) software packages. These data are then decoded from their native format and encoded into WMO BUFR format using decoder software designed by NCO's System's Integration Branch.

b. Most of the satellite data are processed in batch mode as they become available from the various NESDIS servers. Regularly scheduled jobs on the WCOSS transfer "new" files from these servers and encode the data into WMO BUFR format.

See https://www.emc.ncep.noaa.gov/emc/pages/infrastructure/obs-data/satellite_ingest.doc/document.php for more information.

c. NCEP receives "Level 3" radial wind data from 158 NEXRAD radar stations via the radar multicast (NIDS format). At eight minutes past each hour, a script runs to gather up the most recent radar data into eight pairs of files containing the raw data and a station list. At thirty-five minutes past each hour these raw data are superobed and encoded into BUFR format.

d. NCEP receives on-site superobed Level 2 (which we call "Level 2.5") radial wind data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Open Systems Radar Product Generator (ORPG) stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Twice each hour at five and thirty-five minutes past the hour, all newly arrived files are first uncompressed and then encoded into WMO BUFR format.

e. NCEP receives the full complement of Level 2 radial wind and reflectivity data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Radar Data Acquisition stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Four times each hour at ten, twenty-five, forty and fifty-five minutes past the hour, all newly arrived files are first uncompressed, then quality controlled (both radial wind and reflectivity) and then encoded into WMO BUFR format.

f. Mesonet surface data are gathered by NOAA/ERSL/GSL and then pushed to an NCO server several times each hour. NCO converts the data from netCDF to WMO BUFR format.

All of the encoded BUFR data are then appended to the appropriate files in the data base. The files are organized by the WMO BUFR type and local subtype and contain information in 24 hour blocks (based on report time). Observational files remain on-line for up to 10 days before migration to off line cartridges. This allows late arriving observations to be accumulated. While on line, there is open access to them not only for operations, but also for research and study.

2. The various NCEP networks access the observational data base at a set time each day (i.e., the data cutoff time) and perform a time-windowed dump of requested observations. Observations of a similar type [e.g., satellite-derived winds ("satwnd"), surface land reports ("adpsfc")] are dumped into individual BUFR files which maintain the original structure of reports, although some interactive quality control is applied, duplicate reports are removed, and upper-air report "parts" are merged. See http://www.emc.ncep.noaa.gov/emc/pages/infrastructure/obs-data/data_dumping.doc/document.php for more information.

3. The final step in preparing most of the "conventional" observational data for assimilation by the analysis involves the execution of a series of programs which read in the observations from the various dump files, add forecast ("first guess") background and observation error information, perform automated quality control, and finally output the observations in a monolithic BUFR file known as "PREPBUFR". See http://www.emc.ncep.noaa.gov/emc/pages/infrastructure/obs-data/prepbufr.doc/document.php for more information.

4. The PREPBUFR file is read by the Global Statistical Interpolation (GSI) analysis which runs in the Global Forecast System (GFS) and the Global Data Assimilation System (GDAS) (running the Global FV3 model) , the regional North American Model (NAM) [running the NOAA Environmental Modeling System (NEMS) version of the Non-Hydrostatic Multi-scale Model in B-grid (NMMB)], the Rapid Refresh (RAP) [running the RAP-configuration of the WRF model with Advanced Rearch WRF (ARW) core, shared with the High-Resolution Rapid Refresh (HRRR) ], and the Real-Time Mesoscale Analysis (RTMA) and UnRestricted Mesoscale Analysis (URMA) .