Within the framework of numerical analysis, Lorenc and Hammon (1988) study the quality control process in terms of Bayesian probability theory, in which "the error in each datum is either a normal observation error, from a known Gaussian distribution, or a gross error, in which case the datum gives no information". In the present note, what was defined above as "observation error" will be considered to contain the two parts considered by Lorenc and Hammon: 1) instrument and other errors with Gaussian distribution, and 2) rough observation errors with (assumed) uniform distribution. The method of performing the quality control of these data by cqcbufr closely follows the development of Schyberg and Breivik (1997), who made the same assumptions.

The cqcbufr is documented in Collins (1998, 2000?) . Here only a broad outline will be given, especially as it is relevant to the quality control of observation errors (as defined in Section 1). The data are read in for a first time and all residuals are calculated. The residuals are: 1) increments (difference between the observed value and the 6-hour forecast value), 2) vertical residuals (difference between the increment and vertically interpolated increment, using statistical interpolation), 3) horizontal residuals (difference between the increment and horizontally interpolated increment, using statistical interpolation), 4) hydrostatic residuals (difference between a layer thickness, computed by heights and by temperatures and moisture), 5) baseline residual (a form of hydrostatic residual), and 6) lapse rate residuals.

For use by the observation error detection algorithm, the residuals are converted to normal standard deviates:

where is the normalized residual for type t (1-3 above), is the residual value, is its mean, and is its standard deviation. The mean and standard deviation are determined from all the data for the present data time in which a crude attempt is made to eliminate from the computation, data that very likely have a large rough error. In other words, the mean and standard deviation used in the normalization are supposed to represent values appropriate to the part of the error distribution that has a Gaussian distribution.

The data are then read in a second time, one report at a time. The full power of the Decision Making Algorithm (DMA) is used to examine a complete profile of data for communication and computation errors. Only when the complete profile has been examined, and any possible corrections are made, will the DMA examine the data for observation errors. The remainder of this note will discuss that examination, under the assumption that the observation errors are composed of the two parts as discussed above—a part with Gaussian distribution (with parameters now known) and a part with (assumed) uniform distribution.

Observation errors are quite common, but only a few examples will be given. In the first example, the temperatures at three levels

were marked as bad because of observation errors. In the tables below, the shorthand shown below will be used.

In the second example, the height at a single level is flagged since the residuals are excessive in size. And in the last example, the top level height is flagged as bad. All three of these examples are typical.

Table 1. Temperature observation errors for 35700 at 00 UTC 21 March, 2000.

Table 2. Height observation error for 56096 at 00 UTC 21 March, 2000.

Table 3. Height observation error for 48455 at 00 UTC 21 March, 2000.

Statistics are collected from the routine running of cqcbufr to summarize them on a monthly basis. Some information has been extracted on the observation errors detected during February, 2000, in which there were 31760 radiosonde reports collected at NCEP. There were 1773 observation errors detected, of which 217 received quality mark 3 (questionable) and 1556 received quality mark 13 (bad). By contrast, the presently operational code diagnosed 11,463 data with observation errors, with 9480 given quality mark 3 and 1983 give quality mark 13. The new observation error quality control makes more definitive decisions and slightly reduces the number of "bad" quality marks. In pressure, the largest number were at

1000 hPa, with counts trailing off in height. (A minimum value at 925 hPa is likely caused by fewer observations reporting this level.)

The geographic distribution is far from uniform, and is given in Table 4. for the major regions of the globe.

Table 4. Distribution of observation errors by large region for February 2000

The absolute numbers of errors detected is not of primary importance since this can be tuned to some extent by the choice of various parameters. It was the objective to let the data itself determine them as much as possible, and manual examination of the results looks reasonable. In many respects, the result is statistical and one should not expect 100% accuracy. There are bound to be many cases in which bad data are not flagged and many other cases in which good data are flagged. The great majority of the decisions, however, appear to be correct. Using very small values of the mean analysis weight for flagging data as bad is conservative; there are probably more bad data accepted than good data rejected.

Collins, W.G., 1998: The use of complex quality control for the detection and correction of rough errors in rawinsonde heights and temperatures: a new algorithm at NCEP/EMC. NCEP Office Note 419. [Available from NCEP, 5200 Auth Road, Washington, D.C. 20233.]

___________, 2000?: The operational complex quality control of radiosonde heights and temperatures at the National Centers for Environmental Prediction: Part I. Description of the method. Submitted to Journal of Applied Meteorology.

___________, 2000?: The operational complex quality control of radiosonde heights and temperatures at the National Centers for Environmental Prediction: Part II. Examples of error diagnosis and correction and statistics of error determination for a year in operational use. Submitted to Journal of Applied Meteorology.

Lorenc, A.C. and O. Hammon, 1988: Objective quality control of observations using Bayesian methods. Theory, and a practical implementation. Q.J.R. Meteorol. Soc., 114, pp. 515-543.

Schyberg, Harald and Lars-Anders Breivik, 1997: Objective analysis combining observation errors in physical space and observation space. Research Report No. 46, Norwegian Meteorological Institute, 35 pp.