Tables for
Volume C
Mathematical, physical and chemical tables
Edited by E. Prince

International Tables for Crystallography (2006). Vol. C, ch. 7.5, pp. 666-676

Chapter 7.5. Statistical fluctuations

A. J. C. Wilsona

aSt John's College, Cambridge CB2 1TP, England

Chapter 7.5 discusses the effects of statistical fluctuation on the measurement of diffracted intensities. Fixed-time counting and fixed-count timing are compared. Complicating factors, such as detector dead time and source fluctuations are analyzed. The problem of measured-as-negative intensities is discussed, and strategies for optimization of counting time are analyzed.

7.5.1. Distributions of intensities of diffraction

| top | pdf |

Intensities of diffraction have two distinct probability distributions: (1) the a priori probability that an arbitrarily chosen reflection of a particular substance will have a particular `true' intensity (R in the notation used below), and (2) the probability that a `true' R will have an observed value [R_{o}]. The distributions of the first type depend on the symmetry and composition of the material, and are treated in Chapter 2.1[link] of Volume B of International Tables for Crystallography (Shmueli, 2001[link]). The distributions of fluctuations of the second type, variations of the values [R_{o}] observed for a particular reflection, are treated here.

The crude counts or counting rates are rarely used directly in crystallographic calculations; they are subjected to some form of data processing to provide `intensities of reflection' [I_{o}] in a form suitable for the determination of crystal structures, electron densities, line profiles for the study of defects, etc. The resulting intensity for the jth reflection, [I_{o,\,j}], will be directly proportional to the corresponding [R_{o,\,j}]; when no confusion can arise, one of the subscripts will be omitted. The value of the proportionality factor [c_{j}] in [ I_{o,\,j} = c_{j}R_{o,\,j}\eqno (]will be different for different reflections, since they will occur at different Bragg angles, have different absorption corrections, etc.

7.5.2. Counting modes

| top | pdf |

Whatever the radiation, in both single-crystal and powder diffractometry, the integrated intensity of a reflection is obtained as a difference between a counting rate averaged over a volume of reciprocal space intended to include the reflected intensity and a counting rate averaged over a neighbouring volume of reciprocal space intended to include only background. If these intentions are not effectively realized, there will be a systematic error in the measured intensity, but in any case there will be statistical fluctuations in the counting rates. The two basic modes (Parrish, 1956[link]) are fixed-time counting and fixed-count timing. In the first, counts are accumulated for a pre-determined time interval, and the variance of the observed counting rate is proportional to the true (mean) counting rate. In the second, on the other hand, the counting is continued until a pre-determined number of counts is reached, and the variance of the observed counting rate is proportional to the square of the true counting rate. Put otherwise, the relative error in the intensities goes down inversely as the square root of the intensity for fixed-time counting, whereas it is independent of the intensity for fixed-count timing. Each mode has advantages, depending on the purpose of the measurements, and numerous modifications and compromises have been proposed in order to increase the efficiency of the use of the available time. References to some of the many papers are given in Section 7.5.7[link].

In principle, probability distributions can be determined for any postulated counting mode. In practice, they become complicated for all but the simplest modes; this is true even for the single measurement of the total counting rate or the background counting rate, but is even more pronounced for the distribution of their difference (the reflection-only rate). For most crystallographic purposes, however, it is only necessary to know the mean (to correct for bias, if present) and the variance (for the estimation of weights in refinement processes, see Part 8[link] ) of the distribution function.

7.5.3. Fixed-time counting

| top | pdf |

In the absence of disturbing influences (mains-voltage fluctuations, unrectified or unsmoothed high-tension supplies, `dead time' of the counter or counter circuits, etc.), the number of counts recorded during the predetermined time interval used in the fixed-time mode will fluctuate in accordance with the Poisson probability distribution. If the `true' number of counts to be expected in the interval is N, the probability that the observed number will be [N_{o}] is given by [ p(N_{o}) = \exp (-N)N^{N_{o}}/N_{o}!, \eqno (]where all quantities appearing are necessarily non-negative. Both the mean and the variance of [N_{o}] are N. If the `true' number of counts to be expected when the diffractometer is set to receive a reflection is T, and the `true' number when it is set to receive the immediate background is B, the `true' intensity of the reflection is [ R = T - B, \eqno {} \eqno (]provided that the time interval used for the reflection is equal to the time interval used for the background. In practice, the `observed' values of [T_{o}] and [B_{o}] fluctuate with probabilities given by ([link] with T or B replacing N, so that the observed value, [ R_{o} = T_{o} - B_{o}, \eqno (]will also fluctuate, and can, occasionally, take on negative values. The treatment of `measured-as-negative' intensities is discussed in Section 7.5.6[link].

Although the sum of two Poisson-distributed variables is also Poisson, the difference is not, and the probability of [R_{o}] given by ([link] has been shown to be (Skellam, 1946[link]; Wilson, 1978[link]) [ p(R_{o}) = \exp \{-(B+T)\}(T/B)^{R_{o}/2} I_{|R_{o}|}[2(BT)^{1/2}], \eqno (]where [I_{n}] is the hyperbolic Bessel function of the first kind. The mean and variance of [R_{o}] are TB and T + B, respectively. If the times used for reflection and background are not equal, but are in the ratio of k : 1, the mean and variance of [ R_{o} = T_{o} - kB_{o} \eqno (]are TkB and [T + k^2B], respectively. The distribution of [R_{o}] then involves a generalized Bessel function1 depending on k: [ p(R_{o}) = \exp \{-(B+T)\}(T^{2-k}/B)^{R_{o}/2} I^{k}_{|R_{o}|}[2(BT^{k})^{1/2}]; \eqno (]for the properties of the generalized Bessel function, see Wright (1933[link]) and Olkha & Rathie (1971[link]). The probability distributions ([link] and ([link] seem to be very close to a normal distribution with the same mean and variance for [R_{o}] near R, but the probability of moderate deviations from R is less than normal and for large deviations is greater than normal.

In applications, it is usual to work with intensities expressed as counting rates, rather than as numbers of counts, even when fixed-time counting is used. The total intensity τ expressed as a counting rate is [ \tau = T/t, \eqno {} \eqno (]where t is the time devoted to the measurement, and the variance of the counting rate is [ \sigma ^{2}(\tau) = T/t^{2} = \tau /t.\eqno (]Similar expressions apply for the background, with B for the count, b for the time, and β for the counting rate. For the reflection count, the corresponding expressions are [\rho = T/t - B/b, \eqno (] [\sigma ^{2}(\rho) = \tau /t + \beta /b.\eqno (]To avoid confusion, upper-case italic letters are used for numbers of counts, lower-case italic for counting times, and the corresponding lower-case Greek letters for the corresponding counting rates. In accordance with common practice, however, [I_{j}] will be used for the intensity of the jth reflection, the context making it clear whether I is a number of counts or a counting rate.

7.5.4. Fixed-count timing

| top | pdf |

The probability of a time t being required to accumulate N counts when the true counting rate is ν is given by a Γ distribution (Abramowitz & Stegun, 1964[link], p. 255): [ p(t)\,{\rm d}t = [(N-1)!]^{-1}(\nu t)^{N-1}\exp (-\nu t)\,{\rm d}(\nu t). \eqno (]The ratio N/t is a slightly biased estimate of the counting rate ν; the unbiased estimate is (N − 1)/t. The variance of this estimate is [\nu ^{2}/(N-2)], or, nearly enough for most purposes, [(N-1)^{2}/(N-2)t^{2}]. The differences introduced by the corrections −1 and −2 are generally negligible, but would not be so for counts as low as those proposed by Killean (1967[link]). If such corrections are important, it should be noticed that there is an ambiguity concerning N, depending on how the timing is triggered. It may be triggered by a count that is counted, or by a count that is not counted, or may simply be begun, independently of the incidence of a count. Equation ([link] assumes the first of these.

Equation ([link] may be inverted to give the probability distribution of the observed counting rate νo instead of the probability distribution of the time t: [\eqalignno{ p(\nu _{o})\,{\rm d}\nu _{o} &= [(N-1)!]^{-1}[\nu (N-1)/\nu _{o}]^{N-1} \cr &\quad \times \exp \{-(N-1)\nu /\nu _{o}\}\,{\rm d}[\nu _{o}/(N-1)\nu] . & (}]There does not seem to be any special name for the distribution ([link]. Only its first (N − 1) moments exist, and the integral expressing the probability distribution of the difference of the reflection and the background rates is intractable (Wilson, 1980[link]).

7.5.5. Complicating phenomena

| top | pdf | Dead time

| top | pdf |

After a count is recorded, the detector and the counting circuits are `dead' for a short interval, and any ionizing event occurring during that interval is not detected. This is important if the dead time is not negligible in comparison with the reciprocal of the counting rate, and corrections have to be made; these are large for Geiger counters, and may sometimes be necessary for counters of other types. The need for the correction can be eliminated by suitable monitoring (Eastabrook & Hughes, 1953[link]); other advantages of monitoring are described in Chapter 2.3[link] . Voltage fluctuations

| top | pdf |

Mains-voltage fluctuations, unless compensated, and unsmoothed high-tension supplies may affect the sensitivity of detectors and counting circuits, and in any case cause the probability distribution of the arrival of counts to be non-Poissonian. Backlash in the diffractometer drives may be even more important in altering the observed counting rates. As de Boer (1982[link]) says, the ideal distributions represent a Utopia that experimenters can approach but never reach. He observed erratic fluctuations in counting rates, up to ten times as big as the expected statistical fluctuations. When care is taken, the instabilities observed in practice are much less than those of the extreme cases described by de Boer. Stabilizing an X-ray source and testing its stability are discussed in Subsection[link] .

7.5.6. Treatment of measured-as-negative (and other weak) intensities

| top | pdf |

It has been customary in crystallographic computations, but without theoretical justification, to omit all reflections with intensities less than two or three times their standard uncertainties. Hirshfeld & Rabinovich (1973[link]) asserted that the failure to use all reflections, even those for which the subtraction of background has resulted in a negative net intensity, at their measured values will lead to a bias in the parameters resulting from a least-squares refinement. This is, however, inconsistent with the Gauss–Markov theorem (see Section 8.1.2[link] ), which shows that least-squares estimates are unbiased, independent of the weights used, if the observations are unbiased estimates of quantities predicted by a model. Giving some observations zero weight therefore cannot introduce bias. Provided the set of included observations is sufficient to give a nonsingular normal equations matrix, parameter estimates will be unbiased, but inclusion of as many well determined observations as possible will yield the most precise estimates. Requiring that the net intensity be greater than 2σ assures that the value of |F| will be well determined. Furthermore, Prince & Nicholson (1985[link]) showed that, if the net intensity, I, or |F|2 is used as the observed quantity, weak reflections have very little leverage (see Section 8.4.4[link] ), and therefore omitting them cannot have a significant effect on the precision of parameter estimates.

The use of negative values of I or |F|2 is also inconsistent with Bayes's theorem, which implies that a negative value cannot be an unbiased estimate of an inherently non-negative quantity. There are statistical methods for estimating the positive value of |F| that led to a negative value of I. The best known approach is the Bayesian method of French & Wilson (1978[link]), who observe that "Instead of thanking the data for the information that certain structure factor moduli are small, we accuse them of assuming `impossible' negative values. What we should do is combine our knowledge of the non-negativity of the true intensities with the information concerning their magnitudes contained in the data."

7.5.7. Optimization of counting times

| top | pdf |

There have been many papers on optimizing counting times for achieving different purposes, and all optimization procedures require some knowledge of the distributions of counts or counting rates; often only the mean and variance of the distribution are required. It is also necessary to know the functional relationship between the quantity of interest and the counts (counting rates, intensities) entering into its measurement. Typically, the object is to minimize the variance of some function of the measured intensities, say [F(I_{1}, I_{2}, \ldots, I_{j}, \ldots)]. General statistical theory gives the usual approximation [ \sigma ^{2}(F) = {\sum _{i,\,j}} {\partial F \over \partial I_{i}} {\partial F \over \partial I_{j}} {\rm cov}(I_{i},I_{j}), \eqno (]where cov(Ii, Ij) is the covariance of Ii and Ij if [i \neq j], and is the variance of [I_{j}], [\sigma ^{2}(I_{j})], if i = j. There is very little correlation2 between successive intensity measurements in diffractometry, so that cov(Ii, Ij) is negligible for [i \neq j]. Equation ([link] becomes [ \sigma ^{2}(F) = {\sum _{j}} \left ({\partial F \over \partial I_{j}}\right)^{2}\sigma ^{2}(I_{j}). \eqno (]These equations are strictly accurate only if F is a linear function of the I's, a condition satisfied for the integrated intensity, but for few other quantities of interest. In most applications in diffractometry, however, the contribution of each Ij is sufficiently small in comparison with the total to make the application of equations ([link] and ([link] plausible. Any proportionality factors [c_{j}] (Section 7.5.1[link]) can be absorbed into the functional relationship between F and the Ij's.

The object is to minimize σ2(F) by varying the time spent on each observation, subject to a fixed total time [ T = {\textstyle\sum\limits _{j}}\, t_{j}.\eqno (]It is simplest to regard the total intensity and the background intensity as separate observations, so that in ([link] the sum is over n `background' and n `total' observations. With [I_{j}] expressed as a counting rate, its variance is [I_{j}/t_{j}] [equation ([link]], so that ([link] becomes [ \sigma ^{2}(F) = {\textstyle\sum\limits_{j}}\, G_{j}^{2}I_{j}/t_{j},\eqno (]where for brevity G has been written for [|\partial F/ \partial I|]. The variance of F will be a minimum if, for any small variations [{\rm d}t_{j}] of the counting times [t_{j}], [ 0 = - {\textstyle\sum\limits _{j}}\, G_{j}^{2}I_{j}t_{j}^{-2}\, {\rm d}t_{j}, \eqno (]subject to the constancy of the total time T. There is thus the constraint [ 0 = {\textstyle\sum\limits _{j}} {\,{\rm d}}t_{j}. \eqno (]These equations are consistent if for all j [ G_{j}^{2}I_{j}t_{j}^{-2} = k^{-2}, \eqno (] [ t_{j} = kG_{j}I_{j}^{1/2}, \eqno (]where k is a constant determined by the total time T: [ T = k {\textstyle\sum\limits _{j}}\,G_{j}I_{j}^{1/2}. \eqno (]The minimum variance is thus achieved if each observation is given a time proportional to the square root of its intensity. A little manipulation now gives for the desired minimum variance [ \sigma ^{2}_{\rm min}(F) = {1 \over T}\left [{\sum _{j}}\left (\partial F/ \partial I_{j}\right)I_{j}^{1/2}\right] ^{2}. \eqno (]The minimum variance is found to be a perfect square, and the standard uncertainty takes a simple form.

Here, the optimization has been treated as a modification of fixed-time counting. However, the same final expression is obtained if the optimization is treated as a modification of fixed-count timing (Wilson, Thomsen & Yap, 1965[link]).

Space does not permit detailed discussion of the numerous papers on various aspects of optimization. If the time required to move the diffractometer from one observation position to another is appreciable, the optimization problem is affected (Shoemaker & Hamilton, 1972[link], and references cited therein). There is some dependence on the radiation (X-ray versus neutron) (Shoemaker, 1968[link]; Werner, 1972a[link],b[link]). A few other papers of historical or other interest are included in the list of references, without detailed mention in the text: Grant (1973[link]); Killean (1972[link], 1973[link]); Mack & Spielberg (1958[link]); Mackenzie & Williams (1973[link]); Szabó (1978[link]); Thomsen & Yap (1968[link]); Zevin, Umanskii, Kheiker & Panchenko (1961[link]).


Abramowitz, M. & Stegun, I. A. (1964). Handbook of mathematical functions. National Bureau of Standards Publication AMS 55.
Boer, J. L. de (1982). Statistics of recorded counts. Crystallographic statistics, edited by S. Ramaseshan, M. F. Richardson & A. J. C. Wilson, pp. 179–186. Bangalore: Indian Academy of Sciences.
Eastabrook, J. N. & Hughes, J. W. (1953). Elimination of dead-time corrections in monitored Geiger-counter X-ray measurements. J. Sci. Instrum. 30, 317–320.
French, S. & Wilson, K. (1978). On the treatment of negative intensity observations. Acta Cryst. A34, 517–525.
Grant, D. F. (1973). Single-crystal diffractometer data: the on-line control of the precision of intensity measurement. Acta Cryst. A29, 217.
Hirshfeld, F. L. & Rabinovich, D. (1973). Treating weak reflexions in least-squares calculations. Acta Cryst. A29, 510–513.
Killean, R. C. G. (1967). A note on the a priori estimation of R factors for constant-count-per-reflection diffractometer experiments. Acta Cryst. 23, 1109–1110.
Killean, R. C. G. (1972). The a priori optimization of diffractometer data to achieve the minimum average variance in the electron density. Acta Cryst. A28, 657–658.
Killean, R. C. G. (1973). Optimization of scan procedure for single-crystal X-ray diffraction intensities. Acta Cryst. A29, 216–217.
Mack, M. & Spielberg, N. (1958). Statistical factors in X-ray intensity measurements. Spectrochim. Acta, 12, 169–178.
Mackenzie, J. K. & Williams, E. J. (1973). The optimum distribution of counting times for determining integrated intensities with a diffractometer. Acta Cryst. A29, 201–204.
Olkha, G. S. & Rathie, P. N. (1971). On a generalized Bessel function and an integral transform. Math. Nachr. 51, 231–240.
Paciorek, W. A. & Chapuis, G. (1994). Generalized Bessel functions in incommensurate structure analysis. Acta Cryst. A50, 194–203.
Parrish, W. (1956). X-ray intensity measurements with counter tubes. Philips Tech. Rev. 17, 206–221.
Prince, E. & Nicholson, W. L. (1985). The influence of individual reflections on the precision of parameter estimates in least squares refinement. Structure & statistics in crystallography, edited by A. J. C. Wilson, pp. 183–195. Guilderland, NY: Adenine Press.
Shmueli, U. (2001). Editor. International tables for crystallography. Vol. B. Reciprocal space, 2nd ed. Dordrecht: Kluwer Academic Publishers.
Shoemaker, D. P. (1968). Optimization of counting times in computer-controlled X-ray and neutron single-crystal diffractometry. Acta Cryst. A24, 136–142.
Shoemaker, D. P. & Hamilton, W. C. (1972). Further remarks concerning optimization of counting times in single-crystal diffractometry: rebuttal to Killean; consideration of background counting and slewing times. Acta Cryst. A28, 408–411.
Skellam, J. G. (1946). The frequency distribution of the difference between two Poisson values belonging to different populations. J. R. Stat. Soc. 109, 296.
Szabó, P. (1978). Optimization of the measuring time in diffraction intensity measurements. Acta Cryst. A34, 551–553.
Thomsen, J. S. & Yap, F. Y. (1968). Simplified method of computing centroids of X-ray profiles. Acta Cryst. A24, 702–703.
Werner, S. A. (1972a). Choice of scans in X-ray diffraction. Acta Cryst. A28, 143–151.
Werner, S. A. (1972b). Choice of scans in neutron diffraction. Acta Cryst. A28, 665–669.
Wilson, A. J. C. (1967). Statistical variance of line-profile parameters. Measures of intensity, location and dispersion. Acta Cryst. 23, 888–898.
Wilson, A. J. C. (1978). On the probability of measuring the intensity of a reflection as negative. Acta Cryst. A34, 474–475.
Wilson, A. J. C. (1980). Relationship between `observed' and `true' intensity: effect of various counting modes. Acta Cryst. A36, 929–936.
Wilson, A. J. C., Thomsen, J. S. & Yap, F. Y. (1965). Minimization of the variance of parameters derived from X-ray powder diffractometer line profiles. Appl. Phys. Lett. 7, 163–165.
Wright, E. M. (1933). On the coefficients of power series having exponential singularities. J. London Math. Soc. 8, 71–79.
Zevin, L. S., Umanskii, M. M., Kheiker, D. M. & Panchenko, Yu. M. (1961). K voprosu o difraktometricheskikh priemah pretsizionnyh izmerenii elementarnyh yacheek. Kristallografiya, 6, 348–356.

to end of page
to top of page