International
Tables for
Crystallography
Volume F
Crystallography of biological macromolecules
Edited by E. Arnold, D. M. Himmel and M. G. Rossmann

International Tables for Crystallography (2012). Vol. F, ch. 2.2, pp. 64-74
doi: 10.1107/97809553602060000809

Chapter 2.2. Quality indicators in macromolecular crystallography: definitions and applications

H. M. Einspahra* and M. S. Weissb

aPO Box 6483, Lawrenceville, NJ 08648–0483, United States, and bHelmholtz-Zentrum Berlin für Materialien und Energie, Macromolecular Crystallography (HZB-MX), Albert-Einstein-Str. 15, D-12489 Berlin, Germany
Correspondence e-mail:  hmeinspahr@yahoo.com

Many indicators of quality have been developed to monitor a macromolecular crystallography experiment from data collection to refinement of the final model. This chapter offers consensus definitions for most of these indicators and discusses their applications and their impact in the context of current experimental practices. Definitions of the most commonly encountered indicators are presented in a table for quick reference.

2.2.1. Introduction

| top | pdf |

The genesis of this chapter was a perceived need for a single location in the volume in which consensus definitions could be found for the many statistical indicators of quality or figures of merit that have been developed to monitor a macro­molecular crystallography (MX) experiment and its final outcome, a model. The evolving experiment has generated a rich collection of R values, signal-to-noise indicators, correlation coefficients and other figures of merit. As improvements in data collection, processing and other aspects of the experiment continue, we can expect a continued evolution of new or improved indicators of quality with which to monitor the impact of those improvements.

This chapter, then, attempts to provide a comprehensive list of the indicators of quality currently in use and, for each indicator in the list, a precise definition that conforms to consensus interpretations of the literature and current practice. The authors acknowledge that useful indicators may have been missed in the sweep that produced this list, but hasten to point out that the generation of new indicators is an ongoing process with the newest ones often in obscurity for a period before their utility is recognized and adopted by experimenters. There is also a subset of the indicators in the list whose members stand out as universally accepted and crucial indicators that either every experimental description must include or that every experimenter should be familiar with. A summary of these is given at the end of this article in Table 2.2.11.1[link]. The authors also acknowledge that there may be competing definitions for some indicators. Where they occur, these competing definitions will be pointed out and a discussion will be provided which, at a minimum, will attempt to clarify the differences, but will also, where possible, attempt to arbitrate or discriminate among opposing views. It should be noted that the scope of this chapter is primarily concerned with the crystallographic experiment. While quality indicators useful for model refinement are given in Sections 2.2.8[link] to 2.2.10[link][link], the validation of the refined model is covered more extensively in Part 21[link] of this volume.

Before proceeding with the quality indicators themselves, it is necessary to discuss a small collection of parameters whose values, typically left to the experimenter to fix, impact on virtually all the indicators of quality we present here. The reason for this wide impact is that they are used to determine which reflections are included in the final data set in question. In their simplest form, the effect is a limit, applied as a cutoff in intensity or resolution, beyond which reflections are excluded from consideration.

When a cutoff is applied based on intensities, the limiting value may be simply a fixed number, for example zero, or it may vary from reflection to reflection, for example some multiple of the standard uncertainty of the reflection intensity. Reflections with intensities below limiting values are excluded. Reflections excluded in this manner are often referred to as unobserved.

When the objective is to estimate electron-density distributions, the exclusion of intensity terms below very low limiting values is unlikely to have any significant negative impact and may offer a positive savings in computation time. This may not be the case, however, when the objective is refinement. Inclusion of reflections of low intensity, even zero, may have important positive results on the quality of the final model. True, some of these intensities may be poorly known or, if less than zero, physically unrealistic, but given the excellent quality of modern data-collection instrumentation and techniques, few intensities, if any, are without some reasonable estimate of their standard uncertainty, so that weighting procedures can be applied to modulate the impact of individual terms in the final result and judgments can be made about exclusion of intensity values that are truly improbable. With the application of proper weighting procedures, there seems to be little justification for exclusion of reflections from consideration based on intensity (except for the truly improbable mentioned above) when computing indicators of quality. As instrumentation and techniques continue to improve, this is a topic that merits continued attention and debate within the MX community in search of consensus best practices for handling weak reflections and for including them in estimators of quality.

While an acceptable estimate of the nominal resolution of a diffraction data set is widely considered to be a high-value indicator of data quality, assignment of a value to this limit is typically left to the experimenter and is thus prone to subjectivity. Some guidelines for estimation have emerged. One would set the nominal resolution based on the percentage of weak reflections above the limit so that reflections with intensities above that limit would be included. An example might be the resolution at which 70% of unique reflections have intensities above zero or above some multiple of their standard uncertainties. Another way to estimate the nominal resolution which has gained wide acceptance is based on the overall signal-to-noise value where, in its most popular expression, the limit value is the resolution at which the mean signal-to-noise ratio in the outer resolution shell falls to 2. Each of these estimation methods is susceptible to convolution with limits based on intensity leading to, for example, a limit set as the resolution at which the mean signal-to-noise ratio `of observed reflections' falls to 2. The guidelines for estimating resolution limits – the methods to be applied, the constraining values such as 70% or 2, and the proper integration with limits applied based on intensity – also merit further attention and discussion by the MX community, the goal being definition of consensus best practices. We take this opportunity to suggest that the widely accepted estimate, the resolution at which the mean signal-to-noise in the outer resolution shell falls to 2, calculated without imposition of a cutoff in intensity, has much to recommend it. True, some susceptibility to subjectivity remains in the definition of resolution-shell ranges and limits, but the impact is minimal and hardly worth the effort. On the other hand, modifying the indicator to remove the shells so that the mean signal-to-noise ratio applied to all data above an appropriately adjusted limit value would extinguish that source of subjectivity.

When a cutoff is applied based on nominal resolution, then considerations similar to those for intensity-based cutoffs apply. If the objective is to estimate electron-density distributions, exclusion of large numbers of weak intensity terms beyond a limiting resolution is unlikely to be of any significant impact except, perhaps, a favourable one in computation time. If a sparse population of more intense reflections is also excluded, the effect may also be positive by reducing aberrations that might interfere with interpretation. If the objective is refinement, the benefit is less clear, except possibly in computation time. The cost for that may be the exclusion of a few more intense terms that provide positive guidance to refinement. Another positive effect of a sharply defined limiting resolution might be improved estimation of the spherical interference function where it is needed. It appears that, in practice, the consensus is that, in applying a cutoff, the exclusion of a few intense terms is at worst of negligible impact overall.

Because many imposed limits that exclude data are dependent on estimates of standard uncertainty, procedures for estimation of standard uncertainty need to be considered. As suggested earlier, a diffraction data set without a standard uncertainty for each intensity measurement is certainly the exception in current practice. The methods used by data-processing programs to estimate these standard uncertainties may be difficult, even impossible, to discern, but at core they must all be based on counting statistics. With the continuing trend toward data sets of high multiplicity (also referred to as redundancy), however, estimates of standard uncertainties are available from distributions of replicate measurements about means. Both of these estimates have value and express impacts of somewhat different sources of error. It would seem therefore that, where applicable, the best way to accommodate both would be to calculate weighted average values of individual intensities, where the weights are derived from the standard uncertainties from data processing, and then to estimate the standard uncertainties for the weighted averages by application of standard propagation-of-error procedures. This, finally, is the third area we highlight here that deserves focused discussion within the MX community with the objective of defining consensus best practices.

Little remains to be included in these introductory preliminaries. It should be understood that most of these indicators of quality may be cast in terms of either structure factors or intensities. We draw little attention to this as we define individual indicators, except that, where one of the two possibilities seems to dominate in usage, we tend to define that form alone. It is also true that many of these indicators have counterparts in which individual reflection terms are weighted. In current practice, these forms find little use in application to biological macromolecules and we ignore them here. Finally, many of these indicators may be expressed as fractions, that is, as numbers between one and zero. They are also often expressed in the percentage form, that is, as numbers between one hundred and zero. While we confine our use to the former here, we make no statement of preference.

The remainder of this chapter consists primarily of individual sections that reflect the various steps from the crystallographic data-collection experiment to the refinement of the final model. Relevant quality indicators and definitions are given for each of the steps and the most commonly used are collected in Table 2.2.11.1[link].

2.2.2. Quality indicators for diffraction data

| top | pdf |

Once useful crystalline samples have been obtained, the collection of X-ray diffraction data is the next (and the last) experimental step in a structure determination. Although the greatest care may be taken to collect data of as high quality as possible, there remain circumstances and influences that limit the quality of the data. Over time, many indicators have been defined to describe various aspects of diffraction data quality. The most important ones are discussed here.

Nominal resolution, dmin. The resolution of a diffraction data set describes the extent of measurable data and is calculated by Bragg's law [equation (2.2.2.1)[link]] based on the maximum Bragg angle 2θ included in the data set for a given data-collection wavelength λ. As discussed above, the nominal resolution is a limit set by the experimenters and is well known to be prone to subjective judgment. A number of suggestions have been made to reduce the subjectivity associated with this limit. One defines the limit as the resolution within which the intensities of a fraction of the unique reflections, for example 70%, are above a threshold, for example zero or three times their standard uncertainties. Another suggested limiting criterion, discussed further below, recommends that the nominal resolution be set as the midpoint of the resolution range of the shell at which the mean signal-to-noise ratio falls below 2.[d_{\min}=\lambda/2\sin\theta.\eqno(2.2.2.1)]

True resolution, dtrue. The true resolution of a diffraction data set is defined as the minimum distance between two objects in a crystal that permits their images in the resultant electron-density map to be resolved. Often, dtrue is approximated as dmin.

To illustrate this crucial distance, represent two equivalent atoms by equal overlapping Gaussians. One might then consider that the distance between them that just permits distinguishing them as individual atoms might be the distance at which the electron-density value at the midpoint between the atoms drops to a value just below that at the positions of the atoms. For a normal distribution, this distance is 2σ, twice the standard deviation of the distribution.

Another perspective is provided by the realization that, when a Fourier synthesis is terminated at a resolution cutoff dmin, successive spheres of negative and positive density of decreasing amplitude surround the maxima of positive density at atomic positions in that synthesis. It has been shown that the distance from the centre of such a maximum to the first zero is 0.715dmin (James, 1948[link]), which is a useful estimate of the limiting distance between distinguishable features in the electron-density map or dtrue. Similar estimates are used in other areas, notably for defining resolution in astronomy. A more recent re-evaluation suggests that a limit of 0.917dmin is a better value, especially when the effects of form factors and atomic displacement parameters are considered (Stenkamp & Jensen, 1984[link]). Add to that the effects of errors in experimental amplitudes and derived phases, and the approximation of dtrue as dmin seems quite reasonable.

Optical resolution, dopt. The optical resolution dopt is calculated from the standard deviation of a Gaussian fitted to the origin peak of the Patterson function (σPatt) of the diffraction data set and the standard deviation of another Gaussian fitted to the origin peak of the spherical interference function (σsph). This definition is based on Vaguine et al. (1999[link]) and is implemented in the program SFCHECK. The optical resolution is intended to account for uncertainties in the data, atomic displacement factors, effects of crystal quality and series-termination effects by means of a propagation-of-error-like approach (Blundell & Johnson, 1976[link]; Vaguine et al., 1999[link]). It has been suggested that dopt is a better approximation of dtrue than dmin (Weiss, 2001[link]).[d_{\rm opt}=[2(\sigma_{\rm Patt}^2+\sigma_{\rm sph}^2)]^{1/2}.\eqno(2.2.2.2)]

Completeness, C. The completeness C of a diffraction data set is defined as the fraction of the unique reflections in a given space group to a given nominal resolution dmin that have been measured at least once during data collection. C may be given assuming that Friedel symmetry is either applied or not. In the latter case, C is also referred to as anomalous completeness. In the program SCALA (Evans, 2006[link]), the anomalous completeness is defined based on acentric reflections only.

Effective resolution, deff. Since any missing reflection of a data set leads to a deterioration of the model parameters (Hirshfeld & Rabinovich, 1973[link]; Arnberg et al., 1979[link]), an effective resolution may be defined based on the nominal resolution dmin and the cube root of the completeness C of the data set.[d_{\rm eff}=d_{\rm min}C^{-1/3}.\eqno(2.2.2.3)]

Multiplicity (or redundancy), N. The multiplicity or redundancy N of a diffraction data set defines on average how many times a reflection hkl has been observed during the data-collection experiment including symmetry mates and replicate measurements. N may be given assuming that Friedel symmetry is either applied or not.

Merging R factor, Rmerge. The merging R factor of a diffraction data set describes the spread of the individual intensity measurements Ii of a reflection hkl around the mean intensity <I(hkl)> of this reflection. Sometimes Rmerge is also referred to as R factor (observed) (in the program XDS; Kabsch, 1988[link], 1993[link], 2010[link]; Chapter 11.6[link] ), as Rsym or as Rlinear (in the program SCALEPACK; Otwinowski & Minor, 1997[link]). In fractional form, this is[R_{\rm merge}=\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}|I_i(hkl)-\langle I(hkl)\rangle|/\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}I_i(hkl),\eqno(2.2.2.4)]where <I(hkl)> is the mean of the several individual measurements Ii(hkl) of the intensity of reflection hkl. The sums [\textstyle \sum _{hkl}] and [\textstyle \sum _{i}] run over all observed unique reflections hkl and over all individual observations i of a given reflection hkl. It should be noted that alternative definitions of Rmerge exist. In one, Ii(hkl) in the denominator is replaced by <I(hkl)>, thereby producing an expression that is formally equivalent to the one above. In another, Ii(hkl) in the denominator is replaced by |Ii(hkl)| with the suggestion that the denominator is thereby prevented from becoming negative or zero, even in the case of many negative-intensity observations. One should note, however, the counterintuitive side effect: artificial damping of Rmerge values, that is, reducing expected higher Rmerge values of data sets with more weak reflections.

The usefulness of Rmerge as a quality indicator for diffraction data is limited because it is dependent on the multiplicity of a data set (Diederichs & Karplus, 1997a[link],b[link]; Weiss & Hilgenfeld, 1997[link]; Weiss, 2001[link]). The higher the multiplicity of a data set, the higher its Rmerge will be, although, based on statistics, the better determined the averaged intensity values should be. Despite these shortcomings, Rmerge is still widely used today.

Redundancy-independent merging R factor, Rr.i.m. or Rmeas. The redundancy-independent merging R factor Rr.i.m. or Rmeas describes the precision of the individual intensity measurements Ii, independent of how often a given reflection has been measured. Because of its independence of the redundancy (hence its name), it has been proposed that Rr.i.m. or Rmeas should be used as a substitute for the conventional Rmerge (Diederichs & Karplus, 1997a[link],b[link]; Weiss & Hilgenfeld, 1997[link]; Weiss, 2001[link]). In fractional form, this is[\eqalignno{R_{\rm r.i.m.}&=\textstyle \sum \limits_{hkl}\left\{N(hkl)/[N(hkl)-1]\right\}^{1/2}&\cr&\quad\times\textstyle \sum \limits_{i}|I_i(hkl)-\langle I(hkl)\rangle|/\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}I_i(hkl),&\cr&&(2.2.2.5)}]where <I(hkl)> is the mean of the N(hkl) individual measurements Ii(hkl) of the intensity of reflection hkl. As for Rmerge, the sums [\textstyle \sum _{hkl}] and [\textstyle \sum _i] run over all observed unique reflections hkl and over all individual observations i of a given reflection hkl.

Precision-indicating merging R factor, Rp.i.m.. The precision-indicating merging R factor Rp.i.m. describes the precision of the averaged intensity measurements <I(hkl)> (Weiss, 2001[link]). In fractional form, this is[\eqalignno{R_{\rm p.i.m.}&=\textstyle \sum \limits_{hkl}\left\{1/[N(hkl)-1]\right\}^{1/2}&\cr&\quad\times\textstyle \sum \limits_{i}|I_i(hkl)-\langle I(hkl)\rangle|/\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}I_i(hkl),&\cr&&(2.2.2.6)}]where <I(hkl)> is the mean of the N(hkl) individual measurements Ii(hkl) of the intensity of reflection hkl. As with Rmerge and Rr.i.m. or Rmeas, the sums [\textstyle \sum _{hkl}] and [\textstyle \sum _i] run over all observed unique reflections hkl and over all individual observations i of a given reflection hkl.

R factor of merged intensities or amplitudes, Rmrgd-I and Rmrgd-F. An alternative precision-indicating merging R factor, called Rmrgd, is defined as the R factor between two or more data sets or between two subsets of a data set created by randomly apportioning the individual intensity measurements between the two subsets (Diederichs & Karplus, 1997a[link],b[link]). Rmrgd can be calculated for intensities (Rmrgd-I) or structure-factor amplitudes (Rmrgd-F). The latter quantity was suggested to present a lower limit for the crystallographic R factor of a model against the observed data (Diederichs & Karplus, 1997a[link],b[link]). In fractional form[R_{{\rm mrgd\hbox{-}}I}=2\textstyle \sum \limits_{hkl}|\langle I_1(hkl)\rangle - \langle I_2(hkl)\rangle|/\textstyle \sum \limits_{hkl}\langle I_1(hkl)\rangle+\langle I_2(hkl)\rangle,\eqno(2.2.2.7)]where <I1(hkl)> and <I2(hkl)> are the mean intensity values for the individual observations of the reflections hkl, which have been partitioned into the two subsets 1 and 2. The sums [\textstyle \sum _{hkl}] run over all observed unique reflections. Rmrgd-I is related to Rp.i.m. by a constant factor (Rmrgd-I = 21/2Rp.i.m.).

Rmrgd-F is defined analogously to Rmrgd-I (Diederichs & Karplus, 1997a[link],b[link]). In the equation, only the intensities are replaced by structure-factor amplitudes.[R_{{\rm mrgd}\hbox{-}F}=2\textstyle \sum \limits_{hkl}|\langle F_1(hkl)\rangle - \langle F_2(hkl)\rangle|/\textstyle \sum \limits_{hkl}\langle F_1(hkl)\rangle+\langle F_2(hkl)\rangle.\eqno(2.2.2.8)]In order to cope with negative-intensity observations, pseudo-amplitudes had to be introduced just for the purpose of calculating Rmrgd-F (F = I1/2 if I ≥ 0 and F = −|I|1/2 if I < 0).

Note. The approach of comparing randomly partitioned subsets of a given data set is used for a variety of quality indicators. While there is potential for variation in these indicators from one partitioning of the data set to another, an average of several random partitionings should be expected to give a useful estimate. There is also potential for subjectivity, but the principal value of these indicators is to assist the experimenter in proper analysis and they are less often applied to compare experiments from different laboratories and are seldom published.

Pooled coefficient of variation, PCV. The pooled coefficient of variation PCV is the ratio of the sum of the standard deviations to the sum of the reflection intensities (Diederichs & Karplus, 1997a[link],b[link]). PCV is related to Rmeas or Rr.i.m. by the factor (π/2)1/2. In fractional form, this is[{\rm PCV}={{\textstyle \sum_{hkl}\big(\{1/[N(hkl)-1]\}^{1/2}\textstyle \sum_{i}|I_i(hkl)-\langle I(hkl)\rangle|^2\big)^{1/2}}\over{\textstyle \sum_{hkl}\textstyle \sum_{i}I_i(hkl)}},\eqno(2.2.2.9)]where <I(hkl)> is the mean of the N(hkl) individual measurements Ii(hkl) of the intensity of reflection hkl.

Mean signal-to-noise ratio, <I/σ(I)>. The signal-to-noise ratio Ii/σ(Ii) of an individual intensity measurement describes the statistical significance of a measured intensity. As a measure of the overall quality of a data set, the mean signal-to-noise ratio for all reflections is useful as an indication of the robustness of the data, that is, the average intensity as a multiple of the standard uncertainty. In addition, as mentioned above, the mean signal-to-noise ratio for all reflections within the outer resolution shell can be used to define the nominal resolution of a data set. For the data set as a whole or for a resolution shell of that data set, the mean signal-to-noise ratio, <I/σ(I)>, is the sum of the signal-to-noise ratios of all individual reflections hkl within resolution limits divided by the number of individual reflections hkl within those resolution limits.

In principle there are two ways to define a mean signal-to-noise ratio of a data set (or a given resolution shell). The two ways yield different quantities, although, unfortunately, they are both called the mean signal-to-noise ratio. They differ in the manner in which mean signal-to-noise ratios of individual reflections hkl are calculated.

  • (i) <I(hkl)>/σ[I(hkl)]. The mean signal-to-noise ratio of individual reflections hkl may be calculated as the ratio of the mean intensity <I(hkl)> and the r.m.s. scatter of Ii(hkl) about that mean. This is a measure of the average significance of individual observations, but it does not take into account the multiplicity or redundancy of the measurements.[\eqalignno{&\langle I(hkl)\rangle /\sigma[I(hkl)]&\cr&\quad=\langle I(hkl)\rangle/[(1/N)\textstyle \sum \limits_{i}|I_i(hkl)-\langle I(hkl)\rangle|^2]^{1/2}.&\cr&&(2.2.2.10)}]In the program SCALA (Evans, 2006[link]), this value is reported as I/sigma.

  • (ii) <I(hkl)>/σ<I(hkl)>. <I(hkl)> is the average over all observations of the reflection hkl, and is sometimes weighted. σ<I(hkl)> is the propagation-of-error combination of standard uncertainties assigned at data processing for the individual measurements Ii(hkl), that is, a modification of equation (2.2.2.10)[link] in which the term |Ii(hkl) − <I(hkl)>| in the denominator is replaced by σi(hkl), the experimental standard uncertainty for the measurement Ii(hkl).[\eqalignno{&\langle I(hkl)\rangle /\sigma\langle I(hkl)\rangle=\langle I(hkl)\rangle/[(1/N)\textstyle \sum \limits_{i}\sigma_i(hkl)^2]^{1/2}.&\cr&&(2.2.2.11)}]An error model1 is often applied in the denominator here to scale to the r.m.s. scatter in (i) above. In the program SCALA (Evans, 2006[link]), this value is reported as Mn(I)/sd.

Both methods of defining the mean signal-to-noise ratio for the reflection hkl have merit. As suggested for individual intensities in Section 2.2.1[link], perhaps the best approach would be to calculate weighted averages and weighted standard uncertainties of the I(hkl) where weights are the experimental standard uncertainties σi(hkl) for individual measurements Ii(hkl).

Highest possible signal-to-noise ratio, I/σ(I)asymptotic. A relatively recent addition to the collection of diffraction-data quality indicators is the highest possible signal-to-noise ratio of a data set I/σ(I)asymptotic or ISa (Diederichs, 2010[link]). ISa is calculated from the parameters of the error model used for inflating the standard deviations of the reflections with an intensity-dependent term.1 Since ISa is practically independent of counting statistics, it was suggested to be a good measure of instrument errors manifesting themselves in the data set, provided the crystal is close to ideal and radiation damage is negligible. Data sets with ISa values of 25 or greater are considered to be very good and amenable to straightforward structure determination, while data sets exhibiting ISa values of 15 or less are considered marginal at best. The calculation of ISa is implemented in XDS versions of December 2009 or later (Kabsch, 2010[link]).

Anomalous R factor, Ranom. The anomalous R factor Ranom describes the sum of the differences in intensities of Friedel-related reflections (hkl) and [(\bar h\bar k\bar l)] relative to the sum of their mean intensities. In fractional form, this is[R_{\rm anom}=\textstyle \sum \limits_{hkl}|I(hkl)-I(\bar h \bar k \bar l)|/\textstyle \sum \limits_{hkl}\langle I(hkl)\rangle,\eqno(2.2.2.12)]where, in this case, <I(hkl)> is the mean intensity of the Friedel mates of the reflections hkl, or [\textstyle{1\over 2}[I(hkl)+I(\bar h\bar k\bar l)]]. Here, the sums [\textstyle \sum _{hkl}] run over all unique reflections with one of the indices, typically h, greater than zero (h > 0) for which both Friedel mates have been observed at least once.

The ratio of Ranom to Rp.i.m. has been proposed as a possible indicator for the strength of the anomalous signal (Panjikar & Tucker, 2002[link]).

Anomalous correlation coefficient, CCanom. The anomalous correlation coefficient CCanom quantifies the linear dependence of observed anomalous differences in two diffraction data sets. These can be data sets, for example, collected at two different wavelengths in a MAD experiment. In cases where only one data set is available, two randomly partitioned half data sets can be created for comparison.

Note. The correlation coefficient referred to here and elsewhere in this chapter is invariably the Pearson linear correlation coefficient (Rodgers & Nicewander, 1988[link]):[{\rm CC}=\textstyle \sum \left(x -\langle x\rangle\right)\left(y-\langle y\rangle\right)\big/\left[\textstyle \sum \left(x-\langle x\rangle\right)^2\textstyle \sum \left(y-\langle y \rangle\right)^2\right]^{1/2},\eqno(2.2.2.13)]with x and y being, in this case, the anomalous differences [[I(hkl)-I(\bar h\bar k\bar l)]] or [[F(hkl)-F(\bar h\bar k\bar l)]] in the two data sets, <x> and <y> are their averages, and the summations are over all reflections hkl for which observations exist in both data sets across the entire resolution range or within a particular resolution shell. CCanom is a reliable indicator of the strength of the anomalous signal. Values above 0.30 are considered good.

R.m.s. correlation ratio. This is another statistic based on randomly partitioned data sets, which is calculated by the program SCALA (Evans, 2006[link]; Collaborative Computational Project, Number 4, 1994[link]). It is an analysis of the scatterplot of [[I_1(hkl)-I_1(\bar h\bar k\bar l)]] versus [[I_2(hkl)-I_2(\bar h\bar k\bar l)]], where the subscripts 1 and 2 identify the two half data sets. The analysis assumes that the correlation is ideally 1.0. The r.m.s. correlation ratio is defined as the ratio of the r.m.s. widths of the scatterplot distribution along the diagonal and perpendicular to the diagonal. This statistic seems to be more robust than CCanom to the presence of outliers. It cannot, however, be applied to analysing the correlations between different data sets.

Mean anomalous signal-to-noise ratio, <d′′/σ(d′′)>. The anomalous signal-to-noise ratio of an individual reflection measurement [d^{\prime\prime}\!(hkl)/\sigma[d^{\prime\prime}\!(hkl)]] is defined as the ratio of the observed anomalous intensity difference [d^{\prime\prime}=|I(hkl)-I(\bar h\bar k\bar l)|] and the corresponding estimated standard uncertainty [\sigma(d^{\prime\prime})] in the measurement of this anomalous difference. The average of the anomalous signal-to-noise ratios for all reflections within a certain resolution range is used as an indicator of utility for phasing. A value of (2/π)1/2 ≃ 0.8 for mean [d^{\prime\prime}\!/\sigma(d^{\prime\prime})] of a resolution shell, for example, is taken to indicate that no anomalous signal is present (G. Sheldrick & G. Bunkoczi, personal communication).

Decay R factor, Rd. The decay R factor Rd is defined as a pairwise R factor based on the intensities of symmetry-related reflections occurring on different diffraction images (Diederichs, 2006[link]). An increase in Rd as a function of difference in image-collection times is a good indicator of radiation damage occurring during data collection. In fractional form, this is[R_d=2\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{m-n}|I_m(hkl)-I_n(hkl)|/\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}[I_m(hkl)+I_n(hkl)],\eqno(2.2.2.14)]where Im(hkl) and In(hkl) are the intensities of the reflection hkl occurring on images m and n. The only program in which this is currently implemented is XDSSTAT.

Wilson-plot B factor, BWilson. A Wilson plot (Wilson, 1949[link]) is a plot for a contiguous series of resolution shells of the logarithm of the mean intensity in a given resolution shell divided by the sum of the squared atomic form factors for all atoms in the unit cell evaluated at the mean of the resolution limits of the shell. From a least-squares fit of a straight line to the linear part of the Wilson plot, the B factor BWilson can be derived. Typically, data of lower than 4.5 Å resolution are excluded from the fit. The more meaningful determinations of BWilson come from Wilson plots that are linear all the way to the nominal resolution dmin and minimize the occurrence of spikes due to ice rings.[\ln(\langle I_{\rm obs}\rangle/\textstyle \sum \limits_{i}f_i^2)=-\ln K_{\rm Wilson}-2B_{\rm Wilson}/d^2,\eqno(2.2.2.15)]where <Iobs(hkl)> is the mean over the intensities of all observed reflections hkl in a given resolution shell. The sum [\textstyle \sum _i] runs over all atoms in the structure. The parameter d is the midpoint of the resolution shell over which Iobs has been averaged. KWilson is an absolute scale factor.

2.2.3. Comparing different diffraction data sets

| top | pdf |

In MX, there are many instances when two or more diffraction data sets have to be compared to each other. An important decision which has to be made is, for instance, whether data sets collected from different crystals are isomorphous enough so that they can be merged together. Another task is the comparison of a native data set and a heavy-atom derivative. The question here is how much of the observed difference is due to non-isomorphism and how much is due to isomorphous replacement.

Scaling R factor, Rscale. The scaling R factor Rscale between two data sets is defined as the difference in the structure-factor amplitudes of one data set relative to the structure-factor amplitudes of the other data set after the two data sets have been scaled to each other.[R_{\rm scale}=\textstyle \sum \limits_{hkl}|F_1-F_2|/\textstyle \sum \limits_{hkl}|F_1|.\eqno(2.2.3.1)]However, the above formula is asymmetric with respect to data sets 1 and 2. An alternative symmetric formula is[R_{\rm scale}=2\textstyle \sum \limits_{hkl}|F_1-F_2|/\textstyle \sum \limits_{hkl}(F_1+F_2).\eqno(2.2.3.2)]Rscale is also often given based on intensities rather than structure-factor amplitudes.

Gradient from a normal probability analysis. For each reflection hkl in two data sets that have been scaled to each other, the quantity Δ(real) is computed and compared with its expectation value Δ(expected) assuming a normal distribution of errors (Howell & Smith, 1992[link]). If Δ(real) is plotted against Δ(expected), random differences between the two data sets give rise to a slope of 1.0. Significant differences result in slopes significantly larger than 1.0. Such an analysis is implemented in the program SCALEIT (Collaborative Computational Project, Number 4, 1994[link]).[\Delta({\rm real})=(F_1-F_2)/[\sigma(F_1)^2+\sigma(F_2)^2]^{1/2}.\eqno(2.2.3.3)]

2.2.4. Quality indicators for substructure determination

| top | pdf |

The first step of a crystal structure determination after processing and analysing the diffraction data is to use the observed isomorphous and/or anomalous differences in order to determine the isomorphous or anomalous substructure. Since such sub­structures typically consist of rather few atoms, which are relatively far apart, direct-methods approaches have been adopted from small-molecule crystallography and have largely superseded pure Patterson-based methods. The most widely used computer programs for substructure determination are SHELXD (Schneider & Sheldrick, 2002[link]; Sheldrick, 2008[link]), SnB (Weeks et al., 1993[link], 1994[link]) and HySS (Grosse-Kunstleve & Adams, 2003[link]).

Correlation coefficient CC(all). CC(all) measures the correlation of the E values (analogous to normalized structure factors; Hauptman & Karle, 1953[link]) derived from the observed isomorphous or anomalous differences and those calculated from the substructure model. CC(all) is another application of the Pearson linear correlation coefficient [see equation (2.2.2.13)[link]] and is calculated for all reflections. In the equation, x and y are the E values derived from the observed and calculated differences. CC(all) is typically used to assess the successful determination of the isomorphous or anomalous substructure in a macromolecule. A value of CC(all) ≥ 0.3 often indicates that the substructure has been correctly identified (Schneider & Sheldrick, 2002[link]).

Correlation coefficient CC(weak). Like CC(all), CC(weak) is the Pearson linear correlation coefficient [see equation (2.2.2.13)[link]] between the E values (analogous to normalized structure factors as above) derived from the observed isomorphous or anomalous differences and those calculated from the substructure model. In contrast to CC(all), however, CC(weak) is calculated for the weak reflections only. As above, x and y are the E values derived from the observed and calculated differences. The E-value cutoff for defining a reflection as weak can be chosen by the user, but a typical threshold value is 1.5, although lower values may be required for low-resolution data (Schneider & Sheldrick, 2002[link]). A value of CC(weak) ≥ 0.15 often indicates that the substructure has been correctly identified (Sheldrick, 2010[link]).

The minimal function, R(ϕ). The minimal function R(ϕ) is a measure of the mean-square difference between the values of the triplets calculated using a particular set of phases and the expected values of the same triplets as given by the ratio of modified Bessel functions. The minimal function is expected to have a constrained global minimum when the phases are equal to their correct values for some choice of origin and enantiomorph (the minimal principle).[R(\varphi)=\textstyle \sum \limits_{H,K}A_{HK}\left\{\cos\varphi_{HK}-[I_1(A_{HK})/I_0(A_{HK})]\right\}^2/\textstyle \sum \limits_{H,K}A_{HK},\eqno(2.2.4.1)]where [A_{HK}=(2/N^{1/2})|E_HE_KE_{H+K}|] with N being the number of atoms in the corresponding primitive unit cell. The minimal function is the function minimized in the program Shake&Bake, abbreviated as SnB (Weeks et al., 1993[link], 1994[link]).

2.2.5. Quality indicators for phase determination

| top | pdf |

Once the isomorphous or anomalous substructure has been successfully determined, it can be used as reference point for the calculation of phases. The quality of the resulting phases is dependent on the strength of the isomorphous or anomalous signal and the completeness and correctness of the isomorphous or anomalous substructure.

Cullis R factor, RCullis. The Cullis R factor (Cullis et al., 1961[link]) for phase determination by isomorphous replacement is defined as the ratio between the lack-of-closure error [epsilon]P) [equation (2.2.5.1a)[link] below] and the isomorphous difference [|F_{\rm PH}-F_{\rm P}|]. It is the most useful signal for a usable heavy-atom derivative. Values < 0.6 for centrosymmetric data are excellent, while values < 0.9 are still usable.[\eqalignno{\varepsilon(\varphi_{\rm P})&=\big|F_{\rm PH}-|F_{\rm P}+F_{\rm H}|\big|,&(2.2.5.1a)\cr R_{\rm Cullis}&=\textstyle \sum \limits_{hkl}\big|F_{\rm PH}-|F_{\rm P}+F_{\rm H}|\big|/\textstyle \sum \limits_{hkl}|F_{\rm PH}-F_{\rm P}|.&(2.2.5.1b)}]

Anomalous Cullis R factor, RCullis,ano. The Cullis R factor for phase determination by anomalous dispersion is defined as the ratio between the lack-of-closure error and the observed anomalous difference [|F_{\rm PH}(hkl)-F_{\rm PH}(\bar h \bar k\bar l)|]. The lack-of-closure error in the anomalous-dispersion case is the difference between the observed anomalous difference and the calculated anomalous difference 2FH sin αP, where αP is the protein phase. A value of RCullis,ano < 1.0 suggests that a contribution to the phasing from the anomalous data is likely (MLPHARE program documentation; Collaborative Computational Project, No. 4, 1994[link]).[R_{\rm Cullis, ano}={{\textstyle \sum _{hkl}\big||F_{\rm PH}(hkl)-F_{\rm PH}(\bar h \bar k \bar l)|-|2F_{\rm H}\sin\alpha_{\rm P}|\big|}\over{\textstyle \sum_{hkl}|F_{\rm PH}(hkl)-F_{\rm PH}(\bar h\bar k\bar l)|}}.\eqno(2.2.5.2)]

Phasing power, PPiso. The isomorphous phasing power PPiso for phase determination by isomorphous replacement is defined for a particular pair of native and heavy-atom-derivative data sets as the ratio of [|F_{\rm H}|] and [epsilon]P), where [|F_{\rm H}|] is the calculated amplitude of the heavy-atom structure factor and [epsilon]P) is the lack-of-closure error [equation (2.2.5.1a)[link]]. FP + FH is a vector sum of the calculated structure factor FH and the structure factor FP.[{\rm PP}_{\rm iso}=\textstyle \sum \limits_{hkl}|F_{\rm H}|/\textstyle \sum \limits_{hkl}\big|F_{\rm PH}-|F_{\rm P}+F_{\rm H}|\big|.\eqno(2.2.5.3)]There is another, slightly different, definition of PPiso, which is implemented in the program SOLVE. Here, PPiso is defined as the ratio of the r.m.s. of the [|F_{\rm H}|] values and the r.m.s. of the lack-of-closure errors [epsilon]P). For each reflection hkl, a weighted average of [epsilon]P) is calculated by integrating [epsilon]P) over the whole phase circle.[{\rm PP}_{\rm iso}=(\textstyle \sum \limits_{hkl}|F_{\rm H}|^2)^{1/2}/(\textstyle \sum \limits_{hkl}\langle\big|F_{\rm PH}-|F_{\rm P}+F_{\rm H}|\big|\rangle^2)^{1/2}.\eqno(2.2.5.4)]Note. Owing to the cancelling out of the factor N1/2 in the numerator and denominator, the latter PPiso formula does not appear as a ratio of r.m.s. values at first sight.

Anomalous phasing power, PPano. The anomalous phasing power PPano for phase determination by anomalous-dispersion methods is defined as the ratio of the sum of calculated anomalous differences [d^{\prime\prime}_{\rm calc}] and the sum of estimated standard uncertainties [\sigma(d^{\prime\prime}_{\rm obs})] in the measurement of these anomalous differences:[{\rm PP}_{\rm ano}=\textstyle \sum \limits_{hkl}d^{\prime\prime}_{\rm calc}/\textstyle \sum \limits_{hkl}\sigma(d^{\prime\prime}_{\rm obs}).\eqno(2.2.5.5)]As with PPiso (see above), the program SOLVE uses a slightly different definition of PPano. Here, the anomalous phasing power is defined as the ratio of the r.m.s. of the [d^{\prime\prime}_{\rm calc}] values and the r.m.s. of [\sigma(d^{\prime\prime}_{\rm obs})]. For this, a weighted average of [d^{\prime\prime}_{\rm calc}] is computed by integrating over the whole phase circle for each reflection.[{\rm PP}_{\rm ano}=(\textstyle \sum \limits_{hkl}\langle d^{\prime\prime}_{\rm calc}\rangle^2)^{1/2}/[\textstyle \sum \limits_{hkl}\sigma(d^{\prime\prime}_{\rm obs})^2]^{1/2}.\eqno(2.2.5.6)]

Note. As above in the PPiso formula, the factors N1/2 in the numerator and denominator cancel out.

Figure of merit (f.o.m.), m. The figure of merit m is a term used in a number of contexts in X-ray crystallography. In its most common use, it is defined as the weight applied to an individual structure-factor amplitude that, in conjunction with its best phase, gives rise, in a Fourier synthesis, to the electron-density map with the minimum level of noise (Blow & Crick, 1959[link]). Typically, m is given as an average value over all reflections in the data set or in a given resolution shell.[m=\textstyle \int P(\alpha)\exp(i\alpha)\,{\rm d}\alpha/\textstyle \int P(\alpha)\,{\rm d}\alpha=\langle\cos(\Delta\alpha)\rangle,\eqno(2.2.5.7)]where P(α) is the probability of the phase α, initial or refined, being the best phase and Δα = αbest − α is the error in the phase angle at α. The integration is from 0 to 2π and values for m range from 0 to 1.

2.2.6. Quality indicators for density modification and phase improvement

| top | pdf |

After determination of initial phases, a first electron-density map can be computed. It is expected that this map will contain significant errors and improbable features. Additional information, such as the flatness of the electron density in the solvent region or the similarity of electron-density regions of two or more identical molecules in the asymmetric unit, can be exploited to modify the electron density and hence improve the phases.

Figure of merit (f.o.m.), m. The figure of merit m [see equation (2.2.5.7)[link]] is also used to judge the degree of improvement in the phase values. Again, in its most common use, it is defined as the weight applied to an individual structure-factor amplitude that, in conjunction with its best phase, gives rise, in a Fourier synthesis, to the electron-density map with the minimum level of noise. Typically, m is given as an average value over all reflections in the data set or in a given resolution shell.

Density-modification (DM) free R factor. The DM free R factor is defined in the same way as the refinement free R factor Rfree [see equation (2.2.8.1)[link] and the following paragraph describing Rfree]. It describes the disagreement between the observed structure-factor amplitudes of a certain set of reflections and the calculated amplitudes after density modification. It is a weak indicator used in the program DM (Cowtan, 1999[link]) and is helpful mainly in identifying the correct enantiomorph.

Density-modification (DM) real-space free residuals. The DM real-space free residuals are two numbers (real-space free residual for the solvent area and real-space free residual for the protein area) which are calculated by omitting two small spheres of protein and solvent from the density-modification process. The real-space free residual for the solvent area indicates how flat the solvent is in a non-flattened region of solvent and the real-space free residual for the protein area indicates how well the electron density in a non-histogram-matched region of protein obeys the expected histogram. The two residuals can also be combined into the combined real-space free residual by weighted summation, where the weight is defined by the solvent content. The DM real-space free residuals have some value in determining when to stop a density-modification calculation, once no more progress is being made, but limited value otherwise.

Contrast, c. The contrast c between the r.m.s. electron density in the solvent region and the r.m.s. electron density in the macromolecular region can serve as an indication of the presence of a clearly-defined solvent boundary in the electron-density map. It is defined as the standard deviation of the local r.m.s. electron-density values over the entire asymmetric unit (Terwilliger & Berendzen, 1999[link]; Sheldrick, 2002[link]; Terwilliger et al., 2009[link]). The electron-density values are first squared (optionally after truncating very high and very low density values) and then smoothed using a moving local sphere typically with a radius of 6 Å. Local r.m.s. values are then calculated. The contrast c is now defined as the standard deviation σ of the local r.m.s. density values multiplied by a factor to normalize for the fraction of solvent sc in the crystal:[c=[(1-sc)/sc]^{1/2}\sigma.\eqno(2.2.6.1)]

Skewness of electron density, S. A high value for the skewness S of the electron density in an electron-density map indicates the presence of local electron-density maxima with high positive density values. The skewness is defined as the third moment of the electron density:[S=\langle \rho^3\rangle/\langle\rho^2\rangle^{3/2}.\eqno(2.2.6.2)]In order to compute the mean values of ρ3 and ρ2, all density grid points in the asymmetric unit of the electron-density map are taken into account (Terwilliger et al., 2009[link]).

Overlap of NCS-related density, ONCS. The presence of correlated electron density at noncrystallographic symmetry (NCS)-related regions in a map can be used as an indicator for the quality of the electron-density map (Cowtan & Main, 1998[link]; Vellieux et al., 1995[link]; Terwilliger et al., 2009[link]). The overlap (ONCS) between density values at NCS-related locations is also often used to evaluate the presence of local symmetry or non-space-group symmetry:[O_{\rm NCS}=\langle\rho_i\rho_j\rangle.\eqno(2.2.6.3)]ρi and ρj are the normalized electron-density values in the NCS-related regions in the asymmetric unit. The average is calculated over the whole region where NCS is present. This region may be defined as the region where overlap values are 0.3 or greater, or by a mask. If there are more than two NCS groups, the average is taken over all NCS pairs.

R factor, RDENMOD, and phase correlation, mDENMOD, from statistical density modification. The amplitudes and phases of structure factors calculated using statistical density modification can be compared with the observed amplitudes and experimental phases (Cowtan & Main, 1996[link]; Terwilliger, 2001[link]; Terwilliger et al., 2009[link]). These comparisons yield an R value (RDENMOD) for the amplitudes and a mean cosine of the phase difference (mDENMOD) for the phases.[R_{\rm DENMOD}=\textstyle \sum \limits_{hkl}\big||F_{\rm obs}|-|F_{\rm DENMOD}|\big|/\textstyle \sum \limits_{hkl}|F_{\rm obs}|,\eqno(2.2.6.4)][m_{\rm DENMOD}=(1/N)\textstyle \sum \limits_{hkl}\cos(\alpha_{\rm obs}-\alpha_{\rm DENMOD}).\eqno(2.2.6.5)]

Correlation coefficient CC of chain trace against native data. The quality of density modification in SHELXE (Sheldrick, 2002[link], 2010[link]) can be assessed by computing a Pearson linear correlation coefficient [see equation (2.2.2.13)[link]] of the calculated structure-factor amplitudes for a chain trace against the native structure-factor amplitudes. If the poly-Ala trace yields a CC value higher than 0.25 and if the mean chain length of the trace is >10, the solution is almost always correct. This criterion is currently implemented in the program ARCIMBOLDO (Rodriguez et al., 2009[link]).

2.2.7. Quality indicators for molecular replacement

| top | pdf |

In a case where a known structure is assumed to be similar to that of a target molecule (structural similarity is typically inferred by the degree of sequence similarity), the known structure, also termed the search model, can be used to determine the structure of the target molecule. The approach, termed molecular replacement, was first described in 1962 (Rossmann & Blow, 1962[link]). Nowadays, about two thirds of all newly determined structures are determined by molecular replacement (Long et al., 2008[link]).

Rotation function, RF. The rotation function RF is a measure of the overlap or the agreement of the stationary Patterson function P2 calculated from the observed data and the rotated Patterson function P1 from the search model.[{\rm RF}=\textstyle \int \limits_{r}P2\, \underline{R}\,P1\,{\rm d}r.\eqno(2.2.7.1)]In the equation for RF, R is the rotation operator. The integration is performed between a minimum value and a maximum value for the radius r. These values are chosen according to the size of the search model, with the aim of including as many intramolecular Patterson peaks (self vectors) as possible and to exclude as many intermolecular Patterson peaks (cross vectors) as possible. The ratio of the height of a peak in the RF to the background level is used as an indicator of how likely it is that this peak describes the orientation of a molecule in the target structure.

Translation function, TF. There are numerous ways of defining a translation function TF, making it impractical to discuss quality indicators here. For a thorough treatment of translation-function applications, the reader is referred to Chapter 2.3[link] of International Tables for Crystallography Volume B and Chapter 13.3[link] of the present volume.

Log-likelihood gain, LLG. In likelihood-based molecular replacement (McCoy et al., 2007[link]), potential molecular-replacement solutions are evaluated using likelihood, which is defined as the probability P that the observed diffraction data would have been measured if the orientation (and, usually, position) of the model were correct (Read, 2001[link]). The score is reported in terms of the log-likelihood gain (LLG), which is defined as the logarithm of the likelihood score for the model p(Fobs; model) minus the logarithm of the likelihood score for a random-atom Wilson distribution pWilson(Fobs). The LLG measures how much better the data can be predicted from the molecular-replacement model than from a collection of random atoms.[{\rm LLG}=\textstyle \sum \limits_{hkl}\ln[p(F_{\rm obs}\semi{\rm model})]-\textstyle \sum \limits_{hkl}\ln[p_{\rm Wilson}(F_{\rm obs})].\eqno(2.2.7.2)]

LLG-Z score. It is important to note that the LLG depends on the quality of the model and the number of reflections, so the absolute values cannot be compared between different molecular-replacement applications. Instead, the quality of a molecular-replacement solution can be judged by the LLG-Z score, which is defined as the number of standard deviations a score is above the mean score in a particular rotation or translation search.[\hbox{LLG-Z}={\rm LLG} -\langle{\rm LLG}\rangle/[({\rm LLG} -\langle{\rm LLG}\rangle)^2]^{1/2}.\eqno(2.2.7.3)]The translation function Z score (TFZ) for the last component placed in a molecular-replacement search is often a good indicator of the confidence that can be placed in the solution. If TFZ is greater than 8 and there is no translational pseudo-symmetry, the solution is almost always correct.

Detailed descriptions of the background and proper application of molecular-replacement approaches are presented in Chapter 2.3[link] of International Tables for Crystallography Volume B and Chapters 13.2[link] and 13.3[link] of the present volume.

2.2.8. Quality indicators for refinement

| top | pdf |

The last step of a structure determination is the refinement of the model against the observed data. Refinement is in principle a mathematical operation that is applied in order to minimize the discrepancy between the observed structure-factor amplitudes |Fobs| and the calculated ones |Fcalc|.

Crystallographic R factor, R. The crystallographic R factor R is defined as the fractional disagreement between the set of observed structure-factor amplitudes and amplitudes calculated from the structural model. Of course, observed and calculated reflection sets need to be on the same scale.[R=\textstyle \sum \limits_{hkl}\big||F_{\rm obs}|-|F_{\rm calc}|\big|/\textstyle \sum \limits_{hkl}|F_{\rm obs}|.\eqno(2.2.8.1)]

Free R factor, Rfree. The free R factor Rfree is defined in the same way as the crystallographic R factor, but it is based on a set of reflections that have been excluded from the refinement (Brünger, 1992[link]). The excluded set of reflections is called the test set, while the set of reflections used for refinement is called the working set. The test set can be chosen randomly or systematically, either in thin resolution shells or to account for the presence of noncrystallographic symmetry, respectively. In order to minimize the impact on the final model, the test set should be as small as possible. Typically, it contains about 5–10% of the reflections, or at least enough reflections to keep the standard deviation of Rfree below 1%, but there is no need to use more than 2000 reflections (Kleywegt & Brünger, 1996[link]; Brünger, 1997[link]). The standard deviation of Rfree has been empirically estimated to be Rfree/N1/2, where N is the number of reflections in the test set (Brünger, 1997[link]). Of course, there may be concerns about the impact of excluding 5–10% of reflections on the final model, but a few final cycles of refinement against the recombined full data set should allay them.

Correlation coefficients CC(Fobs, Fcalc) and CC(Iobs, Icalc). CC(Fobs, Fcalc) and CC(Iobs, Icalc) are Pearson linear correlation coefficients [see equation (2.2.2.13)[link]] between observed and model-based calculated structure-factor amplitudes or intensities, respectively, that find use from time to time. One advantage of the use of a correlation coefficient instead of an R factor is that it avoids the problem of scaling the two sets of numbers relative to each other.

2.2.9. Quality indicators for the refined model

| top | pdf |

In MX, the observable-to-parameter ratio is mostly unfavourable. Therefore, structure refinements are carried out with boundary conditions, constraints and restraints. Constraints reduce the number of parameters which need to be refined, while restraints provide additional information to the refinement procedure that increases the number of observables. A refined model, therefore, has to fulfil not only the criterion that the crystallographic R factor [see equation (2.2.8.1)[link]] is good and that the model fits well to the electron density, but also that it fits well to the restraints used in the refinement procedure.

Real-space residual, RSR. The real-space residual, RSR (Jones et al., 1991[link]), quantifies the discrepancies between the electron-density maps ρ1, calculated directly from a structural model, and ρ2, calculated from experimental data. RSR can take the form of a real-space R factor RSRF and of a real-space correlation coefficient RSCC.[{\rm RSRF}=2\textstyle \sum \limits_{xyz}|\rho_1-\rho_2|/\textstyle \sum \limits_{xyz}(\rho_1+\rho_2).\eqno(2.2.9.1)]The sum [\textstyle \sum_{xyz}] runs over all grid points of the electron-density maps that are close to the model. A big advantage of RSRF is that is can be calculated on a residue-by-residue basis. It therefore gives a local picture of structure quality. It can also be used throughout model building and refinement in order to follow the improvement of the model locally on a per-residue basis.

RSCC is defined as the Pearson linear correlation coefficient [see equation (2.2.2.13)[link]] between ρ1 and ρ2. Everything said about RSRF above applies to RSCC as well.

R.m.s. deviation from ideal of geometric parameter x. The root-mean-square deviation of a set of geometric parameters x from their ideal values is defined as[{\rm r.m.s.d}(x)=\big\{\textstyle \sum \limits_{i}[x_i({\rm ideal})-x_i({\rm observed})]^2/N\big\}^{1/2}.\eqno(2.2.9.2)]The sum runs over all N instances of the geometric parameter occurring in a structure. The geometric parameters x that are typically considered are bond lengths, bond angles, dihedral angles, chiral volumes, planar groups etc. The ideal values for proteins are typically taken from the study of Engh & Huber (1991[link]) and for nucleic acids from Parkinson et al. (1996[link]).

Z score. A measure of the likelihood that an individual geo­metric parameter is correct is given by its Z score. The Z score is defined as the distance of an individual data point of a distribution from the mean of the distribution expressed in standard deviations. In the case described here, the mean values of the distribution are the ideal values taken from Engh & Huber (1991[link]) and Parkinson et al. (1996[link]).[Z(x_i)=[x_i({\rm observed})-x_i({\rm ideal})]/\sigma(x).\eqno(2.2.9.3)]Ideally, the Z score should be 0. A parameter that exhibits a Z score of less than −4 or greater than +4 is highly unlikely and calls for attention.

Root-mean-square Z score, r.m.s.-Z. Although r.m.s.d. values [see equation (2.2.9.2)[link]] are still popular for use in judging the quality of refined macromolecular models, a much more useful statistic is the r.m.s. value of a distribution of Z scores or the r.m.s.-Z score.[\hbox{r.m.s.-Z}(x)=\textstyle \sum \limits_{i}[Z(x_i)^2/N]^{1/2}.\eqno(2.2.9.4)]The sum runs over all N instances of the geometric parameter x occurring in a structure. A very useful property of Z scores is that the r.m.s. values of Z-score distributions should always be 1. Significant deviations from the ideal value indicate potential problems. R.m.s. Z scores are widely used, for instance, in the program WHAT_CHECK (Hooft et al., 1996[link]).

R.m.s.d. (NCS). The root-mean-square deviation from crystallographic symmetry between two molecules related by non­crystallographic symmetry (NCS) can be calculated from a superposition of the two molecules. It is defined as[{\rm r.m.s.d.(NCS)}=\big(\textstyle \sum \limits_{i}d_i^2/N\big)^{1/2}.\eqno(2.2.9.5)]The sum runs over N equivalent atom pairs with di being the distance between the two equivalent atoms after superposition.

2.2.10. Error estimation for the refined model

| top | pdf |

An important quality indicator for a refined model is the coordinate uncertainty. Short of full-matrix inversion, which is the standard procedure in small-molecule crystallography but which is applicable only in exceptional cases for macromolecules, some methods have been devised for estimating of the overall coordinate uncertainty.

Error estimation according to Luzzati. For most macromolecular structure determinations, atomic standard uncertainties are not available. However, Luzzati (1952[link]) devised a method of estimating the average positional error of a structure. Under the assumption that the atomic positional errors follow a normal distribution, the average error can be estimated by comparing a plot of the crystallographic R factor [see equation (2.2.8.1)[link]] versus the reciprocal resolution (or 2 sin θ/λ) with pre-computed theoretical curves for different average errors. A more recent – and probably better – approach is to use the free R factor instead of the crystallographic R factor.

SigmaA- (σA)-type error estimation. A slightly better estimate of the average positional error of a structure can be obtained by plotting the natural logarithm of the parameter σA versus (sin θ/λ)2 (Read, 1986[link]). The slope of a straight line fitted to the plot provides an estimate of the average positional error of the structure. The parameter σA assumes normally distributed positional errors and takes model incompleteness into account as well.

Diffraction-component precision index, DPI. The diffraction-component precision index DPI is an empirical parameter describing the overall coordinate uncertainty of a structure (Cruickshank, 1999a[link],b[link]). For an atom with an isotropic displacement parameter of average value (Bavg), it is defined as[{\rm DPI}=[N_{\rm atom}/(N_{hkl}-N_{\rm para})]^{1/2}Rd_{\rm min}C^{-1/3},\eqno(2.2.10.1)]where Natom is the number of atoms included in the refinement, Nhkl is the number of reflections included in the refinement, Npara is the number of refined parameters, R is the crystallographic R factor, dmin is the nominal resolution of the data included in the refinement and C is the data completeness. The free R factor Rfree is sometimes used instead of the crystallographic R factor R to calculate the DPI. In this case (NhklNpara) is replaced by Nfree, which is the number of reflections used for Rfree calculation.

The Cruickshank formula for DPI has been recast into several other forms, including[{\rm DPI}=\sigma(x,B_{\rm avg})=0.18(1+sc)^{1/2}V_M^{-1/2}R_{\rm free}d_{\rm min}^{5/2}C^{-5/6}\eqno(2.2.10.2)](Blow, 2002[link]), where sc is the solvent fraction (= Nsolv/Natom, where Nsolv is the number of atoms that are solvent) and VM is the Matthews parameter (Matthews, 1968[link]). The utility of this latter formula in guiding the design of the data-collection experiment to achieve a specified target coordinate uncertainty has been demonstrated (Fisher et al., 2008[link]).

2.2.11. The most commonly used quality indicators

| top | pdf |

A summary of the most commonly used quality indicators and their definitions is presented in Table 2.2.11.1[link] for ready reference.

Table 2.2.11.1| top | pdf |
Definitions of the most commonly used quality indicators

IndicatorDetails
Optical resolution,[d_{\rm opt}=[2(\sigma_{\rm Patt}^2+\sigma_{\rm sph}^2)]^{1/2}\eqno(2.2.2.2)] σPatt and σsph are the standard uncertainties of Gaussians fitted to the origin peak of the Patterson function of the diffraction data set and the origin peak of the spherical interference function, respectively
   
Rmerge (merging R factor), [R_{\rm merge}=\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}|I_i(hkl)-\langle I(hkl)\rangle|/\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}I_i(hkl)\eqno(2.2.2.4)] <I(hkl)> is the mean of the several individual measurements Ii(hkl) of the intensity of reflection hkl
   
Rmeas or Rr.i.m (redundancy-independent merging R factor),[R_{\rm r.i.m.}=\textstyle \sum \limits_{hkl}\left\{N(hkl)/[N(hkl)-1]\right\}^{1/2}\textstyle \sum \limits_{i}|I_i(hkl)-\langle I(hkl)\rangle|/\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}I_i(hkl)\eqno(2.2.2.5)] <I(hkl)> is the mean of the N(hkl) individual measurements Ii(hkl) of the intensity of reflection hkl
   
Rp.i.m. (precision-indicating merging R factor),[R_{\rm p.i.m.}=\textstyle \sum \limits_{hkl}\left\{1/[N(hkl)-1]\right\}^{1/2}\textstyle \sum \limits_{i}|I_i(hkl)-\langle I(hkl)\rangle|/\textstyle \sum \limits_{hkl}\textstyle \sum \limits_{i}I_i(hkl)\eqno(2.2.2.6)] <I(hkl)> is the mean of the N(hkl) individual measurements Ii(hkl) of the intensity of reflection hkl
   
Ranom (anomalous R factor), [R_{\rm anom}=\textstyle \sum \limits_{hkl}|I(hkl)-I(\bar h \bar k \bar l)|/\textstyle \sum \limits_{hkl}\langle I(hkl)\rangle\eqno(2.2.2.12)] <I(hkl)> is the mean intensity of the Friedel mates of the reflection hkl, or [\textstyle{1\over 2}[I(hkl)+I(\bar h\bar k\bar l)]]
   
RCullis (Cullis R factor for isomorphous-replacement applications), [R_{\rm Cullis}=\textstyle \sum \limits_{hkl}\big|F_{\rm PH}-|F_{\rm P}+F_{\rm H}|\big|/\textstyle \sum \limits_{hkl}|F_{\rm PH}-F_{\rm P}|\eqno(2.2.5.1b)] FP, FPH and FH are the structure factors for the protein, the heavy-atom derivative and the heavy atoms alone, respectively
   
PPiso (phasing power for isomorphous-replacement applications), [{\rm PP}_{\rm iso}=\textstyle \sum \limits_{hkl}|F_{\rm H}|/\textstyle \sum \limits_{hkl}\big|F_{\rm PH}-|F_{\rm P}+F_{\rm H}|\big|\eqno(2.2.5.3)]or, as in the program SOLVE,[{\rm PP}_{\rm iso}=(\textstyle \sum \limits_{hkl}|F_{\rm H}|^2)^{1/2}/(\textstyle \sum \limits_{hkl}\langle\big|F_{\rm PH}-|F_{\rm P}+F_{\rm H}|\big|\rangle^2)^{1/2}\eqno(2.2.5.4)] FP, FPH and FH are the structure factors for the protein, the heavy-atom derivative and the heavy atoms alone, respectively
   
R (crystallographic R factor) and Rfree (free R factor), [R=\textstyle \sum \limits_{hkl}\big||F_{\rm obs}|-|F_{\rm calc}|\big|/\textstyle \sum \limits_{hkl}|F_{\rm obs}|\eqno(2.2.8.1)] Rfree is defined as the crystallographic R factor but for a subset of reflections that have been excluded from refinement
   
RSRF (real-space R factor), [{\rm RSRF}=2\textstyle \sum \limits_{xyz}|\rho_1-\rho_2|/\textstyle \sum \limits_{xyz}(\rho_1+\rho_2)\eqno(2.2.9.1)] ρ1 and ρ2 are the electron-density maps calculated from the structural model and from the experimental data, respectively
   
R.m.s.d.'s of geometric parameters x, [{\rm r.m.s.d}(x)=\big\{\textstyle \sum \limits_{i}[x_i({\rm ideal})-x_i({\rm observed})]^2/N\big\}^{1/2}\eqno(2.2.9.2)] xi are the individual values, ideal or observed, of the geometric parameter x and the sum is over all N xi observed. The geometric parameters x may be bond lengths, bond angles, dihedral angles, chiral volumes, deviations from planarity etc.
   
DPI (diffraction-component precision index), [{\rm DPI}=[N_{\rm atom}/(N_{hkl}-N_{\rm para})]^{1/2}Rd_{\rm min}C^{-1/3}\eqno(2.2.10.1)] Natom is the number of atoms in the structure, Nhkl is the number of reflections, Npara is the number of refined parameters, R is the crystallographic R factor, dmin is the nominal resolution and C is the fractional completeness of the data set

Acknowledgements

We gratefully acknowledge the contributions of Kevin Cowtan (York, UK), Kay Diederichs (Konstanz, Germany), Phil Evans (Cambridge, UK), John Helliwell (Manchester, UK), Randy Read (Cambridge, UK), George Sheldrick (Göttingen, Germany) and Tom Terwilliger (Los Alamos, USA).

References

Arnberg, L., Hovmöller, S. & Westman, S. (1979). On the significance of `non-significant' reflexions. Acta Cryst. A35, 497–499.
Blow, D. M. (2002). Rearrangement of Cruickshank's formulae for the diffraction-component precision index. Acta Cryst. D58, 792–797.
Blow, D. M. & Crick, F. H. C. (1959). The treatment of errors in the isomorphous replacement method. Acta Cryst. 12, 794–802.
Blundell, T. L. & Johnson, L. N. (1976). Protein Crystallography. New York: Academic Press.
Brünger, A. T. (1992). Free R value: a novel statistical quantity for assessing the accuracy of crystal structures. Nature (London), 355, 472–475.
Brünger, A. T. (1997). Free R value: cross-validation in crystallography. Methods Enzymol. 277, 366–396.
Collaborative Computational Project, Number 4 (1994). The CCP4 suite: programs for protein crystallography. Acta Cryst. D50, 760–763.
Cowtan, K. (1999). Error estimation and bias correction in phase-improvement calculations. Acta Cryst. D55, 1555–1567.
Cowtan, K. & Main, P. (1998). Miscellaneous algorithms for density modification. Acta Cryst. D54, 487–493.
Cowtan, K. D. & Main, P. (1996). Phase combination and cross validation in iterated density-modification calculations. Acta Cryst. D52, 43–48.
Cruickshank, D. W. J. (1999a). Remarks about protein structure precision. Acta Cryst. D55, 583–601.
Cruickshank, D. W. J. (1999b). Remarks about protein structure precision. Erratum. Acta Cryst. D55, 1108.
Cullis, A. F., Muirhead, H., Perutz, M. F., Rossmann, M. G. & North, A. C. T. (1961). The structure of haemoglobin. VIII. A three-dimensional Fourier synthesis at 5.5 Å resolution: determination of the phase angles. Proc. R. Soc. London Ser. A, 265, 15–38.
Diederichs, K. (2006). Some aspects of quantitative analysis and correction of radiation damage. Acta Cryst. D62, 96–101.
Diederichs, K. (2010). Quantifying instrument errors in macromolecular X-ray data sets. Acta Cryst. D66, 733–740.
Diederichs, K. & Karplus, P. A. (1997a). Improved R-factors for diffraction data analysis in macromolecular crystallography. Nat. Struct. Biol. 4, 269–275.
Diederichs, K. & Karplus, P. A. (1997b). Improved R-factors for diffraction data analysis in macromolecular crystallography. Erratum. Nat. Struct. Biol. 4, 592.
Engh, R. A. & Huber, R. (1991). Accurate bond and angle parameters for X-ray protein structure refinement. Acta Cryst. A47, 392–400.
Evans, P. (2006). Scaling and assessment of data quality. Acta Cryst. D62, 72–82.
Fisher, S. J., Helliwell, J. R., Khurshid, S., Govada, L., Redwood, C., Squire, J. M. & Chayen, N. E. (2008). An investigation into the protonation states of the C1 domain of cardiac myosin-binding protein C. Acta Cryst. D64, 658–664.
Grosse-Kunstleve, R. W. & Adams, P. D. (2003). Substructure search procedures for macromolecular structures. Acta Cryst. D59, 1966–1973.
Hauptman, H. & Karle, J. (1953). Solution of the phase problem. I. The centrosymmetric crystal. Am. Crystallogr. Assoc. Monograph No. 3. Dayton, Ohio: Polycrystal Book Service.
Hirshfeld, F. L. & Rabinovich, D. (1973). Treating weak reflexions in least-squares calculations. Acta Cryst. A29, 510–513.
Hooft, R. W. W., Vriend, G., Sander, C. & Abola, E. E. (1996). Errors in protein structures. Nature (London), 381, 272.
Howell, P. L. & Smith, G. D. (1992). Identification of heavy-atom derivatives by normal probability methods. J. Appl. Cryst. 25, 81–86.
James, R. W. (1948). False detail in three-dimensional Fourier representations of crystal structures. Acta Cryst. 1, 132–134.
Jones, T. A., Zou, J.-Y., Cowan, S. W. & Kjeldgaard, M. (1991). Improved methods for building protein models in electron density maps and the location of errors in these models. Acta Cryst. A47, 110–119.
Kabsch, W. (1988). Evaluation of single-crystal X-ray diffraction data from a position-sensitve detector. J. Appl. Cryst. 21, 916–924.
Kabsch, W. (1993). Automatic processing of rotation diffraction data from crystals of initially unknown symmetry and cell constants. J. Appl. Cryst. 26, 795–800.
Kabsch, W. (2010). XDS. Acta Cryst. D66, 125–132.
Kleywegt, G. & Brünger, A. T. (1996). Checking your imagination: applications of the free R value. Structure, 4, 897–904.
Long, F., Vagin, A. A., Young, P. & Murshudov, G. N. (2008). BALBES: a molecular-replacement pipeline. Acta Cryst. D64, 125–132.
Luzzati, V. (1952). Traitement statistique des erreurs dans la determination des structures cristallines. Acta Cryst. 5, 802–810.
McCoy, A. J., Grosse-Kunstleve, R. W., Adams, P. D., Winn, M. D., Storoni, L. C. & Read, R. J. (2007). Phaser crystallographic software. J. Appl. Cryst. 40, 658–674.
Matthews, B. W. (1968). Solvent content of protein crystals. J. Mol. Biol. 33, 491–497.
Otwinowski, Z. & Minor, W. (1997). Processing of X-ray diffraction data collected in oscillation mode. Methods Enzymol. 276, 307–326.
Panjikar, S. & Tucker, P. A. (2002). Phasing possibilities using different wavelengths with a xenon derivative. J. Appl. Cryst. 35, 261–266.
Parkinson, G., Vojtechovsky, J., Clowney, L., Brünger, A. T. & Berman, H. M. (1996). New parameters for the refinement of nucleic acid-containing structures. Acta Cryst. D52, 57–64.
Pflugrath, J. W. (1999). The finer things in X-ray diffraction data collection. Acta Cryst. D55, 1718–1725.
Read, R. J. (1986). Improved Fourier coefficients for maps using phases from partial structures with errors. Acta Cryst. A42, 140–149.
Read, R. J. (2001). Pushing the boundaries of molecular replacement with maximum likelihood. Acta Cryst. D57, 1373–1382.
Rodgers, J. L. & Nicewander, W. A. (1988). Thirteen ways to look at the correlation coefficient. Am. Stat. 42, 59–66.
Rodriguez, D. D., Grosse, C., Himmel, S., Gonzalez, C., de Ilarduya, I. M., Becker, S., Sheldrick, G. M. & Uson, I. (2009). Crystallographic ab initio protein structure solution below atomic resolution. Nat. Methods, 6, 651–653.
Rossmann, M. G. & Blow, D. M. (1962). The detection of sub-units within the crystallographic asymmetric unit. Acta Cryst. 15, 24–31.
Schneider, T. R. & Sheldrick, G. M. (2002). Substructure solution with SHELXD. Acta Cryst. D58, 1772–1779.
Sheldrick, G. M. (2002). Macromolecular phasing with SHELXE. Z. Kristallogr. 217, 644–650.
Sheldrick, G. M. (2008). A short history of SHELX. Acta Cryst. A64, 112–122.
Sheldrick, G. M. (2010). Experimental phasing with SHELXC/D/E: combining chain tracing with density modification. Acta Cryst. D66, 479–485.
Stenkamp, R. E. & Jensen, L. H. (1984). Resolution revisited: limit of detail in electron density maps. Acta Cryst. A40, 251–254.
Terwilliger, T. C. (2001). Map-likelihood phasing. Acta Cryst. D57, 1763–1775.
Terwilliger, T. C., Adams, P. D., Read, R. J., McCoy, A. J., Moriarty, N. W., Grosse-Kunstleve, R. W., Afonine, P. V., Zwart, P. H. & Hung, L.-W. (2009). Decision-making in structure solution using Bayesian estimates of map quality: the PHENIX AutoSol wizard. Acta Cryst. D65, 582–601.
Terwilliger, T. C. & Berendzen, J. (1999). Discrimination of solvent from protein regions in native Fouriers as a means of evaluating heavy-atom solutions in the MIR and MAD methods. Acta Cryst. D55, 501–505.
Vaguine, A. A., Richelle, J. & Wodak, S. J. (1999). SFCHECK: a unified set of procedures for evaluating the quality of macromolecular structure-factor data and their agreement with the atomic model. Acta Cryst. D55, 191–205.
Vellieux, F. M. D. A. P., Hunt, J. F., Roy, S. & Read, R. J. (1995). DEMON/ANGEL: a suite of programs to carry out density modification. J. Appl. Cryst. 28, 347–351.
Weeks, C. M., De Titta, G. T., Miller, R. & Hauptman, H. A. (1993). Application of the minimal principle to peptide structures. Acta Cryst. D49, 179–181.
Weeks, C. M., DeTitta, G. T., Hauptman, H. A., Thuman, P. & Miller, R. (1994). Structure solution by minimal-function phase refinement and Fourier filtering. II. Implementation and applications. Acta Cryst. A50, 210–220.
Weiss, M. S. (2001). Global indicators of X-ray data quality. J. Appl. Cryst. 34, 130–135.
Weiss, M. S. & Hilgenfeld, R. (1997). On the use of the merging R factor as a quality indicator for X-ray data. J. Appl. Cryst. 30, 203–205.
Wilson, A. J. C. (1949). The probability distribution of X-ray intensities. Acta Cryst. 2, 318–321.








































to end of page
to top of page