Tables for
Volume F
Crystallography of biological macromolecules
Edited by E. Arnold, D. M. Himmel and M. G. Rossmann

International Tables for Crystallography (2012). Vol. F, ch. 13.4, pp. 356-357   | 1 | 2 |

Section 13.4.6. Determining the molecular envelope

M. G. Rossmanna* and E. Arnoldb

aDepartment of Biological Sciences, Purdue University, West Lafayette, IN 47907–1392, USA, and  bBiomolecular Crystallography Laboratory, CABM & Rutgers University, 679 Hoes Lane, Piscataway, NJ 08854–5638, USA
Correspondence e-mail:

13.4.6. Determining the molecular envelope

| top | pdf |

Various techniques are available for determining the molecular envelope within which density can be averaged and outside of which the solvent can be flattened.

  • (1) By assumption of a simple geometric shape, such as a sphere. This is frequently used for icosahedral viruses.

  • (2) By manual inspection of a poor electron-density map which, nevertheless, gives some guidance as to the molecular boundaries. A variety of interactive graphical programs are available to help define the molecular boundary.

  • (3) By use of a homologous structure or other information, such as a cryo-electron-microscopy (cryo-EM) reconstruction at low resolution. The information about a homologous structure may be either in the form of an electron-density grid or, often more conveniently, as an atomic model.

  • (4) By inspection of an averaged map which should have weaker density beyond the limits of the molecular boundary where the NCS is no longer true.

Procedures (2)[link] and (3)[link] are advisable when the NCS redundancy is low. Procedure (4)[link] works well when the NCS redundancy is four or higher. The crystallographic asymmetric unit is likely to contain bits and pieces of molecules centred at various positions in the unit cell and neighbouring unit cells. Therefore, it is necessary to associate each grid point within the p-cell crystallographic asymmetric unit to a specific molecular centre or to solvent.

If the molecular-boundary assignments are to be made automatically, then the following procedure can be used. The number, M, of such molecules can be estimated by generating all centres, derived from the given position of the centre for the reference molecule, [{\bf s}_{p,\, n=1}], and then determining whether a molecule of radius [{R}_{\rm out}] would impinge on the crystallographic asymmetric unit within the defined boundaries. Here, [{R}_{\rm out}] is a liberal estimate of the molecular radius. The corresponding rotation matrices [[\hbox{E}_{m,\, n}]] and translation vectors [{\bf e}_{m,\, n}] can then be computed from ([link]) and ([link]).

Any grid point whose distance from all M centres is greater than [{R}_{\rm out}] can immediately be designated as being in the solvent region. For other grid points, it is necessary to examine the corresponding h-cell density. From ([link]), it follows that (setting [n = 1]) [{\bf x} = [\hbox{E}_{m,\, n=1}'']{\bf y}_{m} + ({\bf s}_{h} - [\hbox{E}_{m,\, n=1}'']{\bf s}_{p,\, m}),]where [[\hbox{E}_{m,\, n=1}''] = [\alpha_{h}][\hbox{R}_{n}^{-1}][\omega][\beta_{p}][\hbox{T}_{m}^{-1}] \eqno(](n can be set to 1, since the h-cell presumably contains an averaged molecular electron density, in which case it does not matter which molecular asymmetric unit is referenced). Thus, ([link]) can be used to determine the electron density at [{\bf y}_{m}] by inspecting the corresponding interpolated density, [\rho ({\bf x})], at x in the h-cell. Transfer of the electron density, [\rho ({\bf x})], from the h-cell to the p-cell using ([link]) is often useful to obtain an initial structure. However, to determine a suitable mask, it is useful to evaluate a modified electron density, [\langle\rho ({\bf x})\rangle], (see below) for the grid points immediately around x in the h-cell.

A variable parameter `CRIT' can be specified to establish the distribution of grid points that are within the molecular envelope. When the modified electron density, [\langle\rho ({\bf x})\rangle], is less than CRIT, the corresponding grid point at y is assumed to be in solvent. Otherwise, when [\langle\rho ({\bf x})\rangle] exceeds CRIT, the grid point at y is assigned to that molecule which has the largest [\langle\rho ({\bf x})\rangle]. If the percentage of grid points which might be assigned to more than one molecule is large (say, greater than 1% of the total number of grid points), it probably signifies that the value of CRIT is too low, that the molecular boundary is far from clear, or that the function used to define [\langle\rho ({\bf x})\rangle] was badly chosen (Fig.[link]). Grid points outside the molecular envelope can be set to the average solvent density.


Figure | top | pdf |

The volume of the molecular mask expressed as a percentage of the volume of the p-cell asymmetric unit, as determined by the density cutoff in the h-cell. When the modulus of the density cutoff is decreased to less than the mean smeared electron density within the protein, the mask volume increases rapidly. Intersection of the tangents suggests the most appropriate density cutoff value for mask generation. [Reproduced with permission from McKenna, Xia, Willingmann, Ilag & Rossmann (1992[link]).]

An essential criterion for the molecular envelope is that it obeys the noncrystallographic point-group symmetry. If the original h-cell electron density already possesses the molecular symmetry (e.g. icosahedral 532, 222 etc.), then the p-cell mask should also have that symmetry. However, if the mask boundaries were chosen manually, masks from different molecular centres might be in conflict and have local errors in the correct molecular symmetry. Such errors can be corrected by reimposing the noncrystallographic point-group symmetry on the p-cell mask. This can be conveniently achieved by setting the density at each grid point that was considered within the molecular envelope to a value of 100, and all other grid points to a density of zero. If the resultant density is averaged using the same routine as is used for averaging the actual electron density of the molecule, then the average density will remain 100 if the interpolated density is 100 at all noncrystallographically related points. However, if the original grid point is near the edge of the mask, finding the density at symmetry-related points may involve interpolation between density at level 100 and at level 0, giving an averaged density of less than 100. Hence, any grid point whose averaged density is below some criterion should be attributed to solvent.

Other improvements to mask generation were discussed by Rossmann et al. (1992[link]). In any event, the molecular-envelope definition should be periodically re-examined after a suitable number of electron-density-averaging cycles.


Rossmann, M. G., McKenna, R., Tong, L., Xia, D., Dai, J.-B., Wu, H., Choi, H.-K. & Lynch, R. E. (1992). Molecular replacement real-space averaging. J. Appl. Cryst. 25, 166–180.Google Scholar

to end of page
to top of page