International
Tables for Crystallography Volume B Reciprocal space Edited by U. Shmueli © International Union of Crystallography 2006 
International Tables for Crystallography (2006). Vol. B, ch. 2.3, pp. 250258
Section 2.3.6. Rotation functions ^{a}Department of Biological Sciences, Purdue University, West Lafayette, Indiana 47907, USA, and ^{b}CABM & Rutgers University, 679 Hoes Lane, Piscataway, New Jersey 088545638, USA 
The rotation function is designed to detect noncrystallographic rotational symmetry (see Table 2.3.6.1). The normal rotation function definition is given as (Rossmann & Blow, 1962) where and are two Pattersons and U is an envelope centred at the superimposed origins. This convolution therefore measures the degree of similarity, or `overlap', between the two Pattersons when has been rotated relative to by an amount defined by The elements of [C] will depend on three rotation angles . Thus, R is a function of these three angles. Alternatively, the matrix [C] could be used to express mirror symmetry, permitting searches for noncrystallographic mirror or glide planes.

The basic concepts were first clearly stated by Rossmann & Blow (1962), although intuitive uses of the rotation function had been considered earlier. Hoppe (1957b) had also hinted at a convolution of the type given by (2.3.6.1) to find the orientation of known molecular fragments and these ideas were implemented by Huber (1965).
Consider a structure of two identical units which are in different orientations. The Patterson function of such a structure consists of three parts. There will be the selfPatterson vectors of one unit, being the set of interatomic vectors which can be formed within that unit, with appropriate weights. The set of selfPatterson vectors of the other unit will be identical, but they will be rotated away from the first due to the different orientation. Finally, there will be the crossPatterson vectors, or set of interatomic vectors which can be formed from one unit to another. The selfPatterson vectors of the two units will all lie in a volume centred at the origin and limited by the overall dimensions of the units. Some or all of the crossPatterson vectors will lie outside this volume. Suppose the Patterson function is now superposed on a rotated version of itself. There will be no particular agreement except when one set of selfPatterson vectors of one unit has the same orientation as the selfPatterson vectors from the other unit. In this position, we would expect a maximum of agreement or `overlap' between the two. Similarly, the superposition of the molecular selfPatterson derived from different crystal forms can provide the relative orientation of the two crystals when the molecules are aligned.
While it would be possible to evaluate R by interpolating in and forming the pointbypoint product with within the volume U for every combination of and , such a process is tedious and requires large computer storage for the Pattersons. Instead, the process is usually performed in reciprocal space where the number of independent structure amplitudes which form the Pattersons is about onethirtieth of the number of Patterson grid points. Thus, the computation of a rotation function is carried out directly on the structure amplitudes, while the overlap definition (2.3.6.1) simply serves as a physical basis for the technique.
The derivation of the reciprocalspace expression depends on the expansion of each Patterson either as a Fourier summation, the conventional approach of Rossmann & Blow (1962), or as a sum of spherical harmonics in Crowther's (1972) analysis. The conventional and mathematically easier treatment is discussed presently, but the reader is referred also to Section 2.3.6.5 for Crowther's elegant approach. The latter leads to a rapid technique for performing the computations, about one hundred times faster than conventional methods.
Let, omitting constant coefficients, and From (2.3.6.2) it follows that and, hence, by substitution in (2.3.6.1) where When the volume U is a sphere, has the analytical form where and . G is a spherical interference function whose form is shown in Fig. 2.3.6.1

Shape of the interference function G for a spherical envelope of radius R at a distance H from the reciprocalspace origin. [Reprinted from Rossmann & Blow (1962).] 
The expression (2.3.6.3) represents the rotation function in reciprocal space. If in the argument of , then h′ can be seen as the point in reciprocal space to which p is rotated by [C]. Only for those integral reciprocallattice points which are close to h′ will be of an appreciable size (Fig. 2.3.6.1). Thus, the number of significant terms is greatly reduced in the summation over p for every value of h, making the computation of the rotation function manageable.
The radius of integration R should be approximately equal to or a little smaller than the molecular diameter. If R were roughly equal to the length of a lattice translation, then the separation of reciprocallattice points would be about . Hence, when H is equal to one reciprocallattice separation, , and G is thus quite small. Indeed, all terms with might well be neglected. Thus, in general, the only terms that need be considered are those where is within one lattice point of h. However, in dealing with a small molecular fragment for which R is small compared to the unitcell dimensions, more reciprocallattice points must be included for the summation over p in the rotationfunction expression (2.3.6.3).
In practice, the equation that is or determines p, given a set of Miller indices h. This will give a nonintegral set of Miller indices. The terms included in the inner summation of (2.3.6.3) will be integral values of p around the nonintegral lattice point found by solving (2.3.6.5).
Details of the conventional program were given by Tollin & Rossmann (1966) and follow the principles outlined above. They discussed various strategies as to which crystal should be used to calculate the first (h) and second (p) Patterson. Rossmann & Blow (1962) noted that the factor in expression (2.3.6.3) represents an interpolation of the squared transform of the selfPatterson of the second (p) crystal. Thus, the rotation function is a sum of the products of the two molecular transforms taken over all the h reciprocallattice points. Lattman & Love (1970) therefore computed the molecular transform explicitly and stored it in the computer, sampling it as required by the rotation operation. A discussion on the suitable choice of variables in the computation of rotation functions has been given by Lifchitz (1983).
The initial step in the rotationfunction procedure involves the orthogonalization of both crystal systems. Thus, if fractional coordinates in the first crystal system are represented by x, these can be orthogonalized by a matrix [β] to give the coordinates X in units of length (Fig. 2.3.6.2); that is, If the point X is rotated to the point X′, then where represents the rotation matrix relating the two vectors in the orthogonal system. Finally, X′ is converted back to fractional coordinates measured along the oblique cell dimension in the second crystal by Thus, by substitution, and by comparison with (2.3.6.2) it follows that Fig. 2.3.6.2 shows the mode of orthogonalization used by Rossmann & Blow (1962). With their definition it can be shown that and where with . For a Patterson compared with itself, .

Relationships of the orthogonal axes to the crystallographic axes . [Reprinted from Rossmann & Blow (1962).] 
Both spherical and Eulerian angles are used in evaluating the rotation function. The usual definitions employed are given diagrammatically in Figs. 2.3.6.3 and 2.3.6.4. They give rise to the following rotation matrices.

Eulerian angles relating the rotated axes to the original unrotated orthogonal axes . [Reprinted from Rossmann & Blow (1962).] 

Variables ψ and ϕ are polar coordinates which specify a direction about which the axes may be rotated through an angle κ. [Reprinted from Rossmann & Blow (1962).] 
(a) Matrix [] in terms of Eulerian angles : and (b) matrix [] in terms of rotation angle κ and the spherical polar coordinates ψ, ϕ: Alternatively, (b) can be expressed as where u, v and w are the direction cosines of the rotation axis given by This latter form also demonstrates that the trace of a rotation matrix is .
The relationship between the two sets of variables established by comparison of the elements of the two matrices yields Since ϕ and ψ can always be chosen in the range 0 to π, these equations suffice to find from any set .
In analogy with crystal lattices, the rotation function is periodic and contains symmetry. The rotation function has a cell whose periodicity is 2π in each of its three angles. This may be written as or where , and are integers. A redundancy in the definition of either set of angles leads to the equivalence of the following points: or These relationships imply an n glide plane perpendicular to for Eulerian space or a ϕ glide plane perpendicular to ψ in polar space.
In addition, the Laue symmetry of the two Pattersons themselves must be considered. This problem was first discussed by Rossmann & Blow (1962) and later systematized by Tollin et al. (1966), Burdina (1970, 1971, 1973) and Rao et al. (1980). A closely related problem was considered by Hirshfeld (1968). The rotation function will have the same value whether the Patterson density at X or in the first crystal is multiplied by the Patterson density at or in the second crystal. and refer to the ith and jth crystallographic rotations in the orthogonalized coordinate systems of the first and second crystal, respectively. Hence, from (2.3.6.6) or Thus, it is necessary to find angular relationships which satisfy the relation for given Patterson symmetries. Tollin et al. (1966) show that the Eulerian angular equivalences can be expressed in terms of the Laue symmetries of each Patterson (Table 2.3.6.2).
^{†}This axis is not unique (that is, it can always be generated by two other unique axes), but is included for completeness.

The example given by Tollin et al. (1966) is instructive in the use of Table 2.3.6.2. They consider the determination of the Eulerian space group when has symmetry Pmmm and has symmetry . These Pattersons contain the proper rotation groups 222 and 2 (parallel to b), respectively. Inspection of Table 2.3.6.2 shows that these symmetries produce the following Eulerian relationships:
Nonlinear transformations occur when using Eulerian symmetries for threefold axes along [111] (as in the cubic system) or when using polar coordinates. Hence, Eulerian angles are far more suitable for a derivation of the limits of the rotationfunction asymmetric unit. However, when searching for given molecular axes, where some plane of κ need be explored, polar angles are more useful.
Rao et al. (1980) have determined all possible rotation function Eulerian space groups, except for combinations with Pattersons of cubic space groups. They numbered these rotation groups 1 through 100 (Table 2.3.6.3) according to the combination of the Patterson Laue groups. The characteristics of each of the 100 groups are given in Table 2.3.6.4, including the limits of the asymmetric unit. In the 100 unique combinations of noncubic Laue groups, there are only 16 basic rotation function space groups.

Notes: (a) This is the number of equivalent positions in the rotation unit cell. (b) Each symbol retains the order . The monoclinic space groups have the b axis unique setting. (c) This is a translation symmetry: e.g. for the case of translation along the axis, goes to and , and . All other equivalent positions in the basic rotation space group are similarly translated. (d) Several consistent sets of ranges exist but the one with the minimum range of is listed.

If the origins are retained in the Pattersons, their product will form a high but constant plateau on which the rotationfunction peaks are superimposed; this leads to a small apparent peaktonoise ratio. The effect can be eliminated by removal of the origins through a modification of the Patterson coefficients. Irrespective of origin removal, a significant peak is one which is more than three r.m.s. deviations from the mean background.
As in all continuous functions sampled at discrete points, a convenient grid size must be chosen. Small intervals result in an excessive computing burden, while large intervals might miss peaks. Furthermore, equal increments of angles do not represent equal changes in rotation, which can result in distorted peaks (Lattman, 1972). In general, a crude idea of a useful sampling interval can be obtained by considering the angle necessary to move one reciprocallattice point onto its neighbour (separated by ) at the extremity of the resolution limit, R. This interval is given by
Simple sharpening of the rotation function can be useful. This can be achieved by restricting the computations to a shell in reciprocal space or by using normalized structure factors. Useful limits are frequently 10 to 6 Å for an average protein or 6 to 5 Å for a virus structure determination.
When exploring the rotation function in polar coordinates, there is no significance to the latitude ϕ (Fig. 2.3.6.4) when . For small values of ψ, the rotation function will be quite insensitive to ϕ, which therefore needs to be explored only at coarse intervals (say 45°). As ψ approaches the equator at 90°, optimal increments of ψ and ϕ will be about equal. A similar situation exists with Eulerian angles. When , the rotation function will be determined by , corresponding to and varying κ in polar coordinates. There will be no dependence on . Thus Eulerian searches can often be performed more economically in terms of the variables and , where which reduces to the simple rotation matrix when .
The computational effort to explore carefully a complete asymmetric unit of the rotationfunction Eulerian group can be considerable. However, unless improper rotations are under investigation (as, for example, crossrotation functions between different crystal forms of the same molecule), it is not generally necessary to perform such a global search. The number of molecules per crystallographic asymmetric unit, or the number of subunits per molecule, are often good indicators as to the possible types of noncrystallographic symmetry element. For instance, in the early investigation of insulin, the rotation function was used to explore only the plane in polar coordinates as there were only two molecules per crystallographic asymmetric unit (Dodson et al., 1966). Rotation functions of viruses, containing 532 icosahedral symmetry, are usually limited to exploration of the and 144° planes [e.g. Rayment et al. (1978) and Arnold et al. (1984)].
In general, the interpretation of the rotation function is straightforward. However, noise often builds up relative to the signal in highsymmetry space groups or if the data are limited or poor. One aid to a systematic interpretation is the locked rotation function (Rossmann et al., 1972) for use when a molecule has more than one noncrystallographic symmetry axis. It is then possible to determine the rotationfunction values for each molecular axis for a chosen molecular orientation (Fig. 2.3.6.6).
Another problem in the interpretation of rotation functions is the appearance of apparent noncrystallographic symmetry that relates the selfPatterson of one molecule to the selfPatterson of a crystallographically related molecule. For example, take the case of αchymotrypsin (Blow et al., 1964). The space group is with a molecular dimer in each of the two crystallographic asymmetric units. The noncrystallographic dimer axis was found to be perpendicular to the crystallographic axis. The product of the crystallographic twofold in the Patterson with the orthogonal twofold in the selfPatterson vectors around the origin creates a third twofold, orthogonal to both of the other twofolds. In real space this represents a twofold screw direction relating the two dimers in the cell. In other cases, the product of the crystallographic and noncrystallographic symmetry results in symmetry which only has meaning in terms of all the vectors in the vicinity of the Patterson origin, but not in real space. Rotationfunction peaks arising from such products are called Klug peaks (Johnson et al., 1975). Such peaks normally refer to the total symmetry of all the vectors around the Patterson origin and may, therefore, be much larger than the peaks due to noncrystallographic symmetry within one molecule alone. Hence the Klug peaks, if not correctly recognized, can lead to erroneous conclusions (Åkervall et al., 1972). Litvin (1975) has shown how Klug peaks can be predicted. These usually occur only for special orientations of a particle with a given symmetry relative to the crystallographic symmetry axes. Prediction of Klug peaks requires the simultaneous consideration of the noncrystallographic point group, the crystallographic point group and their relative orientations.
Unfortunately, the rotationfunction computations can be extremely timeconsuming by conventional methods. Sasada (1964) developed a technique for rapidly finding the maximum of a given peak by looking at the slope of the rotation function. A major breakthrough came when Crowther (1972) recast the rotation function in a manner suitable for rapid computation. Only a brief outline of Crowther's fast rotation function is given here. Details are found in the original text (Crowther, 1972) and his computer program description.
Since the rotation function correlates spherical volumes of a given Patterson density with rotated versions of either itself or another Patterson density, it is likely that a more natural form for the rotation function will involve spherical harmonics rather than the Fourier components of the crystal representation. Thus, if the two Patterson densities and are expanded within the spherical volume of radius less than a limiting value of a, then and and the rotation function would then be defined as Here is the normalized spherical harmonic of order l; is the normalized spherical Bessel function of order l; , are complex coefficients; and represents the rotated second Patterson. The rotated spherical harmonic can then be expressed in terms of the Eulerian angles as where and are the matrix elements of the threedimensional rotation group. It can then be shown that Since the radial summation over n is independent of the rotation, and hence or The coefficients refer to a particular pair of Patterson densities and are independent of the rotation. The coefficients , containing the whole rotational part, refer to rotations of spherical harmonics and are independent of the particular Patterson densities. Since the summations over m and m′ represent a Fourier synthesis, rapid calculation is possible.
As polar coordinates rather than Eulerian angles provide a more graphic interpretation of the rotation function, Tanaka (1977) has recast the initial definition as He showed that the polar coordinates are now equivalent to , and . The rotation function can then be expressed as permitting rapid calculation of the fast rotation function in polar coordinates.
Crowther (1972) uses the Eulerian angles α, β, γ which are related to those defined by Rossmann & Blow (1962) according to , and .
References
Arnold, E., Erickson, J. W., Fout, G. S., Frankenberger, E. A., Hecht, H. J., Luo, M., Rossmann, M. G. & Rueckert, R. R. (1984). Virion orientation in cubic crystals of the human common cold virus HRV14. J. Mol. Biol. 177, 417–430.Blow, D. M., Rossmann, M. G. & Jeffery, B. A. (1964). The arrangement of αchymotrypsin molecules in the monoclinic crystal form. J. Mol. Biol. 8, 65–78.
Burdina, V. I. (1970). Symmetry of the rotation function. Kristallografiya, 15, 623–630.
Burdina, V. I. (1971). Symmetry of the rotation function. Sov. Phys. Crystallogr. 15, 545–550.
Burdina, V. I. (1973). Primitive rotation regions of two Patterson syntheses. Kristallografiya, 18, 694–700.
Crowther, R. A. (1972). The fast rotation function. In The molecular replacement method, edited by M. G. Rossmann, pp. 173–178. New York: Gordon & Breach.
Dodson, E., Harding, M. M., Hodgkin, D. C. & Rossmann, M. G. (1966). The crystal structure of insulin. III. Evidence for a 2fold axis in rhombohedral zinc insulin. J. Mol. Biol. 16, 227–241.
Hirshfeld, F. L. (1968). Symmetry in the generation of trial structures. Acta Cryst. A24, 301–311.
Hoppe, W. (1957b). Die `Faltmolekülmethode' – eine neue Methode zur Bestimmung der Kristallstruktur bei ganz oder teilweise bekannter Molekülstruktur. Acta Cryst. 10, 750–751.
Huber, R. (1965). Die automatisierte Faltmolekülmethode. Acta Cryst. 19, 353–356.
Johnson, J. E., Argos, P. & Rossmann, M. G. (1975). Rotation function studies of southern bean mosaic virus at 22 Å resolution. Acta Cryst. B31, 2577–2583.
Lattman, E. E. (1972). Optimal sampling of the rotation function. Acta Cryst. B28, 1065–1068.
Lattman, E. E. & Love, W. E. (1970). A rotational search procedure for detecting a known molecule in a crystal. Acta Cryst. B26, 1854–1857.
Lifchitz, A. (1983). On the choice of the model cell and the integration volume in the use of the rotation function. Acta Cryst. A39, 130–139.
Litvin, D. B. (1975). The molecular replacement method. I. The rotation function problem, application to bovine liver catalase and STNV. Acta Cryst. A31, 407–416.
Rao, S. N., Jih, J. H. & Hartsuck, J. A. (1980). Rotationfunction space groups. Acta Cryst. A36, 878–884.
Rayment, I., Johnson, J. E., Suck, D., Akimoto, T. & Rossmann, M. G. (1978). An 11 Å resolution electron density map of southern bean mosaic virus. Acta Cryst. B34, 567–578.
Rossmann, M. G. & Blow, D. M. (1962). The detection of subunits within the crystallographic asymmetric unit. Acta Cryst. 15, 24–31.
Rossmann, M. G., Ford, G. C., Watson, H. C. & Banaszak, L. J. (1972). Molecular symmetry of glyceraldehyde3phosphate dehydrogenase. J. Mol. Biol. 64, 237–249.
Sasada, Y. (1964). The differential rotation function. Acta Cryst. 17, 611–612.
Tanaka, N. (1977). Representation of the fastrotation function in a polar coordinate system. Acta Cryst. A33, 191–193.
Tollin, P. (1966). On the determination of molecular location. Acta Cryst. 21, 613–614.
Tollin, P. & Rossmann, M. G. (1966). A description of various rotation function programs. Acta Cryst. 21, 872–876.
Åkervall, K., Strandberg, B., Rossmann, M. G., Bengtsson, U., Fridborg, K., Johannisen, H., Kannan, K. K., Lövgren, S., Petef, G., Öberg, B., Eaker, D., Hjertén, S., Rydén, L. & Moring, I. (1972). Xray diffraction studies of the structure of satellite tobacco necrosis virus. Cold Spring Harbor Symp. Quant. Biol. 36, 469–488.