4D warping and segmentation:
4D image analysis project aims to develop and validate 4-dimensional (4D) image registration and segmentation algorithms, with emphasis on measuring brain and cardiac structure, as well as its evolution over time. The explosive growth of medical imaging has created the need for the development of highly automated methods for analysis of medical imagery. Image registration has attracted particular scientific interest, since it is necessary for integration and comparison of data from individuals or groups, as well as for the measurement of the temporal evolution of structure and function by co-registering different time scans of the same individual.
Although a plethora of 3D image registration methods has been developed and used extensively, there is currently a scarcity of methods for measuring longitudinal change. Longitudinal change was typically achieved by applying standard 3D registration methods to individual scans in a series, and then applying some sort of regression to them. However, applying 3D registration methods independently for each time-point in a longitudinal study typically leads to noisy longitudinal measurements, particularly for small structures such as the hippocampus, due to inconsistencies in atlas warping across different time-points.
We have developed a software package for deformable registration, called HAMMER, and was among the first groups to pioneer its extension to a fully 4D deformable registration method (4D-HAMMER) for aligning serial scans to an atlas, and to demonstrate its superiority over analogous 3D registration in measuring subtle patterns of brain change in serial scans. We have applied this method for longitudinal brain study , simultaneous segmentation and registration of longitudinal brain images , and cardiac motion estimation .
Longitudinal brain study 
The main pipelines of 4D-HAMMER for atlas warping can be briefly summarized here for better demonstration of its superiority in longitudinal brain measurement. To perform 4D atlas warping, we need to construct the 4D images for subject and atlas, respectively. For a subject, its images acquired at different times are first rigidly aligned to the space of the 1st time image using a rigid alignment algorithm, thus obtaining a sequence of rigidly aligned 3D images or a 4D subject image, where time is considered as the fourth dimension in addition to three spatial dimensions. For obtaining a 4D atlas, we simply repeat a 3D atlas as different time images since in our application of longitudinal study the brain changes due to ageing are relatively small. Thus, the goal of our 4D atlas warping algorithm is to align these two 4D images by requiring (1) both spatial and temporal smoothness of the deformation fields estimated from each 3D atlas to its corresponding subject image, (2) the maximized image similarity between the warped 4D atlas and the 4D subject image. By incorporating these two requirements as the energy terms into our 4D atlas warping algorithm, we can register these two 4D images and obtain consistent longitudinal labeling for the subject brain.
The experimental results show the significant improvement of the 4D atlas warping algorithm over our previous 3D atlas warping algorithm (HAMMER) in measuring subtle longitudinal changes of brain structures. For example, for the MR images of 9 elderly subjects aged 59-78, selected from our BLSA project, the 4D atlas warping algorithm measured 5.5% average hippocampal shrinkage during the 5 years, which is very close to the 5.7% by manual definition of an experienced rater. However, the 3D atlas warping method can achieve only 2.1%, and also the obtained longitudinal changes are noisy along different years (Fig 1). This experiment shows that the 4D warping method is more robust and accurate in estimating longitudinal changes.
Fig. 1. Average hippocampal volumes of the 9 subjects, estimated by 4D- and 3D- atlas warping algorithms.
CLASSIC: simultaneous segmentation and registration of longitudinal brain images 
A fundamental first step in morphometric analysis protocols is often the segmentation of brain images into 3 tissues: gray matter (GM), white matter (WM) and cerebrospinal fluid (CSF), so that subsequent tissue-specific volumetric analysis can be carried out. A variety of brain image segmentation algorithms are available for segmentation of 3D brain scans. One of the problems that we have encountered when applying these methods to serial scans is that the segmentation is not longitudinally-consistent, i.e., there are random jumps between the segmentations of consecutive time-points (see the middle row of Fig 2), since 3D scans in a series of scans are treated individually. Noise, limited contrast (especially in elderly individuals) and partial volume effects due to limited voxel size, lead to segmentation inaccuracies, which are especially detrimental in a longitudinal analysis that attempt to measure very subtle changes over periods of 6 months or a year.
Longitudinal analyses can greatly benefit from the availability of multiple scans per individual. Although some form of regression is typically applied to serial measurements in order to smooth out random noise and better estimate longitudinal change, it is far superior to incorporate longitudinal smoothness constraints during the image segmentation procedure, when the image data is being analyzed and available, rather than applying post-hoc smoothness constraints via regression on the segmentation results. We have used this principle to develop a 4D segmentation algorithm referred to as CLASSIC (Consistent Longitudinal Alignment and Segmentation for Serial Image Computing), which is relatively more consistent longitudinally, as well as accurate, compared to 3D segmentation. The details of this method are described in ; we briefly summarize the main intuition behind this algorithm next.
Assume, for a moment, that there is no brain atrophy or other brain change between two or more serial scans of the same person. In this case, combining all the scans in a tissue segmentation procedure would help filter out noise that is not systematic throughout the scans, and amplify the true signal, which is consistent throughout the series of scans. This is somewhat analogous to obtaining multiple excitations during an MRI acquisition and averaging them, in order to improve the signal to noise ratio. Consider, now, the fact that some atrophy has occurred during the series of scans, therefore the data cannot be simply combined/averaged to improve segmentation. However, if we knew the deformation (atrophy) pattern that maps the scan of time t to the scan at time t+1, for all time points t, we could jointly combine all scans in the series to reduce the effect of noise and achieve more accurate and longitudinally consistent segmentation. This idea is utilized by CLASSIC, which simultaneously estimates the longitudinal atrophy and the tissue segmentation, thereby achieving longitudinally consistent and accurate segmentation.
Fig 2 compares a typical segmentation result of BLSA serial scans using CLASSIC and 3D fuzzy tissue segmentation. The top row shows the original serial scans after rigid alignment, the middle row indicates the segmentation results using independent 3D fuzzy tissue classification on each scan, and the bottom row gives the results of the CLASSIC. It can be seen that CLASSIC gives not only spatially-smooth but also temporally-consistent segmentation results, e.g., in the circled regions.
Fig. 2. Comparison on segmentation results of serial brain scans, using 4D segmentation and 3D independent segmentation, respectively.
Consistent cardiac motion estimation 
We have extended 4D-HAMMER for consistent registration of sequences of volumetric MR images to an atlas. We required the deformation fields to be spatially smooth for registering the neighboring frames in the sequence, and also temporally smooth for consistent motion estimation. Promising results for consistent motion estimation were achieved.
We validated our method by comparing our estimated motion with that obtained using tagged MR data. Both tagged and untagged MR images were acquired from healthy volunteers on the same scanner during the same scanning session at end-expiration, thus the datasets were assumed self-registered. To quantize the accuracy of motion estimates, tag-intersection points were manually detected and tracked by an observer in three different SA slices and one LA slice over all time frames between end-diastole and end-systole, including 20 points on the SA slices and 14 points on the LA slices. The root mean square error between the in-plane displacements estimated by our method and the actual in-plane displacements as measured by the observer was within acceptable limits (2mm).
- Dinggang Shen, Christos Davatzikos, "Measuring Temporal Morphological Changes Robustly in Brain MR Images Via 4-Dimensional Template Warping", NeuroImage,21:1508-1517, April 2004.
- Zhong Xue, Dinggang Shen, Christos Davatzikos, "CLASSIC: Consistent Longitudinal Alignment and Segmentation for Serial Image Computing", NeuroImage, 30:388-399, 2006.
- Dinggang Shen, Hari Sundar, Zhong Xue, Yong Fan, Harold Litt, "Consistent Estimation of Cardiac Motions by 4D Image Registration", MICCAI, Palm Springs, California, USA, Oct 26~29, 2005.