We present an algorithm to automatically register magnetic resonance (MR) and positron emission tomographic (PET) images of the human brain. Our algorithm takes an integrated approach: we simultaneously segment the brain in both modalities and register the slices. The algorithm does not attempt to remove the skull from the MR image, but rather uses “templates” constructed from PET images to locate the boundary between the brain and the surrounding tissue in the MR images. The PET templates are a sequence of estimates of the boundary of the brain in the PET images. For each of the templates, the registration algorithm aligns the MR and PET images by minimizing an energy function. The energy function is designed to implicitly model the relevant anatomical structure in the MR image. The template with the lowest energy after registration is the PET brain boundary. The alignment of this template in the MR image marks the MR brain boundary and gives the transformation between the two images. © 1998 John Wiley & Sons, Inc. Int J Imaging Syst Technol, 9, 46–50, 1998