Image Mosaics

Due Nov. 17

One theme in this class is that we can extend the physical limitations of cameras with computational techniques. In this project, you'll extend the field of view of a camera by forming a mosaic from multiple photographs.

When taken from a single viewpoint where there is no parallax between images, two overlapping images can be aligned using a homography. To estimate the homography, you'll need 4 pairs of corresponding points in the overlap region. You'll find them automatically.

The following directory contains the source images for the two examples shown above:

Programming: [20 points]

The first image in each example provided is the central image. It's simplest to construct a mosaic from a central image and a set of peripheral images, since we then need to find just one homography for each peripheral image.

Find the SIFT feature points on the images. Use RANSAC to find the homography from each peripheral image to the center image. A RANSAC sample code using sift features is provided to you. Download the sample code here. Run the code "ransac.m" to see the example result. The function "geth" computes the homography. However, the code does not refine the H after it finds all the matches.

Use the central image's coordinate system for the final mosaic. You need to figure out the min and max pixel coordinates when the peripheral images are mapped into the central image's coordinate system. Simply apply your homographies to the peripheral images' corner points to find the extent of the final mosaic.

Now generate a meshgrid of points to cover the mosaic, send these points through each homography in turn, and use interp2 to extract pixel values from the source images. For each input image, this produces an image the size of the final mosaic; map the 4 corners of each peripheral image to the mosaic and use roipoly to obtain a mask for each image before merging them.

Finally, assemble the mosaic from the remapped source images. The simplest method is to sequentially assign masked warped peripheral images and center image to the mosaic. In this way, at each point of the mosaic, the pixel comes from only one source image.

Writeup: [20 points]

  1. Show the resulting mosaic for one of the provided examples. Show the matching point pairs using the RANSAC method and also the remapped source images before they are blended into the final mosaic.

  2. Show one mosaic of your own creation, with at least 3 source images. Show the matching point pairs for each peripheral image to the center image. Show also the remapped source images before they are blended into the final mosaic.

Extra Credit

  1. Implement an improved blending algorithm. In particular, use the Laplacian pyramid based multiresolution spline to combine the source images into the final mosaic.

  2. Deal with mosaics that don't have a central image that overlaps all other images: for example, a horizontal or vertical chain of more than 3 images.

  3. Extend the automatic mosaicing so that not only does it find corresponding points between two images that are known to overlap, but it figures out which images overlap in the first place. This leads to a truly automatic algorithm that takes a pile of images as input and produces a mosaic as output.

Matlab Tips

  1. Don't forget the hold on command if you want to plot one thing on top of another.

  2. Do no use the full size images. Resize them down so that your debugging goes fast. For your final results, 25% size is sufficient.

  3. You can treat the central image as just another source image by using eye(3) as its homography.

  4. Use 'linear' interpolation with interp2; it's fast and looks good. Just make sure you don't use nearest neighbor interpolation, which looks terrible and may be the default.

What to Hand In

The usual things.