Seam Carving for Image Retargeting

Due Oct. 3

Your task is to implement the seam carving algorithm of Avidan & Shamir. This is a very clever algorithm that was developed for resizing images in a manner that is adaptive to the image content. Typical image resizing scales the image uniformly. The Avidan & Shamir algorithm attempts to remove the boring bits of the image while retaining the important bits. The cue that is used for determining what is important is very low-level, so the algorithm does not always produce good results. It is surprising, however, how well it does work. Apart from implementing the algorithm, your job is also to evaluate its utility.

Programming [30 points]

Implement a function that shrinks an image using seam removal:

function [J] = seamRemove(I,n,isVertical)

where n is the number of seams to remove. isVertical=1 if seams are vertical and otherwise 0.

Write a content removal program to remove a specific region in an image. Use matlab function roipoly to select the region. You can also protect regions that you want to keep.

Writeup [10 points]

  1. Show and discuss one interesting successful result of shrinking an image horizontally, and one vertically. Use two different images. Show side-by-side comparisons with scaling. Also show the seam removal maps.

  2. Same as (1), but show interesting unsuccessful results.

    Show two successful results for object removal using your code.

Extra Credit

Modify the seam removal code to implement image insertion.

Matlab Tips

  1. You can't use transpose to transpose an RGB image. Instead, use the permute function to swap the first two dimensions: permute(I,[2,1,3]).

  2. Do not implement any vertical seam algorithms. The horizontal seam case is simpler to implement in matlab, and you can use horizontal seam code to create and manipulate vertical seams by transposing arguments and results.

  3. If you want to make movies of your results, use the avifile, addframe, and close functions as follows, making sure that each frame is the same size:
    mov = avifile('movie.avi');
    frame.colormap = [];
    for i = 1:nframes,
        frame.cdata = someImage;
        mov = addframe(mov,frame);
    mov = close(mov);

    Note that the movies that matlab creates are enormous uncompressed AVI files. It is a good idea to transcode them into something like DivX or H.264 using something like Quicktime Pro or mencoder if you want to put them in your writeup or on the web.

  4. Use the sprintf function to construct output filenames based on input filenames. For example, sprintf('%s.avi',inputFileName) will create an AVI filename from some input file name.

Code Segments

The following code segment finds a horizontal seam for the image montains.jpg. Use your own image and try the code to see the result. You can use it as a basis for your seam removal function.

fname = 'montains.jpg';
im = imread(fname);
im = im2double(im);
[H,W,D] = size(im);
img = rgb2gray(im);
[fx, fy] = gradient(img);
cmap = abs(fx) + abs(fy);
scmap = imfilter(cmap, ones(5,5)/25, 'same', 'replicate');
cmap = cmap ./ (scmap + 1); % cmap is the cost of removing an image pixel  

prev_node = ones(H-2,W)*2;  % Backpointer
ac_im = ones(H,W);          % The optimal (minimum) value table
ac_im(1:H,1) = cmap(1:H,1); % First column initialization
ac_im(1,:) = 1000;          % Force the seam to stay inside
ac_im(H,:) = 1000;

% Dynamic Programming
for n = 2 : W
    s1 = cmap(2:H-1,n) + ac_im(1:H-2, n-1);
    s2 = cmap(2:H-1,n) + ac_im(2:H-1, n-1);
    s3 = cmap(2:H-1,n) + ac_im(3:H, n-1);
    s = [s1, s2, s3];
    [t, id] = min(s, [], 2);
    prev_node(:,n) = id;
    ac_im(2:H-1,n) = t;

%Convert backpointer index to shift
prev_node(prev_node == 1) = -1;
prev_node(prev_node == 2) = 0;
prev_node(prev_node == 3) = 1;

% Get the seam
sline = [];
[t, id] = min(ac_im(2:H-1, W));
sline = [id+1, sline];

for n = W-1 : -1 : 1
    id = id + prev_node(id, n);
    sline = [id+1, sline];

%Show the seam
hold on;
plot(1:W, sline, 'r', 'linewidth', 5);
hold off;