The Taub Faculty of Computer Science Events and Talks

The growth in the number of digital photographs calls for new and improved image editing tools to help users view, navigate and manipulate them. Many of these tasks can be reduced to instances of organizing visual data to satisfy user constraints. In this talk I will present three such tools that help organize visual data at the pixel, patch and image level: Seam Carving, The Patch Transform, and Infinite Images. Seam Carving attempts to retarget an image to fit a specific display size by judiciously choosing which pixels to keep and which ones to remove. I will show a new energy function that improves the visual quality of the results, compared to the original seam carving algorithm, as well as an extension to video.

The patch transform breaks the image into a collection of patches and then recovers an image subject to user constraints. When no constraints are given this defaults to solving a standard jigsaw puzzle. Image editing operations are mapped to various user constraints such as fixing the spatial location of some of the patches, the size of the target image or the pool of patches to use. We define terms in a Markov network to specify a good image reconstruction from patches: neighboring patches must fit to form a plausible image, and each patch should be used only once. We find an approximate solution to the Markov network using loopy belief propagation, introducing an approximation to handle the combinatorial difficult patch exclusion constraint. We show that this approach allows for various image editing operations such as object cut and paste, hole filling, image retargeting and object removal.

Finally, the Infinite Images project gives users 3D navigation tools to explore large collections of images. Our system does not assume the photographs are of a single real 3D location, nor that they were taken at the same time. Instead, we organize the photos in themes, such as city streets or skylines, and rely on image content to determine how to arrange images to give the impression of a coherent 3D space.

* Parts of this research were done with Michael Rubinstein, Ariel Shamir, Taeg Sang Cho, Moshe Butman, Josef Sivic, Biliana Kaneva, Antonio Torralba and Bill Freeman