Creating 3D worlds from photo albums

Microsoft has developed a photo browser that can turn a collection of pictures into a navigable 3D space.

Representatives from Microsoft Live Labs and a University of Washington work group discussed the Photosynth browser Wednesday at Siggraph 2006, during a speech about the university's Photo Tourism system. Live Labs is developing Photosynth from the Photo Tourism technology as part of Microsoft's push into Internet-based personal services.

(See Photosynth in action in our photo story.)

"Photosynth will be released as a Web client that people can use to explore large collections of photos," said Richard Szeliski, who leads the Interactive Visual Media Group at Microsoft. Photosynth has the potential to create a complete virtual world made up of the world's photos, Szeliski said.

Photosynth lets people import photos, either from their own collection, or via a search of a photo-sharing site like Flickr. Drawing from the imported shots, Photosynth organises the photos into a virtual 3D collage, using an algorithm that measures location information, identifiable features and subsequent 3D points for similarities, differences and viewpoints.

"Unlike other groups who have worked on this before, Photosynth combines image-based rendering with search controls. Our scene reconstruction automatically estimates the position, orientation and focal length of the cameras that took the photo. Once the scene is reconstructed, the user can explore using the photo browser features," said Noah Snavely, a member of the CSE Graphics and Imaging Lab at the University of Washington that developed the Photo Tourism technology.

Viewers can choose to see a detail of the constructed photo scene by clicking on an object or dragging a box around a group of objects. The highest scoring photo for that particular object or view is blown up on the screen. Photos are scored based on the best resolution, lighting and view. Head-on views score higher, as do daytime shots.

Once in detail, viewers can move around and 'step back', and they have the option of seeing other images of the same view within the collection. Thumbnails of these alternates run along the bottom of the screen.

The program also renders transitional views from one photo to the next within the 3D space, giving the illusion that viewers are panning the scene with a video camera. Users can also annotate a particular photo, or portion of the scene. Annotations are shared among images based on the level of detail the image covers. For example, a tag describing the statue of Neptune at the Trevi Fountain in Italy would carry over to other views of the statue but not necessarily panoramic views of the entire Trevi Fountain.

Eventually, the Photosynth team plans to include a time dimension that would let viewers specify the time of day, season or even year they'd like to see with a specific view. The sample shown at Siggraph was a contemporary photo of Yosemite National Park, with a black and white Ansel Adams alternative available for the same vantage point.

Because the tool measures the focal length of each image, Photosynth can determine the exact location the photographer was standing in when she took the photo. 3D overview mode offers a 3D photo scene with these vantage points plotted throughout. Clicking on a vantage point on the rotunda of St. Peter's Basilica brings the viewer to a panoramic view of St. Peter's Square looking out from the rotunda.

According to the Photosynth project blog from Live Labs, there's no set date for a public beta release. Snavely said there are limitations in the program that the work group would like to improve. For one, the program cannot reliably match photos if they are too dark, or of too poor a resolution. The program currently lacks the power to create extremely large-scale renderings of the sort the team would like to achieve. For example, a 3D rendering of New York's Times Square is possible, but not one for the whole of Manhattan.