dynamic surround video

Traditional VR, using millions of polygons to describe the world, is going to take a long time to look like the real world.  I propose an intermediate step, utilizing the mostly-existing technologies of digital video playback and surround picture convolution.

The idea is to record video along a path, such as walking down a street or a mountain footpath, in the same 360º format used by surround-picture software (such as QuickTime VR), then letting the user playback the video at an adjustable rate, such that they have the experience of walking down the path with an unconstrained viewpoint.   This could give most of the navigational benefits of VR, with the photorealism of video.

  • Requirements
    • a video playback format that can play forwards and backwards, smoothly, at a variable rate
    • a way to hook the video output into a convolution software engine
    • a way to generate content, e.g.. a camcorder or set of cameras fitted to a special optical system for capturing 360º imagery
  • Drawbacks
    • dynamic elements, such as other people or moving vehicles, will look very strange on playback, so this approach is best for more static worlds, such as nature hikes
    • stereoscopic content might be impossible to gather, because the camera assemblies can't see through each other
  • Potential
    • good for a wide variety of tourism and educational applications
    • if the camera also accurately tracks positional information, then realtime 3D objects could be superimposed into the view, opening up additional applications

Since i began tracking the subject in 1996, there is no product yet which meets all these requirements.  The following is a list of related projects, products which implement some part of the process, and people that provide (so far, very expensive) services along these lines.

Related Projects

  • FlyCamFlyAbout
    • initial paper: FlyAbout: Spatially Indexed Panoramic Video  Don Kimber, Jonathan Foote, Surapong Lertsithichai, SIGMM 2001
    • closest thing yet to dynamic surround video
    • they used cylindrical, not spherical video, with their own 4-camera system: FlyCam
    • their approach for playback is to use Motion JPEG and have a server (FlyServer) provide frames from it to clients via HTTP - how well does this actually work in practice for navigating in either direction at a variety of speeds?
  • Google Streetview, which isn't video
    • However in 2009, they did demo the 20% Project: Liquid Galaxy which includes streetview captured as video and played back on a wraparound display

Hardware and Softwarehttp://ptgrey.com/products/ladybug2/images/LADYBUG2_white_background_large.jpg

  • Ladybug2 from Point Grey Research
    • six cameras (Sony 1/3" progressive scan CCD) that enable the system to collect video from more than 75% of the full sphere, and an IEEE-1394b interface that allows streaming to disk at 30fps.
    • Resolution: 1024x768
    • Ladybug SDK allows users to fully control the Ladybug2 Spherical Vision camera system and integrate its functionality with their own custom applications
    • costs around $10,000 for a full setup with hardware, software and cables
  • FirstPerson Immersive Video Technology, from Incited Media
    • the technology changed hands a few times, began at a company Enroute which used to make the great 2D stitching software, QuickStitch
    • they did not sell either the software or hardware, they only provided services
    • the sample content and camera array appear more cylindrical than spherical (what is the vertical angle?)
    • claimed the advantage of "high resolution imagery and broadcast quality playback"
    • their most-touted project was Britney Spears using their services for a video in May 2001, and a follow-on videogame in 2002
    • as of 2009, the Incited Media site no longer mentions their surround video technology
  • FullViewFullView
    • their hardware is a system of four cameras and four mirrors, a spin-off from Bell Labs / Lucent
    • their technology page claims some advantages over all the other approaches, such as higher resolution and lower distortion
    • limited FOV of ±25° Vertical, so it's really more cylindrical than spherical
    • they don't say much about their software, other than that it is capable of realtime video
    • according to FullView, the technology is patented and Enroute (while it existed) was a licensee, as well as Microsoft for its 360-degree videoconferencing product called RoundTable, which as of 2009 is sold by Polycom
  • Immersive Media / Dodeca (Portland, OR)DODECA 1000
    • their original DODECA 1000 camera used 11 videocameras in a dodecahedral array - no price, but reportedly high-quality cameras with the ability to synchronize their dynamic range, so that would be very expensive (several hundred thousand dollars)
    • software for playback on a normal PC is called the RoundAbout Viewer, and it uses Motion JPEG (like FlyAbout above) probably because of that format is better designed to support bidirectional play
    • their sample content shows some registration issues (alignment), but it's not that bad
    • As of 2009, the latest version is called the Dodeca 2360. They provide several different software for playback, including a GIS-based viewer, a Flash player, etc.
  • iMove "Multiview Interactive Video technology"
    • an $80,000 capture system?  not sure what the latest price is, they are now apparently marketed just to military users
    • SVS-2500 Spherical Video System includes a specialized 6-lens digital camera, each 768*768 pixels
    • designed for field applications, the three-pound camera is connected via Fire Wire to a ruggedized laptop computer system, with an external PDA for UI
    • the disk storage can hold 30 GB (around 45 minutes) of full surround content, and the system includes a belt battery pack
  • PAL 360° Lens System
    • a lens which attaches to a still-image camera (Nikon COOLPIX 995/990/950/910/800/; Fuji FINEPIX 2900Z) allowing for 55°-80° vertical FOV
    • the lenses cost around $1000
    • doesn't do video off the shelf, but might be customizable to do video
    • despite "PAL" in the product name, doesn't have anything to do with the European video standard PAL
    • included software does the transformation from a single spherical snapshot to a panoramic (cylindrical) image
  • RemoteRealityParamax360 Image (http://www.remotereality.com/ offline as of 2009?)
    • OneShot360™ on Nikon Coolpix 990surround-image capture solutions for both still images and video
    • image capture: OneShot360™, $1300
      • attaches to ~11 prosumer digital cameras
      • images can be saved in the following formats: IBM HotMedia, Apple QuickTime, MGI Java, and their native format (plays in their custom ActiveX control)
    • video capture is marketed as Surveillance Products
    • unlike other companies, these products have no royalties or use restrictions
    • additional hardware option (360Plus™, $700) supports full 360*360 still images
    • no API is mentioned, so it seems unlikely to support an interactive immersion usage
  • SurroundPhoto from VR Interactive Corp.
    • relatively inexpensive: reflector rig costs $100, software is pay-per-use at $7 per image
    • since a single image must be convolved to entire cylindrical projection, there are significant issues with resolution, dynamic range, etc.
    • unfortunately, their revenue model would surely prevent them from releasing an API version of their software, although equivalent convolution may not be difficult to reproduce and open-source
  • cam_live.jpgRoundShot (Switzerland)
    • they have a whole line of cameras which are designed to rotate to take panoramic images
    • however, the panorama requires rotation over time, so you couldn't capture whole images quickly, and no live video
    • despite this limitation, they have a product Panorama Livecam which sounds very nice; for around $12000, it's a standalone system which takes periodic panoramic images and serves them up on the web
  • Specifications ???SpheroVision (Quebec, Canada) (site was http://pages.infinit.net/sphero/ but is apparently gone as of 2003)
    • actually cylindrical, not spherical, but (was) a very promising low-cost option ($700)
    • uses 4 FireWire cameras, 960x180 combined resolution
    • includes software for live streaming and recording

Hardware

omnicam
  • Surround Viewer project at the Multimedia/Visualization Lab of the University of Missouri-Columbia

    • uses a device called the Omnicam which has been prototyped in many configurations

  • various motorized solutions such as IPIX Rotator provide a fixed increment pivot (30°, 180° etc.) for capture of still images for stitching
    • i don't think such a solution is applicable for video - unless your optics give you 360° in a single image, you need to capture multiple synchronized streams
  • a very painstaking process would be to use a still camera, on a pivot, and record multiple images at each point along the path, and generate the video streams from the stills as a separate pass, but that might be very tedious
  • an Infinite Pictures promotional appearance at Spring Comdex, 1998 mentions producing actual surround video, using a Dodeca camera and "smoothmove panoramic movie production capability"

Software

  • Already capable of video
    • Be Here iVideo (now gone?)
      • software products "iVideo Suite" and "Live!" are for developing surround video content, a free web plugin is used for playback
      • sold as a suite, it includes a "360lens", recording software, a digital video camera, and a video capture board
    • Warp VTV® [apparently no more, as of 2007]
      • spherical, 3 degrees of rotation (roll, pitch and yaw)
      • claims to have a demo CD actually demonstrating surround video (!) using Video for Windows, even built content with a dodecahedral camera array
      • basically, it looks like they had the same idea as above, but for some reason the website hasn't been updated in over 2 years
      • existing demo doesn't support 24-bit color (probably easy to address)
      • SDK available, documentation on the website
    • MGI PhotoVista™ (previously Live Picture®) [apparently offline as of 2003]
      • doesn't mention an SDK either
      • people have claimed to be viewing surround video with it
  • Might be capable of video
    • "Surround Video" from Black Diamond <site has been unresponsive>
      • incredibly, this is NOT video - as their FAQ says "Is Surround Video a video?  No, Surround Video is actually one static 360 degree image..."
      • does have a "Surround Video API" which is, again, not video
    • IPIX™
      • supplied software doesn't do video
      • spherical, 2 degrees of rotation
  • Not capable of video
    • QuickTime VR™
      • despite its name, is neither video nor VR, just a Mac/PC convolution viewer for single images
      • a quick search of Apple's website doesn't reveal an SDK for controlling or embedding the QTVR player
      • cylindrical, only 2 degrees of rotation (pitch and yaw)

Have any information on this subject?  Let me know!