Loading Profile...
Create synth from HD camera footage
I want to start a discussion about an idea I had on how to build nice synth quickly and efficiently by leveraging the sheer power of an HD cam as a super fast continuous picture camera.
The idea is to use a good HD consumer camcorder (like my Canon HV20) to record progressive video of a scene while looking around from multiple location in the space. A good tripod/monopod should be used to get stable video from the scene.
You then import this video from the camera and then export it to a folder as 6 pps (pictures per second... 6 is a nice number because it divide the 24 frames per second of my HD footage quite evenly) jpg (you can do this via Quicktime movie to picture function). Simply navigate the 100s (or 1000s) of individual pictures to select the ones you think would by "synthy" and use that to build a nice synth of a scene. One could go crazy here and actually use the 1000s pictures to make the synth... ouch!
What do you think of that? The best would be if photosynth could accept the raw (or h264 converted) footage and analyse it on it's own to pick the best frames based on acceptable pictures differenciation and then use the pictures of what is relevant from it. This way it would save all the work of exporting and selecting.
Photosynth could even slice the inter picture to picture video in small chunk to show nice transitions from selected pictures to pictures. Essentially creating the illusion of walking in the picture path in 3D.
The idea is to use a good HD consumer camcorder (like my Canon HV20) to record progressive video of a scene while looking around from multiple location in the space. A good tripod/monopod should be used to get stable video from the scene.
You then import this video from the camera and then export it to a folder as 6 pps (pictures per second... 6 is a nice number because it divide the 24 frames per second of my HD footage quite evenly) jpg (you can do this via Quicktime movie to picture function). Simply navigate the 100s (or 1000s) of individual pictures to select the ones you think would by "synthy" and use that to build a nice synth of a scene. One could go crazy here and actually use the 1000s pictures to make the synth... ouch!
What do you think of that? The best would be if photosynth could accept the raw (or h264 converted) footage and analyse it on it's own to pick the best frames based on acceptable pictures differenciation and then use the pictures of what is relevant from it. This way it would save all the work of exporting and selecting.
Photosynth could even slice the inter picture to picture video in small chunk to show nice transitions from selected pictures to pictures. Essentially creating the illusion of walking in the picture path in 3D.
I’m happy
Follow this discussion to get notifications on your dashboard.
-
Inappropriate?Oh, one thing that come to mind is that since photosynth use picture still at it's heart it will be essential to crank up the shutter speed to a high enough value (like 1/500th +) to ensure that the captured frames are not blurry and are actually as sharp as can be. Cranking up the shutter speed will also benefit photosynth by increasing the depth of field such that more things are in focus.
Another thing to note is that you will want to use an HD camera that can do "progressive" capture like the HV20. Those Sony that only 1080i at 60fps are no good for this as you will get pictures with a bunch of interlace artifacts... unless you export as individual half frames and resize to full frame at the cost of loosing half the resolution ;-( -
Inappropriate?This sounds like a great set of ideas, and I encourage you to try some of them out and let everyone know how it goes.
There's actually already a related question/discussion going on under the topic of "Why not just take a video?"
Here at Live Labs, we've experimented a bit with creating Video synths. To fully automate the process, it's useful to have at least two elements: figure out when video frames overlap by about the right amount (say 75%) and select frames that are less blurry.
Our work is still in the research phase, but if it pans out, we'll add it to the list of potential future Photosynth features.
P.S.: Drew Steedly wrote a nice research article on how to automaticallly stitch panoramas from video. You can find it by searching for "Steedly Efficiently registering video" on the Web.
I’m content
-
Inappropriate?Interesting abstract. Could apply to how photosynth could leverage video key frames to do the matching on it own and then use photosynth spacial algorythm to produce the 3D scene from video in an efficient manner. This would be great. In the mean time a human can do the heavy work by manually selecting picture frames from the video and use that to do the synth... minus the possibility of the picture to picture animated transition ;-)
This would essentially allow photosynth to recreate a 3D scene from a good HD video source. -
Inappropriate?OK, I created my test synth today from a 9 minutes sequence from my backyard. I had set the camera in 24p mode with a fixed 1/500th shutter speed. That gave me a 13,000+ potential pictures to build my synth from... So clearly this is too much to handle.
For those curious here is my workflow on a Mac to publish a synth via a PC:
1. I imported the video from the camera over firewire by using DVHScap from the Apple Firewire SDK
2. I converted the raw .m2t file to something Quicktime can handle by using mencoder (google for it if you want) with the following command:
./mencoder synthtest.m2t -mc 0 -noskip -fps 30000/1001 -delay -0.222 -oac pcm -vf pullup,softskip,harddup,scale=1280:720 -ofps 24000/1001 -ovc lavc -lavcopts vcodec=mjpeg -o synth.avi
Notice that I reduced the resolution from 1080p to 720p to save some space... might try it later will full res to see if it will improve the synthyness.
3. I opened the resulting synth.avi in Quicktime Pro and used the export function to output as "Movie to Image Sequence" as .jpg at 1fps (1 image per second of source video). This produced 585 pictures in a folder.
4. I select them all and viewed them in "Preview" to delete the pictures I tought where not needed until I got down to about 288 good ones.
5. I select the 288 pic and copied them in a new folder that I putted on a network share.
6. I go to my Vista MCE PC where I openned Photosynth and created a synth from the pictures in the shared folder.
7. Wait about 30 minutes for syhthing to complete
8. Look at log to see if all is good... This particular test is apparently only 47% synthy ;-( But I know it is 100% sinthy ;-) so somewhere the algorythm is not registering things correctly.
9. Wait for synth to show up on web site. It's been 1 hour and I am still waiting.
Note: Since I am generating my pictures from a movie sequence there is not exif data to get the camera info from and Photosynth spit a message about that in the log for every picture... but apparently can do without it. Cold this be a problem on the web site? Could this be why my synth don't show up?2 Comments
Add a comment
-
From David Gedye, Official Rep: "The synther currently uses up to 1.5 megapixels of the photos it is given, so any resolution above that won't change your synthy score." -
This is really interesting. So essentially the HD resolution of 2 megapixel is just fine for creating synth. I guess you are limiting the resolution to 1.5 mega pixels to reduce computing time on the client. Striking the right balance between accuracy and CPU time is a real science.
Is there a technical reason for not allowing one to do a synth in smaller batch of say... 5 pictures, and then adding 5 more, then five move when computation is done for the previous five? Is it something doable? -
Inappropriate?OK, I think I have isolated the issue with my HD based synth. I had to add some more EXIF data to each jpeg before creating the synth. Essentially I took an actual picture in HD format with the Canon HV20 and extracted the EXIF data from it using exiftool.
I then apply this same info to all my jpeg using again exiftool:
exiftool -TagsFromFile canonhv20.xmp <dir>
rm <dir>/*.jpg_original
I then use those files to create the synth. With that no more "Publishing bug".
You can see a sample of a synth created this way below:
DeckHD1080p: Made by outputting at 1 picture per sec.
ToyHD1080p: Made by outputting at 3 picture per sec... very detailed.
If only I could now keep adding to my synth time after time I could build a very high resolution synth containing 1000s of pictured from HD capture.
Photosynth allow you to turn you HD footage into a 3D world. While standing in photosynth at the HD camera location and doing a left to right pan will give you the impression of being virtually behind the camera. Very cool.</dir></dir>
I’m happy
EMPLOYEE
