What should I use to add 3d objects into a video clip?

February 8, 2005 6:53 AM

VideoProductionFilter: I’d like to experiment with adding 3d objects into a video clip. What should I use? [mi]

For example, let’s say I would like to shoot some video of my backyard and digitally insert a 2001 monolith into the video. I would want to animate the monolith to make it appear to maintain perspective with camera movements. (Ultimately, I would like to add 3d objects that have transparency and possibly text objects.)

I’m working with Windows (xp) and I’m familiar with Adobe Premiere and Adobe After Effects but have never worked with 3d models or animation tools. My budget for new tools would probably max at $200 or so. As far as quality goes, “not bad” would be acceptable and “pretty good” would be fantastic. Any hints to get me started?

posted by sexymofo to computers & internet (14 comments total)

On a $200 budget, I'm not sure what you can do. I'm sure you can rig something, but I'm only used to working on high-end school equipment that I could never afford (my tuition has to go somewhere, right?).

Check out 2d3, they make some high-end 3d tracking software. It might not be in your range, but you could get some ideas. Also, run a google search for "After effects 3d motion tracking" and see if you can find a cheaper plug-in that might do what you're looking for. I'd be surprised if you could do it without an external 3d program.
posted by still at 7:04 AM on February 8, 2005

You should be able to do this in After Effects, but I'm not sure how. It seems like there are actually whole books about doing these kinds of effects in AE.
posted by spaghetti at 7:28 AM on February 8, 2005

In my experience AE does 3d compositing (this goes behind this, the camera moves from far to near) but not 3d modeling of objects (this is a cube, now lets turn it).

Maya has an educational download for free, but throws a watermark on the stuff you create.
posted by mzurer at 7:52 AM on February 8, 2005

I wouldn't spend money on this stuff until you've tried some of the free possibilities out there to see if it's the kind of thing you want to dig into. Also, keep in mind that the guys at Weta, when working on LOTR, found it easier to remodel, texture, animate, and render an entire mountain than try to match some of the more obscure camera tracking movements for compositing.

For modeling, I highly recommend Wings3D. It's free, fast, and I've seen relative newbies turn out recognizable models within hours of installing the application. Some basic knowledge of working in 3D space is required to operate the software, though.

For quality rendering (and modeling) with animation capabilities, there is Blender3D, which is also free. The interface is pretty obscure, though, so there is a definite learning curve involved.
posted by xyzzy at 7:52 AM on February 8, 2005

I guess the easiest and cheapest way I can think to do this would be to:

1. create the monolith in 3D studio (or similar; v. easy)

2. animate a camera in your 3D program with movements that mimic your footage (this will drive you insane)

3. Render with a flat green/blue screen to the same video settings as your original footage (easy enough)

4. put it together in after effects, render.

So, as far as tools, you're pretty much set. I don't know of any good free 3D software, but check out some of the computer hobbyist magazines you can get hold of -- many of them sometimes give away old versions of software as a promotional deal. If you find one that does, or has in the past, consider buying a back issue.

When you shoot, either keep the position static and resort to panning only, else figure out a way to rig up a stedi-cam system.

But, a word of warning... don't expect too much... if you want it to look any good, you're going to have to make sure that the camera movements in your 3D program are as close as possible to the real camera movements.

Without any sort of sophisticated movement capture hardware/software, you're going to be doing it by trial and error... and the subtle adjustments/rendering time will probably drive you barmy.

THEN you've got to consider how the lighting in the real scene was set up, and then mimmic THAT in your 3D program.

THEN you better hope that you don't want the monolith behind any trees/other objects in your back-yard.. then you're going to be doing some frame-by-frame editing, which is tedious.

I tried something similar for a project at uni... and the timeframe I had (2 weeks) was nowhere near enough to get it looking as good as I wanted. YMMV, of course.


good luck to you man!
posted by cheaily at 7:54 AM on February 8, 2005

After Effects doesn't really handle 3d tracking that well, so you could try Pixelfarm, they have a product called PFHoe that seems to be aimed to the hobbyist market that could be within your price range. You could also download the Maya PLE (Personal learning addition) and find a post house that can render 3d stuff it out for you on off hours. I'd say pixelfarm and then use after effects to composite it. Most of these type of products have pretty good demos, so try a few.
posted by phirleh at 8:08 AM on February 8, 2005

If you want to learn about this kind of thing, you should really join the Pixelcorps. It's not like other training toward this kind of thing- it's very hands-on hand has active teams working on projects just like this. You even get time-limited versions of lots of apps... I'm using a motion-tracking app right now on some HD footage... Unfortunately I can't say more in a public forum. Check it out here: http://pixelcorps.com/pxc_winter_2004.html
posted by fake at 9:47 AM on February 8, 2005

The terms "matchmoving" or "camera tracking" will bring up tons of stuff on google. The good news: there's a whole bunch of companies that make software for doing this very thing. The bad news: they ain't cheap. Here's a couple of pages with links to matchmoving products (link, link). If you click the links and see where some of the projects that the products have been used in, you see why they charge what they do...'cause they can : \ Based on your budget, methinks you might be out in the cold on this one.

To add on to cheaily's answer, one way to do step 2 (matching the 3D camera movements with your real-world camera) is to add "tracking points" to the scene. Some of the previously mentioned packages can do this after the fact by looking for high-contact areas of the scene; another way is just get some big white stickers and put them on the ground in a pattern, say a triangle. Some 3D packages have components that can take this footage and determine the motion of the camera based on the motion of the tracking points. If your camera is on a tripod and the only motions you make are pan/tilt (while keeping the tracking points in the scene), you make matchmoving a whole lot easier, since you've eliminated the variable of depth, for the most part.
posted by bachelor#3 at 10:51 AM on February 8, 2005

Here's how these things are done.

Usually, in a green room, dots are added to the walls. (this is a virtual set)

You go in and shoot an actor responding to the monolith.

Now, utilizing 3d tracking software (which is often in 3d software) you build the monolith. You tell the tracker to create and move a camera based on these points (on the walls).

The "virtual" camera moves based on the tracked dots around the 3d monolith.

You now use a compositing program (After effects, combustion, Motion) to put the real person in the green room over the monolith. Then you key out the green...and the camera around the monolith matches the move of the real camera.

And then you put a background (a la matrix)


So:

Actor in a green room (goes above)
3d element with transparency (goes above)
Background.
------

Your issue is a bit more difficult:

It's easy to do this (perspective, etc) when the camera doesn't move.

You have the actor work and react to something that's not there (give him a point to focus on). Then you can in 3d make the monolith + shadow.

Use the compositing software (often editing software has some of this ability) to put the 3d item on top of the video background.

Nope, the character can't walk in front of the object easily.

And cheap? nope.
posted by filmgeek at 10:57 AM on February 8, 2005

As still said, 2d3's boujou video analysis software is what you need for doing this. I went on a company tour there a couple of years ago and it became fairly clear that what you're asking to do is one of the hardest problems in video and that their software is pretty special (it doesn't need special markers in the scene). They offer a free trial version if you're just messing about though.
posted by cillit bang at 10:59 AM on February 8, 2005

Thanks everyone, for the great info. More then enough to get started playing around.

The actual effect I'm going for is far simpler than the example I used. Only zoom and pan camera movements, the 3d object won't interact with the live environment and vice versa (no need to match lighting effects, etc.) and it only has to maintain the perspective "pretty close."

But again, thanks for your effort. When I'm ready to do my version of "Sky Captain", I'll know what it takes.
posted by sexymofo at 11:09 AM on February 8, 2005

You'd still have to match the lighting, as the object needs to look realistic in context.

The only time I ever did anything like this was a stupid little video where a CG bird was on a guy's shoulder. I did the bird in 3DSMAX, had it cast a shadow on an invisible cylinder under its legs, then rendered the thing out and composited in After Effects. The guy's shoulder was moving a little, so I just manually moved the bird around in After Effects. Turned out pretty well, considering.
posted by neckro23 at 11:19 AM on February 8, 2005

Found this on popgadget.com:

Total Immersion specializes in augmented reality technologies and its software, D’Fusion, allows for integration of 3D objects into real time video. To get a better idea of how this works check out the video.

Total Immersion
posted by spicynuts at 12:39 PM on February 8, 2005

...by looking for high-contact areas of the scene...

*sigh*...and of course, by that I really mean:

...by looking for high-contrast areas of the scene...

two examples of these are the aforementioned boujou as well as Realviz's product, MatchMover Pro.
posted by bachelor#3 at 5:07 PM on February 8, 2005