Crazy Pulsating Color Filter

February 23, 2006 8:30 PM

This idea for a visual effect has occured to me, and I want to implement it, just to see what it's like. I know Perl and C/C++, but I am heavily biased towards a *nix environment. Description of video transformation inside...

Basically what I want to do is vary the saturation of the primary colors in video frames based on the amplitude of music that would accompany it. So for instance, while the soundtrack was silent the video frames would have no saturation. Let's say I split the audo frequency in midrange, bass, and high end corresponding to yellow, red, and blue respectively. So every time the kick drum is hit, the red saturation peaks for a moment.

How can I implement this? My first thought is to somehow generate average amplitudes of the audio at a sample rate equal to the video frame rate. Then split the video into individual frames and use something like the GD libraries to tweak each frame indivdually.

But that seems like re-inventing the wheel several times over. Is there some existing software that could be extended to do this?

posted by phrontist to technology (7 comments total)

This is exactly the kind of weirdness that MSP and Jitter were designed for.
posted by xil at 8:34 PM on February 23, 2006

You know, I actually have Max/MSP installed, does anyone have reccomendations for learning how to use it...
posted by phrontist at 8:39 PM on February 23, 2006

Just FYI computer screens and graphics in general use the Red/Green/Blue (RGB) color space. Yellow doesn't enter it unless you're printing where CMYK (Cyan, Magenta, Yellow, Key aka Black) is preferrable.
posted by Ryvar at 8:40 PM on February 23, 2006

Phrontists - they come with some MASSIVE pdf files filled with tutorials. (if not theyre on the cycling 74 site). Get reading!
posted by atom128 at 8:53 PM on February 23, 2006

I've never used Max/MSP, but the people who are into it seem to like doing things like this. Jitter, in particular, is an extension designed for working with matrix data (like video frames).

From your description of red, yellow, and blue as "primary colors," I suspect that you may be new to graphics programming. Those are traditional names used in subtractive color (ink on paper); the primary colors in additive color (projected light) are red, green, and blue. This wikipedia article about the RGB color model may be of interest.

The term "saturation" typically applies to the HSV (hue, saturation, value) color space, and can be thought of as the vibrancy or intensity of the color; low-saturation colors are pastels (zero saturation is grey), while high-saturation colors are bright and bold. But saturation in HSV terms applies to the color as a whole, while you're referring to it as a property of individual color channels, so I'm not quite sure what it is you have in mind. When you say that "the video frames would have no saturation" do you mean that the video frames would be solid white, solid black, or colorless (greyscale)?

Varying the overall intensity of the image in response to the audio amplitude could certainly be interesting; the image would go from grey during the quiet parts to bold and colorful during the loud parts. But you want to vary the level of individual color channels based on certain frequency ranges. Sounds like you want to map the actual color levels through an exponential function. Map the audio level to the range 0..1, then subtract from 1, and use that as the exponent. When the audio signal is silent, the image will display unchanged; when the signal peaks, that color channel will flood. You can get an idea of the effect by playing with the Curves dialog in Photoshop.
posted by Mars Saxman at 9:01 PM on February 23, 2006

You could use Adobe After Effects and a rather cool third party commercial plugin, Trapcode Sound keys, to process video footage in exactly the way you describe, isolating amplitude in specific frequency bands and using it to generate keyframes for any filter in After Effects, which includes Hue/Saturation - but not in realtime, it would have to be rendered. For realtime processing of this kind, I suspect Isadora might be the ticket.
posted by dbiedny at 9:19 PM on February 23, 2006

If you happen to have a Mac lying around, this might be doable using the Quartz Composer thingy. (I haven't used it, though.) That would run in real time, but I don't know if it can do the kind of color separation you're after.

I don't think it matters that red-yellow-blue isn't a set of primary colors, btw. Consider the 2-dimensional chroma space as a plane (a disk, say), with saturation=zero at the center and color around the edge. Any pixel value can be represented as a point in this plane, plus an altitude corresponding to luminance --- this is the standard HSV color cylinder. So you can just divide the color plane into arbitrary pie-shaped regions, corresponding to your sound filter bank outputs I guess, and scale the radii of colors accordingly. Smoothly interpolating at the boundaries between sectors wouldn't be too hard, conceptually, but might require more math.
posted by hattifattener at 10:52 PM on February 23, 2006