Left, right, right, cry, steal Mom's glasses and throw them on the floor, left, left, right

September 12, 2011 12:54 PM

What data-coding problems are sufficiently like my data-coding problem to have built good software solutions to handle it? Or can I write something myself?

I'm a developmental psychology grad student who does some work using a paradigm called Preferential Looking with infants and toddlers. The very short version of this task is that you plonk the kid down in front of two tv screens, with something potentially interesting to look at on one/both sides. You wind up with a video of the baby's head, with the baby looking right, left, right, looks down and plays with her shirt,left again, right, left. The measure we use is how many frames/slices the child is looking left, right, or away during a certain period of interest.

To code this, we have been using SuperCoder, a cute and serviceable little program written for exactly this purpose. It displays a digital video and lets you step through it, and hit a key whenever something happens (baby starts looking left, baby stops looking left.) It creates a textfile that has every keycode with the timestamp, which is enough information for us to retrieve the lengths of the looks.

This kind of coding is always slow going, but I've been wondering if there are better (or even just different) tools out there. My problems with this bit of software are very 'picky', but this does seem to be the kind of task where little changes make a big difference in usability and speed. The biggest issue is occasional crashiness, and a few bugs (it's easy to accidentally skip all the way back to the beginning of the clip, and hard to find your place again, for instance.)

Other things that I wish I could change, and would love to find in another piece of software:

- It's cumbersome to switch between real-time and frame-by-frame viewing.

- You can't really code timestamps "live" (as the video plays), which slows coders down a good deal.

- ...It would be especially awesome if there were a half-speed playback option.

- Probably the biggest thing: you can't type in a specific frame/timestep to skip to - you can only scroll the movie bar, and you don't get an updating readout of what frame you're on.

Okay, so after all my nitpicky software requests, I really have three related questions.

What other specific data-collection/coding problems are like my problem? If I knew, for instance, that people needed to do things like accurately record which of four quadrants their mouse is in as it races around its enclosure, then I might be able to find out what solutions they use. Assume that I am looking for a time grain-size of at least 1/24 seconds, but no more than 1/100 seconds.

What other software solutions do you know about that do this? Free or cheap is great, platform-independence is awesome (Mac availability is the most useful, though.)

Can I build this myself? I'm reasonably proficient in python, html, and R but know next to nothing about things like digital video encoding.

(Side note - these are videos of people's babies we're talking about, so putting them on the internet and crowdsourcing our coding is right out. And for passing psycholinguists/dev. psych folks, you can assume that "just use an eyetracker!" is something I hope to get to eventually, but not yet...)

posted by heyforfour to technology (4 answers total)

This really feels like an HCI issue for the coder to me.

If there were a way to plug in a USB game-pad and map the controls so that the the D-pad portion does 'looks left' and 'looks right', etc., while setting some other buttons to play, pause, step forward, step back, scrub forward, scrub back.... I think you'd have the fastest method of input possible, short of an automatic eye tracking unit.

Then you're just mapping the buttons to what I'm hoping is some sort of existing interface. If you're on the Mac, USB Overdrive might be of some help.
posted by Wild_Eep at 1:35 PM on September 12, 2011

If you want to outright replace SuperCoder (which looks like a souped-up version of QuickTime Player to me) you might be best by pulling the movie into any editor that supports 'bookmarking' or 'flagging' a key frame, then having a script output a file based upon the locations of the bookmarks/flags.
posted by Wild_Eep at 1:38 PM on September 12, 2011

This would be a good target for automatic classification via machine vision / machine learning. Basically, you would train the computer with samples of "left looking" and "right looking" frames, and it would classify the rest. If you have a nearby computer science department, recommend emailing professors in AI and specifically machine vision. They may be able to put you in touch with students who would do this as a class project.

As for writing a UI yourself, it's probably pretty easy to do in Python. You shouldn't need to know anything about video encoding, as there are good open source libraries that will do all of that for you. I'd approach this by seeing if there are any open source Python media players you can bastardize, something like this. Worst case, you can break a video into image files for each frame, and image files are very easy to work with.
posted by qxntpqbbbqxl at 2:03 PM on September 12, 2011

Have you looked into Quartz Composer?

It is a graphical programming language for manipulation of audio, video and other input (from your test subjects, say).

You string together widgets with "wires", which operate from left to right. In other words, you start with your input on the left, and as you follow the wires to the right, the input is processed with widgets of your choice.

One such widget runs your JavaScript code of interest, for example (you set the inputs and outputs).

Other widgets perform specific signal processing tasks, such as grabbing signal from input devices (video, keyboard, mouse, etc.) or kernel convolution operations, etc.

It looks simple, and it doesn't get as much attention as the Xcode package it is part of, but it is actually a very powerful signal processing tool.
posted by Blazecock Pileon at 4:42 PM on September 12, 2011