What happens when, and for how long?

April 24, 2006 10:46 AM

Are there any good tools out there for analyzing binary sensor data? I would like to create something like a timeline of data.

I am running a fairly large monte-carlo simulation in which the output of around 200 binary sensors is recorded. Most of the time, all of the sensors are "off." I need to better how the system goes from being all "off" to having some sensors "on."

Previously, I would visually grep through the SQL sensor logs and observe sensors turning on and off. I would like to create a visual representation of the sensors over time so that I can see the shape of the sensor clusters as they turn on and off.

It looks like it might be possible to trick Excel into doing this by explicitly computing the duration of the "off" events, and creating a stacked bar chart. Excel tends to fall over when dealing with really large data samples, so I would prefer to avoid it.

Is there a better way to do this with a tool like GnuPlot or R? I have decently powerful Windows, Linux and OSX machines at my disposal. I am capable of writing code, but would prefer to avoid doing so if this problem has already been solved.

To clarify - by "binary sensor data" I mean that the sensors are either on or off. The actual data is a SQL table with columns for sensor number, start time (in DMY/HMS plus another column for milliseconds) and end time (same format as start time) among others. The underlying data is actually not binary. Sensors take on a range of values, but I am currently examining the case where a given sensor crosses a threshold - yielding binary data.

posted by b1tr0t to computers & internet (7 comments total)

It sounds like a good candidate for custom code, but the nature of the problem is fairly simple. I don't know of anything right out of the box for general purpose data visualization on this particular type of data.

Is the sensor array something that can be graphically depicted in a one or two dimensional form? If it can be shown as a one dimensional string of data, then a time series of the data organized into a waterfall plot would seem to be a good visualization. If two dimensional, then modelling the data as a surface, with three axes labeled time, sensor, and on/off state might also yield a good visualization.

Care to share what the data represents or what tools you are familiar with?
posted by FauxScot at 12:45 PM on April 24, 2006

The data represents the results of a windowing function that enables any of around 200 commodity trading strategies.

I would visualize the data as follows:
1. Along the Y-axis is a list of the 200 sensors
2. Along the X-axis is time
3. For each datapoint, a rectangle is drawn. Its height is always the unit height of a sensor. The width is determined by the starting and ending time.
4. Over time, any sensor can cycle between on and off MANY times. If the sensor only ever went on once, this would be easy to model in excel.
5. The typical duration of a sensor's on period is around 2,000 to 10,000 ms. I am working with about three months of simulated data, but could work on individual days of data. Clearly real time zooming and panning are critical features.

I'm downloading the DirectX SDK now to see how difficult this will be to implement in Direct3D.
posted by b1tr0t at 1:41 PM on April 24, 2006

Your plot sounds like a piano roll plot. At least that's the visualization that comes to my mind first.

200 'Notes' on the vertical axis, time on the horizontal, black=on/white=off.

Doesn't sound all that hard to code with a single control specifying time and a fixed width window horizontally.

What is your take on the SDK?
posted by FauxScot at 7:46 PM on April 24, 2006

check this out, b1tr0t

http://www-camil.music.uiuc.edu/software/cm/installed/cm-2.1/doc/plotter.html


look about 1/2 way down the page at the second plot, not the first one....
posted by FauxScot at 7:50 PM on April 24, 2006

FauxScot - that looks interesting, except that it doesn't seem to be available beyond UIUC.
posted by b1tr0t at 8:30 PM on April 24, 2006

I know. The reason I sent it was that it looked like the display type you were seeking. When I code, 90% of my time is spent figuring out what I want things to look like, and I thought it might be helpful to see a display sample. Is that what you had in mind?
posted by FauxScot at 2:30 AM on April 25, 2006

It is mostly what I want, except that I need vastly more time domain granularity than a music app would provide. I could code this up in Direct3D, but I have enough paying projects that this one will have to go to the back of the pile.
posted by b1tr0t at 10:13 AM on April 25, 2006