Video Archiving Server

May 7, 2007 9:50 AM

I'm looking for a server solution that I can stream video to be archived (in real time) for access at a later date. More after the break...

I'm trying to find an server solution that will take on multiple video streams and archive them. I also am going to want to access these archives and pull a video clip by date/time ranges for download. This is going to be all done over the Interweb.

Also, proprietary camera/PC software solutions are not ideal. We're talking enterprise level solutions here.

Thanks in advance for your ideas!

posted by chrisfromthelc to computers & internet (13 comments total)

I can't help with prepackaged solutions, but I can imagine such a thing being whipped up using mplayer as the active video-handling ingredient and some php/perl/ruby/python scripts munging the metadata into a db.

You don't specify if the streams are local or remote, but mencoder can handle both. Matroska is pretty much designed for the fast-seek you're looking for and it's a nice open archival format. There's definitely some gruntwork to be done to make it all work, but it should be in the realm of a contract job, or even an open-source one, though the timeline there is a bit less concrete.
posted by Skorgu at 10:18 AM on May 7, 2007

I have not come across anything like a turnkey solution for this.

Typically you would have three components: storage, a web interface to access those files, and a player with which to view them. Storage is easy enough and any kid can come up with a directory parser for the web. The player itself is an unknown as most of what i deal with is in flash/flv, but I'm sure there's something out there. Ask people in the porn industry, I'm sure they'd have some pointers.

"Enterprise level solutions" usually require money, so be aware of that.
posted by rhizome at 10:19 AM on May 7, 2007

On poking around some, this page has a good small-scale solution outlined, but it looks like flv streaming isn't too hard either. Scaling this is likely to be a function of programmer time and system performance, either of which might make this uneconomical.
posted by Skorgu at 10:40 AM on May 7, 2007

For an enterprise setup, you would not use one server, rather you would have multiple servers for re-encoding, storage and delivery.

You could use QuickTime Broadcaster for encoding your feed, and Quicktime Streaming Server or Darwin Streaming Server.

The Mac OS X Server product includes database, logging and authentication components for managing and accessing media files. This is sold separately or bundled with the XServe server product.
posted by Blazecock Pileon at 11:03 AM on May 7, 2007

@Skorgu:

The streams are going to be delivered in motion JPEG, MPEG4, or H.264 format via IP, so remotely. I'm looking at a camera solution that will stream MJPEG and either MP4 or H.264 simulataneously over IP.

@rhizome:
I'm not scared of spending money on this project. FLV was my first choice as far as a playback module. I'm wondering how the porn guys archive the video from thier webcam live shows, which, essentially, is what I'm wanting to do on a replicated-many-times-over scale.
posted by chrisfromthelc at 11:10 AM on May 7, 2007

@Blazecock:

Yes, I am aware of that. ;-)

I've thought long and hard about QT, but can it stream and archive the video in realtime,...or at least archive non-stop in realtime? That feature alone would solve many of the problems I'm facing with this project.
posted by chrisfromthelc at 11:12 AM on May 7, 2007

I've thought long and hard about QT, but can it stream and archive the video in realtime,...or at least archive non-stop in realtime? That feature alone would solve many of the problems I'm facing with this project.

QuickTime Broadcaster will save to an H.264 file, which you redistribute with a streaming server.
posted by Blazecock Pileon at 11:46 AM on May 7, 2007

I'm reading a bit into your requirements, but it sounds as if receiving and archiving the videos in realtime is your first priority, and viewing pieces of them is secondary. If this is the case, I would be looking at a video archiving system, full stop. As long as the archive format is something easily manipulable you should be able to stream it from any of the commercial or opensource streamers above.

If you can receive H.264 streams over IP, mencoder can simply dump the stream to a timestamp-named file, and any streaming software should be able to pick that up and redistribute it as needed. One box just dumps streams onto a fat RAID array, then as many streaming servers as you need can pump out the streams.
posted by Skorgu at 12:08 PM on May 7, 2007

@Skorgu:

Yes, that's EXACTLY what I want. ;-)
posted by chrisfromthelc at 12:34 PM on May 7, 2007

Skorgu,

Does mencoder support chopping up the stream for downloading?

Ex: An event happens at XX:XXpm for 5 minutes. I need to download that section of video.

Is this (elegantly) possible?
posted by chrisfromthelc at 12:43 PM on May 7, 2007

*puffs cigar, flicks ash*

I love it when a plan comes together.

The way I imagine this working is a well-connected server running a little program that, every (say) ten minutes starts an mencoder process, dumping the stream that runs for eleven minutes. That way you have a minute of overlap between processes and your worst-case failure is ten minutes of lost video. Obviously substitute your own numbers. Depending on details this could be one computer or ten.

Once you've got the video stored somewhere, retrieving it is essentially a solved problem. I'd have a master index database of all saved streams and a streaming server that knows how to seek inside them. This sounds like the kind of thing that commercial software does well, seeking to arbitrary parts and then chaining consecutive "episodes" automatically. If you want to roll your own it should be pretty trivial with the available streaming servers.

The key part, I think, is making sure that your archival storage is very easy to seek. H.264 sounds like it fits the bill based on Blazecock Pileon's assertion about QT streamer, but there's also Matroska and if you've got plenty of space, MPEG2. Once you've got a pile of indexed, seek-able video files any two-bit coder can hack you up a web-based playback system based on it.
posted by Skorgu at 1:17 PM on May 7, 2007

Yep, that sounds about right.

I'm partial to H.264 because of space and commonality being in my favor.

I'm just trying to come up with something that is "hands-off" on the client end as far as any machinge being set up on thier end. I just want to give them a camera, and let them plug it up.

Sounds like you've done this before, Skorgu.
posted by chrisfromthelc at 1:35 PM on May 7, 2007

I've actually only done the end-result, youtube-ish part of it, and then only on a much smaller scale than it sounds like you're talking about. It sounds good on metafilter, but the real world makes it all so much more complicated :)
posted by Skorgu at 1:47 PM on May 7, 2007