Comments on: How to serve HTML out of a compressed archive?

Question: How to serve HTML out of a compressed archive?

Galvatron — Sat, 17 Nov 2007 09:41:31 -0800

I want to serve lots of different static HTML files out of a single compressed archive, to save disk space on the server. Is there a quick and easy solution for this?

So I got an iPod Touch recently, and could not restrain myself from running the jailbreak on it. There are some nifty third party apps out there, but I'm pretty sure the killer app for me is being able to browse a Wikipedia snapshot without requiring wifi.

Due to the disk space limitations, I'd like to serve the snapshot out of a compressed archive. Compressing each html file individually would work, but it's relatively storage-inefficient, so I'd rather have everything wrapped up in a big gzipped tarball or similar.

Is there an Apache or lighthttpd module that would make this project easy? Or is embedded OS X capable of mounting an archive through a VFS/loopback device? The fallback solution is that I write a cgi script to do the decompression, but I'm looking for something better.

By: Nelson

Nelson — Sat, 17 Nov 2007 09:45:35 -0800

I don't know about the module, but the archive format you want is Zip or something similar. Zip files allow random access to each file instantly; a gzipped tarball requires you scan the entire archive to find where the file you care about starts.

Douwe Osinga did something similar in 2005 with a smartphone: he has some notes online. Sounds like he used Tomeraider, an ebook reader. Just over a gig for Wikipedia in their format.

By: zsazsa

zsazsa — Sat, 17 Nov 2007 10:16:20 -0800

If the iPod touch can mount .dmg image files, those can be made as compressed. The files will just show up as normal, but will be compressed inside the .dmg.

By: cmiller

cmiller — Sat, 17 Nov 2007 10:17:31 -0800

You could do it in not-a-lot of Python.

import zipfile
import BaseHTTPServer
z = zipfile.ZipFile("wikipedia.zip", "r")
...
# listen for requests, for each,
page_source = z.read("path/PageName")
output.write(page_source)
...
z.close()

... or something very close to that.

By: Galvatron

Galvatron — Sat, 17 Nov 2007 10:25:49 -0800

Interesting thought, cmiller; I hadn't considered using Python's built in server. That really would be not-a-lot of python.

By: a_green_man

a_green_man — Sat, 17 Nov 2007 12:14:21 -0800

It'd be pretty cool if you tossed this up on Projects when your done. I'll have one of those gizmos ~Jan.

By: Galvatron

Galvatron — Sat, 17 Nov 2007 13:45:57 -0800

If the project turns out ok then I'll be sure to post a write-up, a_green_man. I wrote a short Python script based on cmiller's suggestion, which pretty well resolves my original question... but now I have to deal with the logistics of decompressing and rearchiving Very Large Quantities of data, and possibly figuring out a strategy to prune it down to something more manageable.

By: Mwongozi

Mwongozi — Sat, 17 Nov 2007 14:08:17 -0800

You could save space by only storing the Wikipedia CD collection.

By: Galvatron

Galvatron — Sat, 17 Nov 2007 14:30:04 -0800

Mwongozi, I did see the CD collection, but I don't think such a limited set of articles would satisfy me. I want to carry the Hitchhiker's Guide in my pocket. Possibly bearing a wallpaper image that says "Don't Panic" in large, friendly letters...