Join 3,380 readers in helping fund MetaFilter (Hide)


Need help solving a zip archive timestamp & GNU make interaction problem on Linux, MacOSX, and other systems.
August 21, 2010 8:53 AM   Subscribe

We ran into the following conundrum while distributing source code for some open-source software we produce. Our code is pretty straightforward C++ stuff with makefiles designed for GNU make. We distribute the package as a zip archive. Works on most systems (Linux, Windows, MacOS, FreeBSD and others). Now, zip stores and unzip restores the file timestamps. Problem is, if someone downloads the archive in an earlier time zone than the one in which it was produced, the unarchived files have a timestamp in the future.

For example, if I create the archive on the east coast and shortly afterwards, someone on the west coast takes it and unpacks it, the files in the archive will have timestamps roughly 3 hours into that person's future because the files were last modified in my timezone (roughly 3 hours ahead of theirs). This confuses make, and the builds fail (in our case, because make's implicit rules do undesired things, although the implicit rules issue could be resolved) because the timestamps of object files it builds are older than the timestamps of the files they're based on (because they're from the future). I tried the -X option to zip, but it doesn't change the file timestamps (both in MacPorts' most recent zip and /usr/bin/zip on MacOS 10.5.8).

How do we work around this problem?

The only things we've found so far is (1) touch all the files after unpacking them or (2) wait 24 hrs before distributing the archive. As far as (1) goes, we don't want to have to tell recipients to do that, because most won't, or probably will miss that portion of the instructions. We've been basically relying on (2), but I'd like to solve this problem in a better way.
posted by StrawberryPie to Computers & Internet (14 answers total)
 
You can use touch to set the modification date of any file to an arbitrary time. Seems the simplest solution would be to set all of the modification times to 24 hours in the past before creating the archive.
posted by grouse at 8:59 AM on August 21, 2010


This is one of the reasons why most open source projects don't use zip files but tarballs (.tar.gz or .tar.bz2) to distribute source. In addition to not being brain dead about time zones, tar preserves file modes which is necessary if you use autotools and/or have executable build scripts.
posted by Rhomboid at 9:52 AM on August 21, 2010


If you decide to package a .tar.gz, you can always suggest that Windows users use 7-Zip to extract it. If they're comfortable enough with using make, they'll be comfortable enough with using .tar.gz.
posted by kdar at 9:58 AM on August 21, 2010


Is there not some way to have the compiling system "touch" the files before using them in the compile?
posted by Biru at 10:19 AM on August 21, 2010


To clarify, you have a script install.sh which essentially contains the following:

# find . | xargs touch
make && make install


or a variation thereon.
posted by Biru at 10:22 AM on August 21, 2010


Don't make end users touch the files -- that is not user friendly and you will only continue to get complaints about the build failing from people that didn't read the directions. If you must stick with zip files instead of using tarballs, do as grouse says and set the time stamps before creating the archive:

find . -print0 | xargs -0 touch -d '1 day ago'

Note that if you don't use -print0 / -0 then any files with whitespace in their names will fail.
posted by Rhomboid at 11:12 AM on August 21, 2010


Moreover, you don't want a build system/script that munges timestamps because then you lose the ability to incrementally build just the objects needing to be remade after making changes. If you're going to munge timestamps you should only do it before creating the source archive.
posted by Rhomboid at 11:16 AM on August 21, 2010


Provide tarballs for the *nix users.
Provide zips for the Windows users.
Provide arj for the DOS users. - Well, ok, maybe not this.
posted by Rendus at 11:22 AM on August 21, 2010


Now why would you do that? All the Windows programs to extract zip files (WinZip, WinRAR, 7-zip, etc.) can extract tarballs just as well, and they would still have build problems with the broken timestamps of the zip file.
posted by Rhomboid at 12:46 PM on August 21, 2010


Tar should record time stamps in UTC, which means no one in the world should have a problem with it since they all are using the same timebase.

You shouldn't have to worry about touching or clock changing or anything if you use tar and your master machine's clock is set correctly.

It also looks like some version of 7zip etc, can store timestamps in UTC format.

Some search terms I used:

timestamp utc zip
timestamp utc tar
posted by bottlebrushtree at 2:20 PM on August 21, 2010 [1 favorite]


All the Windows programs to extract zip files...

Except that you failed to mention the most common one, which is Windows Explorer (in XP or later) itself. Many Windows users don't have any of those utilities installed, because Windows itself deals just fine with ZIP archives. If you hand them a tarball, you're potentially causing them to have to go and download some third-party tool. I would not assume that most users have those.
posted by Kadin2048 at 4:59 PM on August 21, 2010


We're talking about source code here. Are you trying to say with a straight face that a person with a working C++ development environment that knows how to use a makefile is going to be confused by how to work an archive extractor?
posted by Rhomboid at 5:31 PM on August 21, 2010 [1 favorite]


Can't you have your Makefile touch all your source files before you pack them up? That is, on your end, you run 'make package' which touches all your files and then creates the zip archive, then you send it out. Then your user runs 'make all' 'make install' and 'make clean' as necessary, but doesn't have to worry about the timestamp thing.
posted by ctmf at 6:43 PM on August 21, 2010


@ Rhomboid: I neglected to mention that we also provide MSVC project files in addition to the GNU makefiles. Some time back, we had a problem with Windows users who wanted to use an old version of MSVC (v.6 I believe). The person making the distribution (often me, but not always) did it on a Linux system, and the .dsp and .dsw project files ended up having incompatible line endings for the Windows MSVC users. There were probably other/better ways of handling that issue, but the simplest seemed to us to use zip, which has a line-ending conversion facility that fixed this problem for us. To avoid dealing with too many files (we also distribute binary installers for different platforms & machine combos, which makes for a bunch of things at every release), we decided to just distribute one source archive file instead of two. But now, our users have moved on to newer versions of MSVC, which use XML project configuration files, and so I think this is no longer even an issue. In addition, separately, I always was under the impression that Windows users were most comfortable with zip archives and that the free and commonly-available Windows zip utilities wouldn't transparently handle gzip'ed tarballs. I haven't checked it for a long time (I regret I'm not much of a Windows user), so maybe this isn't an issue either.

In any case, our solution now consists of both distributing a gzip'ed tarball and a zip file, as before, and for the zip file, we adjust the timestamps on the files exactly per your suggestion above.

So, thanks!

@ grouse, @ ctmf: That's a great idea, and I'm embarrased I didn't think of such a straightforward solution. However, after implementing it now, I realize there's at least one disadvantage in modifying the file mod times like that. The files themselves may have been modified sooner than the 24 hour window. Since the files often have timestamps written inside of them (as auto-expanded svn $Date$ keywords), this leads to the curious situation that the file mod time is older than what the timestamp inside of it says. I'm not sure if that'll cause any problems, and in any case, we're going ahead with doing this for the zip archive.

Thanks again, everyone!
posted by StrawberryPie at 7:50 PM on August 24, 2010


« Older What is the URL of the music s...   |  What's a good, inexpensive (&g... Newer »
This thread is closed to new comments.