How do I save space by zipping my files?
December 25, 2022 11:03 PM Subscribe
I have a ton of video files (mp4s) on my Windows 11 laptop. I did an experiment by taking one of many folders (this one had 18.1gigs) and ran it through 7zip. About 13 minutes later, I got a zip file that was 18gigs. Surely there's a better way of doing this and saving more than just that .1gig. Any tips?
Response by poster: Thanks for the answer gregjones.
If that's so, does that mean that I only have two options?
1. Delete tons of videos
2. Get a new big external hard drive to store them on
Is there a third option? And just what is zipping for anyway? What file types benefit from zipping and which don't?
posted by ggp88 at 11:21 PM on December 25, 2022
If that's so, does that mean that I only have two options?
1. Delete tons of videos
2. Get a new big external hard drive to store them on
Is there a third option? And just what is zipping for anyway? What file types benefit from zipping and which don't?
posted by ggp88 at 11:21 PM on December 25, 2022
Zipping benefits files with patterns that the algorithm can spot, and results in a file that hasn't got obvious patterns in it. That's pretty true of all compression algorithms, so even a comparatively poorly compressed file will not compress much further with another algorithm. Also, it's quite common that these pattern-less files make the compression algorithm work really hard, so not only do they not get smaller when you compress them but they also take a long time for 7zip to work on.
JPEGs have the additional advantage that they know they're working on an image file so they have a good idea how to look for patterns (go look up discrete cosine transforms if you want to disappear down a deep rabbit hole) and they're also being slightly selective about patterns that matter (ones you can see) and throwing away patterns that don't matter (ones your brain is filtering out for you). Video files are also compressed, as are audio files.
You probably aren't going to make inroads on any of the file types above, so yes, an external drive is probably a good option for you. Alternatives include cloud storage or a network attached storage device.
Also, as a general recommendation, it's a good idea to keep files in two places in case one of the places fails on you (say, two external drives, or your onboard hard disk with a backup for things you don't offload, or in the cloud and also locally). What can happen is that one of the drives dies while it's on the shelf and you don't know until you come to look for the file. The two types of media don't have to be the same.
You may find USB sticks or SD cards work well enough for you. They're comparatively slow, but cheap. If those files are ones you don't use all that much and if you have a good filing system so that you can find what you're looking for without a lot of searching all the data then you don't have to pay for speed. Alternatively, if you want to edit those files or use them from where you put them (eg pointing a photo app at them on their external storage) get something faster - I use USB-C SSDs, which are at the other end of the scale: stupid fast but more expensive. Spinning disks lie somewhere in the middle.
posted by How much is that froggie in the window at 12:09 AM on December 26, 2022 [8 favorites]
JPEGs have the additional advantage that they know they're working on an image file so they have a good idea how to look for patterns (go look up discrete cosine transforms if you want to disappear down a deep rabbit hole) and they're also being slightly selective about patterns that matter (ones you can see) and throwing away patterns that don't matter (ones your brain is filtering out for you). Video files are also compressed, as are audio files.
You probably aren't going to make inroads on any of the file types above, so yes, an external drive is probably a good option for you. Alternatives include cloud storage or a network attached storage device.
Also, as a general recommendation, it's a good idea to keep files in two places in case one of the places fails on you (say, two external drives, or your onboard hard disk with a backup for things you don't offload, or in the cloud and also locally). What can happen is that one of the drives dies while it's on the shelf and you don't know until you come to look for the file. The two types of media don't have to be the same.
You may find USB sticks or SD cards work well enough for you. They're comparatively slow, but cheap. If those files are ones you don't use all that much and if you have a good filing system so that you can find what you're looking for without a lot of searching all the data then you don't have to pay for speed. Alternatively, if you want to edit those files or use them from where you put them (eg pointing a photo app at them on their external storage) get something faster - I use USB-C SSDs, which are at the other end of the scale: stupid fast but more expensive. Spinning disks lie somewhere in the middle.
posted by How much is that froggie in the window at 12:09 AM on December 26, 2022 [8 favorites]
You can put ‘em in the cloud - if you rarely need to access them there are options that allow sort of cool/cold where you don’t stream or work a lot with them in return for very cheap storage. I just got an external drive for my videos, backing up important ones that can’t be replaced on a paid cloud site as well.
The only time I’ve zipped video files is to create a collection of smaller files or because one of the systems didn’t accept video files but did zips.
posted by dorothyisunderwood at 12:10 AM on December 26, 2022 [1 favorite]
The only time I’ve zipped video files is to create a collection of smaller files or because one of the systems didn’t accept video files but did zips.
posted by dorothyisunderwood at 12:10 AM on December 26, 2022 [1 favorite]
What file types benefit from zipping and which don't?
Zipping will save space for text files, executable programs, Word and Excel and PowerPoint documents in old formats; the newer formats for that kind of document (docx, xlsx, pptx) are in fact already Zip files in disguise, as are all the OpenDocument formats used by products like OpenOffice and LibreOffice, and zipping an already-zipped file almost never saves additional space.
It won't typically save space for audio or video or photo files that are already in compressed formats. Video files are typically many orders of magnitude bigger each than the formats that Zip will save space for, and if a drive is getting tight for space it's almost always because it has videos stored on it. Which means that lossless compression formats like Zip or Windows's own inbuilt transparent file compression are almost completely useless when it comes to saving disk space.
The main use for Zip in 2022 is bundling a folder's worth of files together as a single archive file for the purpose of convenient handling. If doing that saves a little space then that's a bonus, but in 2022 it's vanishingly rare for Zip to save enough space for anybody to notice much.
So yes, if your laptop is getting tight for space because it has videos on it, then buying a couple of cheap external USB3 hard disk drives to park those on is your best bet. Always keep identical copies on separate drives. External drives are cheap, and the time you've put into curating your media collection is not.
posted by flabdablet at 3:09 AM on December 26, 2022 [7 favorites]
Zipping will save space for text files, executable programs, Word and Excel and PowerPoint documents in old formats; the newer formats for that kind of document (docx, xlsx, pptx) are in fact already Zip files in disguise, as are all the OpenDocument formats used by products like OpenOffice and LibreOffice, and zipping an already-zipped file almost never saves additional space.
It won't typically save space for audio or video or photo files that are already in compressed formats. Video files are typically many orders of magnitude bigger each than the formats that Zip will save space for, and if a drive is getting tight for space it's almost always because it has videos stored on it. Which means that lossless compression formats like Zip or Windows's own inbuilt transparent file compression are almost completely useless when it comes to saving disk space.
The main use for Zip in 2022 is bundling a folder's worth of files together as a single archive file for the purpose of convenient handling. If doing that saves a little space then that's a bonus, but in 2022 it's vanishingly rare for Zip to save enough space for anybody to notice much.
So yes, if your laptop is getting tight for space because it has videos on it, then buying a couple of cheap external USB3 hard disk drives to park those on is your best bet. Always keep identical copies on separate drives. External drives are cheap, and the time you've put into curating your media collection is not.
posted by flabdablet at 3:09 AM on December 26, 2022 [7 favorites]
Unless you re-encode the videos with a higher compression (and so lower quality), they won't get smaller.
This is the third option - using ffmpeg or similar to shrink the videos by reencoding them. It takes a long time, and results in lower video quality (which may be fine depending on what they are and what kind of screens you plan to be viewing them on). If you know what you're doing you can choose settings that will minimize quality loss while maximizing shrinkage, but I don't know enough about it to offer advice.
And I want to second what How much is that froggie said - if these videos are important at all, make multiple copies. Everything fails at some point, and I've had even solid state memory fail way sooner than it was supposed to, with no warning.
posted by trig at 3:10 AM on December 26, 2022 [2 favorites]
This is the third option - using ffmpeg or similar to shrink the videos by reencoding them. It takes a long time, and results in lower video quality (which may be fine depending on what they are and what kind of screens you plan to be viewing them on). If you know what you're doing you can choose settings that will minimize quality loss while maximizing shrinkage, but I don't know enough about it to offer advice.
And I want to second what How much is that froggie said - if these videos are important at all, make multiple copies. Everything fails at some point, and I've had even solid state memory fail way sooner than it was supposed to, with no warning.
posted by trig at 3:10 AM on December 26, 2022 [2 favorites]
Don't use USB3 SSDs to archive video. For the price of any given amount of solid state storage you can buy disk drives with at least five times as much capacity, and for media playback you don't need and won't see the speed advantage that some SSDs have over disk.
Get multiple disk drives instead, preferably from at least two different manufacturers to minimize your chance of seeing multiple failures at the same time. Western Digital, Seagate and Buffalo all offer drives that are more than good enough.
If it would suit you to keep your main archive at home, with just a selection from it available on your laptop at any given time, look at external drives designed for desktop use, based on the larger and more power-hungry 3.5" mechanisms. These are generally about half the price per gigabyte of the smaller, lighter and somewhat more robust 2.5" ones. The 3.5" drives also come with their own power supplies, so your laptop won't need to power them which is good for battery life.
Biggest capacity currently available in a single external 3.5" disk drive is currently around 20TB. Lowest cost per gigabyte will usually be a couple of generations back from the biggest; 14TB drives are looking quite attractive at the moment. You will occasionally see generic/unbranded "16TB" SSDs advertised at very low prices compared to the big disk drives but given the cost of genuine flash storage chips these really are too good to be true: they are almost certainly fakes with much less actual capacity and a propensity for corrupting anything they do manage to store.
Avoid thumb drives or sticks or memory cards for video archiving. Yes they're very cheap each, but they work out way more expensive per gigabyte than disk drives and they're also much less reliable unless you're in the habit of dropping your drives or otherwise knocking them about. Tiny memory devices are good for caching a few tens of gigabytes for use on the go, but using them for archives is just asking for trouble.
posted by flabdablet at 3:48 AM on December 26, 2022 [4 favorites]
Get multiple disk drives instead, preferably from at least two different manufacturers to minimize your chance of seeing multiple failures at the same time. Western Digital, Seagate and Buffalo all offer drives that are more than good enough.
If it would suit you to keep your main archive at home, with just a selection from it available on your laptop at any given time, look at external drives designed for desktop use, based on the larger and more power-hungry 3.5" mechanisms. These are generally about half the price per gigabyte of the smaller, lighter and somewhat more robust 2.5" ones. The 3.5" drives also come with their own power supplies, so your laptop won't need to power them which is good for battery life.
Biggest capacity currently available in a single external 3.5" disk drive is currently around 20TB. Lowest cost per gigabyte will usually be a couple of generations back from the biggest; 14TB drives are looking quite attractive at the moment. You will occasionally see generic/unbranded "16TB" SSDs advertised at very low prices compared to the big disk drives but given the cost of genuine flash storage chips these really are too good to be true: they are almost certainly fakes with much less actual capacity and a propensity for corrupting anything they do manage to store.
Avoid thumb drives or sticks or memory cards for video archiving. Yes they're very cheap each, but they work out way more expensive per gigabyte than disk drives and they're also much less reliable unless you're in the habit of dropping your drives or otherwise knocking them about. Tiny memory devices are good for caching a few tens of gigabytes for use on the go, but using them for archives is just asking for trouble.
posted by flabdablet at 3:48 AM on December 26, 2022 [4 favorites]
You probably need to explain what you want to do with these files.
If they're just something you want to keep safe for the long term, with infrequent access, then a cloud backup solution (I use Backblaze, but there are many), along with an external drive as a local backup, would be a good approach. I just use a USB-connected dock for bare 3.5" drives. That allows me to back up a drive-full of files, then eject the drive and stash it away in a bag at the back of a cupboard.
If you're just trying to free up a load of space, but still need access to the files, then an external disk drive (mechanical ones are still cheapest) is probably the way to go. Or go a bit more fancy and pick up a cheap NAS to connect to your network (ideally one with more than one disk in RAID configuration). It really depends whether you already have a solid backup regime; if not, it might be worth thinking about alongside the 'I have too many video files' problem.
posted by pipeski at 5:55 AM on December 26, 2022 [1 favorite]
If they're just something you want to keep safe for the long term, with infrequent access, then a cloud backup solution (I use Backblaze, but there are many), along with an external drive as a local backup, would be a good approach. I just use a USB-connected dock for bare 3.5" drives. That allows me to back up a drive-full of files, then eject the drive and stash it away in a bag at the back of a cupboard.
If you're just trying to free up a load of space, but still need access to the files, then an external disk drive (mechanical ones are still cheapest) is probably the way to go. Or go a bit more fancy and pick up a cheap NAS to connect to your network (ideally one with more than one disk in RAID configuration). It really depends whether you already have a solid backup regime; if not, it might be worth thinking about alongside the 'I have too many video files' problem.
posted by pipeski at 5:55 AM on December 26, 2022 [1 favorite]
I just use a USB-connected dock for bare 3.5" drives.
On the other hand, desktop external drives that come in their own USB3 enclosures now usually cost less, often by a hundred bucks or so, than the very same mechanism bought as a bare drive. Plus they also offer a modicum of protection against both physical shock and electrostatic discharge damage.
posted by flabdablet at 6:27 AM on December 26, 2022 [1 favorite]
On the other hand, desktop external drives that come in their own USB3 enclosures now usually cost less, often by a hundred bucks or so, than the very same mechanism bought as a bare drive. Plus they also offer a modicum of protection against both physical shock and electrostatic discharge damage.
posted by flabdablet at 6:27 AM on December 26, 2022 [1 favorite]
I don't know, flabdablet - after a series of external drive failures I've stopped buying them and instead use internal ones with adapters. I can't say for sure what the situation is, but I saw a bunch of sources online claiming that the reason for those lower prices on external drives is that they use lower quality drives, and whether that's true or not it lines up unfortunately well with my experience. I'm fairly confident the cheap external ones don't use higher quality drives, at any rate.
However, this is a bit of a derail ;-)
posted by trig at 6:34 AM on December 26, 2022
However, this is a bit of a derail ;-)
posted by trig at 6:34 AM on December 26, 2022
You might experiment with transcoding the videos to a more efficient codec. Most video you find now is H.264; that's what most consumer video recording systems produce and it's also what you tend to find with pirated video. The newer H.265 codec has similar video quality in 1/3 to 1/4 the space.
Doing this kind of transcoding is not simple; it's easy to get bad results. The rest of this comment is probably more hacky than useful but maybe you like to tinker like this.
Handbrake is a good GUI tool for transcoding video on a desktop computer. unmanic is an *arr style app to automate transcoding (I never could make it work for me.) I transcode on my Linux server with ffmpeg and a little shell script:
Another transcoding option is to lower the quality of the video. If it's 4K from a Blu-Ray source transcoding it to 1080p or even 720p will save some space, although surprisingly not as much as you'd think. You can also lower audio quality, strip foreign audio soundtracks, etc. This all gets pretty fiddly.
posted by Nelson at 8:16 AM on December 26, 2022 [1 favorite]
Doing this kind of transcoding is not simple; it's easy to get bad results. The rest of this comment is probably more hacky than useful but maybe you like to tinker like this.
Handbrake is a good GUI tool for transcoding video on a desktop computer. unmanic is an *arr style app to automate transcoding (I never could make it work for me.) I transcode on my Linux server with ffmpeg and a little shell script:
/usr/lib/jellyfin-ffmpeg/ffmpeg -c:v h264_qsv -i "$1" -map 0 -c copy -c:v hevc_qsv -preset slow -global_quality 22 -look_ahead 1 $2 "${1%.*} nelson265.mkv"Quality is acceptable but with some small degradation if you look for it. This script is tuned to work well with my i5-12600K; I can transcode videos in 4-8x the speed of playback. (Ie, 15-30 minutes for a 2 hour video.) There's alternate options using a GPU if you have it.
Another transcoding option is to lower the quality of the video. If it's 4K from a Blu-Ray source transcoding it to 1080p or even 720p will save some space, although surprisingly not as much as you'd think. You can also lower audio quality, strip foreign audio soundtracks, etc. This all gets pretty fiddly.
posted by Nelson at 8:16 AM on December 26, 2022 [1 favorite]
+1 to Nelson, although in my experience, transcoding an H.264 file to H.265 doesn't have such dramatic results. I recently re-encoded a file using Handbrake: the source file was 1.9 GB, the target file was 1.2 GB (it may be that my settings were not well-optimized). On my pretty new Mac, I'd expect 18 GB of files to take a few hours to transcode.
Throwing money at the problem and buying a big external hard drive is the easiest way to deal with this problem. Hard drives are pretty cheap.
posted by adamrice at 8:55 AM on December 26, 2022 [1 favorite]
Throwing money at the problem and buying a big external hard drive is the easiest way to deal with this problem. Hard drives are pretty cheap.
posted by adamrice at 8:55 AM on December 26, 2022 [1 favorite]
flabdablet: Western Digital, Seagate and Buffalo all offer drives
Note that there are now just three manufacturers of 'spinning rust', conventional hard disk drives: Seagate, Western Digital and Toshiba. Any other brand will have one of those as the actual drive and while you may end up with two drives both from the same manufacturer, it's unlikely they'd be from the same production run which is what you want to avoid.
posted by Stoneshop at 9:11 AM on December 26, 2022 [1 favorite]
Note that there are now just three manufacturers of 'spinning rust', conventional hard disk drives: Seagate, Western Digital and Toshiba. Any other brand will have one of those as the actual drive and while you may end up with two drives both from the same manufacturer, it's unlikely they'd be from the same production run which is what you want to avoid.
posted by Stoneshop at 9:11 AM on December 26, 2022 [1 favorite]
As an aside, depending on how many videos you have in that folder, 18 GB for video doesn't sound like a lot, as I'm used to handling 5-10 GB files for blu-ray movie rips, so I wouldn't expect that re-encoding would necessarily save you all that much space without a significant loss in quality either, depending on the format (or codec) the videos are in, what resolution they have and what their runtime is.
An external drive is definitely the easiest and least time-consuming way out here.
posted by Aleyn at 10:02 AM on December 26, 2022
An external drive is definitely the easiest and least time-consuming way out here.
posted by Aleyn at 10:02 AM on December 26, 2022
Transcoding your video is probably the only thing that can result in significant size savings.
I do a fair bit of Youtubing with my game videos, and what I found was standardizing on 30 fps, as well as reducing audio from 384 kbps to 128 kbps (or even 96 kbps), and sampling rate down to 12000 Hz. That generally had a huge effect on my videos, often reducing the video size by 90%. But then I do have an RTX 2070 and uses NVENC to help encode video in a reasonable amount of time (even hour-long videos are usually transcoded in 10 minutes)
This doesn't always work, and if you have a large video library this will take a long time, for dubious storage savings and SOME loss in video and audio quality.
Ultimately, it's probably easier to buy more storage than to reduce your existing video size.
posted by kschang at 10:15 AM on December 26, 2022 [2 favorites]
I do a fair bit of Youtubing with my game videos, and what I found was standardizing on 30 fps, as well as reducing audio from 384 kbps to 128 kbps (or even 96 kbps), and sampling rate down to 12000 Hz. That generally had a huge effect on my videos, often reducing the video size by 90%. But then I do have an RTX 2070 and uses NVENC to help encode video in a reasonable amount of time (even hour-long videos are usually transcoded in 10 minutes)
This doesn't always work, and if you have a large video library this will take a long time, for dubious storage savings and SOME loss in video and audio quality.
Ultimately, it's probably easier to buy more storage than to reduce your existing video size.
posted by kschang at 10:15 AM on December 26, 2022 [2 favorites]
trig: instead use internal ones with adapters
That way you at least know the exact make and model of the drive you're buying, allowing you to check its specs in detail like whether it's SMR or not, and often find performance and reliability tests. With external drives you rarely know much beyond capacity, rotation speed, seek time and best-case read and write throughput, and tomorrow the drive fitted can be different from today's.
posted by Stoneshop at 11:08 AM on December 26, 2022
That way you at least know the exact make and model of the drive you're buying, allowing you to check its specs in detail like whether it's SMR or not, and often find performance and reliability tests. With external drives you rarely know much beyond capacity, rotation speed, seek time and best-case read and write throughput, and tomorrow the drive fitted can be different from today's.
posted by Stoneshop at 11:08 AM on December 26, 2022
Drives designed around SMR can perform extremely poorly with certain kinds of workload. Such drives will surely cause grief if they end up as part of a RAID set in a NAS, for example. But storing moderate numbers of large files that will be read far more often than rewritten is close to the best possible use case for SMR. It's really not worth avoiding when choosing drives to be used as stand-alone media archives, and might in fact reduce cost per gigabyte to quite a useful extent.
With external drives you rarely know much beyond capacity, rotation speed, seek time and best-case read and write throughput, and tomorrow the drive fitted can be different from today's.
This is all completely true, but media archiving is such an undemanding use case that any modern hard disk drive will perform completely acceptably. If you're choosing a stand-alone drive for a media archive, a low cost per gigabyte and a half-decent warranty are all you really need.
The newer H.265 codec has similar video quality in 1/3 to 1/4 the space.
I have generally seen H.265 (aka HEVC) yield much less dramatic storage savings than that for any given quality level. In my experience, H.265 video works out at around 2/3 to 3/4 of the size of its H.264 (aka AVC) visual equivalent.
H.265 is also much more demanding to decode for playback than H.264, and still not as well supported by hardware acceleration. My own media archive is all H.264 because although H.265 works fine on the Odroid media player that feeds my TV, it causes stuttering and freezing on my shitty old laptop and super cheap phone. Ugrading both of those would cost me more than the disk space H.265 could feasibly save me.
posted by flabdablet at 2:49 PM on December 26, 2022 [2 favorites]
With external drives you rarely know much beyond capacity, rotation speed, seek time and best-case read and write throughput, and tomorrow the drive fitted can be different from today's.
This is all completely true, but media archiving is such an undemanding use case that any modern hard disk drive will perform completely acceptably. If you're choosing a stand-alone drive for a media archive, a low cost per gigabyte and a half-decent warranty are all you really need.
The newer H.265 codec has similar video quality in 1/3 to 1/4 the space.
I have generally seen H.265 (aka HEVC) yield much less dramatic storage savings than that for any given quality level. In my experience, H.265 video works out at around 2/3 to 3/4 of the size of its H.264 (aka AVC) visual equivalent.
H.265 is also much more demanding to decode for playback than H.264, and still not as well supported by hardware acceleration. My own media archive is all H.264 because although H.265 works fine on the Odroid media player that feeds my TV, it causes stuttering and freezing on my shitty old laptop and super cheap phone. Ugrading both of those would cost me more than the disk space H.265 could feasibly save me.
posted by flabdablet at 2:49 PM on December 26, 2022 [2 favorites]
This thread is closed to new comments.
Unless you re-encode the videos with a higher compression (and so lower quality), they won't get smaller.
posted by gregjones at 11:15 PM on December 25, 2022 [11 favorites]