How to auto-delete files in a folder hierarchy to keep total size <12Gb

October 8, 2017 6:59 PM

I have several security cameras which trigger on motion-detect and connect via FTP to my PC where all the video files are deposited into a folder hierarchy. This folder is monitored by pCloud (where I have 12Gb of space available) so anytime video files appear in that folder they are automatically synced to the cloud. Deleting files from the folder of course delete them from the cloud as well. The problem is the cloudspace limit of 12Gb. My cameras will deposit that amount of files within about 7-10 days. Is there any freeware monitoring software (for Windows), or some sort of script I can use, that will constantly monitor this folder hierarchy and upon reaching or approaching 12Gb of total folder+subfolder size, will automatically delete the oldest file(s) in the hierarchy in order to keep the usage under that total limit so I don't eventually blow out my max space on cloud storage? Right now I'm just manually (completely) clearing them out by running a batch file.
posted by tra to Computers & Internet (7 answers total)

How to schedule a batch file to run automatically on windows 7. Don't know if newer versions of Windows have the same capability, but I'd be surprised if they don't.
posted by Homer42 at 8:25 PM on October 8, 2017

I would run a script periodically that checks the space remaining, and if above a certain size, run the cleanup script.

https://stackoverflow.com/questions/51054/batch-file-to-delete-files-older-than-n-days
posted by nickggully at 9:06 PM on October 8, 2017

I don't know how many minutes of video it takes to fill up 12GB, but something to be careful about with an automated script is what if someone breaks in, your cameras film over 12GB of video in one go, and then the script deletes all the files (at the point where you actually need them!).
posted by EndsOfInvention at 2:35 AM on October 9, 2017 [3 favorites]

Best answer: Here's a JScript script that walks through all the files in a folder (and all its subfolders) in order from newest to oldest, adding up the sizes as it goes, then starts deleting files once the total hits the limit. Tested on Windows 7.

Save this as prune.js and schedule it using the same method Homer42 linked to for scheduling a batch file.

// Folder to prune. You can use backslashes instead of forward slashes
// in the pathname if you like, but then you'd have to double them.

var pathname = 'C:/path/to/pcloud/folder';

// Maximum disk space that folder will be allowed to consume
// after pruning, in bytes.

var limit = 11.5 * 1000 * 1000 * 1000;

// Call a function on each member of a collection.

function each(collection, func) {
	var e = new Enumerator(collection);
	for (e.moveFirst(); !e.atEnd(); e.moveNext()) {
		func(e.item());
	}
}

// Call a function on a folder and all its subfolders.
// The most deeply nested subfolders get processed first.

function walkTree(folder, func) {
	each(folder.SubFolders, function(subFolder) {
		walkTree(subFolder, func);
	});
	func(folder);
}

// Return an array of all the files inside a folder
// and all its subfolders.

function filesIn(folder) {
	var files = [];
	walkTree(folder, function(folder) {
		each(folder.Files, function(file) {
			files.push(file);
		});
	});
	return files;
}

// Comparison function for sorting an array of files.

function newestFirst(a, b) {
	return b.DateLastModified - a.DateLastModified;
}

// Delete as many old files as necessary to keep
// disk usage below limit.

var fso = new ActiveXObject('Scripting.FileSystemObject');
var folder = fso.getFolder(pathname);
var files = filesIn(folder).sort(newestFirst);

var totalDiskUsed = 0;
for (var i = 0; i < files.length; ++i) {
	// Round size up to cluster size boundary
	var diskUsed = (files[i].size + 4095) / 4096 * 4096;
	totalDiskUsed += diskUsed;
	if (totalDiskUsed > limit) files[i].Delete(true);
}

// Remove empty subfolders.

walkTree(folder, function(folder) {
	if (folder.SubFolders.Count + folder.Files.Count == 0) {
		folder.Delete(true);
	}
});

posted by flabdablet at 11:44 AM on October 9, 2017 [1 favorite]

Best answer: Sorry, there's a bug in that. I managed to forget that JScript arithmetic is floating-point, not integer, when writing the conversion from file size to disk space used. The result is that it may prune slightly harder than it really needs to. Please substitute the following lines:

	// Round size up to cluster size boundary
	var diskUsed = Math.ceil(files[i].size / 4096) * 4096;

posted by flabdablet at 8:22 PM on October 9, 2017

Best answer: Here's another version of the same algorithm as a cmd+PowerShell polyglot, just because I'm trying to persuade myself to learn PowerShell which is allegedly the most bestest scripting language that ever languaged a script.

To be fair, even with the idiotic cmd section at the beginning that's only needed because PowerShell's default "security" policy prohibits it from actually running any scripts, it's still a fair bit shorter than the JScript version and it does run slightly faster.

Save the whole mess as prune.cmd and schedule it like prune.cmd -limit 11.5e9 -path C:\Path\To\pCloud\Folder

<# :
@set scriptdir=%~dp0
@set scriptps1="%tmp%\%~n0-%random%-%random%-%random%-%random%.ps1"
@copy /b /y "%~f0" %scriptps1% >nul
@powershell -executionpolicy bypass -file %scriptps1% %*
@del /f %scriptps1%
@goto :eof
#>

param(
	[double] $limit = [double]::maxValue,
	[string] $path
)

function depthFirst($action) {
	process {
		$_.GetDirectories() | depthFirst $action
		& $action
	}
}

$folder = get-item $path
$totalDiskUsed = 0

$folder | depthFirst {$_.getFiles()} |
sort lastWriteTime -descending | 
foreach {
	$diskUsed = [Math]::ceiling($_.length / 4096) * 4096
	if (($totalDiskUsed += $diskUsed) -gt $limit) {
		$_.delete()
	}
}

$folder | depthFirst {
	if ($_.GetFileSystemInfos().length -eq 0) {
		$_.delete()
	}
}

posted by flabdablet at 5:41 AM on October 10, 2017

Best answer: And for what it's worth, here's how that algorithm looks on a system that has scripting built into its guts instead of bolted on as an afterthought.

#!/bin/sh
limit=$(( $1 / 512 ))
path=$2

totalBlocksUsed=0
find $path -type f -printf '%T@:%b:%p\n' |
sort -t: -rnk1 |
while IFS=: read -r time blocks name
do
        totalBlocksUsed=$(( totalBlocksUsed + blocks ))
        if [ $totalBlocksUsed -gt $limit ]
        then echo $name
        fi
done |
xargs rm -vf

find $path -depth -type d -empty -exec rmdir -v {} \;

posted by flabdablet at 7:17 AM on October 10, 2017