PowerShell: recursive processing of all files and folders without OutOfMemory exception?

October 8, 2009 8:26 AM

I need to use a PowerShell script to scan through a large amount of files and folders for certain security permissions. The issue is that it seems that the "Get-ChildItem" function insists on shoving the whole folder and file structure in memory. Since the drive has over a million files in over 30,000 folders, the script runs for 30 minutes, then throws out a System.OutofMemoryException without doing anything else.

The script is as follows:

$files = Get-ChildItem c:\

foreach $file in $files {
'Do something here
}

The obvious problem is the first statement. The obvious question remains: how do I walk through a drive recursively without running out of memory? Something of a Get-ChildItem that would execute code on the target without needing to first put the whole structure in memory?

posted by splice to Technology (15 answers total)

Try a depth-first search.

I'm not sure how to implement that in powershell though.
posted by demiurge at 8:31 AM on October 8, 2009

Actually, try pipes too. According to the docs:

However, there is an important difference. When you pipe multiple objects to a command, Windows PowerShell sends the objects to the command one at a time. When you use a command parameter, the objects are sent as a single array object.

So maybe something like

Get-ChildItem c:\ | do-thing

does what you want. I haven't tried it though.
posted by demiurge at 8:42 AM on October 8, 2009

Response by poster: I would love to have an answer from someone who knows how to do it and either has done it before or at least tested their solution.

Trying to fit my program in a pipe format will be hell. do-thing is a dozen lines of code with conditionals and everything. If you can give me an example of how you'd fit that in a piped command line, please go ahead. Otherwise, I'm not very much further than I was.
posted by splice at 8:59 AM on October 8, 2009

What kind of security permissions? ACL's?
posted by 8dot3 at 9:08 AM on October 8, 2009

Response by poster: As a reference, here is the entire script:

$outfile = "C:\AD_group_permissions_found.txt"

$files = Get-ChildItem -recurse j:\

foreach ($file in $files) {
  Write-Host "Scanning " $file.FullName

  $a = Get-Acl $file.FullName
  $aces =$a.GetAccessRules($true, $false, [System.Security.Principal.NTAccount])

  foreach ($ace in $aces) {
    if ($ace.IdentityReference.ToString() -match "^DOMAIN\\GROUP") {
      $file.FullName + ": " + $ace.IdentityReference.ToString() | Add-Content $outfile
    }
  }
}

posted by splice at 9:09 AM on October 8, 2009

Maybe
$files = Get-ChildItem c:\

foreach $file in $files | get-acl | export-csv aclexport.csv
posted by 8dot3 at 9:11 AM on October 8, 2009

(Sorry, didn't preview and hadn't seen your script)
posted by 8dot3 at 9:12 AM on October 8, 2009

You can limit the number of levels recursed, or maybe these guys will have a better solution.
posted by 8dot3 at 9:34 AM on October 8, 2009

Or, some memory helper functions.
I'll stop spamming you now. :)
posted by 8dot3 at 9:38 AM on October 8, 2009

Best answer: Change the foreach block to a function and pipe to that.

I would love to have an answer from someone who knows how to do it and either has done it before or at least tested their solution.

If you're looking for a contract programmer, MeFi Jobs is over there. Otherwise, it's not a perfect world.
posted by rhizome at 9:43 AM on October 8, 2009

Response by poster: If you're looking for a contract programmer, MeFi Jobs is over there. Otherwise, it's not a perfect world.

I'm looking for a pointer to a solution that actually works. I can program fine, I am a programmer by trade, I am simply new to PowerShell.

Limiting the number of levels to recurse would be completely counter-productive, since I really do need to scan everything on the drive. The memory management functions they provide don't look like they will help me any.

I am trying to pipe to a custom function, but it's not working so well (or at all). Hunting for documentation on this and would appreciate pointers.
posted by splice at 10:06 AM on October 8, 2009

Response by poster: And here is an example of why I'm not all keen on suggestions by folks who have never done this: after spending time testing this, I have confirmed that the pipeline method does the exact same thing. Get-ChildItem will shove everything in memory before passing an enumerator of all the objects collected to the function.

It is no solution to my problem, unfortunately.
posted by splice at 10:26 AM on October 8, 2009

Response by poster: But it may have led me to the solution: using "filter" instead of "function". Will try.
posted by splice at 10:31 AM on October 8, 2009

Response by poster: Looks like it'll work with filter! Thanks for the pointer to function pipelining, took some reading and research but I figured it out.
posted by splice at 10:37 AM on October 8, 2009

Unfortunately, that's what happens when you ask strangers on the internet. People are helpful, even if they may not know the 100% answer!
posted by CharlesV42 at 11:42 AM on October 8, 2009