Windows memory usage

November 18, 2007 8:22 AM

How can I force Windows to allow an application (Stata) to allocate itself more memory, when I have plenty of unused physical memory?

I recently upgraded to 2Gb of physical memory, and I'd like Stata to be able to use as much of this as possible so that I can work on large data files. The memory allocated for data is set manually with a command in Stata (set mem), but Windows (XP Home Edition) will only provide a maximum of about 745Mb before Stata reports "op.sys. refuses to provide memory". I was able to use up to 450Mb when my machine had only 768Mb of physical memory, so surely I ought to be able to get far more than 745Mb out of 2Gb. I have no other applications running, and no memory-intensive processes, so ought to have plenty of physical memory free.

posted by matthewr to computers & internet (12 comments total)

If you're trying to allocate a single chunk of data that size, you may be running into memory fragmentation issues.
posted by demiurge at 8:34 AM on November 18, 2007

I had this same problem with stata. My best bet was to start stata (and set mem) right away after rebooting windows, before other things (including system processes) had the chance to get greedy and go for memory. That, or use linux.
posted by a robot made out of meat at 8:36 AM on November 18, 2007

I started Stata just after startup, and 745Mb was the most it would allow. Surely there's some way of allowing one program to use more than 745Mb of memory in a 2Gb system, especially when other processes are at a minimum. Unfortunately, Linux isn't a practical option at the moment.
posted by matthewr at 9:09 AM on November 18, 2007

a google search for your error message links to this mailing list message from someone who works for stata
posted by noloveforned at 9:29 AM on November 18, 2007

You could try playing with the amount of the hard drive used as RAM, virtual memory. That's in control panel->system properties ->advanced -> performance -> settings->advanced->virtual memory->change. I would try both increasing and decreasing and seeing if there is some sort of relationship.
posted by Eringatang at 10:03 AM on November 18, 2007

I'm on XP pro, w/ 2gb ram, and Stata allows me to set mem up to 805m. I'll be interested if you can find a way to allocate more.
posted by langedon at 10:19 AM on November 18, 2007

I saw the StataList message, noloveforned, but unfortunately the Microsoft solution they link to doesn't apply to XP Home. I've already reduced the memory usage of other processes as far as I can.
posted by matthewr at 11:35 AM on November 18, 2007

The most helpful thing I've read so far is this article ("How do I load large datasets (>1 GB) under 32-bit Windows?"). I tried Stata version 9, and found it allowed me about 940m, which suggests I need this hotfix which should allow Stata 10 the same amount.

Judging from the Stata support article, this is the best I'm going to get, so I'll see what I can do in terms of only loading specific variables from the dataset. Thanks, all!
posted by matthewr at 11:54 AM on November 18, 2007

Um, have you actually tried the Microsoft solution, or are you just assuming it won't work on XP Home because it's not specifically listed?
posted by wierdo at 12:19 PM on November 18, 2007

No, I haven't tried the "4GT RAM Tuning" solution because the MSDN page lists XP Pro but not Home. Are you saying this fix does work on Home?
posted by matthewr at 12:33 PM on November 18, 2007

Memory in windows is split in 2 regions, kernel space and user space, both are 2gig if you don't use the /3gb hack which increases the user space to 3gb (can cause you problem because you have less kernel space, especially with your video drivers but people have been known to work with it).

Applications allocate memory from windows using calls to VirtualAlloc which maps a region a virtual address space (all process have their own 2gb virtual address space) to physical memory (the actual thing on your motherboard). So address 0x10000000 that is returned by VirtualAlloc can be mapped to addresses 0x0000F000 on the actual RAM but you don't have to worry about it Windows will take care of it for you. So essentially your application can always allocate memory until it hits the 2gb barrier (if your system has less than 2gb of memory it'll be paged to disk and be very slow). The reason an allocation can fail is either because the 2gb space is all allocated or because it's trying to allocate a contiguous chunk that is too big (ie, you have 5 megs free in 2 blocks of 2 and 3 megs and you want a continuous block of 4 megs, this will not work).

Now the main causes of memory fragmentation are inefficient heap allocators (lots of small blocks mapped and unmapped in virtual memory, your user space is now a swiss cheese), allocations of inefficient size (on most win32 systems the minimum size that should be allocated is 64k not 4k, 4k vallocs will waste 60k due to internal constraints in the virtual memory system) or dlls being loaded in the middle of your address space thus creating fragmentation.

Now it seems that stata isn't doing such a great job at managing memory since you still have user space available (maybe... check the VM Size column in task manager, avalaible by selecting it in View | Select Columns... ) and it can't allocate more memory. This might be because it's trying to allocate it's memory in one big chunk (this is a bad programming practice, but out of your control).

The only thing you can (maybe) control is the dlls base address, if those are creating fragmentation you can rebase them with EDITBIN (comes with Visual Studio, probably available elsewhere). Now to do that you first have to determine if the dlls are causing the fragmentation and which ones to move. You can use Dependency Walker to explore which dlls are used by stata and see their preferred base address (no guarantee they're loaded there). You can use graph paper (or excel) to graph them and move all those that fall in the middle of the user space to the lower regions (make sure they don't collide with other dlls or they'll be rebased elsewhere when loading).

This is a fairly involved process though. I'd try the /3gb first it might be easier, I don't see why it wouldn't work with Home Edition since the differences between windows skus are mostly in the packaged applications and default UI settings, the kernel is the same. But they might have actually disabled it to make sure people who needed it bought the Professional version.

Also consider that if you have 2gb of memory used by stata your system will probably swap to disk excessively and be very unresponsive, but you can probably get more than 754mb of memory used.
posted by coust at 11:43 PM on November 18, 2007

For future Askers: I happened upon an old copy of Stata 6, and found it allocates up to about 1.4g of memory, twice that of Stata 9 or 10. Progress, eh?
posted by matthewr at 12:02 PM on November 30, 2007