Memory usage in Windows Server
September 18, 2011 3:56 AM Subscribe
Why might a process not use all available physical memory under Windows Server 2008 r2 standard? I have 32Gb of RAM acknowledged by Resource Monitor, of which 19.4 are allocated to a process (the IBM solver CPLEX running in command line form from Power Shell) that has just thrown a controlled OOM error despite there being another 12 Gb marked as available.
I'm going to assume (to start with) that the program itself isn't the culprit, but is there a per-user limit switch, a per-process limit switch or something similar in 2008 r2? Any other ideas?
How many cores? How many processes does CPLEX use? Is every bit of everything CPLEX uses true 64 bit code? Spidey sense tingles with 4 worker and 1 aggregator processes each hitting their 4G 32 bit memory limit minus 4 or 5 chunks of memory that aren't actual allocated memory but shared mapped memory for IPC. 4 workers @ 4G plus the master at 4G but 4x128M of that is shared to communicate with the workers == ~19.5G. (could also be a 4G per process limit with 5 processes instead of a 64/32 bit limit).
posted by zengargoyle at 6:59 AM on September 18, 2011
posted by zengargoyle at 6:59 AM on September 18, 2011
Best answer: I know nothing about CPLEX and suspect few people here do. A quick Google search for cplex memory usage turns up a "workmem" parameter that can be configured. Its default is 512kb, so it must be some sort of small working set size and not the full system usage. A search for cplex workmem has many promising leads.
More generically, my first thought was it was a Java program with a fixed memory limit, but CPLEX is not Java. zengargoyle's advice on a 4G process limit is possible but I'm a bit skeptical. But be sure you're running a 64 bit system and program. (Sorry if this answer is not useful; your best bet is to find other CPLEX users and ask them.)
posted by Nelson at 8:40 AM on September 18, 2011
More generically, my first thought was it was a Java program with a fixed memory limit, but CPLEX is not Java. zengargoyle's advice on a 4G process limit is possible but I'm a bit skeptical. But be sure you're running a 64 bit system and program. (Sorry if this answer is not useful; your best bet is to find other CPLEX users and ask them.)
posted by Nelson at 8:40 AM on September 18, 2011
Response by poster: Thanks all - solution is a bit of everything suggested here. CPLEX is absolutely 64-bit throughout. The workaround seems to be a suite of memory efficiency routines of which workmem is one, and which effectively stop a progress trace being held in RAM. Forcing the trace to be written to disk is dog slow, but it does at least run. It's parallel code, and it does look as though there's a preemptive survey of available RAM to allow graceful failure ahead of one core actually failing, which explains the odd limit of RAM observed - it's not an OOM error so much as a HAL-style warning.
Much obliged!
posted by cromagnon at 9:38 AM on September 18, 2011
Much obliged!
posted by cromagnon at 9:38 AM on September 18, 2011
« Older Best advice for non-native speakers teaching in... | Some Questions About Traffic-Light controlled... Newer »
This thread is closed to new comments.
posted by procrastination at 6:16 AM on September 18, 2011