visual studio memory
July 15, 2010 2:03 PM Subscribe
I'm running visual studio 2008 on Vista with 8 GB of memory, and I want to be able to input a very large array from a file to perform calculations on, but it won't let me input a file that is 3GB long. Is there a way to correct this?
Here is the error message I get:
Unhandled exception at 0x76a9f328 in "program name".exe : Microsoft C++ exception" std::bad_alloc at memory location 0x002cf6cc..
and the code:
// newaop -- operator new[](size_t) REPLACEABLE
#include
#if !_VC6SP2 || _DLL
void *__CRTDECL operator new[](size_t count) _THROW1(std::bad_alloc)
{ // try to allocate count bytes for an array
return (operator new(count));
}
#endif /* !_VC6SP2 || _DLL */
/*
* Copyright (c) 1992-2007 by P.J. Plauger. ALL RIGHTS RESERVED.
* Consult your license regarding permissions and restrictions.
V5.03:0009 */
Here is the error message I get:
Unhandled exception at 0x76a9f328 in "program name".exe : Microsoft C++ exception" std::bad_alloc at memory location 0x002cf6cc..
and the code:
// newaop -- operator new[](size_t) REPLACEABLE
#include
#if !_VC6SP2 || _DLL
void *__CRTDECL operator new[](size_t count) _THROW1(std::bad_alloc)
{ // try to allocate count bytes for an array
return (operator new(count));
}
#endif /* !_VC6SP2 || _DLL */
/*
* Copyright (c) 1992-2007 by P.J. Plauger. ALL RIGHTS RESERVED.
* Consult your license regarding permissions and restrictions.
V5.03:0009 */
Are you running 64-bit code on a 64-bit machine with a 64-bit Windows? If not, your process's address space is limited to 2 GB or 3 GB depending on whether it designates itself as "large address space aware." See this MSDN article.
If you're running all 64-bit, then someone else will have to answer the problem.
posted by kindall at 2:08 PM on July 15, 2010
If you're running all 64-bit, then someone else will have to answer the problem.
posted by kindall at 2:08 PM on July 15, 2010
Response by poster: It's a Win32 console application, but I do have 64-bit windows, I just don't know how to create a console application for 64 bit.
posted by amsterdam63 at 2:20 PM on July 15, 2010
posted by amsterdam63 at 2:20 PM on July 15, 2010
How about telling us how you're building the app then -- are you using the IDE or calling the compiler by hand? If the former, then there are build options in the project settings. If the latter then open the VS help reference and look in the index for command line options.
That code snippet is a portion of a standard C++ header and has nothing to do with anything. If that is the location that the IDE brings you to automatically then that simply means that operator new is what's throwing the error, but that's to be expected if you're trying to allocate 3GB in a 32 bit app as that isn't ever going to work.
posted by Rhomboid at 2:49 PM on July 15, 2010
That code snippet is a portion of a standard C++ header and has nothing to do with anything. If that is the location that the IDE brings you to automatically then that simply means that operator new is what's throwing the error, but that's to be expected if you're trying to allocate 3GB in a 32 bit app as that isn't ever going to work.
posted by Rhomboid at 2:49 PM on July 15, 2010
Do you need the entire array in memory? Perhaps try a Windows equivalent of mmap to work with smaller chunks of the array at a time.
posted by Blazecock Pileon at 3:01 PM on July 15, 2010
posted by Blazecock Pileon at 3:01 PM on July 15, 2010
Response by poster: I'm using the IDE, and I will check out the build options. I will also look into using file mapping.
How can I write a console application that is 64 bit?
posted by amsterdam63 at 3:16 PM on July 15, 2010
How can I write a console application that is 64 bit?
posted by amsterdam63 at 3:16 PM on July 15, 2010
This isn't the most intuitive operation (menu names may be slightly different; I'm using VS2010 rather than VS2008 but I doubt it's changed much)
1. Create a new Console app (sounds like you've done this already)
2. Choose "[whatever you named it] Properties" from the Project menu
3. Select "Configuration Properties" in the left-hand-side treeview
4. Click "Configuration Manager..." next to the Platform combo box
5. Click the dropdown for the project row in the "Platform" column, choose ""
6. Select "x64" in the New Platform dialog, click OK
7. Close, OK
Your project should now build as a 64-bit application.
posted by 0xFCAF at 3:32 PM on July 15, 2010
1. Create a new Console app (sounds like you've done this already)
2. Choose "[whatever you named it] Properties" from the Project menu
3. Select "Configuration Properties" in the left-hand-side treeview
4. Click "Configuration Manager..." next to the Platform combo box
5. Click the dropdown for the project row in the "Platform" column, choose "
6. Select "x64" in the New Platform dialog, click OK
7. Close, OK
Your project should now build as a 64-bit application.
posted by 0xFCAF at 3:32 PM on July 15, 2010
Memory mapping doesn't change anything: in order to map a file you need sufficient virtual address space to fit the entire length of the file, even if you only touch a few pages, and unless you use special boot options and enable special compiler flags, you only get 2 GB of usable virtual address space.
posted by Rhomboid at 3:45 PM on July 15, 2010
posted by Rhomboid at 3:45 PM on July 15, 2010
That isn't true, Rhomboid. From the docs on MapViewOfFile:
For files that are larger than the address space, you can only map a small portion of the file data at one time. When the first view is complete, you can unmap it and map a new view.posted by 0xFCAF at 3:59 PM on July 15, 2010
Response by poster: 0xFCAF:
For some reason, I'm not getting the "x64" in the drop down dialog. I have no idea why this would be considering I'm working on 64 bit Vista.
Could this have to do with settings I selected when I installed Visual Studio?
There has to be a way to do this.
posted by amsterdam63 at 4:21 PM on July 15, 2010
For some reason, I'm not getting the "x64" in the drop down dialog. I have no idea why this would be considering I'm working on 64 bit Vista.
Could this have to do with settings I selected when I installed Visual Studio?
There has to be a way to do this.
posted by amsterdam63 at 4:21 PM on July 15, 2010
There has to be a way to do this.
MS does not always make the configuration easy...
Get to the properties page for the solution.
Select the Configuration Manager"
Drop down the menu for "Active Solution Platform".
There should be a selection for x64.
posted by Drasher at 4:32 PM on July 15, 2010
MS does not always make the configuration easy...
Get to the properties page for the solution.
Select the Configuration Manager"
Drop down the menu for "Active Solution Platform".
There should be a selection for x64.
posted by Drasher at 4:32 PM on July 15, 2010
OOPS! sorry...
On the property page, drop down the menu for "Platform"
posted by Drasher at 4:34 PM on July 15, 2010
On the property page, drop down the menu for "Platform"
posted by Drasher at 4:34 PM on July 15, 2010
Well of course you can partially mmap a file. That isn't the point -- it's the same thing as suggesting that he could allocate a 100 MB buffer and read the file in chunks. It doesn't address the question of how you get the full contents of a 3GB file available all at once, for which the only solution is to compile the program in as a 64 bit app.
posted by Rhomboid at 4:46 PM on July 15, 2010
posted by Rhomboid at 4:46 PM on July 15, 2010
Do you actually have Visual Studio or do you have Visual C++ Express? Because Wikipedia informs me that you have to do additional setup to get 64-bit enabled in the IDE.
posted by Green With You at 6:54 PM on July 15, 2010
posted by Green With You at 6:54 PM on July 15, 2010
Think I found it.
Go to Project -> Properties -> Compile
Choose "Advanced Compile Options" at the bottom
Change the "Target CPU" dropdown to x64
Save, compile, etc.
This is 2008 so the menu options should line up better.
posted by wildcrdj at 9:05 PM on July 15, 2010
Go to Project -> Properties -> Compile
Choose "Advanced Compile Options" at the bottom
Change the "Target CPU" dropdown to x64
Save, compile, etc.
This is 2008 so the menu options should line up better.
posted by wildcrdj at 9:05 PM on July 15, 2010
Technically, a 32-bit process running on 64-bit Windows *can* use all 4GB of address space with /LARGEADDRESSAWARE. But you still won't be able to get a single 3GB allocation on the heap, because something (stacks, dlls, locals, etc) will probably be placed somewhere in the middle of memory. But you could probably allocate a bunch of 16MB chunks that added up to nearly 4GB.
posted by IvyMike at 11:04 PM on July 15, 2010
posted by IvyMike at 11:04 PM on July 15, 2010
also consider the STXXL library:
posted by asymptotic at 4:36 AM on July 16, 2010 [1 favorite]
The core of STXXL is an implementation of the C++ standard template library STL for external memory (out-of-core) computations, i.e., STXXL implements containers and algorithms that can process huge volumes of data that only fit on disks. While the compatibility to the STL supports ease of use and compatibility with existing applications, another design priority is high performance.i.e. instead of attempting to load massive quantities of data into an in-memory array an alternative would be to use STXXL in order to load in the array into an "intelligently mmap'd" structure of sorts. i've used this library before and it's remarkably easy to use, however it requires boost.
posted by asymptotic at 4:36 AM on July 16, 2010 [1 favorite]
This thread is closed to new comments.
posted by special-k at 2:06 PM on July 15, 2010