DLL Question

March 15, 2006 10:55 AM

Repeated calls to a user created dll cause stack overflow.

I'm having a really hard tine with this one :( This is all C code, using the Borland Developer 10.

I've created a dll using the .def file method that only contains a function that has a single printf command. Then I've got anther program which calls the fucntion in the dll 100,000 times. After ~60,000 iterations the program stack overflows.

Interestingly, if I change the number of arguments to the function in the dll the less the number of arguments the longer it takes to overflow.

I've tried freeing the library after every call and re-initializing it, but that doesn't seem to help.

This link is almost exactly what I'm seeing:

http://www.swox.com/list-archives/gmp-discuss/2005-November/001954.html

posted by sleslie to Computers & Internet (4 answers total)

Uhh, code sample available? A show of the code which calls the printf() function and the function itself, that is. Without that to look at, it's sheer speculation on anyone's part what might be going wrong.

Can you look at it under the debugger? That should pretty clearly show what happens, if you have the patience and knowledge to interpret what you're seeing. I mean, it's fine if you don't, but the debugger is the right tool here if you can use it properly.

It is possible to write a printf() statement which consumes memory without releasing it due to side-effects. (Or it could be a bug in Borland's stuff, they've had a few).
posted by mdevore at 11:13 AM on March 15, 2006

I haven't done any heavy Win32 development in a few years, but the first thing I would check is if your parameter sizes match. Are you passing a 32-bit int but inside the dll using a 16-bit short?

Also, make sure calling definitions are consitent and the dll is using the stack in the same way your calling program is. I good way to debug is to drop into mixed C/asm mode and make sure the parameter address/size values are consistent.

The Microsoft MSDN archives have a ton of great material to dig around in.
posted by beowulf573 at 11:16 AM on March 15, 2006

Best answer: Sounds like a calling convention mismatch. The DLL is probably using STDCALL mode, as it should, while your declaration of the extern entry point is probably using CDECL mode. In STDCALL mode, the caller is expected to clear the parameters back off the stack after the callee returns; in CDECL mode, the callee clears the parameters off before it returns. You said the behavior changes depending on the number of parameters passed in, which strongly suggests that nobody is clearing off the parameter values, ever. To fix: declare your extern prototypes using STDCALL (the syntax depends on your compiler).
posted by Mars Saxman at 11:51 AM on March 15, 2006

Yes, this screams stdcall vs. cdecl mismatch.
posted by Rhomboid at 4:28 PM on March 15, 2006