Its killing me! Allocating memory in software!

colin · 06-25-2009, 10:38 PM

I cant continue writing my frontend until I get this fixed! It runs off of a message system basically. Each file has a receiving function that gets messages and responds to them, this way it's modular and sort of plugin based. The problem is, I cant pass data between them!

Anybody who knows software, here's where it gets DEEP.

When I want to send a "broadcast" -- a message -- I call the function Broadcast with a Message(string) and some Data(pointer). It gets added to a queue and the pointer gets destroyed.

Thats all fine, except by the time the message gets to the front of the queue, the pointer is destroyed! So, I want to COPY all the data (regardless of what type) into memory somewhere and then use that pointer instead.

How the hell do I do that?! I tried using LocalAlloc, I tried using weird typecasting, I tried so many things! The function has to work for strings, integers, lists, and any generic objects (HID_Device objects too).

Im developing in Delphi but that shouldn't matter.

NSFW · 06-25-2009, 11:04 PM

I picture it working like this:

* allocate the data structure
* put it into the queue
* pull structures off the queue
* pass structure around to each plugin
* when all plugins have processed the structure, release the memory

What is it about your code that's different from what I just described?

2k1Toaster · 06-25-2009, 11:33 PM

I have never been able to get global (system wide) memory allocation to work properly. Is there a reason you arent using COM or something?

ClockWorK · 06-26-2009, 10:18 AM

I could be way off here. If so, sorry.

GlobalAlloc only returns a handle to a memory location. To get a true pointer, you need to call GlobalLock after GlobalAlloc.

Also, that pointer is only useful within your process. Other processes cannot directly access that memory location in your process.

Edit: Also, for the frontend I'm working on (DriveLine), my plan was that the plugins would communicate only with the frontend, via SendMessage. Plugins can set or retrieve certain values based on the params. So one plugin can send a command to the frontend telling it the current speed. Then, another plugin can query the frontend for the speed.

NSFW · 06-26-2009, 01:49 PM

ah... if you're dealing with multiple processes it gets a bit more complicated than what I described. In that case, used named pipes or something.

colin · 06-26-2009, 02:44 PM

Quote: Originally Posted by NSFW

I picture it working like this:

* allocate the data structure
* put it into the queue
* pull structures off the queue
* pass structure around to each plugin
* when all plugins have processed the structure, release the memory

What is it about your code that's different from what I just described?

Nothing. The problem lies in putting it into the queue. If a function allocates memory, as soon as that function completes, the memory is gone. You can't keep it globally because what if that function gets called 100 times and each time puts something into a queue? you need different structures each time, and you need to keep them!

Quote: Originally Posted by 2k1Toaster

I have never been able to get global (system wide) memory allocation to work properly. Is there a reason you arent using COM or something?

COM has unnecessary overhead for my application. Using it simply to pass messages between forms in one application is silly.

Quote: Originally Posted by ClockWorK

I could be way off here. If so, sorry.

GlobalAlloc only returns a handle to a memory location. To get a true pointer, you need to call GlobalLock after GlobalAlloc.

Also, that pointer is only useful within your process. Other processes cannot directly access that memory location in your process.

Edit: Also, for the frontend I'm working on (DriveLine), my plan was that the plugins would communicate only with the frontend, via SendMessage. Plugins can set or retrieve certain values based on the params. So one plugin can send a command to the frontend telling it the current speed. Then, another plugin can query the frontend for the speed.

GlobalAlloc, LocalAlloc and HeapAlloc do return handles, however if you use LPTR or LMEM_FIXED in your flags, you automatically get a pointer because the memory is not movable. I only want it within one process.

Your sendmessage idea is good, except what about when the information you're sending isn't a string or integer or basic type? I need to be able to call ONE function (sendmessage for example) with any type of data. It has to be able to process Integers, Strings, Chars, Pointers, StringLists (delphi thing), HID_Devices, TSongs (custom classes).

ClockWorK · 06-26-2009, 03:31 PM

If you don't call GlobalLock, you will never have a true pointer. Seriously. But maybe you never even need a real pointer. If you call these at the load of your app, and store them in global variables, I don't see why it wouldn't work.

AppStart:
Allocate Memory and store handle into global variable.
Lock Memory and store pointer into global variable.

AppEvents:
Use the global variables to access the allocated memory.

AppClose:
Unlock and free the allocated memory

malcom2073 · 06-26-2009, 04:27 PM

Quote: Originally Posted by colin

I need to be able to call ONE function (sendmessage for example) with any type of data. It has to be able to process Integers, Strings, Chars, Pointers, StringLists (delphi thing), HID_Devices, TSongs (custom classes).

Does delphi not have a Variant type?

Curiosity · 06-26-2009, 09:46 PM

GlobalAlloc and LocalAlloc are the same in current operating systems, so use LocalAlloc just because it's easier. But yeah, your problem is local vs. global variables. locals are allocated on the stack unless declared static. globals are outside the scope of the function. That's pretty much universal for any language.

NSFW · 06-26-2009, 10:19 PM

Quote: Originally Posted by colin

Nothing. The problem lies in putting it into the queue. If a function allocates memory, as soon as that function completes, the memory is gone. You can't keep it globally because what if that function gets called 100 times and each time puts something into a queue? you need different structures each time, and you need to keep them!

So that is what's different from what I described.

You need a way to allocate memory and control when it's released. In C++ you'd use "new" and I'm guessing Delphi has something similar. If you use Local/GlobalAlloc, put the handle into your queued structure, so that the code that pulls structures from the queue can also pass the handle to Local/GlobalFree (I think that's what it's called...) after passing the structure to all of the plugins.

NSFW · 06-26-2009, 10:22 PM

Quote: Originally Posted by colin

I need to be able to call ONE function (sendmessage for example) with any type of data. It has to be able to process Integers, Strings, Chars, Pointers, StringLists (delphi thing), HID_Devices, TSongs (custom classes).

Lots of ways to skin that cat. Declare a base TPluginMessage, for example, and pass those around. Derive types from TPluginMessage that add the extra stuff you need. I don't know if Delphi has run-time type information; if it does not, TPluginMessage needs a field that indicates which derived type it really is.

So:

if (currentMessage.Type == MessageType.SongMessage)
{
TSongMessage songMessage = (TSongMessage) currentMessage;
Play(songMessage);
}

...or whatever. I haven't done much Delphi and that little bit was several years ago, so you'll have to translate some of the syntax.

colin · 06-27-2009, 09:47 PM

Quote: Originally Posted by ClockWorK

If you don't call GlobalLock, you will never have a true pointer. Seriously. But maybe you never even need a real pointer. If you call these at the load of your app, and store them in global variables, I don't see why it wouldn't work.

AppStart:
Allocate Memory and store handle into global variable.
Lock Memory and store pointer into global variable.

AppEvents:
Use the global variables to access the allocated memory.

AppClose:
Unlock and free the allocated memory

From MSDN:

Quote:

GMEM_FIXED
0x0000
Allocates fixed memory. The return value is a pointer.

GMEM_ZEROINIT
0x0040
Initializes memory contents to zero.

GPTR
0x0040
Combines GMEM_FIXED and GMEM_ZEROINIT.

Quote: Originally Posted by malcom2073

Does delphi not have a Variant type?

Yes, but you need to convert things to a variant first I think? Its hard to use, really.

Quote: Originally Posted by NSFW

So that is what's different from what I described.

You need a way to allocate memory and control when it's released. In C++ you'd use "new" and I'm guessing Delphi has something similar. If you use Local/GlobalAlloc, put the handle into your queued structure, so that the code that pulls structures from the queue can also pass the handle to Local/GlobalFree (I think that's what it's called...) after passing the structure to all of the plugins.

Delphi has a NEW function, but the problem is you have to know what you're creating. You cant specify a size and expect Delphi to create the memory.

Quote: Originally Posted by NSFW

Lots of ways to skin that cat. Declare a base TPluginMessage, for example, and pass those around. Derive types from TPluginMessage that add the extra stuff you need. I don't know if Delphi has run-time type information; if it does not, TPluginMessage needs a field that indicates which derived type it really is.

So:

if (currentMessage.Type == MessageType.SongMessage)
{
TSongMessage songMessage = (TSongMessage) currentMessage;
Play(songMessage);
}

...or whatever. I haven't done much Delphi and that little bit was several years ago, so you'll have to translate some of the syntax.

No way. I did try that however, I made 10 different Broadcast functions each with different parameters, it didn't work and was a bigger pain than this is.

NSFW · 06-27-2009, 11:59 PM

Quote: Originally Posted by colin

Delphi has a NEW function, but the problem is you have to know what you're creating. You cant specify a size and expect Delphi to create the memory.

But you do know what you're creating when you allocate the memory, right?

Quote:

No way.

No way, what?

colin · 06-28-2009, 08:17 AM

Quote: Originally Posted by NSFW

But you do know what you're creating when you allocate the memory, right?

Sort of. The program runs off of this broadcast system, so it will broadcast hundreds, if not thousands of messages. I can't say that every similar message will share a data type. Sometimes, the "INIT_DONE" broadcast will have a string attached to it, sometimes it will have an integer. I need a function that doesn't care. Just copies it blindly and sends it back blindly too.

Quote: Originally Posted by NSFW

No way, what?

I tried that, it was a pain and it didnt work right. The problem is just that there are SO many broadcasts, I can't be doing comparisons for every message because there are too many. It will also be harder in the future because if I don't anticipate a data type, I can't pass it later on for plugins. For example, HIDDevice -- I would have completely forgotten about that. What about when I finish writing the CD player units and stuff, I'll probably have to pass over device handles and CD objects. Can't predict that.

TMF · 06-28-2009, 08:24 AM

Follow this link. http://delphi.about.com/od/windowssh...m_copydata.htm

This shows how to use the special Windows Message WM_COPYDATA to pass data from one program to another.

User Name		Remember Me?
Password