[prog] Passing data between modules

[Conor Daly]

On Mon, Jul 28, 2003 at 11:01:59PM +0100 or thereabouts, Meredydd wrote:
> 
> These modules don't necessarily need to be separate processes. It's 
> possible to make dynamically loadable modules very easily - that way, 
> you don't have shared-memory problems, but you still get to pass around 
> your actual data structures. If you do go for separate processes, I'd 
> use either UNIX or TCP sockets (the latter would help scalability over 
> multiple machines if it ever got that big - depends on the app, of 
> course) to communicate between each process. You may want to keep 
> certain modules running in the background so they can "warm start" 
> rather than the overhead of loading up (again, depends on the startup 
> overhead of the app). Sockets in listen mode help again here.

Last time we looked at dynamic libs programs (about 8 years ago), we ended
up with a 0.25x performance hit.  Whether that was down to inexperience or
inherent difficulties in libs.  We've been statically linking ever since.
It may be time to change but we'll have to learn how...  

Sockets are, again, something we haven't touched.  How much new learning
will be involved will again be a factor.

> One big problem is going to be exchanging data between your modules. 
> It's easiest to keep it in memory (which is why I'd recommend 
> in-process modules), and just pass around pointers to well-thought-out 
> data structures. Otherwise, you're going to have to encode each piece 
> of data you want to transfer into a stream of bytes, and decode at the 
> other end. This is boring and error-prone. read() and write() do a good 
> job for simple structures, but the moment you have a pointer in there, 
> it stops working and you have to start explicitly following them.

This is where the files-in-ramdisk idea came from.  Though there's still
essentially the same problem except the stream is into a file rather than a
socket.
 
> As for language, I think C will do the job - won't be pretty, but it's 
> not a horrible monster, and it has the speed. My personal choice would 
> be Java because, if you do want to go the separate-processes route, 
> there's a very neat set of "serialisation" mechanisms, allowing you to 
> just send an object along a bytestream and have the VM at the other end 
> reconstruct it for you. Of course, the overhead (possibly learning it, 
> VM performance penalties, non-Free stuff if that's important to you) 
> could well be prohibitive, but it's the solution that *I* would go for, 
> with my own skillset. Oh yeah, and it has bounds checking, and 
> NullPointerExceptions are considerably friendlier than SIGSEGVs...

One of the reasons for choice of language is the notion that there'll be a C
compiler on any *NIX/ARCH we buy while there may not be perl/python/java.
Knowing we need to train new recruits in only C and (c)sh is of value also.
Our mission is accumulation and preservation of climate data.  We're in this
for the long haul.  Use of the latest "fashionable" language is not
necessarily the way forward...

Free stuff is important if I can get it into use (oh, and if it Just Works (TM)) 

Conor
-- 
Conor Daly 
Met Eireann, Glasnevin Hill, Dublin 9, Ireland
Ph +353 1 8064276 Fax +353 1 8064247
------------------------------------
bofh.irmet.ie running RedHat Linux 11:13:28  up 2 days, 22:13,  7 users,  load average: 0.01, 0.15, 0.15


**********************************************************************
This e-mail and any files transmitted with it are confidential 
and intended solely for the addressee. If you have received
this email in error please notify the sender.
This e-mail message has also been scanned for the
presence of computer viruses.
**********************************************************************