[prog] Passing data between modules

[Conor Daly]

Hi all.

I'm planning a suite of data quality control programs in C to run under
*NIX.  I want to keep everything as modular as possible in order to make the
suite flexible and futureproof.  I'm concerned with passing data between the
modules and how to do this in a portable, accessible manner.

Background:

We maintain a database of Irish Weather data reaching back to about 1941 and
growing all the time.  The database is currently running on ingres software
but that may change.  As a consequence, any programs that interact with the
database may need alteration in the future.  This is the first requirement
for a module:  (IO) Something to get data from and to put data in the database.

The data will be subject to a series of tests.  These tests will be
different depending on the weather element involved (eg. wind speed might be
tested against athmospheric pressure at the time while temperature might be
tested against a time-series at the some location).  This is the second
module requirement:  (TEST) Something to conduct X test on Y data.

Once tested, the data may be fixed (or may be flagged for human
intervention).  Again, the fix will depend on the data type, the test and
the outcome of the test.  This is the third module:  (FIX) Something to fix Y data
with Z fix based on X test.

If human intervention is required, data will need to be retrieved,
displayed, fixed and stored.  The IO module is already there, the fix will,
this time, be manual and the display needs to be done.  This is the fourth
module:  (DISPLAY) Display V, W, X data in human readable format.

Finally, there is a need for a manager to oversee the process.  This will
call the various modules as necessary.  This is the final module:  (MANAGER)
Something to oversee the sensible application of various modules.

Why modular?

I want to come up with a system that is flexible and future-proof.  I would
anticipate maintaining a small collection of each module type (eg.
IO-minute, IO-hour,  TEST-timeseries, TEST-spatial,  FIX-interpolate,
FIX-regression) and be able to mix and match modules to form different QC
routines.  I want to be able to try a new test by adding a module.  I want
to be able to adapt to a new database software (DBMS) by changing only the
IO modules.

So what's your problem?

To effectively use modules, each module should be a stand-alone program (I
may be wrong here).  However, the modules must share the same dataset and be
capable of passing messages to and fro.  Messages may be passed by
manipulating a reserved section of the dataset or by using return codes
(somewhat limited) or by using system messaging (not necessarily useful
here).  Is there a good, simple, portable way to do this?  I'm told that
using the same space in RAM is not easy and is OS/ARCH dependant.  I thought
of using files for transfer but this involves a performance hit, even if a
RAMdisk is used.  Obviously the RAMdisk will reduce the performance hit but
there will still be the RAMdisk-RAM-RAMdisk transfer hit for each module.  

Am I mad?  In over my head?  We have a small team of developers who are used
to writing monolithic applications for individual tasks.  Generalisation of
programs has not caught on here yet!  This application will be additional to
our regular workload so the modular format may actually be useful in that we
should be able to see some results fairly quickly (ie. put together one each
of IO, TEST, FIX modules) while building up the repertoire of modules over
time.

Conor
-- 
Conor Daly 
Met Eireann, Glasnevin Hill, Dublin 9, Ireland
Ph +353 1 8064276 Fax +353 1 8064247
------------------------------------
bofh.irmet.ie running RedHat Linux 08:55:04  up 1 day, 19:54,  4 users,  load average: 0.10, 0.06, 0.01


**********************************************************************
This e-mail and any files transmitted with it are confidential 
and intended solely for the addressee. If you have received
this email in error please notify the sender.
This e-mail message has also been scanned for the
presence of computer viruses.
**********************************************************************