[prog] Passing data between modules

[Conor Daly]

On Tue, Jul 29, 2003 at 09:47:34AM -1000 or thereabouts, Jimen Ching wrote:
> 
> But this is minor overhead with current CPU performance.  Of course, I
> heard that weather processing software is extremely performance sensitive.
> Even a 5% performance hit could mean a difference between running the
> software for 15 hours vs. 10 hours.  

The sort of processing you're thinking of is Numerical Weather Modelling
where we attempt to predict the future state of the athmosphere.  That
requires huge lumps of processing power.  NWM works by using thermodynamic
equations to prodict the state of the athmosphere a short time ahead (eg 5
minutes) based on the current state (as defined by many observations).  Once
that prodiction is complete, it becomes the start point for a new prediction
for a further timestep.  Rinse and repeat until you get to the time for
which you want a prediction.  The two vital elements to such a system are
speed and accuracy.  Speed since it won't be a lot of use to get a
prediction for tomorrow if it takes 24 hours to produce.  Accuracy since
this is an iterative process and any inaccuracies will propagate and
magnify.  This is the main limiting factor in how far out an NWM can predict
and it's impossible to start with the _exact_ state of the athmosphere since
we only have observations at discrete points.

Our (Met Eireann's) NWM server operates on an area covering most of the
North Atlantic and Western Europe and runs out to 48 hours (HIRLAM if you're
interested).  This runs on an 36 processor IBM RS3000 cluster (with
reinforced floor) and takes about 70 minutes to complete.

A number of organisations run global NWMs which require enormous lumps of
porcessing power.  For example, The European Centre for Mediumrange Weather
Forecasts (ECMWF) runs a global NWM out to 10 days and takes about 2 hours
to do so.  This uses a 960 processor (yep, nearly 1k!) IBM cluster detailed
at 

http://www.ecmwf.int/services/computing/overview/ibm_cluster.html

Anyway, the particular application I'm talking about is not performance
sensitive in the NWM sense.  Rather, there'll be lots of processes running
around cleaning up large volumes of data.  Any one process will be fairly
short-lived (ie. seconds rather than hours).  Ease of use and maintenance will
be more important than being superfast...

> But technology has changed a lot in 8
> years, so I think it is time to give dynamic libs another try.  Using it
> should be a lot easier today than 8 years ago.  ;-)

Yep, I guess so.

Thanks all for the input.  I'll no doubt be back with implementation
questions.

Conor
-- 
Conor Daly 
Met Eireann, Glasnevin Hill, Dublin 9, Ireland
Ph +353 1 8064276 Fax +353 1 8064247
------------------------------------
bofh.irmet.ie running RedHat Linux 08:58:42  up 3 days, 19:58,  9 users,  load average: 0.09, 0.13, 0.10


**********************************************************************
This e-mail and any files transmitted with it are confidential 
and intended solely for the addressee. If you have received
this email in error please notify the sender.
This e-mail message has also been scanned for the
presence of computer viruses.
**********************************************************************