The implementation and running of the whole integrated system

WP4 project work package has very closed interactions with WP2 work package "Real time / Off line experiment with different basin scale integrated monitoring systems", whose main purpose is to produce and archive the basic data sets for the inter-comparison exercise.

The implementation and running of the whole integrated system is split into 3 items:

• Implementing and configuring the Opendap catalogues,
• Generating products and exposing them on Internet,
• Operating the whole integrated system.

• Implementing and configuring the Opendap catalogues

  Each model partner implemented tan Opendap aggregation server (CatalogAggServer/0.6 which uses Java1.4, see http://www.unidata.ucar.edu/projects/THREDDS/tech/ aggServer/AggServerStatus.html) to make data available on Internet.

  The main purpose of this distribution server is to read netCDF data files and allow the "aggregation" of multiple datasets into one (virtual) dataset. The Aggregation Server is currently in Beta release. We will mainly used it for temporal aggregation, creating thus a time serie data set accessible by a unique URL to the users - with or without subsetting characteristics (like geographical extraction, one variable) -. It was also known at the beginning of the project that the chosen software was currently in a beta version with known drawbacks (cf. uploading and downloading performances, e.g. the need to reboot the server for taking account new datasets) but:
  

  • the software should be released officially and bugs and bad issues fixed,
  • the aggregation capabilities was worth to use to create a time series product
  
  A typical ocean bulletin from MERSEA models
  

  Model products are no more delivered as they are produced,
  i.e., an ocean bulletin format covering in time 1 week, 2 week or 4 weeks with hindcast, nowcast and forecast data (daily files),
  and delivered every day or once a week, depending on the ocean system of reference.

  The concept of the the best estimate time series has been created,
  it corresponds to the best that each ocean system can produce.
  It is decided by each system freely, that is,
  some systems give the daily averages, others a snapshot.

  Granularity of file is a file per configuration model, per area, per parameter or per class, per day
  ([model]_[config-grid]_[target_area]_[parameter]_[bulletin_date]_ [file_date].nc)

  The same is done for the sixth day forecast time series (weekly data).

  
  The aggregation configuration
  Conception of the time series for ocean system under daily versus weekly production.
  
  

• Generating products and exposing them on Internet

  As part of WP2, each ocean system partner :
  
  

  • adapted their original integrated monitoring system to the needs of the real-time exercise following the intercomparison framework directives (i.e., developed a post-processing on their routine operations), and
    
    
  • set up an operational environment to routinely produce the series of products dedicated to MERSEA strand-1 objectives, and delivered their data via Opendap data servers with the following basic URLs :
    
    
    • Foam UK ocean system (July 1, 2003 - May 31, 2004)
      http://user:password@www.nerc-essc.ac.uk:9090 (password protected)
      
      
    • Mercator French ocean system (June 1, 2003 - May 31, 2004)
      http://user:password@opendap.mercator-ocean.fr (password protected)
      
      
    • MFS Italian ocean system (June 1, 2003 - May 31, 2004)
      http://www.sincem.unibo.it:8080/dodsC/ (public access)
      
      
    • Topaz Norway ocean system (June 1, 2003 - May 31, 2004)
      http://mersea.nersc.no/dodsC/ (password protected)
      
      
    • HYCOM US ocean system (May 28, 2003 - April 21, 2004)
      http://user:password@www.nerc-essc.ac.uk:9090 (password protected)
      
      
  

  To help you use the Opendap and Las e-technolgy, see the Opendap Las quick view user guide (.ppt, 7.6mb)

  
  
  

• Operating the whole integrated system

  The whole integrated sytem has to run and be maintained during the intercomparison period. The real-time experiment goes as follows:

  1. Every week the four forecasting systems produce the required NetCDF files, expose them via their Opendap aggregation server and upgrade their configuration with the previous week analyses and the sixth day forecasts.
     
     
  2. At the same time, each forecasting system produce 'an Opendap/Las synchronisation file called 'notification'. This file is a formatted 'xml' file, which is put on a ftp site (identified by its release date) to communicate the entrance of new data to the manipulation/visualization software (Las). There is an ftp site for each forecasting system.
     
     
  3. Las synchronisation is performed every day of the week: Consequently to this daily interrogation, the new daily fields are automatically charged on the visualization software (LAS), ready for their manipulation and visualization on Internet.
     
     
  4. Once the intercomparison exercise started, we had an important and urgent issue to have better visibility on data products availability and environment, and track in time time series evolutions and report on ocean operational systems versions. To help have a clear view of what is going on, we defined and implemented two new features at the end of the project: a data status and history files. Both features are made available on the operational web site.
     History files are text files released by each system that describe changes away from the systems description (cf. model description). Such file lists all changes that have occurred during the inter-comparison time period June2003- March 2004 in all the systems.
     
     
  
  
  
• Manipulating and visualising products

  To achieve MERSEA strand-1 goals and offer a unique vision, a European GODAE www server has been developed (http://strand1.mersea.eu.org, this web site ). It provides an overall information on:

  • the project
  • the operational ocean forecasting systems
  • the definition of intercomparison framework
  • the implementation and running of the whole integrated system
  • the data access (restricted access to Opendap and Las)
  • the visualisation facilities and a Las tutorial
  • a discussion area (restricted access), a place for exchanging experience before publication (intercomparison results and final assessments available there).

  The European GODAE portal is made of:

  • classical WWW technology (HTML pages) to present the system, maps and results,
  • Opendap/Las/Ferret middleware to distribute, manipulate and visualise the data on Internet.

  A central federating LAS connected to individual OpenDAP data servers has been implemented, one frozen for public access (http://publiclas.mersea.eu.org , public access) and one regularly operated as a GODAE portal to European model products (http://las.mersea.eu.org, restricted access). It federates and unify all individual Opendap catalogues (creating a 'virtual' catalogue of the relevant products - class 1 - from the different data servers distributed over Internet. In short, it allows to visualize outputs from the four ocean forecasting systems. One can chose the model, region parameter, way of plotting, etc.. With a Live Access Server (LAS), you can discover, browse and access data products.

  

  the inter-operable window to do the request (with possibility there to customise the plot)

  the output window (with information at the top of the page on where the user are and what he asked for)

  With a LAS, Discover, Browse ... and Access data products

  The central federating Live Access Server (LAS) enables inter-operability / inter-activity functionalities for distributing ocean data products such as:

  Select the product of interest (i.e.. areas, model, time series, parameter) Sub- setting capabilities (geographical area, x line , y line and more) Aggregation in time (cf. Hovmuller plots) Browsing (on the fly graphics) [or Download (various output formats - however better performance from direct Opendap access -) and therefore not configured] Single data set mode or Inter-Comparison (only 2 at a time, any data - and valid for distributed data on any regular grid) Customize your plot / Define your plot criteria (color scale, min, max, scaling, multiplication factor, etc.)

  Las is a powerful tool to observe/understand/analyse/intercompare the oceans,

  • for outreach (demonstration and education),
  • for real-time and complementary expertise (first tool to check, on the fly, an idea and see if it's worth going on), and
  • for attracting new type of users (a place to shop at least see what's available or what can be done they ask for a service).
  
  

  To help you use the Las, see the Las user guide made available on the MERSEA product web site (http://www.mersea.eu.org/html/strand1/las_tutorial.html).

  Nevertheless, it is not useable for plotting data on large areas and long time scales. For instance, Considering the number of parameters to look at and some others (e.g. 2D (latitude, longitude) monthly mean plots of each Class 1 and 2D (along-track, vertical) Class 2, for the whole area or predefined xyz boxes, eddy kinetic energy plots) and to produce a huge amount of outputs, Las is not the right tool but automatic procedures interrogating directly and off-line the Opendap. Generated plots are stored as GIF format files and made available to the team on MERSEA product web site - directly or in power point presentation files -, discussion area (http://bulletin.mersea.eu.org/html/secure/index.html - password protected).