Open Data Interface, ODI
ESTEC/Contract No. 21964/08/NL/AT
Monthly Progress Report
ODI-MR-5-08/09
Period: 2009-02-01--02-28
Author: Lars Eliasson
1. Progress status
A careful analysis of the metadata of CDF, ISTP, and PRBEM was carried out in order to define the structure and contents of the ODI database. A document was written to capture this and it will be part of a technical note.
A working meeting was held at DH Consultancy, Leuven, Belgium, on Wednesday Feb. 11. Peter Wintoft and Daniel Heynderickx took part in the meeting.
The first Progress Meeting was held at DH Consultancy, Leuven, Belgium, on Thursday Feb. 12. See the separate minutes from the meeting.
A prototype metadata model has been defined that captures the metadata of CDF/ISTP/PRBEM. Three different data sets that span the needs of the ODI system are used: Dst index, ACE real time solar wind data, and Rosetta SREM data.
D. Heynderickx had a very efficient and successful visit to ESTEC 23-24 Feb.
H. Evans has explained how the different functionalities of SEDAT (Java GUI, perl scripts, C codes, IDL codes, ...) interact, and which parts that need to be modified. The GUI calls a set of perl scripts, some of which in turn call a set of stand-alone C executables to extract information about the datasets from STPDF. In the perl scripts, a distinction is already made between SYSTEM data sets and USER data sets. As ODI will only apply to the SYSTEM datasets, the modifications to be made can be pinpointed in the perl codes.
D. Heynderickx managed to run one of the perl scripts on his laptop in command line mode, and also on firefox in command line mode. The call on both systems is identical, which means that it can be developed locally.
The calls to the STPDF functions for SYSTEM datasets need to be replaced by calls to MySQL. D. Heynderickx will investigate whether this can be done directly from perl (through a MySQL or ODBC driver); this would be the preferred solution, as it would avoid having to write, e.g. a php script to handle the MySQL call.
D. Heynderickx has installed the mysqlPP driver on Windows Activeperl. It worked straight away! He did some sample queries on the SEPEM database, worked fine. On Linux, the driver is mysql (without PP), apparenlty that is installed on firefox: He ran test script there, worked fine too! He can use the SEPEM database as a test case, and start on updating the perl scripts. When the ODI database structure is defined, he should only have to update the SQL statements.
We also tried out the IDL DataMiner on firefox. A simple IDL procedure that does a scatter plot of two SEPEM data columns, worked fine! On laptop, it takes two minutes to complete, on firefox it took two seconds... for 120,000 data records.
Several URs are satisfied already (or, at least, it has been demonstrated that the functionalities can be implemented):
- 120,000 records in 2 seconds satisfies "at least 50000 records in less than a second"
- min, max values of table columns can be retrieved with my test procedure, including epoch, which gives the epoch range
- figured out how to obtain the number of records in a table using Dataminer (sounds simple, but required some casting of variables).
Web page http://www.lund.irf.se/odi
WP 100 Core Database System
A draft of the ECSS-E40 documents (D1) is prepared including e.g. Software System Specification.
A draft of the URD has been written.
A draft of the Database Technical Note (D3) has been written.
User manual for the database administrator (D2) will be produced.
Database system and source code (S1) prototype is produced.
WP200 Database
Work with the populated database and source code has started and prototypes have been generated for parts of the system.
MySQL database at T0 + 3.5 according to the schedule
Updated ECSS-E40 document at T0 + 3.5 according to the schedule
Updated Technical Note at T0 + 3.5 according to the schedule
WP300 SAAPS interface
Start T0+3.5 according to the schedule
Updated ECSS-E40 document at T0 + 6 according to the schedule
Software package ready at T0 + 6 according to the schedule
Technical note ready at T0 + 6 according to the schedule
WP400 SEDAT interface
Start T0+3.5 according to the schedule
Updated ECSS-E40 document at T0 + 6 according to the schedule
Software package ready at T0 + 6 according to the schedule
Technical note ready at T0 + 6 according to the schedule
WP500 SPENVIS interface
Start T0+3.5 according to the schedule
Updated ECSS-E40 document at T0 + 6 according to the schedule
Software package ready at T0 + 6 according to the schedule
Technical note ready at T0 + 6 according to the schedule
WP600 Maintenance and support
Start T0+6 according to the schedule
WP700 Administration/management
Monthly progress reports
Final Report ready at T0 + 12
2. Problem areas/reasons for slippages
None
3. Events anticipated to be accomplished during next reporting period
None.
4. Status report on all long lead or critical delivery items
None
5. Action items
The following action items were decided during the PDR teleconference and are all closed.
Action Item PDR-1 Hugh Evans shall send the SEDAT document "RAL_SED_UD_0001" to Peter Wintoft.
Action item PDR-2 Hugh Evans shall contact Paul Buehler/PSI about including the integration time in the SREM datasets.
Action Item PDR-3 Daniel Heynderickx shall clarify the IMP datasets.
Action item PDR-4 Hugh Evans will send comments on Sections 5 and 6 to Peter Wintoft by email.
Action Item PDR-5 Peter Wintoft shall investigate and specify a standard set of metadata and attributes to be included with the indices datasets.
Action Item PDR-6 Hugh Evans shall send details on the SEDAT code to Daniel Heynderickx.
Action Item PDR-7 Hugh Evans shall arrange access to the external servers at ESTEC.
The following action item was decided during the PM1
Action Item PM-1 Hugh Evans shall investigate where the system will be hosted at ESTEC, allocate space, and arrange for the necessary licences and servers to be set up at ESTEC.
6. Milestone payment status
Advance payment has been paid to IRF and subcontractor. Progress payment expected after PM2.
7. Expected dates for major schedule items
The second Progress Meeting will take place May 18 or 19, 2009 at ESTEC.