Bookkeeping Implementation

Sebastien Ponce March 12, 2004

Internal Note

Issue :  1
Revision :  0
Reference :  -
Created :  September 1, 2003
Last modified : March 12, 2004

Internal Note Issue: 1 CONTENTS

Reference:  -
Revision:  0
Last modified: March 12, 2004

Contents

0.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1 Data description and access 4

1.1 Persistent Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.1 The Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.2 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.3 How to generate unique IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2 The transient Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Access to data from Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.3.1 StatusCode and Handle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.2 Main Bookkeeping data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.3 Event types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3.4 Replicas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3.5 A word on the default implementation . . . . . . . . . . . . . . . . . . . . . 11

1.4 Access to data from python and XMLRPC . . . . . . . . . . . . . . . . . . . . . . 11 1.4.1 Python objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.2 Python interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.4.3 XMLRPC interface usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2 Server infrastructure 15 2.1 Server implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2 Server deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Booking of data 17

4 The end-user web interfaces 18

Internal Note Issue: 1 CONTENTS

Reference:  -
Revision:  0
Last modified: March 12, 2004

0.1 Introduction

This document describes the implementation of the bookkeeping tools and services in LHCb.

We first present the persistent data model, which describes the tables and columns declared in the database to store the data.

We then deal with the description of the transient data model and the way applications can access to the data. This includes the definition of abstract interfaces for both data access and data writing.

Finally we present the python and web tools deployed around the main system for easy access by the end user. We also present some tools that were used to transfer, create or update the main database.

Before jumping into the details, here are some prerequisites :

• The core system was implemented in Java. Thus the interfaces described in the “Data Management” document were adapted to this language. Details are given in section ??.

• Since Gaudi is only available in C++ at this time, the system has no link to it. In particular, the interfaces do not inherit from the Interface abstract interface and the services do not inherit from the IService abstract interface. However, the needed Gaudi components were recreated in Java.

• The database used in the implementation is not precisely defined in the rest of this chapter. This is because it may be any relational database having a JDBC interface. MySQL and Oracle are the two databases that were actually tested and the final system is running mainly under Oracle.

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

Chapter 1

Data description and access

1.1 Persistent Data

1.1.1 The Data Model

The persistent data model (Figure 1.1) describes the design of the database used to store bookkeeping data. This database is supposed to be relational. Thus the description is the description of the tables used and of their colums. Each bubble of the figure is a table and the list of column is given inside each one. The associations drawn as lines show where IDs are supposed to match. As an example, the Job ID in Job and in JobOptions match. The dotted line indicates that on the FilesParam side, the EvtType ID is actually stored in the Value column when the Name column contains “EventtType” and not as a separate column.

Figure 1.1: The design of the database tables

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

The whole design was intended to build thin tables on the database side by removing all columns that would not be absolutely necessary for all cases. Lines in one of the parameter tables replace these columns. As an example, the number of event of a given file is only relevant in the case of a file containing data (not in the case of a log file). Thus, is was not added in the Files table but will be a line in the FileParams table with name ”NbEvent” and value the actual number of event of the associated file.

This architecture also allows taking advantage of the indexing capabilities of the underlying database to search for objects. As an example, indexes should be created in the InputFile table, based on the file ID and in the Files table based on the Job ID. This allows an efficient browse of the hierarchy of jobs and files.

In principle, every object in the data model maps to a table and every relation between objects to an ID. There are however some exceptions:

• The Type and Quality objects have actually no members. These are stored as parameters. The names of these parameters are ”Name” and ”Version” for Type and ”Group” and ”Flag” for Quality.

• Types are not mapped to a table since this table would have had no column except the ID one. They only exist through IDs in other tables. As an example, the Type ID is used in the TypeParams and in the Files tables.

• Qualities are not mapped to a table. However, the table would have had two columns: one for File ID and one for Quality ID. For optimization purposes, it was decided that the File ID would be replicated in every line of the QualityParams table and that the Qualities table would be suppressed. However, all the quality parameters defined for a given file stay grouped by the Quality ID and will map to parameters of the same Quality object in the transient world.

• The many to many link that exists for input files requires a specific link table: InputFiles. Each line of this table describes one link between a job and a file, which is seen as an input file of this job. Note that the output files relation is a regular relation and is thus mapped to an ID, namely the Job ID in Files.

1.1.2 Constraints

The ”Type” column of table ”JobParams” cannot be filled with any value. Table 2 give a list of the allowed values and their meaning.

Value  Meaning

”Info” This parameter gives information on the conditions under which

the job ran

”Script” This is a script that was launched before the job was run.

”Environment variable” This is an environment variable that was defined before the job

was run.

”Error” This is a parameter created by the book process when it crashes. It stores what could not be put anywhere else du to the crash.

Many parameter tables are defined in this design that may keep most of the relevant information. It is thus important to agree on the name of the parameters to be able to retrieve the information. Here are the parameters and name that are defined by default. More parameters may be defined in the future. Note that in the case of the ”JobParams” table, these are ”Info” parameters.

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

Table  Name  Description  Needed

JobParams

Location  The laboratory where the job was executed  No
Host  The machine where the job was executed  No

Run The run number for data files that have one

No

Seed1 First random number seed used for this job, if any

No

Seed2 Second random number seed used for this job, if any

No

XmlDDDBVersion Version of the geometry database used for this job, if the XML database is used

No

DbaseVersion Version of the geometry database used for this job, if the old dbase is used

No

Name  The job name.  By convention

XXXXXXXX YYYYYYYY Z where XXXXXXXX is the production number, with 8 digit and prefixed with 0s, YYYYYYYY is the job number (same convention) and Z is the step number

No

Production  The production number  No
Job  The job number in the production  No
ProgramName  The name of the program that was run  No
ProgramVersion  The version of the program that was run  No

FileParams EvtType The type of events contained in this file

(Only in case of a data file). Only an id is given here. The description of the event type can be found in the EvtType table

No

EvtStat The number of events in this file (Only in case of a data file)

No

Size  The size of the file in KB  No

TypeParams Name The name of this type. Example: ”RAW”,

”DST” or ”Log”

Yes

Version The version of this type. Usually a number.

Yes

Description  A short description of this type.  No

QualityParams

Group  The group concerned by this Quality.  Yes

Flag The quality by itself. Allowed values are ”Good”, ”Fair”, ”Bad” and ”Not checked”.

Yes

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

1.1.3 How to generate unique IDs

As you can see in the tables of Figure 1.1, many IDs are defined in the database schema. Each Job, File, Type and Quality can thus be identified by a single and unique number. The problem is to ensure the uniqueness of this number.

Since every database has its own way of solving this problem (if any), it was decided to do it by hand via the use of an additional table called ”NextBookkeepingIDs”. This table has only two columns: ”TableName” and ”ID”. It contains only four lines (with TableNames ”Job”, ”File”, ”Type” and ”Quality”), which keep track of the next ID that should be used when creating a Job, a File, a Type or a Quality. The IBookkeepingEditor interface implementations have to take care that this table is maintained up to date. Note that the table contains a fifth line called “IndexName”. This index is used for the bookkeeping updates, see section ??.

1.2 The transient Data Model

A preliminary version of the transient data model was exposed in the document ”Data Management”. We describe here its current implementation. Lsiting 1.1 shows the implementation of every class used in this model. The code should be pretty obvious to understand.

p u b l i c c l a s s Job {
p u b l i c Job ( i n t jobID , S t r i n g configName ,
S t r i n g c o n f i g V e r s i o n , Date date ) ;
p u b l i c i n t jobID ( ) ;
p u b l i c S t r i n g configName ( ) ;
p u b l i c S t r i n g c o n f i g V e r s i o n ( ) ;
p u b l i c Date date ( ) ;

}

p u b l i c c l a s s F i l e {
p u b l i c F i l e ( i n t f i l e I D , i n t jobID ,
i n t typeID , S t r i n g logName ) ;
p u b l i c i n t f i l e I D ( ) ;
p u b l i c i n t jobID ( ) ;
p u b l i c i n t typeID ( ) ;
p u b l i c S t r i n g logName ( ) ;

}

p u b l i c c l a s s Type {
p u b l i c Type ( i n t typeID , S t r i n g name , S t r i n g v e r s i o n ) ;
p u b l i c i n t typeID ( ) ;
p u b l i c S t r i n g name ( ) ;
p u b l i c S t r i n g v e r s i o n ( ) ;

}

p u b l i c c l a s s Quality {
p u b l i c Quality ( i n t f i l e I D , i n t qualityID ,
S t r i n g group , S t r i n g f l a g ) ;
p u b l i c i n t qualityID ( ) ;
p u b l i c i n t f i l e I D ( ) ;
p u b l i c S t r i n g group ( ) ;
p u b l i c S t r i n g f l a g ( ) ;

}

p u b l i c c l a s s Param {
p u b l i c Param ( i n t ID , S t r i n g name , S t r i n g value ) ;
p u b l i c i n t ID ( ) ;
p u b l i c S t r i n g name ( ) ;
p u b l i c S t r i n g value ( ) ;

}

p u b l i c c l a s s JobOption extends Param {
p u b l i c JobOption ( i n t jobID , S t r i n g name ,
S t r i n g r e c i p i e n t , S t r i n g value ) ;

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

p u b l i c S t r i n g r e c i p i e n t ( ) ;

}

p u b l i c c l a s s TypedParam extends Param {
p u b l i c TypedParam ( i n t jobID , S t r i n g name ,
S t r i n g value , S t r i n g type ) ;
p u b l i c S t r i n g type ( ) ;

}

p u b l i c c l a s s EvtType {
p u b l i c EvtType ( i n t id , S t r i n g desc ,
S t r i n g prim , S t r i n g decay ) ;
p u b l i c i n t ID ( ) ;
p u b l i c S t r i n g d e s c r i p t i o n ( ) ;
p u b l i c S t r i n g primary ( ) ;
p u b l i c S t r i n g decay ( ) ;

}

Listing 1.1: Transient Data Model

1.3 Access to data from Java

The interfaces are actually divided in two groups : the ones providing read access to the data and the ones providing write access. The first ones have their name ending with “Info” while the second ones have their name ending with “Editor”.

The data were split into three pieces : the event types, the replica catalog and the rest of the bookkeeping. Thus, there are 6 interfaces : IEvtTypeInfo, IReplicaInfo, IBookkeepingInfo, IEvt-TypeEditor, IReplicaEditor and IBookkeepingEditor.

On top of the interface themselves, two helper classes were defined in the (tiny) Java part of Gaudi : StatusCode and Handle. They are the only objects returned by the methods of the interfaces.

1.3.1 StatusCode and Handle

The StatusCode class has the same interest in Java that it had in the C++ world: it encapsulates a status and allow to query whether the operation is success or not. The interesting part of its Java definition is shown on Listing 1.2.

p u b l i c c l a s s StatusCode extends Object  {
static p u b l i c StatusCode FAILURE = new StatusCode ( 0 ) ;
static p u b l i c StatusCode SUCCESS = new StatusCode ( 1 ) ;
p u b l i c boolean i s S u c c e s s ( ) ;
p u b l i c boolean i s F a i l u r e ( ) ;

};

p u b l i c c l a s s Handle extends StatusCode {
static p u b l i c Handle FAILURE = new Handle ( 0 , null ) ;
static p u b l i c Handle SUCCESS = new Handle ( 1 , null ) ;
p u b l i c Object o b j e c t ;

};

Listing 1.2: Definition of class StatusCode and Handle

Besides StatusCode, a new return type has been defined that extends it by adding a member of class Object. This allows a function to return at the same time the status and the return value. This new type was called Handle. The Handle class is heavily used in the IBookkeepingInfo and IBookkeepingEditor interfaces.

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

1.3.2 Main Bookkeeping data

Listings 1.3 and 1.4 gives the definition of the IBookkeepingInfo and IBookkeepingEditor interfaces in Java. Some functions are returning a Handle that encapsulates the actual return value. The return value is either a single object (see type definitions in section 1) or a java.util.Vector object containing such objects.

p u b l i c interface IBookkeepingInfo {
p u b l i c Handle j o b s ( ) ;
p u b l i c Handle j o b s ( Date date1 , Date date2 ) ;
p u b l i c Handle j o b s ( S t r i n g configName , S t r i n g c o n f i g V e r s i o n ) ;
p u b l i c Handle j o b s ( F i l e i n p u F i l e ) ;
p u b l i c Handle job ( F i l e o u t p u t F i l e ) ;
p u b l i c Handle job ( i n t id ) ;
p u b l i c Handle f i l e s ( ) ;
p u b l i c Handle f i l e s ( S t r i n g SQLQuery ) ;
p u b l i c Handle f i l e ( S t r i n g logName ) ;
p u b l i c Handle i n p u t F i l e s ( Job job ) ;
p u b l i c Handle o u t p u t F i l e s ( Job job ) ;
p u b l i c Handle f i l e s ( Type type ) ;
p u b l i c Handle q u a l i t i e s ( F i l e f i l e ) ;
p u b l i c Handle q u a l i t y ( F i l e f i l e , S t r i n g group , S t r i n g f l a g ) ;
p u b l i c Handle types ( ) ;
p u b l i c Handle type ( S t r i n g Name , S t r i n g v e r s i o n ) ;
p u b l i c Handle type ( F i l e f i l e ) ;
p u b l i c Handle jobParameters ( Job job ) ;
p u b l i c Handle jobParameters ( Job job , S t r i n g type ) ;
p u b l i c Handle jobParameter ( Job job , S t r i n g name ) ;
p u b l i c Handle jobOptions ( Job job ) ;
p u b l i c Handle jobOption ( Job job , S t r i n g name , S t r i n g r e c i p i e n t ) ;
p u b l i c Handle qualityParameters ( Quality q u a l i t y ) ;
p u b l i c Handle qualityParameter ( Quality q u a l i t y , S t r i n g name ) ;
p u b l i c Handle f i l e P a r a m e t e r s ( F i l e f i l e ) ;
p u b l i c Handle f i l e P a r a m e t e r ( F i l e f i l e , S t r i n g name ) ;
p u b l i c Handle typeParameters ( Type type ) ;
p u b l i c Handle typeParameter ( Type type , S t r i n g name ) ;

}

Listing 1.3: Definition of the IBookkeepingInfo interface

p u b l i c interface IBookkeepingEditor extends IBookkeepingInfo {
p u b l i c Handle createJob ( S t r i n g configName , S t r i n g c o n f i g V e r s i o n ,
Date date ) ;
p u b l i c StatusCode modifyJobConfigName ( S t r i n g configName , Job job ) ;
p u b l i c StatusCode modifyJobConfigVersion ( S t r i n g c o n f i g V e r s i o n , Job job ) ;
p u b l i c StatusCode modifyJobDate ( Date date , Job job ) ;
p u b l i c StatusCode d e l e t e J o b ( Job job ) ;
p u b l i c Handle c r e a t e F i l e ( S t r i n g logName , Job o r i g i n , Type type ) ;
p u b l i c StatusCode modifyFileLogName ( S t r i n g logName , F i l e f i l e ) ;
p u b l i c StatusCode d e l e t e F i l e ( F i l e f i l e ) ;
p u b l i c Handle c r e a t e Q u a l i t y ( F i l e f i l e , S t r i n g group , S t r i n g f l a g ) ;
p u b l i c StatusCode d e l e t e Q u a l i t y ( Quality q u a l i t y ) ;
p u b l i c Handle createType ( S t r i n g name , S t r i n g v e r s i o n ) ;
p u b l i c StatusCode deleteType ( Type type ) ;
p u b l i c StatusCode createJobParameter ( Job job , S t r i n g name ,
S t r i n g value , S t r i n g type ) ;
p u b l i c StatusCode modifyJobParameterValue ( Job job , S t r i n g name ,
S t r i n g value ) ;
p u b l i c StatusCode modifyJobParameterType ( Job job , S t r i n g name ,
S t r i n g type ) ;
p u b l i c StatusCode deleteJobParameter ( Job job , S t r i n g name ) ;

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

p u b l i c StatusCode createJobOption ( Job job , S t r i n g name ,
S t r i n g r e c i p i e n t , S t r i n g value ) ;
p u b l i c StatusCode modifyJobOptionValue ( Job job , S t r i n g name ,
S t r i n g r e c i p i e n t , S t r i n g value ) ;
p u b l i c StatusCode deleteJobOption ( Job job , S t r i n g name , S t r i n g r e c i p i e n t ) ;
p u b l i c StatusCode c r e a t e F i l e P a r a m e t e r ( F i l e f i l e , S t r i n g name ,
S t r i n g value ) ;
p u b l i c StatusCode modifyFileParameterValue ( F i l e f i l e , S t r i n g name ,
S t r i n g value ) ;
p u b l i c StatusCode d e l e t e F i l e P a r a m e t e r ( F i l e f i l e , S t r i n g name ) ;
p u b l i c StatusCode createQualityParameter ( Quality q u a l i t y , S t r i n g name ,
S t r i n g value ) ;
p u b l i c StatusCode modifyQualityParameterValue ( Quality q u a l i t y ,
S t r i n g name ,
S t r i n g value ) ;
p u b l i c StatusCode deleteQualityParameter ( Quality q u a l i t y , S t r i n g name ) ;
p u b l i c StatusCode createTypeParameter ( Type type , S t r i n g name ,
S t r i n g value ) ;
p u b l i c StatusCode modifyTypeParameterValue ( Type type , S t r i n g name ,
S t r i n g value ) ;
p u b l i c StatusCode deleteTypeParameter ( Type type , S t r i n g name ) ;
p u b l i c StatusCode addInputFile ( F i l e f i l e , Job job ) ;
p u b l i c StatusCode removeInputFile ( F i l e f i l e , Job job ) ;

}

Listing 1.4: Definition of the IBookkeepingEditor interface

Note that the two files methods should never be used except for debug purposes. As a matter of fact, the first one would output the full list of available files, which would kill the server for lack of memory on the production system. The second method would execute any SQL query which would allow a user to delete or corrupt data.

The methods are only there for internal debug purposes and should never be exposed to the end user.

1.3.3 Event types

The IEvtTypeInfo and IEvtTypeEditor abstract interfaces are described in listings 1.5 and 1.6

p u b l i c interface IEvtTypeInfo {
p u b l i c Handle evtTypes ( ) ;
p u b l i c Handle evtType ( i n t id ) ;

}

Listing 1.5: Definition of the IEvtTypeInfo interface

p u b l i c interface IEvtTypeEditor extends IEvtTypeInfo {
p u b l i c StatusCode addEvtType ( i n t id ,
S t r i n g d e s c r i p t i o n ,
S t r i n g primary ,
S t r i n g decay ) ;
p u b l i c StatusCode removeEvtType ( i n t id ) ;

}

Listing 1.6: Definition of the IEvtTypeEditor interface

1.3.4 Replicas

The IReplicaInfo and IReplicaEditor abstract interfaces are described in listings 1.7 and 1.8

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

p u b l i c interface I R e p l i c a I n f o {
p u b l i c Handle r e p l i c a ( F i l e f i l e , S t r i n g l o c a t i o n ) ;
p u b l i c Handle r e p l i c a s ( F i l e f i l e ) ;

}

Listing 1.7: Definition of the IReplicaInfo interface

p u b l i c interface I R e p l i c a E d i t o r extends I R e p l i c a I n f o {
p u b l i c StatusCode addReplica ( F i l e f i l e , S t r i n g r e p l i c a ,
S t r i n g l o c a t i o n ) ;
p u b l i c StatusCode removeReplica ( F i l e f i l e , S t r i n g l o c a t i o n ) ;

}

Listing 1.8: Definition of the IReplicaEditor interface

1.3.5 A word on the default implementation

The BookkeepingSvc class is a service that implements all Editor interfaces and as a consequence all Info interfaces. The details of the implementation won’t be described here but some facts need to be known :

• All connections to the database are using JDBC. The only requirement is to give correct connection string, user name and password when creating the service. An example of an initialization string is : ”jdbc:oracle:thin:@oradev:10521:D”

• Every query done on the bookkeeping database corresponds to a SQL statement. The whole list of statements used was put at the very beginning of the Java file.

• The rest of the code is made of very simple calls to JDBC.

1.4 Access to data from python and XMLRPC

All interfaces presented in section ?? are available in python through XMLRPC. The implementation of the XMLRPC server is described in details in section 2.1 of chapter 2. We describe here the interface available in python, the python objects that are defined and the way the interface should be used.

The python interface is very close to the Java interface described in section 1.3 :

• Each java object has a python counterpart which is a dictionnary having the members of the Java object as entries.

• Java vectors are converted to python lists.

• Each java method has a python counterpart with the same name and same arguments.

1.4.1 Python objects

Here is the list of the python objects used by the interface together with the members of each of them and their type :

job

• jobID : number

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

• configName : string

• configVersion : string

• date : python time tuple, as returned by the gmttime function file

• fileID : number

• jobID : number

• typeID : number

• logName : string type

• typeID : number

• name : string

• version : string quality

• qualityID : number

• fileID : number

• group : string

• flag : string parameter

• ID : number

• name : string

• value : string typedparameter

• ID : number

• name : string

• value : string

• type : string joboption

• ID : number

• name : string

• value : string

• recipient : string evttype

• ID : number

• description : string

• primary : string

• decay : string

Take care that these are actually not objects but dictionnaries and the “members” are the entries in the dictionnary.

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

1.4.2 Python interface

There is a little complication in the definition of the python interface. It lies in the fact that python does not include the signature of a method in its definition. In other words, you cannot define two functions with the same name and differents arguments (even different numbers of them) in python.

The solution is to define a single function in python for all java functions with the same name which checks the number of arguments given and their types to choose the right java function. This was done in the actual interface implementation, in file python/BookkeepingSvc.py of the Bookkeeping package. However, we will show here a pseudo interface where different functions may have the same name.

Listing 1.9 gives the full list of methods available in python with the type of the arguments and return type. Note that the given return type is the one that is returned in case of success. If the call fails, a string is returned containing the error message. Thus the returned type should always be checked to detect errors.

l i s t <job >  j o b s ( f i l e f )
l i s t <job >  j o b s ( s t r i n g configName , s t r i n g c o n f i g V e r s i o n )
l i s t <job >  j o b s ( time date1 , time date2 )
job  job ( f i l e o u t p u t F i l e )
job  job ( number id )
l i s t <f i l e >  f i l e s ( s t r i n g SQLquery )
l i s t <f i l e >  f i l e s ( type t )
f i l e  f i l e ( s t r i n g logname )
l i s t <f i l e >  i n p u t F i l e s ( job j )
l i s t <f i l e >  o u t p u t F i l e s ( job j )
l i s t <q u a l i t y > q u a l i t i e s ( f i l e f )
q u a l i t y  q u a l i t y ( f i l e f , s t r i n g group , s t r i n g f l a g )
l i s t <type >  types ( )
type  type ( f i l e f )
type  type ( s t r i n g name , s t r i n g v e r s i o n )
l i s t <typedparameter > jobParameters ( job j )
l i s t <typedparameter > jobParameters ( job j , s t r i n g type )
typedparameter jobParameter ( job j , s t r i n g name)
l i s t <joboption > jobOptions ( job j )
joboption  jobOption ( job j , s t r i n g name , s t r i n g r e c i p i e n t )
l i s t <parameter > qualityParameters ( q u a l i t y q )
parameter  qualityParameter ( q u a l i t y q , s t r i n g name)
l i s t <parameter > f i l e P a r a m e t e r s ( f i l e f )
parameter  f i l e P a r a m e t e r ( f i l e f , s t r i n g name)
l i s t <parameter > typeParameters ( type t )
parameter  typeParameter ( type t , s t r i n g name)
dictionnary <s t r i n g l o c a t i o n , s t r i n g name> r e p l i c a s ( f i l e f )
s t r i n g  r e p l i c a ( f i l e f , s t r i n g l o c a t i o n )
l i s t <evttype > evtTypes ( )
evttype  evtType ( number id )

Listing 1.9: python interface to the Bookkeeping service

1.4.3 XMLRPC interface usage

The usage of the XMLRPC interface is standard. One just needs the name of the service (RPC/- BookkeepingSvc) and the machine and port where to find it. Listing 1.10 shows some code displaying the replicas of a given file.

from xmlrpclib import Server
s e r v e r = Server ( ’ http : / / l b n t s 3 . cern . ch :8100/RPC/ BookkeepingSvc ’ )
f = s e r v e r . f i l e ( ‘ ‘00000141 00000001 6 . oodst ’ ’ )

Internal Note Issue: 1 Data description and access

Reference:  -
Revision:  0
Last modified: March 12, 2004

print s e r v e r . r e p l i c a s ( f )

Listing 1.10: XMLRPC code sample

Internal Note Issue: 1 Server infrastructure

Reference:  -
Revision:  0
Last modified: March 12, 2004

Chapter 2

Server infrastructure

The bookkeeping services are hosted by a central server that deals both with web pages and XMLRPC services. We describe here the server itself as well as its deployment as the time this document was written.

2.1 Server implementation

The Bookkeeping server is based on a piece of python code providing both an XMLRPC server and a web server. This code was initially used in the Production system and was reused with no change in the Bookkeeping. It is implemented in file python/gaudiweb.py of the Bookkeeping package. The server is then customized in file python/BookkeepingServer.py of the Bookkeeping package.

The server handles essentially three things : webpages, servlets and XMLRPC services. The webpages and XMLRPC services just need to be registered to become available, as you can see in methods startBookkeepingWeb and startRPCServices. Most of the functionnalities is actually provided by the underlying gaudiweb server.

For the servlets, the situation is a bit more complex since this concept is java specific and thus not available in python. For this purpose, the Bookkeeping server can actually not run using a regular python implementation. It actually needs Jython, a java implementation of python. On top of that, dedicated classes were defined to handle servlets correctly, reusing the gaudiweb.Service mechanism.

This servlet support is 70% of the code of BookkeepingServer.py. It uses some home made, dummy implementation of a servlet engine provided in the calsses DummyServletConfig, DummyServletResponse and DummyServletRequest. However ever, the servelt engine could be droped as soon as the servlets are hosted by an external web server, like apache or the ORACLE one.

To conclude with the Bookkeeping server, here is the list of entry points in the current server as well as their type and usage.

• Main : A servlet building the main bookkeeping web page, i.e. the one for selecting data. See section ??.

• EvtTypes : A servlet building a list of existing event types. The types for which no events can be found are not listed. See section ??.

• EvTypeInfo : A servlet handling the selection of data and building a web page with the results See section ??.

• DataSets : ? Francoise, could you put something here ?

Internal Note Issue: 1 Server infrastructure

Reference:  -
Revision:  0
Last modified: March 12, 2004

• Select A Servlet allowing to browse the database. See section ??.

• NewConfirm : A Servlet to enter data into the database. See section ??.

• DisplayBookFile : A servlet that displays details of database modification requests. See section ??.

• Manager : A python servlet allowing to shutdown the server from a web interface. See section ??.

• Bookkeeping : A directory containing files used by the generated html pages. Essentially javascripts files.

• images : A directory containing images used by the generated html pages.

• RPC/BookkeepingSvc : An XMLRPC service providing the Bookkeeping API. See section 1.4.

• RPC/BookkeepingQuery : An XMLRPC service providing the BookkeepingQuery API. This API allows to select files from the Bookkeeping. See section ??.

• RPC/GetLogFiles : An XMLRPC service providing easy access to log files. See section ??.

2.2 Server deployment

To be filled by Joel... Merci Joel !

Internal Note Issue: 1 Booking of data

Reference:  -
Revision:  0
Last modified: March 12, 2004

Chapter 3

Booking of data

Internal Note Issue: 1 The end-user web interfaces

Reference:  -
Revision:  0
Last modified: March 12, 2004

Chapter 4

The end-user web interfaces