White dot for spacing only
The Dice Project


RAT Unit Generated Package Lists

ratpkgs.html,v 1.17 2018/07/18 10:03:06 gdutton Exp
Please note: this system is deprecated and was used only to maintain SL5 and F13 packages. This page remains for historical interest only. Package lists are maintained in packages/dice/dice_<os>_rat*.rpms.
See also Package Search procedure.

Structure & Operation

The RAT unit's package list system consists of a number of components, which ultimately with the DICE LCFG subversion repository. These components are:

master package lists

These lists are the only files which should be edited in the course of managing RAT unit packages. They define (almost) all the packages which the RAT unit manage. They contain the master copy of the contents of the multiple package lists which the unit manage.

The package files also contain definitions of how and where the generated package lists should be created.

'taggen'

This script splits the tagged RAT unit master package lists into their constituent parts. The program takes a single input file, which itself contains all the instructions and package lines required to generate multiple output packages.

In normal use, taggen should never need to be called manually, but is invoked on every change to the master package lists.

'ratpkgs' repository

This repository contains the above files and applications, and most importantly also contains 'hook' scripts which track changes to the master package lists and ensure that the generated package lists remain in sync with the DICE LCFG repository.

Until the taggen application is packaged, the repository also contains the master copy of the taggen executable.

Understanding the package lists

This is an excerpt of a package list:

/*?*********************************************************
 * Master Teaching Package File for Research and Teaching  *
 ***********************************************************
 * define dice/dice_sl5_i386_large_rat.rpms @large @i386
 * define dice/dice_sl5_large_rat.rpms @large
 * define dice/dice_sl5_i386_teaching_and_research.rpms @i386
 * define dice/dice_sl5_teaching_and_research.rpms @default
 ***********************************************************/

/*!*********************************************************
 *           Specific Teaching Course Packages             *
 ***********************************************************/
aclisp-8.0-1.inf                                /*             Semester2 */

blender-2.46-1.inf                              /*             Semester2 */
bow-20020213-0.dice.2                           /* Semester1             */

cinelerra-2.1-0.12.20070108.inf                 /*                      @i386 */
  OpenEXR-devel-1.4.0a-4.el5                    /* cinelerra pre-req    @i386 */

/*? useless-packge-2.0/noarch                   /* no longer required.  @large */

[...]

/*! end of file */

An incidence of @tag on a line will write that line into the defined target file. The absence of any tag will write the line to the defined @default file.

The define lines contain the destination (within the LCFG /core/packages/ tree) of the generated files, followed by their tags, in the format @[A-Za-z0-9]+. After definition tags may appear on the same line as a package. Define lines are processed in the order they appear, and all defined tags must be present on a line for it to be written to the given destination.

The only other points of note are the comments of the form /*! ... */ and /*? ... */; these are placed into all or none of the defined package lists, respectively. For example, each generated file has a traditional "eof" comment written into it thanks to the last line in the excerpt above, but no file will have the comments surrounding the 'define' lines written into it. The 'useless package' line will not appear in any output file, even though it has been tagged with @large.

This is an excerpt of the tagged output file, dice_sl5_i386_teaching_and_research.rpms, as defined above:

/*****************************************************************************
 * Automatically generated package list for tagged packages.
 * To edit, please contact the RAT unit or use the master list
 * source: /afs/inf.ed.ac.uk/group/rat-unit/ratpkgs/lists/dice_sl5_rat.master
 * tags: @i386
 *****************************************************************************/
/*!*********************************************************
 *           Specific Teaching Course Packages             *
 ***********************************************************/
cinelerra-2.1-0.12.20070108.inf                 /*              */
  OpenEXR-devel-1.4.0a-4.el5                    /* cinelerra pre-req    */
  OpenEXR-1.4.0a-4.el5                          /* cinelerra pre-req  */

[...]

/*! end of file */

The package lines which had been tagged @i386 are preserved, verbatim, with their comments. The only modification made is to remove the tag text itself: this is replaced by a note in the automatically-generated header comment at the top of the file. Note also that the location of the master list appears in this header for ease of later modification. Like the 'tags: ' line, this is purely for informational purposes.

Tagged output files such as this contain very few blank lines. This is simply because they have not been tagged: blank lines are passed through to the default destination file, though spans of blank lines are collapsed into one or two for clarity.

Finally, note that the public /*! end of file */ comment, which will have been written out to every file including the default.

To understand precisely how comments, tags and define lines are processed, refer to the taggen manual page.

Getting Started

Non-unit members: please note that you will need to be given access to the various shared resources used by the script, before you can use the ./commit-lists script or indeed commit to the repository. Specifically these resources are located in the rat-unit shared AFS space.

To get started with the RAT unit package lists, a copy of the package management repository is required. This is named ratpkgs and can be found on the Informatics subversion server svn.inf.

To get a copy of the package lists:

$ svn co svn+ssh://svn/disk/svn/ratpkgs/
$ tree ratpkgs
ratpkgs/
|-- bin
|   |-- commit-lists
|   `-- taggen
|-- docs
|   |-- Makefile
|   |-- UPDATING
|   |-- taggen.1.gz
|   `-- taggen.pod
|-- lists
|   |-- dice_sl5_rat.master
|   `-- dice_sl5_rat_env.master
`-- output.dir

Alternatively you may use the checked-out copy available in the RAT unit shared space.

Finding a package list

To make a change to a package list, simply edit your chosen master file. You can locate the master file for any given generated list from the generated file's header or by grepping the master lists for the name of the generated list:

$ grep -e 'define' *.master
dice_sl5_rat.master: * define dice/dice_sl5_i386_large_rat.rpms @large @i386
dice_sl5_rat.master: * define dice/dice_sl5_large_rat.rpms @large
dice_sl5_rat.master: * define dice/dice_sl5_i386_teaching_and_research.rpms @i386
dice_sl5_rat.master: * define dice/dice_sl5_teaching_and_research.rpms @default
dice_sl5_rat_env.master:  * define dice/dice_sl5_i386_rat_env.rpms    @i386
dice_sl5_rat_env.master:  * define dice/dice_sl5_i386_rat_env.rpms    @spare
dice_sl5_rat_env.master:  * define dice/dice_sl5_rat_env.rpms    @default

The environments in which these package lists are called can be found in the dice/options/research-and-teaching-packages.h header as before.

To remove a package, simply remove its entry in the master file as before.

To add a package, write the package line as normal. Without tags, this line will appear in the default file (which should generally reach the widest set of machines).

To move a package to a specific list, simply add the appropriate tag in a comment after the package. If a package must appear in two distinct package lists then it may be tagged with both those lists' tags (alternatively a package could be listed twice if tags overlap, but this indicates that the package splitting is not logically arranged).

To comment out a package, it is suggested that you use the secret comment syntax:

/*? potentially-obsolete-package-3.2-5.inf            /* not sure about this pkg */

Simple /* ... */ comments are handled like any other lines: any tags would still be processed and the commented line written to its output file; multi-line traditional comments are therefore inadvisable as they run the risk of mismatched delimiters between output files. Note the redundant /* inside the comment, which is ignored (albeit not good practice) as it would usually be.

Checking in

You can check the effect of the changes you have made by running taggen against each modified file: results will be output to your working directory (or defined subdirectory, if it exists).

To complete the process, check the master file(s) into the repository using the normal subversion processes. This will fire off a precommit script which checks for file errors, then writes all generated files into the location specified by the file output.dir.

It is then crucial that the user then checks in the modified package lists to the main LCFG repository. By default the file output.dir points to a shared checkout of the LCFG package lists, allowing the files to be output in the correct location for a subsequent checkin.

An example:

$ cd ~/ratpkgs
$ vim lists/dice_sl5_rat.master
$ svn st
M      lists/dice_sl5_rat.master

$ svn ci -m 'added a comment with a typo'
Sending        lists/dice_sl5_rat.master
Transmitting file data .svn: Commit failed (details follow):
svn: 'pre-commit' hook failed with error output:
Testing lists/dice_sl5_rat.master...
WARNING: Orphaned tag or list of tags. Adding to default.
  55: /* This is a comment with a tag: @i886 */

$ vim lists/dice_sl5_rat.master # (correct the error)
$ svn ci -m 'added a comment to the list.'
Sending        lists/dice_sl5_rat.master
Transmitting file data .
Committed revision 19.

$ cat output.dir
/afs/inf.ed.ac.uk/group/rat-unit/packages

$ svn up `cat output.dir`
At revision 12344

$ svn st `cat output.dir`
M      /afs/inf.ed.ac.uk/group/rat-unit/packages/dice/dice_sl5_i386_teaching_and_research.rpms
M      /afs/inf.ed.ac.uk/group/rat-unit/packages/dice/dice_sl5_large_rat.rpms
M      /afs/inf.ed.ac.uk/group/rat-unit/packages/dice/dice_sl5_i386_large_rat.rpms
M      /afs/inf.ed.ac.uk/group/rat-unit/packages/dice/dice_sl5_teaching_and_research.rpms

$ svn ci -m 'added a comment to the list.' `cat output.dir`
Sending        group/rat-unit/packages/dice/dice_sl5_i386_large_rat.rpms
Sending        group/rat-unit/packages/dice/dice_sl5_i386_teaching_and_research.rpms
Sending        group/rat-unit/packages/dice/dice_sl5_large_rat.rpms
Sending        group/rat-unit/packages/dice/dice_sl5_teaching_and_research.rpms
Transmitting file data ....
Committed revision 12345.

It would now be wise to check the results of this check-in on the LCFG server.

A Faster Way

Having understood the process, it's worth noting that there's a script which automates some of this for you. Having altered a master package list, rather than command subversion directly you may use the commit-lists script which stages and commits any changes directly to both subversion repositories: execute simply as ./commit-lists.

Note that the success of this script depends on your user having write permissions on the shared AFS space used for the LCFG 'staging copy', as defined in the ./output.dir file.

Common Errors and Limitations

Some common package-list errors can be detected by taggen, though it does not make any explicit effort to validate the output against a package-list specification (perhaps later).

In particular, the script will fail to write files, or return with an error after writing, if it detects any of the following:

The script also considers the following conditions to be suspicious, and will return a visible warning, usually with the file name and line number where relevant:

However, at the time of release the script can not catch a number of logical or syntactic errors which might appear in the generated package lists. In particular it does not detect:

More information

For further information on the current operation of the package list update scripts, please see the document 'UPDATING' in the root of the ratpkgs subversion repository.

For the full details of the taggen utility consult the taggen manual page (until installed, this can be found in the root of the 'ratpkgs' subversion repository.


 : Units : Research_and_teaching : Documentation 

Mini Informatics Logo - Link to Main Informatics Page
Please contact us with any comments or corrections.
Unless explicitly stated otherwise, all material is copyright The University of Edinburgh
Spacing Line