Operational Meeting Research and Teaching Unit Report - 10 August, 2016
The latest unit minutes are available on the DICE Wiki
- lab exams
Thanks to US unit's work we're ready to go for August exams. The only technical changes are listed in the usual place.
- Theon Upgrade
- Last week saw an upgrade to Theon hardware (two new R630s), the OS (SL7), the
PostgreSQL version (9.5), the underlying change management process ("Hypatia"),
protocol (IPv6 — currently supported by postgresql and backend
processing, but not the apache-hosted services), the header infrastructure...
and session rollover to 16/17. All elements of this system had
undergone testing in isolation but as one might expect the integration required
some hand-holding. We're pleased to announce that the changes have mostly gone
unobserved but the unit is stil repaying a little technical debt in terms of
updating documentation and removing legacy headers - and work remains to move
the Portal and UI to (optionally) support IPv6.
- The most severe problem was that the service went without mirroring for a
day due to a clash on the alias; this turned out to be a procedural error (an
#ifdef LIVE_SERVICE) left defined, probably complicated by the
header infrastructure changes. In any event the mirror would likely still not
have run due to the bulk of the old backups not having been copied to the new
server before going live (tripping a change threshold). Backups ran as usual,
but for the period of time when the service had neither mirrors nor streaming
replication we had no protection from single-host failure, so this situation
should be carefully documented to prevent recurrence.
- One other problem was that the careful use of
masked the absence of some of our core database sync scripts on SL7 for a day
or two, resulting in outdated inventory and profile data for a day or two.
- On the data itself, due to the scale of the changes and lack of time for
testing in situe the prometheus account feeds were "held" until some of these
changes had settled down, but as of Tuesday the feed is running normally.
There were also a few fed data inconsistencies (including students presenting
without names) but these were unrelated to the upgrade and resolved
automatically by the feed process.
Please contact us with any
comments or corrections.
Unless explicitly stated otherwise, all material is
copyright The University of Edinburgh