White dot for spacing only
The Dice Project

71: Logging policy and centralised logging -- evaluation

Project #71 is the first part of a two part project. This part looks at our existing logging policy, to see how well it fits with what we would like to do and what we are obliged to do or not do. The outcome of this part will be an Evaluation Document (this document) for discussion.

Following on from the evaluation and subsequent discussion, there will be a second "implementation" project (Project #155), which will implement any central logging requirements identified by the first part. In any case, it is quite possible that individual service managers may require to adjust their services' logging to correspond with any new policy.


In terms of the regulatory framework in which our logging has to operate, the main influences are the Human Rights Act 1998, the Data Protection Act 1998 and the Regulation of Investigatory Powers Act 2000 (see links from index page). In addition, the information we log will (subject to the usual exemptions) be available on request under the Freedom of Information (Scotland) Act 2002.
  1. The Human Rights Act incorporates the European Convention on Human Rights and its various Protocols into UK law. Everything the University does must be in accordance with the ECHR. From the point of view of this project, the most relevant part would seem to be Article 8 of the Convention's rights and freedoms, Right to respect for private and family life: "Everyone has the right to respect for his private and family life, his home and his correspondence."

    Working within the constraints of the Data Protection Act and the Regulation of Investigatory Powers Act, which were set up in part as a result of ECHR issues, and following established industry practice should allow us to argue that we are indeed in accord with the HRA, though we should always bear in mind its underlying principles.

  2. The Data Protection Act 1998 imposes constraints on how we process personal data, as well as providing statutory rights of access and correction for data subjects.

    "'Personal data' means data which relate to a living individual who can be identified- (a) from those data, or (b) from those data and other information which is in the possession of, or is likely to come into the possession of, the data controller ..." (Data Protection Act 1998 §1). In addition, various specified types are known as "sensitive personal data" (DPA §2) and are subject to much more stringent processing constraints. There's a thorough discussion in the JISC Legal Code of Practice for the Further and Higher Education Sectors on the Data Protection Act 1998.

    In terms of the Act's definition, above, the issue is not just whether the log entries identify individuals per se. If it is possible for us to identify to whom the entries relate, possibly using other information available to us, then the log entries are personal data and must be processed accordingly. (Fortunately it's unlikely that any sensitive personal data would be logged in most cases, but it would be as well to be aware of the possibility. It's hard to see which of the Schedule 3 conditions we could possibly meet!)

    All processing of personal data must be done in accordance with the Data Protection Principles set out in Schedule 1 of the DPA (appended below for reference), and for one of the purposes notified in the University's DP registration. "Purpose 5" of the University's registration allows for the "administration and provision of computing facilities", and our processing falls under Schedule 2 condition 6(1): "The processing is necessary for the purposes of legitimate interests pursued by the data controller or by the third party or parties to whom the data are disclosed, except where the processing is unwarranted in any particular case by reason of prejudice to the rights and freedoms or legitimate interests of the data subject."

    The effective requirement here, then, is that processing must be in accordance with the Principles.

  3. "Interception" is subject to the Regulation of Investigatory Powers Act 2000. The definitions are somewhat convoluted, but essentially interception is making "some or all of the contents of the communication available, while being transmitted, to a person other than the sender or intended recipient of the communication." There is a limited exclusion for "traffic data", which amounts to header, addressing and routing information, but it is stated explicitly in the Act's Explanatory Note ¶34 as well as rather obliquely in the Act itself that the traffic data "may identify a server but not a website or page."

    The upshot of all this is that logging associated with a service which has the "purpose of facilitating the transmission of communications" (such as a web service or perhaps a mail service or messaging service) is likely to constitute "interception", and must therefore be "authorized" by the Act or its associated Regulations (in particular the Lawful Business Practice regulations; see link from index page). It is therefore necessary to identify for each service the purpose for which logging is being undertaken to confirm that it is in accordance with the Regulations (or indeed §3(3) of the Act, which allows for interception "connected with the provision or operation" of a service; but note that this must be interpreted tightly).

The University does notify all users fairly regularly that interception and processing of personal data may take place for the purposes given in the notification.

Note that logging must in each case comply with all of the statutory requirements which are relevant. The DPA and RIPA rules must all be followed as required, and following one Act does not obviate the need to also follow the other.

Note also that just because logs are allowed for one purpose, it does not necessarily follow that they can be used for some other purpose. Each case must be assessed separately on its own merits. It may be that some form of redaction or anonymisation is required before they can be used for a secondary purpose. The simple act of passing raw data to someone counts as "processing", and so even in this case the DPA conditions must be met.

There is currently no statutory requirement for us to undertake data retention (JISC Legal overview on data retention); and in any case, there is no requirement to collect data additional to that needed for normal operational purposes. However, for our own purposes and to allow us to monitor compliance with, for example, the JANET AUP, we would likely want to keep some logs for some period. The LINX Best Current Practice on traceability suggests keeping such data for at least 3 months but no longer than 6 months; and once no longer required for this purpose such data should be anonymised or deleted. Following established industry practice in this respect also allows us to argue that we are in accordance with the HRA and the DPA's 5th Principle: "Personal data processed for any purpose or purposes shall not be kept for longer than is necessary for that purpose or those purposes."


The question of whether data on mirrors or backup tapes is still "held" by us is, unfortunately, not clear. The Information Commissioner's guidance on determining what is "data" for the purposes of the DPA notes that: "Neither the DPA nor the [European Data Protection] Directive provides further explanation of what is meant by 'automatic processing'", and hence leaves open whether manual intervention in the selection of tapes, for example, turns them into "category 'e'" data instead. In a FoI decision the Information Tribunal stated that "simple restoration from ... a back-up tape should normally be attempted" (though we are subject to the Scottish Information Commissioner regarding FoI requests, DP is still regulated by the (UK) Information Commissioner and thence the Information Tribunal).

However, the ICO's document The Guide to Data Protection does state: "... there is a significant difference between permanently deleting a record and archiving it. If a record is archived or stored offline, this should reduce its availability and the risk of misuse or mistake. However, you should only archive a record (rather than delete it) if you still need to hold it. You must be prepared to give subject access to it, and to comply with the data protection principles. If it is appropriate to delete a record from a live system, it should also be deleted from any back-up of the information on that system."

It's probably safest, therefore, to assume that backups are still subject to the DPA provisions.

Regarding Freedom of Information requests for data on mirrors or tape backups, the Scottish Information Commissioner's FAQ pages state: "Where a public authority has deleted an e-mail or an electronic file and it can only be retrieved by an IT specialist, the Commissioner takes the view that the information is no longer held by the public authority."

Current status

The tables below list some of the services run (loosely speaking!) by each unit, and what they log. The lists are intended to be indicative of the types of logs encountered, rather than complete. The recommendation below is that in due course service managers should ensure the compliance of all their services with whatever policy is developed.

Sample inf-unit services

Service What/how logged Retention Personal data? RIP? Access Comments
network port traffic counters switch traffic counters; to rrd files every 5 minutes consolidated daily/weekly indexed by port label and/or office no via netmon web pages  
FDB snapshots switch FDBs; to indexed text file every 6/7 minutes rolled on 1st and 15th, with old versions deleted after 90 days MAC addresses linked to users; indexed by port label and/or office and/or hostname no current and immediate past via netmon web pages; others require login to site network infrastructure host MAC-switchport mapping
arpwatch to service database +email 90 days MAC addresses linked to users no via netmon web pages MAC-IP translation
netmon web sites http access and error logs 4 weeks site content as above logs: §3(3) access from within EdLAN only  
switch snmp traps switch and port status changes, on site traphosts via syslog 10 days indexed by port label and/or office and/or hostname no summaries emailed nightly; full access requires login to traphosts switches also keep internal logs for a few days
console servers all connected consoles; to per-host files essentially indefinitely at the moment! usernames; potentially other things potentially requires login to console servers arbitrary commands and their output might be logged
nameservers (DNS; central servers) all queries logged to local files; errors via syslog query logs: typically a couple of hours; error logs: a week query logs traffic data requires login to nameservers query logs indexed by caller's IP address
nameservers (DNS; desktops) queries logged to local files; errors via syslog query logs: typically a couple of hours; error logs: 4 weeks query logs traffic data query logs require root access  
timeservers (NTP) only service statistics logged n/a no no    
DHCP requests and grants; via syslog 5 weeks yes no requires login to DHCP servers  
OpenVPN client-connect and -disconnect, on endpoints via syslog 13 weeks yes traffic data summaries emailed weekly; full access requires login to endpoints  
authentication (kerberos) ticket grants and password changes; to service log files 5 weeks principals (usernames) no requires login to KDCs password information could appear under some circumstances
authentication (cosign) cookie requests 5 weeks usernames §3(3) requires login to cosign servers  
directory services (central servers) connections and queries; to service log files a week yes no requires login to directory servers indexed (indirectly) by IP address
directory services (desktops) logging not normally enabled n/a n/a n/a    
host/service monitoring (nagios) host and service status as recorded by normal nagios operation configuration logs: 5 weeks; service logs: essentally indefinitely no no nagios web pages  
chat ?? ?? probably no ??  

As regards sending to a central loghost, there is no obvious advantage in doing so for the inf-unit services, and good security and confidentiality reasons not to.

Sample mp-unit services

Service What/how logged Retention Personal data? RIP? Access Comments
/var/lcfg/log/auth authentication decisions; via syslog 4 weeks names and principals no root access  
/var/lcfg/log/mail mail transactions; via syslog 4 weeks yes traffic data unrestricted on machine  

A good case could be made for a central loghost to capture syslog-based logs, as this would simplify correlation across machines.

Sample rat-unit services

Service What/how logged Retention Personal data? RIP? Access Comments
database audit logs via syslog ?? inevitably unlikely ??  
sundry licence servers ?? ?? probably indexed by username and IP address §3(3)? ??  
exam laptops via syslog ?? probably §3(3)? ??  

RaT-unit have a number of machines which could usefully be set to log to a central loghost.

Sample services-unit services

Service What/how logged Retention Personal data? RIP? Access Comments
sundry apache-based web services site access and error logs 26 weeks?? indexed by IP address §3(3) login to web servers  
sundry cups-based print services ?? 4 weeks? indexed by username and IP address no? login to print servers; root access  

Sample us-unit services

Service What/how logged Retention Personal data? RIP? Access Comments
RT ?? ?? yes §3(3)? ?? RT database also holds personal data
sundry login/ssh servers ?? 4 weeks? indexed by username and IP address no? login to individual servers; root access for some data  

Interim recommendations

  1. We should have a written policy on logging and retention, probably to include:
  2. What about self-managed services??
  3. A central loghost does look as though it would be useful, though care would have to be given to access, backup and retention periods.
  4. A separate policy on backups and archiving should be developed, paying heed to DP and FoI issues.

Appendix: Data Protection Principles

For reference, the Data Protection Principles as set out in Schedule 1 of the DPA are reproduced below:

  1. Personal data shall be processed fairly and lawfully and, in particular, shall not be processed unless-
  2. Personal data shall be obtained only for one or more specified and lawful purposes, and shall not be further processed in any manner incompatible with that purpose or those purposes.
  3. Personal data shall be adequate, relevant and not excessive in relation to the purpose or purposes for which they are processed.
  4. Personal data shall be accurate and, where necessary, kept up to date.
  5. Personal data processed for any purpose or purposes shall not be kept for longer than is necessary for that purpose or those purposes.
  6. Personal data shall be processed in accordance with the rights of data subjects under this Act.
  7. Appropriate technical and organisational measures shall be taken against unauthorised or unlawful processing of personal data and against accidental loss or destruction of, or damage to, personal data.
  8. Personal data shall not be transferred to a country or territory outside the European Economic Area unless that country or territory ensures an adequate level of protection for the rights and freedoms of data subjects in relation to the processing of personal data.

Evaluation.html,v 1.111 2010/04/01 12:39:40 gdmr Exp

 : Units : Infrastructure : Projects : 71-Logging 

Mini Informatics Logo - Link to Main Informatics Page
Please contact us with any comments or corrections.
Unless explicitly stated otherwise, all material is copyright The University of Edinburgh
Spacing Line