Compromised Machine or User Account Investigation
This document provides guidance on how to go about
investigating a believed-compromised machine or user account.
It's in four main parts:
a section which sets out the context and lists the preliminary steps which
must be done; a section which provides suggestions as to how to go
about an investigation; a collection of useful links; and some
The first section is complete.
The other three are necessarily works-in-progress, and
will be added to in the light of experience.
There will often be a tension between the desire to bring a service back
quickly and the wish to investigate thoroughly. This document does not
attempt to draw any hard and fast rules -- each case will have to be
determined on its own merits. At the very least, however, all
incidents must be investigated in sufficient depth that it becomes possible
to say with reasonable confidence that data integrity has been assured
and that it would be safe to bring the service back.
To ensure a high-quality investigation, all incidents must be
investigated by a member of the Computing Team, that includes any
event which affects a self-managed system.
You should also consult the related
management procedures" document.
This section sets out the basic ground rules for undertaking an
investigation into a believed-compromised machine. While it is understood
that the investigation may have originally started as something else, once
it is suspected that a compromise may have taken place any subsequent steps
must follow the rules laid out here.
The "suggested techniques" in the next section may be followed, as desired. The requirements listed here must be followed as applicable.
- (It is assumed throughout that the Head
of Computing will consult with and take direction from the Head of School
- The Head of Computing (or nominee)
must be informed as soon as it
is realised that a compromise may have taken place.
The Head of Computing may appoint a Lead Investigator, who will be
responsible for coordinating the investigation and the activities of any
other people involved.
- You (the Lead Investigator)
should agree with the Head of Computing how
questions from users will be answered while the investigation is
under way. It is important that users are kept as informed as possible,
though of course there will be some things which you may prefer to hold
back, at least for a while. You should bear in mind that the
of Information (Scotland) Act 2002
will still apply, and if necessary you should take
advice as to its requirements and exemptions.
Note that the Head of School's approval must be obtained
before any FoI(S)A exemption is claimed.
- You should notify
irt@ed of the
incident, at least in general terms, and be prepared to liaise with them
as necessary should the incident turn out to reach beyond Informatics.
You should keep them updated as necessary as your investigation proceeds.
- The University's
Regulations do cover self-managed and personal machines which
are or have been attached to the
network, though there may be practical difficulties in investigating these.
machines policy can be found
- If the investigation requires that users' activities be traced or that
users' filespace be searched then the Head of School's (or nominee's)
must be obtained. It will not in general be practicable,
or even necessarily
desirable, to obtain individual permission in advance from all
the users who might be involved.
- In all cases, data must still be processed in
accordance with applicable legislation (for example, the
Protection Act 2018, the
GDPR, or the
Act 1998). You should be aware
that you may find private, confidential or sensitive material, and you
must treat it appropriately. If in doubt, seek advice.
Note also the School's
- If possible you should make a read-only copy of the
data and work from that. If this is not possible for some reason, you
should be very careful about any steps which might modify the data or
Ideally you should boot from a clean disc, a network
filesystem or a "live USB",
rather than booting a suspected-to-be-compromised machine directly.
You should be aware that working on the original data is likely to
invalidate any evidential status.
- A complete written log of all steps must be maintained.
This should be either handwritten, or else printed out and signed.
It should be clearly dated, and include frequent timestamps throughout.
It must be preserved in a safe place until such time as the Head of
Computing agrees that it may be disposed of.
- If it appears that one or more user accounts should be suspended,
you must inform the Head of Computing, who will
arrange to have this done and will inform any Directors of Studies,
Supervisors, ISS management, line managers and so on as necessary.
- If it becomes apparent that the findings
of the investigation might be used in formal proceedings (whether
School, University or external),
STOP. Before you proceed any further you
must obtain permission from the Head of Computing.
You must be familiar with the ACPO
Practice Guide for Digital Evidence.
You MUST have a second person observing. That second
make their own separate written record. Ideally that second
person would be independent of the School, such as one of the members
irt@ed. You MUST make a copy of
the data before you start; bag it along with a written, signed and dated
description of the contents, and a copy of your notes to date;
seal it; and have it stored in a safe place.
Failure to follow any of these steps may make the material worthless
as evidence. You should be aware that you may be cross-examined over
your findings and any steps which you did or did not take.
- There are certain classes of material (various pornography and
terrorism-related) which are illegal even to possess. If your investigation
turns up any of these then:
- you MUST immediately STOP
your investigation and
make the material inaccessible, even if this results in
a loss of service for some users
- you MUST NOT look at the material as there is only
a very limited
statutory defence and you may open yourself to criminal charges if you do
- you MUST immediately inform the Head of School and
the Head of Computing
- you MUST NOT proceed further without the
explicit permission of the Head of School
- you MUST inform
of what you have found and ask for their advice
- (in the circumstances, you may wish to consider whether you would be
prepared to continue with the investigation)
- Once you have determined the extent of the compromise, or alternatively
concluded that you have investigated as much as you reasonably can, you
must report your findings to the Head of Computing.
It is the Head of Computing's responsibility to decide what steps may then
be necessary as a result of the compromise.
- After the investigation is complete you must write
a detailed report for the Head of Computing, who will circulate it as
appropriate. You should also write a public report,
possibly based on the detailed report, which you must
submit to the Head of Computing and the
Head of School for their approval before publishing.
- Having notified
irt@ed earlier, you should
now inform them of your of your conclusions, at least in general terms,
and point them towards your public report. With the Head of Computing's
agreement, you may provide them with more detailed
information, if this would be in the University's interest.
The National Cyber Security Centre provide some very helpful guidance on incident response
The following suggestions borrow heavily from analysis of
local incidents which occurred in
October 2011 and May 2021
may be useful as a reminder of what you're up against.
- Search engines and incident databases may provide you with useful
- Co-ordination. The response to an incident is likely to involve multiple COs. To co-ordinate activity you could create a separate jabber chatroom and invite interested parties. This means that the standard cos chatroom is available for the usual discussions and other staff are not distracted. It also helps avoid the "too many cooks" situation. All discussions in the incident response chatroom should be logged for future reference.
- Quarantine the machine. Normally this would be
the very first thing to be done when a compromise is suspected. The
simplest way to do this is to disable the network ports using rfe,
that blocks both further incoming traffic and any outgoing traffic
which may be generated on the compromised machine. At the very least,
remove all the machine's firewall holes. When removing firewall holes
you must ensure the firewall rules are actually updated. That can be
done by setting the
ipfilter.export resource to the empty
string, DO NOT just remove the ipfilter component (or
dice/options/ipfilter.h header), when that happens
the system falls out of the spanning map and nothing is updated
immediately. This may alert the perpetrator or trigger some automatic
sanitisation, but in this case safety should normally take priority.
Users may be given a few minutes grace, but the longer they are on the
machine the more chance there will be that some important data might
be leaked or lost or corrupted. When closing firewall holes remember
that existing connections may remain active.
- Create a directory (off-machine) where you can archive copies of all
the log files you look at and dynamic state you obtain.
- When you log in to the machine, try to do this using a mechanism
which doesn't require a password (e.g. kerberos-authenticated ssh). If you
do have to use a password, change it immediately afterwards.
- If you can, obtain a snapshot of the machine's dynamic state, though
be aware that if the machine has been compromised then some or all of the
tools and kernel tables they reference may be incomplete or unreliable:
- process tables
- network interfaces, connections, routing tables, arp cache
- loaded kernel modules
- mounted filesystems
- Boot from a known good OS. This could be done by
PXE booting or a "live USB". Note that PXE won't work if the network
ports have been disabled, a live distro will probably also be more
useful as a wider range of tools will be available. Never trust the
machine's own disc, only boot it if absolutely necessary.
- Make a copy of the machine's partitions. It is
important not to work on the only copy, unless this is absolutely
essential. If it is anticipated that the results of your
investigation might be used in evidence then this step is vital. The
best way to approach this is to get a portable USB drive and use the
dd command to copy the partitions. Using dd
means everything is preserved, including any recently unlinked files,
and can be inspected. Note that due to data transfer speeds this will
be slow, for big partitions it may well be necessary to leave the copy
running over night. One big benefit of taking a copy is that when the
compromised machine is in a server room you can work on the data at
your leisure in a comfortable, quiet office environment.
script or terminal-emulators' log files may provide a
useful way to record your actions and their results. Be aware that you may
have to merge, or at least reconcile, several such logs if you have more
than one window open. Remember to print, date and sign them all.
- Do not trust anything you find on the machine. Dates may have been
changed. Files may have been changed or hidden. The kernel may have been
tampered with. The BIOS may even have been tampered with. Look for things
which corroborate each other; and conversely look for anomalies.
- Run as many rootkit-detection packages as you can find. They all go
about things in slightly different ways, and one may find what another
one may miss.
- Check the local password file and
/home directory for
suspicious or unusual users. For self-managed machines, do all local
accounts have good quality passwords? You don't need to know the
passwords but might need to seek assurance from the people who own
- Check the console logs on the console server.
These may show some unexpected behaviour.
These logs should be reasonably trustworthy against all but a
well-informed targeted intrusion.
- Check the loghost. Again, the logs here should be reasonably
- Check the
wtmp record. Be aware that this is likely to
be incomplete or downright bogus if the machine has been compromised.
Compare the contents against the syslog record, ideally on the loghost.
Any differences here should automatically be treated as suspicious.
- Check the process-accounting record.
Once again, be aware that this may have been tampered with.
Look for differences with
wtmp record or the console or loghost logs. Look for
unexpected processes being run, or being run at unexpected times, or not
being run (or at least not showing as being run) when you would have
anticipated that they should be.
- Check the auditd logs. These can be queried using ausearch and
aureport. They contain a wealth of useful information, in particular
all authentication processes will have auditd log entries. It appears that
auditd is installed as standard on most distributions so can be useful
when investigating self-managed machines as well as DICE. Although you
must consider that this data may be tampered with by attackers it
seems to be less often targetted than wtmp and process accounting.
- Assemble a timeline. The easiest way to get a clear
picture of what occurred is to build a timeline of all the events for
which you have reliable data.
- Compare and contrast all available data. When
authentication data is available from multiple sources is it
consistent? Often an indicator of a compromised account is that
records of activity have been erased from some logs but not all.
- Check the
updaterpms logs. Look for unexpected changes
(deletions as well as additions and updates)
in the installed package set, which may point to things being added
- Verify the installed packages. This probably won't show up anything,
but you never know... Unfortunately this won't check "config" files,
and the designation of some of these may not be quite what you might expect.
- Search the filesystems for files or directories
which are unexpectedly not owned by any package, particularly any in locations
where package ownership would be the norm. These may be malware
which is being "hidden out in the open".
- Compare the package "config" files against the same files from a fresh
install, or at least an expected-to-be-clean machine, looking for
- Pay particular attention to files which would be run automatically
(as root) as part of the machine's boot sequence. The
/etc/systemd/ contents are the most obvious of these, but
remember that many of the standard configuration files (such as those in
/etc/default and subdirectories)
are sourced by the startup scripts and can
contain script fragments as well as straightforward variable assignments.
- Search the filesystems for unexpected setuid-root binaries.
While these would not be strictly necessary so long as a root exploit
exists, they may have been left as a way in after any holes have been closed.
- There may be code or data associated with the compromise stored
"near" some of the files or directories discovered by the previous steps.
If so, these may tell you something about the nature or the purpose
of the compromise.
- Be extremely wary about running anything you find on a compromised
machine. It may not behave as advertised.
- Be careful about contacting sites associated with the incident. It
would be unfortunate to find yourself compromised in the same way.
- You should not normally attempt to probe, scan or otherwise attack
any remote site identified.
- Taking all the evidence together, are you confident that you can
accurately describe the scope of the compromise?
Adjust paths, parameters and options as appropriate throughout.
Some tools are in
CompromisedMachineInvestigation.html,v 1.106 2021/07/07 13:53:41 squinney Exp
Please contact us with any
comments or corrections.
Unless explicitly stated otherwise, all material is
copyright The University of Edinburgh