Compromised Machine Investigation
This document provides guidance on how to go about
investigating a believed-compromised machine. It's in four main parts:
a section which sets out the context and lists the preliminary steps which
must be done; a section which provides suggestions as to how to go
about an investigation; a collection of useful links; and some
The first section now complete.
The other three are necessarily works-in-progress, and
will be added to in the light of experience.
There will often be a tension between the desire to bring a service back
quickly and the wish to investigate thoroughly. This document does not
attempt to draw any hard and fast rules -- each case will have to be
determined on its own merits. At the very least, however, all
incidents should be investigated in sufficient depth that it becomes possible
to say with reasonable confidence that data integrity has been assured
and that it would be safe to bring the service back.
You should also consult the related
management procedures" document.
This section sets out the basic ground rules for undertaking an
investigation into a believed-compromised machine. While it is understood
that the investigation may have originally started as something else, once
it is suspected that a compromise may have taken place any subsequent steps
must follow the rules laid out here.
techniques" in the next section may be followed, as desired. The requirements
listed here must be followed as applicable.
- (It is assumed throughout that the Head
of Computing will consult with and take direction from the Head of School
- The Head of Computing (or nominee)
must be informed as soon as it
is realised that a compromise may have taken place.
The Head of Computing may appoint a Lead Investigator, who will be
responsible for coordinating the investigation and the activities of any
other people involved.
- You (the Lead Investigator)
should agree with the Head of Computing how
questions from users will be answered while the investigation is
under way. It is important that users are kept as informed as possible,
though of course there will be some things which you may prefer to hold
back, at least for a while.
You should bear in mind that the Freedom of Information
(Scotland) Act 2002 will still apply, and if necessary you should take
advice as to its requirements and exemptions.
Note that the Head of School's approval must be obtained
before any FoI(S)A exemption is claimed.
- You should notify
irt@ed of the
incident, at least in general terms, and be prepared to liaise with them
as necessary should the incident turn out to reach beyond Informatics.
You should keep them updated as necessary as your investigation proceeds.
- The University's
Regulations do cover self-managed and personal machines which
are or have been attached to the
network, though there may be practical difficulties in investigating these.
machines policy can be found
- If the investigation requires that users' activities be traced or that
users' filespace be searched then the Head of School's (or nominee's)
must be obtained. It will not in general be practicable,
or even necessarily
desirable, to obtain individual permission in advance from all
the users who might be involved.
- In all cases, data must still be processed in
accordance with applicable legislation (for example, the Data
Protection Act 1998, or the Human Rights Act 1998). You should be aware
that you may find private, confidential or sensitive material, and you
must treat it appropriately. If in doubt, seek advice.
Note also the School's
- If possible you should make a read-only copy of the
data and work from that. If this is not possible for some reason, you
should be very careful about any steps which might modify the data or
Ideally you should boot from a clean disc, a network
filesystem or a "live DVD",
rather than booting a suspected-to-be-compromised machine directly.
You should be aware that working on the original data is likely to
invalidate any evidential status.
- A complete written log of all steps must be maintained.
This should be either handwritten, or else printed out and signed.
It should be clearly dated, and include frequent timestamps throughout.
It must be preserved in a safe place until such time as the Head of
Computing agrees that it may be disposed of.
- If it appears that one or more user accounts should be suspended,
you must inform the Head of Computing, who will
arrange to have this done and will inform any Directors of Studies,
Supervisors, ISS management, line managers and so on as necessary.
- If it becomes apparent that the findings
of the investigation might be used in formal proceedings (whether
School, University or external),
STOP. Before you proceed any further you
must obtain permission from the Head of Computing.
You must be familiar with the ACPO
Good Practice Guide for Computer-Based Evidence
You MUST have a second person observing. That second
make their own separate written record. Ideally that second
person would be independent of the School, such as one of the members
irt@ed. You MUST make a copy of
the data before you start; bag it along with a written, signed and dated
description of the contents, and a copy of your notes to date;
seal it; and have it stored in a safe place.
Failure to follow any of these steps may make the material worthless
as evidence. You should be aware that you may be cross-examined over
your findings and any steps which you did or did not take.
- There are certain classes of material (various pornography and
terrorism-related) which are illegal even to possess. If your investigation
turns up any of these then:
- you MUST immediately STOP
your investigation and
make the material inaccessible, even if this results in
a loss of service for some users
- you MUST NOT look at the material as there is only
a very limited
statutory defence and you may open yourself to criminal charges if you do
- you MUST immediately inform the Head of School and
the Head of Computing
- you MUST NOT proceed further without the
explicit permission of the Head of School
- you MUST inform
of what you have found and ask for their advice
- (in the circumstances, you may wish to consider whether you would be
prepared to continue with the investigation)
- Once you have determined the extent of the compromise, or alternatively
concluded that you have investigated as much as you reasonably can, you
must report your findings to the Head of Computing.
It is the Head of Computing's responsibility to decide what steps may then
be necessary as a result of the compromise.
- After the investigation is complete you must write
a detailed report for the Head of Computing, who will circulate it as
appropriate. You should also write a public report,
possibly based on the detailed report, which you must
submit to the Head of Computing and the
Head of School for their approval before publishing.
- Having notified
irt@ed earlier, you should
now inform them of your of your conclusions, at least in general terms,
and point them towards your public report. With the Head of Computing's
agreement, you may provide them with more detailed
information, if this would be in the University's interest.
(Borrowing heavily from
of the "October 2011" incident.)
may be useful as a reminder of what you're up against.
- Search engines and incident databases may provide you with useful
- Quarantine the machine. Normally this would be the very first thing
to be done when a compromise is suspected. At the very least, remove all
the machine's firewall holes. This may alert the perpetrator or trigger
some automatic sanitisation, but in this case safety should normally
take priority. Users may be given a few minutes grace, but the longer
they are on the machine the more chance there will be that some important
data might be leaked or lost or corrupted. Close firewall holes, but remember
that existing connections will remain active.
- Create a directory (off-machine) where you can archive copies of all
the log files you look at and dynamic state you obtain.
- When you log in to the machine, try to do this using a mechanism
which doesn't require a password (e.g. kerberos-authenticated ssh). If you
do have to use a password, change it immediately afterwards.
- If you can, obtain a snapshot of the machine's dynamic state, though
be aware that if the machine has been compromised then some or all of the
tools and kernel tables they reference may be incomplete or unreliable:
- process tables
- network interfaces, connections, routing tables, arp cache
- loaded kernel modules
- mounted filesystems
- Boot from a network filesystem or a "live DVD" rather than the
machine's own disc, if possible.
- Make a copy of the machine's partitions. It is important not to work
on the only copy, unless this is absolutely essential. If it is anticipated
that the results of your investigation might be used in evidence then this
step is vital.
script or terminal-emulators' log files may provide a
useful way to record your actions and their results. Be aware that you may
have to merge, or at least reconcile, several such logs if you have more
than one window open. Remember to print, date and sign them all.
- Do not trust anything you find on the machine. Dates may have been
changed. Files may have been changed or hidden. The kernel may have been
tampered with. The BIOS may even have been tampered with. Look for things
which corroborate each other; and conversely look for anomalies.
- Run as many rootkit-detection packages as you can find. They all go
about things in slightly different ways, and one may find what another
one may miss.
- Check the console logs on the console server.
These may show some unexpected behaviour.
These logs should be reasonably trustworthy against all but a
well-informed targeted intrusion.
- Check the loghost. Again, the logs here should be reasonably
- Check the
wtmp record. Be aware that this is likely to
be incomplete or downright bogus if the machine has been compromised.
Compare the contents against the syslog record, ideally on the loghost.
Any differences here should automatically be treated as suspicious.
- Check the process-accounting record.
Once again, be aware that this may have been tampered with.
Look for differences with
wtmp record or the console or loghost logs. Look for
unexpected processes being run, or being run at unexpected times, or not
being run (or at least not showing as being run) when you would have
anticipated that they should be.
- Check the
updaterpms logs. Look for unexpected changes
(deletions as well as additions and updates)
in the installed package set, which may point to things being added
- Verify the installed RPMs. This probably won't show up anything,
but you never know... Unfortunately this won't check "config" files,
and the designation of some of these may not be quite what you might expect.
- Search the filesystems for files or directories
which are unexpectedly not owned by any RPM, particularly any in locations
where RPM ownership would be the norm. These may be malware
which is being "hidden out in the open".
- Compare the RPM "config" files against the same files from a fresh
install, or at least an expected-to-be-clean machine, looking for
- Pay particular attention to files which would be run automatically
(as root) as part of the machine's boot sequence. The
/etc/init.d/ contents are the most obvious of these, but
remember that many of the standard configuration files (such as those in
/etc/sysconfig/ and its subdirectories)
are sourced by the startup scripts and can
contain script fragments as well as straightforward variable assignments.
- Search the filesystems for unexpected setuid-root binaries.
While these would not be strictly necessary so long as a root exploit
exists, they may have been left as a way in after any holes have been closed.
- There may be code or data associated with the compromise stored
"near" some of the files or directories discovered by the previous steps.
If so, these may tell you something about the nature or the purpose
of the compromise.
- Be extremely wary about running anything you find on a compromised
machine. It may not behave as advertised.
- Be careful about contacting sites associated with the incident. It
would be unfortunate to find yourself compromised in the same way.
- You should not normally attempt to probe, scan or otherwise attack
any remote site identified.
- Taking all the evidence together, are you confident that you can
accurately describe the scope of the compromise?
Adjust paths, parameters and options as appropriate throughout.
Some tools are in
CompromisedMachineInvestigation.html,v 1.85 2012/06/18 09:04:40 ajs Exp
Please contact us with any
comments or corrections.
Unless explicitly stated otherwise, all material is
copyright The University of Edinburgh