Find a place for your findings. Minimize changes to the existing filesystems.

* external drive
* mount -t tmpfs none /mystuff
* mkdir /dev/shm/mystuff

Problems with these?

----

If possible, send data over the network:

* Listen on server:
  nc -l 6789 >> logfilename.txt
* Send stuff from compromised machine:
  $CMD | nc -w 2 name_or_ip_of_server 6789
  
Untrusted nework? Use openssl! Insert into cmdchain:

nc -l 6789 | openssl enc -aes128 -d -k supersecretpw >> log.txt
$CMD | openssl enc -aes128 -e -k supersecretpw | nc -w 2 name_or_ip_of_server 6789

Or use cryptcat, if available.

----

Want do transfer images of blockdevices?

  dd if=/dev/sdx23 | nc...

Feel like copying stuff?

  cat /usr/bin/rootkit_0.1 | nc...

The later cmdlines presume that you are putting stuff on the local disk (»... > file«). Substitute properly if you want to transfer files to remote locations.

----

Let's collect stuff, starting with the most volatile:

* network state
* process state
* users
* system config

----
# Network State #1

netstat --program --verbose -n > netstat_pTvn.txt
netstat --program --verbose > netstat_pTv.txt

arp -n > arp_n.txt
ip neigh show > ip_neigh_show.txt

for i in link addr route rule neigh ntable tunnel tuntap maddr mroute mrule; do
  ip $i list > ip_${i}_l.txt;
done

----
# Network State #2

for t in filter nat mangle raw; do iptables -v -n -x -L -t > iptables_vnxL_t${t}; done
for table in filter mangle raw; do ip6tables -n -t ${table} -L -v -x > ip6tables_nt_${table}; done
for table in filter nat broute; do ebtables -t ${table} -L --Lmac2 --Lc; done

----

Save process table:

ps auxwwwe > ps_auxwwwe.txt
lsof -b -l -P -X -n -o -R -U > lsof_blPXnoRU.txt
for i in t p c t l; do ipcs -a -${i} > ipcs_a_${i}.txt;done

----
# Users

last > last.txt
lastlog > lastlog.txt
who > who.txt
w > w.txt

----
# System
dmesg > dmesg.txt
cat /proc/mounts > proc_mounts.txt
cat /proc/mdstat > proc_mdstat.txt
lspci > lspci.txt
uname -a > uname_a.txt
uptime > uptime.txt

----
# So you've found a suspicions process?

Stop it!

  export PID=12345 # <- insert correct PID!
  kill -STOP ${PID} # stop process

Save the executable:

  cp /proc/${PID}/exe ${PID}.exe
  ls -l /proc/${PID}/exe > ls_l_proc_${PID}_exe.txt

Dump it's memory:

  gdb -p ${PID}
  # type »gcore«
  # type quit
  # some systems provide a programm calles gcore to do that in one step

Check for shared memory segments:
  less /proc/${PID}/maps # look for /dev/shm

Save state (some of it at least):
  tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,wchan} 

Want to get rid of it (think twice before you do it)?
  kill -9 ${PID}

----
# rw -> ro
# Create a rudimentary timeline:

# heuristic: ymmv. Doublecheck.
sync
for mountpoint in $(sort -r /proc/mounts | grep -E ' (ext[234]|xfs|reiser|vfat|ntfs)' | cut -d' ' -f2)
do
  echo mount -o remount,ro "${mountpoint}"
  find / -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> timestamps.dat
done

----
# Need the system to stay rw while you create the timeline?

Create »aliases«:

  mkdir /mnt/root_ro
  mount --bind / /mnt/root_ro
  mount -o remount,ro /mnt/root_ro
  find /mnt/root_ro -xdev -print0 | xargs -0 stat -c "%Y %X %Z %A %U %G %n" >> timestamps.dat
  umount /mnt/root_ro

Repeat for all other »real« filesystems.

Or: mount -o remount,noatime

----
# timeline-decorator.py

#!/usr/bin/python

import sys, time

def print_line(flags, t, mode, user, group, name):
    print t, time.ctime(float(t)), flags, mode, user, group, name

for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, name) = line.split(" ", 6)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, name)
        else:
            print_line("ma-", m, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)
    else:
        if m == c:
            print_line("m-c", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
        else:
            print_line("m--", m, mode, user, group, name)
            print_line("-a-", a, mode, user, group, name)
            print_line("--c", c, mode, user, group, name)

----

Get these tools:

* chkrootkit ( http://www.chkrootkit.org)
* OSSEC rootcheck (http://www.ossec.net/main/rootcheck)
* rkhunter ( http://rkhunter.sourceforge.net)
* Trojanscan ( http://www.trojanscan.org)
* CVE-Checker ( http://cvechecker.sourceforge.net/)

Go hunting!

----

If applicable, compare checksums of package management with actual files:

  debsums (Debian-based distributions)
  rpm -Va (Redhat-based)

Found mismatch? Look closer!
Found nothing? Binary or database could be compromised -> keep walking, nothing to see here.

----
# Step 2: Offline Forensics

Switch off machine (_no_ shutdown). Create disk image.

* see Toby's slides
* TL;DR:
  * Binaries: strings, hexdump, objdump, elf*, gdb, rec (http://www.backerstreet.com/rec/rec.htm), IDAPro,…
  * Logfiles: grep, sort, log2timeline, …
  * Autosy, rkhunter, …
  * GOTO 1 if necessary