linux-triage.sh: a read-only first-hour collection script for Linux

What it is for

The Windows side of the desk has had a collection script since the beginning. This is the Linux counterpart: what to pull off a host in the first hour, before anybody reboots it, in the order RFC 3227 gives for volatility.

Running it

sudo ./linux-triage.sh /mnt/usb/case-0142

The argument is mandatory and there is no default, on purpose. Output belongs on removable media or a network mount. Writing to the disk you are investigating overwrites unallocated space, which is where deleted files are.

Root sees every process and socket. Without it the script still runs, and each output file records which commands were refused.

sudo ./linux-triage.sh /mnt/usb/case-0142 --quick

--quick leaves out the one step that walks the whole disk, the setuid sweep, which can take minutes on a large server. Without the flag that step is capped at ten minutes wherever coreutils' timeout exists.

What it collects, in order

Stage	What	Why it is early
1	Time, logged-in users, the process tree with start times, TCP, UDP and UNIX sockets with owning processes, open-but-deleted files, ARP, routes, mounts, kernel modules	Gone at reboot, and changing while you read this
1	Processes whose binary has been deleted from disk	The cheapest check there is for a dropped-and-removed implant. It vanishes when the process does
2	Logon history (last, lastb, lastlog), accounts, sudoers, and every authorized_keys file with its times	An added SSH key survives a password reset
3	System and user cron, systemd units and timers, init scripts, /etc/ld.so.preload, profile scripts, and listings of /tmp, /var/tmp and /dev/shm	Persistence
4	Files changed in /etc and the binary directories in the last seven days, and every setuid file, with change times	mtime can be set by the file's owner. ctime is harder to fake, so both are recorded
5	A listing of /var/log, and the tail of the SSH and sudo records	Enough to see what has rolled. Copy the logs in full during imaging
6	Kernel, distribution and installed packages	Last, because it does not expire

Each output file begins with the exact command that produced it, the UTC time and the user ID, so somebody who was not there can interpret it. A SHA-256 manifest is written at the end; the hash manifest tool verifies it later.

What it does not do

It installs nothing, deletes nothing, changes no configuration, sends nothing over the network and does not capture memory. For memory, see the memory forensics cheat sheet. It does not copy shell histories or file contents beyond public SSH keys and the files named above, because a triage script that hoovers up home directories is a privacy incident of its own.

How it was tested, and how it was not

The script passes bash -n, and version 1.1 was run end to end on macOS, which turns out to be a useful stand-in for the awkward case: macOS has none of the GNU options the script prefers (ls --time-style, find -printf, ps --forest, ss), so every one of them failed and every portable fallback ran. Each output file records which form it got. The resulting manifest was then verified twice, with shasum -a 256 -c and with hash_manifest.py, and every file checked clean.

That second test found a real fault in version 1.0: it hashed its own collection log and then appended two more lines to it, so the manifest failed verification against the log it shipped with. Version 1.1 closes the log before hashing.

It has not been run on a Linux host by the author of this page. The GNU forms are the ones mainstream distributions will take, and those are the forms this testing could not exercise. Run it on a test machine of your own build before you rely on it, which is the right advice for any collection script including ones that say they were tested.

The files

linux-triage.sh

Read it first. Every command it runs is in the file, and the file is the documentation.

Download linux-triage.sh · 227 lines · 11.5 KB

#!/usr/bin/env bash
# ═══════════════════════════════════════════════════════════════════════════
#  linux-triage.sh: read-only first-hour collection for a Linux host
#  Security Artifacts, version 1.1, 2026-09-21. Released under CC0.
# ═══════════════════════════════════════════════════════════════════════════
#
#  Collects what expires first, first: processes, sockets and logged-in users
#  before persistence, persistence before logs. Uses only what ships with a
#  normal distribution. Writes text files and a SHA-256 manifest, and prints
#  what it is doing while it does it.
#
#  What it does NOT do: it installs nothing, deletes nothing, changes no
#  configuration, sends nothing over the network and does not capture memory.
#  It does write its output, so point it at removable media or a network
#  mount, never at the disk you are investigating:
#
#      sudo ./linux-triage.sh /mnt/usb/case-0142
#      sudo ./linux-triage.sh /mnt/usb/case-0142 --quick    (skips the whole-disk sweep)
#
#  Read it before you run it. It is short on purpose. Running as root sees
#  every process and socket; running without it still works and the output
#  says which commands were refused.
#
#  Every command is wrapped so that one missing binary costs one file rather
#  than the run. A collection script that stops at the first error collects
#  nothing on exactly the hosts that are most broken.
# ═══════════════════════════════════════════════════════════════════════════

set -u
umask 077
export LC_ALL=C

OUT="${1:-}"
QUICK="${2:-}"
if [ -z "$OUT" ]; then
  echo "usage: $0 <output-directory> [--quick]   (removable media or a network mount, not the evidence disk)" >&2
  echo "       --quick skips the one step that walks the whole disk (the setuid sweep)" >&2
  exit 2
fi

HOST="$(hostname 2>/dev/null || echo unknown-host)"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
DEST="$OUT/${HOST}-${STAMP}"
mkdir -p "$DEST" || { echo "cannot create $DEST" >&2; exit 1; }

LOG="$DEST/_collection.log"
say() { printf '%s  %s\n' "$(date -u +%H:%M:%SZ)" "$*" | tee -a "$LOG"; }

# run <output-name> <command...>
# Records the exact command at the top of each file, so the output can be
# interpreted by somebody who was not there.
run() {
  name="$1"; shift
  file="$DEST/$name.txt"
  if ! command -v "$1" >/dev/null 2>&1; then
    say "skip   $name ($1 not installed)"
    printf '# %s\n# not collected: %s is not installed on this host\n' "$*" "$1" > "$file"
    return 0
  fi
  say "run    $name"
  {
    printf '# %s\n# collected %s as uid %s\n\n' "$*" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$(id -u)"
    "$@" 2>&1
    printf '\n# exit status: %s\n' "$?"
  } > "$file"
}

# try <output-name> <gnu command...> -- <portable command...>
# Several of the most useful options here are GNU extensions: ls --time-style,
# find -printf, ps --forest. They exist on every mainstream distribution and not
# on BusyBox, Alpine or a rescue shell, which are exactly the hosts where you
# are least able to go and find another tool. So the precise form is tried
# first, and if it fails the portable form runs into the same file, which says
# which one it got.
try() {
  name="$1"; shift
  file="$DEST/$name.txt"
  first=""; second=""; seen=0
  for a in "$@"; do
    if [ "$a" = "--" ] && [ "$seen" -eq 0 ]; then seen=1; continue; fi
    if [ "$seen" -eq 0 ]; then first="$first $(printf '%q' "$a")"; else second="$second $(printf '%q' "$a")"; fi
  done
  say "run    $name"
  {
    printf '#%s\n# collected %s as uid %s\n\n' "$first" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$(id -u)"
    if out="$(eval "$first" 2>&1)"; then
      printf '%s\n\n# exit status: 0\n' "$out"
    else
      status=$?
      printf '# the form above failed with status %s on this host, so the portable form was used:\n#%s\n\n' "$status" "$second"
      eval "$second" 2>&1
      printf '\n# exit status: %s\n' "$?"
    fi
  } > "$file"
}

# listing <output-name> <path...>: names and times, never contents.
listing() {
  name="$1"; shift
  try "$name" ls -la --time-style=full-iso "$@" -- ls -la "$@"
}

say "linux-triage 1.1 on $HOST, writing to $DEST"
[ "$(id -u)" -ne 0 ] && say "NOTE   not running as root: other users' processes and sockets will be incomplete"

# ── 1. Volatile: gone at reboot, and changing while you read this ──────────
run 01-date-utc            date -u
run 01-uptime              uptime
run 02-who                 who -a
run 02-w                   w
try 03-ps-tree             ps -eo pid,ppid,user,lstart,etime,tty,stat,cmd --forest -- ps -ef
try 04-sockets             ss -plantu -- netstat -an
run 04-sockets-unix        ss -plx
run 05-open-deleted        lsof -nP +L1
run 06-arp                 ip neigh
run 06-routes              ip route
run 06-addresses           ip addr
run 07-mounts              mount
run 08-modules             lsmod

# A process whose binary has been deleted from disk keeps running, and the
# kernel marks the symlink. This is the single cheapest check for a dropped
# and removed implant, and it disappears when the process does.
say "run    09-deleted-binaries"
{
  printf '# for p in /proc/[0-9]*; readlink exe | grep deleted\n\n'
  for p in /proc/[0-9]*; do
    exe="$(readlink "$p/exe" 2>/dev/null)" || continue
    case "$exe" in
      *"(deleted)"*) printf '%s\t%s\t%s\n' "${p#/proc/}" "$exe" "$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)";;
    esac
  done
} > "$DEST/09-deleted-binaries.txt"

# ── 2. Accounts and logons ─────────────────────────────────────────────────
try 10-last                last -Faiwx -- last
try 10-lastb               lastb -Faiwx -- lastb
run 10-lastlog             lastlog
run 11-passwd              cat /etc/passwd
run 11-group               cat /etc/group
run 11-sudoers             cat /etc/sudoers
listing 11-sudoers-d       /etc/sudoers.d

# authorized_keys is persistence that survives a password reset. Listed for
# every account with a home directory, contents included: these are public
# keys, and which key was added is the finding.
say "run    12-authorized-keys"
{
  printf '# authorized_keys and authorized_keys2 for every home directory\n\n'
  cut -d: -f1,6 /etc/passwd 2>/dev/null | while IFS=: read -r user home; do
    for f in "$home/.ssh/authorized_keys" "$home/.ssh/authorized_keys2"; do
      [ -f "$f" ] || continue
      printf '== %s (%s)\n' "$f" "$user"
      ls -la --time-style=full-iso "$f" 2>/dev/null || ls -la "$f" 2>&1
      cat "$f" 2>&1
      printf '\n'
    done
  done
} > "$DEST/12-authorized-keys.txt"

# ── 3. Persistence ─────────────────────────────────────────────────────────
run 20-crontab-system      cat /etc/crontab
listing 20-cron-dirs       /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly
listing 20-cron-spool      /var/spool/cron /var/spool/cron/crontabs
run 21-systemd-units       systemctl list-unit-files --no-pager
run 21-systemd-timers      systemctl list-timers --all --no-pager
run 21-systemd-running     systemctl list-units --type=service --state=running --no-pager
listing 22-systemd-etc     /etc/systemd/system /usr/lib/systemd/system
listing 23-init-d          /etc/init.d
run 24-ld-preload          cat /etc/ld.so.preload
listing 25-profile-d       /etc/profile.d
listing 26-tmp             /tmp /var/tmp /dev/shm

# ── 4. Recently changed files in the places that matter ────────────────────
# Seven days, names and times only. mtime is settable by anybody who owns the
# file, so an absence here proves nothing; ctime is harder to fake and is
# what the second listing sorts on.
try 30-recent-etc          find /etc -xdev -type f -mtime -7 -printf '%TY-%Tm-%TdT%TH:%TM:%TS\t%CY-%Cm-%CdT%CH:%CM:%CS\t%u\t%m\t%p\n' -- find /etc -xdev -type f -mtime -7 -exec ls -ld {} +
try 30-recent-bin          find /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin -xdev -type f -ctime -7 -printf '%CY-%Cm-%CdT%CH:%CM:%CS\t%u\t%m\t%p\n' -- find /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin -xdev -type f -ctime -7 -exec ls -ld {} +
# The setuid sweep reads every directory on the root file system. On a large
# server that is minutes, and the first hour does not always have them, so it is
# the one step --quick leaves out. Where coreutils' timeout exists it is also
# capped, because a collection that never finishes collects nothing.
if [ "$QUICK" = "--quick" ]; then
  say "skip   31-suid (--quick: walks the whole disk)"
  printf '# not collected: run without --quick for the setuid sweep\n' > "$DEST/31-suid.txt"
else
  CAP=""; command -v timeout >/dev/null 2>&1 && CAP="timeout 600"
  # $CAP is unquoted on purpose: empty, it has to disappear rather than become an argument.
  # shellcheck disable=SC2086
  try 31-suid                $CAP find / -xdev -type f -perm -4000 -printf '%CY-%Cm-%CdT%CH:%CM:%CS\t%u\t%m\t%p\n' -- $CAP find / -xdev -type f -perm -4000 -exec ls -ld {} +
fi

# ── 5. Logs: a listing, and the authentication tail ────────────────────────
# Copying /var/log wholesale is a job for your imaging step. This takes the
# listing (so you can see what has rolled) and the last of the auth records.
listing 40-var-log         /var/log
run 41-journal-auth        journalctl --no-pager -o short-iso-precise -n 5000 _COMM=sshd
run 41-journal-sudo        journalctl --no-pager -o short-iso-precise -n 2000 _COMM=sudo
run 42-auth-log-tail       tail -n 5000 /var/log/auth.log
run 42-secure-tail         tail -n 5000 /var/log/secure

# ── 6. Host identity, last because it does not expire ──────────────────────
run 50-uname               uname -a
run 50-os-release          cat /etc/os-release
run 51-packages-dpkg       dpkg -l
run 51-packages-rpm        rpm -qa --last

# ── Manifest ───────────────────────────────────────────────────────────────
# Hashes of what was written, so a later copy can be shown to be this one.
# The last line the log ever receives. Everything after this point is printed
# to the terminal only: the log is one of the files being hashed, and a line
# appended after hashing makes the manifest fail verification against its own
# collection log, which is what version 1.0 of this script did.
say "hash   manifest (this log is closed; later messages go to the terminal only)"
tell() { printf '%s  %s\n' "$(date -u +%H:%M:%SZ)" "$*"; }
if command -v sha256sum >/dev/null 2>&1; then
  ( cd "$DEST" && find . -type f ! -name 'MANIFEST.sha256' -print0 | sort -z | xargs -0 sha256sum > MANIFEST.sha256 )
elif command -v shasum >/dev/null 2>&1; then
  ( cd "$DEST" && find . -type f ! -name 'MANIFEST.sha256' -print0 | sort -z | xargs -0 shasum -a 256 > MANIFEST.sha256 )
else
  tell "NOTE   no sha256 tool found; manifest not written"
fi

tell "done   $(find "$DEST" -type f | wc -l | tr -d ' ') files in $DEST"
tell "next   record this directory's manifest hash in your case notes, then image the host"