Vulnarex | Elite Cybersecurity Learning Platform

#System Hardening and Incident Response#link

This is the capstone lesson of our course, where we bring together everything you have learned — filesystem navigation, user management, permissions, process monitoring, firewall configuration, SSH hardening, logging, and SELinux — into two critical professional skills: system hardening and incident response. By the end of this lesson, you will be able to harden a Linux system against attacks and respond effectively when a breach occurs.

The System Hardening Methodology

Hardening is the process of reducing a system's attack surface by removing unnecessary components, configuring security controls, and applying patches. It follows a systematic methodology: assess, configure, verify, and maintain. We will walk through each phase with practical commands.

Phase 1 — System Assessment

Before hardening, you need to understand the current state. This assessment reveals what needs to be fixed.

Comprehensive System Assessment

root@vulnarex:~## Check for world-writable files (security risk) find / -perm -0002 -type f ! -path "/proc/*" ! -path "/sys/*" 2>/dev/null # Check for files with no owner find / -nouser -o -nogroup 2>/dev/null # Check for SUID/SGID binaries find / -type f $ -perm -4000 -o -perm -2000 $ 2>/dev/null # Check for open ports ss -tulnp # Check for running services systemctl list-units --type=service --state=running # Check for failed services systemctl --failed # Check kernel parameters for security sysctl -a 2>/dev/null | grep -E "(ip_forward|rp_filter|log_martians|syncookies)" # Check password aging policy login.defs 2>/dev/null; grep -E "(PASS_MAX_DAYS|PASS_MIN_DAYS|PASS_WARN_AGE)" /etc/login.defs # Check for empty passwords awk -F: '($2 == "" || $2 == "!") {print $1}' /etc/shadow 2>/dev/null

info

💡 The password policy above (PASS_MAX_DAYS=99999, PASS_MIN_DAYS=0) is a finding. Passwords should expire every 90 days maximum, and users should not be able to change passwords immediately (minimum 1 day). These are common compliance violations.

Phase 2 — Kernel and Network Hardening

The kernel is the foundation of the OS. Securing kernel parameters (sysctl) provides protection against many network-level attacks.

text

# /etc/sysctl.d/99-security.conf — Kernel Hardening

# Disable IP forwarding (unless this is a router)
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0

# Enable SYN flood protection
net.ipv4.tcp_syncookies = 1

# Disable source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Disable ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0

# Enable reverse path filtering (anti-spoofing)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Log martian packets (suspicious source addresses)
net.ipv4.conf.all.log_martians = 1

# Ignore ICMP broadcast requests (Smurf attack prevention)
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Ignore bogus ICMP errors
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Disable IPv6 if not needed
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

# Increase ephemeral port range
net.ipv4.ip_local_port_range = 1024 65535

# Apply settings
# sudo sysctl -p /etc/sysctl.d/99-security.conf

Phase 3 — Service and User Hardening

With the kernel secured, we harden services and user accounts. This is where most of the practical security work happens.

Service and User Hardening Commands

root@vulnarex:~## Disable unnecessary services sudo systemctl disable --now telnet.socket 2>/dev/null sudo systemctl disable --now ftp.service 2>/dev/null sudo systemctl disable --now rsh.socket 2>/dev/null sudo systemctl disable --now rlogin.socket 2>/dev/null sudo systemctl disable --now rexec.socket 2>/dev/null # Disable unused services (example list) for svc in avahi-daemon cups bluetooth ModemManager; do sudo systemctl disable --now $svc 2>/dev/null echo "Disabled: $svc" done # Set secure password policy sudo sed -i 's/PASS_MAX_DAYS.*/PASS_MAX_DAYS 90/' /etc/login.defs sudo sed -i 's/PASS_MIN_DAYS.*/PASS_MIN_DAYS 1/' /etc/login.defs sudo sed -i 's/PASS_WARN_AGE.*/PASS_WARN_AGE 14/' /etc/login.defs # Lock inactive accounts (30 days) sudo useradd -D -f 30 # Set secure umask (new files: 640, new dirs: 750) echo "umask 027" >> /etc/profile echo "umask 027" >> /etc/bash.bashrc # Secure /tmp with noexec (prevents execution of malware) echo "tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev 0 0" | sudo tee -a /etc/fstab # Set password aging for existing users for user in $(awk -F: '$3 >= 1000 {print $1}' /etc/passwd); do sudo chage --maxdays 90 --mindays 1 --warndays 14 $user echo "Updated password policy for: $user" done

Phase 4 — File Integrity Monitoring

After hardening, you need to detect unauthorized changes. File integrity monitoring (FIM) tools create cryptographic hashes of critical files and alert when they change.

AIDE — File Integrity Monitoring

root@vulnarex:~## Install AIDE (Advanced Intrusion Detection Environment) sudo apt install aide -y # Initialize the baseline database sudo aideinit sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db # Run a manual check sudo aide --check # Update the database after authorized changes sudo aide --update sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db # Schedule daily checks via cron echo "0 3 * * * root /usr/bin/aide --check | mail -s 'AIDE Check' admin@example.com" | sudo tee /etc/cron.d/aide-check

STRICT SECURE AUDIT RULE

⚠️ AIDE detected changes in /etc/passwd and /etc/shadow. This could be legitimate (a new user was added) or malicious (an attacker created a backdoor account). Always investigate FIM alerts before dismissing them. Compare the changes against your change management records.

Incident Response — The Linux Way

Despite all hardening, breaches can still happen. When they do, you need a systematic incident response process. The SANS PICERL framework (Preparation, Identification, Containment, Eradication, Recovery, Lessons Learned) applies directly to Linux incident response.

Identification — Detecting the Breach

The first step is confirming that an incident has actually occurred. Here are the key Linux commands for identification:

Incident Identification Commands

root@vulnarex:~## Check for suspicious network connections ss -tnp | grep ESTAB # Check for processes with deleted executables (common malware technique) ls -la /proc/*/exe 2>/dev/null | grep deleted # Check for hidden processes ps -eo pid | sort -n > /tmp/ps_pids ls /proc/ | grep -E '^[0-9]+$' | sort -n > /tmp/proc_pids comm -23 /tmp/proc_pids /tmp/ps_pids # Check cron jobs for all users for user in $(cut -d: -f1 /etc/passwd); do echo "=== Cron for: $user ===" crontab -u $user -l 2>/dev/null done # Check for recently modified files in critical directories find /etc /bin /sbin /usr/bin /usr/sbin -type f -mtime -1 2>/dev/null # Check for new user accounts awk -F: '$3 >= 1000 {print $1, $3, $7}' /etc/passwd # Check SSH authorized keys for unauthorized entries for user in $(cut -d: -f1 /etc/passwd); do auth_keys=$(eval echo ~$user)/.ssh/authorized_keys if [ -f "$auth_keys" ]; then echo "=== $user authorized_keys ===" cat "$auth_keys" fi done # Check for kernel modules (rootkit detection) lsmod | grep -i -E "(hide|rootkit|backdoor)" # Check /etc/ld.so.preload (library injection) cat /etc/ld.so.preload 2>/dev/null

callout

The output above shows multiple critical indicators of compromise: (1) a deleted executable running as python3 with a reverse shell connection, (2) a hidden cron job in /tmp, (3) an unauthorized SSH key, and (4) a library injection via ld.so.preload. This system is actively compromised and needs immediate containment.

Containment and Evidence Preservation

Once you identify a breach, contain it without destroying evidence. The priority order is: preserve volatile evidence, isolate the system, then begin eradication.

bash

# === CONTAINMENT CHECKLIST ===

# 1. DO NOT power off the system (loses RAM evidence)
# 2. Capture volatile evidence FIRST:

# Memory capture (if LiME is available)
sudo insmod lime.ko "path=/tmp/memdump.lime format=lime"

# Running process list
ps auxww > /tmp/evidence_processes.txt

# Network connections
ss -tnp > /tmp/evidence_connections.txt
ss -ulnp > /tmp/evidence_listeners.txt

# ARP cache
ip neigh show > /tmp/evidence_arp.txt

# Loaded kernel modules
lsmod > /tmp/evidence_modules.txt

# Open files
lsof > /tmp/evidence_openfiles.txt 2>/dev/null

# 3. Isolate the system (but keep it running)
sudo ip link set eth0 down
# OR use firewall to block all traffic:
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo iptables -P FORWARD DROP
# Keep one management interface up for forensics

# 4. Capture disk evidence (after isolation)
sudo dd if=/dev/sda of=/tmp/disk_image.dd bs=4M status=progress
# OR use dcfldd for hashing:
sudo dcfldd if=/dev/sda of=/tmp/disk_image.dd hash=sha256 hashlog=/tmp/disk_hash.txt

# 5. Secure the evidence
sha256sum /tmp/evidence_*.txt > /tmp/evidence_checksums.txt
sha256sum /tmp/disk_image.dd >> /tmp/evidence_checksums.txt

# 6. Document everything
echo "Incident response started: $(date -u)" >> /tmp/ir_timeline.txt
echo "System: $(hostname) - $(uname -a)" >> /tmp/ir_timeline.txt
echo "Analyst: $(whoami)" >> /tmp/ir_timeline.txt

Eradication and Recovery

After containment and evidence preservation, you eradicate the threat and restore the system. In many cases, the safest approach is a full rebuild from known-good media.

▪Remove malicious files and processes (after evidence capture)
▪Remove unauthorized user accounts and SSH keys
▪Clear malicious cron jobs and startup scripts
▪Remove rootkits and kernel modules
▪Patch the vulnerability that was exploited
▪Restore from known-good backup OR rebuild from scratch
▪Re-apply all hardening configurations
▪Verify integrity with AIDE
▪Monitor closely for 30 days for re-compromise indicators

Hardening Checklist Summary

Here is a consolidated hardening checklist that combines everything from this course into a single reference:

Category	Action	Priority
Updates	Apply all security patches; enable automatic updates	Critical
Services	Disable all unnecessary services	Critical
Firewall	Default deny incoming; allow only required ports	Critical
SSH	Key-only auth, no root login, fail2ban, rate limiting	Critical
Users	Strong password policy, remove unused accounts, sudo restrictions	High
Permissions	Audit SUID files, set secure umask, fix world-writable files	High
Kernel	SYN cookies, disable forwarding, enable ASLR, rp_filter	High
Logging	Centralized logging, auditd rules, append-only logs	High
MAC	Enable SELinux/AppArmor in enforcing mode	Medium
FIM	Deploy AIDE or OSSEC for file integrity monitoring	Medium
Network	Disable IPv6 if unused, secure DNS, use TLS	Medium
Monitoring	Deploy IDS (Snort/Suricata), log analysis, alerting	Medium

callout

Congratulations! You have completed Linux for Security Professionals. You now have the skills to navigate the Linux filesystem, manage users and permissions, configure firewalls, harden SSH, implement logging and auditing, enforce mandatory access control with SELinux, and respond to security incidents. These are the foundational skills for every Linux security role — from SOC analyst to penetration tester to incident responder. Continue practicing in your lab, pursue certifications (CompTIA Linux+, RHCSA, LPIC-1), and build on this foundation with advanced topics like container security, cloud hardening, and threat hunting.

challenge BLOCK (★ 100 XP)

Capstone Challenge: Harden and Investigate

Select your proof vectors above

System Hardening and Incident Response

#System Hardening and Incident Response#link

The System Hardening Methodology

Phase 1 — System Assessment

Phase 2 — Kernel and Network Hardening

Phase 3 — Service and User Hardening

Phase 4 — File Integrity Monitoring

Incident Response — The Linux Way

Identification — Detecting the Breach

Containment and Evidence Preservation

Eradication and Recovery

Hardening Checklist Summary

Capstone Challenge: Harden and Investigate

Verification Proof Checkpoint