Vulnarex | Elite Cybersecurity Learning Platform

#Logging, Auditing, and Monitoring#link

If you did not log it, it did not happen. Logging and auditing are the foundation of security monitoring, incident response, and compliance. Linux provides multiple logging subsystems, and understanding each one is essential for detecting attacks, investigating incidents, and meeting regulatory requirements.

The Linux Logging Architecture

Modern Linux systems use two primary logging mechanisms: systemd-journald (the binary journal) and rsyslog (the traditional text-based syslog). They work together — journald collects logs and forwards them to rsyslog for persistent storage.

Log File	Purpose	Security Relevance
/var/log/auth.log	Authentication events	SSH logins, sudo usage, failed attempts
/var/log/syslog	General system messages	Service starts/stops, kernel messages
/var/log/kern.log	Kernel messages	Firewall drops, hardware events
/var/log/apache2/access.log	Web server requests	Web attacks, scanning activity
/var/log/apache2/error.log	Web server errors	Exploit attempts, misconfigurations
/var/log/mysql/error.log	Database errors	SQL injection attempts, auth failures
/var/log/fail2ban.log	fail2ban actions	Banned IPs, attack patterns
/var/log/audit/audit.log	Audit daemon events	File access, system calls, user actions
/var/log/wtmp	Login history	Who logged in and when (binary)
/var/log/btmp	Failed login history	Failed login attempts (binary)

journalctl — Querying the Systemd Journal

journalctl is the primary tool for querying systemd's journal. It provides powerful filtering capabilities that are essential for security analysis.

journalctl Security Queries

root@vulnarex:~## View all logs from the last hour journalctl --since "1 hour ago" # View authentication-related logs journalctl -u sshd --since today # View logs from a specific user journalctl _UID=1000 --since today # View logs with priority warning or higher journalctl -p warning --since "24 hours ago" # View kernel messages (including firewall drops) journalctl -k --since today # Follow logs in real-time (like tail -f) journalctl -f # View logs for a specific process journalctl _PID=2891 # View logs in JSON format (for automated parsing) journalctl -o json-pretty --since "1 hour ago" | head -20

The Linux Audit System (auditd)

While syslog captures application-level events, the Linux Audit system (auditd) captures kernel-level events — file access, system calls, and user actions at a much deeper level. This is the gold standard for security auditing on Linux.

auditd Configuration and Usage

root@vulnarex:~## Install and start auditd sudo apt install auditd -y sudo systemctl enable auditd sudo systemctl start auditd # Add a watch rule for a sensitive file sudo auditctl -w /etc/shadow -p rwxa -k shadow_access # Add a watch for SSH config changes sudo auditctl -w /etc/ssh/sshd_config -p wa -k ssh_config_change # Monitor all commands executed by a specific user sudo auditctl -a always,exit -F uid=1000 -S execve -k user_commands # Monitor privilege escalation attempts sudo auditctl -a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k privilege_escalation # List active audit rules sudo auditctl -l # Search audit logs sudo ausearch -k shadow_access sudo ausearch -m USER_LOGIN --success no sudo ausearch -ts today -k ssh_config_change

info

💡 The auditd system is required for many compliance frameworks (PCI-DSS, HIPAA, SOC 2). The -k flag assigns a 'key' to each rule, making it easy to search for specific events. Always audit /etc/passwd, /etc/shadow, /etc/sudoers, and SSH config files.

Log Analysis Techniques

Raw logs are overwhelming. You need to extract meaningful security intelligence from them. Here are practical log analysis techniques:

Security Log Analysis Commands

root@vulnarex:~## Find all failed SSH login attempts grep "Failed password" /var/log/auth.log | head -20 # Count failed attempts by IP (identify brute-force sources) grep "Failed password" /var/log/auth.log | grep -oP 'from \K[0-9.]+' | sort | uniq -c | sort -rn | head -10 # Find successful logins grep "Accepted" /var/log/auth.log # Find sudo usage (privilege escalation tracking) grep "sudo:" /var/log/auth.log | grep COMMAND # Find account modifications grep -E "(useradd|usermod|userdel|groupadd|passwd)" /var/log/auth.log # Check for log tampering (cleared logs are a red flag) ls -la /var/log/auth.log* grep "logrotate" /var/log/syslog | tail -5 # View login history last -20 # View failed login attempts lastb -20

STRICT SECURE AUDIT RULE

⚠️ The output above shows 847 failed SSH attempts from 203.0.113.50 — a clear brute-force attack. The 'last' command shows root login attempts from that same IP. This is why fail2ban and key-based authentication are essential. Also note: if you find that log files have been truncated or deleted without log rotation, that is a strong indicator of an attacker covering their tracks.

Log Integrity and Protection

Attackers who gain access often try to cover their tracks by modifying or deleting log files. Protecting log integrity is a critical defensive measure.

▪Remote logging: Forward logs to a centralized SIEM (rsyslog remote, Filebeat, Splunk forwarder)
▪Append-only logs: Use 'chattr +a' to make log files append-only (even root cannot delete without removing the attribute)
▪Log monitoring: Use AIDE or OSSEC to detect unauthorized log file modifications
▪Sufficient retention: Maintain logs for at least 90 days (1 year for compliance)
▪Time synchronization: Use NTP to ensure accurate timestamps across all systems

bash

# Make log files append-only (tamper-resistant)
sudo chattr +a /var/log/auth.log
sudo chattr +a /var/log/syslog

# Verify the attribute
lsattr /var/log/auth.log
# Output: ----a-------- /var/log/auth.log

# To remove the attribute (when you need to rotate logs):
sudo chattr -a /var/log/auth.log
# ... perform log rotation ...
sudo chattr +a /var/log/auth.log

# Configure rsyslog to forward to a remote server
# Add to /etc/rsyslog.conf:
*.* @192.168.1.200:514    # UDP forwarding
*.* @@192.168.1.200:514   # TCP forwarding (more reliable)

quiz BLOCK (★ 50 XP)

During an incident investigation, you discover that /var/log/auth.log has been truncated to 0 bytes, and there is no log rotation entry for today. What does this most likely indicate?

Select your proof vectors above