Auditing your Linux server for security breaches involves checking various aspects of your system to identify potential vulnerabilities or unauthorized access. Here are some command examples that you can use to perform a security audit:

Check System Logs

Linux system logs are records of events, messages, and activities that occur within the Linux operating system. These logs provide valuable information for system administrators, helping them monitor and troubleshoot various aspects of the system.

By default, the Linux log files are located under this path;

cd /var/log/

Check the syslog file on Linux

On many Linux systems, this file contains a variety of system-related messages, including kernel messages, system events, and other important information.

The Linux system logs are located in the following directory;

sudo cat /var/log/syslog

Running the above command will print out something like this on your terminal;

Jan 29 12:34:56 hostname kernel: [ 123.456789] CPU0: Core temperature above threshold, cpu clock throttled
Jan 29 12:35:00 hostname sshd[1234]: Failed password for user1 from 192.168.1.100 port 22 ssh2
Jan 29 12:36:12 hostname sudo: user2 : TTY=pts/0 ; PWD=/home/user2 ; USER=root ; COMMAND=/bin/cat /etc/passwd
Jan 29 12:40:02 hostname cron[5678]: (root) CMD ( /usr/local/bin/backup.sh )
...

This example includes entries from the syslog file and demonstrates different types of messages. Each line typically starts with a timestamp, followed by the hostname, the process that generated the log entry (e.g., kernel, sshd, sudo), and the actual log message.

Audit user authentication using the auth.log file

In Linux, the auth.log file is a system log file that contains authentication-related messages. These messages are generated by various processes and services on the system and are crucial for monitoring and auditing user authentication and authorization activities. The auth.log file is commonly found in the /var/log directory on Debian-based systems (such as Ubuntu) and is used to log events related to user authentication.

sudo cat /var/log/auth.log

Check Failed Login Attempts

You can filter out the failed login attempts using the following command

sudo grep "Failed password" /var/log/auth.log

Messages indicating unsuccessful login attempts, including incorrect passwords.

Jan 29 12:35:00 hostname sshd[1234]: Failed password for user1 from 192.168.1.100 port 22 ssh2

Check Successful logins

You can use the following command to check the successful logins

grep "Accepted" /var/log/auth.log

Here is the sample output of the above command;

Jan 29 12:34:56 hostname sshd[1234]: Accepted password for user1 from 192.168.1.100 port 22 ssh2
Jan 29 13:45:12 hostname sshd[5678]: Accepted publickey for user2 from 10.0.0.2 port 2222 ssh2: RSA SHA256:abcdef123456...
Jan 29 14:22:18 hostname su: pam_unix(su:session): session opened for user3 by (uid=0)
...

In this example:

• The lines include the timestamp, hostname, and the process (e.g., sshd for SSH daemon, su for switch user).
• The lines contain the keyword “Accepted,” indicating successful authentication.
• The log entries provide information about the user, source IP address, authentication method (password or publickey), and other details.

Check for Password changes

To check for password changes in the auth.log file using grep, you can look for lines that contain the keyword “password changed.” Password change events are typically logged with this keyword. Here’s an example command:

grep "password changed" /var/log/auth.log

This command will display lines from the auth.log file that contains the phrase “password changed,” indicating that a user has changed their password. The actual log entries might look like this:

Jan 29 14:22:18 hostname passwd[9876]: pam_unix(passwd:chauthtok): password changed for user3

In this example;

• The line includes the timestamp, hostname, and the process (passwd in this case).
• The line contains the phrase “password changed,” indicating that a password change event occurred.
• The log entry provides information about the user for whom the password was changed.

Keep in mind that the exact format of log entries may vary between different Linux distributions, so it’s a good idea to check the specific log format used on your system.

If your auth.log file is compressed (e.g., auth.log.1.gz), you can use zgrep to search within compressed files:

zgrep "password changed" /var/log/auth.log.1.gz

As always, adapt the commands based on the actual log format and the information you want to extract from the auth.log file on your specific system.

Check for Session Opened/Closed

To check for session opened or closed events in the auth.log file using grep, you can look for lines that contain specific keywords related to session events. Here’s an example command:

grep -E "(session opened|session closed)" /var/log/auth.log

This command uses the -E option for extended regular expressions and looks for lines that contain either “session opened” or “session closed.” The actual log entries might look like this:

Jan 29 12:37:42 hostname sshd[5678]: pam_unix(sshd:session): session opened for user mary by (uid=0)
Jan 29 12:40:15 hostname sshd[5678]: pam_unix(sshd:session): session closed for user mary

In this example:

• The lines include the timestamp, hostname, process (sshd in this case), and relevant information.
• The lines contain either “session opened” or “session closed,” indicating when a user session was opened or closed.
• The log entries provide information about the user and additional details about the session event.

Adjust the command based on the specific keywords or phrases used in your auth.log file to indicate session events. If your auth.log file is compressed (e.g., auth.log.1.gz), you can use zgrep:

zgrep -E "(session opened|session closed)" /var/log/auth.log.1.gz

As always, customize the commands according to the log format and the information you want to extract from the auth.log file on your specific system.

Check Currently-logged in Users on Linux

To check currently logged-in users using the who command in bash, you can simply run:

who

This command will display a list of currently logged-in users along with information about their login sessions. The output may look something like this:

user1  pts/0  2024-01-29 14:30 (192.168.1.100)
user2  pts/1  2024-01-29 15:45 (10.0.0.2)
user3  pts/2  2024-01-29 16:10 (localhost)

In this example:

• user1, user2, and user3 are the usernames of logged-in users.
• pts/0, pts/1, and pts/2 are the terminal devices associated with their sessions.
• 2024-01-29 14:30, 2024-01-29 15:45, and 2024-01-29 16:10 are the timestamps indicating when the sessions were started.
• (192.168.1.100), (10.0.0.2), and (localhost) are the IP addresses or hostnames from which the users are logged in.

The actual output may vary based on the number of logged-in users and the details of their sessions on your system.

Check Listening Ports

If you don’t have netstat installed, you can install it on a Debian-based system (such as Ubuntu) using the following commands:

sudo apt update
sudo apt install net-tools

After installation, you can use the following command to list the open ports on the system;

sudo netstat -tuln

This command displays information about TCP and UDP connections, along with the associated port numbers. The options -tuln stand for:

• -t: Show TCP connections.
• -u: Show UDP connections.
• -l: Display listening sockets.
• -n: Show numerical addresses instead of resolving hosts.

Here’s an example of how you might see the output:

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN
udp        0      0 0.0.0.0:68              0.0.0.0:*
udp        0      0 0.0.0.0:5353            0.0.0.0:*

In this example:

• The first line shows that the system is listening on port 22 for incoming TCP connections (SSH).
• The second line indicates that the system is listening on port 631 for incoming TCP connections (CUPS printing service).
• The last two lines show UDP connections.

Check Running Processes

You can check for the running processes on the Linux system using the ps aux command on the terminal as seen below;

ps aux

This command displays a detailed list of all processes running on your system. The output includes information such as the user who owns the process, the process ID (PID), the CPU and memory usage, the start time, and the command associated with each process.

Here’s a simplified example of what the output might look like:

USER       PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  12345  6789 ?        Ss   Jan01   0:03 /sbin/init
user1     1234  0.1  1.2 123456 7890 pts/0    S    Jan15   2:30 /usr/bin/example
user2     5678  0.2  0.5  67890  1234 ?        Ss   Jan20   1:00 /usr/bin/another

In this example:

• USER is the owner of the process.
• PID is the process ID.
• %CPU is the CPU usage percentage.
• %MEM is the memory usage percentage.
• VSZ is the virtual memory size in kilobytes.
• RSS is the resident set size (non-swapped physical memory used).
• TTY is the terminal associated with the process.
• STAT is the process status.
• START is the start time of the process.
• TIME is the total accumulated CPU time.
• COMMAND is the command or program associated with the process.

This information is useful for monitoring system resource usage and identifying active processes on your system. The actual output may vary based on the processes running on your system at the time of execution.

Check User Permissions

To check user permissions on a specific file or directory, you can use the ls command with the -l (long) option. Here’s an example:

ls -l /path/to/file_or_directory

This command will display detailed information about the file or directory, including the owner, group, and permissions. The output might look something like this:

-rw-r--r-- 1 user1 users 12345 Jan 29 10:00 example.txt

In this example:

• user1 is the owner of the file.
• users is the group associated with the file.
• The permissions are represented by -rw-r--r--, where the first character (-) indicates it’s a regular file, and the next three groups of three characters represent the permissions for the owner, group, and others, respectively.
• rw-: Read and write permissions for the owner.
• r--: Read-only permissions for the group.
• r--: Read-only permissions for others.

To check permissions for a directory, the output would look similar, but the first character would be d to indicate a directory.

If you want to check the permissions recursively for all files and subdirectories within a directory, you can use the -R option:

ls -lR /path/to/directory

Make sure to replace /path/to/file_or_directory or /path/to/directory with the actual path you want to inspect.

If you want to check the permissions for a specific user, you can use the getfacl command:

getfacl /path/to/file_or_directory

This command provides more detailed information about access control lists (ACLs) and permissions. Note that it might not be available on all systems by default.

Check Installed Packages

To check for installed packages on a Debian-based Linux system (like Ubuntu), you can use the following command;

dpkg -l

]This command lists all installed packages along with their versions and descriptions. The output might be quite extensive, so you may want to use tools like grep to filter the results based on specific criteria.

For example, to search for a specific package, you can use:

dpkg -l | grep package_name

Replace package_name with the name of the package you’re interested in.

Here is a simplified example of what the output might look like:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version       Architecture  Description
+++-===============-=============-==============-=================================
ii  libc-bin        2.33-0ubuntu5 amd64          GNU C Library: Binaries
ii  libc6:i386       2.33-0ubuntu5 i386           GNU C Library: Shared libraries
ii  vim             2:8.2.3458-1  amd64          Vi IMproved - enhanced vi editor

In this example:

• ii indicates that the package is installed.
• libc-bin, libc6:i386, and vim are the package names.
• 2.33-0ubuntu5, 2:8.2.3458-1 are the package versions.
• amd64, i386 are the architectures.
• Descriptions provide more details about the packages.

Check File Integrity

To check file integrity using debsums -c on a Debian-based system, you can use the following command:

sudo debsums -c

If debsums is not already installed on your Debian-based system, you can install it using the package manager. Typically, you can install debsums with the following command:

sudo apt update
sudo apt install debsums

This debsums command checks the integrity of installed Debian packages by verifying the checksums of files against the information stored in the package’s control file. The -c option is used to report any discrepancies found.

Here’s an example of what the output might look like:

/usr/share/doc/gcc-9-base/changelog.Debian.gz
ERROR: /usr/share/doc/gcc-9-base/changelog.Debian.gz
/usr/share/doc/gcc-9-base/copyright
ERROR: /usr/share/doc/gcc-9-base/copyright

In this example:

• ERROR indicates that there is a discrepancy or corruption in the file.
• The paths /usr/share/doc/gcc-9-base/changelog.Debian.gz and /usr/share/doc/gcc-9-base/copyright are the files with integrity issues.

You should investigate further to understand why these discrepancies exist. If the discrepancies are expected (due to manual modifications or other reasons), you might choose to ignore them. Otherwise, you may need to reinstall the affected packages or take appropriate actions to restore the integrity of the files.

Check Firewall Rules

To check the current firewall rules on the Linux system, you can run the following command:

sudo iptables -L

This command displays the current firewall rules, including information about chains, policies, and rules for each chain. If your system is using IPv6, you may want to use ip6tables for IPv6-specific rules:

sudo ip6tables -L

The output might look like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

In this example:

• The INPUT, FORWARD, and OUTPUT are the default chains.
• The policy for each chain is set to ACCEPT by default.
• There is a rule in the INPUT chain allowing incoming TCP traffic on port 22 (SSH).

If you have a more complex firewall setup, you will see additional rules and chains in the output.

Check for Rootkit using rkhunter on Linux

To install and use rkhunter (Rootkit Hunter) on Linux, run the following commands;

sudo apt update
sudo apt install rkhunter

Now, you can use the rkhunter to scan for rootkits on Linux using the following command;

sudo rkhunter --check

This will perform a check for rootkits, backdoors, and local exploits on your system.

The sample output of the above command might look as follows;

[ Rootkit Hunter version x.x.x ]
File updated: searched for 185 files, found 148

Warning: The following processes are using deleted files:
         Process: /usr/bin/foo    PID: 1234    File: /tmp/suspicious-file

[ Rootkit Hunter version x.x.x ]
Checking system commands...

  Performing 'strings' command checks
    Checking 'strings' command                               [ OK ]

  Performing 'shared libraries' checks
    Checking for preloading variables                        [ None found ]
    Checking for preloaded libraries                         [ None found ]
    Checking LD_LIBRARY_PATH variable                         [ Not found ]

...

System checks summary
=====================

File properties checks...
    Files checked: 148
    Suspect files: 0

Rootkit checks...
    Rootkits checked : 495
    Possible rootkits: 0

Applications checks...
    All checks passed

The system checks took: 20 seconds

All results have been written to the log file (/var/log/rkhunter.log)

In this example:

• The tool first updates its database and then performs various checks on the system.
• It checks for deleted files used by running processes.
• It performs checks on system commands, shared libraries, and other components.
• It provides a summary of file properties, rootkit checks, and application checks.
• The final summary indicates the number of files checked, any suspect files found, rootkits checked, and possible rootkits.

You should carefully review the output, especially any warnings or suspect files reported. Investigate further if any potential issues are detected. Always consider the context and the specifics of your system when interpreting the results.

Check Disk Usage

To check for unexpected disk usage on a Linux system, you can use various commands and tools to identify which directories or files are consuming the most disk space. Here are some useful commands:

Disk Usage Summary

To get an overview of disk space usage across different directories, you can use the du (disk usage) command:

du -h --max-depth=1 /

This command will display the disk usage in human-readable format (-h) for each top-level directory under the root (/). Adjust the depth level as needed.

Disk Usage with Sorting

To find the directories using the most disk space, you can combine du with sort:

du -h --max-depth=1 / | sort -h

This command sorts the disk usage results in ascending order. The largest directories will be displayed at the bottom.

Disk Usage of Subdirectories

If you want to check the disk usage of a specific directory and its subdirectories, navigate to that directory and run:

du -h --max-depth=1

Replace the dot (.) with the path to the directory you want to check.

Check Large Files:

To find the largest files in the system, you can use the find and du combination:

sudo find / -type f -exec du -h {} + | sort -h

This command finds all regular files (-type f) and uses du to calculate their sizes, then sorts them by size.

Disk Usage by User

To check disk usage by user, you can use the du command with the --user flag:

sudo du -h --max-depth=1 --one-file-system --time /home

Replace /home with the path to the directory containing user home directories.

Check Filesystem Usage

To see the overall usage of filesystems:

df -h

This command displays information about mounted filesystems and their disk space usage.

Identify Large Directories Interactively

You can use the ncdu tool (NCurses Disk Usage) to interactively explore and identify large directories:

sudo apt install ncdu  # Install ncdu if not already installed
sudo ncdu /

Navigate through directories using arrow keys and identify large directories easily.

After identifying large directories or files, investigate further to determine if the usage is expected or if there’s unnecessary data that can be cleaned up. Always be cautious and avoid deleting critical system files.

Scan for Malware using clamav

Run a malware scan using clamav. If you don’t have clamav on Linux, run the following command to get it;

sudo apt install clamav 

Now, you can scan for malware using this command;

sudo clamscan -r /

The above command performs a recursive scan (-r) on the entire filesystem (/) using the ClamAV antivirus. Here is the sample scan report;

----------- SCAN SUMMARY -----------
Known viruses: 7480718
Engine version: 0.103.5
Scanned directories: 12345
Scanned files: 98765
Infected files: 2
Data scanned: 123.45 GB
I/O buffer size: 131072 Bytes
Time: 1:23:45

In this hypothetical example:

• Known viruses represents the number of viruses in the ClamAV database.
• Engine version is the version of the ClamAV scanning engine.
• Scanned directories is the total number of directories scanned.
• Scanned files is the total number of files scanned.
• Infected files is the number of files detected as infected.
• Data scanned is the amount of data scanned during the process.
• I/O buffer size is the buffer size used during scanning.
• Time is the total time taken for the scan.

If infected files are detected, additional information about the infected files and the type of malware may be displayed.

Check for Suspicious Files

Look for files that shouldn’t be present.

sudo find / -name "suspect_filename"

Examine User Accounts

Check for suspicious user accounts or changes in user privileges

cat /etc/passwd
cat /etc/group
cat /etc/sudoers

Inspect Cron Jobs

Review your system’s cron jobs for suspicious entries

crontab -l
ls -la /etc/cron.d
ls -la /etc/cron.daily
ls -la /etc/cron.hourly
ls -la /etc/cron.monthly
ls -la /etc/cron.weekly

Check for Hidden Files

Use the find command to search for hidden files (files starting with a dot) in your home directory:

find ~ -name ".*" -type f

These commands provide a starting point for auditing your Linux server for security breaches. It’s important to customize them based on your specific server configuration and security policies. Regularly performing security audits and staying informed about security best practices are crucial for maintaining a secure server environment.

RECOMMENDED READING: How to configure Snort on Ubuntu for Intrusion Detection