In the previous article, we talked about the most common folders and locations where to find 3rd party software logs, kernel logs, and system logs, along with default software and hardware logs, for two of the most popular Linux distributions out there, Debian and CentOS.
In today’s article, we will tackle the topic of how to interpret and analyze log files, but also to check and recognize basic errors.
How to analyze logs using Systemd
Systemd has become the default init software for most of the modern Linux distributions. Since the year 2014, when the Debian and Ubuntu distributions were upgraded to use Systemd, every sysadmin or Linux user has interacted or used Systemd. Because Systemd is a complicated software package, we will talk briefly about the log analysis function present in Systemd. Every task in Systemd is treated as a unit. You can list active units through:
systemctl list-units
By adding the parameter –all, it will display inactive units also.
All the logs created by Systemd can be seen through a tool called Journal. If journalctl is executed without any parameter, it will output the entire Journal content. But you can also see logs just for certain units, such as Apache:
journalctl -u Apache
We can also use the parameters –since and –until like this:
journalctl -u Apache --since "2018-08-22 11:55:59" --until "2019-01-22 23:59:59"
This command will export all Apache logs between those 2 dates, but we can also use keywords such as “Sunday” or “Monday”.
If you need to export logs from multiple units, you can use something like this:
journalctl -u Apache -u lighttpd --since today
This is just a brief explanation of what journalctl can do; if you want to see the entire list of feature you can read the manual page (man journalctl).
Bash tools
Linux gives you the opportunity to use several terminal tools to analyze logs. These tools are handy for small amounts of data, but can take a long time to finish a task if you enter large data fields.
CUT
The cut command allows you to analyze data from delimited logs. Delimiters are used to split fields into pairs.
For example:
pam_unix(su:auth): authentication auth; logname=ubuntu uid=1222 euid=1 tty=/dev/pts/1 ruser=ubuntu rhost= user=root
Through cut, we can get the text after the equal sign. Like this:
grep “authentication auth” /var/log/auth.log | cut -d ‘=’ -f 6
root
ubuntu
debian
horse
guest
Log Management Systems
By using these, you can make your life a little easier by going through large amounts of data in a short process. They can process fast and automatically standard formats for Linux logs and web server logs. That helps you solve a problem faster as you can focus on the solution instead of formulating logs in a way to be read by such a system.
How to Filter Errors
Most sysadmins look at logs to search for errors or when they encounter issues. By default, Linux doesn’t categorize them by their gravity or distinguishes between a small or a large error, making our jobs a little harder.
But with the use of rsyslog, we can modify its configuration file to display the hardness of an error in a way that is easy to read and interpret by a human operator.
Hope this article has helped you learn a little bit more about Linux today.