Monitoring & Server Setup

Monitor server load, disk, CPU and memory with email alerts

A small shell script, run from cron every 5 minutes, that checks four health numbers and emails you when one goes over its limit. It doesn't need an agent or a monitoring server.

What it checks

Check How it is read Default limit
1-minute load average /proc/loadavg 10.00
CPU use vmstat 1 2 (100 minus idle) 98 %
Root disk use df --output=pcent / 98 %
Memory use free (used ÷ total) 98 %

Each limit can be overridden with an environment variable, so one script works on small and large machines:

LOAD_THRESHOLD=4 DISK_THRESHOLD=90 RECIPIENTS="you@example.com" /usr/local/bin/loadmon.sh

Requirements

awk, df, free, vmstat (procps), bc, logger, a mail command (s-nail) with a working mail relay, and optionally sar (sysstat) to include recent history in the alert email.

Check for every tool at start-up, including bc, so a missing tool fails loudly instead of silently skipping alerts:

for c in awk df free vmstat bc mail logger; do
  command -v "$c" >/dev/null || { logger -t loadmon "missing $c"; exit 1; }
done

Reading the numbers

load=$(cut -d' ' -f1 /proc/loadavg)
cpu=$(vmstat 1 2 | awk 'END {print 100 - $15}')
disk=$(df --output=pcent / | tail -1 | tr -dc '0-9')
mem=$(free | awk '/^Mem:/ {printf "%d", $3 * 100 / $2}')
if (( $(echo "$load > $LOAD_THRESHOLD" | bc -l) )); then ... fi

Avoid parsing top -bn1 for CPU. Its first frame is an average since boot, and its column positions shift.

Schedule it

/etc/cron.d/loadmon:

*/5 * * * * root /usr/local/bin/loadmon.sh >/dev/null 2>&1

Tips

Auto-create home folders for Active Directory users on RHEL / Rocky

When Linux servers are joined to Active Directory (SSSD or winbind), domain users who log in for the first time have no /home/<user>.

Recommended: use the built-in feature

RHEL, Rocky and Alma 8/9 ship this already:

sudo dnf install oddjob-mkhomedir
sudo authselect enable-feature with-mkhomedir
sudo systemctl enable --now oddjobd

The home folder is then created from /etc/skel at first login, with the right owner and SELinux context.

Manual fallback (one user)

If you need to create a home folder by hand, quote everything. Group names like domain users contain a space.

#!/bin/bash
set -euo pipefail
u=${1:?usage: mkhome <user>}
id "$u" >/dev/null                      # make sure the user resolves
install -d -m 700 -o "$u" -g "domain users" "/home/$u"
cp -rT /etc/skel "/home/$u"
chown -R "$u":"domain users" "/home/$u"
restorecon -R "/home/$u"

Gotcha

Copying commands from Word or email can turn " into curly "smart quotes". The shell treats them as part of the text, so chown gets a group name that doesn't exist. Retype the quotes if a command fails mysteriously.

A production warning banner, safer aliases and a colour prompt for every login

Drop-in files in /etc/profile.d/ apply to every user's login shell. Here are three small ones.

1. Safer aliases: /etc/profile.d/custom-aliases.sh

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
alias vi='vim'

These only affect interactive shells, not scripts.

2. Production warning: /etc/profile.d/custom-prod-warning.sh

Show it only on hosts that really are production. If every box shows it, people learn to ignore it.

[[ $- == *i* && -t 0 && -t 1 ]] || return
grep -qx "$(hostname -s)" /etc/production-hosts 2>/dev/null || return
RED='\033[1;31m'; NC='\033[0m'
if command -v figlet >/dev/null; then
  figlet -f big PRODUCTION | while IFS= read -r l; do printf "${RED}%s${NC}\n" "$l"; done
else
  echo -e "${RED}***** PRODUCTION SERVER *****${NC}"
fi
echo -e "${RED}This is a LIVE system. Think twice before running commands.${NC}"

3. Two-line colour prompt: /etc/profile.d/custom-prompt-and-colors.sh

if [[ $- == *i* && -t 0 && -t 1 && -z ${SSH_ORIGINAL_COMMAND:-} ]]; then
  R=$(tput sgr0); B=$(tput bold); G=$(tput setab 8)
  RED=$(tput setaf 1); MAG=$(tput setaf 5); GRN=$(tput setaf 2); CYN=$(tput setaf 6)
  PS1="\n\[$B$G$RED\]\u\[$MAG\]@\[$RED\]\h\[$GRN\] : \d \t\[$R\]\n\[$CYN\][\w]\[$R$B\] \\$ \[$R\]"
  export PS1
fi

Notes

List listening ports and the programs using them

Before you publish a new container port, check what is already listening. ss shows everything; this tidies the output into one line per protocol and port.

Quick one-liner

sudo ss -Htulnp | awk '{split($5,a,":"); port=a[length(a)]; print $1, port, $7}' | sort -k2,2n -u

Example output:

tcp 22 users:(("sshd",pid=812,fd=3))
tcp 443 users:(("docker-proxy",pid=1450,fd=4))
udp 53 users:(("dnsmasq",pid=990,fd=5))

Older systems and NAS devices

Where ss is missing, use netstat -tulnp (net-tools). The columns differ: the local address is field 4 and the program is field 7.

Colour (optional)

If you add colour, sort first and colour last. Sorting text that already contains ANSI codes gives the wrong order. Also, match(str, re, arr) in awk is a GNU awk (gawk) feature; busybox and mawk don't have it.

Docker-specific view

docker ps --format '{{.Names}}\t{{.Ports}}' | column -t

Email yourself an internet speed test report

Run Ookla's speed test on a Linux box and email the result, for example to keep a record of your ISP's performance.

Install the Ookla CLI

Install it from Ookla's package repository, rather than piping a web script into sudo bash:

# Download the repo setup script, read it, then run it
curl -fsSLo /tmp/ookla.sh https://packagecloud.io/install/repositories/ookla/speedtest-cli/script.rpm.sh
less /tmp/ookla.sh && sudo bash /tmp/ookla.sh
sudo dnf install speedtest

To check whether a package is installed, use rpm -q speedtest. rpm -qa speedtest always returns success, even when the package is missing.

Run and send

#!/bin/bash
set -euo pipefail
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
speedtest --accept-license --accept-gdpr > "$tmp/speedtest.txt"
enscript -q -p - "$tmp/speedtest.txt" | ps2pdf - "$tmp/speedtest.pdf"
mutt -s "Speed test $(hostname -s) $(date +%F)" -a "$tmp/speedtest.pdf" -- you@example.com < "$tmp/speedtest.txt"

Points to note:

Requirements

speedtest, enscript, ghostscript (ps2pdf), mutt and a working mail relay.

Schedule it (optional)

0 7 * * * root /usr/local/bin/speedtest-report.sh