Linux Administration
Volume 5 — Platform Engineering
Linux Fundamentals covered the concepts; this chapter is the hands-on administration layer — managing users, running services, scheduling recurring jobs, and knowing where to look when a Linux host itself (rather than the application on it) is the thing misbehaving.
1. User and Group Management
sudo useradd -m -s /bin/bash deploy # create a user, with a home directory and bash as their shell
sudo passwd deploy # set/change their password
sudo usermod -aG docker deploy # add an existing user to a group (here, granting Docker access)
sudo userdel -r deploy # delete a user and their home directory
groups deploy # confirm current group memberships
-aG matters specifically — usermod -G docker deploy without -a replaces all of that user's group memberships with just docker, silently removing every other group they belonged to. -a ("append") is what makes it additive instead of destructive. This is a genuinely easy mistake to make and a frustrating one to notice, since the user often doesn't complain until they hit a permission error for a group they used to be in.
2. systemd Services in Depth
A unit file defines a systemd-managed service — what to run, when, and how to handle it if it fails.
# /etc/systemd/system/traxs-agent.service
[Unit]
Description=Traxs Self-Hosted Pipeline Agent
After=network.target
[Service]
Type=simple
User=deploy
WorkingDirectory=/opt/traxs-agent
ExecStart=/opt/traxs-agent/run.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
| Section/Field | Purpose |
|---|---|
After=network.target | Ensures this service only starts once networking is available — startup ordering, not a hard dependency requirement |
Type=simple | The main process itself is the service (as opposed to forking, for processes that daemonize themselves) |
Restart=on-failure | Automatically restart if the process exits with a non-zero code — the mechanism behind a crashed service self-healing without manual intervention |
WantedBy=multi-user.target | Which system state this service should be active in — multi-user.target roughly corresponds to "normal running system, no GUI required" |
sudo systemctl daemon-reload # required after creating/editing a unit file, before systemd will recognize it
sudo systemctl enable traxs-agent # start automatically on boot
sudo systemctl start traxs-agent # start it now
sudo systemctl status traxs-agent # current status, recent log lines, whether it's actually running
sudo systemctl restart traxs-agent # restart it
sudo systemctl stop traxs-agent # stop it
daemon-reload is the step people forget — editing an existing unit file or adding a new one has no effect until systemd re-reads its configuration; systemctl start/restart alone won't pick up file changes on disk.
3. Reading Service Logs
journalctl -u traxs-agent # all logs for this specific service
journalctl -u traxs-agent -f # follow live, same idea as tail -f
journalctl -u traxs-agent --since "1 hour ago"
journalctl -u traxs-agent -p err # only error-level and above
journalctl -xe # recent logs across the whole system, with extra context — a good general "what just went wrong" command
journalctl is systemd's centralized log system — for services managed by systemd, this is generally more reliable and complete than hunting through individual files in /var/log, since systemd captures a service's stdout/stderr directly regardless of whether the application itself does any file-based logging at all.
4. Cron: Scheduled Jobs
cron runs commands on a fixed schedule, independent of systemd — still the standard tool for simple recurring tasks that don't need the fuller service lifecycle a unit file provides.
crontab -e # edit the current user's cron schedule
crontab -l # list current scheduled jobs
# minute hour day-of-month month day-of-week command
0 2 * * * /opt/scripts/cleanup-old-logs.sh
*/15 * * * * /opt/scripts/health-check.sh
| Field | Range |
|---|---|
| Minute | 0–59 |
| Hour | 0–23 |
| Day of month | 1–31 |
| Month | 1–12 |
| Day of week | 0–6 (0 = Sunday) |
A cron job's environment is minimal — it doesn't inherit your interactive shell's PATH, environment variables, or .bashrc. A script that runs perfectly when you execute it directly can fail silently under cron simply because it assumed an environment variable or a PATH entry that only exists in your interactive session. Always use absolute paths inside cron-scheduled scripts, and explicitly set any environment variables the script actually needs rather than assuming they'll be inherited.
5. Disk, Memory, and Process Monitoring
df -h # disk space per mount (also covered in Bash Reference)
free -h # memory usage, human-readable
top # live process view — CPU, memory, per-process
htop # a more readable, interactive version of top, if installed
uptime # how long the system's been running, plus load average
Load average (the three numbers from uptime, representing 1/5/15-minute averages) roughly indicates how many processes are competing for CPU time — a load average consistently above the number of CPU cores available means the system is genuinely CPU-constrained, not just busy.
6. When a systemd Service Is the Right Tool vs. a Container
| Use a systemd service when | Use a Docker container when |
|---|---|
| Managing the host's own long-running processes directly (a self-hosted pipeline agent, a local monitoring daemon) | The workload is an application meant to be portable, reproducible, and isolated from the host's specific package versions |
| Deep OS-level integration is needed (specific kernel features, hardware access) | Consistency across dev/CI/production matters more than deep host integration |
The self-hosted Azure DevOps agent (laptop, in the traxs-self-hosted pool) is a concrete example of the systemd-service case — it's a long-running process tied to a specific host, not something that benefits from container portability the way the API/web applications themselves do.
7. Troubleshooting Playbook
| Symptom | Likely cause | What to check |
|---|---|---|
| Edited a unit file, changes don't seem to apply | Forgot systemctl daemon-reload | Run it, then restart the service |
Service fails to start, systemctl status shows "failed" | Check the actual error, don't guess | journalctl -u servicename -p err for the specific failure reason |
| Script works manually, fails under cron | Cron's minimal environment lacks assumed PATH/env vars | Use absolute paths; explicitly export any needed environment variables inside the script itself |
| User lost access to a group they used to have | usermod -G without -a replaced their group memberships instead of adding to them | usermod -aG going forward; re-add any groups that were dropped |
| Service restarts in a loop | Restart=on-failure combined with a genuine startup crash, not a one-off failure | journalctl -u servicename to find the actual crash reason, not just observe the restart pattern |
| Host is sluggish, unclear why | High CPU, memory pressure, or disk near-full | top/htop for CPU/process, free -h for memory, df -h for disk — narrow down which resource is actually constrained before assuming a specific cause |
8. Quick Reference
| Category | Command | Purpose |
|---|---|---|
| Users | useradd -m -s /bin/bash name | Create a user with a home directory |
| Users | usermod -aG group name | Add to a group (note -a, additive) |
| Services | systemctl daemon-reload | Required after editing/adding a unit file |
| Services | systemctl enable --now name | Enable on boot and start immediately |
| Services | journalctl -u name -f | Follow a service's logs live |
| Cron | crontab -e | Edit the current user's scheduled jobs |
| Monitoring | top / htop | Live CPU/memory/process view |
| Monitoring | free -h | Memory usage summary |
| Monitoring | uptime | System uptime and load average |
Part of the Traxs Engineering Handbook — Volume 5: Platform Engineering. Companion chapters in this volume: Linux Fundamentals, Kubernetes Fundamentals, CI/CD, Observability, Secrets Management.