systemd Service Management, Unit File Configuration, and Troubleshooting

• Introduction
• 1. Official Documentation Sources
• 2. SysVinit vs systemd Comparison
• 3. Basic Unit File Structure
  • 3.1 Unit File Locations
  • 3.2 Unit File Sections
• 4. Service Type Comparison
• 5. Production-Grade Unit File Examples
  • 5.1 Web Application Service
  • 5.2 Oneshot Initialization Script
• 6. Socket Activation
  • 6.1 Socket Unit File
  • 6.2 The Service Paired with the Socket
• 7. systemd Timers (a cron Replacement)
  • 7.1 Timer Unit File
  • 7.2 The Service the Timer Runs
  • 7.3 Timer OnCalendar Syntax
• 8. cgroup Resource Control
  • 8.1 Resource Limit Directives
  • 8.2 Monitoring Resource Usage
• 9. Log Management with journalctl
  • 9.1 Essential Query Commands
  • 9.2 Configuring Persistent Journal Storage
• 10. Dependency Management
  • 10.1 Dependency Directive Comparison
  • 10.2 Visualizing Dependencies
• 11. Essential systemctl Commands
• 12. Operational Pitfalls and Failure Cases
  • 12.1 The Restart=always Infinite Loop
  • 12.2 Dependency Cycles
  • 12.3 Using Shell Features in ExecStart
  • 12.4 A Wrong PIDFile Path
  • 12.5 A Missing EnvironmentFile
• 13. Security Hardening (Sandboxing)
• 14. Troubleshooting Checklist
  • 14.1 Service Fails to Start
  • 14.2 Common Errors and Their Causes
  • 14.3 Service Recovery Procedure
• 15. Drop-in Overrides
• 16. Real-World Debugging Scenarios
  • Scenario 1: The service is Active (running) but does not handle requests
  • Scenario 2: The service does not start after boot
• Conclusion
• References

Introduction

In modern Linux distributions, the core of service management is systemd. Running as PID 1, it handles system initialization and unifies service start/stop, dependency resolution, resource limits, and logging into a single framework.

Unlike SysVinit's sequential, shell-script-based initialization, systemd provides parallel startup, socket activation, on-demand service loading, and cgroup-based resource control by default. But to use these powerful features properly, you have to understand the structure and directives of unit files precisely.

This article works systematically through systemd, from its core concepts to writing production-grade unit files, timers, socket activation, resource limits, journal logging, and the troubleshooting cases you meet in real operations.

1. Official Documentation Sources

Consulting the official documentation is essential for understanding systemd deeply. The main references are as follows.

2. SysVinit vs systemd Comparison

This section compares the core differences between the two systems to show why the move to systemd was necessary.

3. Basic Unit File Structure

3.1 Unit File Locations

systemd loads unit files from several paths, and there is a priority order among them.

Core principle: Do not edit a unit file shipped by a package directly. Use the systemctl edit command to create an override file under /etc/systemd/system/.

3.2 Unit File Sections

Every unit file consists of three main sections.

[Unit]
# Defines the unit's metadata and dependencies
Description=Service description
Documentation=https://example.com/docs
After=network.target        # Start after this unit (ordering)
Requires=postgresql.service  # This unit is required (dependency)
Wants=redis.service          # Nice to have if present (weak dependency)

[Service]
# Defines how the service runs and behaves
Type=notify
ExecStart=/usr/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure

[Install]
# Where the symlink is created on systemctl enable
WantedBy=multi-user.target

4. Service Type Comparison

The Type= directive is the key setting that determines how systemd decides a service has finished starting.

Production recommendation: Use Type=notify where possible. It tells you exactly when the service is actually ready to handle requests, which makes starting dependent services safe.

5. Production-Grade Unit File Examples

5.1 Web Application Service

[Unit]
Description=My Web Application
Documentation=https://wiki.internal.example.com/myapp
After=network-online.target postgresql.service redis.service
Wants=network-online.target
Requires=postgresql.service
ConditionPathExists=/etc/myapp/config.yaml

[Service]
Type=notify
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp

# Environment variables
EnvironmentFile=-/etc/myapp/env
Environment=LANG=en_US.UTF-8

# Execution
ExecStartPre=/opt/myapp/bin/check-config --validate
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
ExecStopPost=/opt/myapp/bin/cleanup

# Restart policy
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5

# Timeouts
TimeoutStartSec=30
TimeoutStopSec=30
WatchdogSec=60

# Resource limits
MemoryMax=2G
MemoryHigh=1536M
CPUQuota=200%
TasksMax=512
LimitNOFILE=65535

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
ReadWritePaths=/var/lib/myapp /var/log/myapp
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

5.2 Oneshot Initialization Script

[Unit]
Description=Initialize application database schema
After=postgresql.service
Requires=postgresql.service
ConditionPathExists=!/var/lib/myapp/.db-initialized

[Service]
Type=oneshot
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/migrate --apply
ExecStartPost=/usr/bin/touch /var/lib/myapp/.db-initialized
RemainAfterExit=true

TimeoutStartSec=120
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

6. Socket Activation

Socket activation is a mechanism that does not start the service up front but brings it up automatically when a request arrives. It speeds up boot and saves resources.

6.1 Socket Unit File

# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket

[Socket]
ListenStream=8080
ListenStream=/run/myapp/myapp.sock
SocketUser=myapp
SocketGroup=myapp
SocketMode=0660

# Connection queue size
Backlog=4096

# Concurrent connection limits
MaxConnections=256
MaxConnectionsPerSource=16

# Socket options
KeepAlive=true
NoDelay=true

# Accept mode
# false: pass the socket fd to the service (recommended)
# true: create a service instance per connection (inetd style)
Accept=false

[Install]
WantedBy=sockets.target

6.2 The Service Paired with the Socket

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target
Requires=myapp.socket

[Service]
Type=notify
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/server
# With socket activation the service receives file descriptors from systemd
# Check via sd_listen_fds() or the LISTEN_FDS environment variable

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Socket activation lets you resolve ordering-dependency problems between services. The socket is created before the service, so connection requests queue up even when the dependent service is not ready yet.

# Enable socket activation
systemctl enable --now myapp.socket

# Check socket status
systemctl status myapp.socket
systemctl list-sockets

# The service starts automatically when a request arrives
curl http://localhost:8080/health
systemctl status myapp.service  # active (running)

7. systemd Timers (a cron Replacement)

systemd timers provide more precise scheduling and better logging than cron.

7.1 Timer Unit File

# /etc/systemd/system/backup-database.timer
[Unit]
Description=Database Backup Timer
Documentation=https://wiki.internal.example.com/backup

[Timer]
# Run every day at 2 AM
OnCalendar=*-*-* 02:00:00

# After boot, run executions that were missed while the server was down
Persistent=true

# Delay the run randomly by up to 15 minutes (prevents many servers firing at once)
RandomizedDelaySec=900

# Use minute-level accuracy instead of exact timing (saves power)
AccuracySec=60

[Install]
WantedBy=timers.target

7.2 The Service the Timer Runs

# /etc/systemd/system/backup-database.service
[Unit]
Description=Database Backup
After=postgresql.service

[Service]
Type=oneshot
User=backup
Group=backup
ExecStart=/opt/backup/bin/pg-backup --full --compress
ExecStartPost=/opt/backup/bin/upload-to-s3

# Notify on backup failure
ExecStopPost=/opt/backup/bin/notify-on-failure

TimeoutStartSec=3600
StandardOutput=journal
StandardError=journal
SyslogIdentifier=db-backup

# Resource limits (so the backup does not affect the service)
CPUQuota=50%
IOWeight=10
Nice=19

7.3 Timer OnCalendar Syntax

# Test timer expressions
systemd-analyze calendar "*-*-* 02:00:00"
systemd-analyze calendar "Mon..Fri *-*-* 09:00:00"

# Manage timers
systemctl enable --now backup-database.timer
systemctl list-timers --all
systemctl status backup-database.timer

# Run immediately (for testing)
systemctl start backup-database.service

8. cgroup Resource Control

systemd uses cgroup v2 to limit per-service resource usage precisely. This is the key feature that keeps one runaway service from affecting the whole system.

8.1 Resource Limit Directives

[Service]
# === Memory limits ===
MemoryMax=2G           # Hard ceiling (OOM Kill when exceeded)
MemoryHigh=1536M       # Soft ceiling (raises memory reclaim pressure when exceeded)
MemorySwapMax=0        # Swap use forbidden
MemoryMin=256M         # Guaranteed memory (never reclaimed below this)

# === CPU limits ===
CPUQuota=200%          # Maximum CPU usage (200% = 2 cores)
CPUWeight=100          # Relative CPU weight (default 100, range 1-10000)
AllowedCPUs=0-3        # Which CPU cores are allowed

# === I/O limits ===
IOWeight=100           # Relative I/O weight (default 100, range 1-10000)
IOReadBandwidthMax=/dev/sda 100M   # Read bandwidth limit
IOWriteBandwidthMax=/dev/sda 50M   # Write bandwidth limit
IOReadIOPSMax=/dev/sda 1000        # Read IOPS limit

# === Other limits ===
TasksMax=512           # Maximum number of processes (threads)
LimitNOFILE=65535      # Maximum open files
LimitNPROC=4096        # Maximum processes
LimitCORE=0            # Core dump size limit (0 = disabled)

8.2 Monitoring Resource Usage

# Check per-service resource usage
systemctl status myapp.service

# cgroup details
systemd-cgtop

# Find a specific service's cgroup path
systemctl show myapp.service -p ControlGroup

# Memory details
cat /sys/fs/cgroup/system.slice/myapp.service/memory.current
cat /sys/fs/cgroup/system.slice/myapp.service/memory.max
cat /sys/fs/cgroup/system.slice/myapp.service/memory.events

# CPU utilization
cat /sys/fs/cgroup/system.slice/myapp.service/cpu.stat

9. Log Management with journalctl

systemd-journald collects every service's stdout/stderr, syslog messages, and kernel logs in a structured binary format.

9.1 Essential Query Commands

# Logs for a specific service
journalctl -u myapp.service

# Follow logs in real time (replaces tail -f)
journalctl -u myapp.service -f

# Last N lines
journalctl -u myapp.service -n 100

# Time-range filter
journalctl -u myapp.service --since "2026-03-14 09:00" --until "2026-03-14 12:00"
journalctl -u myapp.service --since "1 hour ago"

# Error level and above only
journalctl -u myapp.service -p err

# JSON output (for parsing)
journalctl -u myapp.service -o json-pretty

# Only logs since this boot
journalctl -u myapp.service -b

# Kernel logs (replaces dmesg)
journalctl -k

# A specific PID
journalctl _PID=12345

# Check disk usage
journalctl --disk-usage

# Clean up old logs
journalctl --vacuum-time=30d   # Delete entries older than 30 days
journalctl --vacuum-size=1G    # Delete once the journal exceeds 1GB

9.2 Configuring Persistent Journal Storage

By default, on many distributions the journal is stored in /run/log/journal/ and disappears on reboot. To switch to persistent storage, configure it as follows.

# Create the persistent storage directory
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

# /etc/systemd/journald.conf settings
# Storage=persistent
# Compress=yes
# SystemMaxUse=2G
# SystemKeepFree=4G
# MaxRetentionSec=90day
# MaxFileSec=1month
# ForwardToSyslog=yes

# Apply the settings
systemctl restart systemd-journald

10. Dependency Management

10.1 Dependency Directive Comparison

Key point: Requires= and After= are separate things. With only Requires=B, A and B start at the same time. To start A after B is ready, you must use After=B alongside it.

10.2 Visualizing Dependencies

# Dependency tree for a specific unit
systemctl list-dependencies myapp.service

# Reverse dependencies (who needs me)
systemctl list-dependencies myapp.service --reverse

# Visualize the whole boot order
systemd-analyze dot | dot -Tsvg > /tmp/systemd-deps.svg

# Boot time analysis
systemd-analyze blame
systemd-analyze critical-chain myapp.service

11. Essential systemctl Commands

# === Service control ===
systemctl start myapp.service      # Start
systemctl stop myapp.service       # Stop
systemctl restart myapp.service    # Restart
systemctl reload myapp.service     # Reload configuration (ExecReload)
systemctl reload-or-restart myapp  # reload if possible, otherwise restart

# === Autostart at boot ===
systemctl enable myapp.service     # Enable (creates the symlink)
systemctl disable myapp.service    # Disable
systemctl enable --now myapp       # Enable + start immediately
systemctl is-enabled myapp         # Check whether it is enabled

# === Status checks ===
systemctl status myapp.service     # Status, recent logs, PID
systemctl is-active myapp          # active/inactive
systemctl is-failed myapp          # Whether it is failed

# === Unit file management ===
systemctl daemon-reload            # Always run after changing a unit file
systemctl cat myapp.service        # Print the unit file contents
systemctl show myapp.service       # Print all properties
systemctl edit myapp.service       # Create a drop-in override
systemctl edit --full myapp        # Edit the whole unit file

# === Masking (complete disable) ===
systemctl mask myapp.service       # Cannot be started by any means
systemctl unmask myapp.service     # Remove the mask

# === Whole system ===
systemctl list-units --type=service --state=failed  # Failed services
systemctl list-unit-files --type=service            # Installed services
systemctl --failed                                  # Summary of failed units

12. Operational Pitfalls and Failure Cases

12.1 The Restart=always Infinite Loop

One of the most common mistakes is setting Restart=always without limiting the number of start attempts.

Problem scenario: If a configuration file error makes the service crash right after starting, systemd retries the restart endlessly. CPU usage spikes and the logs fill the disk.

Fix: Always set StartLimitIntervalSec and StartLimitBurst together.

[Service]
Restart=on-failure
RestartSec=5

[Unit]
# Move the unit to the failed state after 5 or more restart attempts within 300 seconds (5 minutes)
StartLimitIntervalSec=300
StartLimitBurst=5

# What happens when the start limit is reached
# none: do nothing (default)
# reboot: reboot the system
# reboot-force: forced reboot
# reboot-immediate: immediate reboot
StartLimitAction=none

Caution: StartLimitIntervalSec and StartLimitBurst belong to the [Unit] section (not the Service section). They are ignored if you put them in the wrong section.

12.2 Dependency Cycles

Problem scenario: If A references After=B while B references After=A, a dependency cycle occurs. systemd detects this and prints a warning, but it starts up by ignoring the ordering of one of the units, so the behavior becomes unpredictable.

# Detect dependency cycles
systemd-analyze verify myapp.service
journalctl -b | grep "ordering cycle"
systemctl list-dependencies myapp.service

Mitigation strategies:

• Use socket activation to remove the ordering dependency
• Extract the shared dependency into a target unit
• Drop After= and have the service implement its own dependency wait

12.3 Using Shell Features in ExecStart

You cannot use shell features such as pipes (|), redirection (>), or variable substitution directly in ExecStart=. systemd runs the command directly without going through a shell.

# Wrong example - does not work
ExecStart=/opt/app/bin/server | tee /var/log/app.log
ExecStart=/opt/app/bin/server > /dev/null 2>&1

# Correct example - invoke the shell explicitly
ExecStart=/bin/bash -c '/opt/app/bin/server | tee /var/log/app.log'

# Better example - use systemd's logging features
ExecStart=/opt/app/bin/server
StandardOutput=journal
StandardError=journal

12.4 A Wrong PIDFile Path

In a Type=forking service, if the PIDFile= path differs from where the PID file actually lives, systemd cannot track the service state correctly.

# Correct configuration
[Service]
Type=forking
PIDFile=/run/nginx/nginx.pid
ExecStart=/usr/sbin/nginx
# The pid directive in nginx.conf must point at the same path

12.5 A Missing EnvironmentFile

# The '-' before the EnvironmentFile path means ignore the error if the file is absent
EnvironmentFile=-/etc/myapp/env

# Without the '-', the service fails to start when the file is missing
EnvironmentFile=/etc/myapp/env

13. Security Hardening (Sandboxing)

systemd offers powerful per-service sandboxing options.

[Service]
# === Essential security options ===
NoNewPrivileges=true          # Prevents privilege escalation
PrivateTmp=true               # Isolates /tmp
ProtectSystem=strict          # Read-only filesystem (exceptions via ReadWritePaths)
ProtectHome=true              # Blocks access to /home, /root, /run/user

# === Kernel protection ===
ProtectKernelModules=true     # Blocks loading/unloading kernel modules
ProtectKernelTunables=true    # Blocks writes to /proc/sys and /sys
ProtectKernelLogs=true        # Blocks access to /dev/kmsg and /proc/kmsg
ProtectControlGroups=true     # Blocks writes to /sys/fs/cgroup

# === Network restrictions ===
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX   # Allowed socket types
PrivateNetwork=false          # true blocks networking entirely

# === System call filtering ===
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM   # Return EPERM for blocked syscalls

# === Miscellaneous ===
PrivateDevices=true           # Minimizes /dev
ProtectClock=true             # Blocks changes to the system clock
RestrictNamespaces=true       # Blocks namespace creation
RestrictRealtime=true         # Blocks realtime scheduling
RestrictSUIDSGID=true         # Blocks creation of SUID/SGID files
LockPersonality=true          # Blocks changing the execution domain
MemoryDenyWriteExecute=true   # Enforces the W^X policy

# Check a service's security score
systemd-analyze security myapp.service

# Example output:
#  OVERALL EXPOSURE LEVEL: 2.1 OK
# The lower the score, the more hardened the service is

14. Troubleshooting Checklist

A step-by-step checklist to work through when a service will not start or misbehaves.

14.1 Service Fails to Start

# Step 1: check the status
systemctl status myapp.service -l --no-pager

# Step 2: check the full logs
journalctl -u myapp.service -n 50 --no-pager

# Step 3: validate the unit file syntax
systemd-analyze verify /etc/systemd/system/myapp.service

# Step 4: did you reload after changing the unit file?
systemctl daemon-reload

# Step 5: check dependencies
systemctl list-dependencies myapp.service

# Step 6: does the executable exist, and are its permissions right?
ls -la /opt/myapp/bin/server
file /opt/myapp/bin/server

# Step 7: is SELinux/AppArmor blocking it?
ausearch -m AVC -ts recent  # SELinux
journalctl -k | grep apparmor  # AppArmor

# Step 8: check resource limits
systemctl show myapp.service -p MemoryMax,CPUQuota,TasksMax,LimitNOFILE

14.2 Common Errors and Their Causes

14.3 Service Recovery Procedure

# Reset the failure counter (after hitting StartLimitBurst)
systemctl reset-failed myapp.service

# Force-stop the service (when a clean shutdown fails)
systemctl kill myapp.service
systemctl kill -s SIGKILL myapp.service

# Pick up unit file changes
systemctl daemon-reload

# Restart the service
systemctl restart myapp.service

# Check for processes left behind in the cgroup
systemd-cgls /system.slice/myapp.service

15. Drop-in Overrides

If you edit a package-provided unit file directly, your change is overwritten on the next package update. A drop-in override lets you keep the original and change only specific settings.

# Create the drop-in file (opens an editor)
systemctl edit myapp.service
# Creates /etc/systemd/system/myapp.service.d/override.conf

# Example: add a memory limit and a restart policy
# [Service]
# MemoryMax=4G
# Restart=on-failure
# RestartSec=10

# /etc/systemd/system/myapp.service.d/override.conf
[Service]
# To change an existing ExecStart, you must first reset it to an empty value
ExecStart=
ExecStart=/opt/myapp/bin/server --config /etc/myapp/production.yaml

# Add resource limits
MemoryMax=4G
CPUQuota=300%

Important: When overriding ExecStart=, you must reset it to an empty value first and then specify the new value. Otherwise the new value is appended to the existing one and the command runs twice.

16. Real-World Debugging Scenarios

Scenario 1: The service is Active (running) but does not handle requests

# 1. Check whether the process is alive
systemctl status myapp.service  # Check the PID
ls -la /proc/PID_NUMBER/fd/     # Open file descriptors
strace -p PID_NUMBER -e trace=network  # Trace network syscalls

# 2. Check that the port is listening
ss -tlnp | grep myapp

# 3. Memory/CPU state
systemd-cgtop -n 1

# 4. Search the logs for errors
journalctl -u myapp.service -p warning --since "10 min ago"

Scenario 2: The service does not start after boot

# 1. Check the enabled state
systemctl is-enabled myapp.service

# 2. Check the dependency target
systemctl list-dependencies multi-user.target | grep myapp

# 3. Analyze the boot ordering
systemd-analyze critical-chain myapp.service

# 4. Were the conditions met?
systemctl show myapp.service -p ConditionResult,AssertResult
journalctl -u myapp.service -b | grep -i condition

Conclusion

systemd is not a simple init system but a framework that covers the entire lifecycle of service management. Here are the core principles.

1. Unit files are code: version them, have them reviewed, and test them. Catch syntax errors up front with systemd-analyze verify.
2. Choose the Type correctly: if you do not set a Type that matches how the service actually behaves, systemd cannot track its state correctly.
3. Always cap the restart policy: Restart=always without StartLimitIntervalSec and StartLimitBurst is a time bomb.
4. Set resource limits by default: MemoryMax, CPUQuota, and TasksMax are mandatory settings for a production service. They keep one runaway service from paralyzing the whole system.
5. Use security sandboxing aggressively: NoNewPrivileges=true, ProtectSystem=strict, and PrivateTmp=true are the minimum baseline security.
6. Master journalctl: structured log queries, time-range filters, and priority filters cut troubleshooting time dramatically.
7. Use drop-in overrides: do not edit package unit files directly, use systemctl edit instead.

Handling systemd well is a basic skill for running Linux servers. Adapt the examples in this article to your own environment, keep consulting the official documentation, and build a more stable service operations environment.

References

• systemd Official Manual (freedesktop.org)
• systemd.service(5) - Service Unit Configuration
• systemd.unit(5) - Unit Configuration
• systemd.exec(5) - Execution Environment Configuration
• systemd.resource-control(5) - Resource Control
• systemd.timer(5) - Timer Unit Configuration
• systemd-journald(8) - Journal Service
• Arch Wiki - systemd
• Red Hat Documentation - Managing Services with systemd
• DigitalOcean - Understanding Systemd Units and Unit Files