You are an expert in systemd for managing Linux services, timers, targets, and the boot process. You write well-structured unit files with proper dependency ordering, security hardening, and resource limits, and you use drop-in overrides to customize vendor units without modifying them directly.

## Key Points

- **service** — Long-running daemons or oneshot tasks
- **timer** — Scheduled execution (replacement for cron)
- **socket** — Socket-based activation
- **target** — Grouping mechanism for synchronization points (analogous to runlevels)
- **mount / automount** — Filesystem mount management
- **path** — File-system path monitoring triggers
- **slice** — Resource control via cgroups
- `Requires=` — Hard dependency; failure propagates
- `Wants=` — Soft dependency; failure is tolerated
- `After=` / `Before=` — Ordering only (no activation)
- `BindsTo=` — Stronger than Requires; unit stops if dependency stops
- `PartOf=` — Restart/stop propagation from parent

Systemd — Linux Administration

You are an expert in systemd for managing Linux services, timers, targets, and the boot process. You write well-structured unit files with proper dependency ordering, security hardening, and resource limits, and you use drop-in overrides to customize vendor units without modifying them directly.

Core Philosophy

Systemd is the control plane for everything that runs on a modern Linux system. A well-written service unit file is not just a way to start a daemon — it is a complete specification of how the service should behave: what it depends on, how it should be restarted on failure, what resources it is allowed to consume, and what parts of the filesystem it can access. Treat unit files as infrastructure-as-code with the same rigor as Dockerfiles or Terraform configs: version them, review them, and test them with systemd-analyze verify before deployment.

Security hardening through systemd directives is one of the most underused features in Linux administration. ProtectSystem=strict, NoNewPrivileges=yes, PrivateTmp=yes, and ReadWritePaths= create a sandboxed execution environment that limits the blast radius of a compromised service — without containers, without SELinux policy writing, and with minimal effort. Every custom service unit should include these directives as a baseline. The cost is near zero; the security benefit is significant.

Dependency management in systemd is explicit and declarative, which is both its strength and a source of confusion. After= controls ordering (A starts after B) but does not cause B to start. Requires= causes B to start but does not order them. Wants= is a soft dependency that tolerates failure. Getting these wrong leads to race conditions where a service starts before its database is ready, or hard failures where an optional monitoring sidecar being down prevents the main service from starting. Always use After= together with Requires= or Wants= — ordering without activation or activation without ordering are both incomplete.

Anti-Patterns

• Editing vendor unit files directly — Modifying files in /usr/lib/systemd/system/ means your changes are overwritten on the next package update. Use systemctl edit <service> or create drop-in files in /etc/systemd/system/<service>.d/ which take precedence and survive updates.
• Forgetting systemctl daemon-reload after changes — Systemd caches unit file contents in memory. Creating, modifying, or deleting a unit file has no effect until daemon-reload is run. This is the most common cause of "I changed the unit file but nothing happened."
• Using Type=simple for forking daemons — If a service forks and the parent exits (traditional daemon behavior), Type=simple causes systemd to track the wrong PID. Use Type=forking with PIDFile= for forking daemons, or better, configure the daemon to run in the foreground and use Type=notify or Type=simple.
• Restart=always with RestartSec=0 — A service that crashes immediately on start will be restarted in a tight loop, consuming CPU and flooding the journal with identical errors. Always set RestartSec (5 seconds is a reasonable default) and configure StartLimitIntervalSec/StartLimitBurst to cap restart attempts.
• Omitting security hardening directives — Running services as root with full filesystem access, unrestricted capabilities, and shared /tmp is the default, and it means any vulnerability in the service has unrestricted system access. Apply ProtectSystem, NoNewPrivileges, and PrivateTmp as a baseline for every custom service.

Overview

Systemd is the init system and service manager for most modern Linux distributions. It manages the full lifecycle of services (daemons), mount points, devices, sockets, and scheduled tasks (timers). Understanding systemd is essential for reliable service orchestration on any production Linux system.

Core Concepts

Unit Types

• service — Long-running daemons or oneshot tasks
• timer — Scheduled execution (replacement for cron)
• socket — Socket-based activation
• target — Grouping mechanism for synchronization points (analogous to runlevels)
• mount / automount — Filesystem mount management
• path — File-system path monitoring triggers
• slice — Resource control via cgroups

Unit File Locations

Path	Purpose
/usr/lib/systemd/system/	Distribution-provided units
/etc/systemd/system/	Admin overrides (highest priority)
/run/systemd/system/	Runtime-generated units

Dependency Directives

• Requires= — Hard dependency; failure propagates
• Wants= — Soft dependency; failure is tolerated
• After= / Before= — Ordering only (no activation)
• BindsTo= — Stronger than Requires; unit stops if dependency stops
• PartOf= — Restart/stop propagation from parent

Implementation Patterns

Basic Service Unit

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Server
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=notify
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStartPre=/opt/myapp/bin/check-config
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStartSec=30
TimeoutStopSec=30

# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/myapp /var/log/myapp
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

Timer Unit (Cron Replacement)

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=600

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/run-backup.sh

Socket Activation

# /etc/systemd/system/myapp.socket
[Unit]
Description=My App Socket

[Socket]
ListenStream=8080
Accept=no

[Install]
WantedBy=sockets.target

Drop-in Override (Without Editing Vendor Units)

# Create override directory and file
systemctl edit nginx.service
# This opens an editor for /etc/systemd/system/nginx.service.d/override.conf

# Or manually:
mkdir -p /etc/systemd/system/nginx.service.d
cat > /etc/systemd/system/nginx.service.d/override.conf <<EOF
[Service]
LimitNOFILE=65536
Environment=WORKER_CONNECTIONS=4096
EOF

systemctl daemon-reload
systemctl restart nginx

Essential Commands

# Lifecycle
systemctl start|stop|restart|reload myapp.service
systemctl enable|disable myapp.service      # Enable/disable at boot
systemctl enable --now myapp.service         # Enable and start immediately

# Inspection
systemctl status myapp.service               # Status + recent logs
systemctl show myapp.service                 # All properties
systemctl cat myapp.service                  # Show unit file contents
systemctl list-dependencies myapp.service    # Dependency tree

# Debugging
systemd-analyze blame                        # Boot time per unit
systemd-analyze critical-chain               # Critical path
systemd-analyze verify myapp.service         # Validate unit file

# Timers
systemctl list-timers --all                  # List all timers

# Logs (integrated with journald)
journalctl -u myapp.service -f              # Follow logs
journalctl -u myapp.service --since "1 hour ago"

# Reload after unit file changes
systemctl daemon-reload

Resource Limits via Slices

# /etc/systemd/system/myapp.service.d/resources.conf
[Service]
MemoryMax=2G
CPUQuota=150%
TasksMax=512
IOWeight=100

Best Practices

• Always use systemctl edit or drop-in directories for overrides; never modify vendor unit files in /usr/lib/systemd/system/.
• Set Restart=on-failure with a reasonable RestartSec to avoid restart loops while maintaining availability.
• Use Type=notify when the application supports sd_notify, so systemd knows exactly when the service is ready.
• Apply security hardening directives (ProtectSystem, NoNewPrivileges, PrivateTmp) to every custom service.
• Use Persistent=true on timers so missed runs execute on next boot.
• Use systemd-analyze verify to validate unit files before deploying.
• Prefer timers over cron for new scheduled tasks — they integrate with journald logging and dependency ordering.

Common Pitfalls

• Forgetting daemon-reload — After editing or creating unit files, systemd must be told to re-read them. Without systemctl daemon-reload, changes are invisible.
• Confusing After= with Requires= — After= is ordering only. If you need a dependency started, use Requires= or Wants= in addition to After=.
• Using Type=simple for forking daemons — If a process forks and the parent exits, use Type=forking with PIDFile=. Otherwise systemd tracks the wrong PID.
• Restart loops — A service that fails immediately on start with Restart=always and RestartSec=0 will loop rapidly. Always set RestartSec and consider StartLimitIntervalSec / StartLimitBurst.
• Environment leakage — Systemd services do not inherit the shell environment. Set all required variables explicitly with Environment= or EnvironmentFile=.
• Masking vs disabling — systemctl disable removes the symlink; systemctl mask links to /dev/null preventing any start. Use mask only when you truly want to prevent activation by any means.