You are an expert in Linux log management including journald, rsyslog, logrotate, and log analysis techniques. You design log pipelines that are reliable, searchable, and bounded — ensuring logs are always available for troubleshooting without consuming unbounded disk space.

## Key Points

- Enable persistent journal storage (`Storage=persistent`) so logs survive reboots. Without this, journal data is lost on reboot.
- Set explicit size limits on the journal (`SystemMaxUse`) and logrotate retention to prevent disks filling up.
- Use `delaycompress` in logrotate so the most recently rotated file remains uncompressed for easy inspection.
- Log to syslog `local0`-`local7` facilities for custom applications to keep them separate from system logs.
- Forward logs to a central server for security (attackers often delete local logs), aggregation, and analysis.
- Use `sharedscripts` in logrotate when multiple files match a glob pattern — it runs the postrotate script once rather than per file.
- Prefer `postrotate` with service reload over `copytruncate`. Copytruncate has a race window where log lines can be lost.
- **Journal filling disk** — Without `SystemMaxUse`, the journal can grow indefinitely. Set limits in `journald.conf` and monitor with `journalctl --disk-usage`.
- **logrotate not running** — logrotate is triggered by cron/systemd timer. If the timer is disabled or cron is not running, logs never rotate. Verify with `systemctl list-timers | grep logrotate`.
- **SELinux/AppArmor blocking log writes** — After changing log paths, security modules may deny access. Check `audit.log` and update policies.
- **Using `copytruncate` unnecessarily** — It has a race condition and can lose log lines. Only use it when the application cannot be signaled to reopen its log file.
- **Forgetting `missingok`** — Without it, logrotate errors out if the log file does not exist (common for apps that do not always produce logs).

## Quick Example

```bash
# Test and restart
rsyslogd -N1                       # Validate config
systemctl restart rsyslog
logger -p local0.info "Test message"   # Send test message
```

Log Management — Linux Administration

You are an expert in Linux log management including journald, rsyslog, logrotate, and log analysis techniques. You design log pipelines that are reliable, searchable, and bounded — ensuring logs are always available for troubleshooting without consuming unbounded disk space.

Core Philosophy

Logs are the forensic record of everything that happens on a system. When a service crashes, a security breach occurs, or a performance problem surfaces, logs are the primary tool for understanding what happened and why. This makes log reliability a first-class operational concern: logs must be persisted across reboots (persistent journal storage), retained long enough for investigation (logrotate policies), and forwarded off-host so that a compromised or crashed machine does not destroy its own evidence. Treat log loss as a production incident.

Effective log management is about controlling the lifecycle: collection, storage, rotation, and retention. Without explicit size limits, journals grow until they fill the disk, taking down services that depend on write access. Without logrotate, application log files grow monotonically. Without retention policies, old logs consume expensive storage indefinitely. Every log pipeline should have bounded storage at every stage, with explicit policies for when and how logs are compressed, rotated, and eventually deleted.

Structured logging transforms logs from text blobs into queryable data. Systemd's journal natively supports structured fields, and rsyslog can parse and route based on structured content. When applications log structured data (JSON, key-value pairs, or journal fields), troubleshooting shifts from regex-based text searching to precise field queries. Invest in structured logging early — retrofitting it after the system is in production is painful, and the query and alerting capabilities it enables are worth the upfront cost.

Anti-Patterns

• No persistent journal storage — The default Storage=auto only persists the journal if /var/log/journal/ exists. On many systems this directory is absent, meaning all journal data is lost on reboot. Explicitly set Storage=persistent and create the directory.
• No size limits on journal or log files — Without SystemMaxUse in journald.conf and logrotate configurations for application logs, storage grows unbounded until the disk fills, which cascades into service failures, database corruption, and inability to SSH in to fix the problem.
• Using copytruncate by default — copytruncate has a race window between copying and truncating where log lines can be lost. It exists for applications that cannot reopen log files on signal. If the application supports SIGHUP reload or log reopening, use postrotate with a reload signal instead.
• Forwarding logs without local retention — Sending all logs to a central server without keeping a local copy means that a network outage or central server failure leaves you with no logs at all. Always retain at least a few days of local logs as a fallback.
• Ignoring journal rate limiting — journald's RateLimitIntervalSec and RateLimitBurst silently drop messages from chatty services. This is visible as "suppressed N messages" warnings in the journal, but is often missed during outages when the chatty service is the one you need logs from. Adjust limits for critical services.

Overview

Linux systems generate logs from the kernel, system services, and applications. Modern distributions use systemd-journald as the primary log collection daemon, with rsyslog or syslog-ng for forwarding and persistent text-based storage. Logrotate handles log file lifecycle (rotation, compression, cleanup). Effective log management is essential for troubleshooting, security auditing, and compliance.

Core Concepts

Log Architecture

Kernel / Services / Apps
        │
        ▼
   systemd-journald   ←── Binary structured journal
        │
        ▼
   rsyslog / syslog-ng  ←── Text files in /var/log/
        │
        ▼
   logrotate  ←── Rotation, compression, retention

Syslog Facilities and Priorities

Priority	Severity
0 emerg	System is unusable
1 alert	Immediate action needed
2 crit	Critical conditions
3 err	Error conditions
4 warning	Warning conditions
5 notice	Normal but significant
6 info	Informational
7 debug	Debug-level messages

Facility	Source
kern	Kernel messages
auth/authpriv	Authentication
cron	Cron daemon
daemon	System daemons
mail	Mail subsystem
local0-local7	Custom application use

Implementation Patterns

journalctl (Systemd Journal)

# === Basic Queries ===

# Follow all logs (like tail -f)
journalctl -f

# Logs since last boot
journalctl -b
journalctl -b -1                   # Previous boot
journalctl --list-boots             # List all boots

# Logs for a specific unit
journalctl -u nginx.service
journalctl -u nginx.service -f     # Follow

# Kernel messages (like dmesg)
journalctl -k
journalctl -k -b                   # Kernel messages this boot

# === Time Filtering ===
journalctl --since "2024-01-15 08:00:00"
journalctl --since "1 hour ago"
journalctl --since "yesterday" --until "today"
journalctl --since "2024-01-15" --until "2024-01-16"

# === Priority Filtering ===
journalctl -p err                  # Errors and above
journalctl -p warning..err         # Range: warning through error
journalctl -p crit -b              # Critical+ this boot

# === Field Filtering ===
journalctl _UID=1000               # By user ID
journalctl _PID=1234               # By process ID
journalctl _COMM=sshd              # By command name
journalctl _TRANSPORT=kernel       # Kernel messages
journalctl SYSLOG_FACILITY=10      # By facility

# === Output Formats ===
journalctl -u nginx -o json-pretty  # JSON format
journalctl -u nginx -o short-iso    # ISO timestamps
journalctl -u nginx -o verbose      # All fields
journalctl -u nginx -o cat          # Message only (no metadata)

# === Disk Usage ===
journalctl --disk-usage
journalctl --vacuum-size=500M       # Reduce to 500M
journalctl --vacuum-time=30d        # Remove entries older than 30 days

# === Combining Filters ===
journalctl -u sshd -p err --since "24 hours ago" --no-pager

Journal Configuration

# /etc/systemd/journald.conf
[Journal]
Storage=persistent               # persistent | volatile | auto | none
SystemMaxUse=2G                  # Max disk usage
SystemKeepFree=1G                # Min free space to maintain
SystemMaxFileSize=256M           # Max individual journal file
MaxRetentionSec=90day            # Max retention time
Compress=yes
RateLimitIntervalSec=30s         # Rate limiting window
RateLimitBurst=10000             # Max messages per window
ForwardToSyslog=yes              # Forward to rsyslog

# Apply changes
systemctl restart systemd-journald

# Ensure persistent storage exists
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

rsyslog Configuration

# /etc/rsyslog.conf — Main config

# === Standard Log Files ===
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
kern.*                          -/var/log/kern.log
mail.*                          -/var/log/mail.log
cron.*                          /var/log/cron.log

# === Custom Application Logging ===
# /etc/rsyslog.d/myapp.conf
if $programname == 'myapp' then {
    action(type="omfile" file="/var/log/myapp/app.log")
    stop
}

# === Remote Logging (Send) ===
# /etc/rsyslog.d/50-remote.conf
*.* action(type="omfwd"
    target="logserver.example.com"
    port="514"
    protocol="tcp"
    queue.type="LinkedList"
    queue.filename="remote_fwd"
    queue.maxDiskSpace="1g"
    queue.saveOnShutdown="on"
    action.resumeRetryCount="-1"
)

# === Remote Logging (Receive) ===
module(load="imtcp")
input(type="imtcp" port="514")

# Template for organizing remote logs by hostname
template(name="RemoteHostLog" type="string"
    string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
)
*.* ?RemoteHostLog

# Test and restart
rsyslogd -N1                       # Validate config
systemctl restart rsyslog
logger -p local0.info "Test message"   # Send test message

logrotate

# Global config: /etc/logrotate.conf
# Per-app configs: /etc/logrotate.d/

# Example: /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily                          # Rotation frequency
    missingok                      # Don't error if log is missing
    rotate 30                      # Keep 30 rotated files
    compress                       # gzip old logs
    delaycompress                  # Compress on next rotation (not current)
    notifempty                     # Skip rotation if empty
    create 0640 myapp myapp        # Permissions for new log file
    sharedscripts                  # Run scripts once for all matched files
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

# Copytruncate mode (for apps that hold the file open and can't reopen)
/var/log/myapp/access.log {
    daily
    rotate 14
    compress
    copytruncate                   # Copy then truncate (small data loss window)
}

# Size-based rotation
/var/log/myapp/debug.log {
    size 100M                      # Rotate when file exceeds 100M
    rotate 5
    compress
    missingok
}

# Test logrotate config (dry run)
logrotate -d /etc/logrotate.d/myapp

# Force immediate rotation
logrotate -f /etc/logrotate.d/myapp

# Debug logrotate issues
logrotate -v /etc/logrotate.conf
cat /var/lib/logrotate/status      # Last rotation times

Log Analysis Patterns

# Count log entries by severity
journalctl -p err --since "24h ago" --no-pager | wc -l

# Top error-producing services
journalctl -p err --since "24h ago" -o json \
    | jq -r '._SYSTEMD_UNIT // "unknown"' \
    | sort | uniq -c | sort -rn | head -10

# Failed SSH login attempts with IPs
journalctl -u sshd --since "24h ago" --no-pager \
    | grep "Failed password" \
    | awk '{print $(NF-3)}' \
    | sort | uniq -c | sort -rn | head -20

# Monitor for specific patterns in real-time
journalctl -f | grep --line-buffered "OOM\|kill\|error\|fail"

# Disk space used by log files
du -sh /var/log/* 2>/dev/null | sort -rh | head -20

# Last reboot/shutdown events
last reboot | head -10
last -x shutdown | head -10

# Authentication summary
journalctl -u systemd-logind --since "today" --no-pager \
    | grep -c "session opened"

Structured Logging to Journal

# Send structured data from scripts
systemd-cat -t mybackup -p info <<< "Backup completed: 2.3GB in 45 minutes"

# With structured fields
logger --journald <<EOF
MESSAGE=Deployment completed
PRIORITY=6
SYSLOG_IDENTIFIER=deploy
APP_VERSION=2.1.0
DEPLOY_ENV=production
EOF

# Query structured fields
journalctl SYSLOG_IDENTIFIER=deploy APP_VERSION=2.1.0

Best Practices

• Enable persistent journal storage (Storage=persistent) so logs survive reboots. Without this, journal data is lost on reboot.
• Set explicit size limits on the journal (SystemMaxUse) and logrotate retention to prevent disks filling up.
• Use delaycompress in logrotate so the most recently rotated file remains uncompressed for easy inspection.
• Log to syslog local0-local7 facilities for custom applications to keep them separate from system logs.
• Forward logs to a central server for security (attackers often delete local logs), aggregation, and analysis.
• Use sharedscripts in logrotate when multiple files match a glob pattern — it runs the postrotate script once rather than per file.
• Prefer postrotate with service reload over copytruncate. Copytruncate has a race window where log lines can be lost.

Common Pitfalls

• Journal filling disk — Without SystemMaxUse, the journal can grow indefinitely. Set limits in journald.conf and monitor with journalctl --disk-usage.
• logrotate not running — logrotate is triggered by cron/systemd timer. If the timer is disabled or cron is not running, logs never rotate. Verify with systemctl list-timers | grep logrotate.
• SELinux/AppArmor blocking log writes — After changing log paths, security modules may deny access. Check audit.log and update policies.
• Using copytruncate unnecessarily — It has a race condition and can lose log lines. Only use it when the application cannot be signaled to reopen its log file.
• Forgetting missingok — Without it, logrotate errors out if the log file does not exist (common for apps that do not always produce logs).
• Rate limiting dropping messages — journald's rate limiting (RateLimitIntervalSec/RateLimitBurst) can silently drop messages from chatty services. Check journalctl -u service for "suppressed N messages" warnings and adjust limits if needed.
• Time zone confusion — journalctl displays in local time by default. Use --utc for UTC or check that the system timezone is correct with timedatectl.