Automating Log Rotation with Bash Scripts: A Practical Guide

Introduction

Log files are essential for monitoring and debugging applications, but they can quickly consume disk space, leading to server crashes or performance issues. A common challenge for DevOps engineers is automating log rotation to keep systems healthy. This article provides a step-by-step guide to creating a Bash script for log rotation, compression, and cleanup, ensuring your servers remain efficient and reliable.

Why Automate Log Rotation?

Manually managing log files is time-consuming and error-prone. Common issues include:

• Disk space exhaustion from large, unrotated logs
• Difficulty tracking historical logs for debugging
• Inconsistent cleanup schedules across servers
  Automating log rotation with Bash scripts saves time, reduces errors, and demonstrates your ability to handle critical system administration tasks.

Step-by-Step Guide to Building a Log Rotation Script

1. Understand the Requirements

The script should:

• Identify log files in a specified directory
• Compress logs older than a set number of days
• Delete logs older than a retention period
• Handle running processes safely
• Log its actions for auditing

2. Create the Bash Script

Below is a robust Bash script to automate log rotation. Save it as rotate_logs.sh.

#!/bin/bash

# Configuration
LOG_DIR="/var/log/myapp"
DAYS_TO_COMPRESS=7
DAYS_TO_DELETE=30
COMPRESS_CMD="gzip"
LOG_FILE="/var/log/rotate_logs.log"

# Ensure log directory exists
if [ ! -d "$LOG_DIR" ]; then
  echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: Log directory $LOG_DIR does not exist" >> "$LOG_FILE"
  exit 1
fi

# Ensure log file for script exists
touch "$LOG_FILE"
if [ $? -ne 0 ]; then
  echo "ERROR: Cannot write to $LOG_FILE"
  exit 1
fi

# Compress logs older than DAYS_TO_COMPRESS
find "$LOG_DIR" -type f -name "*.log" -mtime +"$DAYS_TO_COMPRESS" ! -name "*.gz" -exec bash -c '
  for file; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') Compressing $file" >> "'"$LOG_FILE"'"
    '$COMPRESS_CMD' "$file"
    if [ $? -eq 0 ]; then
      echo "$(date '+%Y-%m-%d %H:%M:%S') Successfully compressed $file" >> "'"$LOG_FILE"'"
    else
      echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: Failed to compress $file" >> "'"$LOG_FILE"'"
    fi
  done
' bash {} ;

# Delete compressed logs older than DAYS_TO_DELETE
find "$LOG_DIR" -type f -name "*.gz" -mtime +"$DAYS_TO_DELETE" -exec bash -c '
  for file; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') Deleting $file" >> "'"$LOG_FILE"'"
    rm -f "$file"
    if [ $? -eq 0 ]; then
      echo "$(date '+%Y-%m-%d %H:%M:%S') Successfully deleted $file" >> "'"$LOG_FILE"'"
    else
      echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: Failed to delete $file" >> "'"$LOG_FILE"'"
    fi
  done
' bash {} ;

# Check disk space after cleanup
DISK_USAGE=$(df -h "$LOG_DIR" | tail -1 | awk '{print $5}')
echo "$(date '+%Y-%m-%d %H:%M:%S') Disk usage after rotation: $DISK_USAGE" >> "$LOG_FILE"

exit 0

3. Make the Script Executable

Set permissions to allow execution:

chmod +x rotate_logs.sh

4. Test the Script

Run the script manually to ensure it works:

./rotate_logs.sh

Check the log file (/var/log/rotate_logs.log) for output:

2025-07-16 14:30:00 Compressing /var/log/myapp/app.log
2025-07-16 14:30:00 Successfully compressed /var/log/myapp/app.log
2025-07-16 14:30:01 Disk usage after rotation: 10%

5. Schedule with Cron

Automate daily execution by adding a cron job. Edit the crontab:

crontab -e

Add the following line to run the script daily at 2 AM:

0 2 * * * /path/to/rotate_logs.sh

6. Handle Running Processes

If an application is writing to a log file, compression may fail. To handle this, signal the application to reopen the log file (e.g., with SIGHUP). Modify the script to include:

# Example: Reload nginx logs
if [ -f /var/run/nginx.pid ]; then
  kill -HUP $(cat /var/run/nginx.pid)
fi

Adjust for your specific application.

Common Issues and Fixes

• Permission Errors: Ensure the script runs as a user with access to $LOG_DIR and $LOG_FILE.
• Disk Full Errors: Monitor df -h output and adjust DAYS_TO_DELETE if needed.
• Compression Failures: Verify gzip is installed (apt install gzip or yum install gzip).
• Cron Failures: Check cron logs (/var/log/syslog or /var/log/cron) for errors.

Best Practices

• Log Everything: Always log script actions for auditing and debugging.
• Test Incrementally: Test the script on a small set of files before deploying.
• Monitor Disk Usage: Use tools like prometheus or zabbix to alert on high disk usage.
• Backup Critical Logs: Copy logs to a backup location before deletion if needed.

Conclusion

Automating log rotation with Bash scripts is a critical skill for DevOps engineers, ensuring servers remain healthy and performant. This script provides a reusable, extensible solution for log management, showcasing your ability to automate system administration tasks. Save this guide for your next log rotation task, and keep your servers running smoothly!