Scheduling Magic

The way I look at scheduling ( from the outside looking in ) is that the system runs a program called “cron” every now and then, throughout the day.  It checks to see if there are any tasks for it to run, either tasks created with the “crontab” command, or any installed by the system.  All these tasks are all found in these five files:

    /etc/cron.d
    /etc/cron.daily
    /etc/cron.hourly
    /etc/cron.monthly
    /etc/cron.weekly

Here’s a link to an excellent explanation of how all this works:  

An interesting addition in /etc/cron.daily is the script called “sysstat”, which runs a program called “sa2” that generates ( and later deletes ) system reports, based on configuation settings found in /etc/sysstat.  So that’s where all those system reports come from.  So to find where any report or log file magically appears from, the answer will be in one of these cron files ( or in one of a user’s crontabs in /var/spool/cron/ [crontabs | atjobs| atspool ] ).  If not, there must be a process hanging around asleep ( beside cron ), or someone else is creating it.

And here’s a good explanation for having a php clean up task in /etc/cron.d/php5.…..as follows

# /etc/cron.d/php5: crontab fragment for php5
#  This purges session files in session.save_path older than X,
#  where X is defined in seconds as the largest value of
#  session.gc_maxlifetime from all your SAPI php.ini files
#  or 24 minutes if not defined.  The script triggers only
#  when session.save_handler=files.
#
#  WARNING: The scripts tries hard to honour all relevant
#  session PHP options, but if you do something unusual
#  you have to disable this script and take care of your
#  sessions yourself.
# Look for and purge old sessions every 30 minutes
09,39 *     * * *     root   [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean

The web stats (“AWstats” ) install script, defaults to installing “awstats” in /etc/cron.d.  However, seems like an expensive “over need” for the poor busy server, so I leave this following bit it out.

MAILTO=root
*/10 * * * * www-data [ -x /usr/share/awstats/tools/update.sh ] && 
                       /usr/share/awstats/tools/update.sh
# Generate static reports:
10 03 * * * www-data [ -x /usr/share/awstats/tools/buildstatic.sh ] && 
                      /usr/share/awstats/tools/buildstatic.sh

Instead, I run AWstats program only once a day, as noted in Web Site Statistics and as follows:

#  crontab -e

5 3 * * * /usr/local/src/awstats-7.6/wwwroot/cgi-bin/awstats.pl  -config=jamesharryburton.com -update > /dev/nul

This enters the daily schedule ( say at 5 minutes past 3am each day )  for running the awstats to update.

Log Rotation

Many log files are rotated once a day ( at varying times from day to day ) with a script in /etc/cron.daily called, funny enough, “logrotate”.  This script in turns, relies on the entries in the configuration file called /etc/logrotate.conf”, which is as follows:

# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}
/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}
# system-specific logs may be configured here
~                                                  

This configuation file ( ie /etc/logrotate.conf ), can be seen here to also include all sorts of  configuation files in the directory /etc/logrotate.d.  Application programs typically place their log rotation details in this directory ( ie. /etc/logrotate.d ), instead of mixing them with system type programs.

Common examples are :

apache2,
apt,
aptitude,
clamav-daemon,
clamav-freshclam,
dpkg,
exim4-base,
exim4-paniclog,
fail2ban,
mysql-server,
razor,
rsyslog,
unattended-upgrades

However, this worker machine being installed here, has a few exceptions to the norm.  Firstly, there is no “apache2” or “fail2ban” rotate skips, as these applications had to be compiled from source, and were not loaded in the typical manner, ie with apt-get ( that does everything for you ).

Others like the script “mysql-server”, are there already because MariaDB was loaded the standard way with apt-get.

So then, we simply add in these two rotate scripts ourselves.

/etc/logrotate.d# vi  apache2

/var/log/apache2/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 640 root apache
sharedscripts
postrotate
/bin/systemctl reload apache 1>/dev/null
endscript
}

( which is expecting apache log files in /var/log/apache2 directory )

and

/etc/logrotate.d # vi fail2ban 

/var/log/fail2ban.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    postrotate
         /usr/local/bin/fail2ban-client flushlogs 1>/dev/null
    endscript
    # If fail2ban runs as non-root it still needs to have write access
    # to logfiles.
    # create 640 fail2ban adm
    create 640 root adm
}

( which is expecting a log file simply as /var/log/fail2ban.log )

Lastly, is we have apache disk caching enabled, then this needs adding….

/etc/cron.daily # vi apache2

!/bin/sh # this script will run htcacheclean to clear out the cache set -e set -u type htcacheclean > /dev/null 2>&1 || exit 0 [ -e /etc/default/apache2 ]   || exit 0 # vi /etc/default/apache2 to change this to the follwoing HTCACHECLEAN_MODE=daemon HTCACHECLEAN_RUN=auto HTCACHECLEAN_SIZE=300M HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk HTCACHECLEAN_OPTIONS=”” . /etc/default/apache2 [ “$HTCACHECLEAN_MODE” = “cron” ] || exit 0 [ “$HTCACHECLEAN_RUN” = “yes” ] || ( [ “$HTCACHECLEAN_RUN” = “auto” ] && \   [ -e /etc/apache2/mods-enabled/cache_disk.load ] )  || exit 0 htcacheclean ${HTCACHECLEAN_OPTIONS} \ -p${HTCACHECLEAN_PATH}\ -l${HTCACHECLEAN_SIZE}

All well and good, however …. as we intend to be using AWstats to monitor the number of visitors ( and spammers ) to our site.  This means that the web server’s log files will be read by AWstats and need to be at a time when the log files are full for the day. As logrotate varies somewhat the time it runs from day to day, just a little, this makes it difficult for AWstats to know when the log files are full, as well as not actually covering normal day hours.  Either AWstats needs to run within the apache log rotation program, or set apache’s log rotation to be at a set to run at consistently suitable time each day for our statistics. 

Opting for the latter, create a separate script for rotating the apache log files and place it in crontab to run at the same time each day, as follows.

# vi /etc/logrotate-apache.conf 


# rotate apache log files

/var/log/apache2/*.log {
 daily
 missingok
 rotate 14
 compress
 delaycompress
 notifempty
 create 640 apache apache
 sharedscripts
 postrotate
      /bin/systemctl reload apache 1>/dev/null
 endscript
}

Now add this script to be run at one minute to 3am in the daily cron run with this:

# crontab -e

1  3  *  *  *  /usr/sbin/logrotate   /etc/logrotate-apache.conf

 Lastly, if running a separate logrortate for apache like this, take the apache2 file out of /etc/logrotate.d, that we had added above, previously.

© 2017, James Harry Burton. All rights reserved.