Back to the main page

Logcheck - check my logs for errors and other activities

Everyone knows logging is important, but then there is other thing that comes after it. And many overlook this one - checking your log files. 

So logcheck will simply read your log files and email you messages of interest (and yes, you can chose to ignore some lines).   

For this article, I was using syslog-ng as centralized log server. 

Okay, get logcheck and untar it (I am just using /.0 directory as example, you do it somewhere else). 

/.0> tar -xvf logcheck-1.1.1.tar

See what you get: 
/.0/logcheck-1.1.1> ls
total 112
drwxr-xr-x   4 1000     sasl         512 Oct 31  1999 .
drwxr-xr-x   4 root     root         512 Aug  4 16:24 ..
-rw-------   1 root     root         537 Oct 31  1999 CHANGES
-rw-------   1 root     root        1183 Oct 31  1999 CREDITS
-rw-------   1 root     root       14571 Oct 31  1999 INSTALL
-rw-r--r--   1 root     root       17989 Oct 31  1999 LICENSE
-rw-------   1 root     root        2997 Oct 31  1999 Makefile
-rw-------   1 root     root        5356 Oct 31  1999 README
-rw-------   1 root     root        2097 Oct 31  1999
-rw-------   1 root     root        3459 Oct 31  1999 README.keywords
drwxr-xr-x   2 1000     sasl         512 Oct 31  1999 src
drwxr-xr-x   9 1000     sasl         512 Oct 31  1999 systems
Read INSTALL file. Here is short explanation of files you need to deal with. Here is how it works. 1. Run using cron 2. calls logtail 3. logtail determines when reading was last time and parses off "old" lines 4. greps text from hacking/violations/violations.ignore/ignore files 5. messages found are e-mailed, I guess to you Okay, let's install logcheck now. First, I had to check and do some exercise on Makefile.
# Compiler I use
CC = gcc

# Place where keyword files go
INSTALLDIR = /usr/local/etc/logcheck

# I use gmake instead of make
                gmake install SYSTYPE=sun
And installation now:
/.0/logcheck-1.1.1> gmake sun
gmake install SYSTYPE=sun
gmake[1]: Entering directory `/.0/logcheck-1.1.1'
Making sun
gcc  -O -o ./src/logtail ./src/logtail.c
./src/logtail.c: In function `main':
./src/logtail.c:51: warning: return type of 'main' is not `int'
Creating temp directory /usr/local/etc/tmp
Setting temp directory permissions
chmod 700 /usr/local/etc/tmp
Copying files
cp ./systems/sun/logcheck.hacking /usr/local/etc/logcheck
cp ./systems/sun/logcheck.violations /usr/local/etc/logcheck
cp ./systems/sun/logcheck.violations.ignore /usr/local/etc/logcheck
cp ./systems/sun/logcheck.ignore /usr/local/etc/logcheck
cp ./systems/sun/ /usr/local/etc
cp ./src/logtail /usr/local/bin
Setting permissions
chmod 700 /usr/local/etc/
chmod 700 /usr/local/bin/logtail
chmod 600 /usr/local/etc/logcheck/logcheck.violations.ignore
chmod 600 /usr/local/etc/logcheck/logcheck.violations
chmod 600 /usr/local/etc/logcheck/logcheck.hacking
chmod 600 /usr/local/etc/logcheck/logcheck.ignore
Done. Don't forget to set your crontab.
gmake[1]: Leaving directory `/.0/logcheck-1.1.1'
Now check your script (yes, located now in /usr/local/etc/) Here are some lines you want to pay attention on (again, just my example).


# For centralized log server:
# In email subject replace $HOSTNAME with something like LOGCHECK 
$MAIL -s "Logcheck $DATE system check" $SYSADMIN

# I have syslog-ng as log server
# Depends how did you set it up, you may have many log files
# Maybe separate directory for hosts and services or programs
# I experience issue using *.log for example
# this is one of ideas how to deal with many log files 
# Tip - use command: ls -la /logs/hosts | awk '{print $9}' | egrep log$
# to get list of log files and put them in 'for' loop

touch $TMPDIR/check.$$

for HOSTS in `ls -la /logs/hosts | awk '{print $9}' | egrep log$`
         $LOGTAIL /logs/hosts/${HOSTS} >> $TMPDIR/check.$$

for SERVICES in `ls -la /logs/services | awk '{print $9}' | egrep log$`
        $LOGTAIL /logs/services/${SERVICES} >> $TMPDIR/check.$$
At the end, do not forget to setup cron job, say run it every 15min (except your boss is unreasonable and will ask you to do every 5 min or less, trust me I know people who would ask for this).
# Logcheck
00,15,30,45 * * * * /usr/local/etc/
Postscript: Empty lines in problem/ignore files are problem here (egrep will report syntax error), so below is quick fix how to examine files for empty lines. This "fix" searches for empty lines, and if find any, removes them and sends warning email. Note: Of course, you'll place this in the script before you run logtail, right?
# NEL = Number of Empty Lines

for FILE in ${MYFILES}
        NEL=`grep "^$" ${FILE} | wc -l`
        if [ ${NEL} -eq 0 ]
        echo "There are NO empty lines in ${FILE}" > /dev/null
        echo "There are ${NEL} empty lines in ${FILE} and they will be removed - be careful next time" | \
                ${MAIL} -s "Empty lines in ${FILE}" ${SYSADMIN}
        /usr/bin/cat ${FILE} | /usr/bin/sed '/^$/d' > ${FILE}.out; mv ${FILE}.out ${FILE}
Another thing, say there is message line repeated tons of times. You don't want to get every single line in your Inbox, just one line and notification how many times line is repeated. See below how to do it.
if [ -f "${HW_PROBLEM}" ]; then
        if $GREP -i -f ${HW_PROBLEM} $TMPDIR/check.$$ | $GREP -v -f ${HW_IGNORE} > $TMPDIR/checkoutput.$$
           echo >> $TMPDIR/checkreport.$$
           echo "Hardware Alerts" >> $TMPDIR/checkreport.$$
           echo "=-=-=-=-=-=-=-=-" >> $TMPDIR/checkreport.$$

           #cat $TMPDIR/checkoutput.$$ >> $TMPDIR/checkreport.$$

           # determines how many times message repeat - ignore date/time (first field)
           /usr/bin/uniq -f 1 -c $TMPDIR/checkoutput.$$ \
           | awk '{printf "\nMessage logs " $1 " time(s):"} {for (i=2; i <=NF; i++) printf " "$i} {printf "\n"}' \
           >> $TMPDIR/checkreport.$$
The uniq command compares adjacent lines and writes one copy of each input line on the output. Other copies of repeated adjacent input lines will not be written. -f 1 = I ignore first filed, since it is date/time and is always different. -c = this tells me a number of times that line occurred. What is awk doing? It prints some message first, then first field ($1) which is number of times that line occurred, some message again, and then prints other remaining fields which are from second field to last one (since NF is Number of Field). As you can see this print is done in FOR loop.
Back to the main page