Back to the main page
Logcheck - check my logs for errors and other activities
Everyone knows logging is important, but then there is other thing that comes after it. And many overlook this one - checking your log files.
So logcheck will simply read your log files and email you messages of interest (and yes, you can chose to ignore some lines).
For this article, I was using syslog-ng as centralized log server.
Okay, get logcheck and untar it (I am just using /.0 directory as example, you do it somewhere else).
/.0> tar -xvf logcheck-1.1.1.tar
See what you get:
/.0/logcheck-1.1.1> ls
total 112
drwxr-xr-x 4 1000 sasl 512 Oct 31 1999 .
drwxr-xr-x 4 root root 512 Aug 4 16:24 ..
-rw------- 1 root root 537 Oct 31 1999 CHANGES
-rw------- 1 root root 1183 Oct 31 1999 CREDITS
-rw------- 1 root root 14571 Oct 31 1999 INSTALL
-rw-r--r-- 1 root root 17989 Oct 31 1999 LICENSE
-rw------- 1 root root 2997 Oct 31 1999 Makefile
-rw------- 1 root root 5356 Oct 31 1999 README
-rw------- 1 root root 2097 Oct 31 1999 README.how.to.interpret
-rw------- 1 root root 3459 Oct 31 1999 README.keywords
drwxr-xr-x 2 1000 sasl 512 Oct 31 1999 src
drwxr-xr-x 9 1000 sasl 512 Oct 31 1999 systems
|
Read INSTALL file.
Here is short explanation of files you need to deal with.
- logcheck.sh
- This is main script, executes from cron, looks at log files and use grep command
- logtail
- Executable that remembers position of last reading (tracks size and inodes of log files).
For each log file will create log_file_name.offset file. Do not delete it, otherwise parsing will start from beginning again.
- logcheck.hacking
- It contains keywords that helps you 'catch' attacks on your system.
If you have these ones, expect "ACTIVE SYSTEM ATTACK" warning in your Inbox.
- logcheck.violations
- It contains keywords for negative actions, but also has some positive ones like 'su root'.
If they are captured, it will be reported under "Security Violations".
- logcheck.violations.ignore
- It contains keywords for reversed search of logcheck.violations file.
Try to put detailed and long enough string here, so you will not ignore something you want to see.
- logcheck.ignore
- It contains keywords that will not be reported. As said, put detailed in defining what you want to ignore.
What is not seen here is reported as "Unusual System Activity"
Here is how it works.
1. Run logcheck.sh using cron
2. logcheck.sh calls logtail
3. logtail determines when reading was last time and parses off "old" lines
4. logcheck.sh greps text from hacking/violations/violations.ignore/ignore files
5. messages found are e-mailed, I guess to you
Okay, let's install logcheck now.
First, I had to check and do some exercise on Makefile.
# Compiler I use
CC = gcc
# Place where keyword files go
INSTALLDIR = /usr/local/etc/logcheck
# I use gmake instead of make
sun:
gmake install SYSTYPE=sun
|
And installation now:
/.0/logcheck-1.1.1> gmake sun
gmake install SYSTYPE=sun
gmake[1]: Entering directory `/.0/logcheck-1.1.1'
Making sun
gcc -O -o ./src/logtail ./src/logtail.c
./src/logtail.c: In function `main':
./src/logtail.c:51: warning: return type of 'main' is not `int'
Creating temp directory /usr/local/etc/tmp
Setting temp directory permissions
chmod 700 /usr/local/etc/tmp
Copying files
cp ./systems/sun/logcheck.hacking /usr/local/etc/logcheck
cp ./systems/sun/logcheck.violations /usr/local/etc/logcheck
cp ./systems/sun/logcheck.violations.ignore /usr/local/etc/logcheck
cp ./systems/sun/logcheck.ignore /usr/local/etc/logcheck
cp ./systems/sun/logcheck.sh /usr/local/etc
cp ./src/logtail /usr/local/bin
Setting permissions
chmod 700 /usr/local/etc/logcheck.sh
chmod 700 /usr/local/bin/logtail
chmod 600 /usr/local/etc/logcheck/logcheck.violations.ignore
chmod 600 /usr/local/etc/logcheck/logcheck.violations
chmod 600 /usr/local/etc/logcheck/logcheck.hacking
chmod 600 /usr/local/etc/logcheck/logcheck.ignore
Done. Don't forget to set your crontab.
gmake[1]: Leaving directory `/.0/logcheck-1.1.1'
|
Now check your logcheck.sh script (yes, located now in /usr/local/etc/)
Here are some lines you want to pay attention on (again, just my example).
SYSADMIN=solarisadmin@yourcompany.com
LOGTAIL=/usr/local/bin/logtail
TMPDIR=/usr/local/etc/logcheck/tmp
GREP=egrep
MAIL=mailx
HACKING_FILE=/usr/local/etc/logcheck/logcheck.hacking
VIOLATIONS_FILE=/usr/local/etc/logcheck/logcheck.violations
VIOLATIONS_IGNORE_FILE=/usr/local/etc/logcheck/logcheck.violations.ignore
IGNORE_FILE=/usr/local/etc/logcheck/logcheck.ignore
# For centralized log server:
# In email subject replace $HOSTNAME with something like LOGCHECK
$MAIL -s "Logcheck $DATE system check" $SYSADMIN
# LOG FILE CONFIGURATION SECTION - my SunFire T2000
# I have syslog-ng as log server
# Depends how did you set it up, you may have many log files
# Maybe separate directory for hosts and services or programs
# I experience issue using *.log for example
# this is one of ideas how to deal with many log files
# Tip - use command: ls -la /logs/hosts | awk '{print $9}' | egrep log$
# to get list of log files and put them in 'for' loop
touch $TMPDIR/check.$$
for HOSTS in `ls -la /logs/hosts | awk '{print $9}' | egrep log$`
do
$LOGTAIL /logs/hosts/${HOSTS} >> $TMPDIR/check.$$
done
for SERVICES in `ls -la /logs/services | awk '{print $9}' | egrep log$`
do
$LOGTAIL /logs/services/${SERVICES} >> $TMPDIR/check.$$
done
|
At the end, do not forget to setup cron job, say run it every 15min
(except your boss is unreasonable and will ask you to do every 5 min or less, trust me I know people who would ask for this).
# Logcheck
00,15,30,45 * * * * /usr/local/etc/logcheck.sh
|
Postscript:
Empty lines in problem/ignore files are problem here (egrep will report syntax error), so below is quick fix how to examine files for empty lines.
This "fix" searches for empty lines, and if find any, removes them and sends warning email.
Note: Of course, you'll place this in the logcheck.sh script before you run logtail, right?
# NEL = Number of Empty Lines
MYFILES="
${HACKING_FILE}
${VIOLATIONS_FILE}
${VIOLATIONS_IGNORE_FILE}
${IGNORE_FILE}"
for FILE in ${MYFILES}
do
NEL=`grep "^$" ${FILE} | wc -l`
if [ ${NEL} -eq 0 ]
then
echo "There are NO empty lines in ${FILE}" > /dev/null
else
echo "There are ${NEL} empty lines in ${FILE} and they will be removed - be careful next time" | \
${MAIL} -s "Empty lines in ${FILE}" ${SYSADMIN}
/usr/bin/cat ${FILE} | /usr/bin/sed '/^$/d' > ${FILE}.out; mv ${FILE}.out ${FILE}
fi
done
|
Another thing, say there is message line repeated tons of times.
You don't want to get every single line in your Inbox, just one line and notification how many times line is repeated.
See below how to do it.
if [ -f "${HW_PROBLEM}" ]; then
if $GREP -i -f ${HW_PROBLEM} $TMPDIR/check.$$ | $GREP -v -f ${HW_IGNORE} > $TMPDIR/checkoutput.$$
then
echo >> $TMPDIR/checkreport.$$
echo "Hardware Alerts" >> $TMPDIR/checkreport.$$
echo "=-=-=-=-=-=-=-=-" >> $TMPDIR/checkreport.$$
#cat $TMPDIR/checkoutput.$$ >> $TMPDIR/checkreport.$$
# determines how many times message repeat - ignore date/time (first field)
/usr/bin/uniq -f 1 -c $TMPDIR/checkoutput.$$ \
| awk '{printf "\nMessage logs " $1 " time(s):"} {for (i=2; i <=NF; i++) printf " "$i} {printf "\n"}' \
>> $TMPDIR/checkreport.$$
FOUND=1
fi
fi
|
The uniq command compares adjacent lines and writes one copy of each input line on the output. Other copies of repeated adjacent input lines will not be written.
-f 1 = I ignore first filed, since it is date/time and is always different.
-c = this tells me a number of times that line occurred.
What is awk doing?
It prints some message first, then first field ($1) which is number of times that line occurred, some message again, and then prints other remaining fields which are from second field to last one (since NF is Number of Field).
As you can see this print is done in FOR loop.
Back to the main page