NinjaFocus - New pages [en]

SSH ID Installer

2012-02-19T09:57:15Z

Summary:

[[category:linux]]
[[category:scripts]]
[[category:bash]]

This script, when passed an ssh public id and a list of logins, will use ssh to connect to the logins provided and install the new ssh id.

Although this script contains some basic safety checks, you would be brave to try and use it with out understanding how it works and being very certain that it will work in your environment. When I want to install a new id using a script like this, the first thing I do is open ssh connections to all the servers. Then if there is an error while the script it running, I can get in and fix it.

<nowiki>
#!/bin/bash

key="$1"
shift

if [ "$key" = "$(echo "$key" | sed -e 's/[a-zA-Z0-9 +\/\.=@-]//g')" ]
then
echo supplied key contains invalid characters
exit
fi
if [ "$key" != "$(echo "$key" | egrep "^ssh-(dsa|rsa) ")" ]
then
echo supplied key format looks bad
exit
fi

date=$(date +%F-%T)

echo $date
echo "$key"

for login in $@
do
echo "$login"
if [ "$login" = "$(echo "$login" | sed -e 's/[a-zA-Z0-9 +\/\.=@-]//g')" ]
then
echo "$login Skipped: '$login' contains invalid characters"
continue
fi
ssh $login "if [ ! -f .ssh/authorized_keys ]
then
echo $login Skipped: could not find file;
exit;
fi;
if [ ! -O .ssh ]
then
echo $login Skipped: .ssh does not exist or has incorrect permissions;
exit;
fi;
cp -p .ssh/authorized_keys .ssh/authorized_keys.$date;
if [ ! -f .ssh/authorized_keys.$date ]
then
echo $login Skipped: could not backup backup existing file;
exit;
fi;
if [ ! -O .ssh/authorized_keys ];
then
rm -f .ssh/authorized_keys;
cp -p .ssh/authorized_keys.$date .ssh/authorized_keys;
fi;
echo '${key}' >> .ssh/authorized_keys;
"
echo -n "current ssh key on this machine for $login: "; ssh $login "echo OK"
done
</nowiki>

Spam-Info.py

2012-02-03T17:45:02Z

Summary: New page: [[category:linux]] [[category:python]] [[category:scripts]] Pipe in some exim main log lines, pass the email address of an abused account, get a report of how much mail they sent, how man...

[[category:linux]]
[[category:python]]
[[category:scripts]]

Pipe in some exim main log lines, pass the email address of an abused account, get a report of how much mail they sent, how many recipients it was relayed to and how spammy the mail was.

<nowiki>

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os, sys, time

fromaddr = sys.argv[1]
print fromaddr

received, recipients, rejects, relayed = 0, 0, 0, 0
msgids = []
sa_scores = {}

for line in sys.stdin:
msgid = line[20:36]
sa_index = line.find("Spamassassin score is ")
if (sa_index > -1):
sa_scores[msgid] = float(line[sa_index + 22:])
if (line.find("rejected after DATA: Spam messages are not welcome here") > -1 and line.find("F=<%s>" % (fromaddr,)) > -1):
rejects = rejects + 1
sa_scores.pop(msgid)
continue
if (line.find(" <= ") > -1):
index = line.find(" from <%s> " % (fromaddr,))
if (index > -1):
received = received + 1
rcpt_i = line.find(" for ", index + len(fromaddr) + 8) + 5
recipients = recipients + len(line[rcpt_i:].split())
msgids.append(msgid)
continue
if (msgid in msgids):
if (line.find(" => ") > -1 or line.find(" -> ") > -1):
relayed = relayed + 1

sys.stdout.write("Rejected %d emails\n" % rejects)
sys.stdout.write("Received %d emails, for %d receipients and relayed to %d recipients\n" % (received, recipients, relayed))

for msgid in sa_scores.keys():
if (msgid not in msgids):
sa_scores.pop(msgid)

try:
sa_scores = sa_scores.values()
sa_scores.sort()
sa_total = 0
for sa_score in sa_scores:
sa_total = sa_total + sa_score
sys.stdout.write("SA Scores: min %.1f, med %.1f, max %.1f\n" % (sa_scores[0], (sa_total / len(sa_scores)), sa_scores[-1]))
except:
pass
</nowiki>

Cyrus

2011-07-16T13:42:26Z

Summary:

[[Category:linux]]
[[Category:servers]]

Creating mailboxes and checking/setting permissions:
<nowiki>
$ cyradm --user cyradm imap.example.com
Password:
imap.example.com> cm office.reports
imap.example.com> lam office.reports
bob lrswiktecda
bill lrswiktecda
babs lrswiktecda
anyone p
imap.example.com> sam office.reports becca lrswipd
imap.example.com> lam office.reports
bob lrswiktecda
bill lrswiktecda
babs lrswiktecda
becca lrswipd
anyone p
</nowiki>

imtest [http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/ImtestByHand http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/ImtestByHand]
<nowiki>
imtest -t "" -u bob -a cyradm imap.example.com

-t - keyfile (blank)

-u user for authorization (to simulate access as)

-a user for authentication

hostname
</nowiki>

<nowiki>unexpunge -l user.bob</nowiki>

Exim

2011-07-16T13:37:31Z

Summary:

[[Category:linux]]
[[Category:servers]]

Various little things to do when running an exim mail server.

queue
<nowiki>$ exim -bp</nowiki>

queue summary:
<nowiki>$ exim -bp|exiqsumm</nowiki>

<nowiki>$ exim -brp | exiqsumm | tail
1 0 2252 36m 36m windsorhoteis.com.br
1 1 1126 37h 37h ww.com
1 1 6246 26h 26h yahoo.co.th
4 0 20KB 4m 0m yahoo.co.uk
9 6 22KB 47m 0m yahoo.com
2 2 2252 39h 19m yahoo.de
1 0 110KB 87m 87m yorku.ca
---------------------------------------------------------------
705 309 53MB 5d 0m TOTAL
</nowiki>

check routing:
<nowiki>bash-2.03$ exim -bt bob@example.com
bob@imap.example.com
<-- bob@example.com
router = imapmail, transport = imaptransport
host imap.example.com [10.10.0.34]
</nowiki>

headers
<nowiki>$ exim -Mvh 1Ni4hu-0000ri-KQ</nowiki>
body
<nowiki>$ exim -Mvb 1Ni4hu-0000ri-KQ</nowiki>

Stop attempting to deliver mails that have been on the queue for 4 days or more. (Bounce messages are discarded).
Senders are sent an Error message "cancelled by administrator"
<nowiki>exiqgrep -i -o 345600 | xargs exim -Mg</nowiki>

Remove frozen messages:
<nowiki>exiqgrep -iz | xargs exim -Mrm</nowiki>

<nowiki>
exim -bd -q15m
exim -Mc <msg id>

-d debug
-M <msg id>
-q process queue
-db daemon
-dbf don't fork off
-bP [setting] config dump
-bp [msg ids] show mail queue
-bpr as -bp with messages not sorted in order of arrival
-bpc number of messages in queue

mailq = exim -bp
runq = exim -q
</nowiki>

Testing Clamav

2011-07-16T13:31:27Z

Summary: New page: [[Category:linux]] [[Category:bash]] [[Category:servers]] From a remote machine allowed access clamd, use clamd-stream-client with the clamav test virus files. The test virus files are b...

[[Category:linux]]
[[Category:bash]]
[[Category:servers]]

From a remote machine allowed access clamd, use clamd-stream-client
with the clamav test virus files. The test virus files are built when compiling clamav
http://www.clamav.net/download/third-party-tools/3rdparty-misc

<nowiki>kjw-1:clamav-0.97-test-files kieranwhitbread$ for virus in clam*; do echo "$virus: $(clamd-stream-client -d clamd.example.com < $virus )"; done
clam-aspack.exe: ClamAV-Test-File
clam-fsg.exe: ClamAV-Test-File
clam-mew.exe: ClamAV-Test-File
clam-nsis.exe: ClamAV-Test-File
clam-pespin.exe: ClamAV-Test-File
clam-petite.exe: ClamAV-Test-File
clam-upack.exe: ClamAV-Test-File
clam-upx.exe: ClamAV-Test-File
clam-v2.rar: ClamAV-Test-File
clam-v3.rar: ClamAV-Test-File
clam-wwpack.exe: ClamAV-Test-File
clam-yc.exe: ClamAV-Test-File
clam.7z: ClamAV-Test-File
clam.arj: ClamAV-Test-File
clam.bin-be.cpio: ClamAV-Test-File
clam.bin-le.cpio: ClamAV-Test-File
clam.bz2.zip: ClamAV-Test-File
clam.cab: ClamAV-Test-File
clam.chm: ClamAV-Test-File
clam.d64.zip: ClamAV-Test-File
clam.ea05.exe: ClamAV-Test-File
clam.ea06.exe: ClamAV-Test-File
clam.exe: ClamAV-Test-File
clam.exe.binhex: ClamAV-Test-File
clam.exe.bz2: ClamAV-Test-File
clam.exe.html: ClamAV-Test-File
clam.exe.mbox.base64: ClamAV-Test-File
clam.exe.mbox.uu: ClamAV-Test-File
clam.exe.rtf: ClamAV-Test-File
clam.exe.szdd: ClamAV-Test-File
clam.impl.zip: ClamAV-Test-File
clam.mail: ClamAV-Test-File
clam.newc.cpio: ClamAV-Test-File
clam.odc.cpio: ClamAV-Test-File
clam.ole.doc: ClamAV-Test-File
clam.pdf: ClamAV-Test-File
clam.ppt: ClamAV-Test-File
clam.sis: ClamAV-Test-File
clam.tar.gz: ClamAV-Test-File
clam.tnef: ClamAV-Test-File
clam.zip: ClamAV-Test-File
clam_IScab_ext.exe: ClamAV-Test-File
clam_IScab_int.exe: ClamAV-Test-File
clam_ISmsi_ext.exe: ClamAV-Test-File
clam_ISmsi_int.exe: ClamAV-Test-File
clam_cache_emax.tgz: ClamAV-Test-File
</nowiki>

Telnet to port 3310 on the clamd machine:-

<nowiki>kjw-1:~ kieranwhitbread$ telnet clamd.example.com 3310
Trying 10.10.0.57...
Connected to clamd.example.com.
Escape character is '^]'.
STREAM
PORT 1480
</nowiki>

make another telnet connection to that port number. submit the test string in and close the connection.

<nowiki>kjw-1:~ kieranwhitbread$ telnet clamd.example.com 1480
Trying 10.10.0.57...
Connected to clamd.example.com.
Escape character is '^]'.
$CEliacmaTrESTuScikgsn$FREE-TEST-SIGNATURE$EEEEE$
^]
telnet> Connection closed.
</nowiki>

look at the original connection. It should state that it found the test
signature.

<nowiki>kjw-1:~ kieranwhitbread$ telnet clamd.example.com 3310
Trying 10.10.0.57...
Connected to clamd.example.com.
Escape character is '^]'.
STREAM
PORT 1480
stream: ClamAV-Test-Signature FOUND
Connection closed by foreign host.
</nowiki>

and an entry should be made in the clamd log

<nowiki>$ sudo grep ClamAV-Test-Signature /var/log/clamd
Mon Mar 28 17:42:13 2011 -> stream(138.37.8.42@1480): ClamAV-Test-Signature FOUND
</nowiki>

Squid stats.py

2011-07-16T12:58:50Z

Summary: New page: [[Redirect:Squid-Stats.py]]

[[Redirect:Squid-Stats.py]]

Thunderbird debug

2011-07-16T12:38:14Z

Summary: New page: [[Category:Linux]] [[Category:Desktop]] Use this when thunderbird goes mad and you want to know why. <nowiki> #!/bin/bash export NSPR_LOG_MODULES=IMAP:4,SMTP:4 export NSPR_LOG_FILE=~/Des...

[[Category:Linux]]
[[Category:Desktop]]

Use this when thunderbird goes mad and you want to know why.
<nowiki>
#!/bin/bash
export NSPR_LOG_MODULES=IMAP:4,SMTP:4
export NSPR_LOG_FILE=~/Desktop/tbird.log
/Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin
</nowiki>

Rpm-extract.sh

2011-07-16T12:33:44Z

Summary: New page: [[Category:linux]] [[Category:bash]] [[Category:scripts]] Because I am too dumb to remember the correct switches for cpio <nowiki> #!/bin/sh rpm2cpio $1 | cpio -idmv </nowiki>

[[Category:linux]]
[[Category:bash]]
[[Category:scripts]]

Because I am too dumb to remember the correct switches for cpio

<nowiki>
#!/bin/sh

rpm2cpio $1 | cpio -idmv
</nowiki>

Eximstats.sh

2011-07-16T12:31:25Z

Summary:

[[Category:linux]]
[[Category:scripts]]
[[Category:bash]]
[[Category:perl]]

Call this every night to generate a text based report for your exim server. It uses a little perl script to split the main log in to one log per complete day (from 00:00 hrs to 23:59 hrs). I use this to collect stats from multiple servers, then gather them centrally and convert them in to html reports.

exim-main-log-split.pl:
<nowiki>
#!/usr/bin/env perl

# Take the exim log file specified as the first argument when calling this script,
# then split all entries in to seperate files for each day.
# also pipe logs in

use strict;
use warnings;
use File::Basename;
use POSIX;

my $logtype = "main.log";

my $previousDate = '';
my $currentDate = '';
my $logfile;
while (<>) {
my $line = $_;
$currentDate = '';
if ($line =~ /^(\d{4}-\d{2}-\d{2}) .*/) {
$currentDate = $1;
}
if ($currentDate ne $previousDate || !$logfile) {
if ($logfile) {
close($logfile);
}
open $logfile, ">>", "$logtype.$currentDate"
or die "Could not open $logtype.$currentDate: $!";
}
print $logfile $line;
}

</nowiki>

eximstats.sh:
<nowiki>
#!/bin/bash

# Generate daily reports from the exim-relay logs.
# scp the most recent report to www.exmple.com
# Remove existing reports older than seven days.

LOG_FILE=/var/log/exim/main.log.1.bz2
LOG_SPLITTER=/usr/local/bin/exim-main-log-split.pl

REPORT_DIR=/var/lib/exim/eximstats

SCP_TARGET=eximstat@www.example.com:/var/lib/eximstat/$(hostname)

DATE=$(date +%F)
REPORT_DATE=$(date --date '1 day ago' +%F)

EXIMSTATS=/usr/bin/eximstats

ARGS="$(cat <<EOT
-pattern Spam '/ rejected after DATA: Spam messages are not welcome here./' \
-pattern Viruses '/ rejected after DATA: This message contains a virus/' \
EOT
)"

if [ ! -f $LOG_FILE ]; then
echo Log file $LOG_FILE does not exist
exit 1
fi

cd $REPORT_DIR || exit 1

bzcat $LOG_FILE | $LOG_SPLITTER

$EXIMSTATS $ARGS -txt=$REPORT_DIR/$REPORT_DATE.txt $REPORT_DIR/main.log.$REPORT_DATE || exit

rm $REPORT_DIR/main.log.$REPORT_DATE

scp $REPORT_DATE.txt $SCP_TARGET/

find $REPORT_DIR -name '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].txt' -ctime +14 | xargs rm
</nowiki>

eximstats-html-reports.sh (on www.example.com):
<nowiki>
#!/bin/bash
#
# A script to generate eximstats reports from the exim main logs
# The logs are processed in to daily reports on the mailservers,
# before being copied over to this server each night
#
# This script depends on the mail servers' log rotation jobs,
# and their cron jobs that run aftwards to generate the daily
# reports
#
# The script creates daily, weekly and monthly reports. It also keeps
# the last copy of any given day, week or month to allow comaprisons.
#
# The script needs to run each day, it won't catch up if any days are
# missed. If that happens, weekly and monthly reports will still be
# created but won't be accurate.

################
# Configuration

EXIMSTATS=/usr/local/bin/eximstats
#EXIMSTATS=/opt/local/sbin/eximstats

MAIL_SERVERS="foo bar baz"

TEXT_REPORTS="/var/lib/eximstat"

CURRENT_REPORTS_DIR=/var/www/html/eximstats
PREVIOUS_REPORTS_DIR="${CURRENT_REPORTS_DIR}/previous"

NICENESS=39

DAILY_REPORT_LABEL=daily
WEEKLY_REPORT_LABEL=weekly
MONTHLY_REPORT_LABEL=monthly

########################
# Options for eximstats
ARGS="$(cat <<EOT
-pattern Spam '/ rejected after DATA: Spam messages are not welcome here./' \
-pattern Viruses '/ rejected after DATA: This message contains a virus/' \
-ne \
-nr \
-charts \
-merge
)"

############
# functions

# date adjusts to the given number of days in the past and
# formatted using the provided formatting string
# usage: pastDate <minus no. days> <format>
pastDate()
{
if [ ! $1 ]
then
exit 1
fi
if [ ! $2 ]
then
exit 1
fi
if [ `uname` = "Darwin" ]
then
echo $(date -v -"$1"d +"$2")
fi
if [ `uname` = "Linux" ]
then
echo $(date --date="$1 day ago" +"$2")
fi
}

##############################
# Directory Checks / Creation

# Check required directories are writable - try to create them if missing or
# exit.

if [ ! -d "${CURRENT_REPORTS_DIR}" -o ! -w "${CURRENT_REPORTS_DIR}" ]
then
echo "The reports directory, "
echo "'${CURRENT_REPORTS_DIR}',"
echo "does not exist or is not writable"
exit 1
fi

DAILY_DIR="${CURRENT_REPORTS_DIR}/${DAILY_REPORT_LABEL}"
test -e "${DAILY_DIR}" || test -d "${DAILY_DIR}" || mkdir "${DAILY_DIR}"
if [ ! -d "${DAILY_DIR}" -o ! -w "${DAILY_DIR}" ]
then
echo "The daily reports directory, "
echo "'${DAILY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

WEEKLY_DIR="${CURRENT_REPORTS_DIR}/${WEEKLY_REPORT_LABEL}"
test -e "${WEEKLY_DIR}" || test -d "${WEEKLY_DIR}" || mkdir "${WEEKLY_DIR}"
if [ ! -d "${WEEKLY_DIR}" -o ! -w "${WEEKLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${WEEKLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

MONTHLY_DIR="${CURRENT_REPORTS_DIR}/${MONTHLY_REPORT_LABEL}"
test -e "${MONTHLY_DIR}" || test -d "${MONTHLY_DIR}" || mkdir "${MONTHLY_DIR}"
if [ ! -d "${MONTHLY_DIR}" -o ! -w "${MONTHLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${MONTHLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

if [ ! -d "${PREVIOUS_REPORTS_DIR}" -o ! -w "${PREVIOUS_REPORTS_DIR}" ]
then
echo "The previous reports directory, "
echo "'${PREVIOUS_REPORTS_DIR}',"
echo "does not exist or is not writable"
exit 1
fi

PREV_DAILY_DIR="${PREVIOUS_REPORTS_DIR}/${DAILY_REPORT_LABEL}"
test -e "${PREV_DAILY_DIR}" || test -d "${PREV_DAILY_DIR}" || mkdir "${PREV_DAILY_DIR}"
if [ ! -d "${PREV_DAILY_DIR}" -o ! -w "${PREV_DAILY_DIR}" ]
then
echo "The daily reports directory, "
echo "'${PREV_DAILY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

PREV_WEEKLY_DIR="${PREVIOUS_REPORTS_DIR}/${WEEKLY_REPORT_LABEL}"
test -e "${PREV_WEEKLY_DIR}" || test -d "${PREV_WEEKLY_DIR}" || mkdir "${PREV_WEEKLY_DIR}"
if [ ! -d "${PREV_WEEKLY_DIR}" -o ! -w "${PREV_WEEKLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${PREV_WEEKLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

PREV_MONTHLY_DIR="${PREVIOUS_REPORTS_DIR}/${MONTHLY_REPORT_LABEL}"
test -e "${PREV_MONTHLY_DIR}" || test -d "${PREV_MONTHLY_DIR}" || mkdir "${PREV_MONTHLY_DIR}"
if [ ! -d "${PREV_MONTHLY_DIR}" -o ! -w "${PREV_MONTHLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${PREV_MONTHLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

########
# Dates

# these dates relate to the log files, i.e. yesterdays logs and activity.
DAY_OF_WEEK=$(pastDate 1 %w)
WEEK=$(pastDate 1 %U)
MONTH=$(pastDate 1 %m)
YEAR=$(pastDate 1 %Y)
DATE=$(pastDate 1 %F)
PREV_DATE="$(($YEAR - 1))${DATE:(-6)}"
PREV_PREV_DATE="$(($YEAR - 2))${DATE:(-6)}"

# if today is the 1st of the month, then yesterday was the end of the month
MONTH_ENDED=$(
test $(date +%d) = 01 && echo true
)

###########################
# Create Daily HTML Report

if [ -d "${DAILY_DIR}/${PREV_DATE}" ]
then
echo "Moving previous Daily HTML report"
[ -d "${PREV_DAILY_DIR}/${PREV_PREV_DATE}" ] && rm -rf "${PREV_DAILY_DIR}/${PREV_PREV_DATE}"
mv "${DAILY_DIR}/${PREV_DATE}" "${PREV_DAILY_DIR}/${PREV_DATE}"
fi

echo "Creating Daily HTML Report"

mkdir -p "${DAILY_DIR}/${DATE}"

REPORT_LIST=""
for server in $MAIL_SERVERS
do
if [ -f "$TEXT_REPORTS/$server/$DATE.txt" ]
then
REPORT_LIST="$REPORT_LIST $TEXT_REPORTS/$server/$DATE.txt"
else
echo "WARNING: Missing report $TEXT_REPORTS/$server/$DATE.txt"
fi
done

if [ "$REPORT_LIST" ]
then
nice -n ${NICENESS} "${EXIMSTATS}" \
${ARGS} \
-chartdir "${DAILY_DIR}/${DATE}" \
-html="${DAILY_DIR}/${DATE}/index.html" \
$REPORT_LIST
else
echo "WARNING: Unable to create merged relays report for $DATE. No reports found."
fi
echo

#######################
# Create Weekly Report

if [ $DAY_OF_WEEK -eq 6 ]
then

if [ -d "${WEEKLY_DIR}/${WEEK}" ]
then
echo "Moving previous Weekly HTML report"
if [ -d "${PREV_WEEKLY_DIR}/${WEEK}" ]
then
rm -rf "${PREV_WEEKLY_DIR}/${WEEK}"
fi
mv "${WEEKLY_DIR}/${WEEK}" "${PREV_WEEKLY_DIR}/${WEEK}"
fi

echo "Creating Weekly HTML Report"

REPORT_LIST=""
DATE_ADJUSTMENT=$(( $DAY_OF_WEEK + 1 ))
while [ $DATE_ADJUSTMENT -ge 1 ]
do

REPORT_DATE=$(pastDate $DATE_ADJUSTMENT %F)

for server in $MAIL_SERVERS
do
if [ -f "${TEXT_REPORTS}/$server/${REPORT_DATE}.txt" ]
then
REPORT_LIST="$REPORT_LIST $TEXT_REPORTS/$server/$REPORT_DATE.txt"
else
echo "WARNING: Missing report $TEXT_REPORTS/$server/$DATE.txt"
continue
fi
done

DATE_ADJUSTMENT=$(( $DATE_ADJUSTMENT - 1 ))

done

if [ "$REPORT_LIST" ]
then
mkdir -p "${WEEKLY_DIR}/${WEEK}"
nice -n ${NICENESS} "${EXIMSTATS}" \
${ARGS} \
-chartdir "${WEEKLY_DIR}/${WEEK}" \
-html="${WEEKLY_DIR}/${WEEK}/index.html" \
$REPORT_LIST
else
echo "WARNING: Unable to create merged relays report for week $WEEK. No reports found."
fi

echo
fi

########################
# Create Monthly Report

if [ $MONTH_ENDED ]
then

if [ -d "${MONTHLY_DIR}/${MONTH}" ]
then
echo "Moving previous Monthly HTML report"
if [ -d "${PREV_MONTHLY_DIR}/${MONTH}" ]
then
rm -rf "${PREV_MONTHLY_DIR}/${MONTH}"
fi
mv "${MONTHLY_DIR}/${MONTH}" "${PREV_MONTHLY_DIR}/${MONTH}"
fi

echo "Creating Monthly HTML Report"

REPORT_LIST=""
for server in $MAIL_SERVERS
do
for REPORT in $(ls $TEXT_REPORTS/$server/${YEAR}-${MONTH}-*.txt)
do
REPORT_LIST="$REPORT_LIST $REPORT"

done
done

if [ "$REPORT_LIST" ]
then

mkdir -p "${MONTHLY_DIR}/${MONTH}"

nice -n ${NICENESS} "${EXIMSTATS}" \
${ARGS} \
-chartdir "${MONTHLY_DIR}/${MONTH}" \
-html="${MONTHLY_DIR}/${MONTH}/index.html" \
$REPORT_LIST

else

echo "WARNING: Unable to create merged relays report for month $MONTH. No reports found."

fi

echo

fi

#########################
# Delete old cache files

if [ $MONTH_ENDED ]
then

echo "Removing old text report files"

cd "${TEXT_REPORTS}"

if [ ${MONTH##0} -gt 1 ]
then
RM_MONTH=$(printf "%0#2d" $((${MONTH##0} - 1)) )
RM_YEAR=$YEAR
else
RM_MONTH=12
RM_YEAR=$(($YEAR - 1))
fi

for server in $MAIL_SERVERS
do
for REPORT_FILE in $(ls ${TEXT_REPORTS}/${server}/${RM_YEAR}-${RM_MONTH}-*.txt)
do
rm "${REPORT_FILE}"
done
done

fi
</nowiki>

Exim-linesforclient.sh

2011-07-16T12:26:09Z

Summary: New page: [[Category:linux]] [[Category:scripts]] [[Category:bash]] Pass this an ip address for a client sending mail to your exim submission service <nowiki> #!/bin/bash cat /var/log/exim/main....

[[Category:linux]]
[[Category:scripts]]
[[Category:bash]]

Pass this an ip address for a client sending mail to your exim submission service

<nowiki>
#!/bin/bash

cat /var/log/exim/main.[0-9] | grep " \[$(echo $1 | sed -e 's/\./\\./g')\] ";
</nowiki>

Exim-top-senders.sh

2011-07-16T12:24:08Z

Summary: New page: [[Category:linux]] [[Category:scripts]] [[Category:bash]] <nowiki> #!/bin/bash renice 10 $$ # Some nasty byte escape sequence showed up in an email address # \222 - possibly �� (La...

[[Category:linux]]
[[Category:scripts]]
[[Category:bash]]

<nowiki>
#!/bin/bash

renice 10 $$

# Some nasty byte escape sequence showed up in an email address
# \222 - possibly �� (Latin Capital Letter Thorn)
export LC_ALL='C'

cat /var/log/exim/main.[0-9] \
| egrep "<[=\-] .* \[[^ ]+\] P=[a-z]*smtp.* " \
| sed -e 's/^.* \[$\([[:digit:]]\{1,3\}\.$\{3\}[[:digit:]]\{1,3\}\)\] .*$/\1/' \
| sort \
| uniq -c \
| sort -n \
| tail -20
</nowiki>

Exim-incoming.sh

2011-07-16T12:22:23Z

Summary: New page: [[Category:linux]] [[Category:bash]] [[Category:scripts]] $ exim-incoming.sh foo@example.com <nowiki> #!/bin/bash # return all details logged to the exim main log file for all messages...

[[Category:linux]]
[[Category:bash]]
[[Category:scripts]]

$ exim-incoming.sh foo@example.com

<nowiki>
#!/bin/bash

# return all details logged to the exim main log file for all messages delivered to a particular email address

grep -E "$(
grep "=> [^ ]*@imap.example.com <$1" /var/log/exim/main.log \
| cut -d\ -f 3 \
| sort \
| uniq \
| tr "\n" "|" \
| sed -e 's/|$//'
)" /var/log/exim-relay/main.log

</nowiki>

Cacti ss get snmp

2011-07-16T12:06:08Z

Summary:

[[Category:linux]]
[[Category:scripts]]
[[Category:PHP]]

This is a script that will collect arbitary snmp data for cacti's script host. You specificy which oids to query in named sections of an ini file, and pass the name of the particular dataset you want in as an argument when calling the script.

This means you can start collecting and graphing data for new things on your network with out necessarily needing to write a new script to get the data.

This script is based on the work of the Cacti developers and contributors to the cacti project. It's licensed under the same GNU General Public License.

<nowiki>
<?php

// Cacti Script Server boiler-plate:

/* do NOT run this script through a web browser */
if (!isset($_SERVER["argv"][0]) || isset($_SERVER['REQUEST_METHOD']) || isset($_SERVER['REMOTE_ADDR'])) {
die("<br><strong>This script is only meant to run at the command line.</strong>");
}

$no_http_headers = true;

/* display No errors */
ini_set('display_errors', 1);

if (isset($config)) {
include_once(dirname(__FILE__) . "/../lib/snmp.php");
}

if (!isset($called_by_script_server)) {
include_once(dirname(__FILE__) . "/../include/global.php");
include_once(dirname(__FILE__) . "/../lib/snmp.php");

array_shift($_SERVER["argv"]);
$functionName = $_SERVER["argv"][0];
array_shift($_SERVER["argv"]);

print call_user_func_array($functionName, $_SERVER["argv"]);
}

/*
* SETTINGS
*/
define('SS_GET_SNMP_INI', '/etc/cacti/ss_get_snmp.ini');

// http://www.php.net/manual/en/function.sys-get-temp-dir.php#85261
if (!function_exists("sys_get_temp_dir")) {
function sys_get_temp_dir() {
if( $temp=getenv('TMP') ) return $temp;
if( $temp=getenv('TEMP') ) return $temp;
if( $temp=getenv('TMPDIR') ) return $temp;
$temp=tempnam(__FILE__,'');
if (file_exists($temp)) {
unlink($temp);
return dirname($temp);
}
return null;
}
}

function ss_get_snmp_config($query) {
static $config;
if (!$config) {
$config = parse_ini_file(SS_GET_SNMP_INI, true);
}
return (is_array($config) && isset($config[$query])) ? $config[$query] : null;
}

/**
* Query a group of snmp oids and specified in the ini configuration file for this script
*
* @param string $query a query name specified in the ss_get_snmp.ini configuration file
* @param string $hostname the hostname or ipaddress of the snmp server
* @param string $snmpauth colon delimited list of snmp connection parameters
* @return null
**/
function ss_get_snmp($query, $hostname, $snmpauth)
{
$results = null;
$walk = false;
$oids = ss_get_snmp_config($query);
if (!$oids) {
trigger_error("Not such query: $query", E_USER_WARNING);
return null;
}
if (isset($oids['__walk']) && $oids['__walk']) {
$walk = true;
unset($oids['__walk']);
}

$polling_interval = read_config_option("poller_interval");

$temp_dir = sys_get_temp_dir() . "/" . basename(__FILE__);
$temp_dir = preg_replace('/\.php$/', '', $temp_dir);
if (!is_dir($temp_dir)) {
mkdir($temp_dir);
}
if (!is_dir($temp_dir."/$query")) {
mkdir($temp_dir."/$query");
}
$temp_dir = "$temp_dir/$query";
if (is_file("$temp_dir/$hostname")) {
$cache_stats = stat("$temp_dir/$hostname");
if ( (($cache_stats['mtime'] + $polling_interval) * 0.75) > time() ) {
if ($cache = fopen("$temp_dir/$hostname", 'r')) {
while (!feof($cache)) {
$results .= fgets($cache);
}
}
}
}
if (strlen($results)) {
return $results;
}

$results = ss_get_snmp_query($oids, $hostname, $snmpauth, $walk);

if (is_file("$temp_dir/$hostname") && !is_writable("$temp_dir/$hostname")) {
unlink("$temp_dir/$hostname");
}
file_put_contents("$temp_dir/$hostname", $results);

return $results;
//return (strlen($results)) ? true : false;
}
/**
* generic function to fetch a list of oids and output the results
* in cacti scripts servers fieldname:value format
*
* @param array $oids an assiative array of oids to query. The array keys are used as the field names for cacti in the results.
* @param string $hostname the hostname or ipaddress of the snmp server
* @param string $snmpauth colon delimited list of snmp connection parameters
* @param bool $walk if true, use snmp walk to fetch data, otherwise use snmp get
*
* @return string
**/
function ss_get_snmp_query($query, $hostname, $snmpauth, $walk=false)
{
$snmp = explode(":", $snmpauth);
$snmp_version = $snmp[0];
$snmp_port = $snmp[1];
$snmp_timeout = $snmp[2];
$ping_retries = (strlen($snmp[3]) ? $snmp[3] : 0);
$max_oids = (strlen($snmp[4]) ? $snmp[4] : 0);

$snmp_auth_username = "";
$snmp_auth_password = "";
$snmp_auth_protocol = "";
$snmp_priv_passphrase = "";
$snmp_priv_protocol = "";
$snmp_context = "";
$snmp_community = "";

if ($snmp_version == 3)
{
$snmp_auth_username = isset($snmp[6]) ? $snmp[6] : "";
$snmp_auth_password = isset($snmp[7]) ? $snmp[7] : "";
$snmp_auth_protocol = isset($snmp[8]) ? $snmp[8] : "";
$snmp_priv_passphrase = isset($snmp[9]) ? $snmp[9] : "";
$snmp_priv_protocol = isset($snmp[10]) ? $snmp[10] : "";
$snmp_context = isset($snmp[11]) ? $snmp[11] : "";
}
else
{
$snmp_community = isset($snmp[5]) ? $snmp[5] : "";
}

$output = '';
$cacti_snmp_func = $walk ? "cacti_snmp_walk" : "cacti_snmp_get";
foreach ($query as $name => $oid)
{
$results = $cacti_snmp_func($hostname, $snmp_community,
$oid, $snmp_version, $snmp_auth_username,
$snmp_auth_password, $snmp_auth_protocol,
$snmp_priv_passphrase, $snmp_priv_protocol, $snmp_context,
$snmp_port, $snmp_timeout, $ping_retries, $max_oids,
SNMP_POLLER
);
if ($walk)
{
for ($i = 0; $i < sizeof($results); ++$i)
{
$output .= $name . ($walk ? ($i + 1) : null) . ":" .
$results[$i]['value'] . " ";
}
}
else
{
$output .= "$name:$results ";
}
}
return trim($output);
}
</nowiki>

Squid-Stats.py

2011-07-16T11:59:54Z

Summary: New page: [[Category:linux]] [[Category:python]] [[Category:scripts]] This script will use a local squidclient binary to get stats from squid's cache manager and output them in a way cacti can use ...

[[Category:linux]]
[[Category:python]]
[[Category:scripts]]

This script will use a local squidclient binary to get stats from squid's cache manager and output them in a way cacti can use them. I haven't included the cacti data and graph templates. I might do that at some point.

<nowiki>
#!/usr/bin/env python

squidclient = "/usr/local/bin/squidclient"
tmpdir = "/tmp/squid_stats"

import cPickle, optparse, os, subprocess, sys, time

def cache_stats(stats, hostname, tmpdir):
try:
os.makedirs(tmpdir, 0755)
except:
pass
try:
cache = open(tmpdir + "/" + hostname, 'w')
cPickle.dump(stats, cache)
except:
pass

def fetch_cached_stats(hostname, pollerInterval, tmpdir):
mtime = 0
cacheFile = tmpdir + '/' + hostname
cacheLifetime = pollerInterval * 0.75
now = time.time()
try:
mtime = os.stat(cacheFile).st_mtime
if 0 < mtime > (now - cacheLifetime):
cache = open(cacheFile, 'r')
stats = cPickle.load(cache)
cache.close()
return stats
except:
pass

def fetch_stats(argsList):
squid5min = subprocess.Popen(argsList + ['mgr:5min'], stdout=subprocess.PIPE).communicate()[0]
if len(squid5min):
return squid5min
else:
return False

def filter_stats(stats, query):
queries = {
"requests": [
"aborted_requests",
"client_http.requests",
"client_http.hits",
"client_http.errors",
"server.all.requests",
"server.all.errors",
"server.http.requests",
"server.http.errors",
"server.ftp.requests",
"server.ftp.errors",
"server.other.requests",
"server.other.errors",
],
"transfer": [
"server.all.kbytes_in",
"server.all.kbytes_out",
"server.http.kbytes_in",
"server.http.kbytes_out",
"server.ftp.kbytes_in",
"server.ftp.kbytes_out",
"server.other.kbytes_in",
"server.other.kbytes_out",
"client_http.kbytes_in",
"client_http.kbytes_out",
],
"svctime": [
"client_http.all_median_svc_time",
"client_http.miss_median_svc_time",
"client_http.nm_median_svc_time",
"client_http.nh_median_svc_time",
"client_http.hit_median_svc_time",
"dns.median_svc_time",
],
"syscallsdisk": [
"syscalls.disk.opens",
"syscalls.disk.closes",
"syscalls.disk.reads",
"syscalls.disk.writes",
"syscalls.disk.seeks",
"syscalls.disk.unlinks",
],
"syscallssocket": [
"syscalls.sock.accepts",
"syscalls.sock.sockets",
"syscalls.sock.connects",
"syscalls.sock.binds",
"syscalls.sock.closes",
"syscalls.sock.reads",
"syscalls.sock.writes",
"syscalls.sock.recvfroms",
"syscalls.sock.sendtos",
],
"swap": [
"swap.outs",
"swap.ins",
"swap.files_cleaned",
],
"unlink": [
"unlink.requests",
],
"pagefaults": [
"page_faults",
],
"selectloops": [
"select_loops",
"select_fds",
"average_select_fd_period",
"median_select_fds",
],
}
filteredStats = {}
for item in queries[query]:
if stats.has_key(item):
filteredStats[item] = stats[item]
return filteredStats

def output_stats(stats):
numStats = len(stats)
for stat in stats:
numStats -= 1
output = "%s:" % (stat.replace('.', '_'), )
if stats[stat] == "U":
output = output + "U"
else:
output = output + ("%.2f" % (float(stats[stat]), ))
sys.stdout.write(output)
if numStats > 0:
sys.stdout.write(" ")

def parse_stats(commandOutput, stats):
for line in commandOutput.splitlines():
try:
name, value = line.split(" = ")
except:
continue
if stats.has_key(name):
length = value.find('/')
if length < 0:
length = value.find(' ')
if length < 0:
length = value.find('%')
if length < 0:
length = None
stats[name] = value[0:length]
return stats

def parse_options(parser):
(options, args) = parser.parse_args()
argsList = [squidclient]
if options.hostname:
argsList += ['-h', options.hostname]
if options.port:
argsList += ['-p', options.port]
if options.bind:
argsList += ['-l', options.bind]
if options.timeout:
argsList += ['-T', options.timeout]
if options.username:
argsList += ['-u', options.username]
if options.password:
argsList += ['-w', options.password]
return argsList, options

parser = optparse.OptionParser()
parser.add_option("-H", dest="hostname",
help="Retrieve URL from cache on hostname. Default is localhost.",
default="localhost")
parser.add_option("-l", dest="bind",
help="Specify a local IP address to bind to. Default is none.")
parser.add_option("-p", dest="port",
help="Port number of cache. Default is 3128.",
default="3128")
parser.add_option("-T", dest="timeout",
help="Timeout value (seconds) for read/write operations.",
default="7")
parser.add_option("-u", dest="username",
help="Proxy authentication username")
parser.add_option("-w", dest="password",
help="Proxy authentication password")
parser.add_option("-P", dest="pollerInterval",
help="The polling interval used by Cacti",
default=300, type="int")
parser.add_option("-q", dest="query",
help="The query to run")

argsList, options = parse_options(parser)

stats = {
"client_http.requests": 0,
"client_http.hits": 0,
"client_http.errors": 0,
"server.all.requests": 0,
"server.all.errors": 0,
"server.http.requests": 0,
"server.http.errors": 0,
"server.ftp.requests": 0,
"server.ftp.errors": 0,
"server.other.requests": 0,
"server.other.errors": 0,
"server.all.kbytes_in": 0,
"server.all.kbytes_out": 0,
"server.http.kbytes_in": 0,
"server.http.kbytes_out": 0,
"server.ftp.kbytes_in": 0,
"server.ftp.kbytes_out": 0,
"server.other.kbytes_in": 0,
"server.other.kbytes_out": 0,
"client_http.kbytes_in": 0,
"client_http.kbytes_out": 0,
"client_http.all_median_svc_time": 0,
"client_http.miss_median_svc_time": 0,
"client_http.nm_median_svc_time": 0,
"client_http.nh_median_svc_time": 0,
"client_http.hit_median_svc_time": 0,
"dns.median_svc_time": 0,
"unlink.requests": 0,
"page_faults": 0,
"select_loops": 0,
"select_fds": 0,
"average_select_fd_period": 0,
"median_select_fds": 0,
"swap.outs": 0,
"swap.ins": 0,
"swap.files_cleaned": 0,
"aborted_requests": 0,
"syscalls.disk.opens": 0,
"syscalls.disk.closes": 0,
"syscalls.disk.reads": 0,
"syscalls.disk.writes": 0,
"syscalls.disk.seeks": 0,
"syscalls.disk.unlinks": 0,
"syscalls.sock.accepts": 0,
"syscalls.sock.sockets": 0,
"syscalls.sock.connects": 0,
"syscalls.sock.binds": 0,
"syscalls.sock.closes": 0,
"syscalls.sock.reads": 0,
"syscalls.sock.writes": 0,
"syscalls.sock.recvfroms": 0,
"syscalls.sock.sendtos": 0,
}

cachedStats = fetch_cached_stats(options.hostname, options.pollerInterval, tmpdir)
if cachedStats:
output_stats(filter_stats(cachedStats, options.query))
else:
squid5min = fetch_stats(argsList)
if squid5min == False:
for stat in stats:
stats[stat] = "U"
else:
stats = parse_stats(squid5min, stats)
cache_stats(stats, options.hostname, tmpdir)
output_stats(filter_stats(stats, options.query))

</nowiki>

Squid-Stats.sh

2011-07-16T11:56:32Z

Summary:

[[Category:bash]]
[[Category:linux]]
[[Category:scripts]]

This was a poor attempt to collect squid stats from the cache manager and get them in to cacti. I wrote some python to do this and it's much better, see [[Squid-Stats.py]]. I can't even remember if I managed to finished this script. It's here to remind me that some things shouldn't be attempted in bash because it's a waste of time.

<nowiki>
#!/bin/bash

STATS_DIR=/var/lib/squid-stats

CLIENT=/opt/local/bin/squidclient

HOST=wwwcache.qmul.ac.uk
PORT=3128
URL_FD=mgr:filedescriptors
URL_INFO=mgr:info
URL_CLIENTS=mgr:client_list
URL_PCONN=mgr:pconn
URL_AVG60=mgr:60min

DATE=$(date +%F)
mkdir "${STATS_DIR}/${DATE}"
STATS_FILE="${STATS_DIR}/${DATE}/summary"

function fetch_stats() {
URL="$1"
"$CLIENT" -h "$HOST" -p "$PORT" "$URL" > "${STATS_DIR}/${DATE}/${URL#mgr:}"
}
function add_s() {
NAME="$1"
VALUE="$2"
echo "$NAME $VALUE" >> "$STATS_FILE"
}

fetch_stats "$URL_FD"
fetch_stats "$URL_INFO"
fetch_stats "$URL_CLIENTS"
fetch_stats "$URL_PCONN"
fetch_stats "$URL_AVG60"

FD_INFO=$(grep -A 7 "File descriptor usage for squid:" "${STATS_DIR}/${DATE}/${URL_INFO#mgr:}")
add_s store_fds $(echo "$FD_INFO" | grep "Store Disk files open:" | cut -d: -f 2 | sed -e 's/ //g')
add_s reserved_fds $(echo "$FD_INFO" | grep "Reserved number of file descriptors:" | cut -d: -f 2 | sed -e 's/ //g')
add_s available_fds $(echo "$FD_INFO" | grep "Available number of file descriptors:" | cut -d: -f 2 | sed -e 's/ //g')
add_s used_fds $(echo "$FD_INFO" | grep "Number of file desc currently in use:" | cut -d: -f 2 | sed -e 's/ //g')

add_s socket_fds $(sed -e 's/ */ /g' -e 's/^ *//' < "${STATS_DIR}/${DATE}/${URL_FD#mgr:}" | egrep "^[[:digit:]]+ Socket" | wc -l)

C_CONN_TOTAL=0
C_TOTAL=0
C_CONNECTIONS=$(grep "Currently established connections" "${STATS_DIR}/${DATE}/${URL_CLIENTS#mgr:}" | cut -d: -f 2)
for C_CONN in $C_CONNECTIONS
do
if [ $C_CONN -gt 0 ]
then
C_TOTAL=$(( $C_TOTAL + 1 ))
C_CONN_TOTAL=$(( $C_CONN_TOTAL + $C_CONN ))
fi
done
add_s clients_connected $C_TOTAL
add_s client_fds $C_CONN_TOTAL

#��The number of idle persistent connections per origin server
< "${STATS_DIR}/${DATE}/${URL_FD#mgr:}" \
sed -e 1,15d -e 's/^ *//' -e 's/ */ /g' | \
cut -d\ -f 2,6- | \
grep "^Socket.*idle connection$" | \
cut -d\ -f 3 | \
sort | \
uniq -c | \
sort -nr | \
sed -e 's/^ *//' \
> "${STATS_DIR}/${DATE}/idle_origin_servers"
add_s idle_origin_servers $(wc -l "${STATS_DIR}/${DATE}/idle_origin_servers")
add_s idle_origin_sockets $(< "${STATS_DIR}/${DATE}/idle_origin_servers" cut -d\ -f 1 | tr "\n" "+" | sed -e 's/\+$//' | bc)

# The number of idle persistnent connections per client
< "${STATS_DIR}/${DATE}/${URL_FD#mgr:}" \
sed -e 1,15d -e 's/^ *//' -e 's/ */ /g' | \
cut -d\ -f 2,6- | \
grep "^Socket.*Waiting for next request$" | \
cut -d\ -f 2 | \
cut -d. -f -4 | \
sort | \
uniq -c | \
sort -nr | \
sed -e 's/^ *//' \
> "${STATS_DIR}/${DATE}/idle_clients"
add_s idle_clients $(wc -l "${STATS_DIR}/${DATE}/idle_clients")
add_s idle_client_sockets $(< "${STATS_DIR}/${DATE}/idle_clients" cut -d\ -f 1 | tr "\n" "+" | sed -e 's/\+$//' | bc)

</nowiki>

SpamAssassin Updater

2011-07-16T11:52:11Z

Summary: New page: [[Category:linux]] [[Category:bash]] [[Category:scripts]] This script will check for new spamassassin rule sets, download them, test them, compile them and reload spamd plus keep backups ...

[[Category:linux]]
[[Category:bash]]
[[Category:scripts]]

This script will check for new spamassassin rule sets, download them, test them, compile them and reload spamd plus keep backups of the previous verious. It make me feel a little bit better about spamassassin pulling in new rule sets automatically. Doing that is a good thing but sometimes bad rules are pushed out.

<nowiki>
#!/bin/bash

# This script will automatically update spamassassin rules and compile them
# Backups are kept for at least 28 days

###########
# Settings

SA=/usr/bin/spamassassin
SA_UPDATE=/usr/bin/sa-update
SA_COMPILE=/usr/bin/sa-compile
SA_VAR=/var/lib/spamassassin
SPAMD_PID=/var/run/spamd.pid
BACKUP_OWNER="sa-update"

COMPILED_DIR="${SA_VAR}/compiled"
BACKUP_DIR="${SA_VAR}/backups"

############
# Functions

# date adjusts to the given number of days in the past and
# formatted using the provided formatting string
# usage: pastDate <minus no. days> <format>
pastDate()
{
if [ ! $1 ]
then
exit 1
fi
if [ ! $2 ]
then
exit 1
fi
if [ `uname` = "Darwin" ]
then
echo $(date -v -"$1"d +"$2")
fi
if [ `uname` = "Linux" ]
then
echo $(date --date="$1 day ago" +"$2")
fi
}

##############
# For logging

echo "$(date +"%F %X") spamassassin updater"

#########################################################
# SpamAssassin Version and corresponding rules directory

# spamassassin uses a different directory name to store rules, based on
# it's version number. The version number is converted to a "filesytem
# friendly" name like 3.002005 or 3.003000 for 3.2.5 and 3.3.0.
SA_VERSION=$("${SA}" -V | grep SpamAssassin | sed -e 's/SpamAssassin version //')
SA_MAJOR=$(echo "${SA_VERSION}" | cut -d. -f 1)
SA_MINOR=$(echo "${SA_VERSION}" | cut -d. -f 2)
SA_PATCH=$(echo "${SA_VERSION}" | cut -d. -f 3)
RULE_DIR="${SA_VAR}/$(printf "%d.%0#3d%0#3d" "${SA_MAJOR}" "${SA_MINOR}" "${SA_PATCH}")"

###############
# Basic checks

if [ ! -f "$SPAMD_PID" ]
then
echo "spamd pid file $SPAMD_PID does not exist"
exit 1
fi

if [ ! "$RULE_DIR" ]
then
echo "could not determine directory for spamassassin rules"
exit 1
fi
if [ ! -d "$RULE_DIR" ]
then
mkdir "$RULE_DIR"
fi
if [ ! -d "$RULE_DIR" ]
then
echo "cannot create rule directory $RULE_DIR"
exit 1
fi

if [ "$(egrep "^${BACKUP_OWNER}:" /etc/passwd | wc -l)" -ne 1 ]
then
echo "backup owner \"${BACKUP_OWNER}\" does not exist"
exit 1
fi

if [ ! -d "$BACKUP_DIR" ]
then
mkdir "$BACKUP_DIR"
fi
if [ ! -d "$BACKUP_DIR" ]
then
echo "cannot create backup rules directory $BACKUP_DIR"
exit 1
fi
if [ ! -w "$BACKUP_DIR" ]
then
echo "backup rules directory $BACKUP_DIR is not writable"
exit 1
fi

############
# Spamd pid

SPAMD_PID=$(cat "$SPAMD_PID")
if [ ! "$(ps -p "$SPAMD_PID" --no-headers)" ]
then
echo "spamd is not running"
exit 1
fi

###############
# Backup dates

TODAY=$(date +%F)
FOUR_WEEKS_AGO=$(pastDate 28 %F)

###################################
# Check for availabilty of updates

"$SA_UPDATE" --gpg --checkonly
SA_UPDATE_EXIT_CODE=$?
if [ $SA_UPDATE_EXIT_CODE -eq 1 ]
then
echo "No updates available"
exit 0
elif [ $SA_UPDATE_EXIT_CODE -gt 1 ]
then
echo "sa-update returned error $SA_UPDATE_EXIT_CODE"
exit $SA_UPDATE_EXIT_CODE
fi

########################
# Backup existing rules

mkdir "$BACKUP_DIR/$TODAY"
cp -rp "$RULE_DIR" "$BACKUP_DIR/$TODAY/"
cp -rp "$COMPILED_DIR" "$BACKUP_DIR/$TODAY/"
chown -R $BACKUP_OWNER "$BACKUP_DIR/$TODAY"

###############
# Update rules

"$SA_UPDATE" $GPGKEYS
SA_UPDATE_EXIT_CODE=$?
if [ $SA_UPDATE_EXIT_CODE -eq 1 ]
then
echo "No updates available"
exit 0
elif [ $SA_UPDATE_EXIT_CODE -gt 1 ]
then
echo "sa-update returned error $SA_UPDATE_EXIT_CODE"
exit $SA_UPDATE_EXIT_CODE
fi

################
# Compile rules

"$SA_COMPILE"
SA_COMPILE_EXIT_CODE=$?
if [ $SA_COMPILE_EXIT_CODE -ne 0 ]
then
echo "sa-compile failed with error $SA_COMPILE_EXIT_CODE"
exit $SA_COMPILE_EXIT_CODE
fi

##########################################
# Reload spamassassin spamd configuration

kill -s SIGHUP "$SPAMD_PID"

##########################
# Remove old rule backups

cd "$BACKUP_DIR"
for DIR in *
do
# be paranoid and check that the directory name looks plausible
if [ $(echo "$DIR" | egrep "^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}$") ]
then
# strip the hyphens from the date: YYYY-MM-DD becomes YYYYMMDD, now we can do numerical comparisons
if [ "${DIR//-/}" -lt "${FOUR_WEEKS_AGO//-/}" ]
then
sudo -u $BACKUP_OWNER rm -rf "${BACKUP_DIR}/${DIR}"
fi
fi
done

</nowiki>

Spamd stats

2011-07-16T11:47:30Z

Summary: New page: [[Category:linux]] [[Category:perl]] This script will collect stats from the spamd logs. It's based on another script called exim-stats and has only been modified in a minor way. The scri...

[[Category:linux]]
[[Category:perl]]

This script will collect stats from the spamd logs. It's based on another script called exim-stats and has only been modified in a minor way. The script will output values that can be collected by cacti. I'll leave it to you to create your own data and graph templates. I collect the data over snmp. An snmpd on the same host as the spamd, has an extend directive, that simply cats out the stats file when an certain OID is queried.

<nowiki>
[kieranw@nologo bin]$ cat spamd_stats.pl
#! /usr/bin/perl
#
# spamd stats grabber for NET-SNMP
#
# Based on exim stats grabber for NET-SNMP, which was based on a script
# by Matthew Newton and Copyright (c) University of Leicester, 2005
#
# Exim stats Slightly modified for Timico (http://timico.net) by
# Ian P. Christian. More information about Ian P. Christian's version
# can be found at
# http://pookey.co.uk/wordpress/archives/70-cacti-and-exim-a-mostly-complete-guide-part-1
#
# The spamd version of the script was made at Queen Mary College, Apr 2010
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this program; if not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
# Boston, MA 02110-1301, USA.

use strict;

# Options to set up here.
#
# $conf is where to store the current state file.
# $statistics is the file that contains the current data.
# $maillog is the latest spamd main log.
# $mainlogold is the last rotated out main log.
# $archive is a directory. If it exists then old data is written
# there, too.

my $conf = "/var/lib/spamd/stats/conf";
my $statsfile = "/var/lib/spamd/stats/statistics";
my $mainlog = "/var/log/spamd";
my $mainlogold = "/var/log/spamd.1.bz2";
#my $archive = "/usr/local/exim-snmp-stats/archive";
my $archive = "";

# The regular expressions will need tweaking below, especially the
# IP addresses that determine local or remote. This should be
# moved to a configuration file some day.

# read conf file of when last run and where we got to

my %stats = ();

# set some defaults. If the current-state file does not exist, lets
# start searching through the current mainlog from the beginning.
# Maybe in this case we should really just set $seek to be the last
# position in the file so that the beginning stats are not large?
my $seek = 0;
my $inode = inode_number($mainlog);
my $cantseek = 0;

if (-r $conf) {
open CONF, "< $conf";
while (<CONF>) {
chomp;
s/^\s*//;
s/\s*$//;
next if /^$/;
if (/^inode=(\d+)$/) {
$inode = $1;
}
if (/^seek=(.*)$/) {
$seek = $1;
}
}
close CONF;
}
$stats{"connections"} = 0;
$stats{"spam"} = 0;
$stats{"clean"} = 0;

# see if we can seek to current position in the mainlog. If not, then
# it has most likely been rotated
open LOG, "< $mainlog" or die "cannot open exim log file $mainlog!";
if (!seek(LOG, $seek, 0)) {
$cantseek = 1;
}
close LOG;

if ($inode != inode_number($mainlog) or
$cantseek or
$inode == inode_number($mainlogold) or
$seek > (stat($mainlog))[7] ) {
# we have changed to a new log file; read the previous mainlog first
if ($mainlogold =~ /\.bz2$/) {
my $oldumask = umask;
umask 0077;
`bunzip2 -ck $mainlogold > /tmp/spamd_stats.$$`;
read_log("/tmp/spamd_stats.$$", $seek, \%stats);
unlink("/tmp/spamd_stats.$$");
umask $oldumask;
} else {
read_log($mainlogold, $seek, \%stats);
}
$inode = inode_number($mainlog);
$seek = 0;
}

$seek = read_log($mainlog, $seek, \%stats);

write_stats($statsfile, \%stats);

# create config file
open CONF, "> $conf";
print CONF "inode=$inode\n";
print CONF "seek=$seek\n";
close CONF;

if (defined $archive and $archive ne "" and -d $archive) {
write_stats($archive."/".now_time(), \%stats);
}

exit;

sub now_time
{
my @n = gmtime();
my ($y, $m, $d) = ($n[5]+1900, $n[4]+1, $n[3]);
my ($hr, $mn, $sc) = ($n[2], $n[1], $n[0]);
$m = "0$m" if ($m < 10);
$d = "0$d" if ($d < 10);
$hr = "0$hr" if ($hr < 10);
$mn = "0$mn" if ($mn < 10);
$sc = "0$sc" if ($sc < 10);
return "$y-$m-$d-$hr-$mn-$sc";
}

sub inode_number
{
my $file = shift;
my ($dummy, $inode) = stat($file);
return $inode;
}

sub read_log
{
my ($file, $seek, $stats) = @_;
my $prevline = undef;
my $line;
my ($prevpos, $pos);
local *LOG;

open LOG, "< $file" or die "cannot open spamd log file $file!";
if (!seek(LOG, $seek, 0)) {
close LOG;
return $seek;
}
while ($line = <LOG>) {
if (defined $prevline) {
if ($line =~ /spamd: connection from/) {
$$stats{"connections"}++;
} elsif ($line =~ /spamd: identified spam/) {
$$stats{"spam"}++;
} elsif ($line =~ /spamd: clean message/) {
$$stats{"clean"}++;
}
}
$prevline = $line;
$prevpos = $pos;
$pos = tell LOG;
}
close LOG;

$prevpos = $seek unless defined $prevpos;

return $prevpos;
}

sub write_stats
{
my ($file, $stats) = @_;
local *STATS;

open STATS, "> $file";
print STATS <<EOF;
$$stats{"connections"}
$$stats{"spam"}
$$stats{"clean"}
EOF
close STATS;
}

</nowiki>

Calamaris

2011-07-16T11:41:42Z

Summary: New page: [[Category:linux]] [[Category:bash]] [[Category:scripts]] [[Category:perl]] Calamaris is a log analyser for squid. It makes nice reports that management type like. Actually they're very u...

[[Category:linux]]
[[Category:bash]]
[[Category:scripts]]
[[Category:perl]]

Calamaris is a log analyser for squid. It makes nice reports that management type like. Actually they're very usful in their own right.

This script will run calamaris over the squid access logs every night and build up a collection of Daily, Weekly and Monthly reports going back over the last two years.

It relies on a little perl script to split the access log up in to one log file per day. Most servers rotate their logs in the early hours of the morning. I wanted the reports to represent one whole day from 00:00 hrs to 23:59 hrs

access-log-split.pl:
<nowiki>
#!/usr/bin/env perl

# Take the squid log file specified as the first argument when calling this script,
# then split all entries in to seperate files for each day.

use strict;
use warnings;
use File::Basename;
use POSIX;

sub isoDate {
return strftime("%Y-%m-%d", localtime(shift or time));
}

my $logtype = "access";

my $previousDate = '';
my $logfile;
while (<>) {
my $line = $_;
my $timeStamp = 0;
if ($line =~ /^(\d+)\..*/) {
$timeStamp = $1;
}
my $currentDate = isoDate($timeStamp);
if ($currentDate ne $previousDate || !$logfile) {
if ($logfile) {
close($logfile);
}
open $logfile, ">>", "$logtype.$currentDate"
or die "Could not open $logtype.$currentDate: $!";
}
print $logfile $line;
}

</nowiki>

calamaris.sh:
<nowiki>
#!/bin/bash
#
# A script to generate calamaris log reports from the squid access logs
#
# This script depends on the squid log rotation script,
# /etc/cron.daily/squid.daily.job, and must be run after it.
#
# The script creates daily, weekly and monthly reports. It also keeps
# the last copy of any given day, week or month to allow comaprisons.
#
# The script needs to run each day, it won't catch up if any days are
# missed. If that happens, weekly and monthly reports will still be
# created but won't be accurate.
#
# A simple perl script, "access-log-split.pl", is used to divide log
# data in to one file per day, which is then boiled down by calamaris
# in to daily cache files. These cache files are kept for one month.

################
# Configuration

CALAMARIS=/usr/local/calamaris/calamaris
CONF=/usr/local/calamaris/calamaris.conf

# Logs are picked up from the log archive directory. See /etc/cron.daily/squid.daily.job
LOG_DIR=/var/log/squid
ACCESS_LOG="${LOG_DIR}/access.log"

CACHE_HOSTNAME=squid-cache.example.com

CURRENT_REPORTS_DIR=/var/lib/calamaris/reports
PREVIOUS_REPORTS_DIR="${CURRENT_REPORTS_DIR}/previous"

CACHE=/var/lib/calamaris/cache

BZCAT=`which bzcat`
CAT=`which cat`
LOGSPLIT=/usr/local/bin/access-log-split.pl
RSYNC=`which rsync`

REMOTE_HOST=www.example.com
REMOTE_USER=calamari
REMOTE_PATH=/var/www/html/squid-reports
REMOTE_SHELL="ssh -q -i /var/ib/calamaris/.ssh/id_rsa -o UserKnownHostsFile=/var/lib/calamaris/.ssh/known_hosts"

NICENESS=39

DAILY_REPORT_LABEL=daily
WEEKLY_REPORT_LABEL=weekly
MONTHLY_REPORT_LABEL=monthly

########################
# Options for calamaris

DEFAULT_ARGUMENTS="
--errorcode-distribution-report \
--input-format squid-extended \
--hostname ${CACHE_HOSTNAME} \
--peak-report new \
--response-time-report \
--size-distribution-report 10 \
--status-report \
--type-report-ignore-case"

# When generating cache files, keep don't limit the lenght of reports
CACHE_ARGUMENTS="${DEFAULT_ARGUMENTS} \
--benchmark 50000 \
--domain-report -1 \
--performance-report 60 \
--requester-report -1 \
--type-report -1"

# Limit the length of reports for general consumption - to limit
# privacy issues
OUTPUT_ARGUMENTS="${DEFAULT_ARGUMENTS} \
--benchmark 10000 \
--domain-report 50 \
--requester-report 50 \
--type-report 50 \
--output-format html,graph \
--image-type gif"

############
# functions

# date adjusts to the given number of days in the past and
# formatted using the provided formatting string
# usage: pastDate <minus no. days> <format>
pastDate()
{
if [ ! $1 ]
then
exit 1
fi
if [ ! $2 ]
then
exit 1
fi
if [ `uname` = "Darwin" ]
then
echo $(date -v -"$1"d +"$2")
fi
if [ `uname` = "Linux" ]
then
echo $(date --date="$1 day ago" +"$2")
fi
}

##############################
# Directory Checks / Creation

# Check required directories are writable - try to create them if missing or
# exit.
test -e "${CACHE}" || test -d "${CACHE}" || mkdir "${CACHE}"
if [ ! -d "${CACHE}" -o ! -w "${CACHE}" ]
then
echo "The cache directory, "
echo "'${CACHE}',"
echo "does not exist or is not writable"
exit 1
fi

if [ ! -d "${CURRENT_REPORTS_DIR}" -o ! -w "${CURRENT_REPORTS_DIR}" ]
then
echo "The reports directory, "
echo "'${CURRENT_REPORTS_DIR}',"
echo "does not exist or is not writable"
exit 1
fi

DAILY_DIR="${CURRENT_REPORTS_DIR}/${DAILY_REPORT_LABEL}"
test -e "${DAILY_DIR}" || test -d "${DAILY_DIR}" || mkdir "${DAILY_DIR}"
if [ ! -d "${DAILY_DIR}" -o ! -w "${DAILY_DIR}" ]
then
echo "The daily reports directory, "
echo "'${DAILY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

WEEKLY_DIR="${CURRENT_REPORTS_DIR}/${WEEKLY_REPORT_LABEL}"
test -e "${WEEKLY_DIR}" || test -d "${WEEKLY_DIR}" || mkdir "${WEEKLY_DIR}"
if [ ! -d "${WEEKLY_DIR}" -o ! -w "${WEEKLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${WEEKLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

MONTHLY_DIR="${CURRENT_REPORTS_DIR}/${MONTHLY_REPORT_LABEL}"
test -e "${MONTHLY_DIR}" || test -d "${MONTHLY_DIR}" || mkdir "${MONTHLY_DIR}"
if [ ! -d "${MONTHLY_DIR}" -o ! -w "${MONTHLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${MONTHLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

if [ ! -d "${PREVIOUS_REPORTS_DIR}" -o ! -w "${PREVIOUS_REPORTS_DIR}" ]
then
echo "The previous reports directory, "
echo "'${PREVIOUS_REPORTS_DIR}',"
echo "does not exist or is not writable"
exit 1
fi

PREV_DAILY_DIR="${PREVIOUS_REPORTS_DIR}/${DAILY_REPORT_LABEL}"
test -e "${PREV_DAILY_DIR}" || test -d "${PREV_DAILY_DIR}" || mkdir "${PREV_DAILY_DIR}"
if [ ! -d "${PREV_DAILY_DIR}" -o ! -w "${PREV_DAILY_DIR}" ]
then
echo "The daily reports directory, "
echo "'${PREV_DAILY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

PREV_WEEKLY_DIR="${PREVIOUS_REPORTS_DIR}/${WEEKLY_REPORT_LABEL}"
test -e "${PREV_WEEKLY_DIR}" || test -d "${PREV_WEEKLY_DIR}" || mkdir "${PREV_WEEKLY_DIR}"
if [ ! -d "${PREV_WEEKLY_DIR}" -o ! -w "${PREV_WEEKLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${PREV_WEEKLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

PREV_MONTHLY_DIR="${PREVIOUS_REPORTS_DIR}/${MONTHLY_REPORT_LABEL}"
test -e "${PREV_MONTHLY_DIR}" || test -d "${PREV_MONTHLY_DIR}" || mkdir "${PREV_MONTHLY_DIR}"
if [ ! -d "${PREV_MONTHLY_DIR}" -o ! -w "${PREV_MONTHLY_DIR}" ]
then
echo "The monthly reports directory, "
echo "'${PREV_MONTHLY_DIR}',"
echo " does not exist or is not writable"
exit 1
fi

########
# Dates

# these dates relate to the log files, i.e. yesterdays logs and activity.
DAY_OF_WEEK=$(pastDate 1 %w)
WEEK=$(pastDate 1 %U)
MONTH=$(pastDate 1 %m)
YEAR=$(pastDate 1 %Y)
DATE=$(pastDate 1 %F)
PREV_DATE="$(($YEAR - 1))${DATE:(-6)}"
PREV_PREV_DATE="$(($YEAR - 2))${DATE:(-6)}"

# if today is the 1st of the month, then yesterday was the end of the month
MONTH_ENDED=$(
test $(date +%d) = 01 && echo true
)

#############################
# Process and Cache Log Data

cd "${CACHE}"
if [ -f "${ACCESS_LOG}.${DAY_OF_WEEK}.bz2" ]
then
$BZCAT "${ACCESS_LOG}.${DAY_OF_WEEK}.bz2" | $LOGSPLIT
echo "Generating daily cache file"
$CAT "${CACHE}/access.${DATE}" | \
nice -n ${NICENESS} "${CALAMARIS}" \
${CACHE_ARGUMENTS} \
--cache-output-file "${CACHE}/cache.${DATE}" > /dev/null
rm "${CACHE}/access.${DATE}"
echo
else
echo "WARNING: No cache found for ${DATE}"
fi

###########################
# Create Daily HTML Report

if [ -d "${DAILY_DIR}/${PREV_DATE}" ]
then
echo "Moving previous Daily HTML report"
[ -d "${PREV_DAILY_DIR}/${PREV_PREV_DATE}" ] && rm -rf "${PREV_DAILY_DIR}/${PREV_PREV_DATE}"
mv "${DAILY_DIR}/${PREV_DATE}" "${PREV_DAILY_DIR}/${PREV_DATE}"
fi

echo "Creating Daily HTML Report"

if [ -f "${CACHE}/cache.${DATE}" ]
then
mkdir -p "${DAILY_DIR}/${DATE}"

# Get data from the cache file for the most recent day
nice -n ${NICENESS} "${CALAMARIS}" \
${OUTPUT_ARGUMENTS} \
--no-input \
--performance-report 60 \
--cache-input-file "${CACHE}/cache.${DATE}" \
--output-path "${DAILY_DIR}/${DATE}"
echo
else
echo "WARNING: No cache found for ${DATE}"
fi

#######################
# Create Weekly Report

if [ $DAY_OF_WEEK -eq 6 ]
then

if [ -d "${WEEKLY_DIR}/${WEEK}" ]
then
echo "Moving previous Weekly HTML report"
if [ -d "${PREV_WEEKLY_DIR}/${WEEK}" ]
then
rm -rf "${PREV_WEEKLY_DIR}/${WEEK}"
fi
mv "${WEEKLY_DIR}/${WEEK}" "${PREV_WEEKLY_DIR}/${WEEK}"
fi

echo "Creating Weekly HTML Report"

CACHE_FILES=
DATE_ADJUSTMENT=$(( $DAY_OF_WEEK + 1 ))
while [ $DATE_ADJUSTMENT -ge 1 ]
do

CACHE_DATE=$(pastDate $DATE_ADJUSTMENT %F)

if [ -f "${CACHE}/cache.${CACHE_DATE}" ]
then
if [ $CACHE_FILES ]
then
CACHE_FILES="${CACHE_FILES}:"
fi

CACHE_FILES="${CACHE_FILES}${CACHE}/cache.${CACHE_DATE}"
else
echo "WARNING: No cache found for ${CACHE_DATE}"
continue
fi

DATE_ADJUSTMENT=$(( $DATE_ADJUSTMENT - 1 ))

done
if [ $CACHE_FILES ]
then
mkdir -p "${WEEKLY_DIR}/${WEEK}"

nice -n ${NICENESS} "${CALAMARIS}" \
${OUTPUT_ARGUMENTS} \
--no-input \
--cache-input-file "${CACHE_FILES}" \
--output-path "${WEEKLY_DIR}/${WEEK}"
echo
fi

fi

########################
# Create Monthly Report

if [ $MONTH_ENDED ]
then

if [ -d "${MONTHLY_DIR}/${MONTH}" ]
then
echo "Moving previous Monthly HTML report"
if [ -d "${PREV_MONTHLY_DIR}/${MONTH}" ]
then
rm -rf "${PREV_MONTHLY_DIR}/${MONTH}"
fi
mv "${MONTHLY_DIR}/${MONTH}" "${PREV_MONTHLY_DIR}/${MONTH}"
fi

echo "Creating Monthly HTML Report"

CACHE_FILES=
for CACHE_FILE in $(ls "${CACHE}"/cache.${YEAR}-${MONTH}-*)
do

if [ $CACHE_FILES ]
then
CACHE_FILES="${CACHE_FILES}:"
fi

CACHE_FILES="${CACHE_FILES}${CACHE_FILE}"

done

if [ $CACHE_FILES ]
then

mkdir -p "${MONTHLY_DIR}/${MONTH}"

nice -n ${NICENESS} "${CALAMARIS}" \
${OUTPUT_ARGUMENTS} \
--no-input \
--cache-input-file "${CACHE_FILES}" \
--output-path "${MONTHLY_DIR}/${MONTH}"
echo

fi

fi

#########################
# Delete old cache files

if [ $MONTH_ENDED ]
then

echo "Removing old cache files"

cd "${CACHE}"

if [ ${MONTH##0} -gt 1 ]
then
RM_MONTH=$(printf "%0#2d" $((${MONTH##0} - 1)) )
RM_YEAR=$YEAR
else
RM_MONTH=12
RM_YEAR=$(($YEAR - 1))
fi

for CACHE_FILE in $(ls "${CACHE}"/cache.${RM_YEAR}-${RM_MONTH}-*)
do
rm "${CACHE}/${CACHE_FILE}"
done

fi

###########################
# Copy files to web server

echo "Copying files to ${REMOTE_HOST}"
nice -n ${NICENESS} $RSYNC \
--chmod=D=rx --chmod=F=r --chmod=u+w \
--delete-after \
--recursive \
--rsh="${REMOTE_SHELL}" \
--times \
"${CURRENT_REPORTS_DIR/%\/}/" \
"${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_PATH/%\/}/"
</nowiki>

SpamAssassin-scripts

2011-07-16T11:31:55Z

Summary: New page: [[Category:linux]] [[Category:bash]] [[Category:scripts]] Some simple scripts I user for working with spamassassin. See also [[Clean_Spam]] and [[Test_Spam]]. sa-deploy.sh - sanity check...

[[Category:linux]]
[[Category:bash]]
[[Category:scripts]]

Some simple scripts I user for working with spamassassin. See also [[Clean_Spam]] and [[Test_Spam]].

sa-deploy.sh - sanity check, compile rules, then reload rules:
<nowiki>
#!/bin/bash

sudo spamassassin --lint && sudo sa-compile && sudo kill -HUP $(cat /var/run/spamd.pid)
exit $?
</nowiki>

sa-reload.sh - sanity check and reload rules:
<nowiki>
#!/bin/bash
sudo spamassassin --lint && sudo kill -HUP $(cat /var/run/spamd.pid)

</nowiki>

sa-scantime.sh - pipe your spamd log output in to this. Yes you need that for loop if you want to filter the text stream before getting the message scan times:
<nowiki>
#!/bin/bash

while read line; do
echo $line
done | grep scantime | sed -e 's/Mar 21 $[^ ]*$ .*scantime=$[0-9]*\.[0-9]*$,size=$[0-9]*$.*/\1 time=\2, size=\3/g'
</nowiki>

sa-sts.sh - score, scan time and message size for everythig that's been scanned recently:
<nowiki>
#!/bin/bash
# spamassassin score, time, size

# <time>\t<score>\t<scan time>\t<size>

sudo tail -fn10000000 /var/log/spamd | \
egrep 'spamd: identified spam \([1-9]?[0-9]?[0-9]' | \
sed -e 's/ */ /' | \
cut -d\ -f 3,5,9,13,15- | \
sed -e 's/spamd\[//;
s/\]://;
s/(//;s/\/5.0)//;
s/ bytes\.//;' | \
cut -d\ -f 1,3- | \
tr " " "\t"
</nowiki>

sa-mem-calc.sh - see how much memory all you spamd processes are using:
<nowiki>
#!/bin/bash
echo $(ps axo cmd,sz | grep ^spamd | sed -e 's/ */ /g' | cut -d\ -f 3 ) | sed -e 's/ /+/g' | bc
</nowiki>

Ldap-scripts

2011-07-16T11:21:01Z

Summary:

[[Category:linux]]
[[Category:bash]]
[[Category:scripts]]

These are some little scripts I use to simplify working with an ldap server.

ldap-user.sh - for simple searches, one value against one attribute:
<nowiki>
#!/bin/bash -x

ldapsearch -LLL -W -Hldap://dir.example.com -D"cn=bossman,dc=example,dc=com" -x -b"dc=example,dc=com" \
"(&(|(objectclass=inetorgperson)(objectClass=posixAccount))($1=$2))" $3
</nowiki>

ldap-group.sh - for simple searches, one value against one attribute:
<nowiki>
#!/bin/bash -x

ldapsearch -LLL -W -Hldap://dir.example.com -D"cn=bossman,dc=example,dc=com" -x -b"dc=example,dc=com" \
"(&(objectClass=posixGroup)($1=$2))" $3
</nowiki>

ldap-modify - quickly apply ldif:
<nowiki>
#!/bin/bash -x

ldapmodify -P 3 -f $1 -W -Hldap://dir.example.com -D"cn=bossman,dc=example,dc=com" -x
</nowiki>

user-list-to-filter.sh - simple script to convert a list of space separated user names in to an ldap filter:
<nowiki>
#!/bin/bash
echo "(uid=$(echo $@| sed -e 's/ /)(uid=/g'))"
</nowiki>

Test Spam

2011-07-16T11:15:34Z

Summary: New page: [[Category:linux]] [[Category:bash]] [[Category:scripts]] This is a script I use for testing new spam rules. You can throw you spam corpus at spamd and see how it scores the messages. It...

[[Category:linux]]
[[Category:bash]]
[[Category:scripts]]

This is a script I use for testing new spam rules. You can throw you spam corpus at spamd and see how it scores the messages.

It expexts that you will have an environment variable call $spamd set, which is the ip address or hostname of the spamd server who's rules you want to test.

<nowiki>$ ./test-spam.sh ../blah/test-spam/*.eml</nowiki>

<nowiki>
#!/bin/bash

[ -z "$spamd" ] && echo please set spamd enviroment variable to the ip/name of the spamd to use && exit 1

for mail in $@
do
echo -e "\n$mail\n"
spamc -d $spamd -R < $mail
done
</nowiki>

Clean Spam

2011-07-16T11:12:10Z

Summary: New page: [[Category:Linux]] [[Category:bash]] [[Category:scripts]] This is a simple little script for cleaning up spam. Get the message cleaned up so it looks like it's just come in on a mailrelay...

[[Category:Linux]]
[[Category:bash]]
[[Category:scripts]]

This is a simple little script for cleaning up spam. Get the message cleaned up so it looks like it's just come in on a mailrelay, then pass it to spamassassin so you can test your new rules.

ssed is used here, because I usually run this script on a mac. macs have bsd sed, not gnu sed and bsd sed is not all that super.

<nowiki>
#!/bin/bash
ssed -i -e '/^X-Mozilla-Status/d;
/^X-SpamAssassin-Score/d;
/^X-SpamAssassin-SpamBar/d;
/^X-QM-SPAM-Warning/d;
/^X-Sieve/d' \
-e '/^Received: from mail[12]\.qmul\.ac\.uk .*/,+3d;
/^X-SpamAssassin-Report/,+3d;/^ \* /d;' \
$*

</nowiki>

Ts.sh

2011-03-12T11:38:57Z

Summary:

[[Category:Bash]]
[[Category:Squid]]

a simple script for getting a human readable date from a timestamp. I use this to convert the timestamps in the squid logfile.

<nowiki>
#!/bin/bash

date -d "1970-01-01 $1 sec GMT"
</nowiki>

Bacula

2011-03-12T11:24:40Z

Summary:

[[Category:linux]]
[[Category:bash]
[[Category:scripts]]

Nice backup system for linux [http://www.bacula.org http://www.bacula.org]

Here are some functions I use to check common things in bacula

<nowiki>
[kieranw@hammond ~]$ typeset -f
berroredvols ()
{
echo "list volumes" | sudo /usr/sbin/bconsole | grep Error
}
bjobsrunning ()
{
bstatusdir | grep "^[0-9]\{6\} [^ ]" | grep -E -v "(waiting execution|canceled|max Storage)"
}
bstatusclient ()
{
echo "status client=$1" | sudo /usr/sbin/bconsole
}
bstatusdir ()
{
echo "status dir" | sudo /usr/sbin/bconsole
}
bstatusstore ()
{
echo status storage=tapeworm | sudo /usr/sbin/bconsole
}
btapes ()
{
echo -e "query\n17\nquery\n18" | sudo /usr/sbin/bconsole | grep "^[+\|]"
}
bupdateslots ()
{
echo -e "update slots storage=tapeworm\n1" | sudo /usr/sbin/bconsole
}
bvolumes ()
{
echo "list volumes" | sudo /usr/sbin/bconsole | grep "1 | LTO3"
}

</nowiki>