NAME SpamSuite - Script suite to check and maintain SpamAssassin processed emails SYNOPSIS emailmanut formail -s manutbulk < filename formail -s manutbulks < filename m spamassassin-failed report < email check < email bulklean proclean msmtpclean clean saclean < email strip-att < email(s) mimedecode-sed < email rebuildSA newmail INSTALLATION Several configuration variables and filenames can be set in the config file, in particular, the installation directory $base, and color management, that is adjusted for a black console. Add $base to your $PATH and unpack the last spamsuite tarball at https://luis.impa.br/spamsuite into your $base directory: cd ; wget https://luis.impa.br/spamsuite/spamsuite.tar mkdir $base ; cd $base ; tar xvf $HOME/spamsuite.tar DESCRIPTION SpamSuite is a collection of 30 or so bourne shell scripts for Linux that checks, verifies, maintains and cleans emails and SpamAssassin Bayes files, log and databases with minimum user intervention. It assumes that SpamAssassin does not rewrite subjects, but instead separates the spam into two folders, $HOME/mail/bulk (low score) and $HOME/mail/bulks (high score). PROCESSING BULK AND BULKS MAIL: emailmanut, manutbulk and manutbulks The main program of the collection is emailmanut, that is executed with no arguments and is well suited for cron jobs. It cleans bulk and bulks folders by sending their emails to manutbulk and manutbulks scripts, respectively. After that, both bulk and bulks are deleted, saving only the emails that are not clearly spam according to Bayes (a fraction of the emails in bulk(s)). The scripts process bulk(s) emails according with the following rules: - bulk folder (manutbulk script): o If the email fell into one of the tests BAYES_99, C_ATT1 or C_ATT2, (the last two being custom procmail receipts that indicate dangerous attachments, i.e., executable files and the like), it is reported to DCC, Pyzor and Razor spam-blocker databases -if the email wasn't already reported according to Spamassassin headers-, logs the email From: and Subject: headers into $HOME/.spamassassin/emailmanut-log file and deletes the message. You can change these tests by changing accordingly the variable spamif in the config file. o If the email didn't fall into either tests, it puts it into $HOME/mail/0 folder. - bulks folder (manutbulks script): o The email addresses that belong to our SpamAssassin blacklist but that are not source of spam should be put in goodblk variable at the beginning of manutbulks script; see sample line in the file. These emails are simply discarded with no further action. o All other email is reported to DCC, Pyzor and Razor. The nonempty spam that is not in our blacklist, and that didn't fall into either BAYES_99, C_ATT1 or C_ATT2 tests is saved into $HOME/mail/spam0 folder. After processing bulk and bulks, emailmanut makes the Bayes subsystem to learn the emails contained in $HOME/mail/spam0 and $HOME/mail/ham0 folders, without re-syncing with Bayes database (see rebuildSA for this), and move them into $HOME/mail/spam and $HOME/mail/ham folders, respectively. In the process, emailmanut cleans all SpamAssassin headers, saving only the original email (see saclean). STATUS INFORMATION: m The m script requires no arguments and is quite useful to know your spam files status: it informs the content of your INBOX, 0, spam and ham folders, giving the From and Subject headers of their emails, together with its size in bytes. In addition, if some mail is found in 0 folder, it asks you if it is all spam, and if it is it saves all its emails into bulks folder and calls emailmanut for processing. It also sends a warning message if file spamassassin-failed exists, or if there are errors in msmtp-log file. Remark: msmtp is a small and lightweight program that replaces sendmail for those that use only SMTP relays for sending email and want SMTP authentication and encryption; see https://msmtp.sourceforge.net for more information. You can simply ignore all this if you don't use msmtp. UNPROCESSED EMAILS: spamassassin-failed If you want, you can send unprocessed SpamAssassin mails to a separate $HOME/mail/spamassassin-failed folder instead of your INBOX. Just add the following receipt at the end of your $HOME/.procmailrc file: :0: * !^X-Spam-Checker-Version: SpamAssassin * < 256000 $HOME/mail/spamassassin-failed The script spamassassin-failed takes $HOME/mail/spamassassin-failed folder and sends it through SpamAssassin via procmail, using $base/procmail-failed receipts. It uses sudo command to run spamd to save CPU. If you are not allowed by the root to use spamd via sudo, just comment the sudo lines in spamassassin-failed script, and change spamc for spamassassin in procmailrc-failed file. You can do both things executing this two commands in a console: sed -i 's/^sudo/#sudo/g' $base/spamassassin-failed sed -i 's/| spamc/| spamassassin/g' $base/procmailrc-failed REPORTING AND CHECKING (SPAM-BLOCKERS): report and check - Usage: report [< email] Reports the email to DCC, Pyzor and Razor, if the email didn't fall to the corresponding tests. If no email is supplied, it takes the email from /tmp/aux-emailmanut file. - Usage: check < email It checks if the email is listed in DCC, Pyzor and Razor. CLEANING: saclean, bulklean, proclean, msmtpclean, clean, mimedecode-sed, strip-att saclean < email Deletes SpamAssassin headers and returns the original email. strip-att < email(s) sed -i -f $base/strip-att Deletes unwanted attachemnts (virus/images/video/audio/M$office) from a whole mailbox or a single email. bulklean Deletes emailmanut-log file if no username matches .mutt.aliases file. See it with Emailmanut-log command. proclean Deletes procmail-log file if no error is found. msmtpclean Deletes msmtp-log file if no error is found. clean The last three, together with m script. It first prints a statistic line with the number of bulk, bulks, doubt (when manutbulk isn't sure it puts the email in mail/0 folder) and false- (false negatives) mails. It counts since the last reset if the plain command is executed, or the full historic stats if 'clean -t' is given. This information is contained in the last four columns of the .spamassassin/counters file. The counter file is reseted with the countersreset command. mimedecode-sed < email Decodes quoted-printable from the full text. LEARNING AND SAVING YOUR HAM AND SPAM: rebuildSA rebuildSA re-syncs your Bayes database, and moves your ham and spam folders to /backups/spam+ham/ham and /backups/spam+ham/spam folders, respectively. To use this script, directory /backups/spam+ham must exist and be writable by the user. NEWMAIL: newmail and newmail-aux They send to /dev/console From and Subject headers when new mail arrives. Execute it through your favourite checking email program. MUTT INTEGRATION (Of course you do use mutt, don't you?) For those that use mutt, here you have some macros to integrate mutt with this suite. Add them to your muttrc configuration file: macro index \eh :'set confirmcreate=no'\n\;s=ham0\n:'set confirmcreate=yes'\n "Save as HAM for learning" macro pager \eh i\eh "Save as HAM for learning" macro index \es :'set confirmcreate=no'\n\;s=bulks-\n:'set confirmcreate=yes'\nc!\n "Save in Bulks for learning" macro pager \es i\es "Save in Bulks for learning" PREREQUISITES (aside from usual sed, sh, etc.) Spamassassin, DCC, razor-agents, pyzor, formail, procmail, esdplay. Optionally: msmtp, sudo, mutt. SEE ALSO spamassassin(1), procmail(1), formail(1). FILES $HOME/mail/0 When manutbulk is not sure, it saves the email in this folder $HOME/mail/bulk $HOME/mail/bulks SpamAssassin filtered spams $HOME/mail/bulks- Where false negatives should be saved $HOME/mail/ham0 $HOME/mail/spam0 Ham and Spam that will be learned by emailmanut $HOME/mail/ham $HOME/mail/spam Ham and Spam saved by emailmanut after processing $HOME/mail/ham0 and $HOME/mail/spam0 $HOME/mail/spamassassin-failed Folder where procmail saves email unprocessed by SpamAssassin $base/procmailrc-failed Procmail receipt for email unprocessed by SpamAssassin $HOME/.spamassassin/msmtp-log $HOME/.spamassassin/emailmanut-log $HOME/.spamassassin/procmail-log Log files $HOME/.mutt.aliases Mutt email addressbook /backups/spam+ham/ham /backups/spam+ham/spam Backup ham and spam email for Bayes feeding BUG REPORTS Although the suite runs with no bugs for me, it does so under my own configuration. Please report bugs and suggestions to: luis@impa.br AUTHOR Luis A. Florit, luis@impa.br COPYRIGHT SpamSuite is Copyright (C) 2004-2017 by Luis A. Florit. It is free software, you can redistribute and/or modify it under the terms of the GNU General Public License.