Problem mail queues and how to (sort of) fix them

 
Post new topic   Reply to topic    34SP.com Forum Index // VPS (Virtual Private Server)
View previous topic :: View next topic  
Author Message
garethw
34SP Newbie
34SP Newbie


Joined: 06 Dec 2007
Posts: 54
Location: Stafford, Staffordshire, UK

PostPosted: Wed Mar 07, 2012 8:44 pm    Post subject: Problem mail queues and how to (sort of) fix them Reply with quote
Hi All,

It looks like my VPS has been hit with the same problems other people have been having relating to perl scripts and spam mail. I got in touch with support and was told that as it's a VPS I'm on my own.... I wasn't very happy about that but will worry about that later.

I thought I would be worth making some notes here in case anyone else has this problem.

I did apply the patches as suggested in the middle of Feb (2012) and changed both the root and admin passwords on my vps. The 2 find commands suggested at the time also came back clean. None the less, it's been compromised at some point.

So, I noticed a delay in mail being delivered and had a call from a customer with problems. When I looked in server > statistics on plesk I noticed that my load averages were up significantly (8 - 10 area). I ssh'd onto the box and found loads of processes all mail related and a very over active perl process.

I stopped qmail ( server > service management) and checked the mail queue (#/var/qmail/bin/qmail-qstat), which indicated over 200,000 mails in the queue.

At this point I changed passwords again, restarted the server and stopped qmail again.

Now for the cleanup.....
_________________
--
Gareth Westwood
WFF Systems LTD
Back to top
View user's profile Send private message Visit poster's website
garethw
34SP Newbie
34SP Newbie


Joined: 06 Dec 2007
Posts: 54
Location: Stafford, Staffordshire, UK

PostPosted: Wed Mar 07, 2012 9:05 pm    Post subject: Reply with quote
So, the mail queue is located at /var/qmail/queue, as it was so full I decided to move that out of the way and create a new queue whilst I work out what to do with the old one.
Code:
cd /var/qmail &&
mv ./queue ./queue-orig &&
mkdir queue &&
chown qmailq:qmail queue &&
chmod 750 queue

Next you need to create all the sub directores. qmail uses a number of subdirectores to "split" the mail down into more managable groups. My vps was using 23 split directores. You can check your current queue by;
Code:
ls /var/qmail/queue-orig/info

You will need to find the highest numbered directory and remember what it was, we need it later.

First we will create 1 folder to use as a template, then copy that to make the folders we need. Replace "22" here with the highest numbered directory you found earlier. Note, the ` below is not a single quote it's a the char to the left of 1 on a std keyboard.
Code:
mkdir 34sptmp &&
for num in `seq 0 22`; do
mkdir 34sptmp/$num
done &&
for fred in {info,intd,local,mess,remote,todo}; do
cp -r 34sptmp $fred
done &&
rm -r 34sptmp

We also need some other directories but these don't need the splits in them.
Code:
mkdir {bounce,lock,pid,yanked}

This is the directory structure setup, just need to set permissions now....
_________________
--
Gareth Westwood
WFF Systems LTD
Back to top
View user's profile Send private message Visit poster's website
garethw
34SP Newbie
34SP Newbie


Joined: 06 Dec 2007
Posts: 54
Location: Stafford, Staffordshire, UK

PostPosted: Wed Mar 07, 2012 9:33 pm    Post subject: Reply with quote
Assuming we are still in /var/qmail we need to run the following to set permissions right.
Code:
cd queue &&
chmod -R 700 * &&
chmod -R 755 yanked &&
chmod -R 770 todo mess lock &&
And then set the ownerships
chown -R root:root yanked &&
chown -R qmails:qmail bounce info local remote &&
chown -R qmailq:qmail intd lock mess pid remote

Then last but not least (I think) there are a couple of files that need copying across from lock
Code:
cp ../queue-orig/lock/{sendmutex,tcpto} ./lock

Finally, start qmail again from plesk. If it fails to start it is worth running;
Code:
tail -f /usr/local/psa/var/log/maillog

and checking what errors get thrown up (ctrl + c to quit tail)

Now, that's as far as I have got at the moment. My vps seems to be delivering mail again ok and it's not building a massive queue again. The next job is to attmept to recover the original queue and remove some of the spam from it.... Watch this space.
_________________
--
Gareth Westwood
WFF Systems LTD
Back to top
View user's profile Send private message Visit poster's website
garethw
34SP Newbie
34SP Newbie


Joined: 06 Dec 2007
Posts: 54
Location: Stafford, Staffordshire, UK

PostPosted: Thu Mar 08, 2012 4:53 pm    Post subject: Reply with quote
Last Step....
I checked smtp pendings log for a place to start
Code:
less /usr/local/psa/var/log/smtp_pendings.log

I spotted a couple of address sending loads of mail one of which was the root user for the vps and should not have been sending anything afaik. Make a note of any questionable address.

Check how much space the orig queue is using (to give you an idea of how long this is going to take).
Code:
cd /var/qmail/queue-orig &&
du -sh ./

Backup your original queue (to be safe), this may take some time depending on how big it is.
Code:
cd .. &&
cp queue-orig queue-cleaned &&
cd queue-cleaned

Now, the following code will, recursively from the current working directory, find all files containing a specific string (YOUR_STRING_HERE) and will delete the file. Obviously, please double check your pwd before hitting go and be sure you are sure!

just to be sure (should be /var/qmail/queue-cleaned if everything has gone according to my instructions.
Code:
pwd

Then
Code:
find -type f -exec grep -q 'YOUR_STRING_HERE' '{}' \; -exec rm -f '{}' \;


Once the initial couple commands had gone through I started using du to find out which directories had excessive amounts of mail in them
Code:
du -h ./

I then cd's to those folders and less'd a random selection of the files to find common phrases that were in the spam mail. Examples of strings I found are;
Quote:
@txt.bell.ca
Hudson Bay Company
pcs.rogers.com
root@myvps.mydomain.com


I then ran the find command above with those strings. Eventually I was down to what seemed a little more sensible size. I also stopped finding repeated patterns in the queue files.

I then stopped qmail again, moved the new queue out of the way, moved the clean queue back and ran qstat to see what was left
Code:
cd /var/qmail/ &&
mv queue queue-new &&
mv queue-cleaned queue &&
/var/qmail/bin/qmain-qstat


In my case this gave me a result of 32 emails. I stated qmail again and allowed this queue to clear before copying the new one back.

A few points of note.
1/ This worked for me, do it at your own risk.
2/ There is probably a way to make the find bit look for multiple strings at the same time however at 2AM I decided not to mess around trying to work it out.
3/ You may have to copy the queues back and forth a couple of times to clear all the backlogs.
4/ timestamps in smtp_pendings.log are in unix timestamp format, see http://www.onlineconversion.com/unix_time.htm to convert them
5/ The && at the end of some of the lines of code means if this command finishes without error, run the next command otherwise (if there is an error) stop.
_________________
--
Gareth Westwood
WFF Systems LTD
Back to top
View user's profile Send private message Visit poster's website
Post new topic   Reply to topic    34SP.com Forum Index // VPS (Virtual Private Server) All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Powered by phpBB © 2001, 2002 phpBB Group