I have my little server on the net. It is hosting my imap, trac and this blog as well as some other sites. Normally it runs safe and I don't change anything except some security updates.
However I did an bigger update from debian sarge to the new version etch some weeks ago. I did not discovered that this update changed my smtp server configuration to an open mail relay until last weekend.
The whole story starts at Saturday evening. I logged into my box and made some security updates when my box printed its last line "Segmentation fault. Trying to recovery.". I could still ping the server but anything else went down. Ok, I need a restart it fast. Hopefully there is still someone at the server-park? But what is this? I forgot the password for the web interface and the email-address is hosted by the server in question. I started searching my mail backups and old firefox installations for a hint to the password. I at least found the login name.
Time went by and my restart request would not be processed until Monday at 6.30 am. But I still had no password. I searched two external hard-disks but they both were broken. My laptop was broken too. But since I had an over dimensional soldering iron ready to hand I started to repair my laptop in hope to find my lost password. After about an hour my laptop was running even though five screws were remaining. But still no password.
The last and successful attempt to receive a password was a bit tricky. I set up a mail server on my local machine. Than a quick change of my mx record in the dns zone file and (tada!) everything worked fine. I could request a new password and wait until Monday for my server to restart without missing a mail.
Monday my box was up and running and the syslog showed no problem with the hard-disk recovery. However at Thirsday i had problems sending a mails. The server returned "Too many connections try again later."(or something). I started searching for the cause. The smtp log showed thousands of spam mails relayed for many different IPs each minute. Oh shit. It looked like my server was used as one of many open relays for a spam bot net. It took me some hours to reconfigure the mail server, because I just did it once before.
After the service was restricted to only relay my own mails there were still thousands of messages processed each minute. I was really shocked how many mails were queued up for relay. 1,7e6 mails in the queue! Argh! I removed the queued mails in questions and am still heavily monitoring the system for any activity.
Some zombies are still trying to relay mails but the system is clean now.
I hope no important mail was lost due the process. But I guess not. The mails in the queue were most likely undeliverable mails queued up for retry. And the down time of my server would only cause the mails directed to me to be queued up on another server for retry.
But there are still some lessons I learned:
- Never maintain your server on the weekend.
- Always be paranoid about update changes. Or better just: Always be paranoid.
- Backup everything regularly.