Do you know what a SPF record is?
No?
Neither did I until Microsoft decided to class me as a spammer, and if you read on you might just save yourself from loosing several days of your life trying to implement one.
Me Sir, a ’spammer’?
Anyway, to understand what I’m rambling on about we need a bit of background, and why this ‘SPF Record’ is getting me so wound up.
I, unlike the majority of people with common sense, use Hotmail as my primary email provider, and have done since I first starting using the Internet. In fact I had my Hotmail address before it became part of the Microsoft empire. One thing that annoys me however, is that I am now having to put up with more and more spam, despite efforts to curtail it.
We all know the stress of sorting through spam, and thank the people who work on solutions to filter out or just stop that crap coming through. However, I am sure you will understand my annoyance when I found out that thanks to the configuration of my (dv) dedicated-virtual server I have in fact been branded a ’spammer’ by Microsoft, and as a result they appear to be black holing any mail sent to a Hotmail account from my (dv).
Before I go any further I would just like to clarify that this is in fact nothing to do with the (dv) server as a product or Media Temple, but rather the way in which a virtual server environment works. I have found dozens of references via Google of people complaining of the same problems, and interestingly most seem to refer to people running VPS environments using Plesk.
As with all things, when something goes wrong, you have to learn how it works to be able to fix it, and thus I have been learning some of the ins and outs of running mail servers and the DNS system.
Disclaimer: At this point I would just like to say I only have a (very) basic idea about how either work, so don’t take anything I say as gospel, but rather use it as a loose guide and reference to where you may find further help.
Where is my mail going!?
After getting in touch with the guys at (mt) I decided that I needed to find where my bloody mail was going. I wasn’t getting a bounceback mail from the Hotmail server, and Thunderbird told me that the mail was delivered. Thankfully due to the fact that the (dv) allows you to delve into the OS to see what’s going off, I thought I would interview the SMTP log and see what was going off. The SMTP server in Plesk’s case is called “Qmail’ and the logs are located at # /usr/local/psa/var/log/maillog and can be read in a number of ways. In this case I found the easiest way to track what was going off was to use the tail -f command which spurts out the log information for events as they are happening, and this is what I got when I tried to send an email to my Hotmail account:
Mar 22 17:32:23 as qmail: 1174584743.517414 delivery 437: success: 65.54.244.168_accepted_message./Remote_host_said:_250_ <4602BEEF.1080905@helloian.com>_Queued_mail_for_delivery/
So it would seem that the Hotmail server is accepting the mail, queuing it, but never actually delivering it, due to their spam filtering technology. A quick search on Google showed that plenty people seemed to have experienced the same problem. Interestingly most were using Plesk, and virtually all of them were using Qmail as their SMTP server. Clicking the seemingly never ending list of results, I realised that not one had any comments regarding a working solution, but the acronym SPF kept popping up a lot, so I decided it was worth a look.
The Sender Policy Framework
The Sender Policy Framework allows a domain owner to specify which machines are allowed to send email on its behalf. This kind of mechanism is unfortunately not present in the Simple Mail Transfer Protocol, a fact that allows spammers to send e-mail from forged addresses relatively easily, as there is no inbuilt validation when an email is sent and then received.
Fortunately the remedy is relatively straight forward to implement. The SPF record is applied as a TXT type entry in the domain’s DNS record, and it’s as simple as that. Now, when you send an email, the receiving mail server can use this SPF record to verify that the origin of the email is legitimate. To help illustrate what is happening, below is a MIME header from an email I sent between two accounts on my (dv).
Return-Path:
Delivered-To: 3-sayhello@helloian.com
Received: (qmail 32062 invoked from network);
29 Mar 2007 17:59:58 +0100
Received: from 85-211-13-70.dyn.gotadsl.co.uk
(HELO ?192.168.1.5?) (85.211.13.70)
by distillate-hosting.net with (DHE-RSA-AES256-SHA encrypted)
SMTP; 29 Mar 2007 17:59:58 +0100
Message-ID: <460BF1EE.4020508@distillate.co.uk>
Date: Thu, 29 Mar 2007 18:05:50 +0100
From: Ian Halliday
User-Agent: Thunderbird 1.5.0.10 (Windows/20070221)
MIME-Version: 1.0
The confusion arises when the receiving machine reads the email is claiming to be from the domain ‘distillate.co.uk’ but has been sent via the server ‘distillate-hosting.net’. As far as the machine is concerned, there is no link between the claimed sender and the machine it originated from. There is no way to tell if this information is legitimate or not.
The reason that my initial searches on Google seemed to show that it was mostly VPS users with multiple domains that were suffering from this problem is that by its very nature, a VPS server running by multiple domains will send mail from the mail server of any given domain (in my case distillate.co.uk) through the SMTP server of the host VPS platform (distillate-hosting.net in my case). Unfortunately emails sent using this setup look very similar to ’spam’ messages, and the Hotmail spam filter (known as ‘SmartScreen’) is quick to step in and black hole the email, meaning it never reaches its destination, despite the Hotmail server notifying the sender that the email has been received and delivered.
Fortunately, this is where the SPF record steps in to clear matters up. The SPF record tells the receiving machine that the server ‘distillate-hosting.net’ sends mail on behalf of the mail exchanger for the domain ‘distillate.co.uk’ and this is written as:
v=spf1 mx ip4:XXX.XXX.XXX.XXX mx:mail.YYYYYY.YYY ?all
Where:
v=spf1 Denotes the following as a SPF record.mx States that the Mail Exchanger sends outbound mail for server as stated in the next segment<./li>ip4:XXX.XXX.XXX.XXX Is the IPv4 formatted IP address of the (dv) server.mx:mail.YYYYYY.YYY States that the Mail Exchanger of the domain specified (YYYYYY.YYY) sends mail through the IP previously specified.?all States that any IP’s that fail to meet any of the listed ‘mechanisms’ will return “neutral”, thus will be treated as if a record does not exist.
To clarify, the SPF record for my domain distillate.co.uk is entered in the DNS zone file as:
v=spf1 mx ip4:216.70.127.122 mx:mail.distillate-hosting.net ?all
The Open SPF website explains the above is more detail, and offers a tool to help you set up your SPF record. Microsoft also have a similar tool available which after being referred to by Hotmail technical support, turned out to be more of a hindrance than a help. The Microsoft tool, and many other references recommend that a PTR mechanism is included in the SPF record. The PTR record allows reverse lookup of an IP address; that is identify the domain of an IP address. The reverse lookup is used to verify that the domain name and IP address in the email MIME header actually correlate and have not been faked. Whilst this sounds like a good idea, actually processing a reverse look up takes a considerable amount of time and it is not generally a method employed by large email providers like Hotmail. In fact Hotmail refused my initial SPF record as it included this PTR mechanism. To quote Hotmail technical support:
The specification for SPF records (RFC 4408) discourages use of “ptr” for performance and reliability reasons. This is especially important for Windows Live Mail, Hotmail and other large ISPs as a result of the very high volume of mail we receive each day. We highly recommend you remove the “ptr” mechanism from your SPF record and, if necessary, replace it with other SPF mechanisms that do not require a reverse DNS lookup, such as “a”, “mx”, “ip4″ and “include.”
Troubleshooting
The very nature of the DNS system made this problem a very frustrating one to tackle, as you don’t see instant results from your implementation, but of course have to wait anywhere up to 48 hours for the information to propagate throughout the internet. You can however use some of the tools on the Open SPF website to check your record is configured properly. Once you have confirmed that your record is set up correctly you can also send a blank email to check-auth@verifier.port25.com which will test your SPF record, and email you back the results.
I also found dnstuff.com invaluable in testing my DNS set-up. Whilst it doesn’t check the functionality of your SPF record (it only checks that you have one), then DNS Report tool on dnsstuff.com gives you feedback on all aspects of your DNS configuration and can be an excellent tool for troubleshooting.
SPF Works!
Finally I can email Hotmail users without worrying if it will go through, and if you are running a (dv) or similar setup then I strongly suggest you use a SPF record, even if you are having no problems at the moment. One way of making life even easier for yourself in the future if you use Plesk would be to use your Plesk server as the nameserver for all domains residing on it, and set up a SPF record in the main server DNS page, accessible from the main server configuration page. By doing this all new domains will automatically have the correct SPF record setup for them. If you are only running a few domains, just make the changes in (mt)’s account center and continue to use the (mt) nameservers.
If the above doesn’t work for you, get in touch with your hosting provider and make sure you have run all the tests I mentioned. Unfortunately in the end there is no substitute for really understanding what is going wrong, so I suggest you familiarise yourself with how the DNS system works. Wikipedia has an excellent article and Media Temple’s Knowledgebase has a more concise article available, either of which should put you on the right track.
Update:Well I may have spoken slightly too soon regarding everything being fine. It turns out that my emails are still not guaranteed to go straight through to any given Hotmail inbox, but rather the Hotmail spam filter will take a while to learn that my domains are trustworthy and that the SPF records check out. At the moment some emails go through okay, some go to the Junk folder.
I have been informed by Microsoft that over time (approximately a month) more of my emails should go straight through to the inbox. If anyone gets an email that lands in their junk mail (by subscribing to comment updates for example) you would be doing us both a great favour by checking ‘this is not junk’, which will ensure all mail from my server reaches your inbox in future, and that I will look better in the eyes of the Hotmail spam filter.