2015
02.02

We’ve all experienced the inconvenience of finding an important message sent to us has ended up in the Junk Mail folder in our mailbox and the risk that this might have been removed before we were able to read it. Similarly, for those who run websites, particularly those who send out newsletters and accept orders from visitors, the threat of mail being rejected or junked is something we need to be constantly aware of.

The reputation of an IP address or domain needs to be protected, particularly in mission-critical or e-commerce scenarios, and the threat of being listed by one of the major blacklisting firms or a provider such as Hotmail or Google is a huge risk. There are a multitude of reasons your reputation can take a nose dive, such as a compromised script or a security breach resulting in spam being sent out. It is also necessary with a new IP address to establish a reputation as a responsible owner. In my own case, I’d found that given my IP address was a new one (due to me migrating from my old server a few months ago) some of the messages I was sending via contact forms, or to Manchester Film Co-op newsletter subscribers were unintentionally ending up in Junk Mail folders for Google mailboxes.

There are however a number of things you can do to avoid this: some technical, some non-technical. In the case of bulk mail, you can firstly ensure that whatever service you run complies with the laws on bulk mail. In my case, the newsletter system integrated into Manchester Film Co-op has a confirmation routine and includes an unsubscribe link on every message.

So what then, are the technical measures? I’ll look here at two of these in detail, reverse DNS and SPF. I can’t claim to be an expert in either, as these tips are mostly what I’ve learnt through my own experiences, so do feel free to post any corrections or further suggestions in the comments below.

rDNS

When you type a domain into your web browser, your machine looks up an “A” record for the domain, to discover the IP address of the website. The main purpose of DNS is to translate domain names into IP addresses. When you type “www.bbc.co.uk” your ISP’s DNS server should return an IP of 212.58.244.70. This “forward” A record allows you to reach the website. Reverse DNS works the way you might expect, it sets a host name for your IP address. But why is this important for email delivery?

A large number of mail servers will perform a reverse DNS check when receiving mail from an IP address to see whether the reverse DNS entry for the server matches the forward A record of the domain it is supposedly from. If these do not match, or if no rDNS record exists, senders are liable to have their mail rejected by some recipients.

So for example, if a mail server at 11.22.33.44 identifies itself as mail.yourdomain.com, the recipient mail server will then ask 11.22.33.44 for its rDNS record, sometimes called a PTR record. If the rDNS record of the IP is blank, is not set to mail.yourdomain.com (or is a default PTR such as ‘no-dns-11-22-33-44.myisp.uk.’) then mail is more likely to be rejected. rDNS is usually controlled by your hosting or broadband provider, depending on where you host your domain.

If your IP address is used to host a single website or mail server, you should set your rDNS to match the domain name of this site. In the case of this website, which runs on Bytemark’s BigV platform, we have a rDNS set of “bigv.jonathandavis.me.uk”.

A record: bigv.jonathandavis.me.uk = 213.138.101.210

rDNS record: 213.138.101.210 = bigv.jonathandavis.me.uk

As this server hosts several different sites, and you can only have one rDNS for an IP address, setting a valid domain with a matching forward A record is the best strategy to improve the server’s mail reputation. Lack of an rDNS record is a far bigger issue than the record not matching the sender domain.

Now we come to the next important measure I want to discuss: SPF.

SPF Records

SPF – or Sender Policy Framework – leverages the DNS system to provide additional verification that the server is question is allowed to send mail for your domain name. As is the case for many websites, you might be sending email from a variety of servers. For the most part, these servers will also be the ones on which you receive mail, and you would have MX records set-up (eg: “mail.yourdomain.com”) and this server would have an rDNS record to match, which would verify that this server is related to your domain and that your mail should be accepted as legitimate. However, this isn’t always the case, as most websites also have the potential to generate emails, such as from a contact form, or a forum sign-up page. As most mail and web servers are separated, this poses a problem, as we need to let providers such as Google and Hotmail know that they should be accepting mail from our domain name which is being generated from perhaps several different IP addresses.

SPF makes use of the DNS record type – TXT – to provide a list of server names or IP addresses from which we should expect our mail to originate from. TXT records can be used for a variety of purposes, but SPF is now one of the most common. Creating an SPF record – through your domain registrar – allows a mail provider such as Google to grab this information and compare the IP address the email has been sent from, to the addresses we have specified will be sending mail for our domain.

To use an real-life example, some of the messages I sent out to subscribers of the Manchester Film Co-op, using a database-driven mailing list script were being rejected, as there was nothing to explicitly state that the web server’s IP address will be sending mail from an address at manchesterfilmcoop.uk. So here I added the following SPF record:

manchesterfilmcoop.uk       TXT       “v=spf1 mx a”

This configuration instructs an SPF-compliant server that any IP address listed in the A or MX records is also designated as being allowed to send mail for manchesterfilmcoop.uk. In this case, 213.138.101.210 (the IP my bulk mails are arriving from) is listed as an A record, so Google then has no qualms about accepting this as legitimate.

The syntax of SPF records can sometimes be quite tricky to assemble, so I can highly recommend the services of spfwizard.net, a free online SPF record builder. This is particularly important where adding multiple IP addresses, or a range of addresses from where recipients should expect your domain to send.

Summary

So to round up: whilst there are a huge number of things which can affect the reputation of your IP address and the likelihood of mail being accepted (so much so you could easily write several books on it) we can see that through a few simple measures you can avoid many of the potential reasons that senders find that their mails are never read by their intended recipient.  Since implementing SPF I have found that almost all of my mail is now arriving in inboxes and is dodging the dreaded Junk Mail folder entirely, which although is a useful improvement for me, it could be decisive for an e-commerce company struggling to reach their customers.

2014
09.07

As part of my day-job, I perform a large number of DNS look-ups. Sometimes these are done to diagnose complex DNS issues and tests need to be run from various servers by various methods, but most of the time I’m simply looking for the current information to check everything is configured correctly.

If you’re not sure what DNS is, the Wikipedia article is a good place to start, but essentially every domain name has a number of records behind it (such as A records for web traffic, and MX records for email) which tell it where to direct certain types of traffic. When you type a domain into your web browser, it is DNS which translates this to the IP address of the server, allowing you to reach a website without having to know this information yourself.

I’ve used a wide variety of DNS look-up tools online, all of which have their pros and cons, but for a long time I’d been looking to build one of my own, which I could customize to my specific needs, so I set out to learn how they are built. My PHP skills are barely adequate, so this could also give me some much needed coding practice.

I started by creating a PHP script and adding a very simple HTML form, to provide somewhere for users to enter the domain they’re interested in and POST this ‘domain’ variable back to the script:

<form action=”dns-lookup.php” method=”post”>
<table style=”width:500px”>
<tr>
<td width=”30%” align=”right”>Domain Name:</td>
<td width=”40%”><input type=”text” name=”domain” value=”” /></td>
<td width=”30%”><input type=”submit” value=”Submit” />
</tr>
</table>
</form>

From this point, I wasn’t sure quite how PHP could be used to grab the relevant information on IP addresses and hostnames. Obviously this information is available to the server, so I’d imagined some kind of process where the system runs a dig or similar, then pipes the output back to the script. However this would be incredibly clunky. I soon discovered the dns-get-record function, which PHP can use to perform lookups on various DNS types.

To take the most basic example, if I want to return the IP addresses for the A records on a domain name, I can use the DNS_A parameter. In order to perform the look-up on the domain we entered into our form, we need to use $_POST[‘domain’] to grab the contents of the form we submitted and feed these into the dns-get-record command. As there are commonly multiple A records, we also start a foreach loop, to print the IP address for each one:

$get_A_records = dns_get_record($_POST[‘domain’], DNS_A);

foreach ($get_A_records as $Arecord) {
echo “” . $Arecord[‘ip’] . “\n<br>”;
}

This command will spit out a simple list of IP addresses.

206.190.36.45
98.138.253.109
98.139.183.24

This might be adequate, but we also commonly want to know what the name of this server is, which we can get using the gethostbyaddr command. We set a variable of $hostname and feed each IP address into the foreach loop:

$get_A_records = dns_get_record($_POST[‘domain’], DNS_A);

foreach ($get_A_records as $Arecord) {
$hostname = gethostbyaddr($Arecord[‘ip’]);
echo “” . $Arecord[‘ip’] . ” (” . $hostname . “)” . “\n<br>”;
}

This will return the hostname of the server alongside the IP address:

98.138.253.109 (ir1.fp.vip.ne1.yahoo.com)
98.139.183.24 (ir2.fp.vip.bf1.yahoo.com)
206.190.36.45 (ir1.fp.vip.gq1.yahoo.com)

We can then clean our code up a little more, by sanitizing the input, and returning a value to inform the user when there are no results. We can use the trim command to strip initial and trailing blank characters (which happens often when you’re copying and pasting domain names in) and we can create an if statement which will return a message informing us if no records of that type exist:

$get_A_records = dns_get_record(trim($_POST[‘domain’]), DNS_A);

if (empty($get_A_records)) {
echo “<font color=grey>[None Found!]</font>”;
}
foreach ($get_A_records as $Arecord) {
$hostname = gethostbyaddr($Arecord[‘ip’]);
echo “” . $Arecord[‘ip’] . ” (” . $hostname . “)” . “\n<br>”;
}

We can then use the other parameters of the dns_get_record command to get other information, such as CNAME records, TXT records and MX records. I have included below an example of how the code can be adapted to obtain all of the mail-related MX records for a domain. You should consult the documentation on the function to see which specific arrays are available for each record type.

For example, MX records have a target address and – unlike A records – a priority, which tells mail servers which address to attempt delivery to in the first instance. Upon failure, delivery will be attempted at the next address:

$get_MX_records = dns_get_record(trim($_POST[‘domain’]), DNS_MX);
if (empty($get_MX_records)) {
echo “<font color=grey>[None Found!]</font>”;
}
foreach ($get_MX_records as $MXrecord) {
echo “” . $MXrecord[‘target’] . ” (” . $MXrecord[‘pri’] . “)\n<br>”;
}

Which returns the mail servers, and the priority in brackets:

ASPMX2.GOOGLEMAIL.COM (10)
ASPMX.L.GOOGLE.COM (1)
ALT1.ASPMX.L.GOOGLE.COM (5)

The other functionality I wanted to include in my tool was a whois look-up. This is something which is slightly trickier to do in PHP, as whois is essentially its own protocol running on port 43. From all I’ve read, there doesn’t seem to be any ‘correct’ way to perform a whois look-up in PHP, as every method relies on connecting to the whois server for the top-level-domain (TLD) and manipulating the returned data.

The two primary ways of accessing whois data seem to be using the CURL or FOPEN functions in PHP to open a connection to a remote whois server. I have found that FOPEN seems to work far better for accessing all of the different servers, as I found CURL struggled with the .org and .info servers.

Given that any script requires a manual list of whois servers for each TLD, it is probably better to use an existing script and adapt this for your needs. The only issue is that it is very rare to find a script which has no bugs present. I experimented with a few scripts, some of which I spent hours trying to fix. I ended up using a class from 99webtools.com, which uses FOPEN. I made several modifications to this, including forcing it to return to the user the specific whois server it was using, so they are aware of where the information is coming from. My own script then simply passes the $domain variable to the whois script.

You’ll also need to ensure you update your script as new TLDs are released. For example, I had to add the whois server for .io domains, as this was missing from the script on 99webtools.

Two other things I learnt whilst playing around with whois in PHP is to use the <pre> tag in the HTML to return the data in the exact format it is sent by the whois server (as the servers return pre-formatted text) and I found a neat CSS fix to prevent the text leaping outside of the tables I’d built. The other thing I found is that the whois server for .com domains – unlike any other – performs a wildcard search by default. So when you search for a popular domain such as google.com, many irrelevant results are returned, as they happen to contain the ‘google.com’ string inside them somewhere. Most of the scripts I found were vulnerable to this bug, but it is easy enough to fix using an if statement to prefix the word domain to the search string, where it is being sent to a .com whois server (eg: “domain google.com”).

Here’s my completed DNS tool searching for this domain:

dns-tool

I hope this has proved a useful introduction to performing your own DNS and Whois look-ups with PHP. The code I’ve written above can be used as a useful start on building your own DNS tool which you can then customize to your hearts content. If you have any questions, please comment below.

You can test out my own DNS tool here: http://cpanel.jonathandavis.me.uk/dns