Scaling PHP Applications (2014)

DNS: Why you need to care

The first layer that we are going to unravel is DNS. DNS? What!? I thought this was a book on PHP? DNS is one of those things that we don’t really think about until it’s too late, because when it fails, it fails in the worst ways.

Don’t believe me? In 2009 Twitter’s DNS was hijacked and redirected users to a hacker’s website for an hour. That same year, SoftLayer’s DNS system was hit with a massive DDoS attack and took down their DNS servers for more than six-hours. As a big SoftLayer customer, we dealt with this firsthand because (at the time) we also used their DNS servers.

The problem with DNS downtime is that it provides the worst user experience possible—users receive a generic error, page timeouts, and have no way to contact you. It’s as if you don’t exist anymore, and most users won’t understand (or likely care) why.

As recently as September 2012, GoDaddy’s DNS servers were attacked and became unreachable for over 24-hours. The worst part? Their site was down too, so you couldn’t move your DNS servers until their website came back up (24-hours later).

Too many companies are using their domain registrar or hosting provider’s DNS configuration—that is WRONG! Want to know how many of the top 1000 sites use GoDaddy’s DNS? None of them.

So, should I run my own DNS server?

It’s certainly an option, but I don’t recommend it. Hosting DNS “the right way” involves having many geographically dispersed servers and an Anycast network. DDoS attacks on DNS are extremely easy to launch and if you half-ass it, your DNS servers will be the Achilles’ heel to your infrastructure. You could have the best NoSQL database in the world but it won’t matter if people can’t resolve your domain.

Almost all of the largest websites use an external DNS provider. That speaks volumes as far as I’m concerned—the easiest way to learn how to do something is to imitate those that are successful.

·        Reddit: Akamai

·        Twitter: Dynect

·        Amazon: UltraDNS and Dynect

·        LinkedIn: UltraDNS

You should plan to pay for a well known, robust DNS SaaS. At Twitpic, we use Dynect Managed DNS. It has paid for itself—we’ve had no downtime related to DNS outages since switching. Make sure you choose a DNS provider that has a presence close to your target audience, too, especially if you have a large international userbase.

Here’s what you should look for in a DNS provider:

·        Geographically dispersed servers

·        Anycast Network

·        Large IP Space (to handle network failures)

question

What is Anycast?

Anycast is a networking technique that allows multiple routers to advertise the same IP prefix—it routes your clients to the “closest” and “best” servers for their location. Think of it as load balancing at the network level.

In addition to the providers listed above, Amazon Route53 is a popular option that is incredibly cheap, offers a 100% SLA, and has a very user-friendly web interface. Since it’s pay-as-you-go, you can easily get started and adjust for exact usage needs as you grow.

DNS Load Distribution

Besides scaling DNS, we can use DNS to help us scale, too. The HAProxy load balancer is integral to our scalable stack—but what happens when it loses network connection, becomes overloaded, or just plain crashes? Well it becomes a single point of failure, which equals downtime—it’s a question of when, not if it will happen.

Traditional DNS load balancing involves creating multiple A records for a single host and passing all of them back to the client, letting the client decide which IP address to use. It looks something like this.

 1 > dig A example.com      

 2 ; <<>> DiG 9.7.3-P3 <<>> A example.com

 3

 4 ;; QUESTION SECTION:

 5 ;example.com.                      IN      A

 6

 7 ;; ANSWER SECTION:

 8 example.com.        287     IN      A       208.0.113.36

 9 example.com.        287     IN      A       208.0.113.34

10 example.com.        287     IN      A       208.0.113.38

11 example.com.        287     IN      A       208.0.113.37

12 example.com.        287     IN      A       208.0.113.35

There are a few drawbacks to this, however. First of all, less-intelligent DNS clients will always use the first IP address presented, no matter what. Some DNS providers (Dynect and Route53, for example) overcome this by using a round-robin approach whereby they change the order of the IPs returned everytime the record is requested, helping to distribute the load in a more linear fashion.

Another drawback is that round-robin won’t prevent against failure, it simply mitigates it. If one of your servers crashes, the unresponsive server’s IP is still in the DNS response. There are two solutions that work together to solve this.

1.    Use a smart DNS provider that can perform health checks. Dynect offers this feature and it’s possible to implement it yourself on Route53 using their API. If a load balancer stops responding or becomes unhealthy, it gets removed from the IP pool (and readded once it becomes healthy again).

2.    Even if you remove a server’s IP from the DNS pool, users that have the bad IP cached will still experience downtime until the record’s TTL expires. Anywhere from 60-300s is a recommended TTL value, which is acceptable, but nowhere near ideal. We’ll talk about how servers can “steal” IPs from unhealthy peers using keepalived in Chapter 4.

Dynect has a very intuitive interface for load balancing and performing health checks on your hosts:

DNS Resolution

The last, and very often overlooked, part of scaling DNS is internal domain resolution. An example of this is when a PHP application calls an external API and has to resolve the domain of the API host. Let’s say the application is using the Twitter API—every time you post something to Twitter, PHP has to look up and determine the IP of api.twitter.com.

PHP accomplishes this by using the libc system call gethostbyname(), which uses nameservers set in /etc/resolv.conf to look up the IP address. Usually this is going to be set to a public DNS resolver (like 8.8.8.8) or a local resolver hosted by your datacenter.

So, what’s wrong with this setup? Well, two things:

1.    It’s another server that you don’t control. What happens when it’s down? Slow? They block you? The lowest timeout allowed in /etc/resolv.conf is one second (and the default is FIVE!), which is too slow for a high-volume website and can cause domino-effect failure at scale.

2.    Most Linux distributions don’t provide a DNS cache by default and that adds extra network latency to every single DNS lookup that your application has to make.

The solution is to run a DNS cache daemon like nscd, dnsmasq, or bind. I won’t cover BIND because it’s overkill to run it simply as a cache, but I will talk about nscd and dnsmasq, which work in slightly different ways.

nscd

nscd (nameserver cache daemon) is the simplest solution to setting up your own internal DNS cache. Whether or not it’s installed by default is dependent on your Linux distro (it’s not on Ubuntu or Debian). It’s easy to install and needs zero-configuration. The main difference between nscdand dnsmasq is that nscd runs locally on each system while dnsmasq is exposed as a network service and can provide a shared DNS cache for multiple servers.

1 > apt-get install nscd

2 > service nscd start

Pros

·        Extremely easy to setup

·        Zero configuration

Cons

·        Runs locally only, so each server needs to have it’s own install

dnsmasq

dnsmasq is a lightweight DNS cache server that provides nearly the same feature-set as nscd, but as a network service.

You setup dnsmasq on a server, let’s call it 198.51.100.10, and set 198.51.100.10 as the nameserver for all of your other servers. dnsmasq will still go out onto the internet to look up DNS queries for the first time, but it will cache the result in memory for subsequent requests, speeding up DNS resolution and allowing you to gracefully deal with failure.

Additionally, you can use dnsmasq as a lightweight internal DNS server with the addn-hosts configuration option, allowing you to use local hostnames without having to hardcode IP addresses in your code (i.e, $memcache->connect('cache01.example') instead of $memcache->connect('198.51.100.15')).

Assuming a network setup based on the table below, here is how we’d setup dnsmasq and point our servers to it:

Hosts

IP

dnsmasq server

192.51.100.10

server01.example

192.51.100.16

server02.example

192.51.100.17

On your dnsmasq server:

 1 > apt-get install dnsmasq

 2 > vi /etc/dnsmasq.conf

 3

 4 cache-size=1000

 5 listen-address=198.51.100.10

 6 local-ttl=60

 7 no-dhcp-interface=eth0

 8 no-dhcp-interface=eth1

 9 addn-hosts=/etc/dnsmasq.hosts

10

11 > vi /etc/dnsmasq.hosts

12

13 198.51.100.16 server01.example

14 198.51.100.17 server02.example

15

16 > service dnsmasq restart

local-ttl sets the time-to-live for any hosts you define in /etc/hosts or /etc/dnsmasq.hosts

cache-size defines the size of the DNS cache.

no-dhcp-interface disables all services provided by dnsmasq except for dns. Without this, dnsmasq will provide dhcp and tftp as well, which you do not want in most scenarios.

On EC2, after restarting dnsmasq you may need to add the following line to /etc/hosts:

1 127.0.0.1 ip-10-x-x-x

And then on your other hosts:

1 > vi /etc/resolv.conf

2

3 nameserver 198.51.100.10

4 options rotate timeout:1

The rotate option tells linux to rotate through the nameservers instead of always using the first one. This is useful if you have more than one nameserver and want to distribute the load.

The timeout:1 option tells linux that it should try the next nameserver if it takes longer than 1-second to respond. You can set it to any integer between 1 and 30. The default is 5-seconds and it’s capped at 30-seconds. Unfortunately, the minimum value is 1-second, it would be beneficial to set the timeout in milliseconds.

When do I need this?

There is virtually no negative impact to implementing a caching nameserver early; however, it does add another service to monitor—it’s not a completely free optimization. Essentially, if any of your pages require a DNS resolution, you should consider implementing a DNS cache early.

Want to know how many uncached DNS requests your server is currently sending? With tcpdump you can see all of the DNS requests going out over the network.

1 > apt-get install tcpdump

2 > tcpdump -nnp dst port 53

3

4 09:52 IP 198.51.100.16 > 198.51.100.10.53: A? graph.facebook.com.

5 09:52 IP 198.51.100.16 > 198.51.100.10.53: AAAA? graph.facebook.com.

6 09:52 IP 198.51.100.16 > 198.51.100.10.53: A? api.twitter.com.

7 09:52 IP 198.51.100.16 > 198.51.100.10.53: AAAA? api.twitter.com.

The -nn option ensures tcpdump itself does not resolve IP Addresses or protocols to names.

The -p option disables promiscuous mode, to minimize adverse impact to running services.

The dst port 53 only shows DNS requests sent (for brevity), to see how long the application may have blocked waiting for a response (i.e. to see the request along with the response), exclude the ‘dst’ portion.

The above example shows how a single HTTP GET could cause four DNS requests to be sent to 3rd party APIs that you may be using. Using a caching DNS resolver, such as dnsmasq or nscd, helps reduce the blocking period and possible cascading failures.