Mitigating The Attack
While Dyn probably won’t tell us everything about what they did to defend themselves from this attack, there are a number of mitigation methods they likely employed. The foremost way to defend against a bandwidth based DDoS is to have more bandwidth available than the attackers can marshal to attack with. While new IoT botnets like Mirai are making this difficult by combining small streams from millions of IoT devices like webcams and DVS into attacks reaching huge terabit levels, there are still methods that can help offset them. When you’re a high bandwidth or performance network like Dyn, you can operate multiple data centers, each with multiple methods of connectivity. If your network applications allow, you can further this by using a routing technique know as Anycast: advertising the same ip address range from multiple locations and using local equipment at each location to service that traffic rather than bringing it back to a central server somewhere. This allows you to break up the inbound traffic so each data center only sees a fraction of the total data volume. Another good networking practice involves directly connecting with your customer and provider networks. Typically done by peering with them at public and private interconnection points (IXPs), this also helps break up incoming traffic into more distinct streams. These distinct streams can be easier to police and carry less of the attack traffic, as each peer effectively segments the areas of the internet where traffic comes from. Additionally, they can help ensure you can still talk to services that your systems depend on, keeping your systems from bogging down on constrained access to resources.
Now that you’ve split up the attack and lessened it’s impact at any one location, you still have to deal with the traffic. At this point, you can use a number of techniques including firewalls, scrubbers, proxy servers and services, and just plain server horsepower to handle the traffic. Firewalls can prevent many kinds of traffic from reaching your end servers in the first place, dropping high volume NTP traffic before it gets to your web servers which do not need it. Scrubbers can perform statistical analysis on inbound traffic to weed out fake or highly repeated web requests, preventing them from chewing up resources on your web servers and keeping them from handling legitimate requests. Proxy servers can spread the remaining traffic out over a number of web servers, allowing each to service a bit of the load. More advanced proxying can even prevent any one server from being crushed under high traffic by doing load based traffic distribution or bringing in backup servers during high load events. Basic systems design can provide a good base for this as well. Serving your static web content (like graphics) from a different server pool than the servers that are handling the interactive web elements is already splitting up your traffic, preparing you for further measures. Internet based caching services like CloudFlare can easily extend this concept, allowing you to offset load worldwide.
Designed for Defense
The nature of the Dyn attacks merits elaboration on DNS specific methods, as it points out the importance of considering your application and the technologies involved in any network design. One of the most basic recommendations for DNS configuration is to have 2 or 3 main DNS servers on different physical networks. This prevents an attack on any one network from taking out all of your DNS servers, meaning some of them will still be available to provide answers when queried. DNS also requires the settings of TTLs, or Time To Live settings that allow end users’ name servers to cache results and provide them quickly, even if the authoritative servers are not available. Unfortunately, this can also mean an end user keeps trying the same old server even if you’ve setup a new one. Lower TTLs remedy this, and can also help with geolocation and load balancing, but also increase the load on your DNS servers and shorten the time customers can reach you if your DNS is offline. There’s no hard and fast answer here, as values have to be chosen for your specific needs, but you can make design choices to increase the robustness of even the most complex configuration. These include using load balancing proxy clusters instead of pure DNS load balancing schemes, ensuring you have local DNS servers for your infrastructure instead of relying on remote systems, and even using multiple external DNS providers if your needs are large enough. Consider logically segregating areas of your dns domain space that need specific services so you can use longer TTLs on those that don’t (email servers, say).
These are all part of preparing a resilient infrastructure, but what about reactive measures during an attack? Depending on your setup, your responses may range from bringing additional servers online to diverting incoming traffic to a cloud based scrubbing center all the way up to using Remote Triggered Black Holes (RTBH, a routing technique to tell your upstream ISPs that you don’t want to receive traffic to a particular IP address) to prevent attack traffic from swamping your connections and disrupting other services you provide. While essentially accomplishing the attackers goal, it can be important to allow other services to continue to function. And if you have multiple addresses handling the same site, you may be able to allow some to continue functioning while blocking others. Analyzing the attack in real time can reveal specifics about the nature of the attack that may give you additional ways to fight it. Inbound traffic all udp based when the target is a web server? Work with your internet providers to block inbound UDP to your web server addresses or only allow http & https traffic to them. Most will be able to work with you on this, but it pays to have investigated the communications channels before hand so you know whom to contact.
While I’ve been reviewing general methods that are used in the arena, some of them are beyond the reach or capabilities of smaller businesses. In these cases you are offloading many of these concerns to your bandwidth or colocation provider, so if this is an important concern for you business be sure to discuss the matter with them. Ask them what measures they take to handle traffic floods, if and when they use RTBHs, and if they have any additional services they may provide for scrubbing or other methods of defense. Be aware that more defenses cost more money, so analyze your needs and budget before beginning. It’s important to plan up front and make what preparations you can. Once you’re under attack, your options are much more limited! Allow me to also suggest you ask if your provider uses industry standard best practices like BCP38 and 84. These help ensure that your provider doesn’t perpetuate some of more common DDoS attack types, ensuring you are not inadvertently cut off of the internet because someone else at your chosen datacenter was participating in an attack.
As you can see, there are a lot of options here, and it’s important to consider them while planning any new service. Utilize your network or colocation vendor where possible to help prepare for the event before it happens, and you’ll have taken important first steps to handing any attacks that may come your way.