Blogroll: CloudFlare

I read blogs, as well as write one. The 'blogroll' on this site reproduces some posts from some of the people I enjoy reading. There are currently 32 posts from the blog 'CloudFlare.'

Disclaimer: Reproducing an article here need not necessarily imply agreement or endorsement!

Subscribe to CloudFlare feed
Cloudflare Blog
Updated: 57 min 16 sec ago

Less Is More - Why The IPv6 Switch Is Missing

Thu, 25/05/2017 - 18:30

At Cloudflare we believe in being good to the Internet and good to our customers. By moving on from the legacy world of IPv4-only to the modern-day world where IPv4 and IPv6 are treated equally, we believe we are doing exactly that.

"No matter what happens in life, be good to people. Being good to people is a wonderful legacy to leave behind." - Taylor Swift (whose website has been IPv6 enabled for many many years)

Starting today with free domains, IPv6 is no longer something you can toggle on and off, it’s always just on.

How we got here

Cloudflare has always been a gateway for visitors on IPv6 connections to access sites and applications hosted on legacy IPv4-only infrastructure. Connections to Cloudflare are terminated on either IP version and then proxied to the backend over whichever IP version the backend infrastructure can accept.

That means that a v6-only mobile phone (looking at you, T-Mobile users) can establish a clean path to any site or mobile app behind Cloudflare instead of doing an expensive 464XLAT protocol translation as part of the connection (shaving milliseconds and conserving very precious battery life).

That IPv6 gateway is set by a simple toggle that for a while now has been default-on. And to make up for the time lost before the toggle was default on, in August 2016 we went back retroactively and enabled IPv6 for those millions of domains that joined before IPv6 was the default. Over those next few months, we enabled IPv6 for nearly four million domains –– you can see Cloudflare’s dent in the IPv6 universe below –– and by the time we were done, 98.1% of all of our domains had IPv6 connectivity.

As an interim step, we added an extra feature –– when you turn off IPv6 in our dashboard, we remind you just how archaic we think that is.

With close to 100% IPv6 enablement, it no longer makes sense to offer an IPv6 toggle. Instead, Cloudflare is offering IPv6 always on, with no off-switch. We’re starting with free domains, and over time we’ll change the toggle on the rest of Cloudflare paid-plan domains.

The Future: How Cloudflare and OpenDNS are working together to make IPv6 even faster and more globally deployed

In November we published stats about the IPv6 usage we see on the Cloudflare network in an attempt to answer who and what is pushing IPv6. The top operating systems by percent IPv6 traffic are iOS, ChromeOS, and MacOS respectively. These operating systems push significantly more IPv6 traffic than their peers because they use a routing choice algorithm called Happy Eyeballs. Happy Eyeballs opportunistically chooses IPv6 when available by doing two DNS lookups –– one for an IPv6 address (this IPv6 address is stored in the DNS AAAA record - pronounced quad-A) and then one for the IPv4 address (stored in the DNS A record). Both DNS queries are flying over the Internet at the same time and the client chooses the address that comes back first. The client even gives IPv6 a few milliseconds head start (iOS and MacOS give IPv6 lookups a 25ms head start for example) so that IPv6 may be chosen more often. This works and has fueled some of IPv6’s growth. But it has fallen short of the goal of a 100% IPv6 world.

While there are perfectly good historical reasons why IPv6 and IPv4 addresses are stored in separate DNS types, today clients are IP version agnostic and it no longer makes sense for it to require two separate round trips to learn what addresses are available to fetch a resource from.

Alongside OpenDNS, we are testing a new idea - what if you could ask for all the addresses in just one DNS query?

With OpenDNS, we are prototyping and testing just that –– a new DNS metatype that returns all available addresses in one DNS answer –– A records and AAAA records in one response. (A metatype is a query type in DNS that end users can’t add into their DNS zone file, it’s assembled dynamically by the authoritative nameserver.)

What this means is that in the future if a client like an iPhone wants to access a mobile app that uses Cloudflare DNS or using another DNS provider that supports the spec, the iPhone DNS client would only need to do one DNS lookup to find where the app’s API server is located, cutting the number of necessary round trips in half.

This reduces the amount of bandwidth on the DNS system, and pre-populates global DNS caches with IPv6 addresses, making IPv6 lookups faster in the future, with the side benefit that Happy Eyeballs clients prefer IPv6 when they can get the address quickly, which increases the amount of IPv6 traffic that flows through the Internet.

We have the metaquery working in code with the reserved TYPE65535 querytype. You can ask a Cloudflare nameserver for TYPE65535 of any domain on Cloudflare and get back all available addresses for that name.

$ dig cloudflare.com @ns1.cloudflare.com -t TYPE65535 +short 198.41.215.162 198.41.214.162 2400:cb00:2048:1::c629:d6a2 2400:cb00:2048:1::c629:d7a2 $

Did we mention Taylor Swift earlier?

$ dig taylorswift.com @ns1.cloudflare.com -t TYPE65535 +short 104.16.193.61 104.16.194.61 104.16.191.61 104.16.192.61 104.16.195.61 2400:cb00:2048:1::6810:c33d 2400:cb00:2048:1::6810:c13d 2400:cb00:2048:1::6810:bf3d 2400:cb00:2048:1::6810:c23d 2400:cb00:2048:1::6810:c03d $

We believe in proving concepts in code and through the IETF standards process. We’re currently working on an experiment with OpenDNS and will translate our learnings to an Internet Draft we will submit to the IETF to become an RFC. We’re sure this is just the beginning to faster, better deployed IPv6.

Categories: Technology

Patent Troll Battle Update: Doubling Down on Project Jengo

Thu, 25/05/2017 - 17:00

Project Jengo Doubles In Size
Jengo Fett by Brickset (Flickr)

We knew the case against patent trolls was the right one, but we have been overwhelmed by the response to our blog posts on patent trolls and our program for finding prior art on the patents held by Blackbird Tech, which we’ve dubbed Project Jengo. As we discuss in this post, your comments and contributions have allowed us to expand and intensify our efforts to challenge the growing threat that patent trolls pose to innovative tech companies.

We’re SIGNIFICANTLY expanding our program to find prior art on the Blackbird Tech patents

In a little over a week since we started the program, we’ve received 141 separate prior art submissions. But we know there’s an opportunity to find a lot more.

We’ve been impressed with the exceptionally high quality of the submissions. The Cloudflare community of users and readers of our blog are an accomplished bunch, so we have a number of searches that were done by expert engineers and programmers. In one case that stood out to us, someone wrote in about a project they personally had worked on as an engineer back in 1993, which they are convinced is conclusive prior art to a Blackbird Tech patent. We will continue to collect and review these submissions.

The submissions so far relate to 18 of the 38 Blackbird Tech patents and applications. You can see a summary of the number of submissions per patent here (PDF). You'll see there are still 20 Blackbird Tech patents and applications we’ve yet to receive a submission for.

We’re looking for prior art on 100% of the Blackbird Tech patents. If you are interested in helping, take some time to look into those patents where we don’t have anything yet. We’ll update the chart as we review the submissions with additional information about the number we receive, and their quality, to help focus the search. After the initial review, we’ll start to color code the patents (i.e., red/yellow/green) to demonstrate the number and quality of submissions we’ve received on each patent.

An anonymous benefactor donated another $50K to help invalidate all of Blackbird Tech's patents

And our efforts to cover the field have been re-doubled. We’re excited to report that a friend in the industry who read our blog post and shares our concerns about the corrosive impact of patent trolls has made an anonymous donation of $50,000 to support our efforts to invalidate the Blackbird Tech patents. That means that we are now committing at least $100,000 to the effort to find prior art on and initiate actions to invalidate the Blackbird Tech patents.

We initially dedicated a $50,000 bounty to invalidate Blackbird Tech's patents. We split the bounty so $20,000 was to invalidate the particular patent Blackbird Tech sued us on and $30,000 was to help invalidate any other Blackbird Tech patent. We've received so many prior art submissions on the patent in question in Cloudflare's case that we don't believe we need an additional incentive there. Instead, we're dedicating 100% of the anonymously donated $50,000 to invalidating the other Blackbird Tech patents. This will be used both to boost the bounty we pay to researchers as well as to fund invalidation cases we file with the USPTO. Our goal remains invalidating every one of Blackbird Tech's patents. Again if you want more information about how you can participate, you can find the description here.

And, of course, there will be t-shirts!

Troll Hunter Shirt Design

And it wouldn’t be a cooperative effort in the tech community if we didn’t give out T-shirts to commemorate your participation in the process. You can see the T-shirt design above, all you have to do is provide a legitimate entry of prior art on any of the Blackbird Tech patents and we’ll send one to you (limit one shirt per participant).

Blackbird Tech’s “new model” of patent litigation may be a violation of professional ethics, soon it may also be an explicit violation of law

We think the business operations of the Blackbird Tech attorneys may violate the Rules of Professional Conduct in both Illinois and Massachusetts, where Blackbird Tech’s offices are located and where its co-founders work, and we have asked ethics regulators in those states to undertake a review. But we think it’s worth going a step further and working with innovation-supporting legislators in the states where Blackbird Tech operates to make it absolutely clear this new breed of patent troll is not welcome.

As we mentioned in the original blog post, there have already been several proposals at both the state and federal level to push back and limit the ability of patent trolls to use the courts to bring cases against successful companies. Yet Blackbird Tech is pushing in the other direction and attempting to come up with novel ways to increase the efficiency and effectiveness of patent trolls.

On May 23, 2017, Rep. Keith Wheeler of Illinois introduced a bill (the “Ethics in Patent Litigation Act”) that would make it the public policy of the State of Illinois that attorneys in the state, like Blackbird co-founder Chris Freeman (LinkedIn), should not be able to buy patents themselves for the purpose of suing on them if they are not in the business of any other productive activity. We appreciate Rep. Wheeler’s support of innovation and his stance against patent trolls, feel free to show your support via Twitter below.

In Massachusetts, where Blackbird's other attorney co-founder, Wendy Wendy Verlander (@bbirdtech_CEO; LinkedIn) is based, Sen. Eric Lesser has specifically targeted patent trolls in a bill he introduced earlier this year.

Well done. It's time to stand up to patent trolls. We have a bill in @MA_Senate that will do just that. @ScottKirsner @jonchesto @epaley https://t.co/O2hHB1R3DT

— Eric Lesser (@EricLesser) May 19, 2017

You can show your support for Sen. Lesser’s stance on these issues using the Twitter generator below. We will be working with Sen. Lesser in the weeks and months ahead to address our concern about Blackbird Tech’s “new model” of patent troll.

Even though the patent system may be based on Federal law, states have the ability to set rules for how businesses, and especially lawyers, behave in their jurisdictions. So we’re happy to work with interested lawmakers in other states, including Delaware, to advance new laws that limit the practices of patent trolls, including Blackbird Tech’s “new model.” We can share the information we’ve learned and pull together model legislation. If you are interested or know a legislator who may be, feel free to email us.

Blackbird Tech calls themselves “very much the same” as and “almost identical” to a law firm when it suits their purposes, and “not a law firm” when it doesn’t

As we wrote before, we believe Blackbird Tech's dangerous new model of patent trolling — where they buy patents and then act their own attorneys in cases — may be a violation of the rules of professional ethics. In particular, we are concerned that they may be splitting fees with non-attorneys and that they may be acquiring causes of action. Both practices run counter to the rules of professional ethics for lawyers and law firms.

It is increasingly clear to us that Blackbird’s response to questions about their compliance with the rules of professional conduct will be, at best, based on simple agreements that merely create a shortcut around their ethical obligations, and at worst, directly contradictory.

Blackbird Tech wants to have it both ways. In response to the original blog post, Blackbird Tech denied both that it was a law firm and that it used contingency fee agreements. Specifically:

In a phone conversation with Fortune, Blackbird CEO Wendy Verlander said the company is not a law firm and that it doesn't use contingency fee arrangements for the patents it buys, but conceded "it's a similar arrangement."

Ms. Verlander objects to being characterized as a law firm because if Blackbird is found to be one then their practices would be governed by, and may be a violation of, the rules of professional ethics. Ms. Verlander’s denial that Blackbird Tech doesn’t use contingency agreements, only to quickly concede that what they do is “a similar arrangement” suggests again that Blackbird Tech is finding it convenient to work around the ethical rules.

This runs fundamentally counter to the concept of ethical rules, which are meant to be driven by the spirit of those obligations. Anyone out to intentionally “cut corners” or do the “bare minimum” to comply with only the letter of such obligations are by default in violation of the “special responsibilities” which should be driven by “personal conscience” as described in the preamble of the ABA Model Rules.

Wendy Verlander and Chris Freeman, the Founders of the Blackbird Technologies Law Firm

And Ms. Verlander’s unequivocal assertion that Blackbird Tech is not a law firm can be contrasted with sworn statements submitted by Blackbird Tech attorneys to courts last May asserting how much they operate like a law firm. In Blackbird Tech v. Service Lighting and Electrical Supplies, Blackbird Tech CEO Wendy Verlander, Blackbird Tech co-founder Chris Freeman, and Blackbird Tech employee Sean Thompson, each filed declarations in opposition to a proposed protective order.

Protective orders are important in patent litigation. Often, discovery in those cases involves companies handing over highly confidential information about their most important trade secrets or the history of how they developed valuable intellectual property. In most cases, courts limit access to such materials only to outside counsel, as opposed to the parties’ employees and in-house counsel. In-house counsel generally serve a number of functions at a business that include competitive decision-making, either directly or indirectly. Because in-house counsel may benefit from the additional perspective and insight gained by exposure to sensitive trade secrets of a competitor, and are unable to simply wipe their memories clean, courts in patent litigation cases often limit their review of particularly sensitive documents. In such cases, documents classified as “HIGHLY CONFIDENTIAL—ATTORNEY EYES ONLY” are limited to review by outside counsel, who are less likely to face the same sort of business decisions in the future.

When it served their purposes in opposition to a proposed protective order, the Blackbird Tech attorneys were quick to point out how much they operated only like a law firm and distance themselves from their business roles. Their sworn declarations specifically asserted:

  • “Although the structure of Blackbird is unique, the realities of patent litigation at Blackbird are very much the same as patent litigation on behalf of clients at law firms.” (Verlander at ¶13, Freeman at ¶14)

  • “Thus, in many ways, my role at Blackbird as a member of the Litigation Group is identical to my previous role as outside counsel at a law firm.” (Verlander at ¶13, Freeman at ¶14)(emphasis added)

  • “Blackbird’s Litigation Group operates almost identically to outside law firm counsel. Blackbird’s litigators are presented with patents and possible infringers, just as clients bring to law firms. The Blackbird litigators then bring their litigation expertise to bear and thoroughly analyze the patent and the potential infringement case, ultimately deciding whether to move forward with litigation — just as a law firm would evaluate a case. If the Blackbird litigation team identifies a strong infringement case, the litigators draft Complaints and conduct litigation, acting in the same role as outside counsel.” (Verlander at ¶14, Freeman at ¶15)(emphasis added).

  • “On a day-to-day basis, what I do at Blackbird is the same as what I did when practicing at a firm.” (Thompson at ¶2).

This inconsistency points out once again how Blackbird is attempting to gain an advantage by turning traditional roles on their head. If they were a typical company, that was looking to make products using the patents they own, then we’d be able to seek discovery on their products and operations. Instead, they function as a law firm with no business operations that would be subject to the same sort of scrutiny they will apply to a company like Cloudflare.

And they say that they’re not a law firm, yet they expect all their employees, including their CEO, to be permitted to exercise the special role of an attorney “identical to [their] previous role as outside counsel at a law firm.” But it would be difficult for them to deny that their employees, including their CEO, are engaged in impermissible attorney practices like buying causes of actions and giving a financial interest in litigation to non-parties, which are clearly not “identical” to what they would have done “as outside counsel at a firm.” They can’t have it both ways.

Coverage of the blog post took our arguments even further

In our previous blog posts on patent trolls, we thought we’d said about everything there was to say, or at least exhausted anyone who might have something else to say. But we found that most of the reports about our efforts did much more than merely parrot our statements and ask Blackbird Tech for a response. These reports raise some excellent additional points that we expect to use in our ongoing efforts to defend the case brought by Blackbird Tech.

Several of the reporters noted that Blackbird Tech’s claims seem a bit farfetched and found their own factual basis for contesting those claims. Joe Mullin (@joemullin) at Ars Technica noted that the Blackbird Tech patent—particularly in the overbroad way it is applying it in the case against Cloudflare—has prior art that dates back to the beginning of the last century:

The suggestion that intercepting and modifying electronic communications is a 1998 “invention” is a preposterous one. By World War I, numerous state governments had systems in place to censor and edit telegraph and telephone conversations.

Similarly, Shaun Nichols (@shaundnichols) of the Register notes that the differences between the Blackbird Tech patent and our operations are “remarkable”:

In our view, from a quick read of the documentation, Blackbird's design sounds remarkably different to Cloudflare's approach. Critically, the server-side includes described in the patent have been around well before the patent was filed: Apache, for example, had them as early as 1996, meaning the design may be derailed by prior art.

And beyond the legal arguments in the patent case, Techdirt felt that our arguments questioning the operations of Blackbird Tech itself sounded strikingly familiar to another operation that was found to be legally improper:

Righthaven. As you may recall, that was a copyright trolling operation that effectively "bought" the bare right to sue from newspapers. They pretended they bought the copyright (since you can't just buy a right to sue), but the transfer agreement left all the actual power with the newspapers, and courts eventually realized that all Righthaven really obtained was the right to sue. That resulted in the collapse of Righthaven. This isn't exactly analogous, but there are some clear similarities, in having a "company," rather than a law firm (but still run completely by lawyers), "purchase" patents or copyrights solely for the purpose of suing, while setting up arrangements to share the proceeds with the previous holder of those copyrights or patents. It's a pretty sleazy business no matter what — and with Righthaven it proved to be its undoing. Blackbird may face a similar challenge.

It’s probably best to close this post with a statement from Mike Masnick (@mmasnick) of Techdirt that we may save for a closing argument down the road because it summarized the situation better than we had:

Kudos to Cloudflare for hitting back against patent trolling that serves no purpose whatsoever, other than to shake down innovative companies and stifle their services. But, really, the true travesty here is that the company needs to do this at all. Our patent (and copyright) systems seem almost perfectly designed for this kind of shakedown game, having nothing whatsoever to do with the stated purpose of supporting actual innovators and creators. Instead, it's become a paper game abused by lawyers to enrich themselves at the expense of actual innovators and creators.

We will keep you updated. In the meantime, you can contribute to our efforts by continuing to participate in the search for prior art on the Blackbird Tech patents, or you can engage in the political process by supporting efforts to change the patent litigation process. And support folks like Rep. Wheeler or Sen. Lesser with their proposals to limit the power of patent trolls.

twitterwidget { width: 100% !important; } var twitterInterval = setInterval(function(){ var widget = document.querySelector('twitterwidget'); if (!widget) return; var embedded = widget.shadowRoot.querySelector('.EmbeddedTweet'); if (!embedded) return; clearInterval(twitterInterval); embedded.style.maxWidth = 'none'; }, 100);
Categories: Technology

Reflections on reflection (attacks)

Wed, 24/05/2017 - 19:16

Recently Akamai published an article about CLDAP reflection attacks. This got us thinking. We saw attacks from Conectionless LDAP servers back in November 2016 but totally ignored them because our systems were automatically dropping the attack traffic without any impact.

CC BY 2.0 image by RageZ

We decided to take a second look through our logs and share some statistics about reflection attacks we see regularly. In this blog post, I'll describe popular reflection attacks, explain how to defend against them and why Cloudflare and our customers are immune to most of them.

A recipe for reflection

Let's start with a brief reminder on how reflection attacks (often called "amplification attacks") work.

To bake a reflection attack, the villain needs four ingredients:

  • A server capable of performing IP address spoofing.
  • A protocol vulnerable to reflection/amplification. Any badly designed UDP-based request-response protocol will do.
  • A list of "reflectors": servers that support the vulnerable protocol.
  • A victim IP address.

The general idea:

  • The villain sends fake UDP requests.
  • The source IP address in these packets is spoofed: the attacker sticks the victim's IP address in the source IP address field, not their own IP address as they normally would.
  • Each packet is destined to a random reflector server.
  • The spoofed packets traverse the Internet and eventually are delivered to the reflector server.
  • The reflector server receives the fake packet. It looks at it carefully and thinks: "Oh, what a nice request from the victim! I must be polite and respond!". It sends the response in good faith.
  • The response, though, is directed to the victim.

The victim will end up receiving a large volume of response packets it never had requested. With a large enough attack the victim may end up with congested network and an interrupt storm.

The responses delivered to victim might be larger than the spoofed requests (hence amplification). A carefully mounted attack may amplify the villain's traffic. In the past we've documented a 300Gbps attack generated with an estimated 27Gbps of spoofing capacity.

Popular reflections

During the last six months our DDoS mitigation system "Gatebot" detected 6,329 simple reflection attacks (that's one every 40 minutes). Here is the list by popularity of different attack vectors. An attack is defined as a large flood of packets identified by a tuple: (Protocol, Source Port, Target IP). Basically - a flood of packets with the same source port to a single target. This notation is pretty accurate - during normal Cloudflare operation, incoming packets rarely share a source port number!

Count Proto Src port 3774 udp 123 NTP 1692 udp 1900 SSDP 438 udp 0 IP fragmentation 253 udp 53 DNS 42 udp 27015 SRCDS 20 udp 19 Chargen 19 udp 20800 Call Of Duty 16 udp 161 SNMP 12 udp 389 CLDAP 11 udp 111 Sunrpc 10 udp 137 Netbios 6 tcp 80 HTTP 5 udp 27005 SRCDS 2 udp 520 RIP Source port 123/udp NTP

By far the most popular reflection attack vector remains NTP. We have blogged about NTP in the past:

Over the last six months we've seen 3,374 unique NTP amplification attacks. Most of them were short. The average attack duration was 11 minutes, with the longest lasting 22 hours (1,300 minutes). Here's a histogram showing the distribution of NTP attack duration:

Minutes min:1.00 avg:10.51 max:1297.00 dev:35.02 count:3774 Minutes: value |-------------------------------------------------- count 0 | 2 1 | * 53 2 | ************************* 942 4 |************************************************** 1848 8 | *************** 580 16 | ***** 221 32 | * 72 64 | 35 128 | 11 256 | 7 512 | 2 1024 | 1

Most of the attacks used a small number of reflectors - we've recorded an average of 1.5k unique IPs per attack. The largest attack used an estimated 12.3k reflector servers.

Unique IPs min:5.00 avg:1552.84 max:12338.00 dev:1416.03 count:3774 Unique IPs: value |-------------------------------------------------- count 16 | 0 32 | 1 64 | 8 128 | ***** 111 256 | ************************* 553 512 | ************************************************* 1084 1024 |************************************************** 1093 2048 | ******************************* 685 4096 | ********** 220 8192 | 13

The peak attack bandwidth was on average 5.76Gbps and max of 64Gbps:

Peak bandwidth in Gbps min:0.06 avg:5.76 max:64.41 dev:6.39 count:3774 Peak bandwidth in Gbps: value |-------------------------------------------------- count 0 | ****** 187 1 | ********************* 603 2 |************************************************** 1388 4 | ***************************** 818 8 | ****************** 526 16 | ******* 212 32 | * 39 64 | 1

This stacked chart shows the geographical distribution of the largest NTP attack we've seen in the last six months. You can see the packets per second number directed to each datacenter. One our datacenters (San Jose to be precise) received about a third of the total attack volume, while the remaining packets were distributed roughly evenly across other datacenters.

The attack lasted 20 minutes, used 527 reflector NTP servers and generated about 20Mpps / 64Gbps at peak.

Dividing these numbers we can estimate that a single packet in that attack had on average size of 400 bytes. In fact, in NTP attacks the great majority of packets have a length of precisely 468 bytes (less often 516). Here's a snippet from tcpdump:

$ tcpdump -n -r 3164b6fac836774c.pcap -v -c 5 -K 11:38:06.075262 IP -(tos 0x20, ttl 60, id 0, offset 0, proto UDP (17), length 468) 216.152.174.70.123 > x.x.x.x.47787: [|ntp] 11:38:06.077141 IP -(tos 0x0, ttl 56, id 0, offset 0, proto UDP (17), length 468) 190.151.163.1.123 > x.x.x.x.44540: [|ntp] 11:38:06.082631 IP -(tos 0xc0, ttl 60, id 0, offset 0, proto UDP (17), length 468) 69.57.241.60.123 > x.x.x.x.47787: [|ntp] 11:38:06.095971 IP -(tos 0x0, ttl 60, id 0, offset 0, proto UDP (17), length 468) 126.219.94.77.123 > x.x.x.x.21784: [|ntp] 11:38:06.113935 IP -(tos 0x0, ttl 59, id 0, offset 0, proto UDP (17), length 516) 69.57.241.60.123 > x.x.x.x.9285: [|ntp] Source port 1900/udp SSDP

The second most popular reflection attack was SSDP, with a count of 1,692 unique events. These attacks were using much larger fleets of reflector servers. On average we've seen around 100k reflectors used in each attack, with the largest attack using 1.23M reflector IPs. Here's the histogram of number of unique IPs used in SSDP attacks:

Unique IPs min:15.00 avg:98272.02 max:1234617.00 dev:162699.90 count:1691 Unique IPs: value |-------------------------------------------------- count 256 | 0 512 | 4 1024 | **************** 98 2048 | ************************ 152 4096 | ***************************** 178 8192 | ************************* 158 16384 | **************************** 176 32768 | *************************************** 243 65536 |************************************************** 306 131072 | ************************************ 225 262144 | *************** 95 524288 | ******* 47 1048576 | * 7

The attacks were also longer, with 24 minutes average duration:

$ cat 1900-minutes| ~/bin/mmhistogram -t "Minutes" Minutes min:2.00 avg:23.69 max:1139.00 dev:57.65 count:1692 Minutes: value |-------------------------------------------------- count 0 | 0 1 | 10 2 | ***************** 188 4 | ******************************** 354 8 |************************************************** 544 16 | ******************************* 342 32 | *************** 168 64 | **** 48 128 | * 19 256 | * 16 512 | 1 1024 | 2

Interestingly the bandwidth doesn't follow a normal distribution. The average SSDP attack was 12Gbps and the largest just shy of 80Gbps:

$ cat 1900-Gbps| ~/bin/mmhistogram -t "Bandwidth in Gbps" Bandwidth in Gbps min:0.41 avg:11.95 max:78.03 dev:13.32 count:1692 Bandwidth in Gbps: value |-------------------------------------------------- count 0 | ******************************* 331 1 | ********************* 232 2 | ********************** 235 4 | *************** 165 8 | ****** 65 16 |************************************************** 533 32 | *********** 118 64 | * 13

Let's take a closer look at the largest (80Gbps) attack we've recorded. Here's a stacked chart showing packets per second going to each datacenter. This attack was using 940k reflector IPs, generated 30Mpps. The datacenters receiving the largest proportion of the traffic were San Jose, Los Angeles and Moscow.

The average packet size was 300 bytes. Here's how the attack looked on the wire:

$ tcpdump -n -r 4ca985a2211f8c88.pcap -K -c 7 10:24:34.030339 IP - 219.121.108.27.1900 > x.x.x.x.25255: UDP, length 301 10:24:34.406943 IP - 208.102.119.37.1900 > x.x.x.x.37081: UDP, length 331 10:24:34.454707 IP - 82.190.96.126.1900 > x.x.x.x.25255: UDP, length 299 10:24:34.460455 IP - 77.49.122.27.1900 > x.x.x.x.25255: UDP, length 289 10:24:34.491559 IP - 212.171.247.139.1900 > x.x.x.x.25255: UDP, length 323 10:24:34.494385 IP - 111.1.86.109.1900 > x.x.x.x.37081: UDP, length 320 10:24:34.495474 IP - 112.2.47.110.1900 > x.x.x.x.37081: UDP, length 288 Source port 0/udp IP fragmentation

Sometimes we see reflection attacks showing UDP source and destination port numbers set to zero. This is usually a side effect of attacks where the reflecting servers responded with large fragmented packets. Only the first IP fragment contains a UDP header, preventing subsequent fragments from being reported properly. From a router point of view this looks like a UDP packet without UDP header. A confused router reports a packet from source port 0, going to port 0!

This is a tcpdump-like view:

$ tcpdump -n -r 4651d0ec9e6fdc8e.pcap -c 8 02:05:03.408800 IP - 190.88.35.82.0 > x.x.x.x.0: UDP, length 1167 02:05:03.522186 IP - 95.111.126.202.0 > x.x.x.x.0: UDP, length 1448 02:05:03.525476 IP - 78.90.250.3.0 > x.x.x.x.0: UDP, length 839 02:05:03.550516 IP - 203.247.133.133.0 > x.x.x.x.0: UDP, length 1472 02:05:03.571970 IP - 54.158.14.127.0 > x.x.x.x.0: UDP, length 1328 02:05:03.734834 IP - 1.21.56.71.0 > x.x.x.x.0: UDP, length 1250 02:05:03.745220 IP - 195.4.131.174.0 > x.x.x.x.0: UDP, length 1472 02:05:03.766862 IP - 157.7.137.101.0 > x.x.x.x.0: UDP, length 1122

An avid reader will notice - the source IPs above are open DNS resolvers! Indeed, from our experience most of the attacks categorized as fragmentation are actually a side effect of DNS amplifications.

Source port 53/udp DNS

Over the last six months we've seen 253 DNS amplifications. On average an attack used 7100 DNS reflector servers and lasted 24 minutes. Average bandwidth was around 3.4Gbps with largest attack using 12Gbps.

This is a simplification though. As mentioned above multiple DNS attacks were registered by our systems as two distinct vectors. One was categorized as source port 53, and another as source port 0. This happened when the DNS server flooded us with DNS responses larger than max packet size, usually about 1,460 bytes. It's easy to see if that was the case by inspecting the DNS attack packet lengths. Here's an example:

DNS attack packet lengths min:44.00 avg:1458.94 max:1500.00 dev:208.14 count:40000 DNS attack packet lengths: value |-------------------------------------------------- count 8 | 0 16 | 0 32 | 129 64 | 479 128 | 84 256 | 164 512 | 268 1024 |************************************************** 38876

The great majority of the received DNS packets were indeed close to the max packet size. This suggests the DNS responses were large and were split into multiple fragmented packets. Let's see the packet size distribution for accompanying source port 0 attack:

$ tcpdump -n -r 4651d0ec9e6fdc8e.pcap \ | grep length \ | sed -s 's#.*length \([0-9]\+\).*#\1#g' \ | ~/bin/mmhistogram -t "Port 0 packet length" -l -b 100 Port 0 packet length min:0.00 avg:1264.81 max:1472.00 dev:228.08 count:40000 Port 0 packet length: value |-------------------------------------------------- count 0 | 348 100 | 7 200 | 17 300 | 11 400 | 17 500 | 56 600 | 3 700 | ** 919 800 | * 520 900 | * 400 1000 | ******** 3083 1100 | ************************************ 12986 1200 | ***** 1791 1300 | ***** 2057 1400 |************************************************** 17785

About half of the fragments were large, close to the max packet length in size, and rest were just shy of 1,200 bytes. This makes sense: a typical max DNS response is capped at 4,096 bytes. 4,096 bytes would be seen on the wire as one DNS packet fragment with an IP header, one max length packet fragment and one fragment of around 1,100 bytes:

4,096 = 1,460+1,460+1,060

For the record, the particular attack illustrated here used about 17k reflector server IPs, lasted 64 minutes, generated about 6Gbps on the source port 53 strand and 11Gbps of source port 0 fragments.

We have blogged about DNS reflection attacks in the past:

Other protocols

We've seen amplification using other protocols such as:

  • port 19 - Chargen
  • port 27015 - SRCDS
  • port 20800 - Call Of Duty

...and many other obscure protocols. These attacks were usually small and not notable. We didn't see enough of then to provide meaningful statistics but the attacks were automatically mitigated.

Poor observability

Unfortunately we're not able to report on the contents of the attack traffic. This is notable for the NTP and DNS amplifications - without case by case investigations we can't report what responses were actually being delivered to us.

This is because all these attacks stopped at the network layer. Routers are heavily optimized to perform packet forwarding and have a limited capacity of extracting raw packets. Basically there is no "tcpdump" there.

We track these attacks with netflow, and we observe them hit our routers firewall. The tcpdump snippets shown above were actually fake, reconstructed artificially from netflow data.

Trivial to mitigate

With properly configured firewall and sufficient network capacity (which isn't always easy to come by unless you are the size of Cloudflare) it's trivial to block the reflection attacks. But note that we've seen reflection attacks up to 80Gbps so you do need sufficient capacity.

Properly configuring a firewall is not rocket science: default DROP can get you quite far. In other cases you might want to configure rate limiting rules. This is a snippet from our JunOS config:

term RATELIMIT-SSDP-UPNP { from { destination-prefix-list { ANYCAST; } next-header udp; source-port 1900; } then { policer SA-POLICER; count ACCEPT-SSDP-UPNP; next term; } }

But properly configuring firewall requires some Internet hygiene. You should avoid using the same IP for inbound and outbound traffic. For example, filtering a potential NTP DDoS will be harder if you can't just block inbound port 123 indiscriminately. If your server requires NTP, make sure it exits to the Internet over non-server IP address!

Capacity game

While having sufficient network capacity is necessary, you don't need to be a Tier 1 to survive amplification DDoS. The median attack size we've received was just 3.35Gbps, average 7Gbps, Only 195 attacks out of 6,353 attacks recorded - 3% - were larger than 30Gbps.

All attacks in Gbps: min:0.04 avg:7.07 med:3.35 max:78.03 dev:9.06 count:6329 All attacks in Gbps: value |-------------------------------------------------- count 0 | **************** 658 1 | ************************* 1012 2 |************************************************** 1947 4 | ****************************** 1176 8 | **************** 641 16 | ******************* 748 32 | **** 157 64 | 14

But not all Cloudflare datacenters have equal sized network connections to the Internet. So how can we manage?

Cloudflare was architected to withstand large attacks. We are able to spread the traffic on two layers:

  • Our public network uses Anycast. For certain attack types - like amplification - this allows us to split the attack across multiple datacenters avoiding a single choke point.
  • Additionally we use ECMP internally to spread a traffic destined to single IP address across multiple physical servers.

In the examples above, I showed a couple of amplification attacks getting nicely distributed across dozens of datacenters across the globe. In the shown attacks, if our router firewall failed, our physical servers wouldn't receive more than 500kpps of attack data. A well tuned iptables firewall should be able to cope with such a volume without a special kernel offload help.

Inter-AS Flowspec for the rest

Withstanding reflection attacks requires sufficient network capacity. Internet citizens not having fat network cables should use a good Internet Service Provider supporting flowspec.

Flowspec can be thought of as a protocol enabling firewall rules to be transmitted over a BGP session. In theory flowspec allows BGP routers on different Autonomous Systems to share firewall rules. The rule can be set up on the attacked router and distributed to the ISP network with the BGP magic. This will stop the packets closer to the source and effectively relieve network congestion.

Unfortunately, due to performance and security concerns only a handful of large ISP's allow inter-AS flowspec rules. Still - it's worth a try. Check if your ISP is willing to accept flowspec from your BGP router!

At Cloudflare we maintain an intra-AS flowspec infrastructure, and we have plenty of war stories about it.

Summary

In this blog post we've given details of three popular reflection attack vectors: NTP, SSDP and DNS. We discussed how the Cloudflare Anycast network helps us avoid a single choke point. In most cases dealing with reflection attacks is not rocket science though sufficient network capacity is needed and simple firewall rules are usually enough to cope.

The types of DDoS attacks we see from other vectors (such as IoT botnets) are another matter. They tend to be much larger and require specialized, automatic DDoS mitigation. And, of course, there are many DDoS attacks that occur using techniques other than reflection and not just using UDP.

Whether you face DDoS attacks of 10Gbps+, 100Gbps+ or 1Tbps+, Cloudflare can mitigate them.

Categories: Technology

Introducing Argo — A faster, more reliable, more secure Internet for everyone

Thu, 18/05/2017 - 14:00
Introducing Argo — A faster, more reliable, more secure Internet for everyone

The Internet is inherently unreliable, a collection of networks connected to each other with fiber optics, copper, microwaves and trust. It’s a magical thing, but things on the Internet break all the time; cables get cut, bogus routes get advertised, routers crash. Most of the time, these failures are noticed but inexplicable to the average user — ”The Internet is slow today!” — frustrating user experiences as people go about their lives on the Internet.

Introducing Argo — A faster, more reliable, more secure Internet for everyone

Today, to fix all of this, Cloudflare is launching Argo, a “virtual backbone” for the modern Internet. Argo analyzes and optimizes routing decisions across the global Internet in real-time. Think Waze, the automobile route optimization app, but for Internet traffic.

Just as Waze can tell you which route to take when driving by monitoring which roads are congested or blocked, Argo can route connections across the Internet efficiently by avoiding packet loss, congestion, and outages.

Cloudflare’s Argo is able to deliver content across our network with dramatically reduced latency, increased reliability, heightened encryption, and reduced cost vs. an equivalent path across the open Internet. The results are impressive: an average 35% decrease in latency, a 27% decrease in connection errors, and a 60% decrease in cache misses. Websites, APIs, and applications using Argo have seen bandwidth bills fall by more than half and speed improvements end users can feel.

Argo is a central nervous system for the Internet, processing information from every request we see to determine which routes are fast, which are slow, and what the optimum path from visitor to content is at that given moment. Through Cloudflare’s 115 PoPs and 6 million domains, we see every ISP and every user of the Internet pass through our network. The intelligence from this gives us a billion eyes feeding information about brownouts, faults, and packet loss globally.

Today, Argo includes two core features: Smart Routing and Tiered Cache. All customers can enable Argo today in the Traffic app in the dashboard. Argo is priced at $5/domain monthly, plus $0.10 per GB of transfer from Cloudflare to your visitors.

Introducing Argo — A faster, more reliable, more secure Internet for everyone

Argo Smart Routing

Networks on the Internet rely on legacy technologies like BGP to propagate and calculate routes from network to network, ultimately getting you from laptop-on-couch to video-on-YouTube. BGP has been around for decades, and was not designed for a world with malicious or incompetent actors lurking at every network hop.

In one comical example from 2008, a Pakistani ISP turned a botched censorship order into a global YouTube outage, bringing the fragility of core Internet routing algorithms into the public eye. In the same situation, Argo Smart Routing would detect which transit providers had valid routes to YouTube and which did not, keeping end user experience fast, reliable, and secure.

Metcalfe’s Law states that the value of a network is defined by the square of the number of nodes that make up the network. The existing Internet is incredibly valuable because of the number and diversity of nodes connected to the network. Unfortunately, this makes it difficult to pick up and start over; no Internet started from scratch, with sounder routing and traffic management, would come close to delivering the value provided by the current incarnation without a similar network footprint.

Because of our physical and virtual presence around the world, Cloudflare is uniquely positioned to rebuild the core of the Internet. Every customer we bring on increases the size of our network and the value of that network to each of our customers. Argo is Metcalfe’s Law brought to life.

Argo Smart Routing uses latency and packet loss data collected from each request that traverses our network to pick optimal paths across the Internet. Using this latency data, we’re able to determine which of our transit providers are performing best between any two points on the planet. Cloudflare now sees about 10% of all HTTP/HTTPS requests on the Internet. With Argo, each of those requests is providing the insight necessary to speed up all of its peers.

Introducing Argo — A faster, more reliable, more secure Internet for everyone CC BY 2.0 image by Steve Jurvetson

Enabling Argo (and Smart Routing with it) results in breathtaking reductions in latency. As an example, OKCupid enabled Argo and immediately saw a 36% decrease in request latency, as measured by TTFB (Time To First Byte). Without Argo, requests back to the origin from a Cloudflare PoP traverse the public Internet, subject to vagaries of routers, cables, and computers they will touch on their journey. With Argo, requests back to the origin are tunneled over our secure overlay network, on a path to the origin we've learned the performance of from all the requests that have traversed before it.

Transit over the public Internet is like driving with paper maps; it usually works, but using a modern navigation system that takes current traffic conditions into account will almost always be faster.

Routing over intelligently determined paths also results in significant reliability gains. Argo picks the fastest, most reliable route to the origin, which means routing around flapping links and routers that refuse to do their job. In a real-world illustration of these reliability gains, OKCupid saw a 42% drop in the number of connection timeouts on their site with Argo enabled.

It’s not just OKCupid that’s happy with Argo. 50,000 customers, large and small, have been beta testing Argo over the last 12 months. On average, these Argo Smart Routing beta customers saw a 35% decrease in latency and a 27% decrease in connection timeouts.

Argo Tiered Cache

Argo Tiered Cache uses the size of our network to reduce requests to customer origins by dramatically increasing cache hit ratios. By having 115 PoPs around the world, Cloudflare caches content very close to end users, but if a piece of content is not in cache, the Cloudflare edge PoP must contact the origin server to receive the cacheable content. This can be slow and places load on an origin server compared to serving directly from cache.

Argo Tiered Cache lowers origin load, increases cache hit ratios, and improves end user experience by first asking other Cloudflare PoPs if they have the requested content when a cache miss occurs. This results in improved performance for visitors, because distances and links traversed between Cloudflare PoPs are generally shorter and faster than the links between PoPs and origins. It also reduces load on origins, making web properties more economical to operate. Customers enabling Argo can expect to see a 60% reduction in their cache miss rate as compared to Cloudflare’s traditional CDN service.

Argo Tiered Cache also concentrates connections to origin servers so they come from a small number of PoPs rather than the full set of 115 PoPs. This results in fewer open connections using server resources. In our testing, we've found many customers save more on their cloud hosting bills than Argo costs, because of reduced bandwidth usage and fewer requests to the origin. This makes the service a “no brainer” to enable.

Additional Benefits

In addition to performance and reliability gains, Argo also delivers a more secure online experience. All traffic between Cloudflare data centers is protected by mutually authenticated TLS, ensuring any traffic traversing the Argo backbone is protected from interception, tampering, and eavesdropping.

With Argo, we’ve rebuilt things at the very core of the Internet, the algorithms that figure out where traffic should flow and how. We’ve done all this without any disruption to how the Internet works or how applications behave.

Cloudflare has built a suite of products to address lots of pains on the Internet. Argo is our newest offering.

Go ahead and enable it — you’ll find it in the Traffic tab in your dashboard.

PS. Interested in working on Argo? Drop us a line!
Introducing Argo — A faster, more reliable, more secure Internet for everyone

Categories: Technology

Introducing Load Balancing & Intelligent Failover with Cloudflare

Sun, 14/05/2017 - 16:00

Cloudflare's Enterprise customers have been using our Load Balancing service since March, and it has been helping them avoid website downtime caused by unreliable hosting providers, Internet outages, or servers. Today, we're bringing Load Balancing to all of our customers.

Even the best caching can't escape the fundamental limitations on performance created by the speed of light. Using Load Balancing, Cloudflare's customers can now route requests between multiple origins, allowing them to serve requests from the closest (and fastest) geographic location.

The Cloudflare Load Balancer automatically sends you notifications when things fail, and when they come back up again, so you can sleep well at night knowing we are keeping your website or API running.

If a DDoS attack can bring down your DNS provider or load balancer, it doesn't matter whether your servers are healthy or not. Our load balancing service runs in Cloudflare's 110+ datacenters, and with experience dealing with some of the largest DDoS attacks, we can withstand traffic volumes that smaller providers, virtual machines or hardware appliances can't. This also allows us to help you avoid business-impacting downtime when major cloud compute providers have issues: when we identify a connectivity reaching your application on AWS, we can fail over to your backup infrastructure on Google, a different region, or on-premise servers, and keep your site up and your customers happy.

Further, when proxying traffic through Cloudflare, you immediately benefit from faster failover responsiveness. In building Load Balancing we took a careful look at the existing global load balancing solutions on the market, and found that many rely on DNS to reroute traffic. Cloudflare will work as a DNS load balancer for non-proxied traffic (gray clouded in your DNS control panel) but works as an active proxy with near instant failover for proxied traffic. This means you don’t have to wait for TTLs to expire for traffic to shift.

We know from experience that it's common for a DNS change to take at least 60 seconds to propagate, even when everything is configured perfectly. Sixty seconds can represent thousands of failed requests, frustrated customers and lost sales. Cloudflare’s Load Balancer can reroute proxied traffic almost instantly, giving you the benefits previously reserved for on-premise load balancers but in the cloud.

Even customers with a single origin can benefit from Cloudflare’s Load Balancer. The product includes active and passive monitoring that will now alert you if we have trouble reaching your origin. Cloudflare’s position actively monitoring our customers’ traffic means we can let you know much more quickly than other monitoring services when your site is experiencing a problem

Valérian Saliou, Chief Technology Officer at Crisp, is using Cloudflare to geographically route WebSocket traffic: "When we rolled out Cloudflare Load Balancing to route traffic across our atlas of websocket servers, we immediately got messages from customers in Asia and Oceania thanking us for the improvement."

How Can I Get Started with Load Balancing?

So where do you go from here? You can enable Load Balancing in the Traffic app of your Cloudflare dashboard. We've put together a tutorial on how to get started with failover & health checks between multiple servers.

We want load balancing to be available to everyone. For that reason, Load Balancing starts at $5 per month, and includes 500,000 DNS queries each month (enough for the vast majority of sites).

Setting up a Load Balancer only takes a couple of minutes, and Load Balancers can easily be shared across multiple sites on Cloudflare, avoiding the need for repetitive configuration.

If you want to understand more about Load Balancing on Cloudflare, visit our product page or take a dive through our help articles.

Categories: Technology

Detroit and San Diego Data Centers expand Cloudflare network to 26 North American cities

Fri, 12/05/2017 - 22:29

alt

Cloudflare is excited to announce deployments in Detroit (Michigan) and San Diego (California), which are our 114th and 115th data centers respectively. They join Colombo, Sri Lanka and Cape Town, South Africa in the cohort of four new cities added just this week to our growing global network, which spans 57 countries and counting.

For over 6 million Internet properties, we now serve customer traffic from across 26 North American cities, including 22 in the United States alone. We're not going to stop building our network until we're within milliseconds of every Internet user, and to that end, data centers are already in the works in eight additional North American cities (and many others around the world).

Connections

alt Source: Baja Insider

Detroit and San Diego share something special, as they are immediately adjacent to international borders with Canada and Mexico respectively. Detroit has four border crossings to Windsor, Ontario, including the Ambassador Bridge, which was built in the Roaring Twenties, and accommodates over a quarter of all merchandise trade with Canada.

Founded in 1701, and best known for cars and Motown, Detroit eagerly awaits a 3,000 pound bronze RoboCop statue to watch over Delta City (track progress here). An early member of our technical operations team and fan of Detroit Techno is especially excited that this month's Movement Electronic Music Festival is a Cloudflare customer.

San Diego, the second largest city (by population) in California, is home to deep canyons and its world renowned San Diego Zoo. Later this month, skateboarders across San Diego will celebrate Tony Hawk Day.

Expanding the edge

Projects such as the non-profit Internet exchange DET-IX, enable greater regional interconnection, bringing together hosts, ISPs, and "edge" networks (such as Cloudflare), to exchange traffic locally instead of further away in Chicago. As we turn up peering with additional networks in the coming weeks, we expect to serve a growing share of Internet users in Detroit and San Diego right from their city.

Stay tuned for more data centers. The world's bluest sky will soon welcome the Cloudflare orange cloud.

Categories: Technology

Standing Up to a Dangerous New Breed of Patent Troll

Thu, 11/05/2017 - 16:00

On March 20th, Cloudflare received our first patent infringement claim: Blackbird Tech LLC v. Cloudflare, Inc. Today we’re filing our Answer to that claim in a federal court in Delaware. We have very strong arguments we will present in the litigation, mostly because the patent asserted against us does not have anything to do with our technology.

  • The infringement claim is not a close one. The asserted patent, US 6453335 (‘335 patent) was filed in 1998, and describes a system for monitoring an existing data channel and inserting error pages when transmission rates fall below a certain level. Nothing from 1998—including the ’335 patent—comes close to describing our state-of-the-art product that is provisioned via DNS, speeds up internet content delivery, and protects against malicious attackers. Our technology is innovative and different, and Cloudflare’s technology has about 150 patents issued or in process.

  • We also expect to show that the patent itself is invalid. For example, if the ’335 patent is read broadly enough to cover our system (which shouldn’t happen), it would also cover any system where electronic communications are examined and redacted or modified. But this is not new. Filtering products performing similar functions were around long before the time of the ‘335 patent. Blackbird’s claims under the ‘335 patent are much, much broader than what is justified in light of the prior art.

Blackbird Technologies (@bbird_tech) filed an identical suit against the folks at Fastly.

This claim is a nuisance for us. The lawsuit will take our time, distract us from our work, and it will cost us money to fight. Patent trolls like Blackbird exist to create these headaches so companies will play along and give them money to go away.

There’s no social value here. There’s no support for a maligned inventor. There’s no competing business or product. There’s no validation of an incentive structure that supports innovation. This is a shakedown where a patent troll, Blackbird Technologies, creates as much nuisance as it can so its attorney-principals can try to grab some cash.

Indominus rex, Jurassic World Indominus rex, Jurassic World

Worse still, Blackbird is a new, especially dangerous breed of patent troll. Like the dinosaur in the latest Jurassic Park movie, a synthetic combination of Tyrannosaurs and Velociraptor, Blackbird combines both a law firm and intellectual property rights holder into a single entity. In doing so, they remove legal fees from their cost structure and can bring lawsuits of potentially dubious merit without having to bear any meaningful cost. In other words, Blackbird’s new breed of entity is specifically designed to add leverage and amplify the already widely maligned problem of patent trolling.

Cloudflare does not intend to play along. As explained later in this blog post, we plan to (i) contest the patent lawsuit vigorously, (ii) fund an award for a crowdsourced search for prior art that can be used to invalidate Blackbird patents, and (iii) ask the relevant bar associations to investigate what we consider to be violations of the rules of professional conduct by Blackbird and its attorneys.

Part I — The Patent Troll Problem and Blackbird Tech

A troll is a creature that lives in a cave, under a bridge, or in the hills and serves no productive purpose—other than to be threatening, ugly, and malodorous. In modern parlance, the term is used for someone who merely seeks to annoy others to get a rise out of them, either in the comments section of an Internet page or by making nuisance legal claims. A patent troll is a company formed for only one purpose—to purchase patents and assert them broadly against real, productive companies that are actively engaged and making and selling products. The point isn’t whether their patents are infringed and/or invalid. A perverse incentive structure encourages trolls to bring lawsuits against numerous companies and demand nuisance settlements—settlements well below the millions of dollars it can take a company to protect themselves and defend a patent infringement case.

Freemont Troll "Freemont Troll" by Esther Lee (Flickr)

Although the case against patent trolls is well known, it is worth reviewing.

Patent trolls serve no productive purpose. They don’t make products, conduct research and development, hire employees, or add value to society. This is why they are often called non-practicing entities (NPE). They merely take existing, and largely unexercised patents, and assert them widely against numerous successful companies to try and pry some money for themselves. Often, the trolls lie in wait until after those companies have made irreversible investments in their technology to maximize their leverage. They buy patents solely for the purpose of initiating litigation, and benefit from their non-practicing status to limit their costs in litigation discovery and to limit counterclaims of patent infringement that could be raised if they were actually engaged in productive activity. And increasingly in the world of software or tech patents, the trolls rely on the broadest possible interpretation of vague or generalized patents to sow as much uncertainty as possible.

The social and economic costs of patent trolls are staggering, no matter how you measure them. Estimates in recent years suggest that the number of cases brought by patent trolls have increased more than 500% over the past ten years. And about 70% of all patent infringement claims are filed by patent trolls, a share that has more than doubled in recent years. It is estimated that litigation initiated by patent trolls in U.S. courts cost companies as much as $30 billion in direct costs, a number that has increased more than four-fold over the last ten years. And those direct costs are only the tip of the iceberg, as losses in company value and foregone economic opportunity may be as many as five or ten times that amount.




[See, generally, Chien, Colleen. “Startups and Patent Trolls.” Santa Clara University Legal Studies Research Paper No. 09-12 Working Paper Series, 2012. Bessen, James E. Michael J. Meurer. “The Direct Costs from NPE Disputes.” Boston University School of Law, Law and Economics Research Paper No. 12-34, June 28, 2012. RPX. 2015 Report: NPE Litigation, Patent Marketplace, and NPE Cost.]

Cloudflare’s Support for the Patent System

Make no mistake, Cloudflare is a strong supporter of the patent system. The value and stability of our company is supported by the nearly 150 patents awarded or in process on our technology. We wouldn’t be able to exist, and wouldn’t be able to invest to bring our company to scale, if we didn’t have the security of a well-functioning patent system.

In a fair fight, we’d be able to defend ourselves by pushing back against a practicing entity with the technology set forth in our own patents. If the other entity was actually applying its patent to commercial products and felt hindered in the marketplace because of our operations, we might be able to raise countering arguments that a practicing entity was closer to infringing on our patents than we are to infringing theirs. It would help us to defend the lines around our use of own patented technology and demonstrate whether the infringement was actually happening.

But patent trolls get to live in a fictional world without a business or operations that can be compared or challenged.

And the value of the patent system isn’t one limited to recent developments or to the tech boom. It has supported the innovative spirit of the American people since America’s founding.

It has been reported that during the ransacking of Washington by British soldiers in the War of 1812, then Superintendent of Patents, William Thorton, put himself in front of a British cannon and declared, “Are you Englishmen or only Goths and Vandals? This is the Patent Office, a depository of the ingenuity of the American nation, in which the whole civilized world is interested. Would you destroy it? If so, fire away, and let the charge pass through my body.” Whether or not the result of Superintendent Thorton’s bold stance, the patent building was one of the few government buildings spared by the British army in that attack.

When he redecorated the Oval Office early in his administration, President Barack Obama’s addition of a bust of Martin Luther King, Jr. garnered the most attention. But he also brought in three mechanical devices from the National Museum of American History’s patent collection (models of the telegraph, paddle wheel, and gear-cutter) to reflect the importance of technology and the creative spirit. In that spirit, he later took aggressive action issuing five executive orders to challenge patent trolls (“History Will Remember Barack Obama as the Great Slayer of Patent Trolls,” Wired, 3/20/14).

We are fighting back against patent trolls not because we don’t like the patent system. To the contrary, it’s due to our deep appreciation and need for the patent system that the actions of patent trolls are so offensive to us. Rather than appreciating the economic and social value of an effective patent system, these trolls distort and manipulate the patent system and the court system merely to line their own pockets. Those actions call into question the legitimacy of both systems in a way that is unacceptable to Cloudflare.

Blackbird Technologies

Even though Blackbird Tech is a pure patent troll, which has no operations aside from using patents it purchased to bring litigation against legitimate companies, they have specifically designed their business to create a new breed which poses serious additional concern.

Wendy Verlander and Chris Freeman, the Founders of the Blackbird Technologies Law Firm

Blackbird was formed three years ago by two attorneys who left law firms where they had been engaged in patent defense work — Wendy Verlander (@bbirdtech_CEO; LinkedIn) at WilmerHale, and Chris Freeman (LinkedIn) at Kirkland & Ellis. Notably, both of those firms promote themselves as ready to protect companies from patent trolls. Kirkland trumpets that its IP practice group scored a victory against the “original patent troll,” while WilmerHale has a Patent Troll Initiative that aims to help businesses deal comprehensively with patent trolls.

Having gained valuable experience and training by working for clients who paid their firms handsomely to fight suits brought by patent trolls, Verlander and Freeman were well aware of the harm done to their clients by patent trolls. Yet, Verlander and Freeman decided to cast their lot with the other side and formed a patent troll for themselves.

Blackbird Technologies has filed 107 cases since September of 2014, making it one of the most prolific trolls in the United States. Its website links to a “News” item titled “4 Frequent Filers of IP Suits to Watch this Year” which highlights Blackbird as “a newer entrant on the list of top patent plaintiffs, coming in at fourth place with 48 suits last year in the District of Delaware spanning a wide range of technologies.” Some of the patents at issue include: Bicycle Pet Carrier, Buttock Lift Support, Sports Bra, and Method for Managing a Parking Lot. A complete list of Blackbird's patents is available here. Although they have been very aggressive about filing such claims, they have still not taken a single case through trial. And only a couple of those cases made it to the claim construction phase, where the Court defines the meaning of the patents at issue. Instead, many of Blackbird’s cases have been resolved shortly after filing, suggesting that these cases were never about legal rights or claims but were instead about creating the impetus for a nuisance settlement in the face of significant litigation costs.

In the Cloudflare case, Blackbird is asserting a 1998 patent filed by Oliver Kaufmann. Based on an assignment agreement filed with the USPTO, Blackbird purchased the ‘335 patent from Mr. Kaufmann in October 2016 for $1 plus “other good and valuable consideration.” Because a mere five months later Blackbird used the ‘335 patent as the sole basis for lawsuits against Cloudflare and Fastly, we think the actual but undisclosed compensation between the parties is something considerably more than $1, and may very well involve a contingency arrangement for Mr. Kaufmann.

Mr. Kaufmann appears to be the head of a group of four small IT companies in Augsburg, Germany that share a website — www.exklusiv.de. The companies provide services such as on premise networking and phone, ISP, database design and management, and 3D printing. If Mr. Kauffman ever tried to commercialize the ‘335 patent (in the U.S. or otherwise), we would expect to see some evidence of that on the website. We don’t. Of course, the patent troll system attempts to shield Mr. Kaufmann and his companies’ operations from the lawsuit by having Blackbird become the owner of the patent — even if Mr. Kaufmann maintains an interest in the litigation.

Part II — Cloudflare’s Response

Cloudflare’s mission has always been to help build a better Internet. So it won’t be surprising to frequent readers of this blog that Cloudflare isn’t interested in a short term and narrow resolution of our own interests. We’re not going to reach a settlement that would pay tens of thousands of dollars to Blackbird to avoid millions in legal fees. That would only allow patent trolls to keep playing their game and preying upon other innovative companies that share our interest in making the Internet work better, especially newer and more vulnerable companies.

We are pursuing a four-pronged approach to challenge Blackbird’s actions in this case.

Step 1 — Cloudflare will fight this case in the courts

Cloudflare will not settle this case, and doesn’t plan to settle any patent troll case, ever.

We will litigate this claim as vigorously and expeditiously as possible. Because it is clear to us after even minimal search that there is likely considerable evidence of prior art, we expect to file an Inter Parties Review of the ‘335 patent with the USPTO. If appropriate, we will seek sanctions and fee awards against Blackbird for bringing this case.

Step 2 — Cloudflare will fund a crowdsourced effort to find evidence to invalidate Blackbird’s patents… all of them

In a separate blog post today, Cloudflare is announcing that it will award up to $50,000 to support a search for prior art that can be used to invalidate Blackbird’s patents. Patent trolls like Blackbird have demonstrated that they can’t be trusted to act responsibly or take the public good into account when asserting patents against practicing companies.

The Crowd Crowd by Hamaz Butt (Flickr)

Patent trolls take advantage of a system they assume is tilted in their favor, where they can take vague technology patents issued years ago and apply them as broadly as imaginable to the latest technology. Patent trolls think they can sit back and pick off settlements from companies because their lawsuits are a nuisance and the costs of defending those suits are considerable.

Changing this dynamic and leveling the playing field is going to require an entirely new way of approaching these challenges. Fighting such strong, though perverse, economic incentives is going to require a groundswell. It calls for a broad and active group of innovative folks who are willing to join the cause of helping expose the weaknesses in the patents held by these patent trolls, most of which were issued so long ago and are so broad that they never should have been issued in the first place.

Exposing the weakness of these patents means finding prior art that demonstrates the patents were issued improperly and working to have the USPTO invalidate those patents before they can be weaponized against legitimate companies that are working to innovate, to provide a livelihood to their employees and their families, and to make good products. We aim to invalidate every one of Blackbird’s patents and need your help. Come join us in this effort!

Prior art is any evidence that a patented invention was already known at the time the patent application was filed. Prior art does not need to be a competing patent, it does not necessarily need to exist physically or be commercially available. It is enough that there is evidence that the innovation was publicly known at any point when or before the patent application was filed. Though certainly the more formal and specific the prior art, the better.

As described in more detail in the other blog post, Cloudflare will award up to $20,000 to be shared among the participants who provide us the most useful prior art that can be used in our invalidity challenge to the ‘335 patent in the pending litigation. We will use our outside patent counsel to evaluate the responses and divide the award based on relevance and usefulness. We will also accept information and arguments which invalidate Blackbird patents such as a showing of obviousness or material defects in the patent application.

In addition, we are seeking prior art information on any of the 37 other patents or patent applications owned by Blackbird.

We have a pool of up to $30,000 that we will distribute to people who submit information on any Blackbird patent. Again, the money will be awarded based on relevance and usefulness of the prior art. We will publish all the information that we receive on these patents, so that others may benefit from the information and file challenges on Blackbird’s patents. And where justified, Cloudflare may institute patent office proceedings of any Blackbird patent when there is sufficient prior art to call into question its validity.

These opportunities remain open as long as Blackbird’s case against Cloudflare is still active. If interested, you should go over to the other blog post here, join the cause and be one of a group of talented and motivated experts who will push back against lazy trolls who try to take advantage of innovators by exploiting bugs in the judicial system.

Step 3 — Cloudflare will investigate Blackbird’s operations to develop facts that support our arguments in the litigation and expose how patent trolls really operate

Because patent trolls do not have operations beyond their ownership and litigation of patents for inventions they did not create, it can often be difficult to litigate against them. In addition, because of the unique organization of Blackbird, which appears to actually be a law firm that brings lawsuits on its own behalf and without a real client, there are important facts about this dispute that are difficult or impossible for us to determine at present.

As a result, we expect to investigate and to use the discovery process in this case and to retain a group of private investigators to look into Blackbird’s operations and attorneys to determine what counterclaims and arguments may be available to us in defense of the case. At a minimum, Blackbird suggests that it is engaged in a new and even more ingenious way of operating as a patent troll with maximum leverage and minimal inefficiencies in their operation. At a time when the economic evidence and public sentiment against the work of patent trolls is strong, it is important to bring their operations to the light of day and understand exactly how they work.

Step 4 — File complaints against Blackbird attorneys by bar association disciplinary counsel in Massachusetts and Illinois

The practice of law is supposed to be different from business, with lawyers held to a higher ethical standard. In most states, it is illegal for those who have not been admitted to the bar to provide legal services. This often gives lawyers a privileged position when it comes to handling things like real estate transactions, tax matters, and estate planning that might also be handled by other professionals, and it gives them a near monopoly on access to and use of the court system. In exchange, the legal profession is strictly governed by a series of ethical rules and other rules of professional conduct to make sure the profession sufficiently self-regulates to maintain public confidence and therefore secure the advantages of its monopoly. That’s why admission of new lawyers to the bar is generally premised not only on proof of competence (the bar exam), but also an investigation of character and fitness, and an oath committing to the principles of the profession. Importantly, the rules support the notion that the practice of law is a profession and not a mere business endeavor, this distinction is seen as essential to distinguishing the legal profession and maintaining the monopoly. See, generally Richard L. Abel, American Lawyers (1989).

Specifically, the American Bar Association established its Model Rules of Professional Conduct in 1983, which have subsequently been adopted by bar associations in almost every state, including the two states where Blackbird has its offices, Massachusetts and Illinois. The rules set forth a series of responsibilities that each lawyer must undertake, including duties to the profession generally and duties of candor to the court, which may require actions that run counter to the attorney’s self-interest.

For example, attorneys have an affirmative obligation to disclose when they have knowledge of adverse controlling legal authority (a rule or controlling court opinion that invalidates their proposed outcome), even if such disclosure runs counter to the interests of the attorney’s client. This disclosure is required even if both opposing counsel and the court are otherwise unaware of that authority (Model Rule 3.3(a)(2)). So attorneys are not allowed to take advantage of the judicial system just because they have an opening to do so.

Most of the rules address the extensive obligations owed to a client, to make sure lawyers don’t leverage their monopoly position to take advantage of the client. Among other things, the special nature of the Attorney-Client relationship involves a number of controls on the business relationship between the two — the attorney is to strictly avoid a conflict of interest with her work for the client (Rules 1.6-1.10), and should avoid self-dealing or commingling of funds (Rules 1.8 and 1.15).

As made clear in this blog post, Blackbird’s founders made the decision to leave law firms that were engaged in the defense of clients who were faced with patent lawsuits, and formed a new law firm focused on bringing law suits as a patent troll. The only services promoted on its website (http://www.blackbird-tech.com/) are legal services; the website notes that Blackbird represents a “new model” which provides the benefits of “top law firm experience” offering clients the ability to “litigate at reduced costs.”

Blackbird holding themselves out as a law firm

Of 12 total employees listed on the Blackbird website, 7 are attorneys. The remaining 5 are very junior employees described as “analysts” (3 are current undergraduate students and 2 received Bachelor's degrees last May). As far as we can determine, Blackbird produces no products or services which it makes available to the public. Rather, it offers litigation services and is in the business of filing lawsuits. And its output in that regard is prolific, as it has filed a total of 107 lawsuits since September 2014.

As final confirmation that Blackbird is a law firm marketing legal services, its own website includes a disclaimer about “Attorney Advertising,” which states explicitly “[p]lease note that this website may contain attorney advertising.”

Blackbird Technologies says they advertise as a law firm

Blackbird’s “new model” seems to be only that its operations set out to distort the traditional Attorney-Client relationship. Blackbird’s website makes a direct pitch of its legal services to recruit clients with potential claims and then, instead of taking them on as a client, purchases their claims and provides additional consideration that likely gives the client an ongoing interest in the resulting litigation. In doing so, Blackbird is flouting its ethical obligations meant to protect clients and distorting the judicial process by obfuscating and limiting potential counterclaims against the real party in interest.

As a result, we think Blackbird may be engaged in several activities that violate the rules of professional responsibility in the jurisdictions where they have been admitted as members of the bar. So, Cloudflare is submitting today letters with the bar disciplinary committees in Massachusetts and Illinois asking them to look into the following practices of Blackbird Tech:

  1. Blackbird may have acquired a proprietary interest in the subject matter of the litigation in violation of Rule 1.8(i) — Attorneys have a near monopoly of representing clients in the judicial system. Rule of Professional Conduct 1.8(i) explicitly prohibits an attorney from “acquir[ing] a proprietary interest in a cause of action or subject matter of litigation.” But that is exactly what Blackbird does. Blackbird’s website contains a pitch to recruit clients with potential legal claims under their patents, but then buys those claims and brings them on their own behalf. Wouldn’t that be a violation of Rule 1.8(i)? Doesn’t Blackbird’s attempt to pitch this as a “new model” of being a patent troll ignore the fact that the only non-law firm activity in which they are engaged (buying patents to bring lawsuits) is the exact thing prohibited by Rule 1.8(i)? They shouldn’t be able to use creative contractual or corporate structures to avoid its responsibility under the rules.

  2. Blackbird may be sharing fees or firm equity with non-lawyers in violation of Rule 5.4(a) or 5.4(d) — In order to preserve the integrity of the Attorney-Client relationship, Rule of Professional Conduct 5.4(a) prohibits attorneys from splitting legal fees in individual matters with non-lawyers, and Rule 5.4(d) prohibits providing an equity interest in a firm to non-lawyers. We think Blackbird may be violating both provisions. Although he no longer owns the patent and is not a party to the case, the assignment agreement’s terms (specifying payment of only $1) makes it possible that Mr. Kaufman has a contingency interest in the lawsuit. If that is the case, wouldn’t Blackbird be in violation of Rule 5.4(a)? Similarly, Blackbird has moved very quickly since its founding to file lawsuits against a great number of companies — 107 complaints since September 2014. So far, none of those cases have gone to trial. We intend to examine whether they have used financial support from non-lawyers to fund the very fast start to their operations in exchange for an impermissible equity interest, or have shared an equity interest with patent holders like Mr. Kaufmann, either of which would be in violation of Rule 5.4(d).

In Closing

We've always advised our customers not to pay threatened DDoS attackers' ransom requests but, instead, to mount a real defense. There are a lot of analogies here and we believe it's important to stand up for what is right and against this new breed of patent troll.

If you’ve read this far, then you must share Cloudflare’s passion on this issue. So join us! Write about and talk about these issues. Contact your political representatives and ask them to support all the legislative reform efforts that are out there. And most importantly, take some time and join our crowdsourced effort to identify prior art that will invalidate all of the Blackbird patents.

Watch this space. We’re just getting started.

Categories: Technology

Project Jengo: Cloudflare's Prior Art Search Bounty

Thu, 11/05/2017 - 16:00

Bounty Hunter Attacks Jengo Fett by Brickset (Flickr)

As readers of this blog likely know, especially if you read this post, Cloudflare has been sued by a dangerous new breed of patent troll, Blackbird Technologies, asserting a very old and very vague patent. And we know we are not alone in being frustrated about the way that such patent trolls inhibit the growth of innovative companies. Cloudflare is asking for your help in this effort, and we’re putting our money where our mouth is.

Patent trolls take advantage of a system they assume is tilted in their favor, where they can take vague technology patents issued years ago and apply them as broadly as imaginable to the latest technology. And they do this without the limitations of having to show the original patent holder would have actually exercised the patent, because most of them don’t, at all. Patent trolls think they can sit back and pick off settlements from companies because their lawsuits are a nuisance and the costs of defending those suits are considerable.

Changing this dynamic and leveling the playing field is going to require an entirely new approach. Fighting such strong, though perverse, economic incentives is going to require a groundswell. It calls for a large and active group of innovative folks who are willing to join the cause of helping expose the weakness in the patents held by these patent trolls. Exposing the weakness of these patents means finding prior art that demonstrates the patents were issued improperly and working to have the USPTO invalidate those patents before they can be weaponized against legitimate companies that are working to innovate, to provide a livelihood to their employees and their families, and to make good products.

At Cloudflare, we’re lucky to have the best and most loyal customers on the Internet, a whole bunch of fans who are ridiculously smart and don’t take kindly to lazy trolls who try to take advantage by exploiting bugs in the judicial system. And of course, we’re more than happy to give them a bit of a nudge.

Indiana Jones Warehouse Indiana Jones: Raiders of the Lost Ark

Awards for Prior Art

Cloudflare is committing up to $50,000 to support a search for prior art that can be used to invalidate patents held by Blackbird Tech…all of them. By filing 107 lawsuits against companies since September 2014, Blackbird has demonstrated that it is going to use its patents to sue productive companies. And patent trolls often target newer and more innovative companies, so it is important to investigate and research to discover whether prior art existed on the Blackbird patents and to keep them from filing additional cases against us or against other companies.

By “prior art,” we mean evidence that the patented technology was in use or known before the inventor conceived of or filed for his patent. Often, prior art consists of academic papers, technical websites describing current technology, other patents/patent applications—or, really, any other evidence showing that the technology was in use or known. For example, for the ‘335 patent, we are only asking for evidence that pre-dates the filing date of the patent—July 21, 1998. We will also accept information and arguments which invalidate Blackbird patents such as a showing of obviousness or material defects in the patent application. And, for our purposes, the “patented technology” is defined by the claims that are listed at the end of the ’355 patent.

The $50,000 will fund two separate awards.

The first bounty (up to $20,000) is for prior art which reads on the patent Blackbird is using to sue Cloudflare, the ‘335 patent. $10,000 is guaranteed and will be divided among prior art submissions that raise substantive questions on the ‘335 patent. The remaining $10,000 will be used to compensate prior art submissions that Cloudflare uses as evidence in an invalidation procedure at the USPTO or invalidation at trial. The latest date of prior art on the ‘335 patent would be July 20, 1998.

The larger bounty (up to $30,000) will be spread among those submitting substantial prior art which reads on any of the 34 other outstanding Blackbird patents or their 3 in-flight patent applications and could lead to the invalidation of these dubious patents. Cloudflare will pay the second bounty to people who submit relevant and substantive prior art which, in Cloudflare’s opinion, reads on any other Blackbird patent. The money will be distributed based on the quality of the prior art, the perceived value of the patent, and the extent to which the evidence is used in a proceeding to invalidate one of the Blackbird patents.

We will maintain a list of all the Blackbird patents at cloudflare.com/blackbirdpatents/. The list will provide the number of each patent, the relevant latest date of prior art, and will list germane already-identified prior art. We will update the list periodically as we get new information submitted.

How to participate

You may submit your suggestion for prior art either through cloudflare.com/priorartsearch/. Cloudflare’s experts and attorneys will review each submission for its value in invalidating each patent. Again, the money will be awarded based on relevance and usefulness. Cloudflare will keep the award will be open until the end of our litigation, but we may start making initial partial awards as early as within 90 days.

In order to be eligible for any bounty you need to be the first person to submit a particular piece of prior art and the prior art must not be known to Cloudflare (e.g., the prior art must not be listed on the face of the Blackbird patent or in the patent prosecution’s history). For each submission of prior art, you will need to give us your name and contact information. In your submission, you will need to document:

  • Your name and email address
  • The number of the Blackbird patent at issue
  • What the prior art is.
  • Where you found it.
  • When it was first publicly available.
  • Where it was first publicly available.
  • Why you think it reads on a Blackbird’s patent.
  • Any other information which you think is helpful to our review.

Big hint: don’t go to Google’s patent page and click on the prior art button, we’ve already done that. We will then use our outside patent counsel to evaluate the responses and divide the award based on relevance and usefulness. See here cloudflare.com/priorartrules/ for official rules and details.

Patent trolls like Blackbird have demonstrated that they can’t be trusted to act responsibly or take the public good into account when asserting patents against practicing companies. Making clear that its patents are invalid, or significantly narrower than Blackbird is likely to assert, is another way we can limit Blackbird’s ability to create economic harm.

Categories: Technology

Cape Town (South Africa): Cloudflare Data Center #113

Thu, 11/05/2017 - 05:13
 Cloudflare Data Center #113

 Cloudflare Data Center #113

Five fun facts:
* Cape Town is where the Atlantic Ocean and the Indian Ocean meet deep in the Southern Hemisphere.
* The city is the start of the Garden Route, a 185 mile (300 km) glorious coastal drive brimming with native flowers and stunning vistas.
* Cape Town is the gateway into the Stellenbosch and Franschhoek wine districts that make for very popular weekend excursions.
* The imposing Table Mountain can be seen from most of Cape Town. When you take the cable car to the top of the mountain, the view from up there is even more stunning.
* Cape Town is where Cloudflare has placed its 113rd data center.

Second data center in South Africa

Back in December 2014, Cloudflare opened our first data center in Africa and our 30th datacenter globally. That was in Johannesburg, which has since seen over 10x growth in traffic delivered to South Africa and surrounding countries.

Now, we are expanding into our second city in South Africa — Cape Town, bringing us 870 miles (1,400km) closer to millions of Internet users. Only 15% smaller than Johannesburg by population, Cape Town commands a majority of the tourism business for the country.

For Cloudflare, our newest deployment reinforces our commitment to helping build a better Internet in South Africa and Southern Africa as a whole. Expanding on our existing peering at the JINX and NAPAfrica Internet exchanges in Johannesburg, we are now also a member at NAPAfrica Cape Town. As we enable the interconnections with in-country hosts and ISPs, we will distribute and optimize traffic delivery, while providing in-country redundancy for millions of Internet properties.

Cloudflare now operates six data centers across Africa. Cape Town joins existing facilities in Cairo (Egypt), Luanda (Angola), Mombasa (Kenya), Djibouti (Djibouti), and Johannesburg (South Africa).

Cape Town and Johannesburg equally served

In the graph below, Gauteng is the province containing Johannesburg and the nation's capital Pretoria, whereas the Western Cape province includes Cape Town and the aforementioned wine districts. With the second data center now operational in South Africa, we are seeing near equivalent performance over both major cities.

 Cloudflare Data Center #113 Latency (ms) decreases to Cloudflare customers. Source: Cedexis

Two More Cities

Colombo and Cape Town are the first two of four data centers going live this week. Our next two deployments, yet to be revealed, will provide additional redundancy to existing facilities in North America. Can you guess the cities they are in?

Categories: Technology

How Cloudflare analyzes 1M DNS queries per second

Wed, 10/05/2017 - 22:50

On Friday, we announced DNS analytics for all Cloudflare customers. Because of our scale –– by the time you’ve finished reading this, Cloudflare DNS will have handled millions of DNS queries –– we had to be creative in our implementation. In this post, we’ll describe the systems that make up DNS Analytics which help us comb through trillions of these logs each month.

How logs come in from the edge

Cloudflare already has a data pipeline for HTTP logs. We wanted to utilize what we could of that system for the new DNS analytics. Every time one of our edge services gets an HTTP request, it generates a structured log message in the Cap’n Proto format and sends it to a local multiplexer service. Given the volume of the data, we chose not to record the full DNS message payload, only telemetry data we are interested in such as response code, size, or query name, which has allowed us to keep only ~150 bytes on average per message. It is then fused with processing metadata such as timing information and exceptions triggered during query processing. The benefit of fusing data and metadata at the edge is that we can spread the computational cost across our thousands of edge servers, and log only what we absolutely need.

The multiplexer service (known as “log forwarder”) is running on each edge node, assembling log messages from multiple services and transporting them to our warehouse for processing over a TLS secured channel. A counterpart service running in the warehouse receives and demultiplexes the logs into several Apache Kafka clusters. Over the years, Apache Kafka has proven to be an invaluable service for buffering between producers and downstream consumers, preventing data loss when consumers fail over or require maintenance. Since version 0.10, Kafka allows rack-aware allocation of replicas, which improves resilience against rack or site failure, giving us fault tolerant storage of unprocessed messages.

Having a queue with structured logs has allowed us to investigate issues retrospectively without requiring access to production nodes, but it has proven to be quite laborious at times. In the early days of the project we would skim the queue to find offsets for the rough timespan we needed, and then extract the data into HDFS in Parquet format for offline analysis.

About aggregations

The HTTP analytics service was built around stream processors generating aggregations, so we planned to leverage Apache Spark to stream the logs to HDFS automatically. As Parquet doesn’t natively support indexes or arranging the data in a way that avoids full table scans, it’s impractical for on-line analysis or serving reports over an API. There are extensions like parquet-index that create indexes over the data, but not on-the-fly. Given this, the initial plan was to only show aggregated reports to customers, and keep the raw data for internal troubleshooting.

The problem with aggregated summaries is that they only work on columns with low cardinality (a number of unique values). With aggregation, each column in a given time frame explodes to the number of rows equal to the number of unique entries, so it’s viable to aggregate on something like response code which only has 12 possible values, but not a query name for example. Domain names are subject to popularity, so if, for example, a popular domain name gets asked 1,000 times a minute, one could expect to achieve 1000x row reduction for per-minute aggregation, however in practice it is not so.

Due to how DNS caching works, resolvers will answer identical queries from cache without going to the authoritative server for the duration of the TTL. The TTL tends to be longer than a minute. So, while authoritative servers see the same request many times, our data is skewed towards non-cacheable queries such as typos or random prefix subdomain attacks. In practice, we see anywhere between 0 - 60x row reduction when aggregating by query names, so storing aggregations in multiple resolutions almost negates the row reduction. Aggregations are also done with multiple resolution and key combinations, so aggregating on a high cardinality column can even result in more rows than original.

For these reasons we started by only aggregating logs at the zone level, which was useful enough for trends, but too coarse for root cause analysis. For example, in one case we were investigating short bursts of unavailability in one of our data centers. Having unaggregated data allowed us to narrow the issue down to the specific DNS queries experiencing latency spikes, and then correlated the queries with a misconfigured firewall rule. Cases like these would be much harder to investigate with only aggregated data as it only affected a tiny percentage of requests that would be lost in the aggregations.

So we started looking into several OLAP (Online Analytical Processing) systems. The first system we looked into was Druid. We were really impressed with the capabilities and how the front-end (Pivot and formerly Caravel) is able to slice and dice the data, allowing us to generate reports with arbitrary dimensions. Druid has already been deployed in similar environments with over 100B events/day, so we were confident it could work, but after testing on sampled data we couldn’t justify the hardware costs of hundreds of nodes. Around the same time Yandex open-sourced their OLAP system, ClickHouse.

And then it Clicked

ClickHouse has a much simpler system design - all the nodes in a cluster have equal functionality and use only ZooKeeper for coordination. We built a small cluster of several nodes to start kicking the tires, and found the performance to be quite impressive and true to the results advertised in the performance comparisons of analytical DBMS, so we proceeded with building a proof of concept. The first obstacle was a lack of tooling and the small size of the community, so we delved into the ClickHouse design to understand how it works.

ClickHouse doesn’t support ingestion from Kafka directly, as it’s only a database, so we wrote an adapter service in Go. It read Cap’n Proto encoded messages from Kafka, converted them into TSV, and inserted into ClickHouse over the HTTP interface in batches. Later, we rewrote the service to use a Go library using the native ClickHouse interface to boost performance. Since then, we’ve contributed some performance improvements back to the project. One thing we learned during the ingestion performance evaluation is that ClickHouse ingestion performance highly depends on batch size - the number of rows you insert at once. To understand why, we looked further into how ClickHouse stores data.

The most common table engine ClickHouse uses for storage is the MergeTree family. It is conceptually similar to LSM algorithm used in Google’s BigTable or Apache Cassandra, however it avoids an intermediate memory table, and writes directly to disk. This gives it excellent write throughput, as each inserted batch is only sorted by “primary key”, compressed, and written to disk to form a segment. The absence of a memory table or any notion of “freshness” of the data also means that it is append-only and data modification or deletion isn’t supported. The only way to delete data currently is to remove data by calendar months, as segments never overlap a month boundary. The ClickHouse team is actively working on making this configurable. On the other hand, this makes writing and segment merging conflict-free, so the ingestion throughput scales linearly with the number of parallel inserts until the I/O or cores are saturated. This, however, also means it is not fit for tiny batches, which is why we rely on Kafka and inserter services for buffering. ClickHouse then keeps constantly merging segments in the background, so many small parts will be merged and written more times (thus increasing write amplification) and too many unmerged parts will trigger aggressive throttling of insertions until the merging progresses. We have found that several insertions per table per second work best as a tradeoff between real-time ingestion and ingestion performance.

The key to table read performance is indexing and the arrangement of the data on disk. No matter how fast processing is, when the engine needs to scan terabytes of data from disk and use only a fraction of it, it’s going to take time. ClickHouse is a columnar store, so each segment contains a file for each column, with sorted values for each row. This way whole columns not present in the query can be skipped, and then multiple cells can be processed in parallel with vectorized execution. In order to avoid full scans, each segment also has a sparse index file. Given that all columns are sorted by the “primary key”, the index file only contains marks (captured rows) of every Nth row in order to be able to keep it in memory even for very large tables. For example the default settings is to make a mark of every 8,192th row. This way only 122,070 marks are required to sparsely index a table with 1 trillion rows, which easily fits in memory. See primary keys in ClickHouse for a deeper dive into how it works.

When querying the table using primary key columns, the index returns approximate ranges of rows considered. Ideally the ranges should be wide and contiguous. For example, when the typical usage is to generate reports for individual zones, placing the zone on the first position of the primary key will result in rows sorted by zone in each column, making the disk reads for individual zones contiguous, whereas sorting primarily by timestamp would not. The rows can be sorted in only one way, so the primary key must be chosen carefully with the typical query load in mind. In our case, we optimized read queries for individual zones and have a separate table with sampled data for exploratory queries. The lesson learned is that instead of trying to optimize the index for all purposes and splitting the difference, we have made several tables instead.

One such specialisation are tables with aggregations over zones. Queries across all rows are significantly more expensive, as there is no opportunity to exclude data from scanning. This makes it less practical for analysts to compute basic aggregations on long periods of time, so we decided to use materialized views to incrementally compute predefined aggregations, such as counters, uniques, and quantiles. The materialized views leverage the sort phase on batch insertion to do productive work - computing aggregations. So after the newly inserted segment is sorted, it also produces a table with rows representing dimensions, and columns representing aggregation function state. The difference between aggregation state and final result is that we can generate reports using an arbitrary time resolution without actually storing the precomputed data in multiple resolutions. In some cases the state and result can be the same - for example basic counters, where hourly counts can be produced by summing per-minute counts, however it doesn’t make sense to sum unique visitors or latency quantiles. This is when aggregation state is much more useful, as it allows meaningful merging of more complicated states, such as HyperLogLog (HLL) bitmap to produce hourly unique visitors estimate from minutely aggregations. The downside is that storing state can be much more expensive than final values - the aforementioned HLL state tends to be 20-100 bytes / row when compressed, while a counter is only 8 bytes (1 byte compressed on average). These tables are then used to quickly visualise general trends across zones or sites, and also by our API service that uses them opportunistically for simple queries. Having both incrementally aggregated and unaggregated data in the same place allowed us simplify the architecture by deprecating stream processing altogether.

Infrastructure and data integrity

We started with RAID-10 composed of 12 6TB spinning disks on each node, but reevaluated it after the first inevitable disk failures. In the second iteration we migrated to RAID-0, for two reasons. First, it wasn’t possible to hot-swap just the faulty disks, and second the array rebuild took tens of hours which degraded I/O performance. It was significantly faster to replace a faulty node and use internal replication to populate it with data over the network (2x10GbE), than to wait for an array to finish rebuilding. To compensate for higher probability of node failure, we switched to 3-way replication and allocated replicas of each shard to different racks, and started planning for replication to a separate data warehouse.

Another disk failure highlighted a problem with the filesystem we used. Initially we used XFS, but it started to lock up during replication from 2 peers at the same time, thus breaking replication of segments before it completed. This issue has manifested itself with a lot of I/O activity with little disk usage increase as broken parts were deleted, so we gradually migrated to ext4 that didn’t have the same issue.

Visualizing Data

At the time we relied solely on Pandas and ClickHouse’s HTTP interface for ad-hoc analyses, but we wanted to make it more accessible for both analysis and monitoring. Since we knew Caravel - now renamed to Superset - from the experiments with Druid, we started working on an integration with ClickHouse.

Superset is a data visualisation platform designed to be intuitive, and allows analysts to interactively slice and dice the data without writing a single line of SQL. It was initially built and open sourced by AirBnB for Druid, but over time it has gained support for SQL-based backends using SQLAlchemy, an abstraction and ORM for tens of different database dialects. So we wrote and open-sourced a ClickHouse dialect and finally a native Superset integration that has been merged a few days ago.

Superset has served us well for ad-hoc visualisations, but it is still not polished enough for our monitoring use case. At Cloudflare we’re heavy users of Grafana for visualisation of all our metrics, so we wrote and open-sourced a Grafana integration as well.

It has allowed us to seamlessly extend our existing monitoring dashboards with the new analytical data. We liked it so much that we wanted to give the same ability to look at the analytics data to you, our users. So we built a Grafana app to visualise data from Cloudflare DNS Analytics. Finally, we made it available in your Cloudflare dashboard analytics. Over time we’re going to add new data sources, dimensions, and other useful ways how to visualise your data from Cloudflare. Let us know what you’d like to see next.

Does solving these kinds of technical and operational challenges excite you? Cloudflare is always hiring for talented specialists and generalists within our Engineering, Technical Operations and other teams.

Categories: Technology

Colombo, Sri Lanka: Six million Internet properties now faster for six million Internet users

Tue, 09/05/2017 - 06:13


We are excited to add four new data centers this week to Cloudflare's growing network, beginning with Colombo, Sri Lanka. This deployment is our 112th data center globally, and our 38th in Asia.

Faster Performance


CC BY-NC-ND 2.0 image by Pavel Dobrovsky

Six million Internet properties using Cloudflare are now even faster across the island country of Sri Lanka. Previously, local visitors to Cloudflare customers were served out of our Singapore or Dubai data centers.

Latency (ms) decreases 4x to Cloudflare customers. Source: Cedexis

Sri Lanka added over one million Internet users in the past year alone. At ~30% Internet penetration, there is considerable room to grow.

Next Three Cities

Our deployments to be revealed later this week will provide additional redundancy to existing facilities in North America and Africa.

If you enjoy the idea of helping build one of the world's largest networks, come join our team!

Categories: Technology

Anonymity and Abuse Reports

Sun, 07/05/2017 - 20:35

Last Thursday, ProPublica published an article critiquing our handling of some abuse reports that we receive. Feedback from the article caused us to reevaluate how we handle abuse reports. As a result, we've decided to update our abuse reporting system to allow individuals reporting threats and child sexual abuse material to do so anonymously. We are rolling this change out and expect it to be available by the end of the week.

I appreciate the feedback we received. How we handle abuse reports has evolved over the last six and a half years of Cloudflare's history. I wanted to take this opportunity to walk through some of the rationale that got us to this point and caused us to have a blindspot to the case that was highlighted in the article.

What Is Cloudflare?

Cloudflare is not a hosting provider. We do not store the definitive copy of any of the content that someone may want to file an abuse claim about. If we terminate a customer it doesn’t make the content go away. Instead, we are more akin to a specialized network. One of the functions of the network that we provide is to add security to the content providers that use us. Part of doing that inherently involves hiding the location of the actual hosting provider. If we didn't do this, a malicious attacker could simply bypass Cloudflare by attacking the host directly.

That created an early question on what we should do when someone reported abusive content that was passing through our network. The first principle was we believed it was important for us to not stand in the way of valid abuse reports being submitted. The litmus test that we came up with was that the existence of Cloudflare ideally shouldn't make it any harder, or any easier, to report and address abuse.

Mistakes of Early Abuse Reporting

The majority (83% over the last week) of the abuse reports that we get involve allegedly copyrighted material transiting our network. Our early abuse policy specified that if we received an abuse report alleging copyrighted material we'd turn over the IP address of the hosting provider so the person filing the abuse report could report the abuse directly.

It didn't take long for malicious attackers to realize this provided an effective way to bypass our protections. They would submit a fake report alleging some image on a legitimate site had been illegally copied, we'd turn over the IP address of our customer, and they'd attack it directly. Clearly that wasn't a workable model.

As a result, we revised our policy to instead act as a proxy for abuse reports that were submitted to us. If a report was submitted then we'd proxy the report through and forward it to the site owner as well as the site's host. We provided the contact information so the parties could address the issue between themselves.

While we have a Trust & Safety team that is staffed around the clock, for the most part abuse handling is automated. Various firms that specialize in finding and taking down copyrighted material generate such a flood, often submitting hundreds of reports for the same allegedly copyrighted item, that manual review of every report would be infeasible.

Violent Threats and Child Sexual Abuse

We've always treated reports of violent threats and child sexual abuse material with additional care. Understandably, from the perspective of the individuals in the ProPublica article, it seems callus and absurd that we would ever forward these reports to the site owner. However, we had a different perspective.

The vast majority of times that violent threats or child sexual abuse material were reported to us occurred on sites that were not dedicated to those topics. Imagine a social network like Facebook was a Cloudflare customer. Somewhere on the site something was posted that included a violent threat. That post was then reported to Cloudflare as the network that sits in front of the Facebook-like site.

In our early days, it seemed reasonable and responsible to pass the complaint on to the Facebook-like customer who could then follow up directly. That also met the litmus test of being what would happen if Cloudflare didn't exist. What the policy didn't account for was site owners who could not be trusted to act responsibly with abuse reports including contact information.

Anonymous Reporting

Beginning in 2014, we saw limited, but very concerning, reports of retaliation based on submitted abuse reports. As a result, we adjusted our process to make it so complaints about violent threats and child sexual abuse material would be sent only to the host, not to the site owner.

We’ve confirmed that in the cases reported to the site mentioned in the ProPublica article we followed this procedure. That change largely addressed the problem of people reporting abuse getting harassed. What we didn’t anticipate is that some hosts would themselves pass the full complaint, including the reporter’s contact information, on to the site owner. We assume this is what happened in the ProPublica cases.

Another change we made in 2015 was to clarify exactly what would happen when someone submitted a report by adding disclaimers to our abuse form. These disclaimers appear in multiple places throughout the abuse submission flow:

“Cloudflare will forward all abuse reports that appear to be legitimate to the responsible hosting provider and to the website owner.” "By submitting this report, you consent to the above information potentially being released by Cloudflare to third parties such as the website owner, the responsible hosting provider, law enforcement, and/or entities like Chilling Effects."

In a world without Cloudflare, if you wanted to anonymously report something, you would use a disposable email and a fake name and submit a report to the site's hosting provider or the site itself. We didn't do anything to check that the contact information used in reports was valid so we assumed, with the disclaimer in place, if people wanted to submit reports anonymously they'd do the same thing as they would have if Cloudflare didn't exist.

That was a bad assumption. As the ProPublica article made clear, many people did not read or understand the disclaimer and were surprised that we forwarded their full abuse report to the host who then, in some cases, could forward it to the site owner.

Determining Bad Actors

In reevaluating our policy a key question was when it is appropriate to pass along the full report and when it is not. Again, from the perspective of the author of the ProPublica article, that may seem like an easy distinction. The reality is that requiring an individual working on our Trust & Safety team understand the nature of every site that is on Cloudflare is untenable. Moreover, adding more human intervention that slows down the process of reporting abuse, especially in cases of violent threats and child sexual abuse material, where time may be of the essence, strikes us as a step backward.

Instead, we took the suggestions of many of the comments we received and are implementing a policy where reporters of these types of abuse can choose to submit them and not have their contact information included in what we forward. The person making the abuse report seems in the best position to judge whether or not they want their information to be relayed. Making this change requires some engineering work on our part, but we have prioritized it. By the end of this week, someone submitting an abuse report for one of these categories will have the choice of whether to do so anonymously.

Ongoing Improvements

We are under no illusion that this latest iteration of our abuse process is perfect. In fact, we already have concerns about challenges the new system will create. Anonymous reporting opens a new vector for malicious actors to submit false reports and harass Cloudflare customers. In addition, for responsible Cloudflare customers who want to act on reports, anonymous reports may make it more difficult for them to gather more information from the reporter which may make it more difficult for well-informed action to be taken to address the issue.

We appreciate the feedback on where our previous process broke down. As new problems arise, we anticipate that we'll continue to need to make changes to how we handle abuse reports.

Final Thoughts on Censoring the Internet

While we clearly had a significant blindspot in how we handled one type of abuse reports, we remain committed to our belief that it is not Cloudflare's role to make determinations on what content should and should not be online. That belief comes from a number of principles.

Cloudflare is more akin to a network than a hosting provider. I'd be deeply troubled if my ISP started restricting what types of content I can access. As a network, we don't think it's appropriate for Cloudflare to be making those restrictions either.

That is not to say we support all the content that passes through Cloudflare's network. We, both as an organization and as individuals, have political beliefs and views of what is right and wrong. There are institutions — law enforcement, legislatures, and courts — that have a social and political legitimacy to determine what content is legal and illegal. We follow the lead of those organizations in all the jurisdictions we operate. But, as more and more of the Internet sits behind fewer and fewer private companies, we're concerned that the political beliefs and biases of those organizations will determine what can and cannot be online.

If you're interested, I gave a talk a few years ago about how we think about our role in policing online content. It's about an hour long, but if you're interested in the topic, I encourage you to watch it in order to better understand our perspective.




From time to time an organization will sign up for Cloudflare that we find revolting because they stand for something that is the opposite of what we think is right. Usually, those organizations don't pay us. Every once in awhile one of them does. When that happens it's one of the greatest pleasures of my job to quietly write the check for 100% of what they pay us to an organization that opposes them. The best way to fight hateful speech is with more speech.

I appreciate the feedback on how we can improve our abuse process. We are implementing the changes that were recommended. They take engineering, so they aren't available immediately, but will be live by the end of this week. We continue to iterate and improve on our mission of helping build a better Internet.

Categories: Technology

Meet The Brand New DNS Analytics Dashboard

Fri, 05/05/2017 - 15:55

Have you noticed something new in your Cloudflare analytics dashboard this morning? You can now see detailed DNS analytics for your domains on Cloudflare.

If you want to skip to the punch and start exploring, go check it out here. Otherwise, hop on the DNS magic school bus - and let us show you all the neat stats in your now-available DNS analytics.

DNS analytics dashboard: What does it know? Does it know things? Let’s find out.

At the top of the DNS analytics dashboard you can see your DNS traffic health. This “Queries by Response Codes” graph breaks down queries by what response code Cloudflare DNS answered to the visitor. Like HTTP response codes, DNS response codes give an indication of what is happening behind the scenes. Mostly you will just see NOERROR, the HTTP 200 of DNS response codes, and NXDOMAIN, the HTTP 404 of DNS response codes. NXDOMAIN is particularly interesting - what are people querying for that doesn’t exist?

If you are an enterprise customer and you want to know what all the NXDOMAIN queries are, just scroll down a little bit where we show you the top queries for your domain and top queries for your domain for DNS records that don’t exist (aka top NXDOMAIN queries).

If you are curious then where all of these NXDOMAIN queries are coming from, all Pro, Business and Enterprise plan customers can scroll down a little bit further to the geography section where we show you the breakdown of where your queries come from and also where your queries returning NXDOMAIN come from.

If you are an Enterprise customer and want to dive in deeper just for one or a few names, you can filter down the entire dashboard by hostname. You can even filter the dashboard by names that don’t exist in your DNS records so you can explore traffic for misconfigured records resolvers are looking for that don’t exist.

Lastly, back up at the top, business and enterprise customers can see a breakdown of traffic by record type. If your domain has DNSSEC enabled, you’ll even see queries for DNSKEY records, meaning DNS resolvers that are DNSSEC-aware are asking Cloudflare for your DNSSEC keys to verify DNS records are untampered with.

If you like what you’re seeing and wish you could use the data to piece together your own dashboards? You can. We’ve developed a Grafana plugin you can use to build your own dashboards and an API you can use to create your own tools. Plus the API gives you some added neat information like the distribution of queries over IPv6 vs IPv4 and UDP vs TCP, as well as the distribution of answers by query and response size. (DNS people rejoice -- now you can finally see if there’s a difference in query size between IPv6 and IPv4, and what is the breaking point where Cloudflare DNS answers switch from UDP to TCP.)

What about DNS Firewall and CNAME setup?

DNS Firewall customers, you’re next! Stay tuned! We have a really cool dashboard coming up for you that shows latency and errors on a per origin basis. We can’t wait for you to see it. In the meantime, you can see all your data through the Grafana plugin and API.

If you use Cloudflare but don’t see traffic in the DNS analytics dashboard, it’s because you use Cloudflare through a CNAME setup where your DNS is hosted outside of Cloudflare. Your DNS isn’t on us, so we can’t give you cool DNS analytics. If you’re interested in moving your DNS over to Cloudflare to try out DNS analytics and the newly announced Cloudflare Load Balancing, give our team a holler, they will swap your setup for you.

Let’s get started!

We are excited for you to get started digging into your DNS traffic. Let us know what you think by sending your feedback to dns-analytics-feedback@cloudflare.com. Cloudflare DNS team eagerly awaits your feedback :). Now go on vacation, we’ll look after your DNS!

Categories: Technology

Introducing the new Cloudflare Community Forum

Wed, 03/05/2017 - 22:02

Cloudflare’s community of users is vast. With more than 6 million domains registered, our users come in all shapes and sizes and are located all over the world. They can also frequently be found hanging out all around the web, from social media platforms, to Q&A sites, to any number of personal interest forums. Cloudflare users have questions to ask and an awful lot of expertise to share.

It’s with that in mind that we wanted to give Cloudflare users a more centralized location to gather, and to discuss all things Cloudflare. So we have launched a new Cloudflare Community at community.cloudflare.com.

Who is this community for?

It's for anyone and everyone who uses Cloudflare. Whether you are adding your first domain and don’t know what a name server is, or you are managing 1,000s of domains via API, or you are somewhere in between. In the Cloudflare Community you will be able to find tips, tricks, troubleshooting guidance, and recommendations.

We also think this will be a great way to get feedback from users on what’s working for them, what isn’t, and ways that we can make Cloudflare better. There will even be opportunities to participate in early access programs for new and evolving features.

How do I access it?

Anyone can visit community.cloudflare.com and look around, soaking in all the information and expertise available, but in order to post questions or comments you must have a Cloudflare account. We wanted to keep this forum focused on using and improving Cloudflare, and not about questions from people who visit sites that use Cloudflare services. When users visit the forum and try to sign-in for the first time they will go through the usual Cloudflare login process, but there will be an added step for email verification. Once that is done users are good to go.

How to use the forum

We’ve started off with some broad categories like Performance, Security, Product Feedback, etc., and created some tags to cover some of the more specific products and topics. In time, we could expand to more dedicated categories around things like SSL or WAF, but we didn’t want to separate things off too much up front. There is also a Meta category where members can direct questions or suggestions about the Community. So, just put your topic in the area that you think is best, throw on a tag or two if necessary, and we’ll all figure this out together. But don’t forget to search and see if someone’s already discussing it.

We aren’t degrading our presence on social media or other popular discussion sites. But we are hopeful that having a centralized location will make it even easier for people who want to get together and discuss their Cloudflare experiences to find the conversations they are looking for. So be sure to visit community.cloudflare.com and say hello!

Categories: Technology

How eero mesh WiFi routers connect to the cloud

Wed, 03/05/2017 - 15:10

This is a guest post by Gabe Kassel, Product Manager for Embedded Software at eero.

Relying on a single wireless router to provide internet in every room of the home is like expecting a single light bulb to illuminate the entire house. It’s physics - WiFi radio waves don’t travel through walls or objects easily. The eero Home WiFi System is a new take on home connectivity, bucking the trend of one high-powered device in the center of the home. Instead, eero uses multiple access points that talk to each other via our proprietary mesh technology -- TrueMesh -- to spread coverage and a high throughput connection throughout a home.

eero’s hardware - its distributed access point system - solves the problem of spreading a consistent, stable WiFi signal in a home. But hardware is only part of the puzzle. On the backend of eero’s technology, we face different challenges: how do we build a highly available, high performance infrastructure that’s able to communicate with each eero device? We’ve discussed parts of our architecture previously, but we haven’t yet explored into how we use Cloudflare to eliminate one “single-point-of-failure” in our architecture.

How eeros interact with the cloud

eero devices are in near constant communication with our cloud. They send device diagnostic data to the cloud as well as data to support certain features in our mobile application. An example of this is when we added the ability for eero users to see real time data usage by device. While most products stream this data from a local server, both our user experience and security models rely on relaying that data through our cloud application to aggregate, analyze, and prevent direct local misuse from an unsecured web server.

In addition to streaming data to our cloud, eero devices need to continuously understand whether they have access to the internet. This specific architecture resulted in some special thinking.

First, we needed to ensure eero devices had a secondary source of truth (other than our cloud) to check if they could reach the internet. We originally thought about checking common resources like google.com and twitter.com, however we felt that had negative privacy implications since we didn’t own both sides of that request and connection. We wanted to prevent third-party sites from counting or analyzing eero installations to preserve privacy.

Second, we wanted to make sure that this secondary check was truly “out of band” of our current cloud infrastructure. Since the internet itself is highly distributed, we needed a service that mimicked that and was resilient to swaths of the internet being unreachable.

What Cloudflare allows us to do

Enter Cloudflare. Cloudflare provides CDN, security, and high availability, all in one tool. By storing a small file in Amazon S3, with Cloudflare at that domain, we’re able to serve a secondary internet check to hundreds of thousands of devices. Regardless of whether that connection can reach Amazon S3 and regardless of the health of the internet at-large, we have confidence Cloudflare will deliver the file.

We’re essentially using Cloudflare as a cloud canary. Since Cloudflare exposes a robust API, we’re able to track eero requests to this secondary internet check to create alerts when eeros are not able to resolve our cloud. And, as a bonus, since the file is 99.97% of the time served from Cloudflare’s edge, we can serve hundreds of gigabytes of the same tiny file and not spend anything with S3 to do it.

Cloudflare enables eero to provide an “out of band,” high availability, and low cost method to test internet availability from every eero device. We’re excited to help Cloudflare think about the specific needs of Internet of Things products, and continue to build out features that support scale across millions of devices. To learn more about how eero is solving connectivity in the home visit us at www.eero.com.

Categories: Technology

IoT Security Anti-Patterns

Tue, 02/05/2017 - 14:00
IoT Security Anti-Patterns

From security cameras to traffic lights, an increasing amount of appliances we interact with on a daily basis are internet connected. A device can be considered IoT-enabled when the functionality offered by it’s Embedded System is exposed through an internet connected API.

Internet-of-Things technologies inherit many attack vectors that appear in other internet connected devices, however low-powered hardware-centric nature of embedded systems presents them with unique security threats. Engineers building Internet-of-Things devices must take additional precautions to ensure they do not implement security anti-patterns when addressing new problems, this blog post will investigate four such anti-patterns that have been used by real Internet-of-Things devices.

IoT Security Anti-PatternsAtmel ATMEGA8 Microcontroller Wikimedia Commons - CC BY-SA 3.0

HTTP Pub/Sub

Every time your IoT-enabled alarm clock sounds, you may want it to tell your coffee machine to brew some coffee. In order to do this, your coffee machine may subscribe to messages published by your alarm clock. One such way of doing this is to implement the Publish/Subscribe Pattern within the API of the IoT devices, for this example let's assume our alarm clock and coffee machine communicate through HTTP.

In order to subscribe to messages from the alarm clock, the coffee machine sends a HTTP POST request to https://alarmclock/publish with the following body:

{ "on":"alarmSound", "to":"https:\/\/coffeemachine\/subscribe" }

This JSON request instructs the alarm clock on every “alarmSound” event to send a HTTP request to the coffee machine. Whilst this may seem a simple and effective way of implementing the Pub/Sub pattern in HTTP, this poses a significant security risk.

By not being able to validate if the receiver of the subscribed message wants the message or not, there is effectively a DDOS vulnerability. An attacker with the ability to set subscriptions on the alarm clock can effectively send HTTP messages to any device or internet property they want. If this is done across enough devices, a DDOS vulnerability is created.

Toast popping out of a toaster or a car driving across a road traffic sensor could be the trigger of a future large scale DDOS against a web property.

For further reading, the limitation of Pub/Sub patterns in IoT within the context of the MQTT protocol are discussed in: Limitations of the Pub/Sub pattern for cloud based IoT and their implications - Happ, D. and Wolisz, A. (2016).

IoT Device as TLS Server

As the embedded systems which power IoT devices have gained additional computational resources, more and more IoT developers have chosen to implement TLS to secure communications. Additionally cryptography libraries such as wolfSSL and mbedTLS, better suited to the performance requirements for embedded systems, have removed many of the engineering and performance complexities of adding TLS support to embedded systems.

One anti-pattern in implementing TLS on IoT devices is running devices themselves as web servers and using self-signed server-side certificates to encrypt such requests. Aside from the performance overhead of running web servers on embedded systems, they also pose a severe security risk by failing to maintain trust relationships.

In place of this; IoT devices can communicate through brokers; such brokers can act as web servers with signed TLS certificates installed on web servers which can be authenticated with devices using X.509 encryption. This can be done effectively by simply using the HTTP protocol. Like other kinds of message brokers, they will receive messages, perform validation, transformation and routing before sending them on by recipients.

Protocols like CoAP are specifically designed for brokerless Machine-to-Machine communication in low power environments, in it’s current form CoAP uses DTLS (RFC 4347) which supports pre-shared keys, raw public keys and X.509 certificate authentication.

Unencrypted Bootloader

Far from being in a datacenter, IoT devices can be left in insecure environments - as such extra precautions need to be taken to protect the memory on such a device.

Take the example of an IoT traffic counting device, installed in a locked cabinet at a roadside. A malicious adversary can break into the cabinet and steal the device with physical force. With the device in their possession they can extract software from the embedded system chip on the device to obtain the software running on it. By reverse engineering this software they can learn secrets in the memory of the onboard microcontroller.

When IoT devices can contain sensitive data in memory, locking them away may not always be sufficient. One solution to this problem is to encrypt the bootloaders on such devices - this provides a degree of cryptographic assurance that the secrets on the device will remain secret.

IoT Security Anti-Patterns Documentation from Atmel AVR231: AES Bootloader

There exists dedicated integrated Controller chips that allow for Hardware-based Key Storage at low cost; these chips can be used to prevent cloning and tampering of Embedded Systems. Examples include Atmel’s CryptoAuthentication chips and Microchip’s PIC24F GB2 microcontroller.

Other precautions should of course be taken, ensuring that keys are unique to a device and that revocation systems are in place, should keys be disclosed for whatever reason.

It is not merely enough to leave your secrets unencrypted on an embedded system in the hope no one will attempt to reverse engineer it. Taking appropriate security steps for the secrets your IoT device will store is vital.

Database-as-IPC

Instead of communicating over an abstract protocol like HTTP, developers may choose to use shortcuts by simply directly connecting to a database server to push data. Aside from being making permissions control harder, this opens up performance difficulties.

Lock contention is one such issue, when running UPDATE queries on rows other devices will end up waiting for locks on the database to be released before other queries can be done. Additionally, polling databases for changes can lead to IoT brokers becoming easily overloaded.

This Anti-Pattern can be mitigated by using a message broker service, exposed by a HTTP API. REST APIs are easier to cache, easier to scale and allow you to vary your database implementation independently of your API.

No matter how closed you think the usage of your IoT API is, take pride in it and seek to make it resilient to the forces of change later on. Abstracting your database away from your devices allows you to introduce a greater degree of security and also prevents other architectural headaches later on.

Conclusion

As the technology that powers the internet matures, so will the attacks it faces. IoT devices are set to see a unique set of security vulnerabilities, different to the set seen by other internet connected devices. As more IoT devices start to be unveiled, new security challenges will also unfold.

Categories: Technology

Introducing TLS with Client Authentication

Mon, 01/05/2017 - 16:58

In a traditional TLS handshake, the client authenticates the server, and the server doesn’t know too much about the client. However, starting now, Cloudflare is offering enterprise customers TLS with client authentication, meaning that the server additionally authenticates that the client connecting to it is authorized to connect.

TLS Client Authentication is useful in cases where a server is keeping track of hundreds of thousands or millions of clients, as in IoT, or in a mobile app with millions of installs exchanging secure information. For example, an IoT company can issue a unique client certificate per device, and then limit connections to their IoT infrastructure to only their devices by blocking connections where the client doesn’t present a certificate signed by the company’s certificate authority.

Or in the case of a mobile banking app, where the bank wants to ensure customers’ secure financial data doesn’t get stolen by bots spoofing their mobile app, they can issue a unique certificate to every app install and in the TLS handshake validate requests are coming from their mobile app. Client authentication is also useful for VPNs, enterprise networks or staging sites, where corporations and developers need to lock down connections to only laptops and phones owned by their employees and teammates.

You may be thinking - don’t we have API keys for that? But client certificates offer a layer of security that API keys cannot provide. If an API key gets compromised mid-connection, it can be reused to fire its own valid, trusted requests to the backend infrastructure. However, the private key of the client certificate is used to create a digital signature in every TLS connection, and so even if the certificate is sniffed mid-connection, new requests can’t be instantiated with it.

Handshakes With TLS Client Auth

In a handshake with TLS Client Authentication, the server expects the client to present a certificate, and sends the client a client certificate request with the server hello. Then in the key exchange in the next trip to the server, the client also sends its client certificate. The client certificate is then used to sign the TLS handshake and the digital signature is sent to the server for verification. You can see the whole handshake here:

TLS Client Authentication On The Edge

TLS Client Authentication can be CPU intensive to implement - it’s an additional cryptographic operation on every request. And if there’s a flood of invalid traffic, each request in that traffic flood kicks off a verification step. Companies can move the TLS client authentication to Cloudflare’s edge to offload the expensive verification.

If we are performing TLS Client Authentication for a company, the company sends us the root certificate(s) we should validate the client certificates against. Then the company can set TLS Client Authentication to one of two modes: enforce mode returns a 403 and optional custom JSON or HTML when the client certificate is invalid, and report mode forwards all requests to the origin, even if the certificate is invalid. Cloudflare will send a header including the status of the certificate (none, valid, invalid) and the certificate Subject Key Identifier (SKI) to the origin. For companies that use the client certificate for identification, Cloudflare can also forward any field of the client certificate as a custom header.

We have TLS Client Auth set up on a test domain for you to try, at auth.pizza.

If you curl auth.pizza, you’ll get back a 403 and a custom JSON error, telling you that you are not authorized.

curl https://auth.pizza -H 'Accept: application/json'

However, if you download this certificate: pizza.pem and curl the domain again using that certificate, you will be authenticated.

curl https://auth.pizza -H 'Accept: application/json' --cert pizza.pem

Note that this demo is on the full domain, auth.pizza, but you can also set client auth on a subdomain, such as client.auth.pizza.

Get Started

To use TLS client authentication, you must first set up PKI (Public Key Infrastructure) infrastructure to issue client certificates. If you are interested in running TLS client authentication but don’t have PKI infrastructure set up to issue client certificates, we have open sourced our PKI for you to use. Here is great documentation by our friends at CoreOS on how to use cfssl to issue client certificates. If you prefer not to run your own CA and rely on an established certificate authority, we have partnered with a few certificate authorities who can provide the client certificates for you.

If you are an enterprise customer and would like to get started using TLS client authentication with Cloudflare, reach out to your account team and we’ll help you get setup. If you are not yet an enterprise customer but are interested in trying out TLS client authentication, get in touch.

Within the next year, we’ll be adding TLS client authentication support for all Cloudflare plans. After all, using encryption to make the web more trusted is what we’re about. Stay tuned.

Categories: Technology

Upcoming Cloudflare events: Berlin May 5-7, Austin & Portland May 11.

Fri, 28/04/2017 - 01:40
 Berlin May 5-7, Austin & Portland May 11.

Attending JS Conf EU, CSS Conf, or OSCON in the next couple of weeks? Live in Berlin or Austin or Portland? Come over and join Cloudflare devs in the area at our upcoming events.

 Berlin May 5-7, Austin & Portland May 11. JS Conf EU 2016. Photo by Holger Blank.

In Berlin? Attending JS Conf EU or CSS Conf EU?

If you’re at JS Conf EU (May 6-7) or CSS Conf EU (May 5):

Just happen to be in Berlin? Tweet @qiqing and @IcyApril to come hang out with us in person.

 Berlin May 5-7, Austin & Portland May 11. Cloudflare Apps Preview. Visualize your app here.

In Austin? Attending OSCON?

Join the core developers of Cloudflare Apps for the inaugural Cloudflare meetup in Austin. It will feature an introduction by Zack Bloom (tech lead) to the new Cloudflare Apps including details of how apps get created, moderated, installed, and served to millions of users all over the world.

Bring your laptops!

What: Zack Bloom: “How to make a Cloudflare App: Developer Preview”
Where: Capital Factory, 701 Brazos St Ste 1601, Austin, TX
When: Thursday, May 11, 2017; 6:00 PM to 9:00 PM

RSVP on meetup.com >>

Core devs will be on-hand to help answer questions and serve as teaching assistants. T-shirts and dinner for every attendee. Special goodies for everyone who makes and tests an app.

 Berlin May 5-7, Austin & Portland May 11.

Portland: PDXnode Presentation Night

Eric Teffen Ellis (core dev) will be live-coding a Cloudflare App, followed by Q&A. Bring your laptops! New coders and new friends welcome. Say hi, make noise, and ask questions.

What: Eric Teffen Ellis: “Building apps for the web with Cloudflare.”
Where: Vacasa, 926 NW 13th Ave #200, Portland, OR
When: Thursday, May 11, 2017; 6:00 PM to 9:00 PM

RSVP for PDX Node on meetup.com >>

T-shirts and pizza for everyone.

Where next?

Want to host a Cloudflare event? Want to request a speaker to teach Cloudflare Apps to your group? Drop a line to community@cloudflare.com.

Categories: Technology

Introducing Cloudflare Orbit: A Private Network for IoT Devices

Thu, 27/04/2017 - 14:00

In October, we wrote about a 1.75M rps DDoS attack we mitigated on our network, launched by 52,467 unique IP’s, mostly hacked CCTV cameras.

We continued to see more IoT devices in DDoS attacks, and so we started to put together a security solution to protect the devices from becoming part of the botnet in the first place. Today we’re announcing it: Cloudflare Orbit.

PC-era security doesn’t work in IoT-era computing

As we talked to IoT companies, over and over again we heard the same thing. In the consumer electronics space, IoT manufacturers were telling us that they were shipping patches to their devices, but their end users didn’t always download and install them. (Reserve your judgment, how many times have you pressed ignore when your phone asked you to update its operating system?) In the industrial control, medical and automotive spaces, where devices are used in life-critical functions, we heard a different story. Even if someone wanted to apply a patch, it just wasn’t that easy. For example, even if the manager of a nuclear power plant wants to update software on their thermostats, shutting down operations long enough to do that means the update has to be scheduled.

This is if the device can be patched at all - most devices are patchable, but up to a point. When Jeep was shown to be vulnerable to remote code execution, Chrysler had to recall 1.4 million vehicles.

This model where the end user is responsible for the security of the overall device is a relic of the PC age, where users knew and understood that their computers could have vulnerabilities, and as their software vendors released patches ––on so called “Patch Tuesdays,” for example––users knew to go and download them.

PC security does not work for IoT. Consumers do not share that similar understanding that they need to update their toasters, lightbulbs and cars, because they’ve never needed to in the past. And since we will never write perfect code, we need a better way of securing devices without waiting for users to do it for us.

Introducing Orbit

Thinking about this challenge, we were reminded of when the ShellShock vulnerability was discovered, Cloudflare was able to automatically keep the requests that would trigger the vulnerability from reaching our customers. With Orbit, Cloudflare can do the same thing, only for devices. For example, when Jeep was shown to be vulnerable, instead of recalling 1.4 million vehicles, Fiat Chrysler could have patched the bug in all the vehicles with just a simple rule in Cloudflare's firewall restricting access to the vulnerable DBUS service listening on port 6667 of every Jeep.

Orbit sits one layer before the device and provides a shield of security, so even if the device is running past its operating system’s expiration date, Cloudflare protects it from exploits. And while devices may be seldom patched, the Cloudflare security team is shipping code every day, adding new firewall rules to Cloudflare’s edge. Think of it like changing IoT to I*oT — devices can still access the Internet, but only after passing through Cloudflare where malicious requests can be filtered.

For the last year, Cloudflare has been working with a number of IoT vendors to develop Orbit. Already more than 120 million IoT devices are safer behind Cloudflare’s network. Lockitron is one of the IoT companies using Cloudflare. “Keeping our products and customers secure is our primary concern,” says Paul Gerhardt, co-founder of Lockitron. “Cloudflare provides an extra layer of security that allows us to keep our devices continually updated and ahead of any vulnerabilities.”

Instead of writing and shipping a patch, IoT companies can write logic on Cloudflare’s edge, and write their own firewall rules to run on Cloudflare, and it updates the Cloudflare Orbit layer immediately, for all of their devices, without their users ever being so much as nudged to install something.

Plus, with requests going through Cloudflare, Cloudflare can compress transmitted data and speed up traffic, meaning less time is spent waiting on open connections and more time left in battery.

An Extra Layer of Authentication

A common challenge we heard from IoT device manufacturers was how to authenticate devices and know which connecting clients were authorized company devices, and which were bots only pretending to be. Starting today, Cloudflare now offers Enterprise domains TLS Client Authentication, a TLS handshake where the client authenticates the server’s certificate (as with any TLS handshake) and also the client has a certificate that the server authenticates.

Some IoT vendors already implement their own Client Authentication, but do so at the same origin servers that handle the rest of their IoT infrastructure. Not only is this computationally expensive, but any invalid traffic flood causes a burden on the whole server.

With Client Authentication on Cloudflare, Cloudflare’s edge handles the load of the TLS handshakes, validating the device client certificates and only sending the IoT infrastructure traffic from authorized devices.

What People Are Saying

“This approach of adding security to the network is extremely important for industrial manufacturers. Being able to patch vulnerabilities from the network rather than at the device level is a major shift in the way we secure IoT devices, and one that is completely necessary.” — Sam Cece, CEO of Swift Sensors, an industrial IoT company.

“Car controllers are IoT devices. Karamba Security hardens these devices and prevents cyberattacks with zero false positives to maintain driver and passenger safety. We view Cloudflare’s Orbit as a complementary solution that enables secure connectivity between the cars’ hardened controllers and the car company’s data center for trusted, over-the-air updates.” — Ami Dotan, CEO of Karamba Security, which is building secure platforms for smart automobiles.

“We are at the beginning of a new era in which a vast number of devices will be connecting to the Internet and security will play a critical role in the successful roll-out and adoption of IoT devices. Cloudflare’s Orbit adds another layer of defense that compliments other security measures such as strong hardware-based device security and helps ensure a safer Internet of Things." — Quinn Li, VP and global head of Qualcomm Ventures, the investment arm of Qualcomm Incorporated, the leading supplier of components for IoT devices.

"IoT devices create a distinct security challenge both because of the inability of most end users to update their software, as well as the cost that manufacturers bear if they release an update that bricks devices. This is even worse for legacy devices, many of which are effectively unpatchable. Cloudflare's Orbit provides a unique approach to help with these challenges, by deploying a defensive layer in the network where security updates can be safely made without end-user intervention or on-device changes." — Michael Freedman, professor of computer science at Princeton University and CTO of Timescale.

Get Started

Orbit is available now to all IoT device companies. To learn more or get started, visit the site, or get in touch. We’re excited to hear from you.

Categories: Technology

AES-CBC is going the way of the dodo

Fri, 21/04/2017 - 17:44

A little over a year ago, Nick Sullivan talked about the beginning of the end for AES-CBC cipher suites, following a plethora of attacks on this cipher mode.

Today we can safely confirm that this prediction is coming true, as for the first time ever the share of AES-CBC cipher suites on Cloudflare’s edge network dropped below that of ChaCha20-Poly1305 suites, and is fast approaching the 10% mark.

CC BY-SA 2.0 image by aesop

Over the course of the last six months, AES-CBC shed more than 33% of its “market” share, dropping from 20% to just 13.4%.

Ciphers

All of that share, went to AES-GCM, that currently encrypts over 71.2% of all connections. ChaCha20-Poly1305 is stable, with 15.3% of all connections opting for that cipher. Surprisingly 3DES is still around, with 0.1% of the connections.

The internal AES-CBC cipher suite breakdown as follows:

CBC

The majority of AES-CBC connections use ECDHE-RSA or RSA key exchange, and not ECDHE-ECDSA, which implies that we mostly deal with older clients.

RSA is also dying

In other good new, the use of ECDSA surpassed that of RSA at the beginning of the year. Currently more than 60% of all connections use the ECDSA signature.

Although 2048-bit RSA is not broken, it is generally considered less secure than 256-bit ECDSA, and is significantly slower to boot.

signatures

PFS is king

Last, but not least, 98.4% of all connections are PFS, using ECDHE for key exchange. That's up from 97.6% six months ago.

All in all we see the continuation of the positive trend on the web towards safer and faster cryptography. We believe this trend will continue with the finalization of TLS 1.3 later this year.

Categories: Technology

Pages

Automated Visitors