Blogroll Category: Technology

I read blogs, as well as write one. The 'blogroll' on this site reproduces some posts from some of the people I enjoy reading. There are currently 89 posts from the category 'Technology.'

Disclaimer: Reproducing an article here need not necessarily imply agreement or endorsement!

Cloudflare Access now supports RDP

CloudFlare - 5 hours 15 min ago

Last fall, the United States FBI warned organizations of an increase in attacks that exploit vulnerabilities in the Remote Desktop Protocol (RDP). Attackers stole sensitive data and compromised networks by taking advantage of desktops left unprotected. Like legacy VPNs, RDP configurations made work outside of the office corporate network possible by opening a hole in it.

Starting today, you can use Cloudflare Access to connect over RDP without sacrificing security or performance. Access enables your team to lock down remote desktops like you do physical ones while using your SSO credentials to authenticate each connection request.

Stronger passwords with identity provider integration

The FBI cited weak passwords and unrestricted port access to RDP ports as serious risks that led to the rise in RDP-based attacks. Cloudflare Access addresses those vulnerabilities by removing them altogether.

When users connect over RDP, they enter a local password to login to the target machine. However, organizations rarely manage these credentials. Instead, users set and save these passwords on an ad-hoc basis outside of the single sign-on credentials used for other services. That oversight leads to outdated, reused, and ultimately weak passwords.

Cloudflare Access integrates with the identity credentials your team already uses. Whether your organization uses Okta, or Azure AD, or another provider, your users will be prompted to authenticate with those credentials before starting any RDP sessions. Your team can enforce the same password strength and rotation requirements for RDP connections that you do for all of your critical tools. Need to revoke access? You can do it in a single location instead of chasing down who has credentials for every remote desktop.

Closed ports with Argo Tunnel

Another problem is the ubiquity of the RDP port. Most RDP connections listen on port 3389. Attackers can reasonably guess that number and attempt to reach desktops with misconfigured or overlooked firewall rules. Without those complex rules, remote desktops become as dangerous as a laptop left out on the street.

Cloudflare Access secures RDP ports and connections by relying on Argo Tunnel to lock down any attempts to reach the desktop. Argo Tunnel connects your machine to the Cloudflare network without the need for custom firewall or ACL configurations. Instead, Argo Tunnel ensures that all requests to that remote desktop route through Cloudflare. Once on the Cloudflare network, Access enforces the rules you need to lock down remote desktops.

Protecting your remote desktop

To begin, configure Argo Tunnel on the machine you need to secure by using cloudflared. cloudflared serves as an agent on the machine to open a secure connection from the desktop to the Cloudflare network. Once installed, you can run the following command to associate that connection to a hostname in your Cloudflare account:

$ cloudflared tunnel --hostname --url rdp://localhost:3389

That command will create a proxy to forward traffic to the hostname through port 3389. You can then create rules about who can reach this hostname in the Access tab of the Cloudflare dashboard.

Connecting over RDP

To reach a desktop behind Cloudflare Access, you’ll need the same cloudflared tool. First, install cloudflared on your device with the instructions here. You can then initiate an RDP connection with the following command:

$ cloudflared access rdp --hostname --url rdp://localhost:3389

Running that command will initiate an RDP connection through a proxy to reach the hostname of the machine you configured with Argo Tunnel. cloudflared will open a browser window where you can login with your team’s identity provider credentials. Once logged in, Access will return a token scoped to your user and the target application and store it on your device. You can then configure your RDP client to point to localhost:3389 and reach the protected desktop.

What’s next?

Starting today, all Access customers can start using RDP over Access by following this guide. Every Cloudflare plan includes five free Access seats; follow the link here to get started.

Categories: Technology

Drupal core - Highly critical - Remote Code Execution - SA-CORE-2019-003

Drupal Security - Wed, 20/02/2019 - 19:18
Project: Drupal coreDate: 2019-February-20Security risk: Highly critical 20∕25 AC:None/A:None/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Remote Code ExecutionCVE IDs: CVE-2019-6340Description: 

Some field types do not properly sanitize data from non-form sources. This can lead to arbitrary PHP code execution in some cases.

A site is only affected by this if one of the following conditions is met:

  • The site has the Drupal 8 core RESTful Web Services (rest) module enabled and allows PATCH or POST requests, or
  • the site has another web services module enabled, like JSON:API in Drupal 8, or Services or RESTful Web Services in Drupal 7. (Note: The Drupal 7 Services module itself does not require an update at this time, but you should apply other contributed updates associated with this advisory if Services is in use.)

Versions of Drupal 8 prior to 8.5.x are end-of-life and do not receive security coverage.

To immediately mitigate the vulnerability, you can disable all web services modules, or configure your web server(s) to not allow PUT/PATCH/POST requests to web services resources. Note that web services resources may be available on multiple paths depending on the configuration of your server(s). For Drupal 7, resources are for example typically available via paths (clean URLs) and via arguments to the "q" query argument. For Drupal 8, paths may still function when prefixed with index.php/.

Reported By: Fixed By: 
Categories: Technology

Font Awesome Icons - Critical - Remote Code Execution - SA-CONTRIB-2019-025

Drupal Contrib Security - Wed, 20/02/2019 - 17:56
Project: Font Awesome IconsDate: 2019-February-20Security risk: Critical 18∕25 AC:None/A:User/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Remote Code ExecutionDescription: 

This resolves issues described in SA-CORE-2019-003 for this module. Not all configurations are affected. See SA-CORE-2019-003 for details.

Categories: Technology

Translation Management Tool - Critical - Remote Code Execution - SA-CONTRIB-2019-024

Drupal Contrib Security - Wed, 20/02/2019 - 17:49
Project: Translation Management ToolDate: 2019-February-20Security risk: Critical 16∕25 AC:Basic/A:User/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Remote Code ExecutionDescription: 

This resolves issues described in SA-CORE-2019-003 for this module. Not all configurations are affected. See SA-CORE-2019-003 for details.

  • If you use the TMGMT module for Drupal 8.x, upgrade to TMGMT 8.x-1.7.
Categories: Technology

Paragraphs - Critical - Remote Code Execution - SA-CONTRIB-2019-023

Drupal Contrib Security - Wed, 20/02/2019 - 17:47
Project: ParagraphsDate: 2019-February-20Security risk: Critical 18∕25 AC:None/A:User/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Remote Code ExecutionDescription: 

This resolves issues described in SA-CORE-2019-003 for this module. Not all configurations are affected. See SA-CORE-2019-003 for details.

Categories: Technology

Video - Critical - Remote Code Execution - SA-CONTRIB-2019-022

Drupal Contrib Security - Wed, 20/02/2019 - 17:44
Project: VideoDate: 2019-February-20Security risk: Critical 18∕25 AC:None/A:User/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Remote Code ExecutionDescription: 

This resolves issues described in SA-CORE-2019-003 for this module. Not all configurations are affected. See SA-CORE-2019-003 for details.


Install the latest version:

  • If you use the Video module for Drupal 8, upgrade to Video 8.x-1.4
Categories: Technology

Metatag - Critical - Remote code execution - SA-CONTRIB-2019-021

Drupal Contrib Security - Wed, 20/02/2019 - 17:39
Project: MetatagDate: 2019-February-20Security risk: Critical 18∕25 AC:None/A:User/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Remote code executionDescription: 

This resolves issues described in SA-CORE-2019-003 for this module. Not all configurations are affected. See SA-CORE-2019-003 for details.

Categories: Technology

Link - Critical - Remote Code Execution - SA-CONTRIB-2019-020

Drupal Contrib Security - Wed, 20/02/2019 - 17:38
Project: LinkDate: 2019-February-20Security risk: Critical 18∕25 AC:None/A:User/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Remote Code ExecutionDescription: 

This resolves issues described in SA-CORE-2019-003 for this module. Not all configurations are affected. See SA-CORE-2019-003 for details.


Install the latest version:

  • If you use the Link module for Drupal 7.x, upgrade to Link 7.x-1.6
Categories: Technology

JSON:API - Highly critical - Remote code execution - SA-CONTRIB-2019-019

Drupal Contrib Security - Wed, 20/02/2019 - 17:37
Project: JSON:APIDate: 2019-February-20Security risk: Highly critical 22∕25 AC:None/A:None/CI:All/II:All/E:Theoretical/TD:AllVulnerability: Remote code executionDescription: 

This resolves issues described in SA-CORE-2019-003 for this module.


Install the latest version:

  • If you use the 2.x version of the JSON:API module for Drupal 8.x, upgrade to JSON:API 8.x-2.3
  • If you use the 1.x version of the JSON:API module for Drupal 8.x, upgrade to JSON:API 8.x-1.25
Categories: Technology

RESTful Web Services - Critical - Access bypass - SA-CONTRIB-2019-018

Drupal Contrib Security - Wed, 20/02/2019 - 17:35
Project: RESTful Web ServicesDate: 2019-February-20Security risk: Critical 19∕25 AC:None/A:User/CI:All/II:All/E:Theoretical/TD:DefaultVulnerability: Access bypassDescription: 

This resolves issues described in SA-CORE-2019-003 for this module. Not all configurations are affected. See SA-CORE-2019-003 for details.


Install the latest version:

  • If you use the RESTful Web Services module for Drupal 7.x, upgrade to restws 7.x-2.8
Categories: Technology

Join us for 5 serverless events in SF Bay Area this week

CloudFlare - Wed, 20/02/2019 - 17:20
Join us for 5 serverless events in SF Bay Area this week

Developer Week Bay Area is happening this week and Cloudflare engineers and developer relations team members are delivering several talks around the Bay. Join us in San Francisco and Oakland for the following talks. We’ll hope to see you soon.

Join us for 5 serverless events in SF Bay Area this week

WebAssembly on the Server, npm & genomics tools @ Cloudflare

We've partnered with the WebAssembly SF meetup group to curate three talks from Zack Bloom of Cloudflare, Laurie Voss of npm, and Robert Aboukhalil of Invitae.
Event Type: Meetup
Location: Cloudflare HQ, San Francisco, CA
Date: February 20, 2019

View Event Details & Register Here »

Join us for 5 serverless events in SF Bay Area this week

Serverless: An Inside Look

Cloudflare engineers are delivering three serverless talks in downtown Oakland: How Workers Work, Security: the Serverless Future, and Building a Serverless World (Map) with IoT and Workers.
Event Type: Meetup
Location: At My Sphere, Oakland, CA
Date: February 21, 2019

View Event Details & Register Here »

Join us for 5 serverless events in SF Bay Area this week

Developer Week Bay Area

Cloudflare will be at Developer Week Bay Area. Be sure to check out Single-Process Serverless, Building an Iot World (Map) with Serverless, and Make Your Existing Application Serverless talks.
Event Type: Conference
Location: Oakland Convention Center, Oakland, CA
Date: February 20-24, 2019

Register Here for a free Open conference ticket »

Categories: Technology

Stop the Bots: Practical Lessons in Machine Learning

CloudFlare - Wed, 20/02/2019 - 15:14
 Practical Lessons in Machine Learning

Bot-powered credential stuffing is a scourge on the modern Internet. These attacks attempt to log into and take over a user’s account by assaulting password forms with a barrage of dictionary words and previously stolen account credentials, with the aim of performing fraudulent transactions, stealing sensitive data, and compromising personal information.

At Cloudflare we’ve built a suite of technologies to combat bots, many of them grounded in Machine Learning. ML is a hot topic these days, but the literature tends to focus on improving the core technology — and not how these learning machines are incorporated into real-world organizations.

Given how much experience we have with ML (which we employ for many security and performance products, in addition to bot management), we wanted to share some lessons learned with regard to how this technology manifests in actual products.

 Practical Lessons in Machine Learning

There tend to be three stages every company goes through in the life cycle of infusing machine learning into their DNA. They are:

  • Business Intelligence
  • Standalone Machine Learning
  • Machine Learning Productization

These concepts are a little abstract — so let’s walk through how they might apply to a tangible field we all know and love: dental insurance.

Business Intelligence

Many companies already have some kind of business intelligence: an ability to sort, search, filter and perform rudimentary labeling on a large corpus of data. Business Intelligence is not machine learning per se, but it can serve as its foundation.

  • Imagine your friendly neighborhood dental insurance company, ACMEDental, regularly receives dental claims. For each claim, a trained insurance professional evaluates the incoming X-rays to confirm the dentist’s diagnosis — and marks the claim as accurate or inaccurate.
  • On the surface, this data provides actionable intelligence: an uptick in incorrect diagnoses from a particular dentist or region might warrant further investigation. But this data can be used for something much more interesting.
Standalone Machine Learning

The key to training a machine learning model is to compile a labeled data set: raw data paired with a descriptive label that tells the computer what it’s looking at. These data sets are often a natural byproduct of Business Intelligence, as we’re about to see.

  • Some ingenious engineers at ACMEDental notice that through their day-to-day operations they’ve compiled a large repository of X-rays, labeled either ‘accurate’ or ‘inaccurate’.
  • With several thousand labeled X-rays in hand, they decide to train a machine learning algorithm that can judge X-rays automatically. They use one of several open-source tools to do this using image recognition, yielding an ML algorithm that can scan an X-ray and categorize the claim with impressive accuracy.
Machine Learning Productization

Once a functioning algorithm is in-hand, there is the matter of turning it into a product. This generally involves collaboration between engineering, product design, and ultimately, business development and sales.

  • ACMEDental finds its new ML algorithm to be so effective that its product managers decide to offer it for license to X-ray machine manufacturers. By integrating the ACMEDental algorithm, a dentist could accelerate their workflow and reduce the likelihood of incorrect diagnoses. The machine could also communicate with the insurance company for instant claim approvals.
  • After productizing the algorithm for integration, ACMEDental’s business development team reaches out to manufacturers like Samsung to pursue a partnership.
  • The new X-ray machines, enhanced by ACMEDental’s algorithm, prove to be a hit on the market — allowing dentists to offload routine diagnostics to their assistants.

With this framework in mind let’s explore how we’re leveraging machine learning at Cloudflare.

To inform our machine learning models, we rely on data from the 13 million domains on Cloudflare’s network, which sees more than 660 billion requests per day serving more than 2.8 billion people per month. We employ this huge volume of data to address one of the most urgent security threats on the web: bot attacks.

 Practical Lessons in Machine LearningBusiness Intelligence

We recently analyzed one day’s worth of requests from across all of Cloudflare’s 13 million domains -- 660 billion requests -- and labeled each potential attack with a ‘bot score’ ranging from 0 to 100. Cloudflare already has an extensive array of tools which help inform this score, but in principle, we could manually label each datapoint as ‘bot’ or ‘not bot’, similar to the dentist example above.

One initial takeaway from our analysis was the geographical breakdown of attacks. Rudimentary bot-protection tools rely on blocking IP addresses from countries commonly associated with hostile traffic. But when we ranked the origin of bots across our network, it was obvious that this approach was untenable as most attacks originate from countries with huge volumes of legitimate traffic. A more sophisticated solution allows us to protect against bots without affecting real users.

 Practical Lessons in Machine Learning

Standalone Machine Learning

Having compiled a large dataset sampled from Cloudflare’s network, our team of ML experts developed a state-of-the-art model to predict automated credential stuffing and other bot attacks.

Training: We started with a trillion requests (compiled over multiple days) in our training set, each of them labeled with the aforementioned bot score — and analyzed their features on a CPU/GPU cluster to recognize trends in malicious traffic. We used CatBoost, a gradient boosting library similar to XGBoost.

Validation: Though we perform hundreds of independent validations tests, the ultimate validation is how many login attempts have been challenged with a Captcha, versus how many have solved a Captcha (a solved Captcha likely indicates a false positive).

Over the course of a week, we deployed our solution to 95 websites with WordPress login pages and found that we issued more than 660,000 challenges, of which only 0.32% were solved — meaning our algorithm detected bots with a 99.68% rate of True Positives.

While this work is just beginning, WordPress represents 32.5% of all websites worldwide, so this is a very meaningful step forward. Meanwhile, we found that more than 80% of requests to WordPress login pages are Credential Stuffing attacks, underscoring just how prevalent these attacks are.

Machine Learning Productization

Deployment: Once we became sufficiently confident in the accuracy of our algorithm, we deployed it as one of the many security features constantly running across the 165 server facilities worldwide that comprise Cloudflare’s network edge. Today, it evaluates over 660 billion requests per day — learning and improving from each additional attack observed on the network.

But it wasn’t long before we found ourselves asking: is knowing and stopping credential stuffing attacks in real-time enough? Can we do more?

We used our existing data and began thinking of how to develop an algorithm to predict which companies will be attacked next. This would allow us to proactively warn companies before an attack occurs and prepare them for the worst case scenario, even if they aren’t currently on Cloudflare’s network.

Training: We trained our Machine Learning Model with firmographic information like industry type, number of employees, and the revenue amount to predict the percentage of login attacks.

Results: Amongst the many interesting findings is that smaller companies have a higher fraction of attacks. Intuitively, this makes sense because the opportunistic attack traffic scanning the Internet is equal on all pages while the human traffic on popular sites outweighs the bot attacks. Thus even though small companies represent a smaller traffic volume, they're the most vulnerable to attack.

Do these findings only affect the 13 million Cloudflare websites with 2.8 billion visitors across 237 countries that we had in our model? Likely not. That means Cloudflare can begin thinking about helping all companies by proactively anticipating attacks based on their  risk profile.

Machine learning has been crucial to the development of our next generation of products, and we’ve only scratched the surface. We hope that this post is useful as you map out the trajectory of ML in your own organization: your road will be different, but hopefully you will see some familiar milestones along the way.

Cloudflare is actively developing its machine learning capabilities — if you’re interested in joining us or partnering with us on our mission please get in touch.

Categories: Technology


CloudFlare - Tue, 19/02/2019 - 16:00

We are working really hard to allow you to deploy Workers without having a Cloudflare domain. You will soon be able to deploy your Cloudflare Workers to a, which you can go claim now on!

Why are we doing this?

You may have read the announcement blog post for Workers (or one of the many tutorials and guest posts), and thought “let me give this a try!”. If you’re an existing Cloudflare customer, you logged into the dashboard, and found a new icon called “Workers”, paid $5 and were on your way. If you’re not, you clicked “Sign Up”, but instead of getting to create and deploy a Worker, we asked you for your domain (if you didn’t have one, we had you register one), and move your nameservers.

Since launch, we have had tons of people who wanted to build a new serverless project from scratch or just try Workers out, but found it difficult to get started. We want to make it easier for anyone to get started building and deploying serverless applications.

How did we get here?

The way you get started on Workers today reflects our journey as a company. Our mission to “help build a better Internet” has remained consistent, but the ways in which we can help do that have evolved. We didn’t start as a serverless company, we started as out as a way for you to secure and accelerate your web properties.


After the launch of Workers, it immediately became clear that the possibilities for Workers are endless. We truly believe that Workers are a revolutionary technology that will change how developers think about compute.


Our Serverless Mission

We are still on a mission to help build a better Internet. To us, a world with better Internet means a world where there is no distinction between writing code and deploying global applications.

That is, an independent developer can have access to the same scalable infrastructure as a multi-million dollar business to run their code on.

To remove any obstacles preventing developers from using Workers today, we’re going to allow developers to run Workers on subdomains of

As a part of Google’s TLD launch program, we were lucky enough to obtain, to allow customers to run their Workers on.


You can now go to and read about all the benefits of Workers that we’re really excited about.

Additionally, in you will be able to claim a subdomain (one per user) for you to run Workers on. After choosing your subdomain, you will be asked to verify your email. If you already have a Cloudflare account, please use the same email you used to sign up. We will also use the same email to notify you once we’re ready to let you deploy your first Worker to the subdomain you selected.

Note that is fully served using Cloudflare Workers. All of the following is done using Workers:

  • Serving the static site from storage
  • A/B testing our light vs. dark theme
  • The pre-registration API that allows you to claim a subdomain, validates your email, and reserves it for you (more on that soon).

Though we fully utilize Workers, no Workers were harmed in the making of this site.

We look forward to seeing the great things you build with Workers.
Check out to claim your subdomain, and read about the Workers platform.

Categories: Technology

Critical Release - PSA-2019-02-19

Drupal Public Service Announcements - Tue, 19/02/2019 - 14:11
Date: 2019-February-19Security risk: Highly critical 20∕25 AC:None/A:None/CI:All/II:All/E:Theoretical/TD:UncommonVulnerability: Critical ReleaseDescription: 

There will be a security release of 8.5.x and 8.6.x on February 20th 2019 between 1PM to 5PM America/New York (1800 to 2200 UTC). (To see this in your local timezone, refer to the Drupal Core Calendar) . The risk on this is currently rated at 20/25 (Highly critical) AC:None/A:None/CI:All/II:All/E:Theoretical/TD:Uncommon.

Not all configurations are affected. Reserve time on February 20 during the release window to determine whether your sites are affected and in need of an immediate update. Mitigation information will be included in the advisory.

Contributed module security updates may also be required.

If you are running Drupal 7, no core update is required, but you may need to update contributed modules if you are using an affected module. We are unable to provide the list of those modules at this time.

Neither the Security Team nor any other party is able to release any more information about this vulnerability until the announcement is made. The announcement will be made public at, over Twitter, and in email for those who have subscribed to our email list. To subscribe to the email list: log in on, go to your user profile page and subscribe to the security newsletter on the Edit » My newsletters tab.

Security release announcements will appear on the security advisory page.

Categories: Technology

SOCKMAP - TCP splicing of the future

CloudFlare - Mon, 18/02/2019 - 13:13
SOCKMAP - TCP splicing of the future

Recently we stumbled upon the holy grail for reverse proxies - a TCP socket splicing API. This caught our attention because, as you may know, we run a global network of reverse proxy services. Proper TCP socket splicing reduces the load on userspace processes and enables more efficient data forwarding. We realized that Linux Kernel's SOCKMAP infrastructure can be reused for this purpose. SOCKMAP is a very promising API and is likely to cause a tectonic shift in the architecture of data-heavy applications like software proxies.

SOCKMAP - TCP splicing of the future

Image by Mustad Marine public domain

But let’s rewind a bit.

Birthing pains of L7 proxies

Transmitting large amounts of data from userspace is inefficient. Linux provides a couple of specialized syscalls that aim to address this problem. For example, the sendfile(2) syscall (which Linus doesn't like) can be used to speed up transferring large files from disk to a socket. Then there is splice(2) which traditional proxies use to forward data between two TCP sockets. Finally, vmsplice can be used to stick memory buffer into a pipe without copying, but is very hard to use correctly.

Sadly, sendfile, splice and vmsplice are very specialized, synchronous and solve only one part of the problem - they avoid copying the data to userspace. They leave other efficiency issues unaddressed.

between avoid user-space memory zerocopy sendfile disk file --> socket yes no splice pipe <--> socket yes yes? vmsplice memory region --> pipe no yes

Processes that forward large amounts of data face three problems:

  1. Syscall cost: making multiple syscalls for every forwarded packet is costly.

  2. Wakeup latency: the user-space process must be woken up often to forward the data. Depending on the scheduler, this may result in poor tail latency.

  3. Copying cost: copying data from kernel to userspace and then immediately back to the kernel is not free and adds up to a measurable cost.

Many tried

Forwarding data between TCP sockets is a common practice. It's needed for:

  • Transparent forward HTTP proxies, like Squid.
  • Reverse caching HTTP proxies, like Varnish or NGINX.
  • Load balancers, like HAProxy, Pen or Relayd.

Over the years there have been many attempts to reduce the cost of dumb data forwarding between TCP sockets on Linux. This issue is generally called “TCP splicing”, “L7 splicing”, or “Socket splicing”.

Let’s compare the usual ways of doing TCP splicing. To simplify the problem, instead of writing a rich Layer 7 TCP proxy, we'll write a trivial TCP echo server.

It's not a joke. An echo server can illustrate TCP socket splicing well. You know - "echo" basically splices the socket… with itself!

Naive: read write loop

The naive TCP echo server would look like:

while data: data = read(sd, 4096) write(sd, data)

Nothing simpler. On a blocking socket this is a totally valid program, and will work just fine. For completeness I prepared full code here.

Splice: specialized syscall

Linux has an amazing splice(2) syscall. It can tell the kernel to move data between a TCP buffer on a socket and a buffer on a pipe. The data remains in the buffers, on the kernel side. This solves the problem of needlessly having to copy the data between userspace and kernel-space. With the SPLICE_F_MOVE flag the kernel may be able to avoid copying the data at all!

Our program using splice() looks like:

pipe_rd, pipe_wr = pipe() fcntl(pipe_rd, F_SETPIPE_SZ, 4096); while n: n = splice(sd, pipe_wr, 4096) splice(pipe_rd, sd, n)

We still need wake up the userspace program and make two syscalls to forward any piece of data, but at least we avoid all the copying. Full source.

io_submit: Using Linux AIO API

In a previous blog post about io_submit() we proposed using the AIO interface with network sockets. Read the blog post for details, but here is the prepared program that has the echo server loop implemented with only a single syscall.

SOCKMAP - TCP splicing of the future

Image by jrsnchzhrs By-Nd 2.0 SOCKMAP: The ultimate weapon

In recent years Linux Kernel introduced an eBPF virtual machine. With it, user-space programs can run specialized, non-turing-complete bytecode in the kernel context. Nowadays it's possible to select eBPF programs for dozens of use cases, ranging from packet filtering, to policy enforcement.

From Kernel 4.14 Linux got new eBPF machinery that can be used for socket splicing - SOCKMAP. It was created by John Fastabend at, exposing the Strparser interface to eBPF programs. Cilium uses SOCKMAP for Layer 7 policy enforcement, and all the logic it uses is embedded in an eBPF program. The API is not well documented, requires root and, from our experience, is slightly buggy. But it's very promising. Read more:

This is how to use SOCKMAP: SOCKMAP or specifically "BPF_MAP_TYPE_SOCKMAP", is a type of an eBPF map. This map is an "array" - indices are integers. All this is pretty standard. The magic is in the map values - they must be TCP socket descriptors.

This map is very special - it has two eBPF programs attached to it. You read it right: the eBPF programs live attached to a map, not attached to a socket, cgroup or network interface as usual. This is how you would set up SOCKMAP in user program:

sock_map = bpf_create_map(BPF_MAP_TYPE_SOCKMAP, sizeof(int), sizeof(int), 2, 0) prog_parser = bpf_load_program(BPF_PROG_TYPE_SK_SKB, ...) prog_verdict = bpf_load_program(BPF_PROG_TYPE_SK_SKB, ...) bpf_prog_attach(bpf_parser, sock_map, BPF_SK_SKB_STREAM_PARSER) bpf_prog_attach(bpf_verdict, sock_map, BPF_SK_SKB_STREAM_VERDICT)

Ta-da! At this point we have an established sock_map eBPF map, with two eBPF programs attached: parser and verdict. The next step is to add a TCP socket descriptor to this map. Nothing simpler:

int idx = 0; int val = sd; bpf_map_update_elem(sock_map, &idx, &val, BPF_ANY);

At this point the magic happens. From now on, each time our socket sd receives a packet, prog_parser and prog_verdict are called. Their semantics are described in the strparser.txt and the introductory SOCKMAP commit. For simplicity, our trivial echo server only needs the minimal stubs. This is the eBPF code:

SEC("prog_parser") int _prog_parser(struct __sk_buff *skb) { return skb->len; } SEC("prog_verdict") int _prog_verdict(struct __sk_buff *skb) { uint32_t idx = 0; return bpf_sk_redirect_map(skb, &sock_map, idx, 0); }

Side note: for the purposes of this test program, I wrote a minimal eBPF loader. It has no dependencies (neither bcc, libelf, or libbpf) and can do basic relocations (like resolving the sock_map symbol mentioned above). See the code.

The call to bpf_sk_redirect_map is doing all the work. It tells the kernel: for the received packet, please oh please redirect it from a receive queue of some socket, to a transmit queue of the socket living in sock_map under index 0. In our case, these are the same sockets! Here we achieved exactly what the echo server is supposed to do, but purely in eBPF.

This technology has multiple benefits. First, the data is never copied to userspace. Secondly, we never need to wake up the userspace program. All the action is done in the kernel. Quite cool, isn't it?

We need one more piece of code, to hang the userspace program until the socket is closed. This is best done with good old poll(2):

/* Wait for the socket to close. Let SOCKMAP do the magic. */ struct pollfd fds[1] = { {.fd = sd, .events = POLLRDHUP}, }; poll(fds, 1, -1);

Full code.

The benchmarks

At this stage we have presented four simple TCP echo servers:

  • naive read-write loop
  • splice
  • io_submit

To recap, we are measuring the cost of three things:

  1. Syscall cost
  2. Wakeup latency, mostly visible as tail latency
  3. The cost of copying data

Theoretically, SOCKMAP should beat all the others:

syscall cost waking up userspace copying cost read write loop 2 syscalls yes 2 copies splice 2 syscalls yes 0 copy (?) io_submit 1 syscall yes 2 copies SOCKMAP none no 0 copies Show me the numbers

This is the part of the post where I'm showing you the breathtaking numbers, clearly showing the different approaches. Sadly, benchmarking is hard, and well... SOCKMAP turned out to be the slowest. It's important to publish negative results so here they are.

Our test rig was as follows:

  • Two bare-metal Xeon servers connected with a 25Gbps network.
  • Both have turbo-boost disabled, and the testing programs are CPU-pinned.
  • For better locality we localized RX and TX queues to one IRQ/CPU each.
  • The testing server runs a script that sends 10k batches of fixed-sized blocks of data. The script measures how long it takes for the echo server to return the traffic.
  • We do 10 separate runs for each measured echo-server program.
  • TCP: "cubic" and NONAGLE=1.
  • Both servers run the 4.14 kernel.

Our analysis of the experimental data identified some outliers. We think some of the worst times, manifested as long echo replies, were caused by unrelated factors such as network packet loss. In the charts presented we, perhaps controversially, skip the bottom 1% of outliers in order to focus on what we think is the important data.

Furthermore, we spotted a bug in SOCKMAP. Some of the runs were delayed by up to whopping 64ms. Here is one of the tests:

Values min:236.00 avg:669.28 med=390.00 max:78039.00 dev:3267.75 count:2000000 Values: value |-------------------------------------------------- count 1 | 0 2 | 0 4 | 0 8 | 0 16 | 0 32 | 0 64 | 0 128 | 0 256 | 3531 512 |************************************************** 1756052 1024 | ***** 208226 2048 | 18589 4096 | 2006 8192 | 9 16384 | 1 32768 | 0 65536 | 11585 131072 | 1

The great majority of the echo runs (of 128KiB in this case) were finished in the 512us band, while a small fraction stalled for 65ms. This is pretty bad and makes comparison of SOCKMAP to other implementations pretty meaningless. This is a second reason why we are skipping 1% of worst results from all the runs - it makes SOCKMAP numbers way more usable. Sorry.

2MiB blocks - throughput

The fastest of our programs was doing ~15Gbps over one flow, which seems to be a hardware limit. This is very visible in the first iteration, which shows the throughput of our echo programs.

This test shows: Time to transmit and receive 2MiB blocks of data, via our tested echo server. We repeat this 10k times, and run the test 10 times. After stripping the worst 1% numbers we get the following latency distribution:

SOCKMAP - TCP splicing of the future

This charts shows that both naive read+write and io_submit programs were able to achieve 1500us mean round trip time for TCP echo server of 2MiB blocks.

Here we clearly see that splice and SOCKMAP are slower than others. They were CPU-bound and unable to reach the line rate. We have raised the unusual splice performance problems in the past, but perhaps we should debug it one more time.

For each server we run the tests twice: without and with SO_BUSYPOLL setting. This setting should remove the "wakeup latency" and greatly reduce the jitter. The results show that naive and io_submit tests are almost identical. This is perfect! BUSYPOLL does indeed reduce the deviation and latency, at a cost of more CPU usage. Notice that neither splice nor SOCKMAP are affected by this setting.

16KiB blocks - wakeup time

Our second run of tests was with much smaller data sizes, sending tiny 16KiB blocks at a time. This test should illustrate the "wakeup time" of the tested programs.

SOCKMAP - TCP splicing of the future

In this test the non-BUSYPOLL runs of all the programs look quite similar (min and max values), with SOCKMAP being the exception. This is great - we can speculate the wakeup time is comparable. Surprisingly, the splice has slightly better median time from others. Perhaps this can be explained by CPU artifacts, like having better CPU cache locality due to less data copying. SOCKMAP is again, slowest with worst max and median times. Boo.

Remember we truncated the worst 1% of the data - we artificially shortened the "max" values.


In this blog post we discussed the theoretical benefits of SOCKMAP. Sadly, we noticed it's not ready for prime time yet. We compared it against splice, which we noticed didn't benefit from BUSYPOLL and had disappointing performance. We noticed that the naive read/write loop and iosubmit approaches have exactly the same performance characteristics and do benefit from BUSYPOLL to reduce jitter (wakeup time).

If you are piping data between TCP sockets, you should definitely take a look at SOCKMAP. While our benchmarks show it's not ready for prime time yet, with poor performance, high jitter and a couple of bugs, it's very promising. We are very excited about it. It's the first technology on Linux that truly allows tahe user-space process to offload TCP splicing to the kernel. It also has potential for much better performance than other approaches, ticking all the boxes of being async, kernel-only and totally avoiding needless copying of data.

This is not everything. SOCKMAP is able to pipe data across multiple sockets - you can imagine a full mesh of connections being able to send data to each other. Furthermore it exposes the strparser API, which can be used to offload basic application framing. Combined with kTLS you can combine it with transparent encryption. Furthermore, there are rumors of adding UDP support. The possibilities are endless.

Recently the kernel has been exploding with eBPF innovations. It seems like we've only just scratched the surface of the possibilities exposed by the modern eBPF interfaces.

Many thanks to Jakub Sitnicki for suggesting SOCKMAP in the first place, writing the proof of concept and now actually fixing the bugs we found. Go strong Warsaw office!

Categories: Technology

Introducing Cf-Terraform

CloudFlare - Fri, 15/02/2019 - 20:02

Ever since we implemented support for configuring Cloudflare via Terraform, we’ve been steadily expanding the set of features and services you can manage via this popular open-source tool.

If you're unfamiliar with how Terraform works with Cloudflare, check out our developer docs.

We are Terraform users ourselves, and we believe in the stability and reproducibility that can be achieved by defining your infrastructure as code.

What is Terraform?

Terraform is an open-source tool that allows you to describe your infrastructure and cloud services (think virtual machines, servers, databases, network configurations, Cloudflare API resources, and more) as human-readable configurations.

Once you’ve done this, you can run the Terraform command-line tool and it will figure out the difference between your desired state and your current state, and make the API calls in the background necessary to reconcile the two.

Unlike other solutions, Terraform does not require you to run software on your hosts, and instead of spending time manually configuring machines, creating DNS records, and specifying Page Rules, you can simply run:

terraform apply

and the state described in your configuration files will be built for you.

Enter Cloudflare Terraforming

Terraform is a tremendous time-saver once you have your configuration files in place, but what do you do if you’re already a Cloudflare user and you need to convert your particular setup, records, resources and rules into Terraform config files in the first place?

Today, we’re excited to share a new open-source utility to make the migration of even complex Cloudflare configurations into Terraform simple and fast.

It’s called cf-terraforming and it downloads your Cloudflare setup, meaning everything you’ve defined via the Cloudflare dashboard and API, into Terraform-compliant configuration files in a few commands.

Getting up and running quickly

Cf-terraforming is open-source and available on Github now. You need a working Golang installation and a Cloudflare account with some resources defined. That’s it!

Let’s first install cf-terraforming, while also pulling down all dependencies and updating them as necessary:

$ go get -u

Cf-terraforming is a command line tool that you invoke with your Cloudflare credentials, some zone information and the resource type that you want to export. The output is a valid Terraform configuration file describing your resources.

To use cf-terraforming, first get your API key and Account ID from the Cloudflare dashboard. You can find your account id at the bottom right of the overview page for any zone in your account. It also has a quick link to get your API key as well. You can store your key and account ID in environment variables to make it easier to work with the tool:

export CLOUDFLARE_TOKEN=”<your-key>” export CLOUDFLARE_EMAIL=”<your-email>” export CLOUDFLARE_ACCT_ID=”<your-id>”

Cf-terraforming can create configuration files for any of the resources currently available in the official Cloudflare Terraform provider, but sometimes it’s also handy to export individual resources as needed.

Let’s say you’re migrating your Cloudflare configuration to Terraform and you want to describe your Spectrum applications. You simply call cf-terraforming with your credentials, zone, and the spectrum_application command, like so:

go run cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN --account $CLOUDFLARE_ACCT_ID spectrum_application

Cf-terraforming will contact the Cloudflare API on your behalf and define your resources in a format that Terraform understands:

resource"cloudflare_spectrum_application""1150bed3f45247b99f7db9696fffa17cbx9" { protocol = "tcp/8000" dns = { type = "CNAME" name = "" } ip_firewall = "true" tls = "off" origin_direct = [ "tcp://", ] }

You can redirect the output to a file and then start working with Terraform. First, ensure you are in the cf-terraforming directory, then run:

go run cmd/cf-terraforming/main.go --email $CLOUDFLARE_EMAIL --key $CLOUDFLARE_TOKEN --account $CLOUDFLARE_ACCT_ID spectrum_application >

The same goes for Zones, DNS records, Workers scripts and routes, security policies and more.

Which resources are supported?

Currently cf-terraforming supports every resource type that you can manage via the official Cloudflare Terraform provider:

Get involved

We’re looking for feedback and any issues you might encounter while getting up and running with cf-terraforming. Please open any issues against the GitHub repo.

Cf-terraforming is open-source, so if you want to get involved feel free to pick up an open issue or make a pull request.

Looking forward

We’ll continue to expand the set of Cloudflare resources that you can manage via Terraform, and that you can export via cf-terraforming. Be sure to keep and eye on the cf-terraforming repo for updates.

Categories: Technology

SEO Best Practices with Cloudflare Workers, Part 2: Implementing Subdomains

CloudFlare - Fri, 15/02/2019 - 17:09
Recap Implementing Subdomains

In Part 1, the merits and tradeoffs of subdirectories and subdomains were discussed.  The subdirectory strategy is typically superior to subdomains because subdomains suffer from keyword and backlink dilution.  The subdirectory strategy more effectively boosts a site's search rankings by ensuring that every keyword is attributed to the root domain instead of diluting across subdomains.

Subdirectory Strategy without the NGINX

In the first part, our friend Bob set up a hosted Ghost blog at that he connected to using a CNAME DNS record.  But what if he wanted his blog to live at to gain the SEO advantages of subdirectories?

A reverse proxy like NGINX is normally needed to route traffic from subdirectories to remotely hosted services.  We'll demonstrate how to implement the subdirectory strategy with Cloudflare Workers and eliminate our dependency on NGINX. (Cloudflare Workers are serverless functions that run on the Cloudflare global network.)

Back to Bobtopia

Let's write a Worker that proxies traffic from a subdirectory – – to a remotely hosted platform –  This means that if I go to, I should see the content of, but my browser should still think it's on

Configuration Options

In the Workers editor, we'll start a new script with some basic configuration options.

// keep track of all our blog endpoints here const myBlog = { hostname: "", targetSubdirectory: "/articles", assetsPathnames: ["/public/", "/assets/"] }

The script will proxy traffic from myBlog.targetSubdirectory to Bob's hosted Ghost endpoint, myBlog.hostname.  We'll talk about myBlog.assetsPathnames a little later.

 Implementing SubdomainsRequests are proxied from to (Uh oh... is because the hosted Ghost blog doesn't actually exist)

Request Handlers

Next, we'll add a request handler:

async function handleRequest(request) { return fetch(request) } addEventListener("fetch", event => { event.respondWith(handleRequest(event.request)) })

So far we're just passing requests through handleRequest unmodified.  Let's make it do something:

async function handleRequest(request) { ... // if the request is for blog html, get it if (requestMatches(myBlog.targetSubdirectory)) { console.log("this is a request for a blog document", parsedUrl.pathname) const targetPath = formatPath(parsedUrl) return fetch(`https://${myBlog.hostname}/${targetPath}`) } ... console.log("this is a request to my root domain", parsedUrl.pathname) // if its not a request blog related stuff, do nothing return fetch(request) } addEventListener("fetch", event => { event.respondWith(handleRequest(event.request)) })

In the above code, we added a conditional statement to handle traffic to myBlog.targetSubdirectory.  Note that we've omitted our helper functions here.  The relevant code lives inside the if block near the top of the function. The requestMatches helper checks if the incoming request contains targetSubdirectory.  If it does, a request is made to myBlog.hostname to fetch the HTML document which is returned to the browser.

When the browser parses the HTML, it makes additional asset requests required by the document (think images, stylesheets, and scripts).  We'll need another conditional statement to handle these kinds of requests.

// if its blog assets, get them if ([myBlog.assetsPathnames].some(requestMatches)) { console.log("this is a request for blog assets", parsedUrl.pathname) const assetUrl = request.url.replace(parsedUrl.hostname, myBlog.hostname); return fetch(assetUrl) }

This similarly shaped block checks if the request matches any pathnames enumerated in myBlog.assetPathnames and fetches the assets required to fully render the page.  Assets happen to live in /public and /assets on a Ghost blog.  You'll be able to identify your assets directories when you fetch the HTML and see logs for scripts, images, and stylesheets.

 Implementing SubdomainsLogs show the various scripts and stylesheets required by Ghost live in /assets and /public

The full script with helper functions included is:

// keep track of all our blog endpoints here const myBlog = { hostname: "", targetSubdirectory: "/articles", assetsPathnames: ["/public/", "/assets/"] } async function handleRequest(request) { // returns an empty string or a path if one exists const formatPath = (url) => { const pruned = url.pathname.split("/").filter(part => part) return pruned && pruned.length > 1 ? `${pruned.join("/")}` : "" } const parsedUrl = new URL(request.url) const requestMatches = match => new RegExp(match).test(parsedUrl.pathname) // if its blog html, get it if (requestMatches(myBlog.targetSubdirectory)) { console.log("this is a request for a blog document", parsedUrl.pathname) const targetPath = formatPath(parsedUrl) return fetch(`https://${myBlog.hostname}/${targetPath}`) } // if its blog assets, get them if ([myBlog.assetsPathnames].some(requestMatches)) { console.log("this is a request for blog assets", parsedUrl.pathname) const assetUrl = request.url.replace(parsedUrl.hostname, myBlog.hostname); return fetch(assetUrl) } console.log("this is a request to my root domain",, parsedUrl.pathname); // if its not a request blog related stuff, do nothing return fetch(request) } addEventListener("fetch", event => { event.respondWith(handleRequest(event.request)) }) Caveat

There is one important caveat about the current implementation that bears mentioning. This script will not work if your hosted service assets are stored in a folder that shares a name with a route on your root domain.  For example, if you're serving assets from the root directory of your hosted service, any request made to the home page will be masked by these asset requests, and the home page won't load.

The solution here involves modifying the blog assets block to handle asset requests without using paths.  I'll leave it to the reader to solve this, but a more general solution might involve changing myBlog.assetPathnames to myBlog.assetFileExtensions, which is a list of all asset file extensions (like .png and .css).  Then, the assets block would handle requests that contain assetFileExtensions instead of assetPathnames.


Bob is now enjoying the same SEO advantages as Alice after converting his subdomains to subdirectories using Cloudflare Workers.  Bobs of the world, rejoice!

Categories: Technology

SEO Best Practices with Cloudflare Workers, Part 1: Subdomain vs. Subdirectory

CloudFlare - Fri, 15/02/2019 - 17:09
Subdomain vs. Subdirectory: 2 Different SEO Strategies Subdomain vs. Subdirectory

Alice and Bob are budding blogger buddies who met up at a meetup and purchased some root domains to start writing.  Alice bought and Bob scooped up

Alice and Bob decided against WordPress because its what their parents use and purchased subscriptions to a popular cloud-based Ghost blogging platform instead.

Bob decides his blog should live at at – a subdomain of Alice keeps it old school and builds hers at – a subdirectory of

 Subdomain vs. Subdirectory

Subdomains and subdirectories are different strategies for instrumenting root domains with new features (think a blog or a storefront).  Alice and Bob chose their strategies on a whim, but which strategy is technically better?  The short answer is, it depends. But the long answer can actually improve your SEO.  In this article, we'll review the merits and tradeoffs of each. In Part 2, we'll show you how to convert subdomains to subdirectories using Cloudflare Workers.

Setting Up Subdomains and Subdirectories

Setting up subdirectories is trivial on basic websites.  A web server treats its subdirectories (aka subfolders) the same as regular old folders in a file system.  In other words, basic sites are already organized using subdirectories out of the box.  No set up or configuration is required.

 Subdomain vs. Subdirectory

In the old school site above, we'll assume the blog folder contains an index.html file. The web server renders blog/index.html when a user navigates to the subdirectory.  But Alice and Bob's sites don't have a blog folder because their blogs are hosted remotely – so this approach won't work.

On the modern Internet, subdirectory setup is more complicated because the services that comprise a root domain are often hosted on machines scattered across the world.

Because DNS records only operate on the domain level, records like CNAME have no effect on a url like – and because her blog is hosted remotely, Alice needs to install NGINX or another reverse proxy and write some configuration code that proxies traffic from to her hosted blog. It takes time, patience, and experience to connect her domain to her hosted blog.

 Subdomain vs. SubdirectoryA location block in NGINX is necessary to proxy traffic from a subdirectory to a remote host

Bob's subdomain strategy is the easier approach with his remotely hosted blog.  A DNS CNAME record is often all that's required to connect Bob's blog to his subdomain.  No additional configuration is needed if he can remember to pay his monthly subscription.

 Subdomain vs. SubdirectoryConfiguring a DNS record to point a hosted service at your blog subdomain

To recap, subdirectories are already built into simple sites that serve structured content from the same machine, but modern sites often rely on various remote services.  Subdomain set up is comparatively easy for sites that take advantage of various hosted cloud-based platforms.

Are Subdomains or Subdirectories Better for SEO?

Subdomains are neat. If you ask me, is more appealing than But if we want to make an informed decision about the best strategy, where do we look?  If we're interested in SEO, we ought to consult the Google Bot.

Subdomains and subdirectories are equal in the eyes of the Google Bot, according to Google itself.  This means that Alice and Bob have the same chance at ranking in search results.  This is because Alice's root domain and Bob's subdomain build their own sets of keywords.  Relevant keywords help your audience find your site in a search. There is one important caveat to point out for Bob:

A subdomain is equal and distinct from a root domain.  This means that a subdomain's keywords are treated separately from the root domain.

What does this mean for Bob?  Let's imagine is already a popular online platform for folks named Bob to seek kinship with other Bobs.  In this peculiar world, searches that rank for wouldn't automatically rank for because each domain has its own separate keywords.  The lesson here is that keywords are diluted across subdomains.  Each additional subdomain decreases the likelihood that any particular domain ranks in a given search.  A high ranking subdomain does not imply your root domain ranks well.

 Subdomain vs. SubdirectoryIn a search for "Cool Blog", suffers from keyword dilution. It doesn't rank because its blog keyword is owned by

Subdomains also suffer from backlink dilution.  A backlink is simply a hyperlink that points back to your site. Alice's attribution to a post on the etymology of Bob from does not help because the subdomain is treated separate but equal from the root domain.  If Bob used subdirectories instead, Bob's blog posts would feed the authority of and Bobs everywhere would rejoice.

 Subdomain vs. SubdirectoryThe authority of is increased when Alice links to Bob's interesting blog post, but the authority of is not affected.

Although search engines have improved at identifying subdomains and attributing keywords back to the root domain, they still have a long way to go.  A prudent marketer would avoid risk by assuming search engines will always be bad at cataloguing subdomains.

So when would you want to use subdomains?  A good use case is for companies who are interested in expanding into foreign markets.  Pretend is an American company whose website is in English.  Their English keywords won't rank well in German searches – so they translate their site into German to begin building new keywords on Erfolg!

Other use cases for subdomains include product stratification (think global brands with presence across many markets) and corporate internal tools (think productivity and organization tools that aren't user facing).  But unless you're a huge corporation or just finished your Series C round of funding, subdomaining your site into many silos is not helping your SEO.


If you're a startup or small business looking to optimize your SEO, consider subdirectories over subdomains.  Boosting the authority of your root domain should be a universal goal of any organization. The subdirectory strategy concentrates your keywords onto a single domain while the subdomain strategy spreads your keywords across multiple distinct domains. In a word, the subdirectory strategy results in better root domain authority. Higher domain authority leads to better search rankings which translates to more engagement.

Consider the multitude of disruptive PaaS startups with and  Why not switch to and to boost the authority of your root domain with all those docs searches and StackOverflow backlinks?

Want to Switch Your Subdomains to Subdirectories?

Interested in switching your subdomains to subdirectories without a reverse proxy? In Part 2, we'll show you how using Cloudflare Workers.

Categories: Technology

Solving Problems with Serverless – The Cloudflare LED Data Center Board, Part I

CloudFlare - Thu, 14/02/2019 - 20:00
Solving Problems with Serverless – The Cloudflare LED Data Center Board, Part I

You know you have a cool job when your first project lets you bring your hobby into the office.

That’s what happened to me just a few short weeks ago when I joined Cloudflare. The task: to create a light-up version of our Data Center map – we’re talking more than a hundred LEDs tied to the deployment state of each and every Cloudflare data center. This map will be a part of our booths, so it has to be able to travel; meaning we have to consider physical shipping and the ability to update the data when the map is away from the office. And the fun part – we are debuting it at SF Developer Week in late February (I even get to give a talk about it!) That gave me one week of software time in our San Francisco office, and a little over two and a half in the Austin office with the physical materials.

Solving Problems with Serverless – The Cloudflare LED Data Center Board, Part IWhat the final LEDs will look like on a map of the world.

So what does this have to do with Serverless? Well, let’s think about where and how this map will need to operate: This will be going to expo halls and conferences, and we want it to update to show our most current data center statuses for at least that event, if not updating once a day. But we don’t need to stay connected to the information store constantly-- nor should we expect to, over conference or expo WiFi.

Data Stored about Data Centers

The data stored about each data center has two distinct categories; data relevant to the data center itself, and data about how that data center should be rendered on the map. These are relatively simple data structures, however: for a data center, we want to store what city the data center is in, the latitude and longitude, and the status. We arbitrarily assign an ID integer to each data center, which we'll use to match this data with the data in the other store. We’re not going to pick and choose which data centers we want; just pull them all down and let the microcontroller figure out how to display them.

Speaking of, this is where the data store relevant to the display comes in. LEDs are on strands numbered 0-7, and are represented by an LED numbered 0-63. We need to store the ID of the data center we created for the first store, the strand number, and the LED number in the strand.

Both of these sets of data can be stored in a key-value store, with the ID number as the key and a JSON object representing either the data center or its representative LED on the map as the value. Because of this, coupled with the fact that we do not need to search or index this data, we decided to use Workers KV data stores to keep this information.

The Data Center and Data Center Map API

We needed two APIs around the data centers, and we needed them fast– both in the more immediate sense of having only a few weeks to complete the project, and in the sense that we needed the data to download relatively quickly over non-ideal internet situations. We also know this map will be traveling all over the world– we'd need the API to work and have at least a decent latency no matter where the map was geographically.

This is where the hundreds of LEDs comes in handy– each one represents a data center that we could deploy serverless Workers to. We can deploy the API to the data center before we leave from the comfort of the office, and it'd be ready for us when we hit the conference floor. Workers also, unsurprisingly, work really well with Workers KV data stores; allowing us to rapidly develop APIs around our data.

Our Software Architecture DiagramSolving Problems with Serverless – The Cloudflare LED Data Center Board, Part I

In the end, we ended up with this architecture diagram; 2 Workers KV data stores, and 2 serverless Workers; all of which can be deployed across the world in order to make sure the physical map has the updated data every time we head to a new show.

In the next post in this series, we'll take a look at the physical architecture of the sign:

Solving Problems with Serverless – The Cloudflare LED Data Center Board, Part I

We'll also take a look at the system we built that uses the architecture laid out in this post that consumes this data and turns it into an LED map – so keep an eye out for it next month!

Categories: Technology

Winning the Blackbird Battle

CloudFlare - Thu, 14/02/2019 - 18:03
Winning the Blackbird BattleUS Court of Appeals for the Federal CircuitWinning the Blackbird Battle

Frequent readers of the Cloudflare blog are aware of the efforts we’ve undertaken in response to our first encounter with a patent troll. We’re happy to report that on Wednesday, the U.S. Court of Appeals for the Federal Circuit issued an opinion affirming a lower court decision dismissing the case brought by Blackbird Tech. This is the last step in the process 1, we’ve won.

Winning the Blackbird Battle

In addition to vigorously opposing this case in court, we created and sponsored Project Jengo to push back against the incentives that empower patent trolls like Blackbird Tech. Now that the case is over, we will be wrapping up Project Jengo and will report back with a summary of the Project’s successes in the near future.

But before we move on from the litigation, I want to share a few reflections on this case.

We noted from the very beginning: “The infringement claim is not a close one … if the ‘335 patent is read broadly enough to cover our system (which shouldn’t happen), it would also cover any system where electronic communications are examined and redacted or modified.”

Winning the Blackbird Battle

Our initial observation, which we thought was obvious, was borne out. And we were able to prevail on our arguments as swiftly and cleanly as is possible in the federal court system. The U.S. District Court resolved the case on a motion to dismiss, meaning the court didn’t even consider the factual circumstances of the case, but looked at the face of the complaint and the language of the patent itself and determined that the claims in the patent were too vague to allow anyone to enforce it. It was so obvious to the court that Judge Chabbria’s decision was little more than a single page. You can read our discussion of the lower court decision in a previous blog.

Yet Blackbird appealed that dismissal to the U.S. Court of Appeals for the Federal Circuit, a specialized court based in Washington, DC that hears appeals of all patent cases. A panel of three judges from that court heard arguments on the appeal last Friday, but didn’t ask our attorney a single question about the substance of our argument on the abstractness of the patent. He sat down with almost half of his 15 minutes of argument time left because there was nothing more to say. Yesterday, just three business days after that hearing, the court affirmed the lower court’s decision in summary fashion, which means they didn’t even write about the claims or arguments, they just said “Affirmed” (see below).  

If it were a boxing match, it would have been called by the referee in the first minute of the first round after three knockdowns. It was easy and painless, right?  

Not at all.  

Blackbird filed this case in March 16, 2017. For nearly two years, anyone doing due diligence on Cloudflare might have had questions about whether there was a cloud over our rights to our technology. And we had to go through a lot of briefing, and the related legal expenses, to get to this point. Blackbird’s combined legal filings at the district court and appellate court amounted to more than 650 pages, our responsive briefs were more than 900 pages.

The two courts spent less than two pages describing a result that was obvious to them, but it took us two years of uncertainty and cost to get there. Federal court litigation doesn’t make anything easy. Even if Blackbird had won the case, it is not clear they would have been able to collect significant damages. Our allegedly infringing use was not a product or feature that we charged for or made money from – it was essentially posting interstitial messages for various errors. Even though we were able to win this case early in the legal process and keep our costs as low as possible, it’s possible we spent more money resolving this matter than Blackbird would have been able to collect from us after trial.

This is the dysfunction that makes patent trolling possible. It is why the option for a quick settlement, in the short term, is always so appealing to parties sued by patent trolls. It’s why we exerted efforts on Project Jengo to try and change the long-term calculus and help out others in the community who may soon find themselves in a similar predicament.  

A final note…

Anthony Garza and Steven Callahan of Charhon Callahan Robson & Garza, a litigation boutique in Dallas, are great lawyers. They provided exceptional counseling, perfect legal work, and strong arguments at every step of this process. In every hearing, I was extremely happy that Anthony was the lawyer on our side -- he demonstrated complete control of both the relevant legal authorities and the intricate details of the patent and its various claims. He had the advantage of being right, but he never left any doubt who was making the better argument.  My thanks for their hard work and guidance.

Winning the Blackbird BattleWinning the Blackbird BattleWinning the Blackbird Battle


[1] Yes, I am aware that Blackbird could seek discretionary review of this decision at the U.S. Supreme Court. But the Supreme Court accepts less than 5% of the cases presented to it, so there isn’t even a request to that court in most cases unless the case is particularly significant or reflects a disagreement among federal courts. I don’t reasonably expect they would take this case.

Categories: Technology


Subscribe to aggregator - Technology
Additional Terms