Blogroll: CloudFlare

I read blogs, as well as write one. The 'blogroll' on this site reproduces some posts from some of the people I enjoy reading. There are currently 62 posts from the blog 'CloudFlare.'

Disclaimer: Reproducing an article here need not necessarily imply agreement or endorsement!

Subscribe to CloudFlare feed CloudFlare
The Cloudflare Blog
Updated: 1 hour 54 min ago

Automating Cloudflare Tunnel with Terraform

Fri, 14/05/2021 - 14:00
Automating Cloudflare Tunnel with TerraformAutomating Cloudflare Tunnel with Terraform

Cloudflare Tunnel allows you to connect applications securely and quickly to Cloudflare’s edge. With Cloudflare Tunnel, teams can expose anything to the world, from internal subnets to containers, in a secure and fast way. Thanks to recent developments with our Terraform provider and the advent of Named Tunnels it’s never been easier to spin up.

Classic Tunnels to Named Tunnels

Historically, the biggest limitation to using Cloudflare Tunnel at scale was that the process to create a tunnel was manual. A user needed to download the binary for their OS, install/compile it, and then run the command cloudflared tunnel login. This would open a browser to their Cloudflare account so they could download a cert.pem file to authenticate their tunnel against Cloudflare’s edge with their account.

With the jump to Named Tunnels and a supported API endpoint Cloudflare users can automate this manual process. Named Tunnels also moved to allow a .json file for the origin side tunnel credentials instead of (or with) the cert.pem file. It has been a dream of mine since joining Cloudflare to write a Cloudflare Tunnel as code, along with my instance/application, and deploy it while I go walk my dog. Tooling should be easy to deploy and robust to use. That dream is now a reality and my dog could not be happier.

Automating Cloudflare Tunnel with TerraformOkay, so what?

The ability to dynamically generate a tunnel and tie it into a back end application(s) brings several benefits to users including: putting more of their Cloudflare config in code, auto-scaling resources, dynamically spinning up resources such as bastion servers for secure logins, and saving time from avoiding manually generating/maintaining tunnels.

Tunnels also allow traffic to connect securely into Cloudflare’s edge for only the particular account they are affiliated with. In a world where IPs are increasingly ephemeral, tunnels allow for a modern approach to tying your application(s) into Cloudflare. Putting automation around tunnels allows teams to incorporate them into their existing CI/CD (continuous improvement/continuous development) pipelines.

Most importantly, the spin up of an environment securely tied into Cloudflare can be achieved with some Terraform config and then by running terraform apply. I can then go take my pup on an adventure while my environment kicks off.

Why Terraform?

While there are numerous Infrastructure as Code tools out there, Terraform has an actively maintained Cloudflare provider. This is not to say that this same functionality cannot be re-created by making use of the API endpoint with a tool of your choice. The overarching concepts here should translate quite nicely. Using Terraform we can deploy Cloudflare resources, origin resources, and configure our server all with one tool. Let’s see what setting that up looks like.

Terraform Config

The technical bits of this will cover how to set up an automated Named Tunnel that will proxy traffic to a Google compute instance (GCP) which is my backend for this example. These concepts should be the same regardless of where you host your applications such as an onprem location to a multi-cloud solution.

With Cloudflare Tunnel’s Ingress Rules, we can use a single tunnel to proxy traffic to a number of local services. In our case we will tie into a Docker container running HTTPbin and the local SSH daemon. These endpoints are being used to represent a standard login protocol (such as SSH or RDP) and an example web application (HTTPbin). We can even take it a step further by applying a Zero Trust framework with Cloudflare Access over the SSH hostname.

The version of Terraform used in this example is 0.15.0. Please refer to the provider documentation when using the Cloudflare Terraform provider. Tunnels are compatible with Terraform version 0.13+.

cdlg at cloudflare in ~/Documents/terraform/blog on master $ terraform --version Terraform v0.15.0 on darwin_amd64 + provider v2.18.0 + provider v3.56.0 + provider v3.0.1 + provider v2.2.0

Here is what the Terraform hierarchy looks like for this setup.

cdlg at cloudflare in ~/Documents/terraform/blog on master $ tree . . ├── ├── ├── ├── ├── ├── server.tpl ├── terraform.tfstate ├── terraform.tfstate.backup ├── terraform.tfvars ├── terraform.tfvars.example ├── test.plan └── 0 directories, 12 files

We can ignore the files and terraform.tfvars.example for now. The files ending in .tf is where our Terraform configuration lives. Each file is designated to a specific purpose. For example, the file only contains the scope of the GCP server resources used with this deployment and the affiliated DNS records pointing to the tunnel on it.

cdlg at cloudflare in ~/Documents/terraform/blog on master $ cat # Instance information data "google_compute_image" "image" { family = "ubuntu-minimal-1804-lts" project = "ubuntu-os-cloud" } resource "google_compute_instance" "origin" { name = "test" machine_type = var.machine_type zone = tags = ["no-ssh"] boot_disk { initialize_params { image = data.google_compute_image.image.self_link } } network_interface { network = "default" access_config { // Ephemeral IP } } // Optional config to make instance ephemeral scheduling { preemptible = true automatic_restart = false } metadata_startup_script = templatefile("./server.tpl", { web_zone = var.cloudflare_zone, account = var.cloudflare_account_id, tunnel_id =, tunnel_name =, secret = random_id.argo_secret.b64_std }) } # DNS settings to CNAME to tunnel target resource "cloudflare_record" "http_app" { zone_id = var.cloudflare_zone_id name = var.cloudflare_zone value = "${}" type = "CNAME" proxied = true } resource "cloudflare_record" "ssh_app" { zone_id = var.cloudflare_zone_id name = "ssh" value = "${}" type = "CNAME" proxied = true }

This is a personal preference — if desired, the entire Terraform config could be put into one file. One thing to note is the usage of variables throughout the files. For example, the value of var.cloudflare_zone is populated with the value provided to it from the terraform.tfvars file. This allows the configuration to be used as a template with other deployments. The only change that would be necessary is updating the relevant variables, such as in the terraform.tfvars file, when re-using the configuration.

When using a credentials file (vs environment variables such as a .tfvars file) it is very important that this file is exempted from the version tracking tool. With git this is accomplished with a .gitignore file. Before running this example the terraform.tfvars.example file is copied to terraform.tfvars within the same directory and filled in as needed. The .gitignore file is told to ignore any file named terraform.tfvars to exempt the actual variables from version tracking.

cdlg at cloudflare in ~/Documents/terraform/blog on master $ cat .gitignore # Local .terraform directories **/.terraform/* # .tfstate files *.tfstate *.tfstate.* # Crash log files crash.log # Ignore any .tfvars files that are generated automatically for each Terraform run. Most # .tfvars files are managed as part of configuration and so should be included in # version control. # # example.tfvars terraform.tfvars # Ignore override files as they are usually used to override resources locally and so # are not checked in * * # Include override files you do wish to add to version control using negated pattern # # ! # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan # example: *tfplan* *tfplan* *.plan* *lock*

Now to the fun stuff! To create a Cloudflare Tunnel in Terraform we only need to set the following resources in our Terraform config (this is what populates the file).

resource "random_id" "argo_secret" { byte_length = 35 } resource "cloudflare_argo_tunnel" "auto_tunnel" { account_id = var.cloudflare_account_id name = "zero_trust_ssh_http" secret = random_id.argo_secret.b64_std }

That’s it.

Technically you could get away with just the cloudflare_argo_tunnel resource, but using the random_id resource helps with not having to hard code the secret for the tunnel. Instead we can dynamically generate a secret for our tunnel each time we run Terraform.

Let’s break down what is happening in the cloudflare_argo_tunnel resource: we are passing the Cloudflare account ID (via the var.cloudflare_account_id variable), a name for our tunnel, and the dynamically generated secret for the tunnel, which is pulled from the random_id resource. Tunnels expect the secret to be in base64 standard encoding and at least 32 characters.

Using Named Tunnels now gives customers a UUID (universal unique identity) target to tie their applications to. These endpoints are routed off an internal domain to Cloudflare and can only be used with zones in your account, as mentioned earlier. This means that one tunnel can proxy multiple applications for various zones in your account, thanks to Cloudflare Tunnel Ingress Rules.

Now that we have a target for our services, we can create a tunnel/applications in the GCP instance. Terraform has a feature called a templatefile function that allows you to pass input variables as local variables (i.e. what the server can use to configure things) to an argument called metadata_startup_script.

resource "google_compute_instance" "origin" { ... metadata_startup_script = templatefile("./server.tpl", { web_zone = var.cloudflare_zone, account = var.cloudflare_account_id, tunnel_id =, tunnel_name =, secret = random_id.argo_secret.b64_std }) }

This abbreviated section of the google_compute_instance resource shows a templatefile using 5 variables passed to the file located at ./server.tpl. The file server.tpl is a bash script within the local directory that will configure the newly created GCP instance.

As indicated earlier, Named Tunnels can make use of a JSON credentials file instead of the historic use of a cert.pem file. By using a templatefile function pointing to a bash script (or cloud-init, etc…) we can dynamically generate the fields that populate both the cert.json file and the config.yml file used for Ingress Rules on the server/host. Then the bash script can install cloudflared as a system service, so it is persistent (i.e it can come back up after the machine is rebooted). Here is an example of this.

wget sudo dpkg -i cloudflared-stable-linux-amd64.deb mkdir ~/.cloudflared touch ~/.cloudflared/cert.json touch ~/.cloudflared/config.yml cat > ~/.cloudflared/cert.json << "EOF" { "AccountTag" : "${account}", "TunnelID" : "${tunnel_id}", "TunnelName" : "${tunnel_name}", "TunnelSecret" : "${secret}" } EOF cat > ~/.cloudflared/config.yml << "EOF" tunnel: ${tunnel_id} credentials-file: /etc/cloudflared/cert.json logfile: /var/log/cloudflared.log loglevel: info ingress: - hostname: ${web_zone} service: http://localhost:8080 - hostname: ssh.${web_zone} service: ssh://localhost:22 - hostname: "*" service: hello-world EOF sudo cloudflared service install sudo cp -via ~/.cloudflared/cert.json /etc/cloudflared/ cd /tmp sudo docker-compose up -d && sudo service cloudflared start

In this example, a heredoc is used to fill in the variable fields for the cert.json file and another heredoc is used to fill in the config.yml (Ingress Rules) file with the variables we set in Terraform. Taking a quick look at the cert.json file we can see that the Account ID is provided to it which secures the tunnel to your specific account. The UUID  of the tunnel is then passed in along with the name that was assigned in the tunnel’s name argument. Lastly the 35 character secret is then passed to the tunnel. These are the necessary parameters to get our tunnel spun up against Cloudflare’s edge.

The config.yml file is where we set up the Ingress Rules for the Cloudflare Tunnel. The first few lines tell the tunnel which UUID to attach to, where the credentials are on the OS, and where the tunnel should write logs to. The log level of info is good for general use but for troubleshooting debug may be needed.

Next the first - hostname: line says any requests bound for that particular hostname need to be proxied to the service (HTTPbin) running at localhost port 8080. Following that the SSH target is defined and will proxy requests to the local SSH port. The next hostname is interesting in that we have a wildcard character. This functionality allows other zones or hostnames on the Account to point to the tunnel without being explicitly defined in Ingress Rules. The service that will respond to these requests is a built in hello world service the tunnel provides.

Pretty neat, but what else can we do? We can block all inbound networking to the server and instead use Cloudflare Tunnel to proxy the connections to Cloudflare’s edge. To safeguard the SSH hostname an Access policy can be applied over it.

SSH and Zero Trust

The Access team has several tutorials on how to tie your policies into Terraform. Using this as a guide we can create the Access related Terraform resources for the SSH endpoint.

# Access policy to apply zero trust policy over SSH endpoint resource "cloudflare_access_application" "ssh_app" { zone_id = var.cloudflare_zone_id name = "Access protection for ssh.${var.cloudflare_zone}" domain = "ssh.${var.cloudflare_zone}" session_duration = "1h" } resource "cloudflare_access_policy" "ssh_policy" { application_id = zone_id = var.cloudflare_zone_id name = "Example Policy for ssh.${var.cloudflare_zone}" precedence = "1" decision = "allow" include { email = [var.cloudflare_email] } }

In the above cloudflare_access_application resource, a variable, var.cloudflare_zone_id, is used to pull in the Cloudflare Zone’s ID based on the value of the variable provided in the terraform.tfvars file. The Zone Name is also dynamically populated at runtime in the var.cloudflare_zone fields based on the value provided in the terraform.tfvars file. We also limit the scope of this access policy to using the domain argument in the cloudflare_access_application resource.

In the cloudflare_access_policy resource, we take the information provided by the cloudflare_access_application resource called ssh_app and apply it as an active policy. The scope of who is allowed to log into this endpoint is the user’s email as provided by the var.cloudflare_email variable.

Terraform Spin up and SSH Connection

Now to connect to this SSH endpoint. First we need to spin up our environment. This can be done with terraform plan and then terraform apply.

Automating Cloudflare Tunnel with Terraform

On my workstation I have cloudflared installed and updated my SSH config to proxy traffic for this SSH endpoint through cloudflared.

cdlg at cloudflare in ~ $ cloudflared --version cloudflared version 2021.4.0 (built 2021-04-07-2111 UTC) cdlg at cloudflare in ~ $ grep -A2 '' ~/.ssh/config Host IdentityFile /Users/cdlg/.ssh/google_compute_engine ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h

I can then SSH with my local user on the remote machine (cdlg) at the SSH hostname ( The instance of cloudflared running on my workstation will then proxy this request.

Automating Cloudflare Tunnel with Terraform

This will open a new tab in my current browser and direct me to the Cloudflare Access application recently created with Terraform. Earlier in the Access resource we set the Cloudflare user as denoted by the var.cloudflare_email variable as the criteria for the Access policy. If the correct email address is provided the user will receive an email similar to the following.

Automating Cloudflare Tunnel with Terraform

Following the link or providing the pin on the previously opened tab will complete the authentication. Hitting ‘approve’ tells Cloudflare Access that the user should be allowed through per the length of the session_duration argument in the cloudflare_access_application resource. Navigating back to the terminal we can see that we are now on the server.

Automating Cloudflare Tunnel with Terraform

If we check the server’s authentication log we can see that connections from the tunnel are coming in via localhost ( This allows us to lock down external network access on the SSH port of the server.

Automating Cloudflare Tunnel with Terraform

The full config of this deployment can be viewed here.

The roadmap for Cloudflare Tunnels is bright. Hopefully this walkthrough provided some quick context on what you can achieve with Cloudflare Tunnels and Cloudflare. Personally my dog is quite happy that I have more time to take him on walks. We’re very excited to see what you build with Cloudflare Tunnels and Cloudflare!

Categories: Technology

Humanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madness

Thu, 13/05/2021 - 14:00
Humanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madnessHumanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madness

Select all the buses. Click on bikes. Does this photo have traffic lights? As ridiculous as these questions are, you’re almost guaranteed to have seen one recently. They are a way for online services to separate humans from bots, and they’re called CAPTCHAs. CAPTCHAs strengthen the security of online services. But while they do that, there’s a very real cost associated with them.

Based on our data, it takes a user on average 32 seconds to complete a CAPTCHA challenge. There are 4.6 billion global Internet users. We assume a typical Internet user sees approximately one CAPTCHA every 10 days.

This very simple back of the envelope math equates to somewhere in the order of 500 human years wasted every single day — just for us to prove our humanity.

Today, we are launching an experiment to end this madness. We want to get rid of CAPTCHAs completely. The idea is rather simple: a real human should be able to touch or look at their device to prove they are human, without revealing their identity. We want you to be able to prove that you are human without revealing which human you are! You may ask if this is even possible? And the answer is: Yes! We’re starting with trusted USB keys (like YubiKey) that have been around for a while, but increasingly phones and computers come equipped with this ability by default.

Today marks the beginning of the end for fire hydrants, crosswalks, and traffic lights on the Internet.


In many instances, businesses need a way to tell whether an online user is human or not. Typically those reasons relate to security, or abuse of an online service. Back at the turn of the century, CAPTCHAs were created to do just that. The first one was developed back in 1997, and the term ("Completely Automated Public Turing test to tell Computers and Humans Apart") was coined in 2003 by Luis von Ahn, Manuel Blum, Nicholas J. Hopper, and John Langford.

By their very nature, the challenge-response nature of CAPTCHAs have to be automated: so they can scale across both humans and the bots they need to catch.

Why get rid of CAPTCHAs?

Put simply: we all hate them.

The best we’ve been able to do to date has been to minimize them. For example, at Cloudflare, we’ve continuously improved our Bot management solution to get as smart as possible about when to serve a CAPTCHA to the user. However, over the years the web moved from simple CAPTCHAs based on text recognition against backgrounds to OCRing old books to identifying objects from pictures as AI has improved (see Google paper on Street Numbers). This creates some real problems for the human users of the Internet:

  1. Productivity: Time is lost — as is focus on the task at hand — and often in exchange for some frustration.
  2. Accessibility: Users are assumed to have the physical and cognitive capabilities required to solve the tests, which may not be the case. A visual disability, for example, may make it impossible to perform a CAPTCHA-solving task.
  3. Cultural Knowledge: The people on the planet who have seen a US fire hydrant are in the minority, as are the number who speak English. Cabs are yellow in New York City, and black in London — heck, ‘cabs’ are only cabs in a few places, and ‘taxis’ everywhere else!
  4. Interactions on Mobile Devices: Phones and mobile devices are the primary — and most often only — means of Internet access for a large part of the world. CAPTCHAs put a strain on their data plans and battery usage, in addition to being more difficult on small screens.

In fact, the World Wide Web Consortium (W3C) worked on multiple drafts — as early as 2003 — pointing out the inaccessibility of CAPTCHAs.

And this is just from the user side. Inflicting all these costs on users has very real costs for businesses, too. There’s a reason why businesses spend so much time  optimizing the performance and layout of their websites and applications. That work stops users from bouncing when you want them to register. It stops shopping carts getting abandoned when you want them to end in the checkout. In general, you want to stop customers from getting frustrated and simply not come back.

CAPTCHAs are effectively businesses putting friction in front of their users, and as anyone who has managed a high performing online business will tell you, it’s not something you want to do unless you have no choice.

We started tackling these issues when we moved from Google reCAPTCHA to hCAPTCHA. Today, we are going further.

CAPTCHA without Picture: Cryptographic Attestation of PersonhoodHumanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madnessHardware security keys are devices with an embedded secret that can connect to your computer or your phone

From a user perspective, a Cryptographic Attestation of Personhood works as follows:

  1. The user accesses a website protected by Cryptographic Attestation of Personhood, such as
  2. Cloudflare serves a challenge.
  3. The user clicks I am human (beta) and gets prompted for a security device.
  4. User decides to use a Hardware Security Key.
  5. The user plugs the device into their computer or taps it to their phone for wireless signature (using NFC).
  6. A cryptographic attestation is sent to Cloudflare, which allows the user in upon verification of the user presence test.

Completing this flow takes five seconds. More importantly, this challenge protects users' privacy since the attestation is not uniquely linked to the user device. All device manufacturers trusted by Cloudflare are part of the FIDO Alliance. As such, each hardware key shares its identifier with other keys manufactured in the same batch (see Universal 2nd Factor Overview, Section 8). From Cloudflare’s perspective, your key looks like all other keys in the batch.

There are at most three clicks required to complete a Cryptographic Attestation of Personhood. There is no looping, where a user is asked to click on buses 10 times in a row.

While there is a variety of hardware security keys, our initial rollout is limited to a few devices: YubiKeys, which we had the chance to use and test; HyperFIDO keys; and Thetis FIDO U2F keys.

“Driving open authentication standards like WebAuthn has long been at the heart of Yubico’s mission to deliver powerful security with a delightful user experience,” said Christopher Harrell, Chief Technology Officer at Yubico. “By offering a CAPTCHA alternative via a single touch backed by YubiKey hardware and public key cryptography, Cloudflare’s Cryptographic Attestation of Personhood experiment could help further reduce the cognitive load placed on users as they interact with sites under strain or attack. I hope this experiment will enable people to accomplish their goals with minimal friction and strong privacy, and that the results will show it is worthwhile for other sites to consider using hardware security for more than just authentication.”How does it work?

The Cryptographic Attestation of Personhood relies on Web Authentication (WebAuthn) Attestation. This is an API that has been standardized at the W3C and is already implemented in most modern web browsers and operating systems. It aims to provide a standard interface to authenticate users on the web and use the cryptography capability of their devices.

As the need for stronger security with improved usability increases, we envision the deployment instances of WebAuthn to rise.

Platform Compatible Browsers iOS 14.5 All browsers Android 10 and later Chrome Windows All browsers macOS All browsers Ubuntu All browsers

Assuming you are using a hardware device with a compatible configuration, you might be wondering what is happening behind the scenes.

The elevator pitch

The short version is that your device has an embedded secure module containing a unique secret sealed by your manufacturer. The security module is capable of proving it owns such a secret without revealing it. Cloudflare asks you for proof and checks that your manufacturer is legitimate.

Humanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madnessThe technical explanation

The longer version is that this verification involves public-key cryptography and digital certificates.

Public-key cryptography provides a way to produce unforgeable digital signatures. A user generates a signing key that can sign messages and a verification key that can be used by anyone to verify a message is authentic. This is akin to a signet ring, where the imprint of the ring is the signature and the ring itself the signing key.

Signature schemes are used widely to prove authenticity. Right now, your browser has verified that the server claiming to be “” is legitimate by verifying a signature made by someone who has a signing key associated with “”. To show the verification key is legitimate, the server provides a certificate that links the verification key to “”, itself signed by another verification key in another certificate. This chain goes all the way up to a root certificate from a Certificate Authority built into your browser.

Let's take another example. Alice owns a laptop with a secure module embedded. This module holds a signing key, sk_a. Alice says she sent a love letter to Bob yesterday. However, Bob is suspicious. Despite the letter stating "Hi Bob, it's Alice", Bob would like to be sure this letter comes from Alice. To do so, Bob asks Alice to provide her signature for the following message "musical-laboratory-ground". Since Bob chooses the message, if Alice can provide a signature associated with her verification key (pk_a), Bob would be convinced the love letter is from Alice. Alice does provide the signature, sk_a(“musical-laboratory-ground”). Bob confirms sk_a(“musical-laboratory-ground”) is associated with pk_a. He can now securely engage in their cryptographer relationship.

Humanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madness

Thinking back to the Cryptographic Attestation of Personhood, you now know that your hardware key embeds a signing key. However, Cloudflare does not and cannot know the signing keys of all users of the Internet. To alleviate this problem, Cloudflare requests a different kind of proof. When asked if you are a human, we ask you to prove you are in control of a public key signed by a trusted manufacturer. When shipping devices with a secure module, manufacturers sign the associated attestation public key with a digital certificate.

Digital certificates usually contain a public key, information about the organization they are provisioned for, a validity period, the allowed usage, and a signature from a Certificate Authority making sure the certificate is legitimate. They allow metadata to be associated with a public key and therefore provide information about the issuer of a signature.

Humanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madness

So when Cloudflare asks you to provide a signature, it verifies your public key has been signed by the public key of a manufacturer. Since manufacturers have multiple levels of certificates, your device provides a chain of certificates that Cloudflare is able to verify. Each link in the chain is signed by its predecessor and signs its successor. Cloudflare trusts the root certificate of manufacturers. Because their numbers are limited, we have the capacity to verify them manually.

Humanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madnessPrivacy first

Designing a challenge asking users to prove they are in control of a key from a certain manufacturer comes with a privacy and security challenge.

The privacy properties of the Cryptographic Attestation of Personhood are summarized in the following table.

Property Cloudflare Could Cloudflare Does Get your fingerprints or face NO N/A Know the manufacturer of your key YES - limited to the number of keys in your batch* YES Associate a unique ID to your key YES** NO

* There must be 100,000 or more keys per batch (FIDO UAF Protocol Specification # However, self-signed keys and keys from certain manufacturers have been found to not meet this requirement.

**This would require that we set a separate and distinct cookie to track your key. This is antithetical to privacy on the Internet, and to the goals of this project. You can learn more about how we are removing cookies like __cfduid here.

Attestation without collecting biometrics

The aim of this project: we want to know that you’re human. But we’re not interested in which human you are.

Happily, the WebAuthn API goes a long way to take care of this for us. Not that we want it, but the WebAuthn API prevents the collection of biometrics, such as a fingerprint. When your device asks for a biometric authentication — such as via a fingerprint sensor — it all happens locally. The verification is meant to unlock the secure module of your device, which provides a signature associated with your platform.

For our challenge, we leverage the WebAuthn registration process. It has been designed to perform multiple authentications, which we do not have a use for. Therefore, we do assign the same constant value to the required username field. It protects users from deanonymization.

No hidden work

A common use of CAPTCHA is to label datasets that AI has difficulty identifying. This could be for books, street numbers, or fire hydrants. While this is useful for science, it has also been used as a way for companies to leverage human recognition ability for commercial gain without their users’ knowledge.

With the Cryptographic Attestation of Personhood, this does not happen. We have more flexibility designing the user flow, as we are not constrained by the CAPTCHA challenge model anymore.

What Cloudflare is doing to push privacy even further

While the Cryptographic Attestation of Personhood has a lot of upside in terms of privacy, it is not perfect. Cloudflare still needs to know your manufacturer to let you in. As WebAuthn works with any certificate, we need to make sure Cloudflare receives certificates from untampered hardware keys. We would prefer to not have that information, further preserving your privacy.

We have worked on privacy standards in the past, leading the efforts with Privacy Pass for instance. Privacy Pass allows you to solve a challenge once and provide a proof you passed it, meaning you don’t have to solve multiple CAPTCHAs. It greatly improved the user experience of VPN users, who face more challenges than other Internet users.

For the Cryptographic Attestation of Personhood, we dig into an emerging field in cryptography called Zero Knowledge proofs (ZK proof). It allows our users to prove their manufacturer is part of a set of manufacturers trusted by Cloudflare. Using a  ZK proof, the devices from a single manufacturer become indistinguishable from each other, as well as from devices from other manufacturers. This new system requires more technical details and deserves a dedicated blog post. Stay tuned.

A never-ending quest

Designing a challenge aimed at protecting millions of Internet properties is no easy task. In the current setup, we believe Cryptographic Attestation of Personhood offers strong security and usability guarantees compared to traditional CAPTCHA challenges. During a preliminary user study, users indicated a strong preference for touching their hardware key over clicking on pictures. Nevertheless, we know that this is a new system with room for improvements.

This experiment will be available on a limited basis in English-speaking regions. This allows us to have diversity in the pool of users and test this process in various locations. However, we recognize this is insufficient coverage and we intend to test further. If you have specific needs, feel free to reach out.

Another issue that we keep a close eye on is security. The security of this challenge depends on the underlying hardware provided by trusted manufacturers. We have confidence they are secured. If any breach were to occur, we would be able to quickly deauthorize manufacturers’ public keys at various levels of granularity.

Humanity wastes about 500 years per day on CAPTCHAs. It’s time to end this madness

We also have to consider the possibility of facing automated button-pressing systems. A drinking bird able to press the capacitive touch sensor could pass the Cryptographic Attestation of Personhood. At best, the bird solving rate matches the time it takes for the hardware to generate an attestation. With our current set of trusted manufacturers, this would be slower than the solving rate of professional CAPTCHA-solving services, while allowing legitimate users to pass through with certainty. In addition, existing Cloudflare mitigations would remain in place, efficiently protecting Internet properties.

Final words

For Cloudflare, it always comes back to: helping build a better Internet. The very idea that we’re all wasting 500 years per day on the Internet — that nobody had revisited the fundamental assumptions of CAPTCHAs since the turn of the century — seemed absurd to us.

We’re very proud of the work we’ve done here to release the Cryptographic Attestation of Personhood. This challenge has been built with a user-first approach while maintaining a high level of security for accessing Internet properties sitting behind Cloudflare’s global network. We’re now in the process of augmenting our existing humanity challenge with the Cryptographic Attestation of Personhood. You should expect to see it more frequently as time passes. You can try it out today at

We want to acknowledge the work of other teams at Cloudflare. While this work is led by the Research team, we have been extremely privileged to get support from all across the company. If you want to help us build a better Internet, we are hiring.

Finally: we’re excited to bring about the demise of the fire hydrant on the Internet. It’s no longer needed.

Categories: Technology

Highly available and highly scalable Cloudflare tunnels

Wed, 12/05/2021 - 14:00
Highly available and highly scalable Cloudflare tunnelsHighly available and highly scalable Cloudflare tunnels

Starting today, we’re thrilled to announce you can run the same tunnel from multiple instances of cloudflared simultaneously. This enables graceful restarts, elastic auto-scaling, easier Kubernetes integration, and more reliable tunnels.

What is Cloudflare Tunnel?

I work on Cloudflare Tunnel, a product our customers use to connect their services and private networks to Cloudflare without poking holes in their firewall. Tunnel connections are managed by cloudflared, a tool that runs in your environment and connects your services to the Internet while ensuring that all its traffic goes through Cloudflare.

Say you have some local service (a website, an API, or a TCP server), and you want to securely expose it to the Internet using a Cloudflare Tunnel. First, download cloudflared, which is a “connector” that connects your local service to the Internet through Cloudflare. You can then connect that service to Cloudflare and generate a DNS entry with a single command:

cloudflared tunnel create --name mytunnel --url http://localhost:8080 --hostname

This creates a tunnel called “mytunnel”, and configures your DNS to map to that tunnel. Then cloudflared connects to the Cloudflare network. When the Cloudflare network receives an incoming request for, it looks up the cloudflared running mytunnel and proxies the request there. Then cloudflared proxies those requests to localhost:8080.

Highly available and highly scalable Cloudflare tunnels

With Tunnel, your origin server no longer needs to allow any incoming traffic. In fact, it doesn’t even need a publicly reachable IP address. This is significant because that means that no one can simply bypass Cloudflare to reach your resource either.

Traditionally, Cloudflare customers onboard their sites to our platform with a simple nameserver change. By changing your nameserver, Cloudflare receives any queries to your resource first and leverages this as an opportunity to block malicious traffic and enforce policies and rules you define for your resource within the Cloudflare dashboard. However, if attackers discover your origin IP, they could bypass Cloudflare and your policies and rules.

Instead, with Tunnel, requests for your Internet property are proxied through the already-established outgoing connections from cloudflared to the Cloudflare network. This way any traffic entering your site will have to go through Cloudflare, where you can enforce more granular control with policies for caching, page rewrites, or Zero Trust security (e.g. only users with an email can view the page).

Scaling cloudflared

One feature request we’ve heard quite often is that our users need their software systems to scale. Their database must be scalable. Their web servers must be scalable. And of course, in turn, cloudflared must be scalable, because without cloudflared, our users can’t receive traffic.

For reliability purposes, cloudflared opens connections to four different Cloudflare servers (two in each of two different data centers, for redundancy in case a data center goes down). This way if one goes down, the other three will serve traffic while it reconnects. But what if the cloudflared process itself goes down?

Well, ideally, we would be able to scale or replicate cloudflared itself.

Highly available and highly scalable Cloudflare tunnels

Previously, scaling cloudflared required using Cloudflare Load Balancer to spread traffic across multiple unique tunnels. Each tunnel then had to be manually authenticated, configured, and connected. That poses a challenge for teams who need to autoscale instances, like the resources in a Kubernetes cluster, without manual intervention.

Starting today, you can now create and configure an instance of cloudflared once and run it as multiple different processes in a replica model.

You can still point a DNS record or Cloudflare Load Balancer to a tunnel using its unique ID, but that tunnel is now represented by one or more identical instances of cloudflared - each with a unique connector ID.

Highly available and highly scalable Cloudflare tunnels

When you run your tunnel, cloudflared will log its connector ID.

2021-03-29T18:40:17Z INF Starting tunnel tunnelID=610a53bd-ed0c-4afe-92b5-ca0238153410 2021-03-29T18:40:17Z INF Version 2021.3.5 2021-03-29T18:40:17Z INF GOOS: darwin, GOVersion: go1.16.2, GoArch: amd64 2021-03-29T18:40:17Z INF Generated Connector ID: 14e2e624-0d32-4a21-a88c-64acf9484dac

There’s a new command, cloudflared tunnel info <tunnel name>, to show you each cloudflared running your tunnel.

$ cloudflared tunnel info mytunnel NAME: mytunnel ID: 610a53bd-ed0c-4afe-92b5-ca0238153410 CREATED: 2021-03-26 19:29:34.291328 +0000 UTC CONNECTOR ID CREATED ARCHITECTURE VERSION ORIGIN IP EDGE 71490dec-190f-4652-a70a-cd001fe6fdcf 2021-03-26T19:29:47Z darwin_amd64 2021.3.3 2xDFW, 2xMCI a0737d55-51f5-4fe0-8b53-c25989453c43 2021-03-26T19:29:58Z darwin_amd64 2021.3.3 2xDFW, 2xMCI

At the moment, there’s a limit of 100 simultaneous connectors per tunnel. We think that should be enough for any customer, but if your use-case requires more, please reach out to support and we can raise it for your account.

Cloudflare Tunnel’s new use casesElastic auto-scaling

Don’t let cloudflared become a bottleneck in your system. When your incoming traffic spikes, your origin servers should scale up, and your cloudflareds should scale up too.

With this launch, your team can dynamically start more instances of cloudflared without changing your DNS or Load Balancer configuration. The tunnel will distribute traffic between instances of cloudflared without the need to manually create and enroll new instances.

Graceful restarts

Right now it can be painful to change your cloudflared configuration. You’ll change the config file (or environment variables), then restart cloudflared. The problem is, this restart causes downtime while cloudflared stops accepting requests, restarts, reconnects to the edge, and starts accepting requests again.

Today’s announcement enables zero-downtime config changes. Instead of restarting cloudflared, simply start a second instance. The new instance will read the new configuration from the file. Once it’s connected to the edge and accepting traffic, you can stop the old cloudflared instance. The old instance will stop accepting new connections, wait for existing connections to finish, then terminate. Now 100% of your traffic is going through the new configuration, with zero downtime.

Easier Kubernetes integration

Cloudflared tunnels were previously incompatible with two of the most common Kubernetes scenarios:

  1. Scaling up a service by adding another pod with identical configuration
  2. Gracefully upgrading a service by adding another pod with the new version/configuration, waiting for it to become healthy, then removing the old pod

Unfortunately, neither of these worked with cloudflared, because the new cloudflared pod would fail to start. Instead, it would output an error message, saying it couldn’t run because its tunnel was already running somewhere else.

But now you can run many cloudflared pods, each running the same tunnel. We suggest the easiest way to use cloudflared with Kubernetes is to have your origin server encapsulated in a Kubernetes Service, and then use a separate Kubernetes Deployment for cloudflared. Configure cloudflared’s ingress rules to point at the origin Service. Now you can scale cloudflared and your origin service up or down, independently of each other.

More reliable tunnels

In a modern distributed system, it’s important to avoid having a bottleneck or a single point of failure. Unfortunately, cloudflared sometimes became a single point of failure. Previously, you could mitigate this by running two cloudflareds, with different tunnels but otherwise identical configuration, and load balancing across these tunnels.

However, today you can simply run the same cloudflared multiple times -- whether in the same data center or two different continents -- and avoid the anxiety that comes from relying on a single program to keep your traffic flowing in.

If you’re interested in trying it out for yourself, check out our tutorial to get started today!

Categories: Technology

Designing the new Cloudflare Web Application Firewall

Tue, 11/05/2021 - 14:00
Designing the new Cloudflare Web Application FirewallDesigning the new Cloudflare Web Application Firewall

The Cloudflare Web Application Firewall (WAF) protects websites and applications from malicious traffic attempting to exploit vulnerabilities in server software. It’s a critical piece of the broader security posture of your application. With that in mind, we made sure improvements to the Web Application Firewall dashboard experience made it easier to enable the WAF and configure rules to match the specific requirements of an application. In this post, I’ll share parts of the process we followed and the rationale behind the decisions we took when designing the new Web Application Firewall dashboard experience.

I’ve separated out my design process into three stages:

  1. Identify the tasks customers are trying to complete using the WAF
  2. Prioritise the tasks in such a way that it’s clear what the most common tasks are vs what the more involved tasks are
  3. Define, create, and refine the interface and interactions
Identifying the tasks customers are trying to complete

We support a range of customers — individual developers or hobbyists, small/medium-sized businesses where it’s common for a developer to fulfil multiple roles and responsibilities, through to large global enterprises where often there is an entire department dedicated to information security. Traditionally, product development teams use techniques such as the use of a user persona or a user story to help them focus on a specific problem they’re trying to solve; however each of these methods have their own inefficiencies and importantly aren’t able to scale to match the breadth of our customers. For example, a user persona is typically made by taking the average from a group or a specific selection of demographic indicators (such as age, gender, marital status, and hobbies), presenting it in a document and referring to it as one of multiple user archetypes of the application, but crucially fails to explain why the user decided to use or not use a specific feature. Similarly, user stories make use of user personas, but conflate them with implementation details and desired outcomes all the while failing to describe the situation the user is in.

To help the product development team better empathise with our range of customers, we use a technique known as Job Stories. Job Stories, unlike user persona or user stories allow us to focus on the users situation, motivation, and desired outcome.

We conducted interviews with a handful of customers directly, but we also supplemented them by interviewing members of our Solutions Engineering team. They help customers configure Cloudflare to meet their requirements and therefore have a direct connection to multiple customers themselves. They are in a position of being able to aggregate feedback from multiple customers. From the various interviews we identified the following job stories among many:

  • When onboarding with Cloudflare, I want to quickly turn on the WAF and use the default settings so I can proceed to configuring the rest of the Cloudflare features.
  • When refining and tuning the configuration of my zone, I only want to configure the rules I’m interested in so I can reduce the potential number of false positive results.

Next, we began to analyse each of these uses further. We wanted to understand what was working well from the existing interface and more importantly, what was causing confusion or impacting efficiency.

“I want to turn on the WAF and use the default settings.”

With the legacy dashboard experience, a customer had a choice of different managed rulesets (Cloudflare Managed Ruleset or OWASP ModSecurity Core Rule Set.) Making a ruleset actually run however required some tedious configuration. The specific groups within a ruleset would need to be enabled and a separate switch controlling the overall WAF would also need to be enabled. Now, imagine the use case of wanting to use the Cloudflare Managed Ruleset. The current experience would require the user to enable at least two switches:

  1. The overall Web Application Firewall switch must be enabled
  2. At least a single group from Cloudflare Managed Ruleset must be enabled

As two distinct options needed to be configured, we concluded that this could cause customers to put their applications at an increased risk of being unprotected. They might have enabled the particular groups they’re interested in from the Cloudflare Managed Ruleset. However, if the switch of the Web Application Firewall is in the off position, the configuration of the Cloudflare Managed Ruleset is ignored. Essentially, they’ve misconfigured the WAF into a vulnerable state.

This is where we identified our first opportunity of improvement. The existing user journey had too many opportunities for misconfiguration and should have been very straightforward.

Designing the new Cloudflare Web Application FirewallThe legacy UI of the Web Application Firewall.“I only want to configure the rules I’m interested in.”

On the legacy Managed Rules page, all the groups for each of the rulesets were listed beneath the card. This made it super easy to enable or disable specific rule groups and, from our user research sessions, we learned it was actually a very common workflow. However, scratch beneath the surface slightly and you’ll start to see the complexities of the system. Each group is made up of individual rules. By clicking on a group, a new modal would appear listing all the rules belonging to that group. The legacy UI was built to make it easy to change the action of a single rule — select an option from the dropdown and off you go. Simple. Most users probably even have the patience to change the action of a couple of rules like this. But what if it isn’t just a couple? Some groups contain hundreds of rules. A particularly complicated setup of the WAF could mean having to change the action of each rule. With the capabilities of the legacy UI, that would require hundreds of clicks and waste invaluable time. I’m embarrassed to admit that this was the suggested workflow and users were often required to use our APIs instead.

We now identified our next area of improvement — create the tools necessary for completing bulk edits and easy rule selection.

In summary, users want to easily turn on the WAF and move along. However, the legacy UI had a cumbersome and error-prone process to do so. Secondly, if a user wanted to enable or disable a specific rule of a ruleset, the legacy UI actually made that very simple. However, when it came to changing the action of multiple rules at once, the legacy UI was a time sink.

Prioritising the identified tasks

We’ve identified the tasks or jobs customers are trying to accomplish and understand where the current difficulties and inefficiencies are within the experience. Using our telemetry and analytics tools, such as Amplitude, we determine how often customers are performing each of the job stories. This is a critical step, as the output will help us decide which job stories we should be optimising the interface for. About 76% of zones that are using the WAF today are doing so with its default configuration. From this we can infer that most customers simply turn on the WAF and continue on with their business, potentially continuing to configure other Cloudflare features.

It’s worth pointing out the importance of having sensible defaults within applications. Oftentimes users will continue down the path of least resistance or to whatever helps them complete their goal the quickest — ergo, they’ll usually stick to the default settings of an application as these are usually created with the most common use cases in mind. For this reason, the default state of the Cloudflare Managed Ruleset is such that it exceeds the security requirements of most applications whilst balancing a relatively low false positive rate.

Define, create, and refine the interface and interactions

We use Figma to design the user interface of the dashboard. Using our Component Library, it allows us to quickly create mockups of what an interface could look like. At this stage of the project, a tool like Figma makes it easy for us to iterate through numerous ideas or permutations of a particular interaction.

Designing the new Cloudflare Web Application FirewallThe Figma file used during the development of the new UI.

From the user research and data collection we conducted earlier, it’s clear that we needed to make enabling a particular ruleset better than the legacy experience. It is a very common workflow, but prone to potentially dangerous errors — certainly not a good combination.

Part of our job as designers at Cloudflare is to make the complex and intricate aspects of configuration ridiculously simple and increasing the confidence customers have with the UI and the actions they’re performing. With that in mind, to enable a ruleset like the Cloudflare Managed Ruleset with the new Managed Rules a customer only needs to do a single click.

Designing the new Cloudflare Web Application FirewallThe new Web Application Firewall page.

We wanted to improve the method of having all rules execute the same specific action. With the legacy UI, this required customers to select from a drop-down menu on all the individual rules. This was an extremely tedious and time-consuming process. Sticking to our design principles of maintaining ease of use and increasing simplicity and efficiency, the new Managed Rules allows for a single action to execute across the entire ruleset. We call this a Ruleset Action. A Ruleset Action is an action which all the rules within a ruleset will adhere to.

Designing the new Cloudflare Web Application FirewallReview page with Ruleset Action configured.

The next capability we focused on was having all the rules execute the same specific action. In the legacy UI, changing the mode of multiple rules at once wasn’t possible. With the new dashboard experience, customers can browse through all the rules within the ruleset. Multiple rules can be selected at once using the select box on the left-hand side and the action or status can be set for the entire selection.

Designing the new Cloudflare Web Application FirewallRule Browser page with multiple rules selected and Set Action drop-down menu open.We’re just getting started

We didn’t get to these interactions straightaway, but rather by taking part in numerous design critiques and constantly evaluating the effectiveness of the new interactions against the identified job stories. We’ll be utilising our telemetry and analytics tools to understand how customers are using the new features and continuing to refine the experience further. Watch this space, because more updates are on the way!

Categories: Technology

Branch predictor: How many "if"s are too many? Including x86 and M1 benchmarks!

Thu, 06/05/2021 - 14:00

Some time ago I was looking at a hot section in our code and I saw this:

if (debug) { log("..."); }

This got me thinking. This code is in a performance critical loop and it looks like a waste - we never run with the "debug" flag enabled[1]. Is it ok to have if clauses that will basically never be run? Surely, there must be some performance cost to that...

Just how bad is peppering the code with avoidable if statements?

Back in the days the general rule was: a fully predictable branch has close to zero CPU cost.

To what extent is this true? If one branch is fine, then how about ten? A hundred? A thousand? When does adding one more if statement become a bad idea?

At some point the negligible cost of simple branch instructions surely adds up to a significant amount. As another example, a colleague of mine found this snippet in our production code:

const char *getCountry(int cc) { if(cc == 1) return "A1"; if(cc == 2) return "A2"; if(cc == 3) return "O1"; if(cc == 4) return "AD"; if(cc == 5) return "AE"; if(cc == 6) return "AF"; if(cc == 7) return "AG"; if(cc == 1) return "AI"; ... if(cc == 252) return "YT"; if(cc == 253) return "ZA"; if(cc == 254) return "ZM"; if(cc == 255) return "ZW"; if(cc == 256) return "XK"; if(cc == 257) return "T1"; return "UNKNOWN"; }

Obviously, this code could be improved[2]. But when I thought about it more: should it be improved? Is there an actual performance hit of a code that consists of a series of simple branches?

Understanding the cost of jump

We must start our journey with a bit of theory. We want to figure out if the CPU cost of a branch increases as we add more of them. As it turns out, assessing the cost of a branch is not trivial. On modern processors it takes between one and twenty CPU cycles. There are at least four categories of control flow instructions[3]: unconditional branch (jmp on x86), call/return, conditional branch (e.g. je on x86) taken and conditional branch not taken. The taken branches are especially problematic: without special care they are inherently costly - we'll explain this in the following section. To bring down the cost, modern CPU's try to predict the future and figure out the branch target before the branch is actually fully executed! This is done in a special part of the processor called the branch predictor unit (BPU).

The branch predictor attempts to figure out a destination of a branching instruction very early and with very little context. This magic happens before the "decoder" pipeline stage and the predictor has very limited data available. It only has some past history and the address of the current instruction. If you think about it - this is super powerful. Given only current instruction pointer it can assess, with very high confidence, where the target of the jump will be.


The BPU maintains a couple of data structures, but today we'll focus on Branch Target Buffer (BTB). It's a place where the BPU remembers the target instruction pointer of previously taken branches. The whole mechanism is much more complex, take a look a the Vladimir Uzelac's Master thesis for details about branch prediction on CPU's from 2008 era:

For the scope of this article we'll simplify and focus on the BTB only. We'll try to show how large it is and how it behaves under different conditions.

Why is branch prediction needed?

But first, why is branch prediction used at all? In order to get the best performance, the CPU pipeline must feed a constant flow of instructions. Consider what happens to the multi-stage CPU pipeline on a branch instruction. To illustrate let's consider the following ARM program:

BR label_a; X1 ... label_a: Y1

Assuming a simplistic CPU model, the operations would flow through the pipeline like this:

In the first cycle the BR instruction is fetched. This is an unconditional branch instruction changing the execution flow of the CPU. At this point it's not yet decoded, but the CPU would like to fetch another instruction already! Without a branch predictor in cycle 2 the fetch unit either has to wait or simply continues to the next instruction in memory, hoping it will be the right one.

In our example, instruction X1 is fetched even though this isn't the correct instruction to run. In cycle 4, when the branch instruction finishes the execute stage, the CPU will be able to understand the mistake, and roll back the speculated instructions before they have any effect. At this point the fetch unit is updated to correctly get the right instruction - Y1 in our case.

This situation of losing a number of cycles due to fetching code from an incorrect place is called a "frontend bubble". Our theoretical CPU has a two-cycle frontend bubble when a branch target wasn’t predicted right.

In this example we see that, although the CPU does the right thing in the end, without good branch prediction it wasted effort on bad instructions. In the past, various techniques have been used to reduce this problem, such as static branch prediction and branch delay slots. But the dominant CPU designs today rely on dynamic branch prediction. This technique is able to mostly avoid the frontend bubble problem, by predicting the correct address of the next instruction even for branches that aren’t fully decoded and executed yet.

Playing with the BTB

Today we're focusing on the BTB - a data structure managed by the branch predictor responsible for figuring out a target of a branch. It's important to note that the BTB is distinct from and independent of the system assessing if the branch was taken or not taken. Remember, we want to figure out if a cost of a branch increases as we run more of them.

Preparing an experiment to stress only the BTB is relatively simple (based on Matt Godbolt's work). It turns out a sequence of unconditional jmps is totally sufficient. Consider this x86 code:

This code stresses the BTB to an extreme - it just consists of a chain of jmp +2 statements (i.e. literally jumping to the next instruction). In order to avoid wasting cycles on frontend pipeline bubbles, each taken jump needs a BTB hit. This branch prediction must happen very early in the CPU pipeline, before instruction decode is finished. This same mechanism is needed for any taken branch, whether it's unconditional, conditional or a function call.

The code above was run inside a test harness that measures how many CPU cycles elapse for each instruction. For example, in this run we're measuring times of dense - every two bytes - 1024 jmp instructions one after another:

We’ll look at the results of experiments like this for a few different CPUs. But in this instance, it was run on a machine with an AMD EPYC 7642. Here, the cold run took 10.5 cycles per jmp, and then all subsequent runs took ~3.5 cycles per jmp. The code is prepared in such a way to make sure it's the BTB that is slowing down the first run. Take a look at the full code, there is quite some magic to warm up the L1 cache and iTLB without priming the BTB.

Top tip 1. On this CPU a branch instruction that is taken but not predicted, costs ~7 cycles more than one that is taken and predicted. Even if the branch was unconditional.

Density matters

To get a full picture we also need to think about the density of jmp instructions in the code. The code above did eight jmps per 16-byte code block. This is a lot. For example, the code below contains one jmp instruction in each block of 16 bytes. Notice that the nop opcodes are jumped over. The block size doesn't change the number of executed instructions, only the code density:

Varying the jmp block size might be important. It allows us to control the placement of the jmp opcodes. Remember the BTB is indexed by instruction pointer address. Its value and its alignment might influence the placement in the BTB and help us reveal the BTB layout. Increasing the alignment will cause more nop padding to be added. The sequence of a single measured instruction - jmp in this case - and zero or more nops, I will call "block", and its size "block size". Notice that the larger the block size, the larger the working code size for the CPU. At larger values we might see some performance drop due to exhausting L1 cache space.

The experiment

Our experiment is crafted to show the performance drop depending on the number of branches, across different working code sizes. Hopefully, we will be able to prove the performance is mostly dependent on the number of blocks - and therefore the BTB size, and not the working code size.

See the code on GitHub. If you want to see the generated machine code, though, you need to run a special command. It's created procedurally by the code, customized by passed parameters. Here's an example gdb incantation:

Let's bring this experiment forward, what if we took the best times of each run - with a fully primed BTB - for varying values of jmp block sizes and number of blocks - working set size? Here you go:

This is an astonishing chart. First, it's obvious something happens at the 4096 jmp mark[4] regardless of how large the jmp block sizes - how many nop's we skip over. Reading it aloud:

  • On the far left, we see that if the amount of code is small enough - less than 2048 bytes (256 times a block of 8 bytes) - it's possible to hit some kind of uop/L1 cache and get ~1.5 cycles per fully predicted branch. This is amazing.
  • Otherwise, if you keep your hot loop to 4096 branches then, no matter how dense your code is you are likely to see ~3.4 cycles per fully predicted branch
  • Above 4096 branches the branch predictor gives up and the cost of each branch shoots to ~10.5 cycles per jmp. This is consistent with what we saw above - unpredicted branch on flushed BTB took ~10.5 cycles.

Great, so what does it mean? Well, you should avoid branch instructions if you want to avoid branch misses because you have at most 4096 of fast BTB slots. This is not a very pragmatic advice though - it's not like we deliberately put many unconditional jmps in real code!

There are a couple of takeaways for the discussed CPU. I repeated the experiment with an always-taken conditional branch sequence and the resulting chart looks almost identical. The only difference being the predicted taken conditional-je instruction being 2 cycles slower than unconditional jmp.

An entry to BTB is added wherever a branch is "taken" - that is, the jump actually happens. An unconditional "jmp" or always taken conditional branch, will cost a BTB slot. To get best performance make sure to not have more than 4096 taken branches in the hot loop. The good news is that branches never-taken don't take space in the BTB. We can illustrate this with another experiment:

This boring code is going over not-taken jne followed by two nops (block size=4). Aimed with this test (jne never-taken), the previous one (jmp always-taken) and a conditional branch je always-taken, we can draw this chart:

First, without any surprise we can see the conditional 'je always-taken' is getting slightly more costly than the simple unconditional jmp, but only after the 4096 branches mark. This makes sense, the conditional branch is resolved later in the pipeline so the frontend bubble is longer. Then take a look at the blue line hovering near zero. This is the "jne never-taken" line flat at 0.3 clocks / block, no matter how many blocks we run in sequence. The takeaway is clear - you can have as many never-taken branches as you want, without incurring any cost. There isn't any spike at 4096 mark, meaning BTB is not used in this case. It seems the conditional jump not seen before is guessed to be not-taken.

Top tip 2: conditional branches never-taken are basically free - at least on this CPU.

So far we established that branches always-taken occupy BTB, branches never taken do not. How about other control flow instructions, like the call?

I haven't been able to find this in the literature, but it seems call/ret also need the BTB entry for best performance. I was able to illustrate this on our AMD EPYC. Let's take a look at this test:

This time we'll issue a number of callq instructions followed by ret - both of which should be fully predicted. The experiment is crafted so that each callq calls a unique function, to allow for retq prediction - each one returns to exactly one caller.

This chart confirms the theory: no matter the code density - with the exception of 64-byte block size being notably slower -  the cost of predicted call/ret starts to deteriorate after the 2048 mark. At this point the BTB is filled with call and ret predictions and can't handle any more data. This leads to an important conclusion:

Top tip 3. In the hot code you want to have less than 2K function calls - on this CPU.

In our test CPU a sequence of fully predicted call/ret takes about 7 cycles, which is about the same as two unconditional predicted jmp opcodes. It's consistent with our results above.

So far we thoroughly checked AMD EPYC 7642. We started with this CPU because the branch predictor is relatively simple and the charts were easy to read. It turns out more recent CPUs are less clear.


Newer AMD is more complex than the previous generations. Let's run the two most important experiments. First, the jmp one:

For the always-taken branches case we can see a very good, sub 1 cycle, timings when the number of branches doesn't exceed 1024 and the code isn't too dense.

Top tip 4. On this CPU it's possible to get <1 cycle per predicted jmp when the hot loop fits in ~32KiB.

Then there is some noise starting after the 4096 jmps mark. This is followed by a complete drop of speed at about 6000 branches. This is in line with the theory that BTB is 4096 entries long. We can speculate that some other prediction mechanism is successfully kicking in beyond that, and keeps up the performance up the ~6k mark.

The call/ret chart shows a similar tale, the timings start breaking after 2048 mark, and completely fail to be predicted beyond ~3000.

Xeon Gold 6262

The Intel Xeon looks different from the AMD:

Our test shows the predicted taken branch costs 2 cycles. Intel has documented a clock penalty for very dense branching code - this explains the 4-byte block size line hovering at ~3 cycles. The branch cost breaks at the 4096 jmp mark, confirming the theory that the Intel BTB can hold 4096 entries. The 64-byte block size chart looks confusing, but really isn't. The branch cost stays at flat 2 cycles up till the 512 jmp count. Then it increases. This is caused by the internal layout of the BTB which is said to be 8-way associative. It seems with the 64-byte block size we can utilize at most half of the 4096 BTB slots.

Top tip 5. On Intel avoid placing your jmp/call/ret instructions at regular 64-byte intervals.

Then the call/ret chart:

Similarly, we can see the branch predictions failing after the 2048 jmp mark - in this experiment one block uses two flow control instructions: call and ret. This again confirms the BTB size of 4K entries. The 64-byte block size is generally slower due to the nop padding but also breaks faster due to the instructions alignment issue. Notice, we haven't seen this effect on AMD.

Apple Silicon M1

So far we saw examples of AMD and Intel server grade CPUs. How does an Apple Silicon M1 fit in this picture?

We expect it to be very different - it's designed for mobile and it's using ARM64 architecture. Let's see our two experiments:

The predicted jmp test shows an interesting story. First, when the code fits 4096 bytes (1024*4 or 512*8, etc) you can expect a predicted jmp to cost 1 clock cycle. This is an excellent score.

Beyond that, generally, you can expect a cost of 3 clock cycles per predicted jmp. This is also very good. This starts to deteriorate when the working code grows beyond ~200KiB. This is visible with block size 64 breaking at 3072 mark 3072*64=196K, and for block 32 at 6144: 6144*32=196K. At this point the prediction seems to stop working. The documentation indicates that the M1 CPU has 192 KB L1 of instruction cache - our experiment matches that.

Let's compare the "predicted jmp" with the "unpredicted jmp" chart. Take this chart with a grain of salt, because flushing the branch predictor is notoriously difficult.

However, even if we don't trust the flush-bpu code (adapted from Matt Godbolt), this chart reveals two things. First, the "unpredicted" branch cost seems to be correlated with the branch distance. The longer the branch the costlier it is. We haven't seen such behaviour on x86 CPUs.

Then there is the cost itself. We saw a predicted sequence of branches cost, and what a supposedly-unpredicted jmp costs. In the first chart we saw that beyond ~192KiB working code, the branch predictor seems to become ineffective. The supposedly-flushed BPU seems to show the same cost. For example, the cost of a 64-byte block size jmp with a small working set size is 3 cycles. A miss is ~8 cycles. For a large working set size both times are ~8 cycles. It seems that the BTB is linked to the L1 cache state. Paul A. Clayton suggested a possibility of such a design back in 2016.

Top tip 6. on M1 the predicted-taken branch generally takes 3 cycles and unpredicted but taken has varying cost, depending on jmp length. BTB is likely linked with L1 cache.

The call/ret chart is funny:

Like in the chart before, we can see a big benefit if hot code fits within 4096 bytes (512*4 or 256*8). Otherwise, you can count on 4-6 cycles per call/ret sequence (or, bl/ret as it's known in ARM). The chart shows funny alignment issues. It's unclear what they are caused by. Beware, comparing the numbers in this chart with x86 is unfair, since ARM call operation differs substantially from the x86 variant.

M1 seems pretty fast, with predicted branches usually at 3 clock cycles. Even unpredicted branches never cost more than 8 ticks in our benchmark. Call+ret sequence for dense code should fit under 5 cycles.


We started our journey from a piece of trivial code, and asked a basic question: how costly is  adding a never-taken if branch in the hot portion of code?

Then we quickly dived in very low level CPU features. By the end of this article, hopefully, an astute reader might get better intuition how a modern branch predictors works.

On x86 the hot code needs to split the BTB budget between function calls and taken branches. The BTB has only a size of 4096 entries. There are strong benefits in keeping the hot code under 16KiB.

On the other hand on M1 the BTB seems to be limited by L1 instruction cache. If you're writing super hot code, ideally it should fit 4KiB.

Finally, can you add this one more if statement? If it's never-taken, it's probably ok. I found no evidence that such branches incur any extra cost. But do avoid always-taken branches and function calls.


I'm not the first person to investigate how BTB works. I based my experiments on:


Thanks to David Wragg and Dan Luu for technical expertise and proofreading help.


Oh, oh. But this is not the whole story! Similar research was the base to the Spectre v2 attack. The attack was exploiting the little known fact that the BPU state was not cleared between context switches. With the correct technique it was possible to train the BPU - in the case of Spectre it was iBTB - and force a privileged piece of code to be speculatively executed. This, combined with a cache side-channel data leak, allowed an attacker to steal secrets from the privileged kernel. Powerful stuff.

A proposed solution was to avoid using shared BTB. This can be done in two ways: make the indirect jumps to always fail to predict, or fix the CPU to avoid sharing BTB state across isolation domains. This is a long story, maybe for another time...


1. One historical solution to this specific 'if debug' problem is called "runtime nop'ing". The idea is to modify the code in runtime and patch the never-taken branch instruction with a nop. For example, see the "ISENABLED" discussion on

2. Fun fact: modern compilers are pretty smart. New gcc (>=11) and older clang (>=3.7) are able to actually optimize it quite a lot. See for yourself. But, let's not get distracted by that. This article is about low level machine code branch instructions!

3. This is a simplification. There are of course more control flow instructions, like: software interrupts, syscalls, VMENTER/VMEXIT.

4. Ok, I'm slightly overinterpreting the chart. Maybe the 4096 jmp mark is due to the 4096 uop cache or some instruction decoder artifact? To prove this spike is indeed BTB related I looked at Intel BPUCLEARS.EARLY and BACLEAR.CLEAR performance counters. Its value is small for block count under 4096 and large for block count greater than 5378. This is strong evidence that the performance drop is indeed caused by the BPU and likely BTB.

Categories: Technology

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Tue, 27/04/2021 - 14:00
SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

A few weeks ago I received a Raspberry Pi 400 as a gift. I didn’t have time to do anything beyond plug it in and verify that it works. It’s great that the Pi 400 comes with everything you need except for a screen: there’s the computer itself, mouse, HDMI cable and power adapter.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

The Pi 400 has been sitting gathering dust when Cloudflare launched Auditable Terminal giving me the perfect excuse to get out the Pi 400 and hook it up.

Auditable Terminal gives you a fully featured SSH client in your browser. You authenticate using Cloudflare Access and can log into a computer from anywhere just using the browser and get a terminal. And using Cloudflare Tunnel you can securely connect a computer to Cloudflare without punching holes in a firewall. And you end up with a consistent terminal experience across devices: 256 colours, Unicode support and the same fonts everywhere.

This is ideal for my use case: set up the Pi 400 on my home network, use Cloudflare Tunnel to connect it to the Cloudflare network, use Auditable Terminal to connect to the Pi 400 via Cloudflare and the tunnel using nothing more than a browser.

Here’s what it looks like:

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Before getting into the details of how I set that up, it’s worth stopping and appreciating how cool this is. I am logging into the Pi 400 via SSH but inside the browser window just by visiting a URL in a browser and authenticating using Cloudflare for Teams. And since it’s a URL that SSH session can just be a browser bookmark!

There’s another reason I wanted to do this: I like to use Cloudflare’s own products. I have a number of Cloudflare accounts that I pay for so that I see the true customer experience. I see the emails we send out and what it’s like to navigate our UI for real use. The Pi 400 gave me a chance to experience with Cloudflare Tunnel and Auditable Terminal.

Also this is really cool.

So… if you want to do this yourself, follow along as I take you through the steps I went through to hook a brand new Pi 400 up to Cloudflare and access it from anywhere.

Stage 1: Prepare the Pi

I plugged the Pi 400 into my TV via HDMI, to the Internet via Ethernet and booted it. It comes with the OS pre-installed on an SD card so it only took seconds to be at a welcome screen:

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Inevitably with a brand new machine there were a lot of updates to install so I let the machine run through that before logging in for the first time.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

The Pi 400 doesn’t come with the SSH server enabled, so it’s necessary to run the raspi-config program from the command line (sudo raspi-config). The SSH server is under option “3 Interface Options”:

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

It’s option “P2 SSH” and when turned on will allow SSH access to the machine. By default this will be using SSH with password authentication and so it’s pretty important to change the default pi/raspberry combination (and to go much further and switch to using certificates).

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

But for the rest of this run through I am going to stick with password authentication in SSH. We’ll see later that Cloudflare has a clever solution to setting up SSH security with certificates.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

With SSH access setup and the Pi 400 fully updated, the second step is to set up Cloudflare Tunnel.

Stage 2: Cloudflare Tunnel

Even with SSH setup on the Pi 400 it’s only accessible from inside my home network and I want to get access to it from anywhere. I could open a port in my home firewall, but I hate that idea.

The answer is Cloudflare Tunnel. It’s a (free!) small daemon (called cloudflared) that will connect from the Pi 400 to multiple Cloudflare data centers and maintain connections to them. It’ll be authenticated to my Cloudflare account and once setup I’ll be able to use Cloudflare for Teams to connect to it via the web.

Setting it up is a doddle. I already had a domain name setup on Cloudflare, if I hadn’t I could have set one up quickly. The Cloudflare Tunnel documentation takes you through installing it.

The only thing I didn’t know was the architecture of the Pi 400 (32 bit? 64 bit?); so I ran lscpu which tells me that it’s armv7l (which is 32-bit). How did I know that armv7l was 32-bit? I didn’t. I googled it.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

You can build cloudflared from source if you wish as it’s an open source project. But I chose to go the speedy route and wget it.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

With `cloudflared` installed on the Pi 400 it was time to authenticate it to my account. Authentication is clearly documented and I started by typing cloudflared tunnel login. On a machine with a UI and browser this would have opened the browser ready for me to log into my Cloudflare account and set up cloudflared.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Since I was doing all this while SSH’d into the Pi I had to copy and paste the URL to my browser and then choose which of my domains I would use. (I did this before we changed the name from Argo Tunnel to Cloudflare Tunnel… it’s the same thing).

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Once authorized, the cloudflared command running on the Pi 400 automatically sets up the tunnel. In the dashboard I received confirmation of this:

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

And back on the Pi 400 there was confirmation that cloudflared had been authorized and that a cert.pem file was created.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Now that cloudflared was authorized it was time to go ahead and set up an actual tunnel that can be used for the Auditable Terminal. I chose the incredibly original name ‘terminal’ for this.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

At this point the tunnel is set up but not running yet. So the Pi 400 is not yet connected to Cloudflare. I’ll come back to that. But the next step is setting up Cloudflare for Teams so that users can authenticate to the other end of the tunnel and get access.

Stage 3: Cloudflare for Teams

I didn’t have Cloudflare for Teams setup so the first step was to visit and following the setup guide. I chose the unoriginal team name jgctesting and entered it:

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

And it’s here that things got complicated. I ended up writing this blog post to give anyone a step-by-step guide to getting Auditable Terminal working, but also so that we can learn from my confusion. (The team has put up a complete tutorial on this here.)

If you follow the Cloudflare for Teams documentation you have three choices.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

It wasn’t 100% obvious to me that what I needed to do was add an application (as opposed to a location). The answer is… add an application. The process for doing so is documented here.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

In particular, I’m going to add a Self-Hosted Application (rather than SaaS) since I’m self-hosting it; it’s an SSH server on a Raspberry Pi 400 hidden behind the sofa after all.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

The next step is to give the application a name and specify the URL it will be accessible via. For me that’s “Raspberry Pi 400” (yeah, I’m great with original names) and I’m making it available via

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Cloudflare for Teams provides a lot of different authentication and identity options, but for this demo I am going to use the simplest. When I want to connect to this application (which will be Auditable Terminal ultimately) I’m going to have Cloudflare email me a one-time PIN.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Next up is adding a rule which will let Cloudflare for Teams know who should be allowed access to this application. I am allowing anyone with an email address to get a one-time PIN. Rules can be much more complex than that!

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

And finally I need to enable browser rendering (which will enable the SSH access and Auditable Terminal).

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable TerminalStage 4: Connecting it all together

At this stage I had cloudflared setup and authorized on the Pi 400, Cloudflare for Teams setup so that anyone with an email could get a PIN to access Auditable Terminal on (I’ve since reconfigured everything so going to that URL doesn’t do anything today).

But all these things need connecting together. That means: editing a config file on the Pi 400 and telling Cloudflare to route to the tunnel for

I started with the config file which was /home/pi/.cloudflared/config.yml. The first step was to get the tunnel ID.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

And then edit the config file. Three things needed changing from the defaults: the tunnel (which is just the ID from cloudflared tunnel list), the credentials-file (which is a JSON file in the same location as the config file and with the same name as the tunnel ID) and the hostname (in my case

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

With the tunnel installed, authorized and configured I was able to finally run it and make sure that it can connect to Cloudflare. It worked! And created four connections to Cloudflare. Two to Lisbon and two to Amsterdam. Cloudflare Tunnel manages these connections to ensure that my Pi 400 is reachable across the Cloudflare network.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

And finally… I just needed to tell cloudflared to route to the tunnel (once again using that tunnel ID).

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable TerminalStage 5: Try it!

And then it was time to try it. I went back to my browser and browsed to and was greeted with a login screen. Since I’d authorized any address I entered an email.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

The next step was to enter the one-time PIN that was emailed to me.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

And, since I am using user/password for SSH I was asked for the user name followed by password.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

And, wow, there I was looking at a command prompt on the Pi 400.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

Of course, just doing on my laptop wasn’t enough, so I went ahead and logged in the same way on my phone.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

The absolute last step was to make sure that cloudflared runs automatically which was as simple as typing sudo cloudflared --config ./cloudflared/config.yml service install.


This was pretty fast (especially factoring in my ignorance about all the details of configuring Cloudflare for Teams and Cloudflare Tunnel). Now I can log into that Pi 400 from anywhere in just a browser. So powerful.

But it’s still using username/password for SSH and that can be fixed with short-lived certificates. And my authentication to access is using one-time PINs; for a real application it would be better to use SSO.

Categories: Technology

Project Jengo Redux: Cloudflare’s Prior Art Search Bounty Returns

Mon, 26/04/2021 - 14:00
 Cloudflare’s Prior Art Search Bounty Returns Cloudflare’s Prior Art Search Bounty Returns

Here we go again.

On March 15, Cloudflare was sued by a patent troll called Sable Networks — a company that doesn’t appear to have operated a real business in nearly ten years — relying on patents that don’t come close to the nature of our business or the services we provide. This is the second time we’ve faced a patent troll lawsuit.

As readers of the blog (or followers of tech press such as ZDNet and TechCrunch) will remember, back in 2017 Cloudflare responded aggressively to our first encounter with a patent troll, Blackbird Technologies, making clear we wouldn’t simply go along and agree to a nuisance settlement as part of what we considered an unfair, unjust, and inefficient system that throttled innovation and threatened emerging companies. If you don’t want to read all of our previous blog posts on the issue, you can watch the scathing criticisms of patent trolling provided by John Oliver or the writers of Silicon Valley.

We committed to fighting back against patent trolls in a way that would turn the normal incentive structure on its head. In addition to defending the case aggressively in the courts, we also founded Project Jengo — a crowdsourced effort to find evidence of prior art to invalidate all of Blackbird’s patents, not only the one asserted against Cloudflare. It was a great success — we won the lawsuit, invalidated one of the patent troll’s other patents, and published prior art on 31 of Blackbird’s patents that anyone could use to challenge those patents or to make it easier to defend against overbroad assertion of those patents. And most importantly, Blackbird Technologies went from being one of the most prolific patent trolls in the United States to shrinking its staff and filing many fewer cases.

We’re going to do it again. And we need your help.

Turning the Tables — A $100,000 Bounty for Prior Art

Sable Networks and its lawsuit fit neatly within the same troubling trends we were trying to address the first time we launched Project Jengo. Sable is taking ancient, 20-year-old patents and trying to stretch those patents lightyears beyond what they were meant to cover. It has already sued over a dozen technology companies targeting a wide range of different products and services, and by extending its claims to a company like Cloudflare suggests it may next try to stretch its claims to people that merely use routers … namely, anyone that uses the Internet.

We think Sable’s choice to bring these lawsuits on such a tenuous basis should come with some risk related to the underlying merits of its patents and its arguments, so we are sponsoring another prior-art contest seeking submissions to identify prior art for all of Sable’s active patents. We are seeking the help of the Cloudflare community to identify prior art — i.e., evidence that the patented technology was already in use or known before the patent application was filed — that can be used to invalidate Sable’s patents. And we will make it worth your while, by offering $100,000 to be shared by the winners who are successful in finding such prior art.

Again this time, we are committing $100,000 to be split among entrants who provide what we determine to be the most useful prior-art references that can be used in challenging the validity of Sable’s patents. You can submit prior-art references as long as Sable’s case is pending against us (Sable Networks, Inc. v. Cloudflare, Inc., No. 6:21-cv-00261-ADA (W.D. Tex.)), which means until Sable drops the case fully (and with prejudice — meaning Sable can’t re-file later), there’s a settlement, or the case has been resolved by the court and all appeal rights are exhausted.

Every three months for two years or until the case ends, whichever comes first, we will select winners from the submissions to date, and give out a portion of the $100,000 as awards. Once the case ends, we will select final winners from all submissions and award the remaining funds. We will also make all relevant submissions available to the public.

Here are the four Sable patents asserted against us:

U.S. Patent No. Title Earliest Potential Priority Date 6,954,431 Micro-flow management Apr. 19, 2000 6,977,932 System and method for network tunneling utilizing micro-flow state information Jan. 16, 2002 7,012,919 Micro-flow label switching Apr. 19, 2000 8,243,593 Mechanism for identifying and penalizing misbehaving flows in a network Dec. 22, 2004

And here are the six remaining Sable patents:

U.S. Patent No. Title Earliest Potential Priority Date 6,854,117 Parallel network processor array Oct., 31, 2000 7,428,209 Network failure recovery mechanism June 12, 2001 7,630,358 Mechanism for implementing multiple logical routers within a single physical router July 9, 2001 8,085,775 Identifying flows based on behavior characteristics and applying user-defined actions July 31, 2006 8,817,790 Identifying flows based on behavior characteristics and applying user-defined actions July 31, 2006 9,774,501 System and method for ensuring subscriber fairness using outlier detection May 14, 2012

In addition to helping with our case, we hope that making prior art public on all of Sable’s patents will provide a head start and decrease the costs for others who want to fight back against Sable’s patent trolling. The significant decrease we saw in Blackbird’s staff and filings after publishing prior art on Blackbird’s patents suggests this approach can be an effective way to undermine the threat posed by those patents.

 Cloudflare’s Prior Art Search Bounty Returns

Last time, we received 275 submissions from 155 different people on 49 of the Blackbird patents. In the end, we made payments to 18 participants. We were heartened by how many members of the Cloudflare community not only participated in this effort, but expressed great gusto and appreciation for the opportunity to be involved:

Over the years I've been disappointed and angered by a number of patent cases where I feel that the patent system has been abused by so-called ‘patent trolls’ in order to stifle innovation and profit from litigation. With Jengo in particular, I was a fan of what Cloudflare had done previously with Universal SSL. When the opportunity arose to potentially make a difference with a real patent troll case, I was happy to try and help.
— Adam, Security EngineerI'm pretty excited, I've never won a single thing in my life before. And to do it in service of taking down evil patent trolls? This is one of the best days of my life, no joke. I submitted because software patents are garbage and clearly designed to extort money from productive innovators for vague and obvious claims. Also, I was homeless at the time I submitted and was spending all day at the library anyway.
— Garrett, San FranciscoResurrecting Project Jengo

To understand why we are asking again for your help fighting back, it’s worth taking a closer look at this case and the fundamental problems it represents.

Sable Networks and the “case” against Cloudflare

The patents at issue in this case started with Caspian Networks, a company that tried to commercialize what it called a “flow-based router” in the early 2000s. Caspian was originally founded as Packetcom in 1998, and revealed to the public its flow-based router named Apeiro in 2003. A press story from that time explained that Apeiro routers worked like traditional routers already in existence, but with additional memory and logic for handling packets from the same “flow.” A 2003 slide deck from Caspian distinguished its “flow-based” router from the already existing conventional routers in the following way:

 Cloudflare’s Prior Art Search Bounty Returns

Despite its attempts to tout the benefits of its router, Caspian went out of business in 2006.

That’s when Sable Networks, the company that is suing Cloudflare, enters the picture. Though it doesn’t appear to be much of an entrance. As best we can tell, Sable briefly picked up where Caspian left off and tried to commercialize Caspian’s flow-based routing technology. Sable was equally unsuccessful in doing so, and the last activity of any sort that we could find was from 2011. After a long period of apparent inactivity, Sable’s focus shifted last year to trying to extract money by filing lawsuits through broad application of patents filed on Caspian’s flow-based router technology twenty years ago. In other words, Sable became a patent troll.

In the first round of litigation, Sable filed, and later promptly settled, eight lawsuits asserting infringement of Sable’s router patents. The defendants in those cases (including Cisco and Juniper Networks) provide a range of Internet services, but they all at least manufacture and sell network equipment.

Interestingly, all of those cases were settled just before Sable would have had to do two things that would have actually put its legal claims to the test: (1) respond to an administrative proceeding before the US Patent & Trademark Office (“USPTO”) challenging the validity of its patents; and (2) attend a hearing before the district court where the judge would have determined the proper interpretation and scope of the patent claims. So Sable filed cookie-cutter cases against eight defendants, waited for the defendants to respond, then settled the cases before meaningfully litigating its claims or facing a binding court or administrative ruling, which may have addressed, or likely undermined, Sable’s overly-broad assertion of those patents.

Shortly after settling the original eight cases earlier this year, Sable turned around and filed six new lawsuits against a new batch of technology companies, this time including Cloudflare. Unlike the earlier named defendants, Cloudflare is not in the business of making or selling routers or switches. Sable’s infringement claim therefore is not a close one, and now it’s picked a defendant that is eager to fight back.

This case is a good illustration of how patent trolls operate. All four patents asserted by Sable against Cloudflare were filed between 2000 and 2004, when dial-up Internet access was still common, and are based on Caspian’s “flow-based routing” technology, which is nothing like the technology Cloudflare’s products and services employ. To take one example, one of the patents Sable asserts is U.S. Patent No. 6,954,431, entitled “Micro-Flow Management.” This patent is from April 19, 2000 — almost exactly 21 years ago. Just like Caspian’s Apeiro routers from 2003, the ’431 patent discloses a router that puts a label on the packets for a given flow, and forwards all the packets in that same flow based on the label:

 Cloudflare’s Prior Art Search Bounty Returns’431 Patent, Fig. 3A‌‌

It claims to teach a “[n]ew switching technology [that] relies upon state information for providing a previously unavailable degree of quality of service” — presumably, Caspian’s flow-based routing technology featured in its 2003 Apeiro, which the market rejected over a decade ago.

Sable is now trying to stretch the patents way beyond what they were ever meant to cover. Many of Sable’s infringement claims appear to extend to basic routing and switching functionality known long before any alleged invention date. For the ’431 patent, Sable’s interpretation of the patent appears to stretch the scope so broadly as to cover any kind of packet processing, possibly even the “conventional routers” from the 2000s that routed each packet independently.

Having made this leap, Sable has demonstrated an intent to apply its patents far afield. It’s now a small step for Sable to assert claims against any person or business using a firewall or even a router. On the basis of its logic, any person using a WiFi router in their home may be in Sable’s cross-hairs. And Sable has already claimed that its patents cover firewalls — including firewall software and firewall devices. Again, its wildly broad interpretation threatens not just large companies; anyone trying to protect their networks from outside threats is potentially at risk.

If you think you or your small business are safe because you haven’t really done anything wrong, and no court would ever hold you liable, then you haven’t been paying attention to the current state of patent litigation where small businesses are sued for using copy machines. Without some of us fighting back, the incentives for patent trolls won’t change.

A broken incentive structure….still broken

Patent trolls like Sable proliferate because of a distorted incentive structure fueled by the astronomical costs associated with defending against even bogus patent claims. According to the 2019 Report of the Economic Survey by the American Intellectual Property Law Association, the median litigation cost for defending claims of patent infringement brought by a non-practicing entity through trial and appeal was staggering \$4,500,000 for big cases (i.e., cases with more than \$25 million at risk). Even for small cases that had less than $1 million at stake, the median defense cost was \$750,000.

Knowing that most defendants will settle at a percentage of their expected litigation costs long before infringement claims see the inside of a courtroom, patent trolls see only upside from each additional lawsuit they file. Their business model is built around filing as many lawsuits as possible regardless of the strength of their legal claims, because they know most defendants will pay to settle before the merits of their case are put to the test. They therefore take vague technology patents issued years ago and apply them as broadly as imaginable to new technologies and new companies. Most of these trolls are non-practicing entities who don’t have their own products in the marketplace but merely exist to extract a tax on other companies.

After declining in numbers for a few years, patent lawsuits were up last year about 12% over 2019, and increased activity by non-practicing entities buying patents suggested even further growth among such lawsuits in the future. The Electronic Frontier Foundation observed similar increases in patent troll filings in 2020, highlighted a case where one patent troll decided that the middle of a pandemic was a good time to sue a company that makes COVID-19 tests, and noted that “some patent owners actually saw the rise of the COVID-19 health emergency as a business opportunity.” This trend is showing no signs of slowing down — according to a recent report, there was a 43.3% increase in patent litigation in Q1 of 2021 compared to the same period a year ago, and non-practicing entities had their busiest Q1 since 2015, fueled by the litigation finance industry that is flush with capital.

To reaffirm what we’ve always said: Cloudflare is a strong supporter of the patent system. The 160+ patents we’ve been issued to date give us the ability to run our business with confidence by marketing and providing our services around the world, innovating on our existing products, and developing new products. But patent trolls live in a world where they allege fictional potential uses of their patents without having to make the investment to hire and pay employees or identify and satisfy the demands of the marketplace. Project Jengo is intended to address this distorted incentive structure by imposing real costs on the filing of meritless patent lawsuits. With your help, we can again send the message that by filing the wrong case, patent trolls risk losing not just a single lawsuit but a whole lot more.

We hope you’re all rested and ready to begin again the hunt for prior art!

Categories: Technology

Cloudflare obtains new ISO/IEC 27701:2019 privacy certification and what that means for you

Wed, 21/04/2021 - 14:00
Cloudflare obtains new ISO/IEC 27701:2019 privacy certification and what that means for you

This post is also available in French and German.

Cloudflare obtains new ISO/IEC 27701:2019 privacy certification and what that means for you

Cloudflare is one of the first organisations in our industry to have achieved ISO/IEC 27701:2019 certification, and the first web performance & security company to be certified to the new ISO privacy standard as both a data processor and controller.

Providing transparency into our privacy practices has always been a priority for us. We think it is important that we do more than talk about our commitment to privacy — we are continually looking for ways to demonstrate that commitment. For example, after we launched the Internet's fastest, privacy-first public DNS resolver,, we didn’t just publish our commitments to our public resolver users, we engaged an independent firm to make sure we were meeting our commitments, and we blogged about it, publishing their report.

Cloudflare obtains new ISO/IEC 27701:2019 privacy certification and what that means for you

Following in that tradition, today we’re excited to announce that Cloudflare has been certified to a new international privacy standard for protecting and managing the processing of personal data — ISO/IEC 27701:2019. The standard is designed such that the requirements organizations must meet to become certified are very closely aligned to the requirements in the EU’s General Data Protection Regulation (“GDPR”). So this certification provides assurance to our customers that a third-party has independently verified that Cloudflare’s privacy program meets GDPR-aligned industry standards.

What is ISO/IEC 27701:2019?

The International Organization for Standardization (“ISO”) is an international, nongovernmental organization made up of national standards bodies that develops and publishes a wide range of proprietary, industrial, and commercial standards. In August 2019, ISO published ISO/IEC 27701:2019 (“ISO 27701”), a new international privacy standard about protecting and managing the processing of personal data.

This new standard is a privacy extension to the existing and widespread industry standards ISO/IEC 27001 and ISO/IEC 27002, which were first published by ISO in 2005. They describe how to establish and run an Information Security Management System (“ISMS”), and ISO now reports that over 36,000 organisations in 131 countries are currently independently certified as meeting ISO/IEC 27001. Audited ISO certifications are awarded to organizations that have been assessed by an independent, external auditor to meet a specific, published standard. Auditors are also accredited themselves — with the ISO 27000 series of certifications, to published international ISO standards, too.

The ISO 27701 extension to the ISO/IEC 27001 and ISO/IEC 27002 standards is less than two years old and adapts the ISMS management system concept into the creation of a Privacy Information Management System (“PIMS”). There are requirements to make sure this privacy management system is robust and is also continually improving to meet its defined objectives.

We are excited about this new certification because ISO 27701 maps to the requirements of the GDPR, the EU’s benchmark-setting, comprehensive data protection regulation. Article 42 of the GDPR encourages:

...the establishment of data protection certification mechanisms and of data protection seals and marks, for the purpose of demonstrating compliance with this Regulation of processing operations by controllers and processors.

While Article 42 calls for the development of GDPR certifications, no such official certifications exist yet because none have been approved by either of the official bodies — the European Data Protection Board in the EU, or the UK’s Information Commissioner’s Office in respect of the UK GDPR. However, when the ISO 27701 standard was published, it contained an Annex D detailing how the standard maps to the GDPR:

This annex gives an indicative mapping between provisions of this document and Articles 5 to 49 except 43 of the General Data Protection Regulation of the European Union. It shows how compliance to requirements and controls of this document can be relevant to fulfil obligations of GDPR.

ISO standards often map to — and frequently reference — other international ISO standards, but it’s unusual for them to map to non-ISO standards, especially to one particular region’s regulations. So until the GDPR regulatory bodies adopt an official certification mechanism, ISO 27701 provides an excellent way to demonstrate externally-audited compliance with the regulation.

What does ISO 27701 mean for Cloudflare customers?

Put simply, the ISO 27701 certification provides assurance to our customers that we have a privacy program that has been assessed by a third-party to meet an international industry standard aligned to the GDPR, and that requires us to keep our privacy program under continuous compliance. This certification, in addition to the Data Processing Addendum (“DPA”) we make available to our customers in the dashboard, offers our customers multiple layers of assurance that any personal data that Cloudflare processes will be handled in a way that meets the GDPR’s requirements.

Let us do a deeper dive into some of the requirements under ISO 27701
The standard contains 31 controls identified for organizations that are personal data controllers, and 18 additional controls identified for organisations that are personal data processors. As Cloudflare’s scope is certifying as both a personal data controller and as a personal data processor of customer information, we had to meet all 49 of these controls.

The controls are essentially a set of best practices that data controllers and processors must meet in terms of data handling practices and transparency about those practices, documenting a legal basis for processing and for transfer of data to third countries (outside the EU), and handling data subject rights, among others.

Example Requirement 1:
Organizations should maintain policy and document specific procedures related to the international transfer of personal data.

Cloudflare has implemented this requirement by maintaining an internal policy restricting the transfer of personal data between jurisdictions unless that transfer meets defined criteria. Customers, whether free or paid, enter into a standard Data Processing Addendum with Cloudflare which is available on the Cloudflare Customer Dashboard and which sets out the restrictions we must adhere to when processing personal data on behalf of customers, including when transferring personal data between jurisdictions. Additionally, Cloudflare publishes a list of sub-processors that we may use when processing personal data, and in which countries or jurisdictions that processing may take place.

Example Requirement 2:
Organizations should maintain documented personal data minimization objectives, including what mechanisms are used to meet those objectives.

Cloudflare maintains internal policies on how we manage data throughout its full lifecycle, including data minimization objectives. In fact, our commitment to privacy starts with the objective of minimizing personal data. That’s why, if we don’t have to collect certain personal data in order to deliver our service to customers, we’d prefer not to collect it at all in the first place. Where we do have to, we collect the minimum amount necessary to achieve the identified purpose and process it for the minimum amount necessary, transparently documenting the processing in our public privacy policy.

We’re also proud to have developed a Privacy by Design policy, which rigorously sets out the high-standards and evaluations that must be undertaken if products and services are to collect and process personal data. We use these mechanisms to ensure our collection and use of personal data is limited and transparently documented.

Cloudflare achieves ISO 27701:2019 Certification

Cloudflare’s PIMS was assessed by a third-party auditor, A-LIGN in March 2021. Certifying to the ISO 27701 privacy standard is a multi-step process that includes:

  • understanding and planning for the standard;
  • identifying and adapting the controls the organization will implement;
  • internally auditing against the requirements;  and
  • externally auditing against the standard (itself a two-stage process)

before finally being certified against the standard by the independent auditor. Once certified, the privacy management system is continually evaluated and improved, with internal and external audits on an ongoing annual basis.

Cloudflare has been certified as both a data processor and as a data controller of customer information.[¹] This means that Cloudflare is one of the first organisations in our industry to have achieved this standard, and the first web performance & security company to be certified to ISO 27701 as both a data controller and processor. Alongside Cloudflare’s existing ISO 27001:2013 certificate, Cloudflare’s new ISO 27701:2019 certificate is now available for customers to request from their sales representative.

Cloudflare Certifications

For more information about our certifications and reports, please visit our privacy and compliance pages - You can also reach us at for any questions.

[1]The GDPR defines a “data controller” as the “natural or legal person . . . or other body which, alone or jointly with others, determines the purposes and means of the processing of personal data”; and a “data processor” as “a natural or legal person . . . which processes personal data on behalf of the controller.”

Categories: Technology

Announcing Cloudflare Images beta to simplify your image pipeline

Tue, 20/04/2021 - 18:00
Announcing Cloudflare Images beta to simplify your image pipelineAnnouncing Cloudflare Images beta to simplify your image pipeline

Today, we are announcing the beta of Cloudflare Images: a simple service to store, resize, optimize, and deliver images at scale.

In 2018, we launched Stream to provide a single product that could be used to store, encode, and deliver videos. With Cloudflare Images, we are doing for images what Stream did for videos. Just like Stream, Cloudflare Images eliminates the need to think about storage buckets, egress costs, and many other common problems that are solved for you out of the box. Whether you are building an ecommerce platform with millions of high-res product pictures and videos or a new app for creators, you can build your entire media pipeline by combining Cloudflare Images and Stream.

Fundamental questions for storing and serving images

Any time you are building infrastructure for image storage and processing, there are four fundamental questions you must answer:

  1. “Where do we store images?”
  2. “How do we secure, resize, and optimize the images for different use cases?”
  3. “How do we serve the images to our users reliably?”
  4. “How do we do all of these things at scale while having predictable and affordable pricing, especially during spikes?”

Cloudflare Images has a straightforward set of APIs and simple pricing structure that answers all of these pesky questions. We built Images so your team can spend less energy maintaining infrastructure and more time focusing on what makes your product truly special.

Current state of image infrastructureAnnouncing Cloudflare Images beta to simplify your image pipeline

We talked to many Cloudflare customers who are using Cloudflare to serve millions of images every day. We heard two recurring themes. First, customers wished there was a simpler way for them to securely store, resize, and serve the images. Their current infrastructure generally involves using product A for storage, product B for resizing, and product C for the actual delivery. Combining these products together (often from multiple vendors) quickly becomes messy with multiple points of failure. Moreover, maintaining this infrastructure requires ongoing monitoring and tweaks.

Second, we heard that customers often end up with an ever-growing egress bill due to multiple vendors and products. It blew our minds that the egress bill can be a multiple of the storage cost itself. Every time your pictures move from product A (storage provider) to product B (resizing service) to product C (the CDN), you generally pay an egress cost which can quickly add up depending on the number of pictures and their variants. Multiplied by tens of millions of images and variants, this cost can add up to tens of thousands of dollars per month.

Why Cloudflare ImagesAnnouncing Cloudflare Images beta to simplify your image pipelineEliminate egress cost and storage bucket hell

Each time you upload an image to Cloudflare Images, you receive an :image_id. There are no buckets and folders to manage the originals and the variants. And because of Images built-in support for resizing and delivery, there is no egress cost. If you have internal metadata that you’d like to associate with a picture, you can set the meta field for every upload to any arbitrary JSON value.

Simple APIs catered to your needs

When talking to customers we saw two main patterns of how customers would like to deliver images:

  1. Upload an image and get an :image_uid back that allows future operations on the image. In this case, the image URL would be
  2. Upload images with the filename as their main identifier, e.g. filename reflects SKU.
    In this case, it is up to you to make sure there are no duplicate filenames, as they would be rejected.

Here the image URL would be

Resize and secure your pictures with Variants

Cloudflare Images supports Variants. When you create a variant, you can define properties including variant name, width, height, and whether the variant should be publicly accessible. You can then associate an image with one or more variants using the UI or the API.

Announcing Cloudflare Images beta to simplify your image pipeline

Let’s say you are storing user profile pictures. You could define a variant called “profile-thumbnail” and configure that variant to serve images of a fixed width and height. Once you have a variant, you can associate the profile pictures with the profile-thumbnail variant. Whenever you need to display a profile picture in your app, you simply call or

Variants also offer access control. You might only want logged-in users to view the larger version of the profile pictures. You could create another variant called large-profile-picture and make it require a signed URL token. When a user tries to access the large profile picture with a URL such as, the request will fail because there is no valid token provided.

An indirect upside of using variants is that your organization has a single source of truth for the different ways you are using images across your apps. Different teams can create variants for their use cases, enabling you to audit the security and optimization settings for different types of pictures from one central place. We learned that as our customers' products grow in complexity, separate teams may be responsible for handling various aspects of the product. For example, one team may be responsible for user profile pictures and the different variants associated with it. Another team may be responsible for creator uploads and maintaining different variations that are available to the public and to paid members. Over time, organizations can lose track of this logic with no single source of truth. With variants, this logic is clearly laid out and can serve as the source of truth for all the ways you are securing and optimizing different types of image uploads in your product.

There is no additional cost for using variants. Every picture uploaded to Images can be associated with up to five variants. You will be able to associate an image with a variant using the UI or the API.

Intelligent delivery for every device and use case

Cloudflare Images automatically serves the most optimized version of the image. You no longer need to worry about things like file extensions. When a client requests a picture hosted on Cloudflare Images, we automatically identify the ideal supported format at the Cloudflare edge and serve it to the client from the edge. For example, 93.55% of all users use a web browser that supports webp. For those users, Images would automatically serve webp images. To the remaining users, Images would serve PNGs (and in very rare cases where neither webp or PNGs are supported, it would serve JPGs). In future, we plan to automatically support AVIF for highly-requested images.

When you use Images, you no longer need to worry about cache hit rates, image file types, configuring origins for your image assets.

Simple pricing

To use Cloudflare Images, you will pay a fixed monthly fee for every 100,000 images stored in Cloudflare Images (up to 10MB per image). And at the end of each month, you pay for the number of images served. There are no additional costs for resizing, egress, or optimized routing.

Request an Invite

If you want to be part of Cloudflare Images beta, request an invite. We will start inviting a limited set of users to try out Cloudflare Images in the coming weeks. Pricing and developer docs for Images will be posted at the time we start sending invites.

We can’t wait to see what you build using Cloudflare Images!

Categories: Technology

Start building your own private network on Cloudflare today

Tue, 20/04/2021 - 14:00
Start building your own private network on Cloudflare todayStart building your own private network on Cloudflare today

Starting today, your team can create a private network on Cloudflare’s network. Team members click a single button to connect to private IPs in environments that you control. Cloudflare’s network routes their connection through a data center in one of over 200 cities around the world. On the other side, administrators deploy a lightweight software connector that replaces traditional VPN appliances.

Cloudflare’s private network combines IP level connectivity and Zero Trust controls. Thick clients like RDP software, SMB file viewers, or other programs can connect to the private IPs already in use in your deployment without any additional configuration. Coming soon, you’ll be able to layer additional identity-based network-level rules to control which users, from which devices, can reach specific IPs.

We are launching this feature as a follow-up to Cloudflare’s Developer Week because we are excited to give your development team, and your entire organization, a seamless platform for building and connecting your internal resources. We built this solution based on feedback from customers who want to move to a Zero Trust model without sacrificing some of the convenience of a private network.

We’re excited to give any team the ability to run their internal network on Cloudflare’s global edge. Organizations that have 50 or fewer team members can use this feature, as well as nearly all of Cloudflare for Teams, at no cost by starting here.

Challenges with non-web applications

Over the last three years, Cloudflare Access has helped thousands of organizations replace their VPN with a Zero Trust model. Most of those teams started with web applications like homegrown intranet sites or self-hosted tools. In less than 10 minutes, customers could connect an application to Cloudflare’s network, add Zero Trust rules, and make connectivity seamless and fast for their users.

Web applications make that flow easier thanks to client software that already runs on every device: the browser. Browsers send HTTP requests over the public Internet to the application. Cloudflare’s network checks every request against the Zero Trust rules configured for that application.

Start building your own private network on Cloudflare today

Users are prompted to authenticate and, in some cases, present additional signals like device posture. If the user should be able to reach the application, Cloudflare issues a JSON Web Token (JWT) that the browser stores in the form of a cookie. That token allows for seamless authentication to other applications because they all are available inside of the same web browser.

Start building your own private network on Cloudflare today

Cloudflare's network accelerates traffic to the applications and evaluates every request. Meanwhile, the browser handles authentication storage and HTTP requests trigger Zero Trust checks. No additional client software is required.

Customers gave us two consistent pieces of feedback:

  • “Setup for web applications is seamless.”
  • “What about everything else outside of the browser?”

Use cases outside of the browser introduce two challenges: they each rely on a different piece of client software and they each handle authentication in unique ways. For example, SSH sessions can support client certificates or password authentication. RDP workflows rely on passwords and tend to lack multifactor requirements or SSO integration. Other protocols lack authentication altogether. Exposing any of these directly on the Internet would make them vulnerable to attack.

As a result, organizations hide these types of resources behind a private network as a band-aid. Users toggle their VPN and their client software connects to internal IPs and ports. Administrators suffer through maintaining VPN appliances while their users deal with the slower performance.

Cloudflare attempted to solve this type of use case a couple of years ago. We built an option that relied on a connector, `cloudflared`, that bridged user devices and the environment where the services ran.

The instance of cloudflared running in the data center or cloud environment created a WebSocket connection between the connector and Cloudflare’s edge. End users ran the same connector on their own devices. cloudflared running on the client device exposed a local port which could receive traffic from services like an SMB or RDP client and send it over WebSocket to the corresponding cloudflared in the data center.

Start building your own private network on Cloudflare today

This option was functional, but not viable for small teams without dedicated IT staff or enterprises who do not want to retrain tens of thousands of users. End users had to run a manual command for each service and change the configuration for every client. We had offered full Zero Trust control at the expense of usability.

A private network on Cloudflare’s edge

Today’s announcement combines the usability of a VPN client with the performance and security of Cloudflare’s network while removing the maintenance overhead of networking appliances.

The architecture starts with Cloudflare Tunnel (previously called Argo Tunnel). Cloudflare Tunnel uses the same connector, cloudflared, to create an outbound-only TCP connection from your data center or public cloud environment to two nearby Cloudflare data centers.

Start building your own private network on Cloudflare today

Administrators configure the tunnel to represent a range of IP addresses where applications run in their environment. Those IPs can be RFC 1918 ranges or any IP addresses that cloudflared can address. Teams can also run redundant Tunnels for availability and separate Tunnels in different environments to connect other IP ranges.

Cloudflare’s edge then maps each Tunnel in the organization’s account to the IP range represented. Administrators can review the mapping from any active instance of cloudflared.

Start building your own private network on Cloudflare today

On the client side, end users run an agent, Cloudflare WARP, and authenticate with their identity provider into the same Cloudflare account that administers the Tunnels. They can then click a single button to connect and the WARP agent creates a Wireguard tunnel from the device to Cloudflare’s network.

The Cloudflare WARP agent routes traffic from the device to Cloudflare’s edge. By default, the client excludes traffic to RFC 1918 IP addresses and a few other defaults. In this mode, administrators can configure the client to instead pick up traffic bound for those IP ranges.

When that traffic arrives, Cloudflare’s edge locates the Tunnel in that account that represents the IP range enrolled. If the user connects to the same data center as the Tunnel, Cloudflare proxies a TCP connection by opening a bidirectional stream to the corresponding instance of cloudflared. If the user first reaches a different data center, Cloudflare’s smart routing technology finds the fastest path to the Tunnel.

Client applications that connect to specific IP addresses can continue to do so without any configuration changes. When those applications attempt to reach those IPs, the Cloudflare WARP agent handles routing that traffic to Cloudflare’s edge and to the instance of cloduflared.

cloudflared then operates like a bastion inside of the data center and connects to the services running at those IP addresses.

Security for the rest of the Internet

The Cloudflare WARP agent that connects users to this private network can also keep them safe on the rest of the Internet.

You can start by using Cloudflare WARP to filter DNS queries for devices in any location. We've built that solution on top of the world's fastest DNS resolver,, to stop users from inadvertently connecting to phishing sites, malware, or other threats.

The agent can also help your team adopt a faster Secure Web Gateway and deprecate web filtering hardware. Cloudflare WARP will connect all Internet-bound traffic over a Wireguard tunnel to a nearby data center. Once there, Cloudflare will inspect the HTTP requests and accelerate traffic to its destination on our global backbone network. You can build rules that control where files can be uploaded, filter for viruses inside of traffic, or prevent users from going to certain parts of sites.

How to get started

You can start running your virtual private network on Cloudflare with just four steps.

1. Install and authenticate cloudflared in a data center, public cloud environment, or even on a single server with the command below. Once authenticated, cloudflared will become part of your Cloudflare account and available.

cloudflared tunnel login

2. Create a Tunnel with a name that represents that service or environment.

cloudflared tunnel create grafana

Start building your own private network on Cloudflare today

Next, configure cloudflared to represent the IP address range in your environment. The command below will tell Cloudflare to send traffic from your users to that IP range to this Tunnel.

cloudflared tunnel route ip add 100.64.0/10

Start building your own private network on Cloudflare today

Once configured, you can start the tunnel with a single command or run it as a service.

cloudflared tunnel run grafana

  1. Configure traffic to private IP addresses to be included through WARP, as opposed to being run in the default split tunnel mode.
  2. Enroll your device and enable WARP to connect.
Start building your own private network on Cloudflare today

We've provided a step-by-step tutorial as well to help your team get started. We’d also like to hear how we can keep making this better for your team. Please feel free to send any and all feedback in the community post here.

What’s next?

Available today, security teams can build rules to determine who can enroll into this private network and from which devices. That requirement and the connectivity features available make this option similar to a private network, although one accelerated by Cloudflare.

However, we want to give your team more granular control over who can reach specific resources. We’ll be launching support to build additional Zero Trust rules that apply distinct rules to individual IPs or IP ranges.

Additionally, this flow only works for client-to-server (WARP to cloudflared) connections. Coming soon, we’ll introduce support for east-west connections that will allow teams to connect cloudflared and other parts of Cloudflare One routing.

Categories: Technology

A Full Circle Journey: Introducing Cloudflare Canada

Mon, 19/04/2021 - 14:01
 Introducing Cloudflare Canada Introducing Cloudflare Canada

Today Cloudflare announced that Toronto will be home to Cloudflare’s first Canadian office and team. While I currently live in San Francisco, I was born and raised in Saskatchewan. As a proud Canadian, today feels like a homecoming. Canada has always been an important part of our history and customer base, and I am thrilled to see Cloudflare make a further commitment of expanding officially in the country and opening an office in Toronto. We are hiring a team locally to help our current customers and future customers, and to support even more great Canadian organizations. I wanted to share more about how Cloudflare works with Canadian businesses, what today’s announcement means, and some personal reflections.

How Cloudflare works with Canadian entrepreneurs, businesses, and nonprofits

Cloudflare helps ensure anything connected to the Internet is fast, safe, and reliable. We do this by running a distributed global cloud platform that delivers a broad range of network services to businesses of all sizes—making them more secure, enhancing the performance of anything connected online, and eliminating costs and complexity. We help approximately 25M Internet properties around the world—whether you’re a Canadian entrepreneur trying to spin up your next idea, a healthcare company trying to speed up vaccine distribution, a Global 2000 company, or a non-profit.

Today we work with thousands of customers in Canada including Canadian entrepreneurs, universities, non-profits, large enterprises, as well as small businesses. All of those services need to be fast online, protected from cyber attacks, reliable, and available around the world. Cloudflare helps make that happen—and we’re really good at it. Each day we have blocked an average of 70 billion cyber attacks on behalf of our customers—more than 3.2 billion of those attacks, that we have seen everyday, originate in Canada—and that number has increased by 26% from the end of 2020. We have also seen online usage increasing as well—Internet traffic in Canada is up about 60% compared to one year ago when the world was first shifting to a virtual lifestyle. Canadians are spending a lot more time online in 2021, compared to 2020.

I’m especially proud of how we’ve offered our technology to organizations that may not have the resources to keep up with high traffic and protection from cyberattacks like BullyingCanada, Canada's largest anti-bullying charity serving Canadian youth. Helping keep their website reliable and secure for children and families seeking support, especially when they need it most. It’s also fulfilling to know that we help power COVID-19 vaccine distributors, like Verto Health in Canada and Jane who are helping with the vaccine distribution in British Columbia. We help keep these registration sites accessible, withstand scheduling demands, and efficiently queue and facilitate the distribution of the COVID-19 vaccine.

It turns out whether you are a developer working on a hobby project or a large Canadian organization, every business needs to deliver their service more quickly, more securely, and more reliably. That’s exactly why we started Cloudflare.

Canada is leading digital citizenship

There’s no question that the world has relied on the Internet more than ever before this past year—and that isn’t going away. The Internet has reinvented the way we live and survive. We’ve relied on the Internet to access public information, visit the doctor, get our work done, stay in-touch with friends and loved ones, educate our children, order groceries, and so many other things.

Canada has done a great job at fostering digital citizenship, and is continuing to take this ahead of the curve as one of the most Internet connected countries in the world (#7!). Also, the depth and quality of Canada’s tech talent pool is undeniable, with more than 2.8 million STEM graduates and the world’s highest educated workforce.

There's a strong growing technology ecosystem and entrepreneurship in Canada. This isn’t just a moment, it’s a movement. And it’s gaining steam. Since 2013, Toronto has added more tech jobs than any other place in North America, including Silicon Valley. There are numerous communities helping propel this. As a Charter Member of The C100 it’s great to see how this community of Canadians in tech are supporting, inspiring, and connecting with Canadian entrepreneurs across the globe. There are plenty of other amazing communities and resources including Next 36, Co.Labs, Elevate and events like Collision Conference bringing together the vast technology industry—I’ll be speaking about Canadian entrepreneurship tomorrow alongside Ariel Garten.

Canada is also a strong research hub that’s progressing global standards. We’ve worked with a number of research teams and academic communities, such as with the University of Waterloo, to evolve global cybersecurity, cryptography, and privacy. Now having our team on the ground presents an even stronger opportunity to deepen this work.

Homecoming Introducing Cloudflare Canada

I grew up in Prince Albert, Saskatchewan, and my journey from the North to Silicon Valley included stops in Montreal and Toronto, before I headed to Boston for business school and, eventually California.

Saskatchewan is all about community and hard work, which gave me the foundation to be an entrepreneur and really help build what Cloudflare is today.

Toronto is a special place for me because it was where I fell in love with startups. I worked for an early-stage startup, and that’s where I learned about the power of what a small group of motivated people can accomplish together. It’s also where I got my first experience working in technology. I soon saw how pragmatic and actionable it was working in technology, and I was instantly drawn to how it could make an impact on the world.

I went to Harvard Business School to pursue my MBA. It was there that I met a super smart serial entrepreneur Matthew Prince. We were classmates and we started to work on Cloudflare together. We eventually moved to San Francisco to join our third co-founder, Lee Holloway and to grow Cloudflare.

Fast forward to almost 11 years later, I’m excited for Cloudflare to expand our team to Canada. We are here to help build a better Internet—for Canadian organizations and their online users.

How we can all work together

I’m thrilled that we are doubling down on hiring local talent to further support local customers and partner with more businesses in the region. As new neighbors in the region, don’t hesitate to reach out if we can help:

  • Universities/Research: For research industry professionals and Universities interested in working with us, or if you want to learn more about Cloudflare’s approach to research, check out or connect with the team:
  • Project Galileo: We proudly secure public interest groups with our highest level of cybersecurity, at no cost, through Cloudflare’s Project Galileo. If you are an organization working in the arts, human rights, civil society, journalism, or democracy, you can apply for Project Galileo to get this protection, free of charge.
  • Project Fair Shot: Around the world, governments, hospitals, and pharmacies are struggling to distribute the COVID-19 vaccine. We want to help and have created a feature for vaccine registration sites to keep up with demand and use digital queues to safely scale their efforts, at no cost. If you are an organization facilitating the distribution of the vaccine, reach out and we’ll work with you quickly.
  • Businesses: If your business needs Internet security, performance, and reliability services, our team is ready to help. We have affordable plans for startups, non-profits, and small businesses at $20 and $200/month where you can get started now, or if you are a larger enterprise that wants to talk to our customer team, in Canada, contact us.
  • Careers: Finally, if you are interested in joining an ambitious mission to help build a better Internet, get in-touch with our team and checkout Cloudflare careers: Cloudflare Canada Careers.
Categories: Technology

Why I'm helping Cloudflare grow in Canada

Mon, 19/04/2021 - 14:00
Why I'm helping Cloudflare grow in CanadaWhy I'm helping Cloudflare grow in Canada

I am incredibly excited to join Cloudflare as Head of Sales for Canada to expand the company’s growth in the region as part of its mission to help build a better Internet. This is an important milestone for Cloudflare to better serve the needs of our Canadian customers, recruit local talent, and build on the regional successes we’ve had around the globe.

Internet security, privacy, and performance are key drivers for every business, individual, and public sector organization. Universal dependency on the Internet has significantly increased, with web commerce, remote learning, distributed teams, remote work, virtual meetings etc. — and this is all here to stay.

Over the last few years we have seen the industry move from on-premise infrastructure and applications to a cloud-first approach with cloud and SaaS architectures. As this significant and inevitable transition has accelerated, it has introduced new complexities, challenges, and opportunities for organizations with the evolution to heterogeneous environments across public cloud, on-premise, and hybrid deployments.

At the same time, a company’s digital assets (data, web properties, applications, etc.) have become their most valuable assets. How an organization uses the Internet to serve their customers, partners, and employees has become a strategic priority for organizations around the world. With the rapid shift to remote work caused by the pandemic, businesses of all sizes are looking to adopt a Zero Trust architecture. According to an October 2020 commissioned study conducted by Forrester Consulting, 76% of businesses today say their organization’s security approach is “antiquated” and that they need to shift to a Zero Trust framework. The Cloudflare team in Canada is dedicated to helping Canadian organizations successfully scale and optimize their businesses in this landscape.

At the same time, the threat landscape has evolved faster than ever before. In the first quarter of 2021 alone, Cloudflare blocked an average of 3.2 billion attacks per day originating in Canada—a 26.53% increase from the previous quarter. There is a need to secure websites, corporate and personal data, users, applications, and entire networks in a different way. In a way that provides cloud-based security and network services—to scale with organizations and their business needs.

The volume and sophistication of network attacks can strain the defensive capabilities of even the most advanced enterprises, especially as many organizations have traditional security and network architectures with individual equipment for specific functions in corporate data centers. Leveraging cloud-based security and networking functionalities provides an architectural shift with better security, performance, and reliability. This is how Cloudflare’s global network, one of the largest in the world, protects enterprise networks, applications, and users in a way that reduces complexity, and enables organizational agility.

A little about meWhy I'm helping Cloudflare grow in Canada

My background includes years of working with enterprise and public sector customers to address their business needs and technology challenges with innovative software and services. My career has included leadership positions with Cisco, Splunk, CenturyLink, and Microsoft with responsibilities across sales leadership, solutions engineering, business development, and alliances. I’ve had the pleasure of leading teams to serve enterprise customers in Canada, the Americas, and across the globe, with technologies in security, networking, cloud, and XaaS. Providing solutions to customers, to enable them to address their unique needs, and ultimately transform their organizations, is something I’m excited to help achieve with Cloudflare.

Why Cloudflare

I joined Cloudflare because I believe in its mission to help build a better Internet and have seen how the company delivers truly disruptive and innovative solutions for customers—of all sizes in various industries. The team at Cloudflare has this mission rooted in everything they do, to benefit customers, developers, and Internet users all across the globe. Some of the reasons I’m most excited to embark on this journey with Cloudflare:

Industry solutions with product excellence:

Cloudflare is at the forefront of industry trends, enabling customers to use cloud-based security and networking solutions, and providing solutions for data privacy and data residency. These are solutions that are easy to use and easy to deploy across the world. This edge computing network architecture provides security and performance to all users at incredible scale.

Technology leadership and pace of innovation:

At Cloudflare, the pace of innovation is amazing. Leveraging its unique global network, the company is continuously innovating to release new products and features in the cloud that are immediately available at scale—to its customers and all users worldwide. Some of the latest products disrupting traditional IT approaches include:

  • Magic Transit uses Cloudflare's global infrastructure to protect organizations’ entire networks, not just their web-facing applications, to protect office networks and keep companies running
  • Cloudflare One, a platform to connect and secure companies and teams anywhere (remote and across offices) and on any device regardless of location
  • Cloudflare Workers, the serverless solution redefining how applications are deployed at the network edge—instantly across the globe with exceptional performance and reliability
  • Argo Smart Routing, which acts as Waze for the Internet, can significantly cut the amount of time users online spend waiting for content
  • Remote Browser Isolation provides security at the “last mile” of Internet connectivity: web browsers, which are one of your biggest attack surfaces
  • Regional Services give customers the ability to control data flows and manage where their traffic is handled, maintaining regional control over their data to comply with data residency requirements or their customer preferences
  • Culture and Values:
  • Cloudflare’s intent to “do the right thing” is reflected in several initiatives that give back to society, such as through Project Fair Shot and the Athenian Project. Project Fair Shot is a no-charge program offering a digital Waiting Room service to any organization involved with the COVID-19 vaccine scheduling and distribution. We’ve had Project Fair Shot initiatives underway around the world, including several in Canada and we’re here to help any government, pharmacy, or organisation that’s distributing the vaccine do so efficiently and withstand demands. And Cloudflare’s Athenian Project provides its enterprise-class service for free to any state or local government elections' officials, in order to protect services like voter registration, poll location, precinct reporting, and official results.
  • Our co-founder, President & COO, Michelle Zatlyn, is originally from Canada and is committed to Cloudflare’s further investment in the region, working with Canadian businesses and organizations, and to the growing technology landscape in Canada. Here she shares more about this and Cloudflare Canada.
  • Overall, Cloudflare’s entire culture has a clear focus that revolves around its customers, product excellence, innovation, and collaboration. We have an extremely talented and passionate team, and I am thrilled to be part of it to help Cloudflare grow even more in Canada.
Cloudflare in Canada

There are several thousand Cloudflare customers in Canada already, including large enterprises, small companies, individuals, and public sector organizations. We are eager to bring even more Internet security, reliability, and more speed to users in the region. We recently established a local team in Toronto and our team will continue to grow in Canada. The initial functions prioritized for our office are Engineering, Research and Development, Customer Success, Sales, and Partnerships, and more, to work with our current and future customers of all sizes and industries across Canada, and to support the overall Canadian Internet landscape.

Our opportunity in Canada

I am especially excited about the opportunity for Cloudflare to grow in the region to help Canadian customers with essential and unique security and networking. If you are a business in Canada that would like to chat about your cybersecurity or Internet performance, or explore how Cloudflare can assist, please get in touch with us or send me a message directly at If you are interested in exploring careers at Cloudflare, our team in Canada is growing(!), and you can check out our careers page to learn more.

I have been impressed by what Cloudflare has built globally in a short period of time, and I am even more excited by what lies ahead for our Canadian customers, partners, and team.

Categories: Technology

DDoS attack trends for 2021 Q1

Mon, 19/04/2021 - 12:00
DDoS attack trends for 2021 Q1DDoS attack trends for 2021 Q1

Last week was Developer Week at Cloudflare. During that week, our teams released a bunch of cool new products, including a bunch of improvements to Workers. And it's not just our customers that love deploying apps with Workers, but also our engineering teams. Workers is also what powers our Internet traffic and attack trends on Cloudflare Radar. Today, along with this deep-dive analysis blog, we’re excited to announce the new Radar DDoS Report page, our first fully automated data notebook built on top of Jupyter, Clickhouse, and Workers.

Last month, we introduced our autonomous edge DDoS (Distributed Denial of Service) protection system and explained how it is able to drop attacks at wire speed without impacting performance. It runs in our networks’ edge, analyzes traffic asynchronously to avoid impacting performance, and pushes mitigation rules in-line immediately once attacks are detected. All of this is done autonomously, i.e., without requiring centralized consensus.

Today, we’d like to share the latest DDoS insights and trends that are based on attacks that our system mitigated during the first quarter of 2021. When we analyze attacks, we calculate the “DDoS activity” rate, which is the percent of attack traffic out of the total traffic (attack + clean). This allows us to normalize the data points and avoid biases towards, for example, a data center that sees more traffic and therefore also more attacks.

HighlightsApplication-layer DDoS attacks
  • In 2021 Q1, the country with the highest percentage of HTTP attack traffic was China. This was followed by the United States, Malaysia, and India.
  • The telecommunication industry was the most attacked in Q1, followed by Consumer Services, Security and Investigations, Internet, and Cryptocurrency.
  • The most attacked Internet properties were of companies based in China, the US, and Morocco.
Network-layer DDoS attacks
  • On the Cloudflare network, the highest DDoS activity was observed in our data centers in Rwanda, China, and Brunei.
  • Almost 44% of all of the attacks in Q1 occurred in January.
  • Top emerging threats include attacks targeting Jenkins and TeamSpeak3 servers, which increased by 940% and 203% QoQ, respectively.
  • Additional emerging threats include floods of QUIC version negotiation packets that may have been an attempt to disrupt Cloudflare’s infrastructure.
Application-layer DDoS attacks

Application-layer DDoS attacks, or HTTP DDoS attacks, are attacks that aim to disrupt an HTTP server by making it unable to process requests. If a server is bombarded with more requests than it can process, the server will drop legitimate requests or even crash.

DDoS attack trends for 2021 Q1DDoS attack activity per industry

When we break down DDoS activity by our customer’s market industry, we can see that Telecommunication was the most targeted industry in Q1. This is a significant jump from sixth place in 2020 Q4. Following in second place is the Consumer Services industry, and in third place the Security and Investigations industry.

DDoS attack trends for 2021 Q1DDoS activity by source country

As opposed to network-layer attacks, the source IP cannot be spoofed in an HTTP attack. A connection must be established. By looking up the location of the source IP of the client, we can identify the source country. A high DDoS activity rate in a given country indicates large botnets operating from within. Both in 2020 Q4 and 2021 Q1, China came in first place, with the US not far behind.

DDoS attack trends for 2021 Q1DDoS activity by target country

In order to identify which countries are being attacked the most, we break down the DDoS activity by our customer’s billing country. Similar to the attack source breakdown, China and the US come in first and second places, respectively. Interestingly enough, in the previous quarter, India dethroned China from the first place, perhaps due to the elections in India that also occurred throughout 2020 Q4.

DDoS attack trends for 2021 Q1Ransom attacks

As we’ve seen, our customers on the non-Enterprise plans were the most targeted by DDoS attacks. However, it’s not just the quantity of attacks that is high, but these customers also reported the highest number of Ransom DDoS attacks (RDDoS). In 2021 Q1, 13% of surveyed Cloudflare customers that were hit by a DDoS attack reported they were either extorted by an RDDoS attack or received a threat in advance. Of those, 62% are on the Pro plan and 33% on the Business plan. This is a continued trend from 2020 Q4 where the number of extorted customers was 17%, including a Fortune Global 500 company that was targeted by a group claiming to be the Lazarus Group, a company which we onboarded and protected.

DDoS attack trends for 2021 Q1Network-layer DDoS attacks

While application layer attacks strike the application (Layer 7 of the OSI model) running the service end users are trying to access, network layer attacks target exposed network infrastructure (such as in-line routers and other network servers) and the Internet link itself.

Number of attacks

On a monthly basis, January was Q1’s busiest month for attackers, constituting 42% of the total attacks observed in the quarter. Followed by March with 34.2% and February with 23.8%.

DDoS attack trends for 2021 Q1

In February, we did however see the largest attacks of Q1 peaking at 300-400 Gbps.

DDoS attack trends for 2021 Q1Size of attacks

There are different ways of measuring a L3/4 DDoS attack’s size. One is the volume of traffic it delivers, measured as the bit rate (specifically, gigabits-per-second). Another is the number of packets it delivers, measured as the packet rate (specifically, packets-per-second). Attacks with high bit rates attempt to saturate the Internet link, while attacks with high packet rates attempt to overwhelm the routers or other in-line hardware devices.

In 2021 Q1, a vast majority (over 97%) of the L3/4 attacks observed were smaller than 1 mpps and 500 Mbps.

This is a continuation of the trend we observed all of last year. However, this does not imply that these attacks are harmless.

Attacks under 500 Mbps are often sufficient to create major disruptions for Internet properties that are not protected by a cloud-based DDoS protection service. Many organizations have uplinks provided by their service providers with less bandwidth capacity than 1 Gbps. Assuming their public facing network interface also serves legitimate traffic, you can see how even DDoS attacks under 500 Mbps can easily take down Internet properties.

DDoS attack trends for 2021 Q1DDoS attack trends for 2021 Q1Duration of attacks

Over 90% of attacks lasted under one hour in duration. Short burst attacks may attempt to cause damage without being detected by DDoS detection systems. DDoS services that rely on manual analysis and mitigation may prove to be useless against these types of attacks because they are over before the analyst can even identify the attack traffic.

DDoS attack trends for 2021 Q1

Short attacks are often also used to probe the cyber defenses of the target. Load-testing tools and automated DDoS tools, widely available on the dark web, can generate short bursts of, say, a SYN flood, and then following up with another short attack using an alternate attack vector. This allows attackers to understand the security posture of their targets before they decide to potentially launch larger attacks at larger rates and longer durations.

Attack vectors

An attack vector is the attack method that the attacker utilizes. In 2021 Q1, SYN attacks continued to remain the most popular attack vector used by attackers, followed by RST, UDP, and DNS amplification attacks.

What is a SYN flood attack? It’s a DDoS attack that exploits the very foundation of a TCP connection. A stateful TCP connection between a client and a server begins with a 3-way TCP handshake. The client sends an initial connection request packet with a synchronize flag (SYN). The server responds with a packet that contains a synchronized acknowledgment flag (SYN-ACK). Finally, the client responds with an acknowledgment (ACK) packet. At this point, a connection is established and data can be exchanged until the connection is closed. This stateful process can be abused by attackers to cause denial of service events.

By repeatedly sending SYN packets, the attacker attempts to overwhelm a server or the router’s connection table that tracks the state of TCP connections. The router replies with a SYN-ACK packet, allocates a certain amount of memory for each given connection, and falsely waits for the client to respond with the final ACK. Given a sufficient number of connections occupying the router’s memory, the router is unable to allocate more memory for legitimate clients, causing the router to crash or preventing it from handling legitimate client connections, i.e., a denial of service event.

Similarly, a RST amplification flood attack exhausts the target servers by depleting their system resources used to look up incoming packets for a current session.

DDoS attack trends for 2021 Q1Emerging threatsDDoS attack trends for 2021 Q1

While SYN attacks remain popular, this quarter we’ve seen an enormous uptick of 940% in attacks targeting Jenkins servers. Jenkins is a free open-source automation server. It helps engineering teams facilitate software development. A vulnerability in an older version of the server (Jenkins 2.218 and earlier) aided the launch of DDoS attacks. This vulnerability was fixed in Jenkins 2.219 by disabling UDP multicast/broadcast messages by default. However, there are still many vulnerable and exposed devices running UDP-based services which are being harnessed to generate volumetric amplification attacks.

Cloudflare also observed a 433% increase in L3/4 DDoS attacks over the QUIC protocol, a new encrypted-by-default Internet transport protocol that runs over UDP. Version negotiations packets sent by a server to client allow the server to indicate the version of QUIC it supports to the client. Since UDP is stateless, it is easy for attackers to mimic Version Negotiation packets by spoofing the source IP address and overwhelm a client.

The attacks targeting Cloudflare may have meant to impact Cloudflare’s infrastructure — perhaps by downgrading the versions being used — rather than targeting specific customers. You can learn more about QUIC amplification attacks here.

The third emerging threat vector observed was TeamSpeak, a proprietary voice-over-Internet Protocol (VoIP) that runs over UDP to help gamers talk with other gamers in real time. This emerging threat increased by 203% QoQ.Talking instead of just chatting can significantly improve a gaming team’s efficiency and help them win. DDoS attacks that target TeamSpeak servers may be launched by rival groups in an attempt to disrupt their communication path during real-time multiplayer games and thus impact their team’s performance.

DDoS activity by Cloudflare data center country

Looking at country-based distribution for network layer DDoS attacks, Rwanda, China, and Brunei observed the most number of L3/4 DDoS attacks. Unlike application layer DDoS attacks, attackers can (and typically do) spoof the source IP address to obfuscate the source location of the DDoS attack. For this reason, when analyzing L3/4 DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations where the traffic was ingested, and not by the location of the source IP. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.

DDoS attack trends for 2021 Q1

To view all regions and countries, check out the Radar DDoS Report dashboard’s interactive map.

Helping build a better Internet

Cloudflare was founded with the mission to help build a better Internet — one where the impact of DDoS attacks is a thing of the past. Over the last 10 years, we have been unwavering in our efforts to protect our customers’ Internet properties from DDoS attacks of any size or kind. CCP Games and Panasonic are two of many customers that benefit from Cloudflare’s DDoS Protection.

Cloudflare was also recently named a leader in The Forrester WaveTM: DDoS Mitigation Solutions, Q1 2021. You can download a complimentary copy of the report here.

According to the report, written by Forrester Senior Analyst for Security and Risk, David Holmes, “Cloudflare protects against DDoS from the edge, and fast… customer references view Cloudflare’s edge network as a compelling way to protect and deliver applications.”

There are three key reasons Cloudflare DDoS Protection is recognized by customer and industry analysts alike:

  1. Cloudflare’s network architecture: Cloudflare doesn't operate scrubbing centers, as we believe that the scrubbing center model is a flawed approach to DDoS protection. Scrubbing centers cause delays and cost too much to build and run. Instead, we run DDoS protection from every server in every data center in our network. Our Anycast-based architecture makes our capacity equivalent to our DDoS scrubbing capacity, the largest in the market at 59 Tbps. This means Cloudflare detects and mitigates DDoS attacks close to the source of attack. Better yet, Cloudflare’s global threat intelligence acts like an immune system for the Internet — employing our machine learning models to learn from and mitigate attacks against any customer to protect them all.
  2. Fast performance: Our customers constantly tell us that they want robust security but not at the expense of performance. From its inception, Cloudflare was architected so that customers do not incur a latency penalty as a result of attacks. Our Anycast architecture allows us to mitigate attacks closest to the source and analyze traffic out-of-path, ensuring that our DDoS mitigation solution doesn’t add any latency to legitimate traffic. The rule is applied at the most optimal place in the Linux stack for a cost efficient mitigation, ensuring that there's no performance penalty. Performance tests over Cloudflare’s network show that the latency decreased by 3 ms and packet loss was nearly zero when traffic was routed over Cloudflare Magic Transit.
  3. Cloudflare’s support: Every Cloudflare enterprise customer account is assigned a team (including an Account Executive, Solution Engineer, and Customer Success Manager) that actively supports customers through onboarding and beyond to help identify areas for optimization in customer configurations.

Cloudflare’s 24x7x365 global “follow the sun” support team is always ready to pick up the phone and provide instant human response when our enterprise customers request urgent support.

To quote Grant Ingersoll, CTO of the Wikimedia Foundation, “Cloudflare has reliable infrastructure and an extremely competent and responsive team. They are well-positioned to deflect even the largest of attacks.”

To learn more about Cloudflare’s DDoS solution contact us or get started.

Categories: Technology

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021

Mon, 19/04/2021 - 00:00
Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021

Letzte Woche fand die Cloudflare Developer Week statt – ein willkommener Anlass für unsere Teams, eine Reihe von spannenden neuen Produkten und nicht zuletzt auch einige Verbesserungen für Workers vorzustellen. Die Qualitäten dieser Lösung für den Einsatz von Applikationen wissen übrigens nicht nur unsere Kunden zu schätzen: Das Tool erfreut sich auch bei unseren eigenen Entwicklern großer Beliebtheit. Unter anderem basiert auch unsere Untersuchung von Internet- und Bedrohungstrends mithilfe von Cloudflare Radar auf Workers. Wir freuen uns, dass wir Ihnen heute (zusätzlich zu diesem Blogbeitrag mit detaillierten Analysen zu diesem Thema) unseren neuen Radar DDoS Report präsentieren können, unser erstes komplett automatisiertes Daten-Notebook auf der Grundlage von Jupyter, Clickhouse und Workers.

Letzten Monat stellten wir unser autonomes, am Netzwerkrand (Edge) betriebenes Schutzsystem gegen DDoS-Angriffe (Distributed Denial of Service) vor und erläuterten, wie es mit dieser Lösung gelingen kann, Attacken verzögerungsfrei und ohne Performance-Einbußen abzuwehren. Dieses System vermeidet Leistungsabfälle durch eine asynchrone Analyse des Datenverkehrs und leitet bei Angriffen sofort und direkt im Datenstrom Gegenmaßnahmen ein. All dies geschieht autonom am Netzwerkrand, eine separate Prüfung über eine zentrale Stelle ist nicht nötig.

Heute möchten wir Sie nun auf der Grundlage der Angriffe, die unsere Systeme im ersten Quartal 2021 abwehren konnten, über die neuesten DDoS-Bedrohungstrends informieren. Wenn wir Attacken analysieren, berechnen wir unter anderem die „DDoS-Aktivitätsrate“, also den prozentualen Anteil des Angriffs-Traffics am gesamten Datenverkehr. Auf diese Weise erhalten wir normalisierte Datenpunkte und können statistische Verzerrungen vermeiden, die sonst zum Beispiel dadurch entstehen könnten, dass ein Rechenzentrum mehr Datenverkehr und damit eben auch mehr Angriffe verzeichnet.

Die wichtigsten Erkenntnisse auf einen BlickDDoS-Angriffe in der Anwendungsschicht
  • Im letzten Quartal entfiel der größte Anteil am gesamten HTTP-Flooding-Angriffsverkehr auf China, gefolgt von den Vereinigten Staaten, Malaysia und Indien.
  • Am häufigsten geriet die Telekommunikationsbranche ins Visier der Angreifer, aber auch Anbieter von Verbraucherdienstleistungen, Sicherheits- und Ermittlungsdienste, Internetdienste und Kryptowährungen wurden stark attackiert.
  • Die beliebtesten Angriffsziele waren Websites von Unternehmen in China, den USA und Marokko.
DDoS-Angriffe in der Netzwerkschicht
  • Unsere Rechenzentren in Ruanda, China und Brunei waren die Cloudflare-Standorte mit der höchsten DDoS-Aktivitätsrate.
  • Fast 44 % aller Angriffe im ersten Quartal wurden im Januar durchgeführt.
  • Bei den Angriffen auf Jenkins- und TeamSpeak3-Server verzeichneten wir im Quartalsvergleich besonders deutliche Zuwächse von 940 % bzw. 203 %.
  • An neuen Bedrohungen registrierten wir unter anderem hohe Mengen an Datenpaketen zur Verhandlung über eine gemeinsame QUIC-Version (Version Negotiation); dabei könnte es sich um einen Versuch gehandelt haben, die Infrastruktur von Cloudflare aus dem Tritt zu bringen.
DDoS-Angriffe in der Anwendungsschicht

DDoS-Angriffe in der Anwendungsschicht – auch HTTP-DDoS-Angriffe genannt – sind Versuche, Störungen an einem HTTP-Server hervorzurufen, sodass er keine Anfragen mehr verarbeiten kann: Wenn ein Server mit einer Flut von Requests überlastet wird, übergeht er legitime Anfragen oder stürzt sogar ab.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Aufschlüsselung der DDoS-Angriffe nach Branchen

Wenn man sich ansieht, welchen wirtschaftlichen Bereichen unsere angegriffenen Kunden angehören, dann zeigt sich, dass im ersten Quartal vor allem die Telekommunikationsbranche ins Visier der DDoS-Angreifer geriet – eine deutliche Veränderung gegen über dem Vorquartal, als dieser Sektor nur an sechster Stelle lag. Am zweit- und dritthäufigsten wurden diesmal Anbieter von Verbraucherdienstleistungen und Sicherheits- und Ermittlungsdienste attackiert.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Aufschlüsselung der DDoS-Angriffe nach Herkunftsland

Im Gegensatz zu Angriffen in der Netzwerkschicht kann bei einem HTTP-Flooding-Angriff die Quell-IP nicht verschleiert werden, weil eine bidirektionale Kommunikation erforderlich ist. Das bedeutet auch, dass der Standort der Quell-IP des Clients und damit auch das Ursprungsland identifiziert werden können. Wenn viele DDoS-Attacken einem bestimmten Land zugeordnet werden können, deutet das darauf hin, dass dort große Botnetze betrieben werden. Sowohl im vierten Quartal 2020 als auch im ersten Quartal 2021 lag China dabei auf dem ersten Platz, dicht gefolgt von den USA.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Aufschlüsselung der DDoS-Angriffe nach Zielland

Um herauszufinden, welche Länder am häufigsten angegriffen werden, schlüsselten wir die DDoS-Angriffe nach dem Abrechnungsland unserer Kunden auf. Genauso wie bei den Herkunftsländern liegen China und die USA diesmal an erster und zweiter Stelle. Im Vorquartal allerdings hatte Indien interessanterweise China vorübergehend vom ersten Platz verdrängt, was vielleicht etwas mit den Wahlen in Indien zu tun hatte, die ebenfalls im vierten Jahresviertel 2020 stattfanden.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Ransom-Angriffe

Wir konnten feststellen, dass Cloudflare-Kunden ohne Enterprise Plan besonders stark von DDoS-Angriffen betroffen waren. Dies gilt allerdings nicht nur für die Gesamtzahl der Störungsversuche, denn aus diesem Kundenkreis kamen auch die meisten Berichte über Erpressungsversuche durch Ransom-DDoS-Angriffe (RDDoS). 13 % der befragten Cloudflare-Kunden mit DDoS-Attacken im ersten Quartal 2021 ließen uns wissen, dass sie entweder durch einen RDDoS-Angriff erpresst wurden oder im Vorfeld eine entsprechende Drohung erhalten hatten. 62 % dieser Kunden hatten sich für unseren Pro Plan entschieden, 33 % nutzten unseren Business Plan. Damit bestätigte sich ein Trend aus dem Vorquartal, als der Anteil der erpressten Kunden sogar 17 % betrug, einschließlich eines Fortune Global 500-Unternehmens, das von einer Gruppe angegriffen wurde, die behauptete, die Lazarus Group zu sein. Das Unternehmen gehört zu unseren Kunden und wird durch unsere Systeme geschützt.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021DDoS-Angriffe in der Netzwerkschicht

Angriffe in der Anwendungsschicht (Layer 7 des OSI-Modells) nehmen die Applikationen ins Visier, auf die die Benutzer des angebotenen Dienstes zugreifen möchten; Attacken in der Netzwerkschicht hingegen sollen die ungeschützte Netzwerkinfrastruktur (z. B. die Inline-Router und andere Netzwerkserver) und die eigentliche Internetverbindung stören.

Anzahl der Angriffe

Im ersten Jahresviertel war der Januar der Monat mit den meisten Angriffen (42 %), gefolgt von März (34,2 %) und Februar (23,8 %).

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021

Die Attacken mit den höchsten Datenübertragungsraten (300–400 GBit/s) verzeichneten wir allerdings im Februar.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Dimension der Angriffe

Es gibt verschiedene Möglichkeiten, einen L3/4-DDoS-Angriff zu bemessen. Man kann sich das reine Volumen des Angriffs-Traffics ansehen, das als Bitrate (in Gigabits pro Sekunde, GBit/s) ausgedrückt wird; oder man stellt auf die Paketrate (die Anzahl der pro Sekunde gelieferten Datenpakete, Mpps) ab. Angriffe mit hohen Bitraten sollen die Internetverbindung auslasten, während man mit hohen Paketraten die Router oder andere Inline-Hardwaregeräte an ihre Grenzen bringen möchte.

Im ersten Quartal 2021 belief sich das Volumen von über 97 % der beobachteten L3/4-Attacken auf weniger als 1 Mpps bzw. 500 MBit/s.

Damit setzt sich ein Trend fort, den wir bereits im gesamten letzten Jahr beobachten konnten. Allerdings heißt das nicht, dass diese Angriffe harmlos waren:

Nicht selten lassen sich Internetpräsenzen bereits mit weniger als 500 Mbit/s massiv stören, wenn sie nicht durch eine cloudbasierte Lösung vor DDoS-Angriffen geschützt werden. Viele Unternehmen erhalten von ihren Providern Uplinks mit Kapazitäten von weniger als 1 Gbit/s. Wenn man davon ausgeht, dass auch der legitime Traffic über deren öffentliche Netzwerkschnittstelle laufen muss, wird schnell klar, dass selbst DDoS-Angriffe mit weniger als 500 MBit/s Websites und Webapplikationen ohne Weiteres lahmlegen können.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Angriffsdauer

Über 90 % der Angriffe dauerten weniger als eine Stunde. Derartige Blitzattacken sind ein Versuch, einen Schaden zu verursachen, ohne von DDoS-Abwehrsystemen erkannt zu werden. Lösungen zum Schutz vor DDoS-Attacken, die auf manuellen Analysen und Gegenmaßnahmen basieren, erweisen sich bei dieser Art von Angriffen gegebenenfalls als nutzlos, weil die Störung schon vorbei ist, bevor der Analyst den Angriffsverkehr überhaupt identifizieren konnte.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021

Oft soll mit Blitzattacken auch zuerst einmal getestet werden, wie gut das Angriffsziel geschützt ist. Tools zur Durchführung von Belastungstests und automatisierten DDoS-Attacken sind im Darknet ohne Weiteres erhältlich. Mit ihnen lassen sich zum Beispiel kurze SYN-Floods durchführen, gefolgt von einem weiteren schnellen Angriff mit einem anderen Angriffsvektor. Auf diese Weise können Hacker herausfinden, wie ihre Opfer sicherheitstechnisch aufgestellt sind, um dann gegebenenfalls größere und länger andauernde Attacken mit höheren Datenmengen in die Wege zu leiten.


Mit dem Begriff „Angriffsvektor“ bezeichnet man die Methode, die bei einer Attacke zur Anwendung kommt. Im ersten Quartal 2021 verzeichneten wir vor allem SYN-Angriffe, aber auch RST-, UDP- und DNS-Verstärkungsangriffe waren sehr beliebt.

Bei einem SYN Flood-Angriff handelt es sich um eine DDoS-Attacke, die bei den Grundvoraussetzungen einer TCP-Verbindung ansetzt. Der Aufbau einer zustandsabhängigen TCP-Verbindung zwischen einem Client und einem Server beginnt mit einem TCP-Handshake in drei Schritten: Der Client sendet zuerst ein Verbindungsanforderungspaket mit einem Synchronisierungsflag (SYN-Flag). Der Server antwortet mit einem Paket, das die Anforderung bestätigt (ACK) und seinerseits ein Synchronisierungsflag (SYN) enthält. Abschließend antwortet der Client wiederum mit einem Bestätigungspaket (ACK). Damit ist eine Verbindung aufgebaut und der Datenaustausch kann stattfinden, bis die Verbindung wieder getrennt wird. Doch genau dieser zustandsabhängige Prozess kann von Hackern missbraucht werden, um Denial-of-Service-Attacken durchzuführen.

Der Angreifer sendet dabei immer wieder neue SYN-Pakete und versucht auf diese Weise, einen Server in die Knie zu zwingen oder die Routingtabelle, die den Status von TCP-Verbindungen enthält, zu überfüllen. Der Router antwortet mit einem SYN-ACK-Paket, weist jeder einzelnen Verbindung eine bestimmte Menge Speicherplatz zu und wartet vergeblich darauf, dass der Client mit dem abschließenden ACK-Paket antwortet. Wenn eine ausreichend hohe Anzahl an Verbindungen den Speicher des Routers komplett in Anspruch nimmt, ist er nicht mehr in der Lage, legitimen Clients weiteren Speicherplatz zuzuweisen: Der Router stürzt ab oder ist zumindest unfähig, Verbindungen zu den legitimen Clients aufzubauen – es kommt zu einer Dienstverweigerung (Denial of Service).

Ein RST-Verstärkungs-Flood-Angriff funktioniert ganz ähnlich: Er führt zu einer Überlastung der Systemressourcen der Zielserver, die für die Suche nach den eingehenden Paketen einer aktiven Sitzung vorgesehen sind.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021Neue BedrohungenEntwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021

Neben den bei Hackern nach wie vor sehr beliebten SYN-Angriffen mussten wir im letzten Quartal auch einen enormen Anstieg um 940 % bei den Attacken auf Jenkins-Server feststellen. Bei Jenkins handelt es sich um einen kostenlosen Open-Source-Automatisierungsserver, der Programmierern die Softwareentwicklung erleichtert. Eine Sicherheitslücke in älteren Versionen des Servers (bis Jenkins 2.218) erlaubte die einfache Durchführung von DDoS-Angriffen. Diese Schwachstelle wurde zwar mit der Version 2.219 durch eine standardmäßige Deaktivierung der UDP-Multicast/Broadcast-Nachrichten behoben, allerdings gibt es nach wie vor viele angreifbare und ungeschützte Geräte, auf denen UDP-basierte Dienste laufen, sodass sich mit ihnen volumetrische Verstärkungsangriffe generieren lassen.

Bei den L3/4-DDoS-Angriffen verzeichnete Cloudflare außerdem einen Anstieg der Attacken über das QUIC-Protokoll um 433 %. Bei QUIC handelt es sich um ein neues und standardmäßig verschlüsseltes Internet-Transportprotokoll, das auf UDP basiert. Dabei sendet der Server Versionsverhandlungspakete (Version Negotiations Packets) an den Client, um die unterstützte QUIC-Version anzugeben. Weil es sich bei UDP um ein verbindungsloses Protokoll handelt, können Angreifer diese Pakete durch Spoofing der Ursprungs-IP-Adresse relativ leicht imitieren und so den Client überlasten.

Bei diesen Angriffen auf Cloudflare hatte man es möglicherweise nicht auf bestimmte Kunden abgesehen, sondern auf unsere Infrastruktur, vielleicht durch ein Downgrade der verwendeten Versionen. Klicken Sie bitte hier, wenn Sie mehr über QUIC-Verstärkungsangriffen erfahren möchten.

Der dritte neue Angriffsvektor, den wir beobachten konnten, betraf TeamSpeak, ein proprietäres Voice-over-Internet Protocol (VoIP), das ebenfalls auf UDP basiert und Gamern die Möglichkeit bietet, in Echtzeit miteinander zu sprechen. Bei dieser neuen Bedrohung verzeichneten wir einen Zuwachs um 203 % im Quartalsvergleich. Die Mitglieder eines Gaming-Teams können deutlich effizienter agieren und ihre Gewinnchancen erhöhen, wenn sie miteinander reden, anstatt sich schriftlich per Chat auszutauschen. Dementsprechend gehen DDoS-Angriffe auf TeamSpeak-Server möglicherweise auf Sabotageversuche von Gamern zurück, die bei Echtzeit-Multiplayer-Spielen die Kommunikation gegnerischer Teams stören wollen.

Aufschlüsselung der DDoS-Angriffe nach Standort der Cloudflare Rechenzentren

Was die Verteilung der DDoS-Angriffe in der Netzwerkschicht auf verschiedene Länder anbelangt, wurden die meisten L3/4-DDoS-Attacken in Ruanda, China und Brunei beobachtet. Anders als bei den DDoS-Angriffen in der Anwendungsschicht haben Hacker hier die Möglichkeit, die Quell-IP-Adresse und damit den Ursprungsort des DDoS-Angriffs zu verschleiern (und in der Regel tun sie dies auch). Deshalb haben wir bei der Analyse der L3/4-DDoS-Angriffe den Standort der Quell-IP außer Acht gelassen und die Störversuche stattdessen den Cloudflare-Rechenzentren am Netzwerkrand zugeordnet, die den jeweiligen Datenverkehr aufgenommen haben. Cloudflare löst das Problem der verschleierten IPs also dadurch, dass wir Angriffsdaten zusammen mit dem Standort des Cloudflare-Rechenzentrums anzeigen, in dem die Attacke registriert wurde. Da wir Rechenzentren weltweit in über 200 Städten betreiben, sind die geografischen Angaben in unserem Bericht sehr genau.

Entwicklung der DDoS-Bedrohungslandschaft im ersten Quartal 2021

Auf dem Dashboard des Radar DDoS Report sehen Sie eine interaktive Karte, mit der Sie sich einen Überblick über alle Regionen und Länder verschaffen können.

Unser Ziel: ein besseres Internet

Von Anfang an stand bei Cloudflare das Ziel im Vordergrund, ein besseres Internet zu schaffen – ein Internet, mit dem man DDoS-Angriffe und ihre Auswirkungen nicht mehr fürchten muss. Dementsprechend haben wir uns in den vergangenen zehn Jahren unermüdlich darum bemüht, die Websites und Anwendungen unserer Kunden gegen DDoS-Attacken jeglicher Größe und Art abzuschirmen. Heute profitieren unter anderem CCP Games, Panasonic und eine Vielzahl weiterer Unternehmen von unserem Schutz vor DDoS-Attacken.

In „The Forrester WaveTM: DDoS Mitigation Solutions, Q1 2021“ wurde Cloudflare kürzlich als führender Anbieter eingestuft. Sie können den Bericht hier kostenlos herunterladen.

David Holmes, Verfasser des Berichts und Senior Analyst for Security and Risk bei Forrester, äußert sich darin wie folgt: „Cloudflare bietet reaktionsschnellen Schutz vor DDoS-Angriffen am Netzwerkrand ... Kunden des Unternehmens bezeichnen das Edge-Netzwerk von Cloudflare als überzeugende Lösung für die sichere Bereitstellung von Anwendungen“ („Cloudflare protects against DDoS from the edge, and fast … customer references view Cloudflare’s edge network as a compelling way to protect and deliver applications“).

Der Schutz vor DDoS-Angriffen von Cloudflare überzeugt Kunden und Branchenanalysten vor allem aus drei Gründen:

  1. Netzwerkarchitektur: Cloudflare betreibt keine Scrubbing Center, weil wir glauben, dass das hinter diesen Zentren stehende Modell ein verfehlter Ansatz für den Schutz vor DDoS-Attacken ist. Scrubbing Center verursachen nicht nur Verzögerungen, sondern auch hohe Einrichtungs- und Betriebskosten. Unsere Lösung wird vielmehr von jedem Server in jedem Rechenzentrum unseres Netzwerks ausgeführt. Dank unserer Anycast-Architektur entspricht unsere Kapazität unserer DDoS-Scrubbing-Kapazität, die mit 59 Tbit/s auf dem Markt ihresgleichen sucht. Das bedeutet, dass Cloudflare DDoS-Angriffe bereits nahe an deren Ursprungsorten erkennt und entschärft. Doch damit nicht genug: Die Bedrohungsinformationen, die Cloudflare aus aller Welt zusammenträgt, wirken wie ein Immunsystem für das Internet. Wir setzen maschinelle Lernmodelle ein, um jeden Angriff auf unsere Kunden abzuwehren und gleichzeitig in neue Erkenntnisse zu verwandeln, die dem Schutz Aller dienen.
  2. Hohe Performance: Von unseren Kunden hören wir immer wieder, dass sie zwar gut geschützt sein wollen, die Performance darunter aber nicht leiden darf. Deshalb wurde Cloudflare von Anfang an so konzipiert, dass unseren Kunden selbst bei Angriffen hohe Latenzzeiten erspart bleiben. Unsere Anycast-Architektur ermöglicht es uns, Angriffe in nächster Nähe ihres Ursprungsorts zu entschärfen und die Analyse des Datenverkehrs in einen anderen Teil des Netzwerks auszulagern. Dadurch wird sichergestellt, dass unsere Lösung zur Abwehr von DDoS-Attacken die Latenz des legitimen Datenverkehrs nicht erhöht. Im Interesse einer kosteneffizienten Entschärfung wird die Regel an der idealen Stelle im Linux-Stack angewandt, sodass Leistungseinbußen vermieden werden. Bei Performance-Tests unseres Netzwerks zeigte sich, dass sich die Latenz um 3 ms verringerte und der Paketverlust gegen Null ging, wenn der Datenverkehr über Cloudflare Magic Transit geleitet wurde.
  3. Umfassender Support: Bei Cloudflare ist jedem Enterprise-Account ein Team zur aktiven Unterstützung unserer Kunden zugeordnet, dem unter anderem ein Account Executive, ein Solution Engineer und ein Customer Success Manager angehören. Dabei beschränkt sich der Support nicht auf das Onboarding, sondern umfasst auch Hilfestellungen bei der Suche nach Möglichkeiten zur Optimierung der kundenseitigen Konfiguration.

Unser Support-Team ist weltweit, täglich und rund um die Uhr per Telefon erreichbar. Es wartet nur darauf, unseren Enterprise-Kunden einen persönlichen Ansprechpartner zu bieten, wenn unsere Unterstützung dringend gebraucht wird.

Das überzeugt unter anderem auch Grant Ingersoll, CTO der Wikimedia Foundation: „Cloudflare verfügt über eine zuverlässige Infrastruktur und ein äußerst kompetentes und reaktionsschnelles Team. Das Unternehmen ist hervorragend aufgestellt, um selbst die größten Angriffe abzuwehren.“

Kontaktieren Sie uns, wenn Sie weitere Informationen zu Cloudflares DDoS-Lösung wünschen. Sie können aber auch gleich loslegen.

Categories: Technology

Containers at the edge: it’s not what you think, or maybe it is

Sat, 17/04/2021 - 14:00
 it’s not what you think, or maybe it is it’s not what you think, or maybe it is

At Cloudflare, we’re committed to making it as easy as possible for developers to make their ideas come to life. Our announcements this week aim to give developers all the tools they need to build their next application on the edge. These include things like static site hosting, certificate management, and image services, just to name a few.

Today, we’re thrilled to announce that we’re exploring a new type of service at the edge: containers.

This announcement will be exciting to some and surprising to many. On this very blog, we’ve talked about why we believe isolates — rather than containers on the edge — will be the future model for applications on the web.

 it’s not what you think, or maybe it isIsolates are best for Distributed Systems

Let us be clear: isolates are the best way to do edge compute, period. The Workers platform is designed to allow developers to treat our global network as one big computer. This has been a long-held dream of generations of engineers, inspiring slogans like "The Network is the Computer" — a trademark which, incidentally, we now own. Isolates and Durable Objects are finally making that vision possible.

In short, isolates excel at distributed systems. They are perfect for treating the network as one big computer.

Isolates are great for distributed systems because, by being extremely lightweight, they enable us to reduce the unit of compute to a very fine granularity. That in turn allows work to be more effectively distributed across a large network. It is completely reasonable and efficient (takes just a few milliseconds, less than a TLS handshake) to spin up an isolate to handle one single HTTP request on the edge, which means we can choose the ideal location for each request to be processed. In contrast, because containers and virtual machines are heavier weight, it's necessary to centralize traffic on a small number of instances to achieve economies of scale.

But there's still a place for containers

Some applications are not really meant to be distributed. Consider, for example, a modern, single-player 3D video game. Such a game can be processing dozens of gigabytes of data every second, which by some measures sounds like "Big Data." Can we make games like that better by running them as a distributed system across a cluster of machines? It turns out… probably not. The problem is that all that data is being compiled down into a single output stream (video frames) which must be delivered in a specific sequence with minimal latency. With today's technology, it just doesn't make sense to distribute this work across a network. As such, isolates don't offer much benefit for this use case.

Meanwhile, at least today, isolates present a challenge when supporting legacy systems. The ecosystem of tooling and technology stacks for isolates is still young and developing. Writing a new application on isolates is great, but taking a complex existing codebase and porting it to run in isolates takes considerable effort. In the case of something like a 3D game, it may not even be possible, as the APIs to access GPUs may not be available. We expect this to improve, but it won't happen overnight.

Isolates Containers Distributed/global systems Web application servers Big data (e.g. MapReduce) Legacy/single-user applications 3D rendering CI builds We needed them too

We even have a small confession to make: we already built the capability to run containers at the edge for ourselves, specifically for our Browser Isolation product. This product lets you run your web browser on Cloudflare's servers and stream the graphics back to your client machine, increasing security and performance. We didn't build our own browser for this — our technology is based on Chromium.

Chromium is a big existing codebase that cannot realistically run inside isolates today. In fact, the "isolate engine" that Workers is built on — V8 — is itself a piece of Chromium. It's not designed to nest within itself — maybe someday, but not today.

Moreover, a web browser is another example of an application that doesn't make sense to be "distributed." A browser is extremely complex, but serves only one user. It doesn't need to be infinitely scalable or run all around the world at once.

So, instead of trying to build Browser Isolation on Workers, we deployed a container engine to our edge to run Chromium.

Another way to run isolates at the edge

“The edge”, of course, doesn’t have to mean running in all 200+ data centers all the time. We’ve also been able to use containers on the edge ourselves by running them in off-peak locations and for non-latency sensitive tasks. The scheduler for scheduled Workers, for example, runs on our internal container service. Since scheduled events don’t have an end user waiting on a timely response, we’re able to run events in data centers where it’s nighttime and the traffic levels are low.

Another great use case is running CI builds on the edge, though not for the reason you think. Web traffic in any particular location goes through daily cycles. During off-peak hours, a lot of compute is not used. These off-peak locations would be perfect for running batch work like builds in order to maximize compute efficiency.

What about migrating my containers to the edge to make them faster?

While there are some use cases better suited for containers, moving your container workload from its centralized location to the edge may not be the silver bullet you were hoping for.

A container-based web application running in Node.js or Django, for example, is unlikely to reap the same benefits from running on the edge. Due to the high overhead required by containers, your application will experience hundreds of milliseconds and often upwards of seconds of cold starts even when running on the edge. In that context, the saved network latency becomes negligible.

Even if the average request to a warmed-up container was faster, would you be willing to pay a premium for distributing it to 200+ data centers, rather than your current one or two?

Another thing to keep in mind is that being at the edge may introduce considerable cognitive overhead for legacy server stacks in containers. Managing the state of your application running in 200+ locations around the world is very different from managing it in one, two, or even three data centers. We've specifically designed Workers and Durable Objects to abstract away these concerns, but with classical server stacks running in containers, it may not be so easy.

With Cloudflare Workers and now Durable Objects — which were built with the edge in mind — we believe we have the right abstractions to allow developers to build for the edge first.

Container support is for a more limited class of applications that can’t be easily migrated today.

Still can’t contain your excitement?

If you have a use case for running containers at our edge, we’d love to know about it! Sign up for our early access (currently restricted to our enterprise plans) and let us know.

Categories: Technology

Cloudflare’s Partnership with HashiCorp and Bootstrapping Terraform with Cf-Terraforming

Sat, 17/04/2021 - 14:00
Cloudflare’s Partnership with HashiCorp and Bootstrapping Terraform with Cf-TerraformingCloudflare’s Partnership with HashiCorp and Bootstrapping Terraform with Cf-Terraforming

Cloudflare and HashiCorp have been technology partners since 2018, and in that time Cloudflare’s integration with HashiCorp’s technology has deepened, especially with Terraform, HashiCorp’s infrastructure-as-code product. Today we are announcing a major update to our Terraform bootstrapping tool, cf-terraforming. In this blog, I recap the history of our partnership, the HashiCorp Terraform Verified Provider for Cloudflare, and how getting started with Terraform for Cloudflare developers is easier than ever before with the new version of cf-terraforming.

Cloudflare and HashiCorp

Members of the open source community wrote and supported the first version of Cloudflare's Terraform provider. Eventually our customers began to bring up Terraform in conversations more often. Because of customer demand, we started supporting and developing the Terraform provider ourselves. You can read the initial v1.0 announcement for the provider here. Soon after, Cloudflare’s Terraform provider became ‘verified’ and we began working with HashiCorp to provide a high quality experience for developers.

HashiCorp Terraform allows developers to control their infrastructure-as-code through a standard configuration language, HashiCorp Configuration Language (HCL). It works across a myriad of different types of infrastructure including cloud service providers, containers, virtual machines, bare metal, etc. Terraform makes it easy for developers to follow best practices when interacting with SaaS, PaaS, and other service provider APIs that set up infrastructure. Like developers already do with software code, they can store infrastructure configuration as code in git, manage changes through code reviews, and track versions and commit history over time. Terraform also makes it easier to roll back changes if developers discover issues after a deployment.

Our developers love the power of Cloudflare + Terraform for infrastructure provisioning. IT operations teams want a platform that provides complete visibility and control. IT teams want a platform that is easy to use, does not have a steep learning curve, and provides levers to control resources much better. Cloudflare + Terraform platform provides just that.
– Dmitry Zadorojnii, Chief Technology Officer, Autodoc GmbH

Since the 1.0 release of Cloudflare’s Terraform provider, Cloudflare has continued to build out the capabilities exposed in the provider while HashiCorp has expanded its ecosystem by developing additional features like the Terraform Registry. This helps developers find documentation on Cloudflare’s Terraform provider or any others. Terraform itself has also matured greatly–Terraform’s v0.12 release had many changes, outlined here.

“Leveraging the power of Terraform, you can codify your Cloudflare configuration. Codification enables version control and automation, increasing productivity and reducing human error. We are pleased to have Cloudflare as a technology partner and look forward to our ongoing collaboration.”
– Asvin Ramesh, Director, Technology Alliances, HashiCorpGetting started with Cloudflare’s Terraform Provider

Here are some great resources for developers looking to better understand how to get started using Terraform with Cloudflare:

Bootstrapping Terraform configuration with Cf-Terraforming

We released the first version of cf-terraforming in early 2019 in this blog post. Since then we learned a few lessons about building and maintaining such a tool. In this section I’ll recap why we built such a tool in the first place, lessons learned over the last two years, and what is new and improved with the version we are launching today.

Why terraform-ing and why would a developer need this tool?

The name for the cf-terraforming library comes from another tool created by dtan4 on github: The original tool allowed users to generate tfconfig and tfstate files for existing AWS resources. This made it much easier for existing AWS customers to begin using Terraform. Terraform generally expects to be authoritative about the configuration it manages. Effectively, it expects that you only make changes to that config through Terraform and not anywhere else, like an API or a dashboard. If you were an existing customer of AWS (or Cloudflare) and already had everything configured via API or a UI, this posed a challenge: How do I quickly and correctly get all of my existing config into Terraform so that it can be the authoritative source of truth? For AWS resources, before dtan4’s terraforming you had to manually write the tfconfig for every object and the run import commands to generate the corresponding tfstate. For sizable deployments, this could be nigh impossible.

cf-terraforming served to solve the same problem for Cloudflare services. I had many conversations with customers who had been using Cloudflare for years and who were interested in migrating control of their Cloudflare configuration to Terraform. Cf-terraforming gave them a way to quickly convert all of their existing Cloudflare usage into tfconfig and tfstate.

cf-terraforming was one of the enabling technologies that we used to bootstrap Cloudflare into our Terraform Git-ops workflow. We had thousands of records to port, import, and manage, and doing this by hand would have been an arduous and error-prone task. Using cf-terraforming to generate our initial set of resources allowed our engineers to submit Cloudflare changes, enabling our product engineers to be infrastructure operators.
Sean Chittenden, Infrastructure, DoorDashWhat we have learned

After having cf-terraforming available for some time, we’ve learned quite a bit about the challenges in managing such a tool.

Duplication of effort when resources change

When Cloudflare releases new services or features today, that typically means new or incremental changes to Cloudflare’s APIs. This in turns means updates to our Terraform provider. Since Terraform is a golang program, before we can update our provider we have to first update the cloudflare-go library. Depending on the change, this can be a couple lines in each repo or extensive changes to both. Once we launched cf-terraforming, we now had a third library that needed synchronous changes alongside the provider and go library. Missing a change meant that if someone tried to use cf-terraforming, they may have incomplete config or state, which would not work.

Impact of changes to Terraform for the tool

Not only did our own API changes create additional work, but changes to Terraform itself could mean changes for cf-terraforming. The Terraform 0.12 update was a massive update that required a lot of careful testing and coordination with our provider. It also meant changes in HCL and in provider interactions that cf-terraforming had to account for. Such a massive one-time hit was very difficult to account for, and we’ve struggled to ensure compatibility.

TF State management

The ability to have cf-terraforming generate a tfstate file was both incredibly important and also experimental. In general a developer never really needs to concern themselves with what is in the tfstate file but needs to know it contains the actual state of those resources such that references in the config can be resolved and managed correctly. We opened up that black box, which meant that we were involved in state file implementation details that we ultimately shouldn’t be.

Given these lessons, we looked at how we could update cf-terraforming to alleviate these problems and provide a better tool for both customers and ourselves. After some prototyping to validate our ideas, we came up with a new model that has been productized and is now available for customers.

What’s new in cf-terraforming

Today we are launching a new version of cf-terraforming that improves upon our previous work. This new version makes it easier for us to support new resources or changes to existing resources and simplifies the workflow for developers looking to bootstrap their Cloudflare Terraform configuration.

Simplified management

Instead of hand crafting how to generate both the tfconfig and tfstate for each of the 48 or so resources supported in the Terraform provider, we now leverage Terraform’s capabilities to do more auto generation of what’s needed for similar resource types. HashiCorp has a great CLI tool called terraform-exec that provides powerful out-of-the-box capabilities we can take advantage of. Using terraform-exec we get access to `terraform providers schema -json`, which gives us the json schema of our provider. We use this to auto generate the fields we need to populate from the API. In many cases the API response fields map one to one with the json schema, which allows us to automatically populate the tfconfig. In other cases some small tweaks are necessary, which still saves a lot of time to initially support the resource and lowers the burden for future changes. Through this method, if the terraform provider changes for any reason, we can build new versions of cf-terraforming that will fetch the new schema from terraform-exec versus us having to make a lot of manual code changes to the config generation.

For tfstate, we simplify our approach by outputting the full set of terraform import calls that would need to be run for those resources instead of attempting to generate the tfstate definition itself. This removes virtually any need for future library changes since the import commands do not change if Cloudflare’s API or provider changes.

How to use the new cf-terraforming

With that, let’s look at the new cf-terraforming in action. For this walkthrough let’s assume we have an existing zone on Cloudflare with DNS records and firewall rules configured. We want to start managing this zone in Terraform, but we don’t want to have to define all of our configuration by hand.

Our goal is to have a ".tf" file with the DNS records resources and firewall rules along with filter resources AND for Terraform to be aware of the equivalent state for those resources. Our inputs are the zone we already have created in Cloudflare, and our tool is the cf-terraforming library. If you are following along at home, you will need terraform installed and at least Go v1.12.x installed.

Getting the environment setup

Before we can use cf-terraforming or the provider, we need an API token. I’ll briefly go through the steps here, but for a more in-depth walkthrough see the API developer docs. On the Cloudflare dashboard we generate an API token here with the following setup:

Zone:Firewall Services:Read

Zone Resources: (my zone, but this should be your own)

Valid until: 2021-03-30 00:00:00Z

Note: I set an expiration date on the token so that when I inevitably forget about this token, it will expire and reduce the risk of exposure in the future. This is optional, but it’s a good practice when creating tokens you only need for a short period of time especially if they have edit access.

Cloudflare’s Partnership with HashiCorp and Bootstrapping Terraform with Cf-TerraformingAPI Token summary from the Cloudflare Dashboard

Now we set the API Token we created as an environment variable so that both Terraform and cf-terraforming can access it for any commands (and so I don’t have to remove it from code examples).

$export CLOUDFLARE_API_TOKEN=<token_secret>

Terraform requires us to have a folder to hold our Terraform configuration and state. For that we create a folder for our use case and create a config file with a provider definition for Cloudflare so Terraform knows we will be using the Cloudflare provider.

mkdir terraforming_test cd terraforming_test cat > <<'EOF' terraform { required_providers { cloudflare = { source = "cloudflare/cloudflare" } } } provider "cloudflare" { # api_token = "" ## Commented out as we are using an environment var } EOF

Here is the content of our file if you would rather copy and paste it into your text editor of choice:

terraform { required_providers { cloudflare = { source = "cloudflare/cloudflare" } } } provider "cloudflare" { # api_token = "" ## Commented out as we are using an environment var }

We call terraform init to ensure Terraform is fully initialized and has the Cloudflare provider installed. At the time of writing this blog post, this is what terraform -v gives me for version info. We recommend that you use the latest versions of both Terraform and the Cloudflare provider.

$ terraform -v Terraform v0.14.10 + provider v2.19.2

And finally we install cf-terraforming with the following command:

$ GO111MODULE=on go get -u

If you’re using Homebrew on MacOS, this can be simplified to:

$ brew tap cloudflare/cloudflare $ brew install --cask cloudflare/cloudflare/cf-terraforming Using cf-terraforming to generate Terraform configuration

We are now ready to start generating a Terraform config. To begin, we run cf-terraforming to generate the first blocks of config for the DNS record resources and append it to the file we previously created.

cf-terraforming generate --resource-type cloudflare_record --zone <zone_id> >>

Breaking this command down:

generate is the command that will produce a valid HCL config of resources

--resource-type specifies the Terraform resource name that we want to generate an HCL config for. You can only generate configuration for one resource at a time. In this example we are using cloudflare_record

--zone specifies the Cloudflare zone ID we wish to fetch all the DNS records for so cf-terraforming can create the appropriate API calls


$ cf-terraforming generate --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414 >> $

Success will return with no output to console. If you want to see the output before adding it to the config file, run the command without >> and it will output to console.

Here is the partial output in my case, if it is not appended to the config file:

$ cf-terraforming generate --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414 resource "cloudflare_record" "terraform_managed_resource_db185030f44e358e1c2162a9ecda7253" { name = "api" proxied = true ttl = 1 type = "A" value = "x.x.x.x" zone_id = "9c2f972575d986b99fa03c7bbfaab414" } resource "cloudflare_record" "terraform_managed_resource_e908d014ebef5011d5981b3ba961a011" { ...

The output resources are given standardized names of “terraform_managed_resource_<resource_id>”. Because the resource id is included in the name, the object names between the config we just exported and the state we will import will always be consistent. This is necessary to ensure Terraform knows which config belongs to which state.

After generating the DNS record resources, we now do the same for both firewall rules and filters.

cf-terraforming generate --resource-type cloudflare_firewall_rule --zone <zone_id> >> cf-terraforming generate --resource-type cloudflare_filter --zone <zone_id> >>


$ cf-terraforming generate --resource-type cloudflare_firewall_rule --zone 9c2f972575d986b99fa03c7bbfaab414 >> $ cf-terraforming generate --resource-type cloudflare_filter --zone 9c2f972575d986b99fa03c7bbfaab414 >> $ Using cf-terraforming to import Terraform state

Before we can ask Terraform to verify the config, we need to import the state so that Terraform does not attempt to create new objects but instead reuses the existing objects we already have in Cloudflare.

Similar to what we did with the generate command, we use the import command to generate terraform import commands.

cf-terraforming import --resource-type cloudflare_record --zone <zone_id>

Breaking this command down:

import is the command that will produce a valid terraform import command that we can then run
--resource-type (same as the generate command) specifies the Terraform resource name that we want to create import commands for. You can only use one resource at a time. In this example we are using cloudflare_record
--zone (same as the generate command) specifies the Cloudflare zone ID we wish to fetch all the DNS records for so cf-terraforming can populate the commands with the appropriate API calls

And an example with output:

$ cf-terraforming import --resource-type cloudflare_record --zone 9c2f972575d986b99fa03c7bbfaab414 terraform import cloudflare_record.terraform_managed_resource_db185030f44e358e1c2162a9ecda7253 9c2f972575d986b99fa03c7bbfaab414/db185030f44e358e1c2162a9ecda7253 terraform import cloudflare_record.terraform_managed_resource_e908d014ebef5011d5981b3ba961a011 9c2f972575d986b99fa03c7bbfaab414/e908d014ebef5011d5981b3ba961a011 terraform import cloudflare_record.terraform_managed_resource_3f62e6950a5e0889a14cf5b913e87699 9c2f972575d986b99fa03c7bbfaab414/3f62e6950a5e0889a14cf5b913e87699 terraform import cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d 9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d$

The output of this will be ready to use terraform import commands. Running the generated terraform import command will leverage existing Cloudflare Terraform provider functionality to import the resource state into Terraform’s terraform.tfstate file. This removes the tedium of pulling all the appropriate resource IDs from Cloudflare’s API and then formatting these commands one by one. The order of operations of the config then state is important as Terraform expects there to be configuration in the .tf file for these resources before importing the state.

Note: Be careful when you actually import these resources, though, as from that point on any subsequent Terraform actions like plan or apply will expect this resource to be there. Removing the state is possible but requires manually editing the terraform.tfstate file. Terraform does keep a backup locally in case you make a mistake though.

Now we actually run these terraform import commands to import the state. Below shows what that looks like for a single resource.

$ terraform import cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d 9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Importing from ID "9c2f972575d986b99fa03c7bbfaab414/47581f47852ad2ba61df90b15933903d"... cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Import prepared! Prepared cloudflare_record for import cloudflare_record.terraform_managed_resource_47581f47852ad2ba61df90b15933903d: Refreshing state... [id=47581f47852ad2ba61df90b15933903d] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform.

With cloudflare_record imported, now we do the same for the firewall_rules and filters.

cf-terraforming import --resource-type cloudflare_firewall_rule --zone <zone_id> cf-terraforming import --resource-type cloudflare_filter --zone <zone_id>

Shown with output:

$ cf-terraforming import --resource-type cloudflare_firewall_rule --zone 9c2f972575d986b99fa03c7bbfaab414 terraform import cloudflare_firewall_rule.terraform_managed_resource_0de909f3229341a2b8214737903f2caf 9c2f972575d986b99fa03c7bbfaab414/0de909f3229341a2b8214737903f2caf terraform import cloudflare_firewall_rule.terraform_managed_resource_0c722eb85e1c47dcac83b5824bad4a7c 9c2f972575d986b99fa03c7bbfaab414/0c722eb85e1c47dcac83b5824bad4a7c $ cf-terraforming import --resource-type cloudflare_filter --zone 9c2f972575d986b99fa03c7bbfaab414 terraform import cloudflare_filter.terraform_managed_resource_ee048570bb874972bbb6557f7529e094 9c2f972575d986b99fa03c7bbfaab414/ee048570bb874972bbb6557f7529e094 terraform import cloudflare_filter.terraform_managed_resource_1bb6cd50e2534a64a9ec698fd841ffc5 9c2f972575d986b99fa03c7bbfaab414/1bb6cd50e2534a64a9ec698fd841ffc5 $

As with cloudflare_record, we run these terraform import commands to ensure all the state is successfully imported.

Verifying everything is correct

Now that we have both the configuration and state in place, we call terraform plan to see if Terraform can verify everything is in place. If all goes well then you will be greeted with the following “nothing to do” message:

No changes. Infrastructure is up-to-date. This means that Terraform did not detect any differences between your configuration and real physical resources that exist. As a result, no actions need to be performed.

You now can begin managing these resources in Terraform. If you want to add more resources into Terraform, follow these steps for other resources. You can find which resources are supported in the README. We will add additional resources over time, but if there are specific ones you are looking for, please create GitHub issues or upvote any existing ones.

It has never been easier to get started with Cloudflare + Terraform

Whether you are an existing Cloudflare customer and have been curious about Terraform or you are looking to expand your infrastructure-as-code to include Cloudflare’s services, you have everything you need to get building with Terraform, the Cloudflare provider, and cf-terraforming. For questions, comments, or feature requests for either the provider or cf-terraforming, see the respective github repos.

Categories: Technology

Announcing Cloudflare’s Database Partners

Fri, 16/04/2021 - 14:00
Announcing Cloudflare’s Database PartnersAnnouncing Cloudflare’s Database Partners

Cloudflare Workers is the easiest way for developers to deploy their application’s code with performance, scale and security baked in. No configuration necessary. Worker code scales to serve billions of requests close to your users across Cloudflare’s 200+ data centers.

But that’s not the only interesting problem we need to solve. Every application has two parts: code and state.

State isn’t always the easiest to work in a massive distributed system. When an application runs in 200+ data centers simultaneously, there’s an inherent tradeoff between distributing the data for better performance, availability, scale, and guaranteeing that all data centers see the same data at a given point in time.

Our goal is to make state at the edge seamless. We started that journey with Workers KV, which provides low-latency access to globally distributed data. We’re since added Durable Objects, with strong consistency and the ability to design coordination patterns on top of Workers. We’re continuing to invest in and build out these products.

However, some use cases aren’t easily implemented with Workers KV or Durable Objects. Think querying complex datasets, or communicating with an existing system-of-record. Even if we built this functionality ourselves, there will always be customers who want to use a different tool for data storage and access.

Today, we’re making our data strategy clear so that developers can build on Workers with confidence. First, we’re announcing a partnership with two distributed data companies, Macrometa and Fauna, to help developers choose an edge-first database when they build a new application on Workers. Next, we’ve introduced tutorials and guides for databases that already support HTTP connections to make connecting to existing databases even easier.

With this, the Workers runtime is the easiest way to handle both parts of your application that matter: code and state.

Announcing Cloudflare’s Database Partners

Cloudflare is excited to announce a partnership for global data with Macrometa. Macrometa provides a serverless data platform for developers to build rich, geo-distributed, stateful applications on the edge with Cloudflare Workers. The Macrometa Global Data Network (GDN) is an ultra-fast, globally replicated noSQL database with integrated features for search, pub/sub, streams, and stream processing.

Macrometa complements Cloudflare Workers by enabling stateful functions to run, store, query, mutate, and serve data globally with local read/write latencies. Tables replicate across Macrometa’s global data centers with configurable consistency and without additional code, cost, or complexity. Data calls between Workers and Macrometa are automatically routed and served from the closest Macrometa region relative to the Cloudflare Worker invocation. We have seen that end-to-end latency for a database request from a client to the edge and back is <75ms for 99th percentile.

Macrometa combines typically disparate data services and APIs (each with their own data model and copy of data) into a single, integrated, and intuitive data API (with a single, unified common data set and model). At a high level, one can think of Macrometa as combining a noSQL database together with pub/sub, streaming data and event processing along with search into a seamless, globally replicated, low latency data platform.

The following multimodal interfaces are available to the developer via a SQL-like query language:

  • Key/value database with high write performance, low latencies, and Time to Live (TTL) support
  • JSON document database with a SQL-like query language
  • Graph database for storing and retrieving relationships between entities
  • DynamoDB API compatible database
  • Low latency edge cache for database and API results with global cache invalidation and Time to Live (TTL)
  • Search
  • Pub/Sub & queues with Kafka, Pulsar and MQTT message compatibility
  • Streaming data and complex event processing - evaluate events, aggregate, enrich, filter, summarize, pattern match on stream data in real time.

Apps written using Cloudflare Workers consume Macrometa via a JavaScript SDK or REST API. Developers have a choice of using one of two SDKs tightly integrated with Cloudflare Workers:

  • A Cloudflare Worker specific JavaScript SDK which provides the full set of capabilities mentioned above
  • The Dynamo client SDK that enables developers to use Macrometa as a drop-in replacement AWS DynamoDB (or as an edge cache).
Example Use-casesAnnouncing Cloudflare’s Database PartnersWith Cloudflare and Macrometa, developers can now build and deliver powerful and compelling data-driven experiences in a way that the centralized clouds will never be able to. The world deserves a better cloud - an edge cloud.
- Chetan Venkatesh, Co-founder/CEO, Macrometa

If you are interested in learning more about using Macrometa and Cloudflare Workers together, please register here.

Case study for Cloudflare Workers + Macrometa: A bookstore demo app

To showcase the power and performance of Cloudflare and Macrometa together, we re-built AWS’s demo application — a complete E-Commerce backend that serves an online bookstore. eCommerce backends for large global sites are complex and require many different types of databases. These complex architectures are necessary to store, perform multi-table or multi-data store queries, and serve product catalogs, pricing, search, recommendations, order management, cart operations, fulfillment, etc.

The bookstore is a serverless app built as a Cloudflare Worker to serve a React single page application and uses Macrometa for storing and serving product/catalog data, orders, cart operations, etc. It uses streams and complex event processing to provide a real-time leaderboard of the top-selling books and uses the Macrometa search service to deliver near-instant search results. You can test drive the end-user experience here. Our tests show orders of magnitude latency improvement for a user vs. the same use case built on a legacy cloud provider.

To check out the source code, head over to GitHub.

Announcing Cloudflare’s Database PartnersAnnouncing Cloudflare’s Database Partners

Cloudflare is excited to announce a new partnership for transactional workloads with Fauna, a data API for modern applications. Together, Cloudflare and Fauna enable developers to build more powerful stateful applications that scale dynamically and maintain strong consistency without any additional configuration.

Fauna is a flexible, developer-friendly, global transactional database delivered as a secure, global, cloud API — no database operations required. With Fauna, developers can simplify code and ship faster by replacing their database infrastructure with a data API that combines the flexibility of relational and document-centric querying with the power of custom business logic, and the ease of GraphQL at the edge.

Fauna is a simple yet scalable backend for applications built on Cloudflare’s services such as Workers and Pages. By leveraging a web-native (HTTP-based) invocation model with support for modern security protocols, Fauna eliminates the connection limits introduced by traditional databases and can be integrated directly with serverless functions and applications running at the edge.

Fauna complements Cloudflare KV and Durable Objects by providing a global, queryable, strongly consistent persistence layer for the mission-critical data required to build modern web and mobile applications. It’s underlying globally distributed storage engine is fast, consistent, and reliable. Developers can rely on Fauna’s unique global transactional capabilities to ensure that mission-critical business data is always strongly consistent with minimal latency.

Fauna, a serverless offering, is easy to get started with for free, and lets developers experience freedom from database operations at any scale. Fauna is available through GraphQL or drivers that work with Workers in several popular programming languages.

If you’re interested in learning how to use Fauna with Cloudflare Workers, please visit this step-by-step tutorial for a hands-on experience! If you are interested in learning more about using Cloudflare Workers and Fauna together, please register here.

The integration of Cloudflare workers with Fauna allows our joint customers to simply and cost effectively bring both data and logic to the edge. Fauna was developed from the ground up as a globally distributed, serverless database and when combined with Cloudflare Workers serverless compute fabric provides developers with a simple, yet powerful abstraction over the complexities of distributed infrastructure.
- Evan Weaver, Co-founder/CTO, FaunaAnnouncing Cloudflare’s Database PartnersAnnouncing Cloudflare’s Database PartnersCase study for Cloudflare Workers + Fauna: MeetKai, An AI-Powered Voice Assistant

Let’s take a look at a current customer of Fauna and Cloudflare who benefits from Fauna’s ability to offer globally-distributed transactional data access at low latency.


MeetKai is an AI-powered voice assistant that makes a consumer’s life easier through conversation, personalization & curation. Deployed by enterprises with Monthly Active Users in the hundred million plus range, MeetKai uses Fauna and Cloudflare together to deliver a responsive, global experience.

Cloudflare Workers act as the proxy to all downstream requests. Doing so enables MeetKai to perform much smarter load balancing at the edge, using the Worker rather than some other competing solutions. When combined with Fauna, this serverless-first approach delivers scale and performance at predictable costs, while reducing time to market. By pushing data to the edge with Fauna, MeetKai are able to inject personalization information before a request ever hits the backend! Learn more about how MeetKai uses Cloudflare Workers and Fauna in this technical case study.

Combining Cloudflare Workers with Fauna allows us to bring both data and logic to the edge, enabling us to deliver a differentiated search engine while still meeting high operations requirements in latency and uptime -- all at a fraction of the cost and complexity of non-serverless solutions.
- James Kaplan, Founder/CEO MeetKaiAnnouncing Cloudflare’s Database PartnersConnecting to existing databases

In addition to our partners, Cloudflare Workers can work with any database that accepts connections over HTTP.

In particular, we’ve documented connections to both DynamoDB and AWS Aurora, supporting either Postgres or MySQL. These connections let you access already existing databases directly from Cloudflare’s edge, making it easy to build new applications on top of existing data.

Connect a Worker to DynamoDB

Connect a Worker to AWS Aurora

In addition, we’ll be adding more tutorials to document how to connect to other databases that work with Workers, like Google’s Cloud Firestore.

Advancing state at the edge

The Workers platform has built a new paradigm for developers for serverless. Across our own Durable Objects and Workers KV, our partner databases, and connections to existing providers, we’re bringing that same paradigm shift to state at the edge. As we continue to build out our database partnerships and data platform, we want to hear from you! Send your ideas and feedback to the Workers Product team to let us know what databases and features you’d most like to see us build.

Categories: Technology

Node.js support in Cloudflare Workers

Fri, 16/04/2021 - 14:00
Node.js support in Cloudflare WorkersNode.js support in Cloudflare Workers

We released Cloudflare Workers three years ago, making edge compute accessible to the masses with native support for the world’s most ubiquitous language — JavaScript.

The Workers platform has transformed so much since its launch. Developers can not only write sandboxed code at our edge, they can also store data at the edge with Workers KV and, more recently, coordinate state within our giant network using Durable Objects. Now, we’re excited to share our support of an 11 year old technology that’s still going strong: Node.js.

Node.js made a breakthrough by enabling developers to build both the frontend and the backend with a single language. It took JavaScript beyond the browser and into the server by using Chrome’s JavaScript engine, V8.

Workers is also built on V8 Isolates and empowers developers in a similar way by allowing you to create entire applications with only JavaScript — except your code runs across Cloudflare’s data centers in over 100 countries.

Our Package Support Today

There is nothing more satisfying than importing a library and watching your code magically work out-of-the-box.

For over 20k packages, Workers supports this magic already: any Node.js package that uses webpack or another polyfill bundler runs within our environment today. You can get started with the greatest hits packages like node-jose for encryption, itty-router for routing, graphql for querying your API, and so much more.

And rather than finding out by trial and error, we made a catalogue of libraries that you can rely on. All you have to do is pick one and boom: it runs.

Node.js support in Cloudflare Workers

Once you select a package, you can use webpack to bundle it all up with our wrangler CLI and deploy onto Workers. Webpack is a module bundler that takes your JavaScript files, including third-party dependencies, and makes them usable in the browser.

For an example of bundling dependencies in action, see this example of getting started with Gatsby.

Our Next StepsIncreasing Worker sizes

Using webpack can get you far, but that can cause the Worker to exceed the size limit quickly. Node.js was designed with the assumption that servers, unlike the client, are amenable to code bloat, resulting in an ecosystem of packages that are generous in size.

We plan to support raising the 1MB size limit for Workers soon, so users don’t have to worry about the size of their dependencies. Please share what you’re building in the Workers Unbound channel of our Discord if you’d like that limit raised.

Supporting Native APIs

But why stop there? We want to go even further and support the most important modules, even if they do rely on native code. Our approach will be to reimplement supported modules and polyfill package functionality directly into the Workers runtime. This doesn’t mean we’re shifting our runtime to run on Node.js. In fact, here are two important security and design reasons why we are not:

  • Node.js was not designed to be a sandbox, which was made apparent by their vm module that says “do not use it to run untrusted code.”
  • For proper sandboxing, Node.js would’ve forced us to build a container-based runtime that both doesn’t scale and isn’t as performant as Isolates. Without containers, we were able to design a system that has 0ms cold starts.

However, there are other ways we can be Node.js compatible without necessarily supporting the entire runtime. What’s up first? We’ll support Stripe.js SDK and Twilio Client JS SDK. We’ll also build support for the net module, so you can run popular database libraries.

But we want to hear from you! We created a leadership board for you to vote on which popular libraries/APIs matter the most. Are statistics packages your jam? Do you need an email utility? What about a templating engine? We want to hear directly from you.

We won’t stop until our users can import popular Node.js libraries seamlessly. This effort will be large-scale and ongoing for us, but we think it’s well worth it.

We’re excited to support developers as they build all types of applications. We look forward to hearing from you!

Categories: Technology

Cloudflare Stream now supports NFTs

Fri, 16/04/2021 - 14:00
Cloudflare Stream now supports NFTsCloudflare Stream now supports NFTs

Cloudflare Stream has been helping creators publish their videos online without having to think about video quality, device compatibility, storage buckets or digging through FFmpeg documentation. These creators want to be able to claim ownership of their works and assert control over how that ownership claim is transferred. Increasingly, many of those creators are looking to Non-Fungible Tokens (NFTs).

NFTs are a special type of smart contract that allows provable ownership of the contract on the blockchain. Some call NFTs collectibles because like coins or stamps, collectors who enjoy them buy, sell and trade them. Collectors keep track of NFTs on the Ethereum blockchain which acts as a shared source of truth of all the activity.

Today, we’re introducing a new API that takes a ERC-721 token ID and contact address and sets it on a video so every video on Stream can be represented with an NFT.

curl -X POST -H "Authorization: Bearer $AUTH_TOKEN" --data '{"contract":"0x57f1887a8bf19b14fc0d912b9b2acc9af147ea85","token":"5"}'$ACCOUNT_ID/stream/$VIDEO_ID/nft

Once you set it, you cannot change these values so be sure to set it to an NFT you own! If you set a video you own to an NFT you don’t own, the owner of the NFT can claim your video and delete it. As the owner of the linked NFT changes, admin rights on videos can be transferred between users without the involvement of the original user who uploaded the video.

How can I represent a video with a NFT?

Stream doesn’t create NFTs for you, however, you can use online platforms like OpenSea and  Foundation or a command line tool like Minty to create them. These tools don't actually host your video but only allow you to create a NFT on the blockchain. For example, if you shot a cool video and want to turn it into an NFT, you'd use a third party service to get a token ID and contact address. Starting today, you can associate the token ID and contact address with your video on Stream.

You should consider storing a copy of the video you shot in a storage system like IPFS or Filecoin as they can help with long term storage for the NFT. A link from a decentralized storage system should be the one to use in your NFT metadata URL (We’re working on making a copy of the Stream videos available automatically in IPFS to facilitate long term storage of videos).

You can use Stream with your NFT to store high quality or longer versions of the video that may cost too much to store and distribute through decentralized storage. Stream offers accessibility features such as captions, compatibility with mobile and desktop devices and automatic quality switching.

Cloudflare Stream now supports NFTsWhy NFTs?

NFTs allow for unlimited options for unique business models around media: videos can be owned by an entity with many owners (made into a crypto coin with voting rights proportional to ownership), crowdfunded (for example as done for the $ESSAY) or simply traded (like cryptokittes or NBA Top Shot).

Digital ownership is not new (domain names are a good example that existed for decades) but it has never been this popular. We’ve been watching the explosive growth of the NFT community closely. If you’ve heard of CryptoKitties a while back or visited any news website recently you’ve probably heard of the vibrant community forming around digital ownership of art and the growth of creators that are able to monetize their work.  By permitting creators to embed their NFT claim to ownership into their video on Cloudflare Stream, we hope this will be a helpful step as acceptance of the process for minting, validating, and utilizing NFTs moves forward.

If you visit one of the many NFT marketplaces, you will see that most tokens are either images or short animations yet there are no films. We think this is partly because it’s hard to host videos on the Internet and get it to work on every device and connection speed. Storage technologies such as IPFS are not yet fit to support video streaming and require work on the creators’ part to make accessibility features such as captions to work.

Cloudflare Stream now supports NFTsWe've been building OpenSea since the inception of NFTs in 2017. NFTs provide a brand-new format for media with new and exciting properties, including scarcity, provable ownership, and interoperability between platforms. But turning videos into NFTs has always been a difficult task to scale. Cloudflare is well-positioned to help here, and is stepping up to this problem by adding NFT support to Cloudflare Stream, allowing video creators to produce and globally distribute high-quality video NFTs. We're excited for the new applications this will enable!
- Alex Atallah, CTO, Opensea.ioCloudflare Stream now supports NFTsDapper Labs is excited that Cloudflare continues to make it easier for folks to adopt Blockchain, the most democratizing technology since the creation of the Internet itself. Cloudflare Stream creates new opportunities for creators to showcase and monetize their work using NFTs.
- Dieter Shirley, CTO at Dapper Labs and creator of ERC-721A model for decentralized ownership

By tagging token ID and contract on Stream videos, you make it possible to transfer admin rights on a video. If a change in ownership of the ERC-721 token happens on the Ethereum blockchain, the new owner of the token may claim control over the video on Cloudflare Stream by proving they own the token associated with the video. Once the video has been claimed via the API call, the video may be deleted or transferred to the new owner’s Cloudflare Stream account.

Here’s how the transfer API looks like:

curl -X POST -H "Authorization: $AUTH_TOKEN" --data '{"signature": "0xcac010d5ac8ef1f325c14fa0e7df5bb3a1920a205d827bd727e96b32b918875c26d8635dd778e0a1fa77f2cb547259e05fb40f3cb5c2673b77d77e372cc22f091b", "contract":"0x57f1887a8bf19b14fc0d912b9b2acc9af147ea85","token":"5"}'$ACCOUNT_ID/stream/nft/claim

Signature is the following string signed with the wallet that owns the NFT:

“I claim ownership of NFT #$TOKEN_ID under contract $CONTRACT and this message expires at $EXPIRY_TIME”

$EXPIRY_TIME is a RFC3339 formatted time in the soon future.

See an example signature

Cloudflare Stream now supports NFTs

Cloudflare Stream doesn’t do anything complicated besides storing a token ID and contract address pair for each video in our database. When an NFT is set, we verify compatibility with ERC-721 using Cloudflare’s Ethereum Gateway. When ownership on an NFT associated with a video is claimed, we update control over the video.

After making this API call, the video claimed shows up in the new account where the new owner of the NFT can change settings for the video, restrict views, add captions: anything that the original user who uploaded the video can do.

NFTs on the video player

Big Buck Bunny by Blender Foundation (Creative Commons Attribution 3.0 license) associated here with an example NFT.

An important part of digital ownership is linking back to the contract and token that governs the ownership. If a video on Stream is linked to an NFT, the Stream video player will show a link to the token and contract.

We’re hiring

We’re working on more ways to support creators and platforms that enable creators to do their best work. If you’re interested in working on technology for creators, Stream team at Cloudflare is hiring both in the United States and in Europe.

Categories: Technology

Introducing, custom builds, and improved logging for Workers

Fri, 16/04/2021 - 14:00
Introducing, custom builds, and improved logging for WorkersIntroducing, custom builds, and improved logging for Workers

Cloudflare Workers® aims to be the easiest and most powerful platform for developers to build and deploy their applications. With Workers, you can quickly solve problems without having to wonder: “is this going to scale?”

You write the JavaScript and we handle the rest, from distribution to scaling and concurrency.

In the spirit of quickly solving problems, we’re excited to launch three new improvements to the Workers experience, so you can take your next idea and ship it even faster.

Introducing... workers.newIntroducing, custom builds, and improved logging for Workers

First, we’re introducing, a shortcut that takes you directly to a JavaScript editor for creating a new Worker. Anytime you have a cool idea, need a quick fix to a problem, or just want to debug some JavaScript, you now have a simple way to go from idea to prototype. What’s more is you don’t even need to deploy the Worker to try it out!

Introducing, custom builds, and improved logging for Workers

We’ve also updated the default Worker template to help you go a few steps beyond the typical “Hello, World!”. When you open the editor, you’ll now see a few examples that demonstrate how to redirect requests, modify headers, and parse responses.

Customize your build scriptsIntroducing, custom builds, and improved logging for Workers

For developers who want to deploy more complex applications, like those with external libraries or written in other languages like TypeScript, we recommend using our command-line tool, wrangler.

Previously, wrangler has supported three project types: JavaScript, Rust, and webpack. By far, we’ve found that webpack is the most widely used among the community, since it has built-in support for bundling npm packages. Yet, as more developers started to use wrangler, they’ve wanted to customize their webpack configurations or even use other JavaScript bundlers.

That’s why we’re excited to release custom builds for wrangler! Now you can provide wrangler with a custom build script and upload directory, which works with all of the wrangler commands including wrangler dev and wrangler publish.

Introducing, custom builds, and improved logging for Workers

This means you can use any JavaScript bundler you want: webpack, Rollup, Parcel, esbuild, and many more! We’ve put together a few templates that you can copy if you want to get started with custom builds. Make sure to update wrangler to release 1.16 or later and check out our documentation about custom builds and additional configuration options.

Viewing logs and exceptions

Ever wanted to see the console.log() output or uncaught exceptions from a deployed Worker?

In case you missed it, wrangler tail allows you to see a live stream of your Worker’s logs and exceptions. Today, we’re improving wrangler tail by making it easier to use and more accessible to developers.

First, we wanted to improve the terminal output. Previously, wrangler tail would only output log entries as JSON. While this was useful when piping through to tools like jq, it was generally hard to digest. That’s why we created a new “pretty” format that is much easier to read and takes full advantage of your terminal’s color codes.

$ wrangler tail --format=<json|pretty> Introducing, custom builds, and improved logging for Workers

If you have suggestions for how we can improve wrangler tail, feel free to pop in the #wrangler channel on the Workers Discord. We’re excited to see what you ship with these improvements!

A sneak preview

As we like to say at Cloudflare, we’re just getting started. While we have plans to add improvements to wrangler... where else could we introduce logs?

Introducing, custom builds, and improved logging for Workers

Stay tuned.

Categories: Technology


Additional Terms