Overcoming memory issues installing Drupal with Composer

Note: Last updated July 2020 to update the Composer package for installing core on a new site to those currently recommended, which removes the need for creating Drupal scaffold files separately. Also, for reasons that I'll blog separately, I no longer fight Drupal going into a web subdirectory, so I've removed that part of the post.

Drupal 8 uses Composer for package management. You can still install a Drupal 8 site by downloading a tarball, but we're all encouraged to use Composer to download Drupal core and other dependencies and to keep things up to date.

I'm just starting to get my head around how Drupal 8 works, so that I can reach the point where I can build new sites in 8.x rather than 7.x, and in time migrate existing sites over. Composer is part of the learning curve for this.

If you start off with a downloaded tarball, it's possible to bring the site back into the Composer fold at a later date, but it's non-trivial. So, although it means I have more to learn and will therefore take longer to reach the point where I start using Drupal 8, it seems it would be worth mastering enough of Composer before I start.

I'd learnt how to download Drupal 8 using Composer, and some of what I'd need to do to maintain it. I've still got a lot to learn, but the basics are coming into place; I like what I'm finding. Then I hit an issue with the amount of memory Composer requires to do this, and I've worked out how to work around that.

So I thought I'd write this up. As usual, part of my reason for doing so is as personal documentation, and part is so that I might help others who hit a similar problem. Because comments are open, I'm also taking the opportunity to solicit feedback, so that I can improve my process further.

How it's supposed to work

The Drupal website has an excellent page entitled "Using Composer to manage Drupal site dependencies".

I won't distract this post by discussing the options of how to do this. My preference is also the recommended one on that page: Use the drupal/recommended-project. I also prefer the "modified install", so that it doesn't install in the web subdirectory.

Step 1: Download the project from git

git clone https://github.com/drupal/recommended-project.git my_site_name_dir

Usually, I want "my_site_name_dir" simply to be ".", i.e. the current directory.

Step 2: Tweak the composer.json file that's been downloaded.

For example I have a few bash scripts that I run when Composer's post-install-cmd event fires. I won't document that here.

Step 3: Install Drupal (and its dependencies)

composer install

That installs Drupal, the project (in Composer terms). Now all the files are in place, you're back in the normal place for installing a Drupal site. Either go to the site and run through the installation wizard, or install Drupal (the website) via Drush.

Either way, away we go!!!!!!!!

Out of Memory

Until, that is, you try to do this on a server with memory limitations.

It's widely documented that Composer needs a lot of memory to crunch what dependencies are required. The more the composer.json file fails to be specific about exactly what versions of each library are required, the more possibilities it has to consider, and the more memory it needs to do so.

So it is that you get this message:

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in phar:///opt/cpanel/composer/bin/composer/src/Composer/DependencyResolver/Rule.php on line 76

There are various things you can do about that, and I read lots of pages on Stack Exchange, Answers, Reddit and elsewhere that suggested the same things. I'll mention them here in case they help

Give more memory to PHP

Put something like this in your .bashrc file:

alias composer='php -d memory_limit=-1 /path/to/composer $1'

(obviously replacing "path/to/composer" with the full path to the server's composer binary).

That tells PHP to use no memory limits when running composer, and may work

Enable Swap

A lot of the reports of this problem come from users on low-end virtual servers from providers like Digital Ocean or Amazon EC2. These, apparently, come with no swap enabled. By enabling swap on the VPS, you can use more memory for burst tasks like this.

What if System Memory is the Problem?

Neither of these will solve the problem if the issue is a lack of system memory.

You'll observe that the problem above came when composer needed to use more than 1 GB of RAM (1073741824 bytes). That's a lot of memory just for a package manager to use.

The two tips above make sure that the server has the maximum memory available to it (enable swap), and that PHP can use as much of that memory as possible (stop limiting what PHP can use). But if your actual system has a hard memory limit, PHP will not be able to use more than the physical memory available.

I can think of a couple of scenarios where this might be the case:

• Someone installing a site on a VPS using OpenVZ (or similar) will find that swap is not something they can set up. Their provider will have allocated their VPS a certain amount of RAM, and a certain amount of "vswap". It is what it is, and you don't have control over it.
• Someone using shared hosting will almost certainly be operating within a CloudLinux LVE (lightweight virtual environment), where the amount of physical memory you can use is capped to prevent one user from impacting the performance of other users on the shared server.

In both cases, we need to note that 1 GB is a fairly hefty amount of RAM. Good shared hosts will give you up to that amount of RAM, but I've not seen anyone offer higher. For the VPS option, I've run a Drupal site on a VPS with as little as 256 MB of RAM (and with some careful tuning), so there may well be people on virtual servers where the 1 GB limit is a real one that they will hit.

So, how to overcome this?

Use a Separate Development Server

When you run Composer Install, it does two things in one.

According to the built in help (what you see if you just run the command "composer" with no arguments):

composer install = "Installs the project dependencies from the composer.lock file if present, or falls back on the composer.json." 

composer update = "Updates your dependencies to the latest version according to composer.json, and updates the composer.lock file."

So, composer.json contains an editable description of what projects you want Composer to install and maintain. Those projects will have dependencies. Composer has to do a lot of thinking as to exactly what version of every project and library is required if those dependencies are to be fulfilled. The results of that thinking get stored in a file called composer.lock.

If you run composer update, it will look at your composer.json file, work out what dependencies are required, and write that information into composer.lock. But it won't install any of them yet.

If you run composer install, it will first look to see if you have a composer.lock file. If you do, the thinking has already been done, and it will install the exact versions detailed in composer.lock. If you don't, it will look at composer.json, do the thinking, create a composer.lock, and then install the required versions.

So "composer install" first runs "composer update", and then installs the resulting packages.

It's only the first of those two steps that need all that RAM.

So if we offload "composer update" to another machine, the main web server can run with the limitation of only that 1GB of RAM, and it won't be a problem. That "other machine" could be a local desktop development computer, or another higher specified (virtual) server. It doesn't matter.

So here's my new workflow

Step 1: Download the project from git

This is pretty much as above, but we now do this in two places. On the server that will host the site, and on the development server. When we tweak composer.json to alter the deployment directory, we do so in an identical way.

Step 2: Update the project on the dev server

On the development server, run "composer update".

All the hard thinking has been done, and has been stored in the composer.lock file. So:…

Step 3: Copy the lock file to production

Use scp, rsync, ftp, it doesn't matter. Get the composer.lock file that now exists on the development server, and copy it back to the correct directory on the production server

Step 4: Run composer install

Now, on the production server, run "composer install". This should fly through — there's no thinking to do, it just has to install all the required packages.

Maintaining the site

Once you've got a vanilla Drupal 8 site up and running, there'll be other steps you need to perform using Composer. These include updating things, and installing a new module or theme from contrib.

Let's say you want to install the Chaos tools suite, a fairly likely requirement.

In theory, all you do is run:

composer require drupal/ctools

The problem is, that will run the whole dependency analysis, and so break down for lack of memory.

So, instead, run that command on the development server. You then need to copy back to the production server both composer.lock and composer.json otherwise the two will be out of sync. You can then run composer install, and all is well.

Over to you

I hope this has helped if you were looking to solve the same problem I hit.

I'm very much a composer newbie, and this is what I've managed to figure out for myself.

I'd be very grateful to have readers (with more experience than I do) correct my mistakes, pass on words of caution, or point out what I've missed. Thanks in advance!