Optimizing Magento2 php-fpm configuration parameters

In your configuration file, you usually set up the max amount of children process you allow fpm to run. If this is too high, your server will run out of memory and some process may crash (especially if you are running your DB on the same server). To avoid that, you need to properly calculate how much memory you will allow fpm to use. In this article we will go through a method to optimize these parameters.

How does the configuration file look

The file typically is created in /etc/php/fpm/pool.d/magento.conf and looks like:

[magento]
user = magento
group = www-data
listen.owner = magento
listen.group = www-data
listen = /var/run/php/php7.3-fpm-magento.sock
pm = ondemand
pm.max_children = 50
pm.process_idle_timeout = 10s
pm.max_requests = 500
chdir = /

All the parameters we want to set up in our magento fpm conf file are related to the process manager (pm).

The most important one is max_children as it will control how many child process do you allow to be created at maximum.

How do we calculate max_children

Let’s say that we have a VM with 8GB RAM, we run MariaDB with a small catalog of 100 products and we have Varnish cache configured to have 256Mb max cache storage.

Typically MariaDB will consume 1Gb (to be on the safe side), Varnish 256Mb and 256Mb more for OS processes. That leaves us with 6,5Gb of free memory.

Now we want to understand how much memory does one child consume. To do that we are first using a cache warmer script that will try to access every URL of our sitemap. This way we can look at the memory while it runs and determine memory usage of FPM. Here is the code for your bash script:

#!/bin/bash
URL='magentosite.com'

wget --quiet https://$URL/sitemap.xml --no-cache --output-document - | egrep -o "http(s?):\/\/$URL[^] \"\(\)\]*" | while read line; do
    time curl -A 'Cache Warmer' -s -L $line > /dev/null 2>&1
    echo $line
done

Immediately after running that script you can look at your fpm process and sort them by memory usage:

ps -ylC php-fpm — sort:rss

Look at the RSS column to get the value you are looking for.

Or you can run the following that will calculate the average consumption of your fpm processes:

ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"Mb") }'

A typical value is between 60 and 90Mb.

So if we follow the logic, we need to divide the available memory by let’s say 80, ie 83 max_children.

The hickup

When you start to have concurrent connections to your magento shop, your DB will start to consume more resources than it does in the example we ran above, because we don’t simulate concurrency.

It is quite difficult to give a general rule here as it will largely depend on your Mysql or MariaDB InnoDB engine parameters. You have tools such as MYSQLTuner which will give you good advices on how to tune your DB settings.

If you really want to fine tune, you can use tools like Apache JMeter from several small VMs simultaneously, and you’ll get a clearer picture of how much memory is really available for FPM to run.

So for the example above, instead of using 83 as a max children value, I lowered it down to 50.

Indications that you allow too few/many processes:

There is one very strong indication that your max_children value is too high, your DB runs out of memory and crashes randomly every 24/48hrs. The logs are not very verbose, but I systematically had this issue when setting up too high max_children value.

A typical example would be a 2Gb VM with 50 max_children. Magento 2 crashes randomly every day and the logs dont’t show much. Reducing to 5 fixed the issue.

If on the contrary you set up too few max_children, you will not notice it immediately but the browsing experience for your customers would be terrible. A good indicator is to look at the logs of php-fpm.log. If you notice the following error most of the time then you have set up a too small value:

[11-Oct-2019 10:05:24] WARNING: [pool www] server reached pm.max_children setting (35), consider raising it

Stress test your server to define the best value possible

We talked above about using a cache warmer script or/and Apache JMeter to create load on your server. There is one aspect which we haven’t considered is Magento CRON. Every minute a cron is run, and you also have the hourly and daily one. To fully stress test your server I suggest to run the cache warmer plus one JMeter small instance configured for concurrent calls, and running your daily CRON manually at the moment of the test. All together will load your DB and FPM quite much and should help you understand how your server can handle the load or if you need to fine tune FPM.

A nice monitoring tool I also use (and which is open source) is netdata. But stick to PS for a good trace of process memory consumption.

https://www.netdata.cloud/

Need help with your Magento installation, performances or strategy, I am also providing consulting and implementation services. Contact me by email v.teyssier@gmail.com