Re: php-fpm processes max out even though CPU is not that high

Hello

I was/am in similar situation where it works for a while then suddenly it
chokes with Nginx 502 for few minutes then it's fine even at higher users
per seconds than at choke time.

I discovered lately that we should put the listen.backlog = 4096 or bigger
usually the same as your sysctl somaxconn

you can set that like
echo 'net.core.somaxconn=4096' >> /etc/sysctl.conf
sysctl -p

Also use unix sockets if nginx and php-fpm are on the same box, they are
faster than TCP 127.0.0.1:9000 thing.

I changed these things like 4 days ago and I am clear of the problem so far..

Also set php-fpm slow logging, it's pretty useful I can see exact script
line and function that slow things down
request_terminate_timeout = 60s
request_slowlog_timeout = 30
slowlog = /var/log/php-fpm/www-slow.log

I have it like this


Make the php-fpm static as suggested above
Put the pm.max_requests = 500 so that it kills php threads faster and
prevent memory leaks. Better respawn than waste resources.

Also make sure mysql is not the cause, many times it is. Mine is on a
different box and never reported any issue, but I didn't manage to catch it
in the act, as it's a matter of minutes usually after 1AM (obviously :)).

Also play with timeouts, maybe increasing those for debug will pop-up the
real error in another log than php-fpm. As far as 502, don't bother
searching for it directly, it's not relevant. You can shut down mysql and
Nginx say 502. :P


---------------------------------------------------------------
Cristian Rusu
Web Developement & Electronic Publishing

======
Crilance.com
Crilance.blogspot.com


On Thu, Jan 24, 2013 at 10:54 AM, Jérôme Loyet <jerome@loyet.net> wrote:

> Hi there
>
> Static pm is definitely the choice to make for high traffic websites.
>
> To figure out the bottleneck you can also try the slow request feature
> Le 24 janv. 2013 09:37, "Alexey A. Rybak" <alexey.rybak@gmail.com> a
> écrit :
>
> Hi!
>>
>> To determine what exactly causes the problem you should figure out
>> what's going with/inside PHP workers.
>> All the suggestions made (disk - db, sessions) are reasonable, but you
>> have to know for sure.
>> AFAIR 502 as bad gateway on nginx side can be:
>> * workers just don't work at all for some reason (like fpm is not
>> started, or global shared memory crash et cetera). I don't think it's
>> your case.
>> * workers stall getting data from other resource, so all you workers
>> wait in some IO and there's no worker to serve new request.
>>
>> So assuming PHP workers just stall getting data from somewhere you
>> have to figure out exactly what operation time is increased at this
>> threshold.
>> You can either do it with manual timers and manual aggregation or
>> pinba. Possibly just looking in PHP error log can help: sometimes all
>> the workers stall just connecting to some resource and wait for a
>> connect timeout, so your PHP log will be full of corresponding errors.
>> But debugging with timers will definitely give you much more
>> information.
>>
>> Also having dedicated box for just one project "pm = ondemand" is not
>> the best choice, "static" is better.
>>
>> Hope this helps.
>>
>> On Thu, Jan 24, 2013 at 2:10 AM, Jim Hackett <jimh@exelate.com> wrote:
>> > My company is looking at a combination of PHP-FPM/APC/nginx to respond
>> to a
>> > very large number of requests, very quickly, globally. In general we
>> see a
>> > huge performance increase over lighttpd, however once we get to around
>> 200
>> > req/sec things get wacky. All of the idle processes are taken up and we
>> > start responding with 502 errors. We've increased shared memory, toyed
>> > around with nginx/php-fpm config to no avail. Until we high around 200
>> > req/sec everything looks really great, CPU levels are fine, memory is
>> fine,
>> > but once we hit that mark we get a lot of socket errors in the nginx
>> error
>> > log and many many 502 error. Any help would be GREATLY appreciated.
>> Below I
>> > have included some of our config, please let me know what else you need:
>> >
>> > php-fpm.conf
>> >
>> > [global]
>> > pid = /var/run/php-fpm/php-fpm.pid
>> > error_log = log/php-fpm_error.log
>> > log_level = debug
>> > ;emergency_restart_threshold = 0
>> > ;emergency_restart_interval = 0
>> > ;process_control_timeout = 0
>> >
>> > process.max = 500
>> >
>> > include=/etc/php-fpm.d/*.conf
>> >
>> > [www]
>> >
>> > user = nobody
>> > group = nobody
>> > listen = /tmp/pool1.socket
>> > listen.allowed_clients = 127.0.0.1
>> > slowlog = /var/log/php-fpm/www-slow.log
>> > request_slowlog_timeout = 1
>> > catch_workers_output = yes
>> > php_admin_value[error_log] = /var/log/php-fpm/www-error.log
>> > php_admin_flag[log_errors] = on
>> > rlimit_files = 50000
>> > request_terminate_timeout = 30s
>> >
>> > pm = ondemand
>> > pm.max_children = 300
>> > pm.process_idle_timeout = 2s
>> > pm.max_requests = 5000
>> >
>> > pm.status_path = /monitor/status.fpm
>> > listen.backlog=0
>> >
>> > nginx.conf
>> >
>> > user nobody nobody;
>> > worker_processes 10;
>> > pid /var/run/nginx.pid;
>> >
>> > events {
>> > worker_connections 1024;
>> > }
>> >
>> >
>> > http {
>> > include mime.types;
>> > default_type application/octet-stream;
>> > index index.html index.php;
>> > log_format main '$remote_addr - $remote_user [$time_local]
>> > "$request" '
>> > '$status $body_bytes_sent "$http_referer" '
>> > '"$http_user_agent" "$http_x_forwarded_for"
>> > $request_time';
>> > error_log /var/log/nginx/error.log;
>> > access_log off;
>> > sendfile on;
>> >
>> > keepalive_timeout 0s;
>> > fastcgi_read_timeout 360s;
>> >
>> > upstream phpfarm {
>> > server unix:/tmp/pool1.socket weight=100 max_fails=5 fail_timeout=5;
>> > }
>> >
>> > server {
>> > listen 80;
>> > listen 443 default_server ssl;
>> > root /srv/www/;
>> > fastcgi_busy_buffers_size 256k;
>> > fastcgi_buffers 4 256k;
>> > fastcgi_buffer_size 128k;
>> > fastcgi_temp_file_write_size 256k;
>> > proxy_buffering off;
>> > tcp_nopush on;
>> > tcp_nodelay on;
>> > auth_digest_user_file user.passwd;
>> > auth_digest_expires 300s;
>> > location /monitor {
>> > auth_digest 'Authorized users only';
>> > location /monitor/status {
>> > extended_status on;
>> > access_log off;
>> > allow all;
>> > }
>> > location /monitor/status.fpm {
>> > fastcgi_pass phpfarm;
>> > fastcgi_param SCRIPT_FILENAME
>> $document_root$fastcgi_script_name;
>> > fastcgi_param PATH_INFO $fastcgi_script_name;
>> > include fastcgi_params;
>> > }
>> > location /monitor/status/php-fpm {
>> > alias /usr/share/fpm/;
>> > allow all;
>> > }
>> > location ~ \.php$ {
>> > try_files $uri =404;
>> > fastcgi_pass phpfarm;
>> > fastcgi_param SCRIPT_FILENAME
>> $document_root$fastcgi_script_name;
>> > fastcgi_param PATH_INFO $fastcgi_script_name;
>> > include fastcgi_params;
>> > }
>> > }
>> >
>> > location ~ \.php$ {
>> > try_files $uri =404;
>> > fastcgi_pass phpfarm;
>> > fastcgi_param SCRIPT_FILENAME
>> $document_root$fastcgi_script_name;
>> > fastcgi_param PATH_INFO $fastcgi_script_name;
>> > include fastcgi_params;
>> > }
>> >
>> > location = /favicon.ico {
>> > return 204;
>> > access_log off;
>> > log_not_found off;
>> > }
>> > }
>> > }
>> >
>> > --
>> >
>> >
>> >
>>
>>
>> --
>>
>> wbr,
>> Alexey Rybak
>> Badoo Development (badoo.com)
>>
>> --
>>
>>
>>
>> --
>
>
>
>

--