I have my production drupal website running on EC2. Recently we noticed that our EC2 CPU utilization is hitting 100%. I checked my site traffic, not much users are there. I checked the process running in EC2 using the top command. I saw one command jbd2_sda1-8. I couldn't understand what is the command is used for. I saw the command was invoked by user www-data and it is showing 200% as CPU usage for that command. As far as I know, www-data means the command was invoked by some application running in my machine. I am using PHP-7.1 and Nginx for my drupal site. I ran the command sudo service php7.1-fpm restart. Then I checked the processes. The process got killed. So I assume, the command was invoked by some php process. I checked EC2 monitoring, my CPU usage went down to 3%.
Everything seemed fine for around an hour, suddenly I again got alert from AWS saying high CPU usage. I did the same process to debug. This time I saw some different command kjournald. This process is taking high CPU, that also from user www-data.
I am confused, I tried to find out the meaning of this command, I didn't understand anything, and I didn't find any relation between the old process and new process.
This issue is keep happening. If I restart the php, the process will get killed and after some time it will re-appear again. I don't know what is the issue.
My trouble shoot experiments:
- I copied the code and placed it in my staging environment, in that machine, I am not facing the issue (Since it is staging, no traffic at all). I tried to map, the production DNS to staging, since it was not showing any issue, but after DNS mapping, same issue occurred in the second environment also. First instance became normal.
- I enabled access monitoring in nginx and checked any un-usual requests coming at the time of 100% CPU(To check if my site was under some attack). I couldn't find any suspicious request at the time of CPU spike.
- I enabled access monitoring in ELB and checked the requests using AWS athena. Couldn't find anything there also.
I am stuck here. No idea what is going on. Can anybody help?