Questions about Hits, Entry Processes and Virtual Memory Usage

Server Config
1

Hi,

I checked my website’s resource usage this morning and noticed that my site had a temporary issue yesterday where the entry processes and virtual memory usage hit the limits. (For the record, I am on a shared server with 20 entry process and 4GB virtual memory limits.) You can see the resource usage charts from yesterday below:

480×1040 70 KB

When I checked my access logs, I noticed that there was one IP that caused lots of page requests in a single second. Below is the logs:

45.55.164.167 - - [06/Mar/2016:17:09:05 -0500] "GET / HTTP/1.1" 301 235 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:05 -0500] "GET / HTTP/1.1" 200 11285 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:13 -0500] "GET /robots.txt HTTP/1.1" 301 245 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14"
45.55.164.167 - - [06/Mar/2016:17:09:13 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-a/ HTTP/1.1" 200 5048 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-b/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-c/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-d/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-e/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-f/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-g/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-h/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-i/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-j/ HTTP/1.1" 301 245 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-k/ HTTP/1.1" 200 6414 "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-l/ HTTP/1.1" 200 12435 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-m/ HTTP/1.1" 200 19852 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-n/ HTTP/1.1" 301 245 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-o/ HTTP/1.1" 200 18041 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 - "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 - "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-p/ HTTP/1.1" 200 5170 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-q/ HTTP/1.1" 200 5802 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-r/ HTTP/1.1" 200 29505 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 5068 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-s/ HTTP/1.1" 200 19566 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 5068 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 5068 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 5068 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 5068 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /page-t/ HTTP/1.1" 200 7703 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 5068 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
45.55.164.167 - - [06/Mar/2016:17:09:14 -0500] "GET /500.php HTTP/1.1" 508 5068 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"

I did an IP lookup, no information was available other than the host of the IP range this IP belongs to. Legit search engine spiders seem to not make this many requests in a second, therefore my conclusion is that this might be a scraper bot. Either case, I don’t want my website to became unaccessible and hit my server resource limits due to this type of things. So, I need to understand better how things work and what I can do to improve my website’s resource usage and block this type of scrapers if possible.

My website is a custom coded (PHP) very light website that displays pages from a database (MySQL) like a blog. No user interaction whatsoever (registration, comments, etc.).

Here are some questions that I will appreciate if more experienced webmasters could answer (as detailed as possible):

  • What do entry processes consist of? My hosting’s help article says that when a user opens a PHP page, that counts as an entry. Do hits on resource files such as images, CSS and JS files (or other) also count as an entry process? Or only scripts that communicate with applications on the server such as the PHP module or the database server, in my case the php file that displays the page?

  • If the entry processes consist of resource file hits too, does that mean I cannot have more than 20 items on my page that will cause a hit? Or else, visitors will get a 500 error?

  • When a user opens a page on my website, let’s say that has 7 items that cause a “hit”. Do all these hits happen simultaneously or do they happen as a sequence?

  • I have a header.php, footer.php and sidebar.php files that I include in my page.php file. Does that mean when someone opens a page, that counts as 4 entry processes?

  • If I put my JS code (953 bytes) into my page file rather than putting in a separate JS file, will it have any practical advantage in terms of performance and resource usage, since it eliminates one hit?

  • I read on the help article but still don’t have a clear idea about what physical memory and virtual memory are used for. In this incident, why the virtual memory usage is so high? I mean what would cause that? My pages are just about 10-20 kb in HTML content stored in the database.

  • My page file uses 2 functions (each with a single SQL query) to get its content from the database. It seems to be very simple but I guess I can optimize my queries or maybe optimize my database tables with the help of keys/indexes? Anything you could suggest or any source I can check?

  • I learned that there are some caching plugins on WordPress which help reduce the load on the database server by serving cached versions of the pages. WordPress is also built with PHP so I am guessing that I can build my own caching mechanism. Any tips where to start? I will surely search on the web, just asking if anyone has specific experience with caching on custom coded PHP websites.

  • In a way I am asking what steps you would take to fight with this type of issues that would cause your website to be unaccessible and hit your resource limits (other than moving to a higher hosting package which I will do when my website grows larger)?

Thanks for your all your input.

2

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.