I don’t see how concurrent accesses could be the issue because each request is logged sequentially one after another and they are always at least a few microseconds apart.
I don’t know what you mean by “ban list increase” - my goal is not to increase the ban list but to decrease requests by banned IPs. My system is not complicated and seems to work well - every 5 minutes a cron job runs and counts the number of visits per each IP within the last 10 minutes. If an IP has more than 300 visits (which is one visit per two seconds on average) then it is added to the ban list to .htaccess for 24 hours. I log each request to a database table, which then makes it very easy to count visits later on, much easier and faster than parsing Apache logs.
Now back to the topic:
After closer examination of all access logs I have available I must admit I have misinterpreted the actual problem. Just after the bot is banned in .htaccess it doesn’t access the site with 200 status as I previously thought. Those 200 status entries in the access log were still before the ban took place - so nothing abnormal here. The moment the ban takes place there are no more requests from that IP either in access log nor in error log - the bot simply disappears. Because it is extremely unlikely that a bot would simply stop visiting my site just one second before it is banned I conclude that my Apache server simply does not log 403 requests from that bot at all. However, it logs 403 fine from other IPs, I even tried spoofing the UA (which is “Zend_Http_Client”) but still could not reproduce the failure to log.
So I have a different mystery to solve. Do you know what could prevent apache from saving requests to access log? I’ll be investigating this with my admin but maybe there are some obscure techniques of visiting servers without leaving any traces in access logs? Perhaps by sending special http headers or other trickery?