Problem with raw access logs

[TechnoBear]

Hope this is the right for this question.

All my sites are on shared hosting using cPanel. I have them all set to save the raw access logs, which I download at the end of every month. In September, the logs for two sites, both on the same server and the only two I have on that server, were incomplete, ending in mid-month. I contacted the hosting company, and they sorted it out. In October - same problem, same sites. I contacted the hosting company, they sorted it out. They also promised that they had fixed it so that November's logs would run to the end of the month, which they did. Come December - logs end on 24th for these two sites. I contacted the hosting company, and they say

We have investigated this issue further and found that the cPanel system has some inbuilt functionality for logs options. This is one of our shared server on which your hosting account is hosted, cPanel log rotation system automatically delete the old logs so the available disk space on server will be sufficient to use to clients. You need to download these logs in each 5 days if you wish to see full month logs. You can not download full month logs at once.

Please be sure, you will download logs of 1st to 5th of January on 5th January and 5th to 10th January on 10th January and so on on each 5 days.

Does that seem reasonable? They've never mentioned this problem before and it doesn't affect any other site I have. Also, if their system is deleting old logs, why do I have the logs for the start of the month but not the end?

[guido2004]

Also, if their system is deleting old logs, why do I have the logs for the start of the month but not the end?

That's a very good point. I'd ask them.

[Immerse]

It almost seems as though apache stops logging stuff after those dates. How's your disk usage on the server? Is there enough room for the logs to actually grow?

[TechnoBear]

Yes. Both sites have an allocation of 400MB each; one is using 126MB and the other 15MB, so I'm not pushed for space. The largest zipped archive file I have is 780KB - and that was for about a month and a half from last time they fixed the problem. I'm not exactly a power user.

[guido2004]

Like I said, I think you'll have to talk to them some more. It doesn't make sense to download your logs every 5 days, if they stop logging half way through the month. It would make sense if they would eliminate the old logs every x days to free up space, but from your explanation it appears they don't do that.

[TechnoBear]

Like I said, I think you'll have to talk to them some more.

Yes, I did - I'm just not sure we're talking the same language. First reply I got said

I went through your ticket thoroughly and just to let you know that to process the access logs or any stats cron service is must and needs to be running all the time. Being shared server normally the option "Delete each domain’s access logs after stats run" is enabled on cpanel server in order to maintain the server disk space resources under control.

But cpanel user's have options to store those logs after each stats are updated/complete and in your case I can see that "Archive logs in your home directory at the end of each stats run" is enabled and logs zip files are stored in cpaneluser/logs/ folder.

For a test I've checked the access logs of last month Dec-2011 and found that it started archiving logs couple of days before the start of Dec month (29th and 30th Nov) and ended some time in 25th Dec.

Honestly this shouldn't happen with the monthly raw access logs that includes previous months logs, that is older entries of last month gets added in the current month logs. i'll open up a ticket to cpanel support and ask them for an explanation on this log issue.

followed by

The web server does not store logs that way. Apache simply writes logs out to a text file and cpanel save that log file as it processes it each day. Where it starts and ends is determined by when stats last happen to run.

It would take extra cpu time to chop the log up by days like that, which isn't something we would want to do by default. On a server with several hundred domains this could add several minutes per domain, on a server with several thousand domains it could add up to an hour or more extra time for stat runs.

As I'm sure is obvious, I don't really understand all this. All I know is that I don't have this problem with nine other sites hosted with this company but on different servers, or with the sites I have hosted elsewhere.

Frankly, I don't care if I have an odd day or two of one month in another month's logs, my problem is with the bits that are missing altogether. For one of these sites, there is still no January 2012 archive and even the AWStats end at 25th December and I get an error message if I try to update them. To me, that looks like a problem on the server, but what do I know?

[TimIgoe]

It does't sound completely unreasonable to delete "old" logs, but normally a log rotation will keep at least 3 or 4 logs so you can always refer to the one just before it was rotated.

Personally, downloading and processing log files every month I find as a painful task - is there no way you could automate grabbing them every few days like they suggest? that way you've always got them if something goes wrong with one of the servers.

[TechnoBear]

It does't sound completely unreasonable to delete "old" logs, but normally a log rotation will keep at least 3 or 4 logs so you can always refer to the one just before it was rotated.

I agree. However, the missing information is from the later dates, not the earlier dates, which doesn't look to me like "deleting old logs". Also, the archived logs are stored on my allocated disc space, where there is ample room for them.

Personally, downloading and processing log files every month I find as a painful task - is there no way you could automate grabbing them every few days like they suggest?

I don't think so. The majority of my sites are very small - under ten pages - so the files are not huge. I have only ever had a problem with the two sites on this server.

The relevant bit of my last e-mail to the hosting company read:

The current situation is this:
The December 2011 archive for onedomain.co.uk ends on 24th December 2011. The January 2012 archive starts on 24th December 2011 and ends on 27th December 2011; there is nothing archived after that date.
The December 2011 archive for otherdomain.co.uk ends early on 25th December 2011. There is no January 2012 archive at all. Moreover, although the AWStats say they were last updated on 27th December, there are actually no records after 25th, and every time I click the "Update Now" button, I get the following error message:

Error: Log is currently being processed in the background. Cannot update.

As I said, that looks to me like a problem on the server. (Webalizer stats also stop on 25th December.) I've had this happen before and they've managed to supply the missing logs and apparently "fix" the problem, but they don't seem bothered about it this time.

Their reply is:

We've checked with cpanel team and they did mentioned the same about how cpanel works when it comes with updating stats. Depending upon the number of accounts on shared server and the traffic/visits/hits each account have and if such domains have high daily stats logs then it'll definitely take time to update the webstats.

On this shared server we've 250 shared accounts and by default the web traffic statistic update process is set to every 24 hours and bandwidth statistics update is set to every 2 hours which means all 250 accounts needs to complete the updatelog process inside 24 hours but it happens that it goes beyond 24 hours so it skips the next cycle for updating stats thus you see that some stats logs stops and then it continue when the next cron process is in action.

More importantly cron service is necessary to run/update the statistics on daily basis, along with the stats update cronjob there are few other cronjob which runs on daily basis which include cpanel update, backups, package update, cpaddons update, dnsqueue check, eximdb update and few other as well. Also whenever stats process runs on the server for individual user it checks CPU/memory and server load usage and if the server load is high (out of the threshold limit) then stats process halts for a while until the server load comes back to normal.

If webstats are important for your account then I would recommend you to go with either VPS or a Dedicated server where you can have limited number of account and the server's resources will fully utilized by your accounts and not shared by other account as it is on shared server.

Apart from the last paragraph, I don't entirely understand what they're saying.

Am I simply being unreasonable in expecting to use the in-built archive function in cPanel to archive my log files with a reasonable degree of accuracy?

[TimIgoe]

Sounds like they are trying to make excuses a little as to why it doesn't work without actually looking into it / fixing the underlying problem.

They aren't making it up about the crons, but log rotation etc should not be broken at a certain date / period through month after month - it'll either work and rotate or it won't

Its entirely possible somethign on the server is overloading the processing and causing it to not finish in time and then having a knock on effect on those sites that come after in the processing - not completely untrue but if thats the case then they need to do something about it.

[TechnoBear]

Thanks, Tim, that all makes sense to me.

Sounds like they are trying to make excuses a little as to why it doesn't work without actually looking into it / fixing the underlying problem.

That was my feeling. They advertise AWStats and raw access logs among the features of their shared hosting, so I feel they should provide them, not tell me to get a dedicated server.

I'm in the process of moving my sites away from this company anyway, but I can really only afford to move them as they fall due for renewal, so I'm just trying to keep things running in the meantime.

[TimIgoe]

Ah, yeah. Migrating sites isn't a fun task - its one i'm mid doing myself (moving from a dedicated server to a colocated one... time consuming)

[Ted French]

Hi TechnoBear,

I've just joined this forum to say quickly (because dinner's nearly ready!): you're not the only one having this problem with the Raw Access Logs on cPanel! I've been chasing up the same odd behaviour with MY host for the past week since I discovered it, and they've also been a bit flakey as to why it's doing it and what can be done to correct it (basically, nothing as far as they're concerned). They've told me it's a change of default behaviour caused by a recent cPanel update, but I'm not convinced. I've contacted Customer Support at cPanel to see what they've got to say about it. I don't have time to write more now but will try to do so over the weekend. We should compare notes! I really need my Raw Access Logs back to their full, monthly logging!

Best Wishes,

Ted

PS the bit where your host said "If webstats are important for your account then I would recommend you to go with either VPS or a Dedicated server" sounds almost identical to what MY host wrote to me today! Hmmm ... same host, I wonder? In the UK? Too?

[TechnoBear]

Hi Ted French, and welcome to SitePoint.

Sorry to hear you're having this problem (although reassured to discover it's not just me. )

I have domains hosted on four different servers with this (UK) company, and have only ever had this problem with the two domains hosted on this server, which makes me think it's a server problem rather than a cPanel problem.

The December 2011 archive for otherdomain.co.uk ends early on 25th December 2011. There is no January 2012 archive at all. Moreover, although the AWStats say they were last updated on 27th December, there are actually no records after 25th, and every time I click the "Update Now" button, I get the following error message:

Error: Log is currently being processed in the background. Cannot update.

The hosting for this domain was due for renewal in mid-January, so a few days after my last post, I moved it to another company. I did a backup through cPanel (took several attempts - I kept getting "back-up failed" messages) and asked the new hosts to upload it. Miraculously, I found I had almost-complete log files. The January backup, missing altogether on my old host, started on 29th December, so I actually only lost four days' logs. (Thanks to my new hosts, it runs until 31st January, when the February log starts. )

The December 2011 archive for onedomain.co.uk ends on 24th December 2011. The January 2012 archive starts on 24th December 2011 and ends on 27th December 2011; there is nothing archived after that date.

This was due for renewal in early February, so I also moved that to a (different) new host. Again, I did a cPanel backup and again had to make several attempts before it would complete. The January backup (downloaded from my new host) starts on 24th December and runs until 3rd January, after which there are no further entries until 26th January, and none after that. (I'm no expert, but I still think they have a wee problem there...) I moved the site on 3rd February. February's logs start on 1st February and continue up to today without a problem.

As I say, I'm no expert (or I wouldn't have needed to ask for advice here ), but if it works everywhere except one server, I'd say there's a fair chance there's a problem with the server.

Out of interest, does your hosting company have (in late February) a special offer from Santa advertised on their web site? If so, I'd bet we're talking about the same bunch.

[Ted French]

Hi Technobear, I'll have a look tomorrow - I've been sorting my new iPhone 4S this afternoon, so have been mightily distracted! I received a reply from cPanel Product Support this morning (having emailed them yesterday), and their reply certainly suggests that my host's explanation for the change in logging behaviour (that the default was changed in a recent major update - i.e it's not their problem) is not the case.

I'll (hopefully!) have time to write more tomorrow, so watch this space!

[Ted French]

Hi Technobear -

Well, no sign of Santa at my host's web site (though I gave a clue as to who they are in my first post, when I said "In the UK? Too?" - but I guess if that didn't ring any bells, maybe it's not them! )

The latest situation with me is that I'm awaiting a reply from cPanel, who were quite quick to respond when I emailed them on Friday afternoon. The relevant bit of their reply, when I relayed what my host had suggested about the default behaviour being changed in a "recent major update of cPanel", is:

"While it would be uncharacteristic of us to modify a preference just as part of an update, I am currently pursuing this internally to see if anyone is aware of any situations where this could happen."

When I passed this comment on to my host, they responded with:

Our system admins have ... verified that the setting was the server-wide default. Please let us know if you hear anything back from cPanel support regarding the issue."

I passed this back to cPanel, and that's as far as I've got so far. Am awaiting cPanel's response. Will let you know when I hear more.

[Ted French]

I've heard back from my contact at cPanel. I've passed his comments on to my host, thus:

Hello,

I've had a further reply from cPanel. They write:

*****

I have heard back from our product manager who oversees our changelog and all modifications to our software. We have not changed these settings on any servers as part of our update to version 11.32.

Keep in mind, changing a setting and changing a default are two totally different things.

Changing a default means for all brand new installations of our software, how that setting is set out-of-the-box has changed. This would only affect installations of our software on freshly deployed servers, not servers updating from a prior version of cPanel&WHM. For existing deployments of our software being updated, settings would not change regardless of what the new default is.

Changing a setting means if you had an option selected, we would unselect it or vice-versa. This is exceptionally rare and is something we prefer not to do. Your experience seems to indicate this is precisely what happened with the archive logs setting hence we feel it is worthy of investigation as this was not an intended part of our update.

Someone with root access to that server must submit a support request so we can investigate this issue to determine what caused this to happen so corrective action can be taken. Anyone else experiencing this issue should have whomever has root access to the server (typically their hosting provider) contact us as well."

*****

I've written back to say that this is all well beyond my expertise and I'm going to ask you to contact cPanel Product Support yourselves, as suggested, and raise a Ticket with them, as a matter of great urgency. Plainly, something is definitely wrong.

I note also that the cPanel software is now up to version 11.32, while the version deployed on my site's server is 11.30.6 (build 3). I would imagine it's much better to be using the latest version (perhaps there is a bug in 11.30.6 that is not in 11.32), so I've said that I'll encourage you to update as soon as possible!

I look forward to hearing that you've contacted cPanel to follow-up on this matter.

If you're still corresponding with your host about this, Technobear, you might want to request that they also contact cPanel with an official Ticket, to see what can be done?

[TechnoBear]

I've heard back from my contact at cPanel. I've passed his comments on to my host...

I'm glad you seem to be getting somewhere. Hopefully, things will be sorted out for you soon.

If you're still corresponding with your host about this, Technobear, you might want to request that they also contact cPanel with an official Ticket, to see what can be done?

Oh, I love the sound of optimism. One of the reasons I'm in the process of moving sites away from this hosting company (not the same as yours - I finally tumbled to your clue) is that they're fine until there's a problem, then they don't want to know. Basically, everything is the customer's fault/responsibility (even when their database was hacked and I received spam e-mail from their address ). I can't see them caring - and I've moved the two sites affected.

FWIW, all but one of my sites, over three different hosts, are using cPanel 11.30.6 (build 3). I don't know what the problematic sites were using, but I'd guess it was the same. I have three sites with a different company, all of which were moved recently to brand-new servers, and two of these are still using 11.30.6 (build 3). The third one is now using 11.32, although I'm sure that upgrade has only happened in the last week or so. (The log-in screen looks quite different, so I'd have noticed it straight away.)

[Ted French]

FWIW, all but one of my sites, over three different hosts, are using cPanel 11.30.6 (build 3). I don't know what the problematic sites were using, but I'd guess it was the same. I have three sites with a different company, all of which were moved recently to brand-new servers, and two of these are still using 11.30.6 (build 3). The third one is now using 11.32, although I'm sure that upgrade has only happened in the last week or so. (The log-in screen looks quite different, so I'd have noticed it straight away.)

Thanks for that info - might come in handy at some point!

My lot have been quite good, generally (though years ago I must say they were pretty dreadful, but the robustness of their servers has improved a lot in the last five years or so). I've been with them for many years now, but this is the first time I've had to enter into a protracted correspondence with them about anything as complex as this, so I really hope they don't let me down by dropping the ball and not contacting cPanel themselves. As their customer, I don't really see why I should be doing all the chasing to get this fixed!

[Ted French]

What a time I've had since I last posted here on 27 Feb!

Much correspondence has passed between me and what was my then-host (I've since moved, as you'll see). In this correspondence, my old host finally admitted that they had changed a cPanel setting and prevented the logging of archive files. In fact, they were adamant that they had done this "months ago" and they were not going to change it back. This led to me moving my web sites to another host. This move was completed in the last few days.

You can imagine my surprise when I began to discover that THEIR Raw Access Logs appeared to be behaving in the same way i.e. only the first 24 hours'-worth of logging was being stored in the archive log file for March 2012!

To cut a long story short, my correspondence with my new host soon uncovered the fact (in the space of just one email back and forth, in fact) that there is nothing wrong with the log files at all - the problem lies with WinZip, the program I've been using to unzip the files. What seems to be happening is that WinZip, in decompressing the files, only decompresses the data that was logged in the initial 24-hour period - although further chunks of 24-hour logging data are appended to the log file (it grows in size accordingly, as does the "packed size" of the compressed file within), they do not appear in the decompressed file when viewed in apps such as Winsyntax, Notepad, Word, etc!

My new host suggested I try using the free, open source app 7zip to unzip the files, which I did. Hey presto! All the data appears in all the log files - even the log files at the old host!

Why this should be the case is unclear, but my theory runs along these lines:

Probably nothing has changed at my old host, but what has definitely changed since the last time I opened any gzipped log files is:

1. I've moved from Windows XP on a 32-bit PC to Windows 7 on a 64-bit PC;

2. WinZip has gone through a few upgrades (from the 32-bit version I was using on XP), and is currently on version 16.0 (9715 - 64-bit version).

The problem must lie somewhere within those two areas. Either the current (and, perhaps recent past) version(s) of WinZip 64-bit itself has a specific bug that prevents it decompressing data appended to the initial data stored in a file of this kind when the gzip file is over-written after the first 24-hour logging period, or a combination of WinZip 64-bit running on a Windows 7 64-bit PC causes the problem.

What's so amazing is my old host's dogged insistence that they had changed a cPanel setting themselves to prevent proper logging. (It was this insistence that threw me completely off the scent and stopped me even beginning to consider some other reason.) I've even sent a seven-page letter to their MD (posted in the mail just prior to uncovering the real cause of the problem) bemoaning the fact that they admitted they'd made this change "months ago" without informing any of their customers, and demanding a pro rata refund of the annual fees I'd paid in January because they'd changed one of the fundamental functions of my hosting package without informing me!

Well. I'll be interested to see what the MD replies (I'm saying nothing in the meantime!), but I thought you would want to know that the problem, in the end, was nothing whatsoever to do with cPanel's software (or with my old host's logs, for that matter!).

I hope this info proves helpful to anyone else who encounters this problem!

Best wishes,

Ted.

[TechnoBear]

Thanks for the update Ted, and well done to you and your new hosts on tracking down the problem.

For the record, I'm running Ubuntu Linux and using Archive Manager to extract the compressed files, so that wasn't the problem here.