Server's apache and multiple services fail to load, can't figure out why

[schwim]

Hi there everyone!

Debian 10

Recently, my server’s websites failed to load. I did a bit of digging and found that many of the systems are failing to load on the machine. I can still connect via SSH and my SSHFS still load but practically everything else does not.

Trying to force an apache2 restart:

root@adolf:/var/log# systemctl status apache2.service
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: failed (Result: resources)
Docs: https://httpd.apache.org/docs/2.4/

Dec 07 20:54:16 adolf.schwimserver3.com systemd[1]: apache2.service: Failed to run 'start' task: Read-only file system
Dec 07 20:54:16 adolf.schwimserver3.com systemd[1]: apache2.service: Failed with result 'resources'.
Dec 07 20:54:16 adolf.schwimserver3.com systemd[1]: Failed to start The Apache HTTP Server.
Dec 07 20:54:29 adolf.schwimserver3.com systemd[1]: apache2.service: Failed to run 'start' task: Read-only file system
Dec 07 20:54:29 adolf.schwimserver3.com systemd[1]: apache2.service: Failed with result 'resources'.
Dec 07 20:54:29 adolf.schwimserver3.com systemd[1]: Failed to start The Apache HTTP Server.
root@adolf:/var/log# journalctl -xe
Dec 07 20:55:04 adolf.schwimserver3.com mailmanctl[10590]: KeyError: 'getgrnam(): name not found: list'
Dec 07 20:55:04 adolf.schwimserver3.com systemd[1]: mailman.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit mailman.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Dec 07 20:55:04 adolf.schwimserver3.com systemd[1]: mailman.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit mailman.service has entered the 'failed' state with result 'exit-code'.
Dec 07 20:55:04 adolf.schwimserver3.com systemd[1]: Failed to start Mailman Master Queue Runner.
-- Subject: A start job for unit mailman.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit mailman.service has finished with a failure.
--
-- The job identifier is 302448 and the job result is failed.

root@adolf:/var/log#

My syslog can be found here: https://schw.im/syslog.txt . It’s shows thousands of clamav related failures but I am failing at finding something else to point to my issue.

Any information or pointers on what I should do to troubleshoot this issue would be most welcome.

Thanks for your time!

I’m at a loss and don’t know what else to look at.

[rpkamp]

@schwim

Dec 07 20:54:16 adolf.schwimserver3.com systemd[1]: apache2.service: Failed to run 'start' task: Read-only file system

This line makes me suspect that the system could not mount / properly and has fallen back to read-only mode mounting instead.

You should be able to see this in the output of the mount command.

[m_hutley]

either that or someone’s been playing with the user table or file permissions (accidentally mass-set a recursive permission on / ?) and suddenly the apache user cant write to its own directories…

[schwim]

Trying to write a file to the filesystem shows that it’s read-only:

root@adolf:/var/log# touch /test.txt
touch: cannot touch ‘/test.txt’: Read-only file system

I wanted to look at /etc/fstab and found this:

/var/log/ispconfig/httpd/dogwoodthicket.com /var/www/clients/client1/web11/log none bind,nobootwait 0 0

That dogwoodthicket path is the last site I added to a web hosting control panel and doing that somehow wiped my fstab, there are no other lines in that file.

So I need to restore my fstab but have never written one from scratch. How do I go about restoring the mount lines in fstab? I have this from my tests, will this suffice in getting me the mount lines I need?:

[m_hutley]

filesystem - How to fix “sudo: unable to open … Read-only file system”? - Ask Ubuntu

Have you seen this question over on askubuntu (yes i know it’s not debian, but.)

[schwim]

Thanks very much for the link. I ended up borrowing the mount line from my desktop debian install and I have a writable file system again. Unfortunately all my processes are failing to start: https://schw.im/syslog.txt

That file shows all of them but as an example, here’s mariadb:

Dec    8 09:03:39 adolf systemd[473]: mariadb.service: Failed to determine group credentials: No such process
Dec    8 09:03:39 adolf systemd[473]: mariadb.service: Failed at step GROUP spawning /usr/bin/install: No such process
Dec    8 09:03:39 adolf systemd[1]: mariadb.service: Control process exited, code=exited, status=216/GROUP
Dec    8 09:03:39 adolf systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec    8 09:03:39 adolf systemd[1]: Failed to start MariaDB 10.3.22 database server.

Failed to determine group credentials: No such process
Failed at step GROUP spawning /usr/bin/install: No such process

I’m googling but if anyone has any insight on where to start with solving this poopstorm, I’d be so appreciative