The first bug of the year showed up right on schedule.
A couple of days ago, the site I maintain, Hanabi Gakuen, went down. It has a lot of unrelated things running on it, so I assumed it was the usual temporary hang and rebooted it without thinking much. After the reboot, it stayed down. The database would not connect, and I had no idea why.
I asked the other person who can log into the server whether he had touched the database. It turned out he had not logged in for months. That was the point where I started to panic. I tried starting MySQL by hand, but the black terminal gave me a cold Job for mysql.service failed. The detailed error was not much help either; it only said mysql.service: Main process exited, code=exited, status=1/FAILURE, which is just a formal way of saying it failed to start. Everyone already knew that.
At that point I had to ask the previous admin for help. After whatever operation he performed, we finally found the real error: something had gone wrong with the user table in the database. He reset the user table, and MySQL finally came back up, barely.
I could probably have put the site back online with a few more configuration tweaks, but I thought about it and decided to migrate the whole thing instead. If I left it as it was, I would keep running into messy problems. A clean migration would also let me update the system environment properly.
Rebuilding from scratch
First, I rented a new server. The old one ran Ubuntu, but a year had passed, and I wanted to think about stability a bit more, so I switched to CentOS. To make configuration easier, I used the LEMP application that the hosting provider had already set up. That way I would not have to worry about installing the web server stack myself.
While the new server was being deployed, I went back to the old one and packed up the site’s database and files so they would be ready to move. The new server came up soon after, and I transferred everything with a single scp. Direct server-to-server transfers are fast; several GiB moved in just a few minutes. It was also a reminder of how much the site had grown. When I took it over, the data was under 2 GiB. Now it was already several GiB.
Next I imported the database, unpacked the site files, and configured Nginx. As expected, the site worked normally after that. Then came DNS, Cloudflare CDN, and the usual pile of settings. Fortunately, I had set up the firewall whitelist before, so that part was easier this time.
But I was not finished. Since I had a new server and had learned more about server management over the past year, I wanted to do some optimization. First I installed BBR. I did not want to deal with too many manual steps, so I used a one-click script. I also added Swap. The server only had 1 GiB of memory, so it made sense to give it some extra breathing room. It might be slower in some cases, but I preferred stability, and since the disk was an SSD, the impact should be small.
At that point I considered the basic environment done.
Sockets, ports, and logs
The next day I felt like doing more optimization. My first goal was to replace internal TCP/IP sockets with UNIX domain sockets, since that can reduce system overhead. I started with PHP-FPM.
The change itself was simple: in the PHP-FPM configuration, I changed the listen value from an IP address to a file, and in Nginx I changed fastcgi_pass to point at the same file with the unix: prefix. I thought that was all, but reality gave me a 502 Bad Gateway. Apparently it was not that simple.
I looked more carefully at the comments in the PHP-FPM configuration and found this warning:
; WARNING: If you switch to a unix socket, you have to grant your webserver user
; access to that socket by setting listen.acl_users to the webserver user.
Right. I needed to set listen.acl_users. I added the nginx user, restarted the services, and it worked.
After that I thought about optimizing MySQL. I was not sure how to make MySQL use UNIX domain sockets, so I was ready to leave it alone. Then I heard that using 127.0.0.1 in database connections can be better than localhost because it avoids one resolution step. That sounded reasonable, so I changed the database address in the site configuration.
Later I looked it up and found that MySQL treats localhost specially. With localhost, the connection goes through a UNIX domain socket; with 127.0.0.1, it goes through TCP/IP sockets as usual. That annoyed me a little, so I changed the database address back. I also decided to close port 3306. According to what I found, adding skip-networking to the configuration file would do it.
Then I checked the open ports. Besides 443 and my SSH port, there was an odd port 25. I did not like the look of it. It turned out to be postfix, so I uninstalled postfix without much ceremony.
Next I looked at the Nginx logs and noticed the size was off. The server had only just been set up, and the logs were already 400–500 MiB. Hanabi Gakuen does get a fair amount of traffic, but that still seemed excessive. The cause was that I had not excluded static resources from the logs. I adjusted the configuration file, and that made me feel much better.
With that, the optimization work on Hanabi Gakuen came to an end.
Afterthoughts
I probably should have read the logs more carefully. A bug that appears out of nowhere could be an attack, though when I went through the logs I did not see any sign of that.
I also should not have put a project that was still under development on the same server as the main site. At the very least, eggs should not all go in one basket. This time I plan to finish all the main site’s configuration and make sure it runs properly.