I woke up on Jul 23, an ordinary day. I checked to see if my favorite games had any updates (I had seen a new item announced), and then I went to upload a YouTube video I've been wanting to upload for a while.
Then I remembered:
So, like any sysadmin, I logged onto the server to make sure everything was okay. I logged on, it worked. The server was up! uptime. The server was rebooted and not just "paused" like QEMU can.
Then I tried my website. Connection refused. I checked nginx. Code 1. journalctl -xe. No nginx logs lately. service nginx start then journalctl -xe. Something's listening on 80. !lsof from that weird time I needed it last. Then adjust to port 80. Lighttpd was listening on port 80! systemctl disable lighttpd. systemctl stop lighttpd.
Then 443. I had forgotten to pm2 save when I decommissioned my old Discord bot which ran its own HTTPS server. That started back up and used 443, but then refused connections. pm2 rm nanobot. Nope, need pm2 del nanobot. Better. Now only NanoBot 2 is running. pm2 save.
service nginx start. Profit!
docker ps -a. VPN is stopped. I should set a --restart always. docker update --restart always gallant_feynman. docker start gallant_feynman. 1194 is already bound. systemctl disable openvpn. systemctl stop openvpn. Okay, now test? docker start gallant_feynman. It works!
pm2 ls. Most things are working, but some things are infinitely restarting. Okay, that http interceptor can go. pm2 del it. Now, Last Seen is infinitely restarting... What is running it? Oh, of course! Systemd and pm2 are fighting. systemctl disable lastseen && service lastseen stop. Now I can start it in pm2. pm2 start lastseen. pm2 ls a few times to make sure it's not restarting and it's not. Good!
Now I wait for the messages to come in telling me what's broken... :(
The lesson of this story is to reboot frequently and be capable of rebooting. I should have been rebooting regularly for updates. It had not been rebooted in more than 400 days!
Thanks for reading.