+- Jiggmin's Village (https://jiggmin2.com/forums)
+-- Forum: Mt. Olympus (https://jiggmin2.com/forums/forumdisplay.php?fid=11)
+--- Forum: Announcements (https://jiggmin2.com/forums/forumdisplay.php?fid=16)
+--- Thread: A tale about loss, redemption, and reflection (/showthread.php?tid=5239)
A tale about loss, redemption, and reflection - bls1999 - 31st December 2025
Two days ago, a slightly more naive @bls1999 set out to fix PR2 verification on the JV Discord bot. In order to do this, I needed to update the framework version I was using. No problem.
When trying to update the framework version, I learned that it isn't supported on the version of node I was using. No problem, I'll just update node.
When trying to update node, I learned that it isn't supported on the version of Linux the server hosting JV is running. It also just so happens that the version of Linux the server is running is 7 years old, had EOL'd 3.5 years ago, and had been surpassed by 4 major OS release versions since then.
Mistake number zero:
- What I should've done: regular updates.
- What I did: ignored OS updates for 5 years.
Mistake number one:
- What I should've done: stop, back everything up (including taking a snapshot with my hosting provider), prepare for the update, announce downtime.
- What I did: "Who needs a backup anyway? We're due for an upgrade anyway."
(I did back up the databases. This is important later.)
The first version upgrade goes somewhat well, but JV is now inaccessible via the internet.
Mistake number two:
- What I should've done: stop, fixed the issue, backed everything up.
- What I did: "That's fine, right? I'll just do all the updates and then fix everything."
The second version upgrade bricked SSH. For those who don't know, SSH is the conventional, main method of accessing your server. Without this, you can't get into the server to see any of the files or run any commands.
And remember the database update I did earlier? Yeah, I exported that to the local filesystem on the server, meaning I couldn't access that data either.
"****." What had started as a small fix had spiraled into a potential data loss scenario.
Okay, time for damage control. Turns out my hosting provider has something a fail-safe mode of sorts, where it boots another OS onto the server, sends you a password, and you're able to access all of the files. Excellent! From here, I downloaded the entire server partition onto my local machine. I decided that, since I had all the data and files, the best route would be to wipe the OS and restore the data. That way, I wouldn't have to deal with fixing broken things on a 5-year-old OS, I could just start fresh.
I did exactly that, installed and configured the latest versions of webserver/php/db, and went to jiggmin2.com. Nothing. Turns out, the firewall was misconfigured. I fixed that. Went to jiggmin2.com, nothing. Webserver was misconfigured. Fixed that and went to jiggmin2.com, got a MyBB error saying it couldn't load the database plugin. Turns out, the file permissions were wrong. Fixed those, went to jiggmin2.com; I was greeted by a mess of errors on the homepage and the forums not even loading.
"****." Even though the data was all there, nothing was working.
Through hours of painstaking trial and error, I figured out that not only were my webserver/SSL settings still misconfigured, the PHP version I had installed wasn't supported by some of the MyBB plugins I'd been using.
Mistake number three:
- What I should've done: checked to make sure everything was compatible with the latest versions of the webserver/PHP/db software.
- What I did: downloaded the latest versions, because new software is best software right? just look at windows 11
I downgraded PHP, fixed the webserver/SSL configuration errors, and we have the place up and running again.
So, morals of the story:
- Do software updates incrementally so each one isn't a huge leap
- Always back up critical services before performing major updates
- Warn stakeholders (you!) about potential downtime before maintenance begins
- Use a hosting provider that has a fail-safe mode
- Make mistakes in a lower-stakes environment
All of this being said, most services are back up and running; however, there are some things that still need attention:
- Discord bots are still down (lol)
- All multiplayer game servers hosted on the site are down (e.g., Platform Racing)
- Highscores databases haven't been re-created, so submission systems and the viewing page will be broken
- 2FA for forum logins is currently disabled (when enabled, it should be restored exactly as it was)
- Unclear if there are any fun MyBB things lurking in the shadows
I believe everything else is fixed. Please LMK if you find any other issues, though!
RE: A tale about loss, redemption, and reflection - bls1999 - 2nd January 2026
Update:
- The discord bot works now (+ PR2 verification is fixed)!
- The multiplayer game servers are back up!
- Highscores pages and submissions are back up!
- 2FA is re-enabled!
- I think I got most of the MyBB shenanigans... but there are always fun things to discover. 🙃
If you find any issues, please let me know!
RE: A tale about loss, redemption, and reflection - Adulock77 - 30th January 2026
SUPER late on the reply of this but yikes! that whole situation sounded like a heart attack on top of a heart attack on top of another heart attack that came with... a... cardiac arrest O_O
Glad you were able to figure it out by the end. This website must continue to exist! i cannot live without looking at the old PR1 CHAT with all the profanity that used to be there always and forever! ðŸ˜
and dem bubbles