About the downtime

[Gryzor]

Hello guys!


I apologise for the downtime. Getting the system up and running again took longer than expected, so let me give you the story.


The problem
On Saturday our site certificate expired. This should normally not be a problem because Letsencrypt/certbot was set up to auto-renew. Alarm bells went off when this didn't happen and some of you landed on the site to be greeted with stern browser messages about hackers and Elbonians stealing your data.


So, ok, let's renew manually, that's easy, right? Well yes, if you have Letsenrypt installed. In our case, it was gone. Turns out, the OS update we did a while back did not agree with Letsencrypt and it was removed.


So, ok, let's update the server, right? Right. And that's what was done, down to the kernel. Except the new OS version came with PHP7, and our forum version did not support PHP7 (in fact that was one of the reasons we had stayed back a few versions).


So, ok, let's update the forum software! Have done this dozens of times over the years, this should be a breeze (actually started doing it on my mobile during a grocery run). But no! It shouldn't be that easy, fate decided, and a key file was corrupted. Tracking down that issue alone (which was giving an Error 500 when visiting) took me a whole day. Once understood, it was an easy fix, so last night I finished the installation and put the forum in working mode this morning.


The status
We are now on the latest stable forum software version. Because that was a large update, most of the mods have been removed. Only those that rely on hooks survived, those that depended on changing code, or the manual code changes I had done, are gone. That was inevitable because the installation replaced all modified files with the official ones. More on this later in this post.


A major pain point at the moment: if you look at your browser's address bar you'll see a not-so-friendly URL. Pretty URLs was a mod we had running that provides human-readable URLs, and it'll have to be reinstalled. Not a huge issue by itself, but it means that all URLs that are out there (bookmarks, links from other sites, Google results) give a 404. So that'a  priority.


As a bonus, though, we can now update the wiki software, too. That side is also stale because newer mediawiki versions required more up-to-date platforms. It will take some time because I'll have to be very careful and set up basic functionality like email, but we'll get there.


The Future
Since the discussion we had some time ago about changing platforms did not bear any fruits, we're staying with SMF.


To minimize disruption from OS updates in the future we're looking into making a docker container out of the platform. This comes with its own set of problems but has very specific benefits in managing the forum.


Also: the 2.0 branch of SMF is, by all measures, pretty old-fashioned. There's a 2.1 in the horizon which is much more modern (and has a proper mobile version), which is now at RC4 and is nearing release. Now, we could jump onto it because it looks like it's pretty useable, *but* going from a RC to the stable release later on would cause the loss of any mods, again. So I'm trying to decide what to do...


Current issues/mods
As I said, most mods are missing. Because we have amassed quite a few modifications over the years I'm not sure I remember all of them. Please let me know of any functionality you're missing and I'll put it in the list below along with a status.


Thanks for listening,
T


Mods list (done ones with green)

   
• Pretty URLs
• Likes - won't do as it affects performance in a big way. Will wait for the next release of SMF which includes such functionality as core.
• Mobile theme
• Mentions
• Flags
• Media embedding
• Unread posts (https://www.cpcwiki.eu/forum/unread/?all;start=0)
• Donations
• Inline attachments

[pelrun]

*goes to like the post* Oh wait... 

[Gryzor]

*goes to like the post* Oh wait... 

Oh right.

[luckpro]

Don't worry, these things happen. Thank you very much Gryzor.

[thomas]

Welcome to the Upgrade Hell ... thank you Gryzor!

[Gryzor]

Likes reinstalled. Worth saying that the upcoming smf version has such functionality built into it so I'm not sure what will happen then.

[ervin]

All good Gryzor, great to have you and the wiki back!
I have to admit I did panic a little when I found the website dead.

Where is the donations button?
I feel like throwing you some moneys after what you went through!

[Gryzor]

Likes removed from now, causing issues with server performance; will revisit.

Donations mod must be reapplied, but no need really

[reidrac]

What an adventure!

The forum takes a lot to load now, at least for me.

e.g. this post took 8 secs to get the response. Then the modify was ~500ms. Then the "save" operation 39 seconds.

Just FYI.

[Gryzor]

Well, I think that now the forum is faster than ever. URLs have returned too, so if you have any bookmarks feel free to try them out!

[Gryzor]

Testing embeds

https://twitter.com/krakout/status/1442731093342539777

[Gryzor]

Testing mentions. Paging dr @Bryce ...

[Gryzor]

Ok, some mods installed. I've had enough for now, if you run into any issues let me know

[eto]

Mobile theme

Not sure if that is really a good idea to get it back. Imho it made more problems than it helped. Without it, I can at least fully access the forum with my mobile phone. With the mobile theme I could hardly use it at all.

[pelrun]

Donations are back! *kaching* 

[SkulleateR]

I think mobile Theme would be nice, right now I got massive differences in text size, buttons below are so small I cannot even read whats on there ...

[eto]

I think mobile Theme would be nice, right now I got massive differences in text size, buttons below are so small I cannot even read whats on there ...

you can zoom in and out. With the mobile theme, depending on your exact resolution it could happen that elements were completely hidden.

It's a bit annoying to zoom in, but at least you get what you want. But if the element is simply not there, you are doomed.

[Gryzor]

Mobile view IMO, though lacking in features, offered a pretty nice mobile experience. Sure, every now and then I'd want to use a feature that was missing but basic functionality was there... so you were doing maybe 90% of the stuff you usually do at 100% instead of 100% of features at a 50% ease of use.

Yes, donations are back, thanks so much @pelrun !! Much, much appreciated

Something I forgot to mention:
At no point was the database at risk. All files had been specifically backed up before I started tinkering with the site (note to self: delete the backup, that's several 10s of GB on the disk), and both files and the db are on a daily rsync job and transferred to an external server as redundancy, with a rotation of several days' worth of backups. As a matter of fact when it became clear a file was corrupted I just pulled the latest backup and fixed it in seconds.


This, just in case anyone was worried.

[Gryzor]

Now let me ask a question.

The reason it wasn't fixed a bit sooner and indeed the reason why nowadays it takes me a bit more time to do stuff on the server (well, apart from the toddler that climbs on me the moment I enter our home), is that the best times to do such stuff is at work: I have lots of hours in front of a PC so taking a little break here and there is the best opportunity to work on the server.

However, at my current job the wise IT stuff have decided to close all "non-standard" ports, which means that SSH at port 22 (or indeed any other random port) is a no-go. Any ideas how I could access the server with such a restriction in place? Cheers!

[pelrun]

What the hell do they consider "standard", then? Lemme guess, only 80 and 443, because anything that isn't web browsing must be hacking 
Nobody said you had to send only HTTP over port 80 though...  https://daniel.haxx.se/docs/sshproxy.html

[Gryzor]

Yeah that's their logic, naturally.

But if we do it on 80 then what happens with the web server's 80?

[Skunkfish]

Great to see the site back up! And thanks for all the work that you put in Gryzor on keeping this thing afloat, we all appreciate it! 

[Animalgril987]

Great to see the site back up! And thanks for all the work that you put in Gryzor on keeping this thing afloat, we all appreciate it! 

Agreed

[reidrac]

Well, I think that now the forum is faster than ever. URLs have returned too, so if you have any bookmarks feel free to try them out!

It is, indeed! Whatever was happening, it is gone now!

[lmimmfn]

Called it on TLS cert issue(deal with that crap regularly lol), great to see the site back, great job @Gryzor