Issues with the forum

[SkulleateR]

though it seems to be running fine now for me.

"Worked" fine until today morning, now back on endless loading times 

[Gryzor]

Can confirm, it's a bit erratic... 

[SerErris]

Hey guys, sorry for the late reply - my little one was sick for a few days and then it was his birthday, so I didn't have much time...

Though I *was* monitoring the server and it does this even on a very light load. If it's the same as last time then they may have changed something again with the way the certificates work; though it seems to be running fine now for me.

No worries, take your time.

It is hit and miss. At times it is very fast and at others it seems to not work at all. If it is sluggish it is sluggish on all computers/iPads/Phones whatever I throw at it. Even different network (mobile) with different IP etc. 

[darkhalf]

Also to add to this, it is very intermittent (but seems to be more frequent in the last few weeks). Not limited to just the forum. For example https://www.cpcwiki.eu/index.php/M4_Board page came up with an error the first time, but then refresh the page and it was okay (just now)

[eto]

Any update regarding this? I have the impression it gets worse. It sometimes takes minutes to get into the forum and check for updates. Just today uploading a 75KB attachment failed due to time-outs.

Btw: If this at any point crashes finally, is there a backup of all the content?

[Maniac]

I've found the forum to be quite unusable in recent weeks which has caused me to engage less with it than I normally would do. @Gryzor is there anything that we can help with to improve this please? There was talk of migrating to a different platform a while ago.

[Shaun M. Neary]

Sorry to add to this, but just confirming that the forum is running at the speed of an arthritic slug on valium for whatever reason.

Bogey plugin?

[Maniac]

Ironically today since I posted my message this morning, it's been a very speedy speedy thing for me! 

[eto]

Ironically today since I posted my message this morning, it's been a very speedy speedy thing for me! 

sometimes during the day speed is acceptable. 

I just checked the network activity during a request. What is outstanding here is: all requests are sequential. Nothing was received in parallel. When I look at other sites, e.g. cpcwiki.de, there are 8 threads in parallel which are all loading resources at the same time. That's the expected behavior for a website. 

with my limited understanding of servers, I would assume, there is some issue with the number of parallel requests to the Wiki. 

[Gryzor]

Apologies for the radio silence, been ill for a few weeks now with infection after infection, leading to a nice little bronchitis which has torn me to shreds. I may have some news tonight if I manage to croak something that sounds like speech, after a call with the backend provider.

@eto I don't think it's that, because it's the first/initial request that times out - if that gets through, everything is pretty fast; but just in case, which tool did you use? 

[eto]

Apologies for the radio silence, been ill for a few weeks now with infection after infection, leading to a nice little bronchitis which has torn me to shreds. I may have some news tonight if I manage to croak something that sounds like speech, after a call with the backend provider.

@eto I don't think it's that, because it's the first/initial request that times out - if that gets through, everything is pretty fast; but just in case, which tool did you use?

That's not my experience. If the first request times out I need to reload. Then, sometimes the request is answered and then the side is loaded - but not always fast. At the moment e.g. it's quite slow, taking over 10 seconds to load and render. 

This would (imho) match to a scenario, where the server does not have enough threads available to answer requests in parallel.

1) I will see sequential downloads as the other threads are blocked by other users.
2) in case there are not enough threads available, my request times out. Once another user has downloaded his page, the thread is free to answer my request.

Actually - I just tried again and find attached the timeline of requesting the same page. In one case I saw 2 threads in parallel. The page load time was 8 seconds after the first request started downloading. With only a single thread (the same page requested just a few seconds later) it takes 13 seconds until the page is finished loading. 

In both cases it took quite some time until it started. The second try took over 30 seconds until download started - so all in all almost a minute until the page has fully rendered. 

Tool used: Chrome developer tools

PS: Uploading a 31KB attachment to this post failed several times.

[eto]

just to throw it in: did you consider a DOS attack? Depending on the size of your server, an attacker that just requests (even little stuff) very often and in parallel could consume too many threads. Could also be a crawler. 

[Maniac]

Apologies for the radio silence, been ill for a few weeks now with infection after infection, leading to a nice little bronchitis which has torn me to shreds. I may have some news tonight if I manage to croak something that sounds like speech, after a call with the backend provider.

Sorry to hear you haven't been feeling well. I hope you feel better soon.

[eto]

btw: this is no complaint, I'm just curious what happens and wanted to share what I experience, with the hope that i helps. 

Get well soon - that's much more important than a slow forum.

[eto]

btw: right now it is perfectly fast. I checked in DevTools again and currently it loads resources in parallel threads.

[eto]

And now slow again - and again just a single thread. 

[eto]

And now fast again...


fyi: I always measured the exact same page

[Gryzor]

So...

The thing with the first request is that it does the SSL handshake. From my understanding, there's no parallel anything here - if this doesn't happen, then nothing can continue. Once connection is established the rest flow easily.

There have been times that indeed the site might be slow - it is, after all, a pretty old server by now so at times it may handle things a little slow. But for me, once connection is established (and it doesn't time out), the rest always loads pretty quickly. (not sure - can you see the last line at the footer describing how long it took to create the page? Or is it only available to admins?). But I'll take your word for it, I can see how some times it can be slow.

The idea about an attack is not a bad one. I must research how to detect one - unless you got any ideas?

For the time being I have updated the blocklist for Asian IPs to try and prevent Chinese and other bad-behaving crawlers. I also threw some Latin America IP ranges in for good measure.

Ok, as for the server: yes, backups are kept continuously. Last night I managed to have a talk with our server co-admin. Looks like the new server which has been standing ready for quite some time now is already ripe for some updating  But what we will do is create a subdomain, copy all data to it and try how things go with our current platforms (SMF/mediawiki) and the new environment. If all goes well (the forum should be ok mostly, but the wiki requires two migration steps to reach current version) we may complete the transfer. Anyone willing to test the new environment?

PS @eto , thanks for the tip about the Dev Tools but my view is different - can you list the steps you took to produce these screenshots?

[eto]

Anyone willing to test the new environment?

PS @eto , thanks for the tip about the Dev Tools but my view is different - can you list the steps you took to produce these screenshots?

1) open an example page. I always used this page for consistency:
2) open DevTools and click the network tab. Clear tab if any data is there
3) hit reload on the page and see what's happening


I also totally agree that the SSL handshake can only have a single thread. Afaik a server has a maximum of threads it can handle. If all of those are blocked already, the initial request cannot happen and we get the timeout. The tests I made always had the same pattern: if page load times are good then the requests after the handshake are always parallel. If the handshake was already slow then always subsequent requests are serial or max. two parallel threads. This is of course not a clear proof that threads are an issue but to me it sounds logical that it could be related to the issue or lead to the underlying reason.


btw: at the moment it's perfectly fast - and again parallel threads.

[Gryzor]

Anyone willing to test the new environment?

PS @eto , thanks for the tip about the Dev Tools but my view is different - can you list the steps you took to produce these screenshots?

1) open an example page. I always used this page for consitency:
2) open DevTools and click the network tab. Clear tab if any data is there
3) hit reload on the page and see what's happening


btw: at the moment it's perfectly fast

Damnit I hat hit Performance instead of Network

Yes, it's faster now, let's see if it holds. I also discovered a number of connections from Singapore, will try to block those IPs too.

[andycadley]

Anyone willing to test the new environment?

PS @eto , thanks for the tip about the Dev Tools but my view is different - can you list the steps you took to produce these screenshots?

1) open an example page. I always used this page for consistency:
2) open DevTools and click the network tab. Clear tab if any data is there
3) hit reload on the page and see what's happening


I also totally agree that the SSL handshake can only have a single thread. Afaik a server has a maximum of threads it can handle. If all of those are blocked already, the initial request cannot happen and we get the timeout. The tests I made always had the same pattern: if page load times are good then the requests after the handshake are always parallel. If the handshake was already slow then always subsequent requests are serial or max. two parallel threads. This is of course not a clear proof that threads are an issue but to me it looks like something that sounds logical.


btw: at the moment it's perfectly fast - and again parallel threads.

Those are client side threads, not server side. If data is being served up slowly a single browser thread can easily keep up, if data is coming down at a more normal rate then the browser will spin up more threads to process it all.

[Gryzor]

So, question for those who are familiar with htaccess files:

I have added the following:

Code Select Expand

Require not ip 47.128.0.0/16
because we have tons of connections from those IPs (Singapore). Yet, the connections are still there... Am I missing something? (Yes of course I restarted Apache).

[Shaun M. Neary]

Zipping along now! More of this please 

[cwpab]

So, question for those who are familiar with htaccess files:

I have added the following:

Code Select Expand

Require not ip 47.128.0.0/16
because we have tons of connections from those IPs (Singapore). Yet, the connections are still there... Am I missing something? (Yes of course I restarted Apache).

I just checked my IP ( https://www.myip.com/ ) and it wasn't me!

Seriously, this could be some kind of attack. Sometimes they use other computers, so this is why I checked. Someone in my family inadvertedly added the computer to a bot network of China to launch cyberattacks or something just because SOPcast or a similar program was installed to watch football.

I also googled the IP you pasted and indeed, it seems some kind of bot attack called Bytespider: 

from r/flask

https://www.reddit.com/r/flask/comments/161fqml/what_to_do_about_bytespider/

[reidrac]

So, question for those who are familiar with htaccess files:

I have added the following:

Code Select Expand

Require not ip 47.128.0.0/16
because we have tons of connections from those IPs (Singapore). Yet, the connections are still there... Am I missing something? (Yes of course I restarted Apache).

I'm guessing; if you are filtering at app level (Apache), the operating system still accepts the request.

You may need to use iptables instead to drop those connections.