Author Topic: Why does this site grind to a halt randomly?  (Read 3970 times)

0 Members and 1 Guest are viewing this topic.

Offline Bignumbas

  • CPC664
  • ***
  • Posts: 128
  • Country: au
  • If you break it you get to keep both halves.
  • Liked: 55
  • Likes Given: 6
Why does this site grind to a halt randomly?
Especially when I want to post or message.
Are you Europeans punishing us antipodeans?
We have faster internet down under now but its useless on this site when it decides to stop.
100mbits of nada!
The cutting edge of technology is covered in blood!

Offline Dagger

  • CPC664
  • ***
  • Posts: 122
  • Country: gb
  • Liked: 41
  • Likes Given: 29
Re: Why does this site grind to halt randomly?
« Reply #1 on: 11:53, 23 May 17 »
It just happens now and then. Does not normally last long so shouldn't stop you enjoying the site
Life's a bitch. You marry one or two then you die !

Offline pelrun

  • Supporter
  • 6128 Plus
  • *
  • Posts: 579
  • Country: au
    • index.php?action=treasury
  • Liked: 296
  • Likes Given: 163
Re: Why does this site grind to a halt randomly?
« Reply #2 on: 13:37, 23 May 17 »
We have faster internet down under now


Our PM Brian Trumble says that, but it's a bald faced lie...

Offline Bryce

  • The Hardware Guy.
  • Supporter
  • 6128 Plus
  • *
  • Posts: 10.980
  • Country: wf
  • It's not broken, it just hasn't been fixed yet.
    • index.php?action=treasury
  • Liked: 3862
  • Likes Given: 401
Re: Why does this site grind to a halt randomly?
« Reply #3 on: 13:39, 23 May 17 »

Our PM Brian Trumble says that, but it's a bald faced lie...

They do the same trick as in other countries: Give mega fast internet to a few select companies / areas in the middle of a big town, then calculate the average speed for the country and claim that everything has got faster.

Bryce.

Offline AMSDOS

  • Supporter
  • 6128 Plus
  • *
  • Posts: 3.689
  • Country: au
    • index.php?action=treasury
    • Programs for Turbo Pascal 3
  • Liked: 929
  • Likes Given: 1626
Re: Why does this site grind to a halt randomly?
« Reply #4 on: 13:59, 23 May 17 »

Our PM Brian Trumble says that, but it's a bald faced lie...


Funny I can read that while listening to Don Henley online.  :laugh:


Please don't bring the Poliies in here, their all a pack of Bastards!
* Using the old Amstrad Languages :D   * with the Firmware :P
* I also like to problem solve code in BASIC :)   * And type-in Type-Ins! :D

Home Computing Weekly Programs
Popular Computing Weekly Programs
Your Computer Programs
Updated Other Program Links on Profile Page (Update April 16/15 phew!)
Programs for Turbo Pascal 3

Offline Bignumbas

  • CPC664
  • ***
  • Posts: 128
  • Country: au
  • If you break it you get to keep both halves.
  • Liked: 55
  • Likes Given: 6
Re: Why does this site grind to a halt randomly?
« Reply #5 on: 15:43, 23 May 17 »
Our internet down under is not superfast, its much faster in other countries.
The problem here is the distances so we are lucky to get anything.


I worked in IT for the last 30 years and it would seem the hosting is the problem.
Currently  I am a sys admin in volunteer org with a mix of virtual servers, vpns, ms 365 , etc, etc, etc.
Evidently I am to old to be employable

 :(


Is the site being done on a raspberry pi and a dialup modem?
That would explain the bottleneck
If we all gave money would that help fix it?
« Last Edit: 15:46, 23 May 17 by Bignumbas »
The cutting edge of technology is covered in blood!

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #6 on: 19:38, 28 May 17 »
Not a money issue, but a much harder one, it would seem. The machine is powerful enough certainly, and bandwidth does not seem to be the problem. We've worked many, many hours on this issue and things have arguably gotten much better than before, but we're out of aces to pull.


Basically, "something" stalls the db every now and then; we haven't been able to pinpoint what causes the problem though - there's no consistency to be found, just random queries becoming very slow to execute and the db crawls to a halt. I've pored over the query queues so long my brain turned into mushed potatoes to no avail...


If you got any ideas, I'll certainly be glad to hear from you :)

Offline GeoffB17

  • 6128 Plus
  • ******
  • Posts: 551
  • Country: gb
  • PCW since '85
  • Liked: 115
  • Likes Given: 7
Re: Why does this site grind to a halt randomly?
« Reply #7 on: 19:58, 28 May 17 »
Well, I don't know what system is supporting your server, or the OS, but in my experience regarding a network I look after.


Two things in particular cause the sort of problems you seem to have.


Firstly, from time to time (on schedule) the server will 'wander off' to take care of some housekeeping, re-arranging it's buffers, doing 'shadow copies' or whatever they're called, etc.   These things will be barel;y noticeable if the system is quiet.   If the system is busy at that time, then the users will notice, and also the problem process will take a lot longer!   So it will be more noticeable.   You need to check/review any such 'background' tasks.   That might include what any anti-virus software is doing.


Secondly, regarding specifically the db, what sort of indexing is involved, and how are indexes maintained.   Indexes WILL get in a mess as time goes on, which certainly reduces performance.   The db system may need to do some re-indexing from time to time, there may be some of this happening automatically.   Yes, it's a problem, a system like this will not have much 'quiet' time, as it's being accessed from around the globe.


Geoff

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #8 on: 20:12, 28 May 17 »
Here's the thing, though: even with background tasks etc (even with a powerful enough machine and reinstalls) CPU never touches critical levels. SQL will take a significant chunk itself when it's stuck, but the system is ok, overall - nothing else appears to be doing any damage.


Regarding indexing, I have to look into it again, but I had studied that side of the issue and found nothing in particular. But, even if it's done automatically, it wouldn't be as frequent as the problem...


Also something of interest, the problem appeared *after* we moved to a new server (a dedicated box) and copied everything over, but nothing of significance was changed - same OS, same config, so that in itself is puzzling.

Offline GeoffB17

  • 6128 Plus
  • ******
  • Posts: 551
  • Country: gb
  • PCW since '85
  • Liked: 115
  • Likes Given: 7
Re: Why does this site grind to a halt randomly?
« Reply #9 on: 20:46, 28 May 17 »
Hmm, two interesting things you say there.


First of all, you refer to system being OK except when SQL is 'stuck'.  Are you referring to the 'cpu useage' figure?  If you see that when there's a problem, and it seems to be a problem with SQL, then the problem DOES appear to be with SQL.   What might SQL be doing.


I'd also wonder about your comment about copying things over, esp the 'same config' thing.   I'm not too familiar with SQL (specifically) so I don't know what sort of things get configured, but if you're using a config set/defined/appropriate for the old machine on a new machine (that is HOW different ?) then isn't that a possible source of problem.   Was SQL reconfigured on the new machine?


Geoff

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #10 on: 11:57, 29 May 17 »
Your assumptions are correct - even when SQL is stuck overal CPU usage is not that high, so it's not something else.


There's not much configuring to do regarding the db, to be honest, just installed the engine, imported the db as it was and optimised it (cache levels etc etc). We didn't build it anew. So...

Offline GeoffB17

  • 6128 Plus
  • ******
  • Posts: 551
  • Country: gb
  • PCW since '85
  • Liked: 115
  • Likes Given: 7
Re: Why does this site grind to a halt randomly?
« Reply #11 on: 01:02, 30 May 17 »
The only thing I can think to suggest is...


Does the system allow you to force a re-build of all the indexes.   Various things can cause indexes to get messed up, not to the point that they fail to work, but yet past the point where they work efficiently.


I don't know how efficient  the SQL index system is, but I've had problems with other index systems where the overall efficiency (and hence performance) depends on the distribution of the data used for keys.   If you get an imbalance, such that you have a disproportional lump of records with a VERY similar key, then any sort of update to that part of the index becomes a struggle for the system.   How you would tell this, mind you, I cannot say.


Is the system SQL, or is it MySQL, or what?


Geoff

Offline Audronic

  • 6128 Plus
  • ******
  • Posts: 1.253
  • Country: au
  • I know RST3 . I now know a little More.
  • Liked: 287
  • Likes Given: 622
Re: Why does this site grind to a halt randomly?
« Reply #12 on: 01:53, 31 May 17 »
Please FIX


These are the screen pics before i tried to Log in. after 3 attempts i went to the Third Picture ??


Please FIX




Thanks     Ray
Procrastinators Unite,
If it Ain't Broke PLEASE Don't Fix it.
I keep telling you I am Not Pedantic.
As I Live " Down Under " I Take my Gravity Tablets and Wear my Magnetic Boots to Keep me from Falling off.

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #13 on: 09:48, 31 May 17 »
@Audronic : when reporting a problem please use a new thread instead of hijacking another one...


In any case, we're aware of the problem, caused by an update, a solution is in the works.

Offline Audronic

  • 6128 Plus
  • ******
  • Posts: 1.253
  • Country: au
  • I know RST3 . I now know a little More.
  • Liked: 287
  • Likes Given: 622
Re: Why does this site grind to a halt randomly?
« Reply #14 on: 09:57, 31 May 17 »
@Gryzor


Oops  Sorry


I thought that it might help Diagnosing some faults ?


Ray
Procrastinators Unite,
If it Ain't Broke PLEASE Don't Fix it.
I keep telling you I am Not Pedantic.
As I Live " Down Under " I Take my Gravity Tablets and Wear my Magnetic Boots to Keep me from Falling off.

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #15 on: 10:00, 31 May 17 »
No, that's a totally different issue :)

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #16 on: 10:43, 31 May 17 »
I think the issue has been resolved, can you confirm?

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #17 on: 10:53, 31 May 17 »
The only thing I can think to suggest is...

Does the system allow you to force a re-build of all the indexes.   Various things can cause indexes to get messed up, not to the point that they fail to work, but yet past the point where they work efficiently.

I don't know how efficient  the SQL index system is, but I've had problems with other index systems where the overall efficiency (and hence performance) depends on the distribution of the data used for keys.   If you get an imbalance, such that you have a disproportional lump of records with a VERY similar key, then any sort of update to that part of the index becomes a struggle for the system.   How you would tell this, mind you, I cannot say.

Is the system SQL, or is it MySQL, or what?

Geoff
 


The system is running on a Mariadb. IIRC it had some MyISAM tables that were converted to InnoDB but I'm not 100% sure by now.


We did optimize all sorts of things in the db itself quite a while ago; I could perhaps write a script to defrag and optimize all the tables again, but honestly speaking, given the randomness of the issue, I don't think it'll do much good. Well, it can't hurt either...

Offline Bryce

  • The Hardware Guy.
  • Supporter
  • 6128 Plus
  • *
  • Posts: 10.980
  • Country: wf
  • It's not broken, it just hasn't been fixed yet.
    • index.php?action=treasury
  • Liked: 3862
  • Likes Given: 401
Re: Why does this site grind to a halt randomly?
« Reply #18 on: 11:10, 31 May 17 »
I think the issue has been resolved, can you confirm?

Yup, login from the home page is working again.

Bryce.

Offline Bignumbas

  • CPC664
  • ***
  • Posts: 128
  • Country: au
  • If you break it you get to keep both halves.
  • Liked: 55
  • Likes Given: 6
Re: Why does this site grind to a halt randomly?
« Reply #19 on: 11:57, 31 May 17 »
I sympathise with your troubleshooting woes.
I have just revived two dead cpc's.
Nothing like that feeling when the ready prompt comes up for the first time.
I have also struggled with squid proxy, mysql and websites etc, etc over the years.
Have you looked at '' innodb_adaptive_flushing_method' ?

Sent from my SM-T550 using Tapatalk

The cutting edge of technology is covered in blood!

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #20 on: 12:05, 31 May 17 »
No, but this is used for high-volume IO bursts. I hope some day we get there, but the traffic is not that high yet :D Plus the db does have all the memory it could need and buffers are pretty much ok...


Yeah, the wonders of troubleshooting!

Offline Bignumbas

  • CPC664
  • ***
  • Posts: 128
  • Country: au
  • If you break it you get to keep both halves.
  • Liked: 55
  • Likes Given: 6
Re: Why does this site grind to a halt randomly?
« Reply #21 on: 16:29, 01 June 17 »
Yes , I was thinking it was for short floods of data.
When I cannot get any response from the site it lasts for an hour or more.
Just like when a computer crashes.
You said this started on the new server.
This might make it a hardware issue or with linux a combination of both.

Linux kernels and modules can be a problem, sometimes you have to recompile to get compatibility especially with newer hardware.
Maybe a randomly faulty network port.
That could explain the unresponsiveness at random times
Where I work we have a lot of "techs" and occasionaly one plugs a patch cable  back into the same switch.
Really stuffs up a network with a loop back like that.
We just had to change to pfsense to get our incoming vpn to work, could not get server 2012 to do it.
Why? ???
« Last Edit: 16:32, 01 June 17 by Bignumbas »
The cutting edge of technology is covered in blood!

Offline Bignumbas

  • CPC664
  • ***
  • Posts: 128
  • Country: au
  • If you break it you get to keep both halves.
  • Liked: 55
  • Likes Given: 6
Re: Why does this site grind to a halt randomly?
« Reply #22 on: 14:41, 25 June 17 »
What have you done?
The site does not seem to be freezing up anymore.  :D


???


Dumb luck?
Or really clever stuff?


Cheers
The cutting edge of technology is covered in blood!

Offline arnoldemu

  • Supporter
  • 6128 Plus
  • *
  • Posts: 5.332
  • Country: gb
    • Unofficial Amstrad WWW Resource
  • Liked: 2246
  • Likes Given: 3477
Re: Why does this site grind to a halt randomly?
« Reply #23 on: 16:42, 25 June 17 »
What have you done?
The site does not seem to be freezing up anymore.  :D


 ???


Dumb luck?
Or really clever stuff?


Cheers
It's been stalling for me :(
It never got better :(

My games. My Games
My website with coding examples: Unofficial Amstrad WWW Resource

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.074
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2967
  • Likes Given: 5216
Re: Why does this site grind to a halt randomly?
« Reply #24 on: 17:33, 25 June 17 »
Hahaha :D


No, indeed, it's the same, and rather random as always. Been meaning to get to your previous post, @Bignumbas , but sadly very little time...