Author Topic: Why does this site grind to a halt randomly?  (Read 3958 times)

0 Members and 1 Guest are viewing this topic.

Offline reidrac

  • Supporter
  • 6128 Plus
  • *
  • Posts: 688
  • Country: gb
  • Trying to gamedev!
    • index.php?action=treasury
    • usebox.net
  • Liked: 1198
  • Likes Given: 725
Re: Why does this site grind to a halt randomly?
« Reply #50 on: 17:04, 11 November 17 »
I've been experiencing the same for quite some time now.

Yes, it works after some minutes, so... I just come back later :)
Released The Return of Traxtor, Golden Tail, Magica and The Dawn of Kernel for the CPC.

If you like my games and want to show some appreciation, you can always buy me a coffee.

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #51 on: 13:57, 13 November 17 »
No news ever since. Nobody has offered to help unfortunately, and I'm really at my wits' end regarding that issue...

Offline GeoffB17

  • 6128 Plus
  • ******
  • Posts: 545
  • Country: gb
  • PCW since '85
  • Liked: 116
  • Likes Given: 7
Re: Why does this site grind to a halt randomly?
« Reply #52 on: 17:23, 13 November 17 »
Hello,
 
I'd LOVE to help, but how?
 
I get this problem frequently. Lasts for a period, then OK again.  To me, shows all the signs of your server/system being busy.   Don't know why.   Could be the server is doing some housekeeping job.   Could be a number of downloads in progress.
 
Do you get any sort of log as to what happens.   So that if, once I can access again, I can send a message saying that 'about time xx:xx the system was not accessible.   Lasted for xx minutes'?   There's no point to doing this unless you've got something to check.
 
Of course, as with any DOS attack, the problem will be compounded because anythime there is a bit of a problem, if there are people trying to log on, and they keep repeat trying, then it just adds to/multiples the problem.
 
Geoff

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #53 on: 18:30, 13 November 17 »
I do have external services set up, and I know when the site gets unresponsive.


It's not a DOS attack; in the past we've had a couple but it looked more like spambots gone wild rather than DDOS attacks, and were repelled quite fast (see how an entire continent is disallowed :D ).


The problem has to do with the db; for some reason sometimes, otherwise perfectly normal queries become slow and take a long time to complete. Haven't managed to find out why since it's not specific scripts/functions/calls that cause the trouble!

Offline GeoffB17

  • 6128 Plus
  • ******
  • Posts: 545
  • Country: gb
  • PCW since '85
  • Liked: 116
  • Likes Given: 7
Re: Why does this site grind to a halt randomly?
« Reply #54 on: 19:17, 13 November 17 »
I think I asked this before, some time ago.   I seem to remember that you said what the db was.   Can't remember.  But even just db systems need to do housekeeping from time to time, esp if indexes are involved, and sometimes specific sorts of key can lead to problems meaning the system has to do a LOT of work to update the indexes.   Memory available will have an effect as well.   I mean memory specifically available to the db, not just the memory available on the machine.
Geoff

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #55 on: 19:39, 13 November 17 »
System specs are fine, memory allocated to db were upped and re-upped to accommodate things, nothing seemed to work. We're on MariaDB.


The weird thing is, the db was lifted and replicated from the previous server where we didn't have such issues (well, we did have performance issues to the server's age, but not db-related); and yet, the problems appeared out of the blue.


To tell you the truth, we've done so many things to try and make the db behave better, that I don't remember all of the details now; if you feel you can help, let's take this off the forum and do it through email - I'm willing to take it up again :)


Thanks!

Offline Poliander

  • Supporter
  • CPC6128
  • *
  • Posts: 164
  • Country: de
  • Liked: 81
  • Likes Given: 69
Re: Why does this site grind to a halt randomly?
« Reply #56 on: 19:46, 13 November 17 »
Hi Gryzor,

from what I've observed the database theory sounds very valid to me. I believe it might have something to do with an exceeded connection limit which may occur if the application, or some part of, does not close the db connection for some reason. This is of course absolutely not trivial to debug and hard to explain how to do this.

I suppose you're running Ubuntu 14.04 (that's the version with Apache 2.4.7 IIRC). You might want to try mysqlprimer and/or mysqltuner, the DB should be already up and running for a while, then those scripts will fetch the database statistics and give you some hints how to improve your DB tuning.

By the way, switching to nginx is not a bad idea. But that's another thing.

In case you're not sure whether it's really the database, you could use a watchdog/website monitoring (e.g. pingdom or something like that) for tracking that down. Place a plain text "test.txt" file, a "echo ping" test.php script and a test-db.php file which opens (and closes) a db connection somewhere in your document root. Then configure the website monitor to call each of them every few minutes. When the txt file is not reachable, you can forget PHP and the database. If the txt file is reachable, but PHP not, it's probably the PHP FCGI/FPM backend etc.

Cheers
Schneider CPC 664 • X-MEM • X-MASS • CTM 644 • DMP 2160 — Schneider CPC 6128 • M4 Board • Z-MEM • OSSC

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #57 on: 20:42, 13 November 17 »
You're right in your assumptions about the system. I'll try mysqlprimer and mysqltuner, never used them before; got to study a bit and see if we'llneed a backup to restore the system to previous state if it all goes to shit when I run them :D


Pretty skeptical about nginx; always wanted to try it, but preferably not on our valuable production server!


Connection limits, like allocated mem, have been raised again and again (I think in the beginning it was pretty tight, but raising them didn't help). The thing is, queries do finish and connections are closed -it just takes a looong time for it to happen, randomly. I mean, I've seen the thing hanging with such trivial things as fetching a filename to display an attached file, or quering to see how many posts a user has. Go figure. For instance, this page I'm typing on took 0.099 seconds to create with 37 queries. The same page will take a minute or two some times - and then the whole system will freeze for everyone. System resources do hold up - very frequently I'm logged in and I can watch stuff and in general it's very calm - except for the db.


However, employing a watchdog is not a bad idea. I mean, I am pretty sure it's the db's fault, but it never hurts to go back to basics and try and take it from there. Do you have any service in mind? At this point I wouldn't mind paying for a month or two to make things better.


Thanks once again :)


PS I would do it now, but one thing at a time preferably - currenty running a scheduled malware scan - and let me tell you, it takes quite some time to check the almost 900,000 files we've got sitting there!

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #58 on: 13:09, 15 November 17 »
Ok.


Over the last few dates I did some tweaking on the databases and on the functionality of the forum and wiki. We did gain some stuff - for instance, a couple of absolutely huge tables were slashed and unnecessary functionality removed to reduce the number of db hits, but it remains to be seen whether these will result in any improvement in real life.


On the other hand, I did run the scripts suggested above. If @Poliander wants to take a look at the logs and discuss, I'm all ears :)

Offline Audronic

  • 6128 Plus
  • ******
  • Posts: 1.252
  • Country: au
  • I know RST3 . I now know a little More.
  • Liked: 288
  • Likes Given: 620
Re: Why does this site grind to a halt randomly?
« Reply #59 on: 00:48, 16 November 17 »
@Gryzor.


Ok I have had similar problems on the old setup as the current setup, I believe the it is the same problem that i have had for years.


It fell over today (This connection)  :- Logged in to the site. Then went to put Name /Password in Pressed "Return" wait - wait - wait etc
The original connection started up fast BUT ----


I hope that this is of some help.
I am in a MAC OS 10.11.6


Thanks        Ray
Procrastinators Unite,
If it Ain't Broke PLEASE Don't Fix it.
I keep telling you I am Not Pedantic.
As I Live " Down Under " I Take my Gravity Tablets and Wear my Magnetic Boots to Keep me from Falling off.

Offline GeoffB17

  • 6128 Plus
  • ******
  • Posts: 545
  • Country: gb
  • PCW since '85
  • Liked: 116
  • Likes Given: 7
Re: Why does this site grind to a halt randomly?
« Reply #60 on: 01:16, 16 November 17 »
Just out of interest, what were the 'couple of abs huge tables', what sort of data was being stored in them, how was the data being accessed?   Were the files indexed?

Just because the file is HUGE doesn't make it a problem.   A massive log file, no index, use always just adding new records (on the end) COULD be no problem at all.

Hopefully, I be able to investigate more soon.   My main PC died, waiting for replacement.   Laptop is a help, but it's W7, and less comfortable to use.

Geoff

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #61 on: 09:25, 16 November 17 »
Two tables had to do with how the system monitors which threads and which messages you have read or not. With a lot of inactive users, over the years, this becomes problematic since every new post and thread is kept in an entry as unread for them.


On the other hand, there was wiki one caching data for, um, probably 11 years or so. I had my eye on it for quite some time now because I was seeing hits to it being rather slow, but that was not the 'serious' problem we've been experiencing. Just went ahead and did it in anyway.

Offline GeoffB17

  • 6128 Plus
  • ******
  • Posts: 545
  • Country: gb
  • PCW since '85
  • Liked: 116
  • Likes Given: 7
Re: Why does this site grind to a halt randomly?
« Reply #62 on: 23:49, 17 November 17 »
Hmm!

I was just thinking, the site has been OK the last few times I've tried to connect.   Is it better?

BUT...

Just now, 9:32pm UK time, I tried to connect, and it just sat thumbing it's nose at me.   Took a few minutes to connect.   Not too long, OK, but still longer than normal.

Was the system doing anything special at that time.   Any hints of specific activity you could identify.

One process I do worry about, as it can cause me problems for my support work.   When the system starts doing the (in Windoze) 'shadow copies'.   I note that Linux systems can do something similar (using LVM, or similar).   Does your system do anything like that.   When, how often?

Such processes should not cause major problems, BUT, this can be dependant on various peculiarities regarding number of files, size of files, and how the files change.   Hence, you could have one setup that copes OK, and another that has problems.

Do you use such a process?

By the way, further to an earlier mention, I've now got a 'new' machine, getting it fully set up.   I say 'new' as it's far from NEW, I do a LOT of DOS stuff, and need a system that allows reasonable DOS capability, and XP is pretty much the last system that works for me, but it's almost impossible to get a NEW system that comes like that.   The 'new' box, though, is looking good!

Geoff

Offline Audronic

  • 6128 Plus
  • ******
  • Posts: 1.252
  • Country: au
  • I know RST3 . I now know a little More.
  • Liked: 288
  • Likes Given: 620
Re: Why does this site grind to a halt randomly?
« Reply #63 on: 00:40, 19 November 17 »
@Admin


This site Grrrr.
It took 30 Minutes to do 1 Post today.
4 attempts to log in 3 attempts log in as audronic.
It fell over trying to read some postings.
It fell over trying to add a reply.


Very Frustrated
If Possible Please FIX


Thanks    Ray
Procrastinators Unite,
If it Ain't Broke PLEASE Don't Fix it.
I keep telling you I am Not Pedantic.
As I Live " Down Under " I Take my Gravity Tablets and Wear my Magnetic Boots to Keep me from Falling off.

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #64 on: 00:52, 19 November 17 »
You don't say? Oh my, better get on top of this, stat!

Offline reidrac

  • Supporter
  • 6128 Plus
  • *
  • Posts: 688
  • Country: gb
  • Trying to gamedev!
    • index.php?action=treasury
    • usebox.net
  • Liked: 1198
  • Likes Given: 725
Re: Why does this site grind to a halt randomly?
« Reply #65 on: 01:16, 19 November 17 »
Not sure if add anything to the puzzle, but is not just the forum. I've had the same issue with the wiki; but if there's already some theory that this could be the DB, it would make sense (assuming both forum and wiki share DB server).
Released The Return of Traxtor, Golden Tail, Magica and The Dawn of Kernel for the CPC.

If you like my games and want to show some appreciation, you can always buy me a coffee.

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #66 on: 01:18, 19 November 17 »
Yup, it's the db alright; been doing north of 100% cpu today, and I'm still hunting the why... :( too late here, heading to bed for now...

Offline Audronic

  • 6128 Plus
  • ******
  • Posts: 1.252
  • Country: au
  • I know RST3 . I now know a little More.
  • Liked: 288
  • Likes Given: 620
Re: Why does this site grind to a halt randomly?
« Reply #67 on: 01:20, 19 November 17 »
Yup, it's the db alright; been doing north of 100% cpu today, and I'm still hunting the why... :( too late here, heading to bed for now...


Thanks that is appreciated.


Ray
Procrastinators Unite,
If it Ain't Broke PLEASE Don't Fix it.
I keep telling you I am Not Pedantic.
As I Live " Down Under " I Take my Gravity Tablets and Wear my Magnetic Boots to Keep me from Falling off.

Offline steve

  • Supporter
  • 6128 Plus
  • *
  • Posts: 1.263
  • Country: gb
  • Liked: 178
  • Likes Given: 210
Re: Why does this site grind to a halt randomly?
« Reply #68 on: 01:35, 19 November 17 »
I have not been looking at the site much for quite a while now, my unread posts had gotten past 60 pages and the thought of reading them was quite daunting, now my unread posts are down to 4 pages, so, thank you? Gryzor :), did I miss anything earth shattering?.

Offline Gryzor

  • Administrator
  • 6128 Plus
  • *****
  • Posts: 15.071
  • Country: gr
  • CPC-Wiki maintainer
    • CPCWiki
  • Liked: 2996
  • Likes Given: 5210
Re: Why does this site grind to a halt randomly?
« Reply #69 on: 01:37, 19 November 17 »
Well, quite a few things actually, but you wouldn't be able to find them probably? [emoji849]

Sorry about that; but of course I guess it did make some sense...

Sent from my HTC 10 using Tapatalk