PLEASE READ: Regarding forum performance issues

[TFM]

Or maybe the problem is that people constantly upload pictues in the size of many Megabytes?. PMs are only text. 

[||C|-|E||]

Picture size is always a problem, even if the server is responding OK. I like to upload pictures from time to time, but I always try to keep them below 400KB to let the server live 

[TFM]

Picture size is always a problem, even if the server is responding OK. I like to upload pictures from time to time, but I always try to keep them below 400KB to let the server live 

Especially since we're already forced to crunch 200 KB disc images and 16 KB ROM images, else we can't upload and the typed in text of the post is lost when the system displays a file-type error. 

[Bryce]

If it was a problem with PMs or uploaded pictures it would always be there. However the problem is that the server goes from very responsive to (essentially) offline for 5 to 7 minutes at a time several times a day. This has nothing to do with the amount of PMs or pictures.

Bryce.

[arnoldemu]

If it was a problem with PMs or uploaded pictures it would always be there. However the problem is that the server goes from very responsive to (essentially) offline for 5 to 7 minutes at a time several times a day. This has nothing to do with the amount of PMs or pictures.

Bryce.

My thoughts were that people who had a high number of PMs happened to access the forum at the same or similar times. The server was quering the database to see if there are new messages for each person and this was causing a stall. All of this is speculation of course.

[TFM]

If it was a problem with PMs or uploaded pictures it would always be there. However the problem is that the server goes from very responsive to (essentially) offline for 5 to 7 minutes at a time several times a day. This has nothing to do with the amount of PMs or pictures.

Bryce.

And why not? If there is a time (or times) when everybody looks at a thread with lots of pictures then if overwhelms the servers and it locks up. If you disagree, what could it be then??

[Bryce]

You can see how many people are online and even reading the same thread as you. The slow downs aren't related to either.

Bryce.

[SRS]

... PMs are only text. 

Well, modern computers like CPC store text as bitmap on screen so 500 chars need 16k instead of 0,5

[||C|-|E||]

The pictures are definitely not killing the server  They can cause a slowdown if they are huge and in general they can make a post to load slower, so it is always good to keep them withing a reasonable size. However, in this case we suffer from complete blackouts, that is something different  .

[1024MAK]

So I have a total of 15 messages, and often the thread I am accessing has no large or indeed any pictures apart from the icons and pictures in people's signatures.


So I don't think either is relevant. Also the long pause appears to occur anywhere in the downloading process. Sometimes half a page loads. Other times very little of the page loads. And worst of all, sometimes it does it when attempting to send a reply 

[Gryzor]

Ok, the dropout yesterday was at 13:05. Today it was at 13:00. That's close enough for me to be predictable. Let's see what happens tomorrow between 13:00 and 13:05.

Just a coincidence, actually, from the looks of it. Some times I wake up and find alerts that the server was down during the night, sometimes I get notifications while I'm having dinner and so on. Just took a look at the graphs, they don't show a pattern.

What DB engine is it? SQL tuning tends to be very specific to the system being used. I'm mostly an MS SQL person, but happy to help if I can.

We use MariaDB. If you have any idea about troubleshooting slow queries, do let me know

@Gryzor: this is almost related. Do you know if personal messages are causing a problem with the database?

I have 55 pages of personal messages.  Sorry.

I will try and delete them all this weekend.

Highy, highly unlikely. The queries I had seen don't show something of the sort, plus in any case there are threads longer than 55 pages. This gave me an idea, I lowered the number of posts/threads per page, just to check things out....

My thoughts were that people who had a high number of PMs happened to access the forum at the same or similar times. The server was quering the database to see if there are new messages for each person and this was causing a stall. All of this is speculation of course.

If we had thousands of users doing that all at once it'd make some sense, but as it is it doesn't. As I said before, for some reason some perfectly sensible queries just take huge amounts of time to execute - it's not that there's a huge number of queries. As a matter of fact I've optimised the forum since a long time ago to reduce the number of queries used as much as possible, and I'd say we're at very ok levels.

Picture size is always a problem, even if the server is responding OK. I like to upload pictures from time to time, but I always try to keep them below 400KB to let the server live 

True. Serving pictures would be a bandwidth issue and (to a much lesser extent) a CPU issue as Apache is pushing things around. But we're nowhere near our limits, and the pictures are not stored in the database...

[TotO]

Before, I had to waid many minutes and sometime the loading failled.
Now, I have to waid some seconds... It is cool.

[Audronic]

This wait time was happening on the Previous server.
I just thought i was to far away ?


Ray

[Gryzor]

This wait time was happening on the Previous server.
I just thought i was to far away ?


Ray

Sorry, not sure what you mean? 

[Audronic]

Some months ago i thought that you changed servers or providers, the Problem was there on the old server- Provider.
Does that make sense or have i imagined it ?


Ray

[Gryzor]

Ah no, we changed servers because the old one was getting waaay too old and was very expensive for what it was offering...

[Audronic]

OK, when we were on the OLD server the same problems were happening there.
I hope that this helps.


Ray

[Gryzor]

Well, depends on the period, we did have some issues but nothing as serious as this.

[Bryce]

If you think our Forum is slow, try connecting to xe.com or any of the other currency conversion sites this morning

Bryce.

[robcfg]

Heh, 36 seconds before giving an error....


xe.com is indeed on fire today! 

[Gryzor]

Heheh we noticed the XE issues first thing in the morning (I work for a financial company)... it's a bit better now, I guess they upped some limits for their database


By the way, I realised I said something that wasn't true some pages earlier: when the problem occurs, the CPU spikes indeed, but again it's nowhere near any levels that should cause issues. We're talking maybe about 20 or 30%. It's just that it takes a looong time to process the query...

[SRS]

Sounds like file resp. database i/o performance issues to me.

[Grim]

It's just that it takes a looong time to process the query...

Just some random thoughts about this:

• Is there anything clogging up files-system IO when it happens? (eg. backup script or stats/monitoring script)
• And nothing in the DB's logs would indicate if it is often (re-)building huge index/temp tables? (you might have to change the log level setting and restart the DB to get these information).
• Try the EXPLAIN command on a query that sometime takes too long to execute.
• Maybe check if there's no unusual amount of HTTP requests when it happens? (such as h4x0rz scripts brute-forcing the auth to the admin panel or other well-known vulnerabilities)
• Also the default MariaDB configuration might not be adequate anymore with the growing size/traffic of the forum and would require some tweaking?
  

Tough problem to figure out, may Nelson be with you! :)

[AMSDOS]

Not sure if it relates to the performance issues, but I just went into my account settings to view my posts and when I clicked on Drafts, I found 4 Pages of stuff, some of that was old too.


So I'm just wondering if that's happening, how many people are unknowingly building up this clutter? Can Admin see those posts and/or cleanse (get rid of) those kinds of posts?

[Bryce]

Hey, I just checked too and had 5 pages of drafts??? Why is the system creating drafts at all?

Bryce.