Server Issues

[PapaFrizArcade]

This last weekend we experienced some server issues...

I was told:

I found that the site was hitting PHP-FPM pool limits. I have increased those considerably which should resolve the issue.

That seemed to resolve the issue for a few hours... but then the site went down again.  I re-inquired.  After a short while everything went back online and it has been since.  A day or two later my server admin emailed me:

How have things been performing since your last reply?

We ended up increasing the limits on another setting within an hour or so of your last reply. Have things been normal since then?

Another thing they mentioned is that there was a lot of traffic to a specific query that day that seemed like a possible indicator of a code-related issue. We're not 100% positive but something wasn't adding up that day.

The overall traffic isn't what was really concerning but the amount of requests for specific things were much higher than some of the other sites on the server that have a lot more traffic (as a comparison).

For example, here's one comment from the datacenter:

"When looking at traffic from 67.XXX.112.XXX, I found quite a few of the following requests:

67.XXX.112.XXX - - [09/May/2022:16:10:24 -0400] "GET /index.php?action=profile;area=alerts_popup;counter=6;u=436 HTTP/1.1" 200 486 "https://www.NOFOLLOW.com/index.php?action=badgeawards;sa=badgedetail&id=699" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"

You may want to run that by the site developer to see if there have been recent changes."

That particular IP (user) above had about 5000 requests in the timeframe that day we were reviewing.

The IP in question above is one of my BADGEMASTERS.  I asked him about that day... if he noticed anything... and asked what he was doing.  He replied:

That's odd for sure.

I was browsing around the master badge list looking at previous badges I have missed as I'm thinking of starting a thread of Zombie badge hunting for stuff that I may have missed being late to the AHA game.

I probably had 15 or so tabs open with various badges from the list and the threads they came from. It certainly didn't feel like much of anything excessive though.

I cannot begin to imagine how I could have generated 5000 requests for anything from the site over a particular time period though, unless that period was days to weeks long.

I had a fair number of problems on Monday when the site was having issues, but so did everyone.

Anything I can do to assist in troubleshooting though, let me know.

It seems that messing around in the MASTER BADGE list may have caused too many PHP calls or something?!? I have no idea... hence my post here.  Keep in mind we are nearly at 1000 badges now.

Any way to prevent this in the future?  Is my hypothesis correct?  Am I going to have to start DELETING badges? (say it isn't so!!!)



Anyway,  whatever they did on their side seems to have worked as the site is up and we've not had any more issues since...

[SMFHacks]

It looks like it is with the alert mod based on if all the requests start with
 /index.php?action=profile;area=alerts_popup;counter=6;u=436 and if the badge id is the same.
I think the alerts mod got struck in a loop or something similiar.


That page on the badges system just shows badge details.

[PapaFrizArcade]

Ah,  but the alerts "mod" is no longer a mod but part of SMF.  I guess I need to report this issue over there?

[SMFHacks]

Yeah would like to see more of the requests to see if there is a pattern or what happened.

[PapaFrizArcade]

I can ask the server admin ... or if you can direct me to where I may be able to create such a log in Cpanel (if doable) I can do it myself.

[SMFHacks]

Haven't used cpanel in a long time. There should be a logging section though. Like apache logs.

[PapaFrizArcade]

Found a bunch of different ways to access historical data,  unfortunately it appears to typically only go back a couple days or so.  I'll ask the server admin for more info if possible.