Page 1 of 1

Forum Outage

Posted: Thu Oct 24, 2019 10:27 pm
by Geoff the Medio
For most of the last 16-24 hours, the forums were inaccessible.

This happened because the host disabled the MySQL database for the forums. They did that because the forums were being flooded with requests originating from Chinese IP addresses, which was causing problems for the shared server. They tried blocking individual IPs, but were unable to slow the traffic, so disabled the database to prevent it from affecting any other users.

We've since set up a free tier of CloudFlare distributed cache, and I've banned a range of IPs in the forum software itself that were apparently the source of most of the issues, which dropped the stable 5-min concurrent users from about 300 (of which 99+% were guest users) to 50, with the 50 mostly being CloudFlare-related IPs, based on the whois info the forum shows me.

Hopefully this combination of tactics will reduce the issues for the host and the server and bandwidth usage for freeorion.org.

Re: Forum Outage

Posted: Fri Oct 25, 2019 7:19 am
by AndrewW
I noticed.

Hopefully it all clears up. Wonder if this was really a target or they are just testing some stuff out.

Re: Forum Outage

Posted: Sat Oct 26, 2019 4:32 pm
by Adrian
AndrewW wrote:
Fri Oct 25, 2019 7:19 am
Hopefully it all clears up. Wonder if this was really a target or they are just testing some stuff out.
I think it's more likely that there's a financial (hacking/phishing) interest behind it, rather than a deliberate DDoS intended to harm the site.

Re: Forum Outage

Posted: Sat Oct 26, 2019 8:01 pm
by Vezzra
What can be possibly be (financially) gained by launching a DDoS attack against our forums? This is an open source project, no money involved at all here, nothing to gain...

Re: Forum Outage

Posted: Sat Oct 26, 2019 9:47 pm
by Geoff the Medio
I think the proposed motive is to get access and post spam, phishing links, or search engine optimization links.

Re: Forum Outage

Posted: Sun Oct 27, 2019 12:27 am
by AndrewW
Just hit the database problem again (max connections), though cleared up when I reloaded the page.

Code: Select all

SQL ERROR [ mysqli ]

User tzlaine_4 already has more than 'max_user_connections' active connections [1203]

Re: Forum Outage

Posted: Sun Oct 27, 2019 9:54 am
by Ophiuchus
Forum is almost not usable for me right now.

4 out of five tries i GET the 'max_user_connections' active connections issue

Re: Forum Outage

Posted: Sun Oct 27, 2019 10:49 am
by Oberlus
Ophiuchus wrote:
Sun Oct 27, 2019 9:54 am
4 out of five tries i GET the 'max_user_connections' active connections issue
In my case it's variable. Sometimes it needs ten retires per page, sometimes I only need to retry one every few pages.

I wonder if there is a way to ban the IPs that are constantly crawling the forum and are not useful bots.

Re: Forum Outage

Posted: Sun Oct 27, 2019 11:02 am
by Geoff the Medio
I already banned all the IPs in the range from the Hong Kong ISP that is causing the issues. The remaining connections are coming through the CDN now so can't be blocked in the forum software. Tyreth should be setting up a CAPTCHA for those IPs in the CDN though, qhich should deter bot connections before they reach the forum server itself.

Re: Forum Outage

Posted: Sun Oct 27, 2019 1:14 pm
by Vezzra
Geoff the Medio wrote:
Sat Oct 26, 2019 9:47 pm
I think the proposed motive is to get access and post spam, phishing links, or search engine optimization links.
Well, I'm not really an IT security expert, but how is a rather slow/lazy DDoS attack like this supposed to get you into the forum to do these things? To achieve this, I'd rather expect outright hacking attempts, and who would waste such efforts on our forum?

Or ARE all these access attempts by bots actually automated hacking attempts...? But even if, that's a lot computer power wasted on such a small, insignificant forum... :?

Re: Forum Outage

Posted: Sun Oct 27, 2019 1:16 pm
by Vezzra
Ophiuchus wrote:
Sun Oct 27, 2019 9:54 am
Forum is almost not usable for me right now.

4 out of five tries i GET the 'max_user_connections' active connections issue
Same here, although right now it's even worse...

Re: Forum Outage

Posted: Sun Oct 27, 2019 2:37 pm
by AndrewW
Vezzra wrote:
Sun Oct 27, 2019 1:16 pm
Same here, although right now it's even worse...
Was variable for me on my previous post, sometimes ok, other times several reloads where needed. Seems to be fine at the moment though.

Re: Forum Outage

Posted: Sun Oct 27, 2019 2:58 pm
by Vezzra
AndrewW wrote:
Sun Oct 27, 2019 2:37 pm
Was variable for me on my previous post, sometimes ok, other times several reloads where needed. Seems to be fine at the moment though.
Tyreth has just enabled some countermeasures (CAPTCHA for China and Hong Kong) on the cloudflare tier level, if I understand correctly. Immediately afterwards the issues were gone. So it looks like that worked. I can use the forum just fine now.

Re: Forum Outage

Posted: Sun Oct 27, 2019 3:43 pm
by Oberlus
Vezzra wrote:
Sun Oct 27, 2019 2:58 pm
Tyreth has just enabled some countermeasures (CAPTCHA for China and Hong Kong) on the cloudflare tier level, if I understand correctly. Immediately afterwards the issues were gone. So it looks like that worked. I can use the forum just fine now.
Thank you all for the arrangements. Now this is smooth again.