Forum Outage

Discussion about the project in general, organization, website, or any other details that aren't directly about the game.
Post Reply
Message
Author
User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12469
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Forum Outage

#1 Post by Geoff the Medio » Thu Oct 24, 2019 10:27 pm

For most of the last 16-24 hours, the forums were inaccessible.

This happened because the host disabled the MySQL database for the forums. They did that because the forums were being flooded with requests originating from Chinese IP addresses, which was causing problems for the shared server. They tried blocking individual IPs, but were unable to slow the traffic, so disabled the database to prevent it from affecting any other users.

We've since set up a free tier of CloudFlare distributed cache, and I've banned a range of IPs in the forum software itself that were apparently the source of most of the issues, which dropped the stable 5-min concurrent users from about 300 (of which 99+% were guest users) to 50, with the 50 mostly being CloudFlare-related IPs, based on the whois info the forum shows me.

Hopefully this combination of tactics will reduce the issues for the host and the server and bandwidth usage for freeorion.org.

AndrewW
Juggernaut
Posts: 788
Joined: Mon Feb 04, 2013 10:15 pm

Re: Forum Outage

#2 Post by AndrewW » Fri Oct 25, 2019 7:19 am

I noticed.

Hopefully it all clears up. Wonder if this was really a target or they are just testing some stuff out.

User avatar
Adrian
Space Floater
Posts: 49
Joined: Thu Jan 26, 2012 8:40 pm
Location: Eridanus Supervoid

Re: Forum Outage

#3 Post by Adrian » Sat Oct 26, 2019 4:32 pm

AndrewW wrote:
Fri Oct 25, 2019 7:19 am
Hopefully it all clears up. Wonder if this was really a target or they are just testing some stuff out.
I think it's more likely that there's a financial (hacking/phishing) interest behind it, rather than a deliberate DDoS intended to harm the site.

User avatar
Vezzra
Release Manager, Design
Posts: 5028
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Forum Outage

#4 Post by Vezzra » Sat Oct 26, 2019 8:01 pm

What can be possibly be (financially) gained by launching a DDoS attack against our forums? This is an open source project, no money involved at all here, nothing to gain...

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12469
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Forum Outage

#5 Post by Geoff the Medio » Sat Oct 26, 2019 9:47 pm

I think the proposed motive is to get access and post spam, phishing links, or search engine optimization links.

AndrewW
Juggernaut
Posts: 788
Joined: Mon Feb 04, 2013 10:15 pm

Re: Forum Outage

#6 Post by AndrewW » Sun Oct 27, 2019 12:27 am

Just hit the database problem again (max connections), though cleared up when I reloaded the page.

Code: Select all

SQL ERROR [ mysqli ]

User tzlaine_4 already has more than 'max_user_connections' active connections [1203]

Ophiuchus
Programmer
Posts: 1002
Joined: Tue Sep 30, 2014 10:01 am
Location: Wall IV

Re: Forum Outage

#7 Post by Ophiuchus » Sun Oct 27, 2019 9:54 am

Forum is almost not usable for me right now.

4 out of five tries i GET the 'max_user_connections' active connections issue
Any code or patches in anything posted here is released under the CC and GPL licences in use for the FO project.

Furthermore, I propse... we should default to four combat rounds instead of three ...for the good of playerkind.

User avatar
Oberlus
Cosmic Dragon
Posts: 1579
Joined: Mon Apr 10, 2017 4:25 pm

Re: Forum Outage

#8 Post by Oberlus » Sun Oct 27, 2019 10:49 am

Ophiuchus wrote:
Sun Oct 27, 2019 9:54 am
4 out of five tries i GET the 'max_user_connections' active connections issue
In my case it's variable. Sometimes it needs ten retires per page, sometimes I only need to retry one every few pages.

I wonder if there is a way to ban the IPs that are constantly crawling the forum and are not useful bots.

User avatar
Geoff the Medio
Programming, Design, Admin
Posts: 12469
Joined: Wed Oct 08, 2003 1:33 am
Location: Munich

Re: Forum Outage

#9 Post by Geoff the Medio » Sun Oct 27, 2019 11:02 am

I already banned all the IPs in the range from the Hong Kong ISP that is causing the issues. The remaining connections are coming through the CDN now so can't be blocked in the forum software. Tyreth should be setting up a CAPTCHA for those IPs in the CDN though, qhich should deter bot connections before they reach the forum server itself.

User avatar
Vezzra
Release Manager, Design
Posts: 5028
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Forum Outage

#10 Post by Vezzra » Sun Oct 27, 2019 1:14 pm

Geoff the Medio wrote:
Sat Oct 26, 2019 9:47 pm
I think the proposed motive is to get access and post spam, phishing links, or search engine optimization links.
Well, I'm not really an IT security expert, but how is a rather slow/lazy DDoS attack like this supposed to get you into the forum to do these things? To achieve this, I'd rather expect outright hacking attempts, and who would waste such efforts on our forum?

Or ARE all these access attempts by bots actually automated hacking attempts...? But even if, that's a lot computer power wasted on such a small, insignificant forum... :?

User avatar
Vezzra
Release Manager, Design
Posts: 5028
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Forum Outage

#11 Post by Vezzra » Sun Oct 27, 2019 1:16 pm

Ophiuchus wrote:
Sun Oct 27, 2019 9:54 am
Forum is almost not usable for me right now.

4 out of five tries i GET the 'max_user_connections' active connections issue
Same here, although right now it's even worse...

AndrewW
Juggernaut
Posts: 788
Joined: Mon Feb 04, 2013 10:15 pm

Re: Forum Outage

#12 Post by AndrewW » Sun Oct 27, 2019 2:37 pm

Vezzra wrote:
Sun Oct 27, 2019 1:16 pm
Same here, although right now it's even worse...
Was variable for me on my previous post, sometimes ok, other times several reloads where needed. Seems to be fine at the moment though.

User avatar
Vezzra
Release Manager, Design
Posts: 5028
Joined: Wed Nov 16, 2011 12:56 pm
Location: Sol III

Re: Forum Outage

#13 Post by Vezzra » Sun Oct 27, 2019 2:58 pm

AndrewW wrote:
Sun Oct 27, 2019 2:37 pm
Was variable for me on my previous post, sometimes ok, other times several reloads where needed. Seems to be fine at the moment though.
Tyreth has just enabled some countermeasures (CAPTCHA for China and Hong Kong) on the cloudflare tier level, if I understand correctly. Immediately afterwards the issues were gone. So it looks like that worked. I can use the forum just fine now.

User avatar
Oberlus
Cosmic Dragon
Posts: 1579
Joined: Mon Apr 10, 2017 4:25 pm

Re: Forum Outage

#14 Post by Oberlus » Sun Oct 27, 2019 3:43 pm

Vezzra wrote:
Sun Oct 27, 2019 2:58 pm
Tyreth has just enabled some countermeasures (CAPTCHA for China and Hong Kong) on the cloudflare tier level, if I understand correctly. Immediately afterwards the issues were gone. So it looks like that worked. I can use the forum just fine now.
Thank you all for the arrangements. Now this is smooth again.

Post Reply