Member
Gnome
Group: Admin
Posts: 236
Joined: 25-July 05
Class: Warlock
Professions: Alchemy, Herbalism
|
Maybe amateurs shouldn't host their own sites. I do anyway, because it was a fun little project, and does continue to be, even though it has been on the back burner for a very long time.
So, confession time. For several years, my only method of SQL backups was a cron job on the server that saved a mysqldump locally, with a fixed filename (so it'd overwrite on the weekly schedule it was on). I would manually copy that over to my desktop every few months, or much less often. In February 2016, for whatever reason, I wisened up. Maybe I was bored, or maybe after several years of neglect, the slowly mounting fear of losing all my data due to an inadequate backup scheme had finally reached the point of prompting some kind of action. Probably both. I ended up researching a very quick way to set up a timed job on my Windows10 desktop that pulled the mysqldump file over to the desktop, and in the process named the file based on the copy date. It was possible without installing any new software, just vanilla Windows10, so that made me happy.
After 3 ½ weeks of running this setup, I checked up on how it was working. I was delighted to learn it had worked precisely as intended, but I also learned of an incredibly alarming trend in the filesize of the dump file.
A file from October 2013 that I had kept around as reference was around 11 megabytes. A manually pulled file from beginning of February was 17 megabytes, which I didn't have reason to doubt, it had been more than 2 years after all. But the pattern of the next weeks was jarring:
2016-02-15, 20 Mb
2016-02-22, 31 Mb
2016-02-29, 62 Mb (!!!)
This was the point at which I realized something was terribly wrong. It was apparent that something was flooding the server with spam. I run several very old unlicensed IPB forums (the version I run does allow it, so I shouldn't be breaching their terms for my noncommercial use), and I had to lock them down several years ago due to spambots having their way with them. It had been a sufficient strategy to stop new member registrations and disable guest posting, but now I got a bit worried if perhaps new exploits had been found that circumvented those controls.
I could have researched how to analyze the database to find out where the spam is, but I wanted to first try my luck with something as simple as a look into the access log. Typically the spambots try all kinds of different attacks at the IPB or the server in general, which tend to leave errors in the access log. This happened to be the case here as well. Upon simply glancing at the tail of the access log, I immediately spotted suspect behavior. Now, errors left by spambots are not a red flag as such - like said, they are a commonplace thing to have all the time. But the part that caught my attention was that the errors were arising from a subdomain that I did not honestly remember even existing.
That's a huuuge red flag. I realize I have an IPB there, and log on. There are thousands of threads and hundreds of "members" - on a board which, from what I could tell by looking at the oldest available threads, was completely empty until around December 2015.
I start with the usual lockdown, removing posting privileges from non-admin member groups. Since the board had been empty, I also use the IPB board on/off switch to turn it off altogether - no point even leaving behind a view access since this was an unused board. Luckily the IPB admin panel, while a bit crude in a 2003-dated IPB version, did have a few interesting statistics I could check out before commencing with the great purge.
The flow of members and threads had started late December 2015, with low volumes throughout January. I guess the bots had propagated the address over time slowly into their network, after discovering their spam was getting through. From February the volumes surged to hundreds of posts daily. There's no reason to suspect anything less than exponential growth would have continued to take place, had I not gotten incredibly lucky and set up my weekly backup task at just the right time to reveal the build-up of a tidal wave of spam, before it severely compromised my server.
It's not just that my bandwidth and storage would have been overwhelmed, it's scary to imagine what kind of crap could have ended up on the site that I could potentially have been liable for. Now I managed to plug the hole within just over 2 months of exposure.
I was ever so happy that an "empty board" feature existed in the admin panel, which got rid of all the spam in one fell swoop. Pruning the userbase took a little more effort, and I once accidentally removed the admin user with the IPB controls (yeah, oops...). Still, after restoring the admin user manually by way of ad-hoc sql, and a few more bulk runs on the Delete User tool in the IPB, I had cleaned up every user except the admin. Problem solved.
At the end of it, I pulled a manual sqldump. 13 megabytes.
On the bright side, seems the server as a whole is still not too exposed to crap, despite running on quite old software - after all, the leak was really mostly a configuration error on my part. Still, I've known I should probably rebuild the server and recode everything in the not-so-distant future. You know, maybe pick up some new but inexpensive, fault-tolerant, passively cooled, energy-efficient hardware. Get a new O/S and infrastructure (apache, php, sql) and finish by rewriting the sites not to run on a more than a decade old IPB.
I still need to do all of those things. I don't consider it just a chore, mind you. It should actually be reasonably fun. As long as I can do it in a way that the old system runs on the side, so I can take my time with it. But I sure have been putting it off, no doubt. Guess this was a reminder that I really can't put it off indefinitely. |
