Saturday morning I got the urge to hunt for some parts on Xianyu and swap them into a desktop that had been in service for more than seven years and had lately been protesting about retirement. I was still messaging a seller when texts and emails started arriving with bills.
What happened
Sometime between 7 and 10 a.m., my site was hit by a DDoS attack. Traffic stats showed about 15TB over a few hours, with nodes from Asia-Pacific and Europe.
The whole site sits behind Alibaba Cloud's CDN. The bill came out to just over 8,000 yuan.
For storage I use Qiniu Cloud's OSS. That bill came out to just over 1,000 yuan.
I still use Alibaba Cloud for my domain and email, and occasionally for servers and cloud SMS tests. After talking with customer service, the outcome was tolerable. All pay-as-you-go services are now unusable until the bill is paid off. Annual services like the domain and server are unaffected, but automatic deductions no longer work, so I have to renew manually. Email was free anyway, so there is no arrears issue there.
On Qiniu I only use object storage. Since that is pay-as-you-go, and even though low-frequency access is free, I cannot use it until the account is cleared.
The damage to the site is considerable and annoying. All the rules I had built for the CDN and OSS stopped working. A lot of the related code turned into an unmaintainable pile.
Guessing at the reason
After thinking it over, there were two recent things that could have offended people in this circle.
The first was a blog aggregator site that had a site aggressively pushing a WeChat public account. I mentioned it in a group of bloggers, and the aggregator's owner promptly removed the account-promotion site. If that site's owner was also in the group, there would be a motive.
The second was a conversation on another blogger's site about handling malicious comments. I said the backend can see IPs, and probably the commenter's site as well, so if someone really meant harm, just DDoS them back. If that comment was heard by someone who took it personally, then anyone visiting that blogger's site would have a reason to attack.
The plan
The quickest fix would be to switch CDN and object storage providers. But I cannot predict whether the attack will come back, or when. Anyone familiar with DDoS/CC attacks knows I cannot even find out who did it. If there is more to come, changing vendors would just mean two more bills.
For now I have pointed DNS back to the server's origin IP. Worst case, I shut it down. The code is the painful part: many rules cannot be adapted to a local deployment. The code was changed on and off over several years. There are development records, but the mess is already there, and I do not have the energy to fix it all.
For access rules, I have temporarily blocked overseas visitors and added suspected high-frequency visitors to a blacklist.
The domain expires in January next year, and the server in May. If I am not too busy in January, I might replace the server, domain, and program entirely.
I cannot guess which outcome is the right one, but either way, this is a lesson. Maybe I will gradually reduce how much I interact.
09.05
More follow-up. The database still records images as being on Qiniu, with a lot of parameters that do not mean much. If I do not change that, continuing to call Qiniu will make all images disappear. If I do change it, the backend throws a lot of errors, and email notifications get dragged down too. After a day of poking around, the problem is temporarily solved. But I do not know when or where a new problem will pop up.
09.06
The effects continue. Most of the attacking IPs were in Europe. The 15TB of traffic drew attention from some small European companies: site builders, traffic generators, bulk email promoters. Some left comments on the site, some emailed me. Annoying.