Yesterday I was just casually looking over my WordPress administration panel and out of habit I clicked on the Users page even though I already knew in my mind there was gonna be only one user turns out there were in fact _two_ users. Mine, and… the automated script’s one that just exploited my site.
As to how it happened exactly is something that I’m still investigating but the culprit seems to be that there might be an unknown vulnerability/exploit in the W3 Total Cache plug-in. What I just said right now is pure speculation.
As you would expect from the name of the plugin its function is to just cache and thus improve the site performance. This was a new plug-in I introduced to the site when I moved to LXD/LXC.
Why is W3 Total Cache the main suspicious actor in this case? Because I’ve been blogging for over five years and the only new introduction I have made to the site was just that plugin. Everything else is pretty much vanilla. I also use a plug-in for Amazon Simple Email Service (SES) but I have been using that plug-in for over 3-4 years as well.
Because the site is deployed within a linux container no other site was really impacted by it. It was just completely isolated within the container.
I still had to change MySQL user password and the WordPress user password as it has already harvested my password. It was harvesting any login attempt.
But that wasn’t enough for me. I noticed that the attack created multiple database tables. That left a sour taste in my mouth because I don’t know what else it did in the database without looking at it closely. At this time I haven’t looked at it.
All I did on my end was just zip tainted files. Grab all the logs I could. Export WordPress using the exporter provided by WordPress itself. Inspect there wasn’t any hidden vulnerability in them, which thankfully there wasn’t.
The hack also did something very peculiar with the .htaccess and index.php file. I could not delete or edit the file. There was something executed that thwarted any efforts to write or delete the file to the point that not even root could delete them. Even if you deleted the root folder containing folder and recreated it it would appear back. There was some sort of issue at the permission layer. I don’t think the container was ever compromised to the point it already took over the configurations of the server. There was no breach in the auth.log. There was no sense of history anywhere in the logs. If the automated attack was as sophisticated to the point it would cover it tracks then… I have no idea what to say but just feed my paranoia.
There was however something interesting. I forced the delete of the folder. Then restarted the container, recreated the folder and there was no longer any reference to .htaccess and the tainted index.php.
What was in the attack?
An insane amount of stuff.
Here’s a few screenshots of what I have seen so far:
What you see here is usually how PHP infected files look internally.
Once decoded the file would carry on gathering inputs and sending that input somewhere in the internet. The file would contain a lot of nested base64 encoded text so you wouldn’t just have to decode this it would show you some code then you’d meet new base64 text.
The automated attack failed in a lot of areas as well. As seen on these logs:
Overall… I’m completely annoyed this happened. I’m happy that I found out before something could happen I learned a lot from how this attack worked (not completely).
Apparently it was committing database dumps somewhere in the internet. Thankfully this is just a personal blog so there was no impact at all.
I use a password manager, namely 1Password, and because I don’t re-use password all I just had to do is change my password.
The automated attack would actually list your password publicly which makes it all scary. You would access these passwords on the license.txt or readme.html URL.
It also gave the attacker manual entry points in case the person just wanted to come by the site and do whatever they wanted.
… there’s still so much to learn from this attack. I know it embedded a virus (apparently) from what Bitdefender told me to one of the Gutenberg files.
The first thing you will notice about this exploit is that your site performance would just completely drop. Everything would be incredibly slow at first then it would speed up a tiny bit. But you would feel each page load to be really slow compared to when you did the first installation.
Wrapping this quick overview. I think there are several lessons learned on this and hopefully in the future I will share more details on the attack as I decode and learn more on how it behaved.
For now I’m just happy I was able to bring back my personal site quick.
I would just say that just because you are running your site on a container such as Docker or LXC it doesn’t really make it completely safe. However, you would minimize or just minimized a whole lot of damage if you had shared sites within the same user running on the same server. I’m still convinced that going with containers is the best way to go even if there’s a bit of overhead. But knowing that you are using one of the most popular blogging software ever known to the web it’s bound to be an attraction. Be wary of the plug-ins you choose.