Re: Interesting webserver intrusion (apache 1.3.31, mod_ssl 2.8.18, php 4.3.7)

["nathan c. dickerson" <nathan () pro net>]

Thanks for the reply!

>It would certainly help knowing what OS you are running Apache on. From
>the statement about running the latest kernel, I assume you are running

> Linux. I'm not familiar with that environment as I'm a BSD guy :)
> However, one thing that would definitely help are tighter firewall rules
> so that the attacker can not spawn shells back to them or log into an
> IRC channel for control.

Linux - 2.4.26 #2 Wed Jun 30 22:47:39 PDT 2004 i686 i686 i386 GNU/Linux

It was based from a basic Redhat 9 for the utils, but I compiled all the 
server software and kernel from source. I definitely want to move 
towards Slackware, or even OpenBSD, having used both at home, but the 
company uses Redhat in general.

Since then, I have blocked the common IRC ports, and the firewall was 
fairly tight(only allowing 4 ports in), but perhaps I could tighten it 
down even further.

> In my BSD world, I have Apache running in a jail. There is no outbound
> connectivity from that jail to the Internet (on servers that need
> certain outbound access, it is allowed but limited to those
> connections/IP that are required, nothing more). So even if the jailed
> httpd got compromised, there is no way to establish connection back out
> to the attacker or some IRC channel
>  
>
> Even if you can't jail Apache, you can tighten your firewall rules such
> that outbound access is restricted. Allow yourself to open the fw if you
> need to performance maintenance (downloading patches, etc), but lock it
> back up once you're done.
>  
I'll look into a setup like this. I am not exactly sure how the jails 
control outbound ports though. Apache needs to open the outbound to 
communicate, and a user exploiting apache and gaining it's permission 
could probably open up a port as well. How do the jails work? Is it like 
a proxy? I'll definitely have to read more about the subject.

> Good job catching that sucker, but it wasn't quite clear how you
> detected him. I always enjoy hearing from people how they detected
> attacks in progress. Perhaps you can shed some light on that. Please
> also list the other modules you have running on Apache so that we might
> be able to guess on how they got into the box.
>  
As far as apache, it be:

Apache/1.3.31 (Unix) PHP/4.3.7 mod_ssl/2.8.18 OpenSSL/0.9.7d

mod_auth_mysql, mod_php4, mod_ssl, mod_setenvif, mod_so, mod_auth, 
mod_access, mod_rewrite, mod_alias, mod_userdir, mod_actions, mod_imap, 
mod_asis, mod_cgi, mod_dir, mod_autoindex, mod_include, mod_status, 
mod_negotiation, mod_mime, mod_log_agent, mod_log_config, mod_env, 
mod_vhost_alias, http_core

Detection wasn't that special, having had dealt with hackers before, it 
wasn't a shock, although it was annoying and frustrating, since I 
thought the new web server was fairly secure.

I noticed a spike on the cpu with some monitoring software. High cpu 
tends to cause ftp launched via xinetd to respond slowly, which the 
monitoring software also detects. I pretty much keep a terminal open at 
home and at work, so I checked out what was going on.

ps -aux showed the process (perl using a high percent of CPU), and 
netstat -apn showed what was communicating with it. I noticed right away 
port 6667, and searched for the place they were storing their files, 
copied over all the scripts before they were deleted, killed the 
processes, and filtered the ips. Couldn't see anything in the logs, so I 
kept my eye on the server for any tiny spike in cpu, having a feeling 
they'd be back, and the hole still existed.

A week later, they tried it again, and I was hoping to get some decent 
logs to see how it was being done. At that point I had some IPs, some 
scripts, basic logs, but nothing pointing toward the hole, so I went to 
check these guys out on IRC to get a better idea of who they were. When 
one of the guys on the channel messaged me, I suggested that he stop, 
and since then, it has stopped.

I still want to track down the point of entry however;

Thanks,
Nathan