[Techtalk] Re: Server was hacked into; looking for tips on how to secure it

[Raven Alder]

Heya --

Quoth jennyw (Sun, Feb 23, 2003 at 09:07:58PM -0800):
> My server was hacked into. For the curious, it's 66.93.78.112 ... It is back
> up now in its hacked state. 

	Um... I wouldn't post that to a public mailing list, if I were
you.  While most of my interactions with people from this list have been
positive, for all we know there could be a thousand silently lurking
black hats on list.  And keeping a hacked box online after you know it's
been owned can possibly open you up to legal liability if the box is
used to launch further attacks.  (I know you have a firewall ruleset
that disallows outgoing traffic... but they could get around that,
depending on what rules you have, or compromise another box on your
network and go out from that.)

	I would suggest not publicly posting the IP.  If you want to
share a look at that box in its hacked state, make people e-mail you and
ask for the IP, and tell you where they'll be coming from to look at it.
That way you can separate out your curious friends from the attackers.

> I have tethereal running on it hoping to catch something that will
> give me more clues (if the person tries coming back) but I'm not too
> hopeful about that (especially since they'll quickly discover that all
> outgoing traffic is blocked by an external firewall). 

	How did you do that?  If you just blocked outgoing SYNs from
that IP, they can still launch UDP or ICMP based attacks.  If you
blocked SYNs, ICMP, and UDP, they can still launch ACK or FIN scans or
DoS.  And if you've blocked all outgoing traffic from that IP, you won't
see them if they do try to come back since the return traffic to them
won't be allowed and they will be unable to establish a TCP connection.
All you'll see will be the incoming SYN.

> The main issue is that I can't figure out how they got in.  This same
> person has gotten in twice (I assume it's the same, since they send
> spam for something called websalesjet), so obviously there's some
> vulnerability on the machine.

	Look at your logs for each service you were running.  Many
common attacks will leave telltale log entries that you can use to
identify the attacked service and what exploit was used.  Buffer
overflows often show up like this, and many common Web attacks will
generate messages in the error_log or access_log.

	This is, of course, assuming that the logs have not been wiped.  

	At this point, I would suggest figuring out what you're most
interested in.  If you've messed with the machine then you won't be able
to construct a forensic chain of evidence that would hold up in a US
court.  (IIRC, you are in the US, yah?)  Obviously you don't want your
machine to be used to attack anyone else or as a launching pad to attack
the other machines on your network.  I would seriously consider
unplugging that Ethernet cable and then doing analysis from the console.
 
> The first time, I had ipchains on the machine. This time I had an external
> firewall -- a Sonicwall.

	Check your firewall logs and see if you have anyone scanning
that particular machine near the time of the hack.  That might give you
a starting point.  Do you know with any precision when the hack
happened?

> I had opened only DNS, http, https, imap, imaps, and smtp.  I was
> running Postfix, Courier-IMAP, BIND9 (the first time I was running
> BIND 8), Apache, MySQL, and PHP4.  The only dynamic site I had up was
> PostNuke Phoenix, which is pretty recent. I took down all other
> dynamic sites in case they were the way they got in (I was running an
> older PostNuke, IMP, Gallery, and phpBB 1.2).

	Hopefully your logs will tell you which service was attacked.
If you log to an external host as well, I would compare your local logs
to the ones on the remote host to see if anything was edited or removed.

	My money would be on BIND or on a web vulnerability.  But it
could be anything.  Also, are you sure that the server was owned?  There
are webforms that get abused by spammers to send their mail through
without requiring actual server compromise -- versions of formmail
are a common target, for example.

http://www.securiteam.com/securitynews/Formmail_pl_Can_Be_Used_As_An_Open_Mail_Relay.html
http://www.linuxfw.org/feature_stories/fingerprinting-http-page3.html
 
> With the new system, I'm installing Postfix with SMTP-Auth. This will help
> with spamming if they're not careful, but that's hardly much.  I'm also
> installing Integrit, which will hopefully let me know what files get
> modified. But that's only finding out that I've been attacked, not how it
> was done.

	If you haven't already blown away the system and want to try to
find out what happened, taking a forensic image and analyzing it with
The Coroner's Toolkit might be helpful.  But the more you have
changed/looked at/touched/rebooted, the less likely you are to be able
to find out what happened.

	Given the www-data processes, I'd check anything on your Web
site that sends mail.  Forms, CGI, any automated process... anything
that runs as your www-data user.
 
> Anyway, thanks in advance for any suggestions!
 
	No problem!  Let us know what you find.

Cheers,
Raven