[Techtalk] Re: Server was hacked into; looking for tips on how to secure it

[jennyw]

On 2/24/03 12:57 AM, "Raven Alder" <raven at oneeyedcrow.net> wrote:
> Um... I wouldn't post that to a public mailing list, if I were
> you.  While most of my interactions with people from this list have been
> positive, for all we know there could be a thousand silently lurking
> black hats on list.  And keeping a hacked box online after you know it's
> been owned can possibly open you up to legal liability if the box is
> used to launch further attacks.  (I know you have a firewall ruleset
> that disallows outgoing traffic... but they could get around that,
> depending on what rules you have, or compromise another box on your
> network and go out from that.)

I'm taking it down now. If anything useful came through, hopefully it came
through in the last 36 hours.

I thought I was safe because the Sonicwall uses SPI, so theoretically any
traffic initiated from inside the box won't be able to get out at all, but
if traffic were initiated from the outside, outgoing packets that are part
of that session would be allowed.  It is on the DMZ port and there are no
other machines on the DMZ port.  Would someone still be able to launch an
attack from it?

> Look at your logs for each service you were running.  Many
> common attacks will leave telltale log entries that you can use to
> identify the attacked service and what exploit was used.  Buffer
> overflows often show up like this, and many common Web attacks will
> generate messages in the error_log or access_log.

I'm looking through the logs now.  I made two backups of the computer while
it was offline -- one using Norton Ghost and then two backups using rsync.
Unfortunately, I made the first (partial) backup using rsync while booting
the server itself; I wanted to make the backup offline (which is why I
resorted to Ghost first), but couldn't think of a way to do it.  Then
halfway through the rsync, it occurred to me that I could use Knoppix, which
is what I did for the second (full) backup of the server.  Of course, since
I have the Ghost backup, I could probably restore the image, boot into
Knoppix, and then run rsync from there.  I may end up doing that.

> This is, of course, assuming that the logs have not been wiped.

The logs may have been tampered with, but they're present.  Since they
didn't replace ps or anything too fancy, it might be that they didn't play
with the logs, either.  I can always hope! ;-)

> At this point, I would suggest figuring out what you're most
> interested in.  If you've messed with the machine then you won't be able
> to construct a forensic chain of evidence that would hold up in a US
> court.  (IIRC, you are in the US, yah?)  Obviously you don't want your
> machine to be used to attack anyone else or as a launching pad to attack
> the other machines on your network.  I would seriously consider
> unplugging that Ethernet cable and then doing analysis from the console.

I'll probably do the analysis from the computer I rsynced to -- unless
there's an advantage to running the system in its compromised state?

> Check your firewall logs and see if you have anyone scanning
> that particular machine near the time of the hack.  That might give you
> a starting point.  Do you know with any precision when the hack
> happened?

Unfortunately, I get a bunch of port scans every day.  I do have a general
idea of when the attack happened, so I'll take particular note of that range
of time, but if they got through through the Web server, it might not have
required a port scan at all to prepare for their attack.

> Hopefully your logs will tell you which service was attacked.
> If you log to an external host as well, I would compare your local logs
> to the ones on the remote host to see if anything was edited or removed.

I only have had one Linux box on the Internet at once; I should probably
consider setting up another one (electricity consumption is an issue,
though, hence my other questions about ITX which will hopefully allow me to
run a lower power consumption server).

> My money would be on BIND or on a web vulnerability.  But it
> could be anything.  Also, are you sure that the server was owned?  There
> are webforms that get abused by spammers to send their mail through
> without requiring actual server compromise -- versions of formmail
> are a common target, for example.

I'll check out the BIND vulnerabilities.

> http://www.securiteam.com/securitynews/Formmail_pl_Can_Be_Used_As_An_Open_Mail
> _Relay.html
> http://www.linuxfw.org/feature_stories/fingerprinting-http-page3.html

The only thing that could have been used was PostNuke, and most of the
modules were not available (although someone who knew PostNuke well could
have accessed them with a URL).  But since I found other processes in ps
including one run as root, I suspect if anything they used the Web server to
get a shell on my computer rather than using PostNuke to send mail.

>> With the new system, I'm installing Postfix with SMTP-Auth. This will help
>> with spamming if they're not careful, but that's hardly much.  I'm also
>> installing Integrit, which will hopefully let me know what files get
>> modified. But that's only finding out that I've been attacked, not how it
>> was done.
> 
> If you haven't already blown away the system and want to try to
> find out what happened, taking a forensic image and analyzing it with
> The Coroner's Toolkit might be helpful.  But the more you have
> changed/looked at/touched/rebooted, the less likely you are to be able
> to find out what happened.

I'll check out Coroner's Toolkit.

> Given the www-data processes, I'd check anything on your Web
> site that sends mail.  Forms, CGI, any automated process... anything
> that runs as your www-data user.

I'll also ask on the PostNuke forums.

Any other suggestions on how to prevent breakins? I'm also looking into
Snort. One reason I want to put an IPCop firewall up is to have Snort
running on it; at this point I doubt firewalling makes a difference to
whatever this attack was.

Thanks so much!

Jen