[Techtalk] Re: Server was hacked into; looking for tips on how to secure it

[Raven Alder]

Heya --

Quoth jennyw (Mon, Feb 24, 2003 at 07:49:54AM -0800):
> I'm taking it down now. If anything useful came through, hopefully it came
> through in the last 36 hours.

	What I would do... unplug the box from the network in general
and connect it to your workstation via a crossover cable.  Then nmap it.
That way you can see whether it's listening on any ports that it
shouldn't be (that you're aware of).  That will give you a clue as to
whether your attacker installed a backdoor.  If you see any unknown
ports, try connecting to them with telnet or ssh, or throw them into
Google to see if they're well known backdoor/trojan ports.  It's pretty
common to find a high port that responds to telnet connections and gives
you a password prompt on connection.

	If you do find such a port, then look through your firewall logs
for attempted connections to that port.  If it's listening for telnet
connections on TCP 20202 or something, and you see one IP address
repeatedly trying to connect to TCP 20202 and nothing else, odds are
good that that's someone who knows how you were compromised and is
trying to get back in.
 
> I thought I was safe because the Sonicwall uses SPI, so theoretically any
> traffic initiated from inside the box won't be able to get out at all, but
> if traffic were initiated from the outside, outgoing packets that are part
> of that session would be allowed.  It is on the DMZ port and there are no
> other machines on the DMZ port.  Would someone still be able to launch an
> attack from it?

	That all depends on how the Sonicwall does stateful packet
inspection.  Sadly, I am not familiar with the Sonicwall product in
particular, so I don't know how it handles these things.  Perhaps
someone else here knows.  I can't see anything on their site that would
explain it.

	Basically, if the stateful packet inspection does not allow the
hacked box to initiate connections out, it's probably depending on the
fact that all normal TCP connections start with an outbound packet from
with the SYN flag set from the machine initiating the connection.  So if
your Sonicwall drops all outgoing SYNs from your hacked box, the hacked
box will not be able to start any new connections, such as requesting a
web page, initiating an FTP download, etc.

	However, dropping all outgoing SYNs is not the same as dropping
all outgoing traffic.  If you have a knowledgeable hacker who's logged
in to the box (more on how that is possible in a second), they could
send outgoing traffic that does not have the SYN flag set but is still
TCP.  Nmap with the -sF flag set, for example, will initiate a scan with
the FIN flag set but no SYNs at all.  So they could still FIN-scan other
networks from your box.  Alternatively, there are Denial of Service
tools that will send TCP traffic with the ACK flag set but no SYN...
that would also get by your firewall, and you could be used as an agent
for attacking other computers.

	If you are indeed dropping all outgoing traffic from the box,
then it is true that your box will not be able to be used for evil.
[grin]  Mostly because nobody could communicate with it at all.  In
order to establish any sort of connection, you need two way data flow.
And though there are various Stupid Packet Tricks you can play to
simulate that/spoof other hosts/etc., it's not the same as actual
communication with your compromised host.  So really, it all depends on
what traffic exactly your Sonicwall is dropping and what it's letting
through.

	Another possibility is that your attacker is not depending on
TCP to communicate with the compromised host.  They may be tunneling
their remote commands or control connection through ICMP or UDP.  There
are well known ways to hide a connection inside "DNS" UDP packets
(assuming you control both ends of the connection, which your attacker
would), so what is really a "start attacking 10.1.1.17 now!" command
looks like simple DNS queries.  Again, depending on exactly how the
Sonicwall does stateful packet inspection, it may or may not catch this.
Without more in-depth knowledge of how the Sonicwall works, I couldn't
say what its capabilities were.
 
> The logs may have been tampered with, but they're present.  Since they
> didn't replace ps or anything too fancy, it might be that they didn't play
> with the logs, either.  I can always hope! ;-)

	Worth looking at, at least, to see if there's anything
interesting there.
 
> I'll probably do the analysis from the computer I rsynced to -- unless
> there's an advantage to running the system in its compromised state?

	There are things that you can see from the compromised system's
state that you would not be able to see when just mounting the files on
another box.  Processes that started and were running on the compromised
system may not start, things like that.  You'll still be able to look at
log files and such, but if you're not running the same kernel and init
scripts, your list of processes and all will likely be different.

	Also, depending on your needs, you may want your analysis
station's kernel to not support modules.  This will keep it from loading
malicious Loadable Kernel Modules -- there are a non-trivial number of
LKM rootkits out there for Linux.  Allowing such things to be loaded can
severely hamper your knowledge of what's going on.  So I'd look at a
process list and a full portscan of the box in its compromised state,
and then mount the files on another machine and analyze them further
from there.
 
> Unfortunately, I get a bunch of port scans every day.  I do have a general
> idea of when the attack happened, so I'll take particular note of that range
> of time, but if they got through through the Web server, it might not have
> required a port scan at all to prepare for their attack.

	Yeah -- a smart attacker would have done any recon necessary
some time ago, and then come in from a different IP when they were
planning to make the attempts.  That would evade most NIDS tracking.
Slow scans, as much footprinting as possible, and then one targeted
strike from a different IP.  Really, your best bet might be to look at
all the traffic to that box from before the projected time of the hack
onwards, and see if you can pick out patterns.
 
> But since I found other processes in ps including one run as root, I
> suspect if anything they used the Web server to get a shell on my
> computer rather than using PostNuke to send mail.

	Sounds sensible.
 
> I'll check out Coroner's Toolkit.

	It's kind of intimidating at first and has a steep learning
curve (or at least, it did for me), but was well worth the effort.
 
> Any other suggestions on how to prevent breakins? I'm also looking into
> Snort. One reason I want to put an IPCop firewall up is to have Snort
> running on it; at this point I doubt firewalling makes a difference to
> whatever this attack was.

	If they come in and exploit a service that you're running and
allow through your firewall, then no, the firewall won't really make a
difference.  If you log permitted connections as well then you might be
able to see it... but logging everything eats disk space quickly on even
a reasonably busy network, and any permitted connection should show up
in the logs on that host anyway.  Firewalls are good for keeping out
people from services you do not want accessible, but they won't protect
you from things you do want people to be able to reach.  This is one of
the reasons why assiduous patching, log reading, host-based security,
etc.  are also important.  Firewalls are a useful tool for security
admins, but the best firewall in the world still won't protect you from
everything.

	I know I'm probably preaching to the choir here... [grins] 
 
Cheers,
Raven

"Don't give me whys and wherefores, reasons for surprise,
 I don't care for words that don't belong.
 And I don't care what you're called.  Tell me later, if at all.
 I can wait a long, long time before I hear another love song."
 -- Sisters of Mercy, "Some Kind of Stranger"