[Courses] [Security] The useful netstat

[Raven, corporate courtesan]

Heya --

Quoth Rosie and jennyw:
> > sound so good to me. And I also like the suggestion "How can I tell if my
> > linuxbox has been hacked". With a Follow-up "How do I clean up my hacked
> > linuxbox".
> 
> Yes! That would be great. The box I have my Web sites on got hacked (it's
> BSD) and I didn't discover it immediately. I *think* I cleaned out
> everything, but it's hard to tell whether they left any surprises. I found
> out because I saw a file that I didn't recognize, and then used last to see
> when they had been in. Using last probably isn't the best way to detect
> whether someone's been in your system, though!

	Fortunately, this fits really well into our topic of the week.
Hurrah for serendipity.

	In order to know when your box has been hacked, you have to know
what your box is like under normal conditions.  Something's got to
trigger the "that ain't right" reflex.  [grin]  There are times when the
hack is really obvious -- things like your website being replaced by
"Chinese hackers overcome pitiful US defenses!", to paraphrase a recent
common phenomena.  Other times, it's not so obvious.  One of my servers
was hacked this past summer, with surpassing subtlety.  If I hadn't been
really paranoid (and somewhat lucky), I never would have realized they
were there.
 
	So, what do you look for?  Well, let's start with the list of
services that your box is running.  Most of the time, successful remote
hacks are exploits against a service that your box is running.  (I'm
talking attacks that give the black-hat priviliges and access to your
server, not ones that will just drop your server off the Internet map
for a while.)  So it makes sense not to run more network services than
you need.  Running everything under the sun increases your chances of a
hole being found in one of those services, and your box becoming
vulnerable to attack.

	On most Linux systems, you'll have a command called netstat.
Simply put, it lets you know the status of your network.  Running just
the bare command, like so:

[root at linuxbox /root]# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address
State
tcp        0      0 www.linuxbox.:2944 astro.dal.net:ircd
ESTABLISHED
tcp        0      0 www.linuxbox.o:ssh dragonhome.humanit:1594
ESTABLISHED
tcp        0    256 www.linuxbox.o:ssh scarlett.demon.di:39735
ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  0      [ ]         DGRAM                    157112 /dev/log
unix  0      [ ]         DGRAM                    272  

shows you the open sockets -- the connections between a local IP/port
pair and a remote IP/port pair.  It doesn't include the servers, which
is what we're really interested in here.  But we can see that someone on
this box is connected to dal.net for IRC, and that two users are ssh'd
in -- one from dragonhome.humanit[something], port 1594, and one from
scarlett.demon.di[something], port 39735.  (The latter is me connected
to the box to use it as an example.)  You can see that the remote
hostnames are truncated if they're too long.  Both of these are.

	Much more helpful is to run netstat with the -pl options.  This
shows you the server processes that are listening too, and the processes
that are associated with them.

[root at linuxbox /root]# netstat -pl
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address
State       PID/Program name
tcp        0      0 *:ftp                   *:*
LISTEN      532/tcpserver
tcp        0      0 *:smtp                  *:*
LISTEN      462/tcpserver
tcp        0      0 *:www                   *:*
LISTEN      436/httpd
tcp        0      0 *:ssh                   *:*
LISTEN      421/sshd
raw        0      0 *:icmp                  *:*
7           -
raw        0      0 *:tcp                   *:*
7           -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program
name    Path  

	So we can see that this server is running ftp, mail, www, and
ssh.  I'm using tcpserver to run my ftp and mail daemons, and Apache and
ssh are running standalone, without a superserver.

	There are lots more options out there -- check out the man page
for netstat for a rather exhaustive list.  If anyone feels like posting
the netstat info from their system for comment, we can go over what you
should and shouldn't see here.  For the most part, if you don't know
what it is, you probably shouldn't have it listening on a port here.

	Try killing and restarting some of your services.  You'll see
the ports appear and disappear from netstat if they're running
standalone.  (Some superservers answer for the service they maintain
whether it's up or not.)

	Really common things that are hackable that you often see here
by default:

	Rpcbind and other rpcservices -- portmap.  If you're
deliberately running them, make them only accessible to your internal
network or whoever needs them with a firewall.  If you don't know what
they are, you probably don't need them.

	Sendmail.  Make sure you have a recent version, and keep it
patched.  Securing sendmail is a whole other week's worth of discussion.

	Bind.  Older versions are notorious for having holes.  If you
don't need it, turn it off.

	Xwindows services.  Unless you're acting as an X server to
remote Xwindows clients, X doesn't need to be listening on a port.  You
can forward X connections over ssh if you need to -- it's much safer.

	To remove something from here, you first have to kill the
process and then remove it from your startup scripts (usually in
/etc/init.d and then symlinked to /etc/rc?.d and/or /etc/rc.d/rc?.d).

	If you want to run these services but don't want to have them
listening on a port, there's usually some way to do that within the
configuration for the service itself.  We can get into detail for those
if anyone has a specific service in mind.

	There are other programs that do the same sort of thing -- lsof,
for example.  But one thing that's really important is to check your
system both from the inside and from the outside.  Use nmap or a similar
tool to portscan yourself.  You should see the same list of services
available.  If you see a service in one and not the other, start to be
suspicious.  (I'll cover nmap in another mail -- this one's getting
long.)

> I'd love to learn more about stuff like tripwire (never used it) or anything
> that scans for changes to the list of running processes.
 
	I was thinking that would be a good topic for discussion too.
It's one of the few ways you can be relatively sure that nothing else
has been changed on your system.

> It'd also be great to have a general overview of the types of things
> security things that exist, so we have an idea of what's out there. I mean,
> just a discription of what a firewall is, what an IDS is, what a honeypot
> is, what a whatever kind of thing tripwire is is, etc. Not a treatise, but
> just like a sentence that says that so and so exists. We don't have to
> actually talk about these things, but it's good to know what words to look
> for in case we have an issue in the future.

	A firewall is a device that will selectively intercept packets,
and forward, drop, or reject them according to configuration.  An IDS is
a system that looks for patterns that might be attacks among the network
traffic, and notifies you when it thinks it sees something attack-like.
A honeypot is a system that's deliberately left hackable and then
watched.  People do this either to learn about black-hat methods and
behaviour, or to lure them away from the rest of your network.  Tripwire
is an integrity checker.  It makes sure that the files that were there
last week haven't been monkeyed with, and are still the same files
today.

	I hope we'll get into all of these in the coming weeks. 
 
> It would also be good to know the weaknesses of products. A lot of people
> say ipchains is a weak firewall and that iptables is better. I don't know
> anything about iptables so I couldn't even say. Also, there are tons of
> Linux-based firewalls out there like Smoothwall that would be nice to know
> about.

	Short answer: ipchains is a packet filter.   It decides whether
to forward, drop, or reject packets based upon the ports and IP
addresses alone.  Iptables lets you do more than that because it allows
stateful filtering.  It can keep track of sessions that originate from
inside the firewall, and allow all traffic in a session to pass through,
but block traffic trying to come in from outside that's not part of a
session.  With the strings patch to iptables, you can even filter based
on the contents of the packet -- blocking Nimda and Code Red attempts at
the firewall, for example.

	We'll be all over this next week.
 
Cheers,
Raven 
 
"I am so very girly."
  -- RavenBlack, on 'feminine' and 'masculine' traits