[Techtalk] undeletable files

[Raven, corporate courtesan]

Heya --

Quoth Walt (Tue, Mar 26, 2002 at 03:01:43PM -0500):
> >I'm curious why you're not just nuking the whole thing and
> >reinstalling. Is there stuff on there you can't afford to
> >lose?

	Throwing in my two cents with everyone else's -- this really is
what you should do if you ever want to trust that system again.  Not
just trusting it for your own files and system integrity, but also
trusting it to not DDoS other machines, spy on the rest of your network,
harvest passwords, get your account canned by your ISP, send threatening
mail from youraccount at yourdomain.com to the President, thereby earning
you some really unfriendly attention... there are lots of really nasty
things that can happen to you because of a hacked box.
 
> Ummm hrrm. I probably would do that except
> that the server is offsite now which is making
> maintenance and whatnot a real pain. And
> actually, it's kind of the opposite with the files
> it has: there's nothing super-important on it,
> but I am not thrilled with having to go through
> all the reinstalling mess either. User accounts,
> preserving email, web pages, dns, fun fun. *Grrrr*

	It's awful to do, but it really is worth it in the long run.
Back up what you can (no binaries), and if you restore anything from
backups of the system-in-a-hacked-state, check everything you put back
on there to make sure it's okay.  Force new passwords for your users
when the box comes back online.  It's painful, but I've seen boxes
re-hacked almost immediately after coming back online because the black
hat had a user account's passwords.

	If you are bound and determined on saving the system the way it
is, there are things you can do to make it less of a risk.  Checksum
every binary.  Boot from a boot disk or cd, and use the tools on that to
get all new binaries, libraries, etc. and install them all by hand.
(That in and of itself is probably more work than just reinstalling.)
Get new libs, then build a new kernel from scratch.  Make sure you have
the latest version of all your networked services.

	But I would really reinstall instead.  Even with all that work,
you'll never be *sure* you got everything.  The last time I tried to
save a system that had been compromised without reinstalling, there were
things hiding in the kernel that I'd never seen before.  (And a kernel
module to keep me from seeing them.)  I had to admire the black hat, as
much as I was angry at them.  Impressive skill, turned to an unfortunate
result.  The keystroke logger would have been very easy to miss.  I
ended up reinstalling anyway, after two weeks of trying to save that
system.

> >[files] altered by crackers include: ls, top, ps, w, who, and even syslogd.
> 
> I don't think ls was, because I could see the
> files in /dev/.../bdos though ps may have been
> and the intruder was definitely running his own
> syslog daemon.

	You could see the /dev files with what?  Ls on the compromised
hard drive?  I wouldn't trust it.  Even if the checksums match, md5sum
may be trojaned to lie to you, or the trojaned ls may have been tailored
to be the same size as the original.  (Check MAC times with known good
media -- that will give you a better clue.  And, as someone else has
suggested, chkrootkit sometimes finds rootkits.  It's batting about .600
for the systems I've seen that I know to have been rootkitted.)
 
Cheers,
Raven 

"Incoming packet over rabbit. SYN."
"Incoming packet over duck. quACK!"
  -- me and Tiff, flinging stuffed animals and tech humor