[techtalk] RedHat, glibc and your friend sash.

[Magni Onsoien]

Some may remember me complaining about glibc before, more specifically a
couple of weeks ago when a glibc-upgrade broke tcsh. A prompt reply and
fix from RedHat fixed my problems, so I was quite satisfied with their
service.

Now I am less satisfied again.

Due to more security holes in glibc, I decided to upgrade a little bit
on some of the servers we are responsible for. Since RedHat made a few
mistakes last time they upgraded glibc, I expected this errata to be nice
and working, so I could safely install it even at noon and on a
production server. So, upgrade, here we go!

I randomly chose butler to be my first upgrade-victim, maybe not _that_
smart concidered that he is the main mailserver, but heck, I trusted
RedHat and their RPMS now!

Ouuuppss.... [insert lots of "faen" which is a bad Norwegian word]
RPM suddenly quit.
error in loading shared libraries
libc.so.6: cannot open shared object file: No such file or directory

How nice of it.
RedHat has also - for reasons unknown to me - been so terrily smart that
their system consist almost 100% of dynamically linked binaries.
Exceptions are NOT ls, cp, mount, sh (link to /bin/bash) and cpio.
Actually all the binaries I checked in /bin and /sbin except rpm, fsck,
ldconfig, *.static (about 5 of them) - and sash.

Due to this it was impossible to install new libraries - rpm required 
cpio - impossible to mount a floppy disk with new libraries (impossible
or at least more complicated anyway, libc-2.0.7.so is about 3MB) etc.

Luckily we discovered sash:
       The  sash  program  is a stand-alone shell which is useful
       for recovering from certain types of system failures.   In
       particular, it was created in order to cope with the prob­
       lem of missing shared libraries or important  executables.

It has a great bunch of useful builtins: ls, mount, ln, grep, ed, dd,
cp. To make it even easier we already had a filesystem mounted on butler
and another RH-box, so it was quite simple to copy libc-2.0.7.so once we
had something to move it from the remote to the local filesystem. Tada:

> -dd if=libc-2.0.7.so of=/lib/libc-2.0.7.so
5997+1 records in
5997+1 records out
> cd /lib
> -ln -s libc-2.0.7.so libc.so.6 

Puh.

sash is bloody good for you (and your customers).
I will never ever upgrade glibc again. Never.

And I no longer trust RPM. This probably happened because / was too full
- only 6MB left - so in the middle of the upgrade rpm had removed some
old libraries and the suddenly discovered the lack of space and just
quit. Without replacing the deleted/removed/disappeared libraries. How
nice of it. I'll try to express to RedHat that I concider it RPMs job to
be sure it has enough space for an upgrade, then tell me before it
gracefully quits without touching my files. Just have to relax and get
my blood pressure and adrenalide level down to a healthy level first.



Magni :)
-- 
sash is very good for you.