[Techtalk] Maintaining a production system

[Kai MacTane]

At 10/10/01 11:52 AM , Subba Rao wrote:
>I have a linux system that is rolled out into production. It is mounted
>onto the rack and humming away. Now I would like to upgrade the kernel.
>How do I go about upgrading the kernel on a system that is in operation?
>Is it kind of the weekend NT/VMS/MVS maintainence window thing? ;-)

If you're slick, and you know what you're doing (i.e., you're *sure* your 
new kernel will compile and run properly), then you could conceivably do it 
at any time, simply letting users know that there will be a 5-15 minute 
downtime at such-and-so time/date.

For example, if you're simply upgrading to the latest stable kernel 
release, and you've done kernel upgrades before, then you could probably 
get away with a minimal-downtime situation. ("Minimal-downtime" meaning, 
roughly, "however long it takes the machine to go down and then come back 
up" -- say 30-60 seconds for a modern rackmount box. But you might as well 
tell the users 5-15 minutes, just to give yourself a little extra room.)

Depending on what the user base for the system is (local file/print/MP3 
server, for example), you may be able to get away with a few minutes of 
downtime in the middle of the day. Otherwise, you're looking at the 
late-night/weekend maintenance window.

If you've never done a kernel reinstall before, you may want to try it 
first on an expendable machine, if you've got any around. (My first kernel 
upgrade was on a production machine; I didn't have anything else available. 
It went smooth as silk, by why take the risk if you don't have to?) Then 
you can become confident enough to run it on a production machine.

Did you need a list of the actual steps to perform? That, I can provide 
too, after a bit of thinking and remembering (and maybe getting a couple of 
people on this list to remind me of the one or two steps I've forgotten!).

                                                 --Kai MacTane
----------------------------------------------------------------------
"Playing dead and sweet submission,
  Cracks the whip deadpan on cue."
                                                 --Siouxsie and the
                                                   Banshees,
                                                  "Peek-a-boo"