[Techtalk] End key no longer goes to EOL

[Almut Behrens]

On Thu, Jan 10, 2002 at 11:47:58PM -0500, Jeff Dike wrote:
> grs at theneteffect.com said:
> >   KeyRelease event, serial 29, synthetic NO, window 0x1600001,
> >     root 0x35, subw 0x0, time 1404496059, (198,393), root:(501,532),
> >     state 0x0, keycode 103 (keysym 0xff57, End), same_screen YES,
> >     XLookupString gives 0 characters:  "" 
> 
> > but the 3rd line is identical on both machines, which is the part I
> > would've expected to be different, no?
> 
> Yeah :-)
> 
> If they both give 'keysym 0xff57, End', then I dunno.  That's the part you
> could have used xmodmap to match up.

well, at least you've verified that the X server is still sending the
same keycode/keysym. Unfortunately, that's not the whole story ;)

Other things involved are the following:

The terminal emulator (xterm, kterm, rxvt or whatever) gets this
keysymbol from the X server and transforms it into a so-called
"terminal escape sequence" for the special keys like cursor-keys, home,
end, etc.  This is a multibyte sequence starting with the "escape
char", which is ASCII char 27 decimal (or "033" octal, or "1b" in
hexadecimal notation), followed by several other characters (usually
up to 4).

You can see what your terminal is generating by issuing "cat -v" and
then typing the key(s) in question. You should see an escape sequence
something like ^[[8~ or ^[[4~ for the "end"-key (could be something else,
still, depending on your terminal). The ^[ in these sequences is one of
the ways to represent the escape key (27), similarly to writing ^C for
key combination Ctrl-c, etc. (other ways to represent the escape char
are for example "\E"). The other chars in the escape sequence [8~ are to
be taken as is, i.e. the 3 chars "[" (ASCII 91), "8" (56) and "~" (126).
So the escape sequence here consists of the 4 ASCII chars/bytes:
27 91 56 126, or in hex notation: 1b 5b 38 7e.  I'm mentioning this in
such excrutiating detail, because this can definitely become confusing
at times.
BTW, instead of "cat -v" you could also use "od -c" for similar output.
Both commands can be aborted with ^C, as usual.

The idea behind these escape sequences is to be able to represent/transmit
special characters in the limited 7-bit ASCII-space 0-127, which would
otherwise soon run out of unique codes.

Now, you could compare the escape sequences you get in your old and new
RH installations. They could be different, but I expect that they are
the same, too, just like the keysyms from the X server. But even if they
did differ, that wouldn't mean too much on its own. Why?

Unix has a very flexible mechanism for handling the multitude of
incompatible different terminals that have accumulated in the 30+ years
of unix history -- in fact so flexible that it usually doesn't work as
intended ;)  Anyway, it is supposed to work as follows:

The terminal announces its name via the environment variable TERM (that
you can of course also set manually). So if you type "echo $TERM", you'd
probably get something like "xterm" or so.  The application that's using
the terminal (pine in your case) reads out $TERM and looks up a list of
associated terminal features in the terminal capability database.
Among those are the above mentioned key-to-escape-sequence mappings and
various other control sequences that are being used to move the cursor
around in the display, scroll lines, set colors, etc. (A text-mode
application should thus always be written to dynamically determine
which control/escape sequences to use, and never hard-code any assumed
mappings. This is usually made sure by using the appropriate libraries,
such as ncurses, slang, readline, etc.)

Actually, there a two versions of this database: termcap and terminfo.
termcap is the ancient one. It's a single huge file (usually /etc/termcap)
that contains all the terminal descriptions in ASCII form, so you could
use your text editor to view and modify them.  terminfo is the newer
concept, that tries to do things in a more flexible and sane way. It
consists of a bunch of directories (usually under /usr/lib/terminfo/)
that contain many small binary files, one for each terminal. They are
sorted in directories named by the first letter of the terminal name,
so you'd find the terminfo file for "rxvt" under /usr/lib/terminfo/r/,
for example. There are also ASCII versions of those terminfo files that
can be compiled using "tic", the terminfo compiler, to yield the
required binary files.

One of the main problems with this is that those feature descriptions
have to kept in sync with how the terminals and terminal emulators are
really behaving. They are not being read as a sort of config file by the
terminal emulator that would setup how the terminal operates (although
this would be a reasonable idea). This is mainly due to the fact that
the original terminals were hardware devices that simply couldn't read
a config file and adjust themselves accordingly.
This means that the description has to be made to match the terminal.
How the terminal itself operates is configured elsewhere. Depending on
the terminal this can be one of: being hardcoded in the sources, via X
resources, settings in the desktop environment and just about any
typical unix way like config files, commandline options and environment
variables.  Thus, confusion is preprogrammed.

Just in case you feel like checking or messing around with the terminal
capability descriptions, you could look for an entry "@7=..." (e.g.
@7=\E[4~ -- "@7" being the very intuitive abbreviation for the end-key)
somewhere in xterm's capability string, which is a colon-separated list
of such assignments split over several lines. That's for termcap.
terminfo uses different abbreviations and a different format. As you
probably expected, the manpages have all the details -- in particular,
"man terminfo" will tell you more than you ever wanted to know :)

As there are two such databases, there's also always the question of
which one to use, or rather "which one is being used?". As usual, this
depends on how it's configured -- else it wouldn't be unix ;)  I guess
I don't have to repeat the usual list of possibilities.  A typical
default is to first look for termcap and then for terminfo, in case
termcap is not found. And this finally brings us back to your original
problem:
It *could* be that you simply don't have a /etc/termcap any more in
the recent version of RH, so that terminfo is being used, which might
contain other (to avoid having to say which ones are the correct ones)
capability entries...  At least for SuSE I've heard rumours that they
no longer ship termcap with their most recent version -- not sure,
though, and even less sure about RH.
So that's where I would start looking. And don't hesitate to make use
of strace (also see the recent posting by Laurel Fan on that) to find
out what's really going on, e.g. whether termcap or terminfo is being
read and such...

I know, I could have simply said in recipe form: try copying over
/etc/termcap (and possibly libtermcap.so.*) from the old RH version,
which might in fact have worked right away.  Yet, I thought that a
deeper understanding of the mechanics behind this could help you to
debug such problems yourself next time you encounter them, which you
certainly will ;)  

Hoping to have clarified things a bit,

Almut


PS: other commands in the immediate vicinity of the issue include:
stty, tset and tput -- so you might want to read up on those, in case
the termcap stuff doesn't get you any further...