[Techtalk] Looking for ideas on how to test status of SSH TCP tunnel

[Almut Behrens]

On Thu, Nov 03, 2005 at 10:55:18AM -0800, Bruce McPeek wrote:
> 
> I am planning to script the SSH connection on the client side to
> reconnect should the connection drop. This should be a fairly trivial
> task. Unfortunately I have seen long running SSH tunnels in a state
> where they appear to be connected but no data flows through the tunnel
> or to the login shell. I would like test for this condition in my script
> but I am unsure which approach to take.
> 
> I could conceivably try to connect through the tunnel to the server
> using some utility but which one? I could conceivable try using the
> Perforce client but would rather not consume a license to do this.
> Perhaps I could open have a second tunnel open just to test the
> connection, but what would be good to use?

For detecting connection related problems, ssh itself provides two
mechanism: (a) TCP keep alives, and (b) sending periodic 'ignore'
messages, i.e. null packets. The latter is typically the preferred
method, because it's more reliable and faster (and not spoofable).

The related config options have changed their names with time and
ssh versions, so check your ssh_config manpage for any of
ServerAliveInterval/ServerAliveCountMax, TCPKeepAlive, KeepAlive,
ProtocolKeepAlives, or some such.

This merely tests the connection, though  (TCP, or the encrypted
channel), which means it wouldn't necessarily detect some other obscure
problem with the tunnel (e.g. one of the auxiliary sockets no longer
responding, etc.) -- i.e. it doesn't do a full roundtrip check via the
forwarded port(s) themselves.
OTOH, this might be considered a somewhat academic issue...

Anyhow, if things don't work as expected, for whatever reason (or you
generally don't have too much faith in those ssh builtin mechanisms),
you could of course just use a second tunnel and write two little
'heartbeat' scripts that communicate through that tunnel (via sockets).
For example, the client-side script could periodically send some random
number, which the server simply echoes back.  So, if the client is
returned its self-generated numbers within some timeout period, you can
be pretty sure that tunnel is OK.
(The added benefit of this approach would be that you could also more
easily detect server-side whether the client is still alive, and take
appropriate measures otherwise, if that should be an issue...)

In theory, you could also plug in two generic port forwarders / proxies
_in series_ with the actual tunnel, but they would need to be
absolutely transparent with respect to what's being transmitted.
In addition to forwarding every piece of original data to be tunnelled,
they would exchange seperate messages of their own to test if the other
side is still alive.  That would probably be the the most reliable
technique, though somewhat harder to implement...

IOW, I'd first take a closer look at ServerAliveInterval... ;)

Cheers,
Almut