[Techtalk] Is it the server???

Thu Apr 4 16:33:04 EST 2002

At 01:14 PM 4/4/2002, Raven, corporate courtesan wrote:
>Heya --
>
>Quoth Michelle Murrain (Thu, Apr 04, 2002 at 12:15:46PM -0500):
> > This is SO strange.
>
>         But those are the interesting problems!  [grin]

I agree, this is very interesting, and I'm learning a lot. I'm just not 
happy getting phone calls from my colleagues whose mail is delayed!

> > I'm still getting a lot of "unreachable" errors in the tcpdumps, and
> > there is no question that the null connections match with the
> > unreachable errors.  Mail is still getting delayed.
>
>         Good diagnostic information.  Is the ICMP unreachable error
>actually *to* 192.168.1.1, or is that just an address you swapped in to
>substitute for your real IP?

No - I didn't swap that.

>  If that's the actual error, that's telling
>you that your box doesn't know how to get to that private-space network.

Hmmmm - so maybe it  *is* the server.

>If your box is in the normal-routable-space, it shouldn't be trying to
>access private space IP directly.  (Normally.  I would need to know more
>about your network topology to say for sure.)

The way the network is set up at this moment, it has a private IP, with the 
192.168.1.1 as the gateway/firewall/router. I haven't yet set up a DMZ. I 
just got the T1, and moved the server, and the first priority was getting 
people up and running. I've pretty much figured out what I'm going to set up.

>         Can you ASCII art up a network map for us, complete with your
>mail server, all boxes you're seeing the errors on in tcpdump, and your
>DNS server?  And let us know which boxes actually have routable IPs and
>which don't?

                                 mailserver #2
                                         |
                                         |
web/mail server #1  ----------------- router/firewall 
----------------------   Internal network
                                         |
                                         |
                                 ISP 
CSU/DSU

Web/mail server #1 is the only one seeming to have trouble, although ping 
floods to unreachable addresses from mailserver 2 seem to drop about as 
many packets. Mailserver 1 and 2 are the ones with routable IPs.

Both servers are running named, but I'm in fact using my ISPs DNS right now 
- I hadn't finished the DNS configurations yet.

>         Large packets dying and small ones being okay is usually either
>connectivity or firewalls.  Your ISP needs to take care of that, since
>you won't have access to the relevant equipment.  Was the pingflooding
>showing packet loss because their pipe isn't as big as yours?  (If
>you're on a T3 and you pingflood a machine on a T1, you're going to see
>serious packet loss simply because their pipe can handle only 1/30th the
>bandwidth that yours can, and the other 29/30ths are getting dropped.)

I'm on a fractional T1 - so I kinda doubt that's a problem, since many of 
the unreachable hosts are big deal mail servers, like aol's or topica's.

>         Have you tried replacing all the physical connections in your
>path with known good ones?  (I don't think it's your side of things, but
>it never hurts to be sure.)  Change ports on the switch or hub, as well
>as swapping out the Ethernet cable.

There are two things I could try, and I tried them both - the port on the 
router (it has an 8 port hub) and the ethernet cable. No difference.

> > Suggestions? I have a trouble ticket in with my ISP, but they seem a bit
> > clueless.
>
>         Honestly, that's not unexpected.  When I was in a
>customer-support job at an ISP, I would have been totally stymied by
>something like this.  Most people with the necessary understanding of
>networking and protocols won't take a customer-facing phone-answering
>job.  Escalate within your ISP if necessary, keep sending them all the
>evidence you can to help them troubleshoot, and hopefully you'll
>eventually get someone clueful on the phone.

I actually have done that, and they are doing some testing. I haven't heard 
back from them yet.

>
> > Just 'cause I was curious, I did a ping flood from a different box
> > within the same network, and guess what - way, way, less packet loss.
> > (like 1%) So it seems like it's the server, right? If so - is it a bad
> > ethernet card?  Or can something else be going on?
>
>         It could be the server, or it could be connections from your
>local network to those remote sites.  Do you still see the packet loss
>when going from the non-server machine on your local network to the
>machines you're having the issues with on the remote network?  Do they
>get their DNS from the same place?

There is a little packet loss between machines inside the network. They all 
do get their DNS from the same place.

>         I am wondering if this is what's happening.
>
>Remote mail server begins to connect to local mail server.
>Local mail server queries the DNS server, to make sure the remote mail
>server is who it says it is.  "Where's this remote mail server?
>192.168.1.1."
>DNS server says, "Uh, what?  I don't have a mapping for that.
>ServFail."
>Mail server says "Fine.  Router, connect me to 192.168.1.1".
>Local router says, "The hell?  I don't know how to get to 192.168.1.1!
>ICMP error -- network unreachable."
>
>         Lather, rinse, repeat.
>
>         There are a few problems with this hypothesis, though.  If this
>kept happening, mail would never get through.  Obviously, it is getting
>through, if somewhat delayed.  The configuration error could be on the
>DNS server (IME, ServFails usually are borked DNS setups), or on the
>mailserver (asking for bad information).  But if it's on the mailserver,
>I wonder why it was working before and isn't now.  Maybe the change in
>IP address needs to be reflected somewhere, and hasn't been?  So there's
>something else going on, too.  If you have tcpdump info from a good SMTP
>connection from those same servers, could you post that, too?  (And let
>me know what addresses have been changed to RFC 1918 addys and what
>haven't.)

Um, I'm a little lost here. Lemme give you some more details. The server 
has an internal IP address, which is mapped to an external address via NAT 
on the router/firewall (a netopia box). The IP address of the server having 
trouble (the one I moved), is the same internal 192 address, it's just 
mapped to a new external IP (since it's with a new ISP, and different net).

Here's a set of tcpdumps for successful SMTP packets:

16:33:08.174659 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: . ack 1 win 16060 <nop,nop,timestamp 
1650370 488638534> (DF)
16:33:08.653713 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: P 1:94(93) ack 1 win 32120 <nop,nop,timestamp 
488638568 1650370> (DF)
16:33:08.653743 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: . ack 94 win 16060 <nop,nop,timestamp 
1650417 488638568> (DF)
16:33:08.654026 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 1:29(28) ack 94 win 16060 
<nop,nop,timestamp 1650418 488638568> (DF)
16:33:08.724165 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: . ack 29 win 32120 <nop,nop,timestamp 488638589 
1650418> (DF)
16:33:08.731954 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: P 94:296(202) ack 29 win 32120 
<nop,nop,timestamp 488638589 1650418> (DF)
16:33:08.732257 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 29:85(56) ack 296 win 16060 
<nop,nop,timestamp 1650425 488638589> (DF)
16:33:08.830401 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: . ack 85 win 32120 <nop,nop,timestamp 488638600 
1650425> (DF)
16:33:08.837643 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: P 296:355(59) ack 85 win 32120 
<nop,nop,timestamp 488638600 1650425> (DF)
16:33:08.837788 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 85:116(31) ack 355 win 16060 
<nop,nop,timestamp 1650436 488638600> (DF)
16:33:08.931197 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: . ack 116 win 32120 <nop,nop,timestamp 
488638610 1650436> (DF)
16:33:09.144431 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: P 355:404(49) ack 116 win 32120 
<nop,nop,timestamp 488638631 1650436> (DF)
16:33:09.144582 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 116:122(6) ack 404 win 16060 
<nop,nop,timestamp 1650467 488638631> (DF)
16:33:09.207941 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: P 404:454(50) ack 122 win 32120 
<nop,nop,timestamp 488638637 1650467> (DF)
16:33:09.209088 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 122:1146(1024) ack 454 win 16060 
<nop,nop,timestamp 1650473 488638637> (DF)
16:33:09.209340 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 1146:1937(791) ack 454 win 16060 
<nop,nop,timestamp 1650473 488638637> (DF)
16:33:09.349963 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: . ack 1937 win 32120 <nop,nop,timestamp 
488638652 1650473> (DF)
16:33:09.350001 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 1937:1940(3) ack 454 win 16060 
<nop,nop,timestamp 1650487 488638652> (DF)
16:33:09.409900 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: . ack 1940 win 32120 <nop,nop,timestamp 
488638658 1650487> (DF)
16:33:09.438836 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: P 454:508(54) ack 1940 win 32120 
<nop,nop,timestamp 488638660 1650487> (DF)
16:33:09.439832 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: P 1940:1946(6) ack 508 win 16060 
<nop,nop,timestamp 1650496 488638660> (DF)
16:33:09.592612 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: P 508:560(52) ack 1946 win 32120 
<nop,nop,timestamp 488638666 1650496> (DF)
16:33:09.592925 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: F 1946:1946(0) ack 560 win 16060 
<nop,nop,timestamp 1650511 488638666> (DF)
16:33:09.593716 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: F 560:560(0) ack 1946 win 32120 
<nop,nop,timestamp 488638666 1650496> (DF)
16:33:09.593750 nanuuq.ursa-minor.com.1221 > 
xx5.mail.simpleservers.com.smtp: . ack 561 win 16060 <nop,nop,timestamp 
1650511 488638666> (DF)
16:33:09.639324 xx5.mail.simpleservers.com.smtp > 
nanuuq.ursa-minor.com.1221: . ack 1947 win 32120 <nop,nop,timestamp 
488638680 1650511> (DF)

.Michelle

---------------------------------------
Michelle Murrain
tech at murrain.net
AIM/Yahoo Messenger:pearlbear0
ICQ:129250575
http://www.murrain.net/ for pgp public key