[Techtalk] 3 IP blocks, one cisco switch

[Maria Blackmore]

On Fri, 3 Jan 2003, Jen B wrote:
> Hey TechTalkers,

Hi :)

> In one of my co-located racks I've got a cisco 2900XL switch.
> Drop #1 has 2 seperate IP blocks routed to it (209.73.y.xxx &
> 209.10.z.xxx)
> VLAN1 has an IP of 209.73.y.xx1 for cli administration.
> The switch happily services both IP blocks.

The switch itself doesn't actually care what the IP addresses are, it just
switches packets.

Switches work on OSI layer 2, which is packets on the wire
IP is OSI layer 3, which is stuff inside the packets on the wire.

To summarise, the switch itself only looks at where the packets are going,
and makes sure that they get there.

> I managed to get the cage right next to mine too, and with that comes the
> IP block 209.11.w.xxx. I plugged the drop from the new cage into the
> switch next door and the macine I IP'd on the new block (209.11.w.xxx)
> worked fine but the servers with addresses on the other blocks became
> unavailable from outside the cage.

When you say "drop" are you referring to an IP transit connection?

To clarify, as soon as you plugged the connection to the outside world
from the second cage into the switch, the outside world was no longer
available to the machines in the IP blocks for the first connection?

hmm

are both connections from the same transit provider?
if they are, then I think the reason for this happening is simple, it's
STP (Spanning Tree Protocol) doing it's thing.

STP is designed to stop loops from being formed on ethernet networks. [1]

Loops on ethernet are bad, when a broadcast happens on an ethernet switch
it is "flooded" to all the ports on the VLAN. This means that if you have
a loop, the broadcast is constantly flooded to all ports because it goes
around in circles, this is bad.  It makes your network into a notwork, and
has been known to make devices attached to the network get their knickers
in a twist.

When STP detects a loop, it blocks one of the ports that would have formed
the loop, so that there remains only a single path between any two points
on the network.  This is neat, because it means that you can make
redundant links on your network, and if one fails everything will
immediately start using the other one.  A good example of this is at work,
where there is a switch called "ayl-sw-1" which is connected to another
switch called "ayl-core-1", due to its importance, there are two
connections, but only one is used.

ayl-sw-1 is a cisco 2924XL, probably much like your own, Jen.

to look at what's going on, log in and type "show spanning-tree", this is
an example from ayl-sw-1 with the two upstream links to ayl-core-1

Interface Fa0/21 (port 21) in Spanning tree 1 is FORWARDING
   Port path cost 10, Port priority 128
   Designated root has priority 32768, address 0001.3078.8b00
   Designated bridge has priority 32768, address 0001.3078.8b00
   Designated port is 9, path cost 0
   Timers: message age 2, forward delay 0, hold 0
   BPDU: sent 318, received 17129649

Interface Fa0/23 (port 23) in Spanning tree 1 is BLOCKING
   Port path cost 10, Port priority 128
   Designated root has priority 32768, address 0001.3078.8b00
   Designated bridge has priority 32768, address 0001.3078.8b00
   Designated port is 11, path cost 0
   Timers: message age 1, forward delay 0, hold 0
   BPDU: sent 13, received 17129342

port 23 is the second uplink, and as you can see, is being blocked by
spanning tree.

If, of course, it turns out to not be this, I'll have to have a little
think

> At the time, I didn't have the switch logging anywhere, and I can't
> recreate the scenario since it would make my VPN & critical databases
> unavailable.

If you really can't afford to try again, I can suggest two things.
1) setup VLANs on the cat 2924 and completely seperate the two
or
2) use the network connection from the first cage for the machines in the
second cage, and get rid of the network connection for the second.

> I have other racks with 2900 switches that have multiple network
> drops from different providers and didn't have to configure anything
> special.

*shudder*

It's kind of nasty having different IP subnets flying around in the same
broadcast domain in a production environment.  There's nothing technically
wrong with it i suppose, it just feels kind of icky, especially when
they're all externally routable IP addresses.

If you have many of these, why don't you get your own network together?
Get your own routers together and an IP range and an AS#, get a couple of
transit providers and let BGP take the strain :) then you'll never have
problems like this again :)  plus you will have consolidated quite a lot,
and could well end up getting bandwidth cheaper.

> Are there any notions of what I might be doing wrong here?

see above .. you may get some clues by looking at the ARP tables on the
machines involved

This is the kind of thing to troubleshoot in the whee small hours of the
morning on site :)  (so it doesn't matter when you break stuff :)

good luck :)

Maria

1: Let me know if you want more info about Spanning Tree Protocol, and
i'll see what I can do