If all you wanted was another simple tutorial, then you'd be better off somewhere else.
This is not just another PowerPoint presentation. Neither is it endless lines of config scrolling
past your eyes.
No. I'm sorry to say that this is NOT the video you will be watching.
This is a cautionary tale about routing and vPC. I bring this to light in hopes that you
can avoid the pitfalls yourself.
Early in 2017, a colleague and I were given the task of installing a new pair of Nexus
switches into a production data centre. This included migrating a few vPC attached
routers and firewalls. No problem! We can do that! There'll be a
small outage, but how bad could it be? So I planned it out, configured the switches,
and prepared myself to spend a night in the data centre.
During the migration, we saw something strange. The routers and firewalls weren't peering
with the switches. The routers were up, and we could ping them,
but OSPF peering was unstable.
Let me use a lab demo to show you what was happening.
vPC has already been configured on a pair of Nexus 9000 switches, and it is working
correctly. There are two routers connected with vPC
As you can see, there is nothing special about the vPC configuration. It is all quite normal
On the OSPF side, Switch-1 is peering with three different neighbours. It is peering
with Switch-2 twice in this example Switch-2, however, has problems. It is peering
with Switch-1, but the two routers are stuck in INIT and EXSTART
If we debug the OSPF adjacencies, we can see that the neighbours are looping between INIT
and EXSTART. The routers would show similar errors in their debugs
We spent hours troubleshooting through the night, but we couldn't find the cause of the problem
Finally, the time came. The decision was made. We began to implement the rollback plan.
We later returned to the office to put some thought into it.
After a couple of days of research and testing. We finally found our solution.
To understand what causes this issue, we first need to have a quick review of LAGs and vPC.
A LAG is a bundle of several physical links, connecting two devices. Each flow of traffic
gets assigned to one of the physical links.
A vPC is an enhanced LAG, where two Nexus switches are used instead of one. Each flow
of traffic is still assigned to one link in the vPC.
Sometimes traffic meant for Switch-1 will arrive on Switch-2. Switch-2 can have a feature
enabled called peer-gateway. This lets it respond on Switch-1's behalf.
So, how does this fit into our problem?
To answer this, we need to have a high-level overview of how
Routing Protocols interact with vPC to answer this.
Routing Protocols (aside from BGP)
expect routers to be directly connected. To enforce this, their packets are sent with
a TTL of 1.
Throughout the process of forming neighbours, a series of multicast and unicast packets
are sent.
If Switch-2 receives some of the packets meant for Switch-1, it naturally decrements the
TTL. The TTL is now zero, and the packet is discarded.
This causes the neighbour relationship to not form completely. Next, we'll have a look
at some ways to solve this problem.
All of this so far has just been for a router peering WITH the Nexus OVER vPC. As Cisco's
routing over vPC document shows, there are some other topologies that we must consider.
The exact behaviour will depend upon your Nexus platform, so be sure to use their document
when designing your solution.
The first topology we'll consider is where routers want to peer with each other. The
routers are connected to the switches with vPC.
This scenario works, as the routers are connected at layer-2. The switches do not decrement
the TTL of the peering traffic. The switches are not peering with anything
in this topology, they are only used for transport.
But what if you want the switches to participate in dynamic routing? In this topology, the
router does not use vPC. Instead it is connected to an orphan port.
If you want to peer with both switches, you may need to add an additional layer-2 inter-switch
link. This depends on your platform, so check the Routing over vPC document.
If you do need this link, the VLAN used for peering would need to be pruned from the peer-link,
to force peering traffic over the new link.
Of course, you can use a hybrid approach. In this topology, a router peers with one
or both switches over an orphan port. It is also able to peer with the vPC connected
router. An alternative to to change the links to routed
ports with the no switchport command. These are routed orphan ports, and allow the routers
to peer with the switches. Now the routers are more than one hop away
from each other. They can no longer peer with each other.
There are some times when the previous 'out of the box' solutions won't work. Imagine
a case where you have a cluster of ASA's. The recommended way to attach these is with
VSS or vPC. There are a few options here. First, think
about whether you really need dynamic routing. If the cluster is on the edge of the network,
you may only need a default route outbound, and some static routes inbound.
If you do need dynamic routing, you may consider running BGP. In many cases, this may not be
an ideal solution, as it would need to be tuned, and you would probably need redistribution
into another routing protocol. There is one final option, and I think this
is the one you've been waiting for...
Let me show this one in the lab. As you probably remember, this is a very standard vPC config
vPC itself is working, and the interfaces are up
Now for the magic command. In the vPC config, we add the 'layer3 peer-router' command on
both sides. This enables vPC connected routers to peer with the switches
We can now see that the OSPF relationships are stable on both switches.
This is a simple solution, but there is also a simple catch. You need the right NXOS version,
and if you have line-cards, there are some hardware constraints.
Make sure to check out Cisco's routing over vPC document to see what is supported for
your platform.
That's it for routing over vPC. I hope this will help you avoid the problems that I encountered
at the beginning of the video. If you want to know more about vPC, I recommend
the 'How vPC Works' and 'vPC Configuration' videos. If you still want more, head over
to networkdirection.net Please remember to subscribe or use Twitter
to be notified of new content.