Layer 2 extension using VXLAN with Brocade VDX

Nov 03, 2015 in BROCADE • NSX • VCS • VDX
6740 6940 nsx overlay vdx vxlan
4 min read

VXLAN. I’m sure you’ve heard a lot about it, and you might have even configured it. For those not in the know: it’s a fairly easy way to overlay L2 networks over a L3 infrastructure, allowing you to stretch a L2 domain between L3 PODs or even between sites. One of the applications could be vMotion for example, allowing you to transfer VMs on the fly between PODs.

In short VXLAN overlays are tunnels between VXLAN Tunnel End Points (VTEPs), with the underlying L3 infrastructure not having to know anything about VXLAN. One thing to note is that due to the VXLAN encapsulation, the frame size increases (+8 bytes) so you might have to adjust the MTU size on your routers as VXLAN packets should not be fragmented.

For those interested, here’s the RFC (RFC7348).

Brocade VDX6740 and VDX6940 platforms support layer 2 extension using VXLAN in two ways:

Layer 2 extension mode
VCS Gateway for NSX

The first option works without any controllers, and using the VDX platform as the VTEPs and to build the tunnels. The second option has the VDXs connect to the NSX Controller to automatically create tunnels in conjunction with NSX. This blog post focuses on option 1, but don’t worry: I’ll dedicate one of my next articles to NSX and the integration with VDX as this is becoming an increasingly hot topic.

There are a few ways you can set this up, check with your local SE what the best way is to do this.

I’ve got it running in our lab like this, with VDX 6940s as the spine, and 6740s as leaf switches:

What you see is two PODs, connected via L3. There’s a VXLAN gateway setup on each of the VDX 6940s in the different sites (RBridge 101 & RBridge 201), which are VTEP 1 and VTEP 2 respectively. I only have 1 device per site available, you could have multiple of these running in VRRP-E mode for example.

There are two servers which are connected to VDX 6740s in different sites, with a different VLAN per site but both in the same subnet. I could have used the same VLAN on both sides, but for illustration I used different VLANs to show that the one thing that really matters is the VXLAN Network Identifier (VNI).

To get the layer 2 extension to work, the following is required:

Layer 3 connectivity between the VTEPs.
Define what switch(es) you want to use as VXLAN gateway(s).
Specify site names, one for each site you want to connect to.
Define the VLAN to VXLAN Network Identifier (VNI) mapping,. This is a 1:1 mapping, the VLAN per site can be different but the VNI needs to match on both sides.
On each site, define the VLANs you want to extend.

To make it work, you need to configure this as follows:

RBridge 101

<th class="column-2">
  Explanation
</th>

<td class="column-2">
  Define the name of the overlay-gateway.
</td>

<td class="column-2">
  Type of the Overlay Gateway, either NSX or Layer2 Extension.
</td>

<td class="column-2">
  Specify the source L3 interface for the tunnel, either a Ve interface or Loopback interface.
</td>

<td class="column-2">
  Specify the RBridge IDs of the switches that need to be used for the VXLAN gateway instance.
</td>

<td class="column-2">
  Maps the VLAN to a VXLAN Network Identifier (VNI). This VNI needs to match on both sides!
</td>

<td class="column-2">
  Name of the site you want to connect to.
</td>

<td class="column-2">
  The IP address of the other site.
</td>

<td class="column-2">
  The VLAN(s) you want to extend between these sites.
</td>

<td class="column-2">
  Activate the configuration.
</td>

RBridge 201

overlay-gateway vxlan-gw1
type layer2-extension
ip interface Ve 250 vrrp-extended-group 250
attach rbridge-id add 201
map vlan 199 vni 5000
site POD1
ip address 192.168.150.100
extend vlan add 199
activate

We now have a VXLAN tunnel running over our L3 network, extending the L2 domain between the two PODs over VNI 5000 and allowing the servers to connect to each other.

Last but not least some verification commands, to check the status of our VLANs and the associated tunnel (one per VNI):

That’s it! Hope you learned a thing or two about getting started with VXLAN, feel free to reach out if you have any questions and stay tuned for some updates on this topic!

Related Articles