Inter-VRF Routing with VRF Lite

By stretch | Monday, March 29, 2010 at 4:29 a.m. UTC

In Intro to VRF lite, we looked at how virtual routing and forwarding (VRF) instances can be employed to logically separate the layer three topologies of unrelated entities sharing a single physical infrastructure. VRFs work at layer three much like VLANs work at layer two, by assigning interfaces to a virtual domain isolated from other virtual domains at the same layer. This is a simple concept to grasp so long as each of the VRFs are maintained in isolation. However, things get a bit more complex when you need to route traffic from one VRF to another. Consider the following topology, which might be found at a small business park or college campus.

The network pictured serves three customers: we'll call them customers Red, Green, and Blue. Each customer has its own Internet connection, housed in building one. Customer Green has employees in building two, customer Blue has employees in building three, and customer Red has employees in both buildings two and three. Additionally, VOIP service and connection to the PSTN (represented for now by a loopback interface) will be provided in the future by the network operator to customers Red and Green, but not Blue. Buildings two and three are connected to building one via 802.1Q trunk links. All three customers rely on the same physical network infrastructure for connectivity, but the traffic from each customer must be isolated from the others for security reasons. The IP prefixes in use are as follows: Red: 172.16.0.0/16 Green: 172.17.0.0/16 Blue: 172.18.0.0/16 VOIP Services: 192.168.99.0/24

The design question posed by this scenario is as follows: how can we provide local VOIP connectivity to some customers while keeping all of the customers isolated from one another?

Modifying the Multilayer Switches' SDM Template

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/ Page 1

Before we can implement VRFs on our multilayer switches (Catalyst 3550s, in this case), we have to modify how the internal ternary content-addressable memory (TCAM) is partitioned to store IP routes. Observe what happens if we attempt to enable VRF with the default SDM template: S1(config)# ip routing S1(config)# ip vrf Red S1(config-vrf)# %L3TCAM-3-SIZE_CONFLICT: VRF requires enabling extended routing VRF routes require that an additional unique identifier, a route distinguisher, is prepended to each VRF route in the routing table (we'll cover this in more detail shortly). To accommodate for this, the TCAM must be repartitioned. (This step is only necessary when configuring VRFs on certain multilayer switches.) S1(config)# sdm prefer extended-match Changes to the running SDM preferences have been stored, but cannot take effect until the next reload. Use 'show sdm prefer' to see what SDM preference is currently active. S1(config)# ^Z S1# show sdm prefer The current template is the default template. The selected template optimizes the resources in the switch to support this level of features for 8 routed interfaces and 1K VLANs. number number number number number number of of of of of of unicast mac addresses: igmp groups: qos aces: security aces: unicast routes: multicast routes: 5K 1K 1K 1K 8K 1K

The template stored for use after the next reload is the default extended-match template. The selected template optimizes the resources in the switch to support this level of features for 8 routed interfaces and 1K VLANs. number number number number number number of of of of of of unicast mac addresses: igmp groups: qos aces: security aces: unicast routes: multicast routes: 5K 1K 1K 1K 4K 1K

Note that the supported number of unicast routes has been halved from 8K to 4K. We need to apply this change to and reload all three switches before proceeding.

Creating VRFs
We need to create four VRFs on S1: one for each of the three customers, and one for the shared VOIP network. Each VRF must be assigned a unique route distinguisher. A route distinguisher is simply a unique number stored beside each route in the routing table to associate it with its VRF and maintain uniqueness should two or more VRFs have overlapping address space (for example, if all three of our customers were addressed out of the 10.0.0.0/8 network).

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 2

There are two formats in which we can define a route distinguisher: <ASN>:<number> or <IP address>:<number>, where number is an arbitrary decimal number. Which method we choose is arbitrary. While the format of route distinguishers is an important design consideration in "real" VPNs, in VRF lite (i.e. VRFs without MPLS) it is only a locally-significant value. We'll use the <ASN>:<number> format with the private ASN 65000 (which will also be used for our BGP configuration later in the article) and number customers sequentially beginning from 1. S1 ip ! ip rd ! ip rd ! ip rd ! ip rd routing vrf Red 65000:1 vrf Green 65000:2 vrf Blue 65000:3 vrf Shared 65000:99

Next, we assign the physical and logical interfaces to VRFs and address them appropriately. We're using a loopback interface in place of a physical interface on S1 to emulate the shared VOIP network, however there is no effective difference with regard to VRF configuration. Note that interfaces must be assigned to a VRF before being addressed; assigning an interface to a VRF wipes any IP addresses already configured on that interface. S1 vlan 16 vlan 17 vlan 18 ! interface Loopback99 description VOIP Services ip vrf forwarding Shared ip address 192.168.99.1 255.255.255.0 ! interface FastEthernet0/1 no switchport ip vrf forwarding Red ip address 172.16.1.2 255.255.255.252 ! interface FastEthernet0/3 no switchport ip vrf forwarding Green ip address 172.17.1.2 255.255.255.252 ! interface FastEthernet0/5 no switchport ip vrf forwarding Blue ip address 172.18.1.2 255.255.255.252 ! interface FastEthernet0/13 switchport trunk encapsulation dot1q

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 3

switchport mode trunk ! interface FastEthernet0/16 switchport trunk encapsulation dot1q switchport mode trunk ! interface Vlan16 ip vrf forwarding Red ip address 172.16.0.1 255.255.255.0 ! interface Vlan17 ip vrf forwarding Green ip address 172.17.0.1 255.255.255.0 ! interface Vlan18 ip vrf forwarding Blue ip address 172.18.0.1 255.255.255.0 The physical trunk interfaces F0/13 and F0/16 are not assigned to a VRF, because they are not routed interfaces. VRF configuration is only relevant at layer three. We can use show ip vrf to examine the VRFs and their assigned interfaces: S1# show ip vrf Name Blue Green Red

Default RD 65000:3 65000:2 65000:1

Shared 65000:99 S1# show ip vrf interfaces Interface IP-Address VRF Fa0/5 172.18.1.2 Blue Vl18 172.18.0.1 Blue Fa0/3 172.17.1.2 Green Vl17 172.17.0.1 Green Fa0/1 172.16.1.2 Red Vl16 172.16.0.1 Red Lo99 192.168.99.1 Shared

Interfaces Fa0/5 Vl18 Fa0/3 Vl17 Fa0/1 Vl16 Lo99 Protocol up up up up up up up

We only need to create two VRFs on each of the two other switches, as each connects only two customers, and assign the appropriate interfaces. S2 ip ! ip rd ! ip rd ! routing vrf Red 65000:1 vrf Green 65000:2

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 4

vlan 16 vlan 17 vlan 216 vlan 217 ! interface FastEthernet0/13 switchport trunk encapsulation dot1q switchport mode trunk ! interface Vlan16 ip vrf forwarding Red ip address 172.16.0.2 255.255.255.0 ! interface Vlan17 ip vrf forwarding Green ip address 172.17.0.2 255.255.255.0 ! interface Vlan216 ip vrf forwarding Red ip address 172.16.2.1 255.255.255.0 ! interface Vlan217 ip vrf forwarding Green ip address 172.17.2.1 255.255.255.0 S3 ip routing ! ip vrf Red rd 65000:1 ! ip vrf Blue rd 65000:3 ! vlan 16 vlan 18 vlan 316 vlan 318 ! interface FastEthernet0/13 switchport trunk encapsulation dot1q switchport mode trunk ! interface Vlan16 ip vrf forwarding Red ip address 172.16.0.3 255.255.255.0 ! interface Vlan18 ip vrf forwarding Blue ip address 172.18.0.3 255.255.255.0 ! interface Vlan316 ip vrf forwarding Red ip address 172.16.3.1 255.255.255.0 http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/ Page 5

! interface Vlan318 ip vrf forwarding Blue ip address 172.18.3.1 255.255.255.0

Configuring OSPF
Next we'll configure OSPF to exchange routes among the three buildings. Three independent OSPF processes are to be run, one per customer VRF. OSPF configuration here is pretty straight forward, as we can simply place all interfaces in area 0 within each VRF. S1 router ospf 1 vrf Red network 0.0.0.0 255.255.255.255 area 0 ! router ospf 2 vrf Green network 0.0.0.0 255.255.255.255 area 0 ! router ospf 3 vrf Blue network 0.0.0.0 255.255.255.255 area 0 S2 router ospf 1 vrf Red network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan216 ! router ospf 2 vrf Green network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan217 ! S3 router ospf 1 vrf Red network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan316 ! router ospf 3 vrf Blue network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan318

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 6

We can verify that OSPF adjacencies have been appropriately established with show ip ospf neighbor. Note that this command is not VRF-specific. S1# show ip ospf neighbor Neighbor ID 172.18.3.1 172.17.2.1 172.16.2.1 172.16.3.1 Pri 1 1 1 1 State FULL/DR FULL/DR FULL/BDR FULL/DR Dead Time 00:00:31 00:00:32 00:00:32 00:00:31 Address 172.18.0.3 172.17.0.2 172.16.0.2 172.16.0.3 Interface Vlan18 Vlan17 Vlan16 Vlan16

At this point, all customers in all buildings should have full connectivity within their respective VRFs. S2# show ip route vrf Red Routing Table: Red ... 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks C 172.16.0.0/24 is directly connected, Vlan16 O 172.16.1.0/30 [110/2] via 172.16.0.1, 00:00:31, Vlan16 C 172.16.2.0/24 is directly connected, Vlan216 O 172.16.3.0/24 [110/2] via 172.16.0.3, 00:00:31, Vlan16 S2# show ip route vrf Green Routing Table: Green ... 172.17.0.0/16 is variably subnetted, 3 subnets, 2 masks O 172.17.1.0/30 [110/2] via 172.17.0.1, 00:00:37, Vlan17 C 172.17.0.0/24 is directly connected, Vlan17 C 172.17.2.0/24 is directly connected, Vlan217 S3# show ip route vrf Red Routing Table: Red ... 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks C 172.16.0.0/24 is directly connected, Vlan16 O 172.16.1.0/30 [110/2] via 172.16.0.1, 00:01:04, Vlan16 O 172.16.2.0/24 [110/2] via 172.16.0.2, 00:01:04, Vlan16 C 172.16.3.0/24 is directly connected, Vlan316 S3# show ip route vrf Blue Routing Table: Blue ... 172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks C 172.18.3.0/24 is directly connected, Vlan318 C 172.18.0.0/24 is directly connected, Vlan18 O 172.18.1.0/30 [110/2] via 172.18.0.1, 00:01:05, Vlan18

Configuring Multiprotocol BGP

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 7

Multiprotocol BGP (MP-BGP or BGP-MP) will be used only to exchange routes between VRFs, and therefore needs to be configured only on S1. We'll use the private AS number 65000 for the BGP process. Because all of our routed interfaces on S1 have been assigned to VRFs, we'll create a loopback interface not in a VRF to keep BGP from complaining about not being able to find a router ID. (In the real world, one might consider using the IP of a management interface as the BGP router ID.) A router ID can also be configured per BGP address family. S1(config)# interface loopback0 S1(config-if)# ip address 192.0.2.1 255.255.255.255 To configure MP-BGP, we configure a separate address family within BGP for each VRF and simply redistribute all connected and OSPF-learned routes within that VRF. S1 router bgp 65000 no synchronization bgp log-neighbor-changes no auto-summary ! address-family ipv4 vrf Red redistribute connected redistribute ospf 1 vrf Red no synchronization exit-address-family ! address-family ipv4 vrf Green redistribute connected redistribute ospf 2 vrf Green no synchronization exit-address-family ! address-family ipv4 vrf Blue redistribute connected redistribute ospf 3 vrf Blue no synchronization exit-address-family ! address-family ipv4 vrf Shared redistribute connected no synchronization exit-address-family We can verify that each BGP address family now maintains routes for its respective VRF. (To view the BGP routes for only a specific VRF, use the command show ip bgp vpnv4 vrf.) S1# show ip bgp vpnv4 all BGP table version is 47, local router ID is 192.0.2.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 65000:1 (default for vrf Red) *> 172.16.0.0/24 0.0.0.0 0 32768 ?

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 8

*> 172.16.1.0/30 0.0.0.0 0 *> 172.16.2.0/24 172.16.0.2 2 *> 172.16.3.0/24 172.16.0.3 2 Route Distinguisher: 65000:2 (default for vrf Green) *> 172.17.0.0/24 0.0.0.0 0 *> 172.17.1.0/30 0.0.0.0 0 *> 172.17.2.0/24 172.17.0.2 2 Route Distinguisher: 65000:3 (default for vrf Blue) *> 172.18.0.0/24 0.0.0.0 0 *> 172.18.1.0/30 0.0.0.0 0 *> 172.18.3.0/24 172.18.0.3 2 Route Distinguisher: 65000:99 (default for vrf Shared) *> 192.168.99.0 0.0.0.0 0

32768 ? 32768 ? 32768 ? 32768 ? 32768 ? 32768 ? 32768 ? 32768 ? 32768 ? 32768 ?

Enabling VRF Route Import and Export

Our final task is to configure route import and export among the Red, Green, and Shared VRFs. VRF route import and export are configured as two explicit, independent functions. Under VRFs Red and Green, we need to: Export local routes Import routes from VRF Shared

Under VRF Shared: Export local routes Import routes from VRFs Red and Green

To do this, we configure each VRF with import and export route targets. A route target is similar in format to a route distinguisher, but serves a different purpose. Whereas any given VRF route in our scenario has exactly one route distinguisher to uniquely identify it, a route can have zero or more route targets as arbitrary identifiers for reference by import and export policies. This allows for much greater flexibility than if only the route distinguisher was used. Although used only locally in our example, route targets are communicated with MP-BGP neighbors as extended communities. Route targets can be defined in the same formats as route distinguishers. Here each route target will reuse the same format and value of its VRF's route distinguisher, just to keep things simple. However, remember that there is no direct relation between the two. Additionally, although each VRF is exporting only one route-target in our example, multiple route-targets can be exported simultaneously (resulting in multiple communities attached to an MP-BGP route). ip vrf Blue rd 65000:3 ! ip vrf Green rd 65000:2 route-target export route-target import ! ip vrf Red rd 65000:1 route-target export route-target import ! ip vrf Shared rd 65000:99 route-target export route-target import

65000:2 65000:99

65000:1 65000:99

65000:99 65000:1

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 9

route-target import 65000:2 Now VRF Shared has routes from VRFs Red and Green, and VRFs Red and Green each have the route from VRF Shared (it may take a minute for the routes to appear in the table): S1# show ip route vrf Shared Routing Table: Shared ... 172.17.0.0/16 is variably subnetted, 2 subnets, 2 masks B 172.17.1.0/30 is directly connected, 00:00:24, FastEthernet0/3 B 172.17.0.0/24 is directly connected, 00:00:24, Vlan17 172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks B 172.16.0.0/24 is directly connected, 00:00:24, Vlan16 B 172.16.1.0/30 is directly connected, 00:00:24, FastEthernet0/1 C 192.168.99.0/24 is directly connected, Loopback99 S1# show ip route vrf Red Routing Table: Red ... 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks C 172.16.0.0/24 is directly connected, Vlan16 C 172.16.1.0/30 is directly connected, FastEthernet0/1 O 172.16.2.0/24 [110/2] via 172.16.0.2, 00:05:38, Vlan16 O 172.16.3.0/24 [110/2] via 172.16.0.3, 00:05:38, Vlan16 B 192.168.99.0/24 is directly connected, 00:00:44, Loopback99 S1# show ip route vrf Green Routing Table: Green ... 172.17.0.0/16 is variably subnetted, 3 subnets, 2 masks C 172.17.1.0/30 is directly connected, FastEthernet0/3 C 172.17.0.0/24 is directly connected, Vlan17 O 172.17.2.0/24 [110/2] via 172.17.0.2, 00:05:43, Vlan17 B 192.168.99.0/24 is directly connected, 00:00:49, Loopback99 VRF Blue, however, still contains only its own routes, as we did not configure import or export for this VRF: S1# show ip route vrf Blue Routing Table: Blue ... 172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks O 172.18.3.0/24 [110/2] via 172.18.0.3, 00:06:02, Vlan18 C 172.18.0.0/24 is directly connected, Vlan18 C 172.18.1.0/30 is directly connected, FastEthernet0/5

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 10

Lastly, we need to configure the VRF OSPF processes to redistribute BGP-learned routes to OSPF peers on S1: S1(config)# router S1(config-router)# S1(config-router)# S1(config-router)# ospf 1 vrf Red redistribute bgp 65000 subnets router ospf 2 vrf Green redistribute bgp 65000 subnets

These routes are now propagated via OSPF to S2 and S3 within their respective VRFs. On S3, we can verify that VLAN 316 (VRF Red) has access to the VOIP network, but VLAN 318 (VRF Blue) does not: S3# show ip route vrf Red Routing Table: Red ... 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks C 172.16.0.0/24 is directly connected, Vlan16 O 172.16.1.0/30 [110/2] via 172.16.0.1, 00:10:15, Vlan16 O 172.16.2.0/24 [110/2] via 172.16.0.2, 00:10:15, Vlan16 C 172.16.3.0/24 is directly connected, Vlan316 O E2 192.168.99.0/24 [110/1] via 172.16.0.1, 00:10:15, Vlan16 S3# show ip route vrf Blue Routing Table: Blue ... 172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks C 172.18.3.0/24 is directly connected, Vlan318 C 172.18.0.0/24 is directly connected, Vlan18 O 172.18.1.0/30 [110/2] via 172.18.0.1, 00:25:17, Vlan18 S3# ping vrf Red 192.168.99.1 source vlan316 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.99.1, timeout is 2 seconds: Packet sent with a source address of 172.16.3.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms S3# ping vrf Blue 192.168.99.1 source vlan318 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.99.1, timeout is 2 seconds: Packet sent with a source address of 172.18.3.1 .....

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 11

Success rate is 0 percent (0/5) The final configurations of all three switches are made available here: S1.txt S2.txt S3.txt

Posted in Routing, VPN

http://packetlife.net/blog/2010/mar/29/inter-vrf-routing-vrf-lite/

Page 12