Understanding Virtual Router and

Virtual Systems

PAN-OS 6.0

Humair Ali
Professional Services





Content

Table of Contents
VIRTUAL ROUTER .................................................................................................................................. 5
CONNECTED ............................................................................................................................................. 8
STATIC ROUTING .................................................................................................................................... 9
OSPF ......................................................................................................................................................... 11
BGP ........................................................................................................................................................... 17
IMPORT RULE ........................................................................................................................................................... 21
REDISTRIBUTE RULE .............................................................................................................................................. 28
IBGP .......................................................................................................................................................................... 41
ROUTE REFLECTOR ................................................................................................................................................. 45
HIGH AVAILABILITY ........................................................................................................................... 49
ACTIVE-PASSIVE ...................................................................................................................................................... 49
Purpose of this Document

The purpose of this document is to explain how Virtual Router and Virtual system works and some example
on they need to be configured.

It is not a complete guide for every scenario but is aimed at giving pointers on how to achieve some of
Routing or Virtual System configurations

This document is mostly focus on examples where the Palo Alto Networks Firewall is configured in Layer 3
mode and not for Layer 2 or Vwire .
The Inter-vsys is inherent as most Virtual Routers are in their own Virtual Systems, so most of the Routing
that we will see are setup across Multi-VSYS

Lab Setup
All the commands and output shown in this document is based on the PA-4020 unless
specifically said otherwise

PANOS 6.0.10
PA-4020 in HA Active/Passive

Physical Setup

INTERNET

.22
.200
Ubuntu Desktop PA-VM-300
Default G W 10.193.56.1
10.193.57.x/24
10.193.58.x/24

Cisco SG500 Switch

Eth1/3 10.193.56.x/24
.2
1 3 5 7

POWER

FAN

ALARM
STATUS

HA

TEMP
PA-500

Default G W t o PA-500
10.193.56.2
2 4 6 8 MGT CONSOLE USB
Vsys1

PA-500
Vsys2
EBGP

Eth1/1 Eth1/2

.200 .2 .2
1 3 5 7 9 11 13 15 17 19 21 23

POWER ALARM

STATUS FANS
HA PWR 1
TEMP PWR 2
2 4 6 8 10 12 14 16 18 20 22 24 HA2

PA-4020

USB 1 / 2 MGT HA1

CONSOLE

PA-4020
Virtual Routers in Multi-VSYS
environment

Virtual Router
The firewall uses virtual routers to obtain routes to other subnets by manually defining a route
(static routes) or through participation in Layer 3 routing protocols (dynamic routes).

All routes learned are first place in the routing information base (RIB)

The best route obtained through these protocol are used to populate the firewalls Forwarding
information base (FIB).

When a packet is destined for a different subnet, the Virtual Router obtains the best route from
this FIB and forwards the packet to the next hop router defined in the table.

You can configure the virtual router to participate with dynamic routing protocols (BGP, OSPF,
or RIP) as well as adding static routes.

You can also create multiple virtual routers (VR), each maintaining a separate set of routes that
are not shared between virtual routers, enabling you to configure different routing behaviors for
different interfaces.

1 VSYS can have all multiple VRs as well as multiple VSYS can share the same VR

Each Layer 3 interface, loopback interface, and VLAN interface defined on the firewall must be
associated with a virtual router. While each interface can belong to only one virtual router,
multiple routing protocols and static routes can be configured for a virtual router.

The firewall uses Ethernet switching to reach other devices on the same IP subnet.

The common command in Virtual Router will start with the following:

show routing route
show routing fib
show routing protocol xxx
Show routing route type xxxx


Show routing route is essentially the RIB of the Firewall

Routing Table in Management Plane
Show routing FIB is essentially the FIB of the Firewall
Best routes selected and installed in the DataPlane , and is the routes use for actual
packet forwarding


In my lab , the output on PA-4020

> show routing route

Show routing fib

If you look at the Show routing route, it will display the RIB table from the Firewall
All the routes that are active are marked with an A in the flags column.

If you look at the show routing fib , all the route entries in it are the one marked with A in
show routing route and only those ones.

CONNECTED
Connected routes are IP Subnet belonging to the Firewall interfaces itself.
All interfaces configured in the Firewall will belong to this category.
Connected are the most commonly routes used for Redistribution in other Routing Protocols
with Static routes.
**Important** any /32 interface (including Loopback) configured on the Firewall are not
considered Connected but Host, and therefore cannot be redistributed.

As you can see from above screenshot, Host are not define in the Redistribution Profiles and
cannot therefore be redistributed into OSPF.

The workaround is to advertise those interface define the IP address /32 in Export rules (see
ospf section)

STATIC ROUTING
To configure a static route
NextHop: As you would expect in any static route , you have to define a directly connected next
hop for reachability to external network

**Important** Palo Alto Device does not support recursive lookups, so all next hop IP must be
directly connected.
If a non-directly connected IP has to be define, then a second static route has to be define to
advise how to reach that non-directly connected Next hop IP.

Next-VR: to forward traffic to a destination that is not in the local VR but another VR
Used often when dealing with Multiple VSYS scenario, where each VSYS has its own VR.

Discard: often use when needing to blackhole traffic, this is the equivalent next hop Null in
other Vendors.

**Important*, in 5.x, 6.0 and 6.1 , Discard route were not to be redistributed to OSPF because
they dont have a valid next hop.
This behavior has been modified, where now you can redistribute Next-Hop Discard in OSPF
from PAN-OS 6.1.4 (backported to 6.0.10 and above as well).

Discard routes can be redistributed in BGP in any version without issues.

None: often use for NAT and/or Global Protect Tunnel interface

NAT: When doing NAT to a subnet/IP address that is not directly connected (
meaning not part of any Firewall Interface subnets), it allows to install the route
so that IP Lookup during NAT evaluation can occur successfully for the NAT
translation to be completed.

Global Protect: When doing Global Protect, it is often required to have a static
route pointing to the GP Subnet allocated with Interface tunnel.x and use the
next hop none .
This is allows the GP subnet to be installed in the Routing table so FW know
where to forward the GP user packet back to, and it knows from the interface
configure that it is reachable via tunnel.x interface (and doesnt need next ip
for that hence the none).

OSPF
OSPF is a Link State Routing protocol based on AREAs to segment a customer network.

The segmentation is based on:

1. Backbone network (aka Area 0)
2. Area Border router (ABR- Firewall/Routers having at least an interface in backbone and
another in a different area),
3. Internal ( Firewall/routers only belong to one area )

OSPF uses IP multicast to send link-state updates. This ensures less processing on routers that
are not listening to OSPF packets. Also, updates are only sent in case routing changes occur
instead of periodically. This ensures a better use of bandwidth.
OSPF has better convergence than RIP. This is because routing changes are propagated
instantaneously and not periodically.
OSPF allows for a logical definition of networks where routers can be divided into areas. This
limits the explosion of link state updates over the whole network. This also provides a
mechanism for aggregating routes and cutting down on the unnecessary propagation of subnet
information.
OSPF allows for routing authentication by using different methods of password authentication.
OSPF allows for the transfer and tagging of external routes injected into an Autonomous
System.
In 7.0, with the addition of ECMP feature , OSPF will allow for better load balancing.

The minimum basic item to configure to get OSPF up and running:

-Define Router ID
-Define Area ID
-Define Area Type (normal by default)
-Define interface to be part of the OSPF
-Define Link Type (Broadcast by Default)
After setting up the neighborship
As you can see in the above output, all /32 address are seen as Host (via the flags H),
The 1.1.1.1/32 is a static route (Discard) redistributed into OSPF (we are on 6.0.10), yet we are
not seeing our loopback ip 192.168.0.2/32 in the table.
This means any loopback required in OSPF cannot be redistributed,
The workaround is to Export the /32 IP so it can be advertise in OSPF.
From the above note, since Loopbacks cannot be redistributed into OSPF,

It leads to the conclusion that you cannot have OSPF running between different VSYS/VR
instances, however within a single VSYS/VR , OSPF can be established.

The workaround is to use a Physical cable across 2 physical interfaces assigned in their
respective VR in the device or use a 3rd party switching/routing equipment in between the 2
VRs.
BGP
BGP is a Distance vector Protocol based on AS-PATH ,
It is compose of Attributes , some are transitive (mean are kept across AS-PATH) , some are
non transitive (mean only relevant in the local AS)
Any BGP session between 2 different AS are called External BGP (EBGP) and any BGP session
within one AS are called Internal BGP (IBGP)
The rule is all IBGP (in the same AS) neighbors must be full mesh to avoid routing loop,
The only way to overcome this rule is to create Route Reflector or Confederations.
Here we will cover only Route Reflector as Confederation are not as commonly used in the
market.

In general, BGP is the most flexible routing protocol for inter-VR/VSYS routing , it gives you
much more granularity in controlling routes than any other protocols.
However Palo Alto Networks preferred method for Inter-VR/VSYS communication remains via
Static Routes.

To configure BGP with the default settings,

Graceful Restart is enable by default

The default for the PEER Group configuration is below

When doing EBGP,

In my opinion the key thing that should usually be change straight away by the Admin on the
BGP configuration is to change the :
Import Next Hop: from Original to Use Peer
Export Next Hop: from Resolve to Use Self

This is always to be double check depending on the customer topology and routing as in some
case, the changes of Next Hop may not be required, but in my opinion it should be set as Use
PEER and Use Self by default

The reason for this By default when a route is advertised to an Ebgp outside of the AS, the
router will make sure that the next-hop attribute reflects its original IP address (basically
preserving the announcer IP address).
As a result when a route is advertised to an Ibgp speaker and sourced into the BGP as-group, all
Ibgp routers will have for next-hop the ip address of the Ebgp neighbor.

But what happen if the Ebgp speaker is not reachable?

What happen if the IBGP peer does not know how to get to the EBGP speaker announcing the
route (not IP reachability)?.

To prevent this, we can make sure that a route advertised to an Ibgp router; echoes the IP
address of the router sourcing that route into the AS to the Ibgp neighbors; and not the IP
address of the Ebgp speaker which originally advertised this route.
As you can see, a lot of the information are left by default and those timers should work in most
case for your customer.

If a faster failure detection is required for BGP, you can set the BGP to be more aggressive by
lowering Keep Alive and Hold time respectively to 10 and 30 (instead of default 30 and 90
sec)
Note the row highlight in RED MULTI-HOP
By default , the number of Hop is 0, which is usually a TTL of 1 ,
TTL of 1 mean it will establish BGP only with a directly connected peer (it will not do more than 1
hop).
In other to have more than 1 hop you can increase this value to 2 (recommended for EBGP) or
255.
2 or more Hop (or TTL if you prefer) can be useful when doing BGP Neighborship (whether
EBGP or IBGP) between the PA loopback and the peering router Loopback .

The above information should be enough to setup most basic/regular BGP

For any advance BGP, you will need to start playing with the features specs.

Import Rule
So lets create an import rules from Static to BGP (redistribute 2 static route into BGP) in VR-
VSYS1 and we will then advertise them to VR-VSYS2
I am importing the 2 subnet 10.194.56.0/24 and 10.193.58.0/24 and I want to tag them with the
Community Attribute of 65002:1111 and replacing the default Local Pref from 100 to value of 200
So now let s check our BGP Neighborship (here on VR-VSYS2 , but same for VR-VSYS1)
Cool, see that even without manipulating too many options, BGP Neighborship comes up , so
the default values are good enough to get us up and running.
(the import policy is just to have routes in BGP as this is a lab environment, and see if they are
advertise to the BGP peer, they are not mandatory to setup for simple BGP Neighborship)

Let us check the BGP routes learned

No Routes ??!!
Anybody figured out why yet?

If you go back to the 1st screenshot in the BGP section

The Install Route option is not ticked (it is unticked by default)

If I enable the tick next to Install Routes and commit
Voila !!! ;-)

You see that the receiving end (VR-VSYS2 BGP is receiving BGP routes) from VR_VSYS1 IP
10.193.57.2

Let us check the BGP route details to ensure how Attributes (Local Pref, Communities) are also
correctly advertise

So In VR-VSYS1, which is the one who own the routes and will be advertising to VR-VSYS2,
I have highlight the key info relative to 10.193.56.0/24 route specifically but that applies to all the
other routes
In VR-VSYS2
We verify for the same route 10.193.56.0/24
We now have a next hop 10.193.57.2 (on VR-VSYS1 , it shows local as expected)

We also added an AS-PATH: 65002 (which is VR-VSYS1 AS number to reach that destination)
We also received now correctly the Community Attribute tag for this 10.193.56.0/24 define in
VSYS1 (65002:1111) so we now will maintain that Attribute and advertise to our BGP Peer

However we do not have the same advertised Local Preference: here we have 100, in our
output from VR-VSYS1, we configure it to be 200 .

Anybody know why?

As discuss in intro, in BGP you have transitive and non-transitive attribute, which is essentially
meaning which BGP attribute are share with other BGP Peer AS (so which attribute can transit to
another AS and which attribute Cannot).
In intro, we specify that Local Pref is a powerful attribute but is non-transitive and has only Local
significance, meaning it only is relevant within its own AS (here only within VR-VSYS1 AS)

So all in all, it looks like we are doing ok so far.

Redistribute Rule
Redistribute Rule allows to redistribute route from one protocols into another protocol

While we saw some test previously on Communities,

Lets see the maximum community can be added when redistributing route into BGP
From the above Screenshot, we can see that the Add button is greyed out after 10
communities added

That tell us we cannot have more than 10 Communities redistributed to a route.

Again Palo Alto is a firewall and not a router and 10 should be more enough for 99.9% of cases.

On VR-VSYS1, we do see correctly the 10 Communities

Lets verify on VR-VSYS2, and Yes we do receive all 10 Communities
Now this shows that when redistributing from one protocol into BGP, we can add up to 10
Communities.
But when Advertising from one BGP peer on the PA FW to another BGP Peer, we can only
append 1 community
I have tried different ways to append additional Communities but we just cannot.
So the main recommendation should all communities tagging required should be done by an
upstream/downstream router so can forward it to the next hop.

So What if we try to have 2 export rules to see if we can append 2 communities list,

Lets change the setup so that the same prefix (lets try with 10.193.56.0/24) be use in 2 different
export profile

Creating 2 export rule (in below order) in VR-VSYS1 to same neighbour for the same prefix
(10.193.56.0/24) to be advertise to VR-VSYS2 10.193.57.120 , each with different communities
(purposefully change to different communities to validate the redistribution)

export rule 1 : 10.193.56.0/24 with community 65010:9876
export rule 2 : 10.193.56.0/24 with community 65010:5432

This is the Export rule for community 65002:9876
This is the Export rule for community 65002:5432

In RED, the advertise communities from the Export Rules,

in GREEN the advertised communities from the Redistribution rules, since we are appending we
should see both the Export rules communities and the Communities from the Redistribution
Rules.

Notice we dont see the 2nd export rule communities 65002:5432

If I swap the rule order to make Export rule community 65002:9876 below 65002:5432
Same behavior, we now see the top order export rule with community 65002:5432 but not the
bottom export rule 65002:9876

So the Export rule is matching the first Export rule but not processing the bottom one,

Lets see if this is because each are Export rule based on same attributes (Communities) by Pre-
pending AS-PATH which is a different Attribute.

From the current output, we only have 1 AS-PATH

Lets create our AS-PATH prepending policy

We have place it at the bottom in the Rule order , below the 2 Community export rules
We should now expect the AS 65002 to be prepending 5 times
In the output, it looks the export rule did not work as we still only 65002 once.
But we still see the Community 65002:5432 in the community list (who is at the top of the order
in Export rules)

If we place the AS-Path Prepending export rule at the order (so the 2 Community export rules
are below)
Now we see the AS-Path Prepending (first rule in the order list) work, but we lost all the other
Export List communities (there is no 65002:9876 nor 65002:5432 community anymore in
community ).

If we create a 4th filter, this time with a different prefix (10.193.58.0/24) with AS-PATH
PREPENDING 5
Here our prefix 10.193.58.0/24 Export rules is at the bottom

And when we verify

Now using a different subnet in the match criteria, we are having the AS-PATH prepending
happening.

That confirms the Export rules do a first match action and stop processing any following export
rules.

IBGP
So now that we have tested BGP successfully, lets try and setup iBGP between the 2 VSYS.
We currently have EBGP between VR-VSYS1 and VR-VSYS2,
We will now create a 3rd VSYS with its own VR called VR-VSYS3 and have IBGP between the
2.
We create another loopback for VR-VSYS3 with IP 192.168.0.4/32

The config on VR-VSYS2 is still the same (loopback IP is 192.168.0.3/32) and AS is 65120

So VR-VSYS2 config looks like

Created a new Peer Group with BGP type as IBGP

As you can see , we define Peer AS 65120 , which is the same as our Local AS ,
This is normal as IBGP is BGP peering between 2 neighbour within the same AS.
So the expected behavior is all the EBGP routes learn from VR-VSYS1 by VR-VSYS2 , VR-
VSYS2 will then advertise all those routes to VR-VSYS3 (as they are IBGP neighbor, they both
advertise to each other all their local BGP routes
On VR-VSYS3 , same configuration of BGP , but with the IPs other way around
Let s verify the Neighborship

On Vr-VSYS2 , we see it established

On Vr-VSYS3 , we see it established as well and receiving the BGP routes


VR-VSYS3 is now receiving the BGP routes from VR-VSYS2 (VR-VSYS2 learned those route from
VR-VSYS1).
Also we notice that VR-VSYS2 is preserving the AS-PATH prepending done by VR-VSYS1 and is
advertising the same AS=PATH length to VR-VSYS3.

Route Reflector
Now that we are adding VSYS and VR to achieve the RR scenario, our logical topology now
changed.
As a reminder, the current topology now looks like this
 The Route Reflector is a BGP routing components that offer an alternative to having all IBGP
peer to be in full Mesh (as full mesh doesnt scale well for large networks )

A fundamental point in Route Reflector (since it is not in full mesh anymore) is the Loop
Avoidance and 2 new attributes gets generated. In RR the 2 attributes for this:

Originator-ID
Cluster ID

The Router ID field is for Originator-ID and Reflector Cluster ID is for the Cluster ID

Route Reflector Configurations

Here we use the Next-hop Use Self options for the purpose of the lab, but in IBGP you
probably want to keep the Original next-hop.
As you can see, above screenshot show the config of the RR itself (192.168.0.3) and it has the
Reflector Client option set to Client.
It does not mean the RR is a client itself, but that the iBGP peering to him (here 192.168.0.5 )
will be his client
The router ID is referring the RR attribute Originator ID
The Reflector Cluster ID is referring the RR Attribute Cluster ID

High Availability

Active-Passive
When doing Active-Passive, it is important to remember that the passive member is in standby
mode, not responding to Arp and does not own any IP as the IP address are still owned by the
primary .

If we take the routing table output of our lab device Active in the HA
We have our 3 Virtual router , with the routes installed in each Virtual Router.
If we look at the flags column, we see the Host (H) ,Directly Connected (C ) , Static routes (S) ,
OSPF (Oi), BGP (B) route entries.
So that confirms all routes are correctly present in the Routing RIB tables.

If we now go to our Passive device, to verify the state of the routing entries
If you look on passive, we do not see the same route entries,
No OSPF, no BGP, more interestingly not Directly connected,
If you look only Host and Static entries,

No dynamic routing (as when the failover occurs, the Firewall will re-establish the Neighborship,
learn the routes from its neighbor and recalculate the SPT),

The directly connected not being present, actually make sense since the Interface and the IPs
are only active on the other Firewall (who is currently the active member and the one who own
the IP addresses).