eBOOK

Thou Shalt Not Pass… I Think?!

How to be a Cisco® ASA ace

by Destiny Bertucci
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Table of Contents
INTRODUCTION 3

Topics this eBook will cover 4

Is this eBook for you? 4

1 VIRTUAL PRIVATE NETWORK (VPN) 5

History 5

Problems and issues 7

Breakthroughs in monitoring and managing VPN tunnels 8

2 CISCO ADAPTIVE SECURITY APPLIANCE (ASA) 12

History  12

Problems and issues 13

Breakthroughs in monitoring and managing ASAs 16

3 ACCESS CONTROL LIST (ACL) 19

History  19

Problems and issues 19

Breakthroughs in monitoring and managing ACLs 21

4 ABOUT THE AUTHOR 23

page 2
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Introduction
If you have ever been tasked with monitoring and managing VPN concentrators or the com-
bined technology within devices like Cisco ASAs, you understand this task could be challenging.
Welcome to a much-needed safe zone where I plan on venting my frustrations and offering
ways to overcome common issues. Don’t worry—this is not going to be a whiney or unhelpful
manifesto about how ASAs can be hard, and nothing can make it better. Instead, I am going
to discuss things like ACL management and remote access VPN troubleshooting to give you a
solid understanding of the complications that tend to arise. Sharing ways you can fix (or at least
improve) the situation with a little bit of insight from monitoring and management software.

Writing this eBook was a huge eye-opener for me, and I hope it presents new ideas for you as
well. Although we (meaning any network/security individual or team) deal with security, we do
not have the opportunity to see what it is that is secured visually. It can be incredibly frustrating
running around opening this port or shutting down that port for a new application without being
able to address security concerns or see the whole picture. In that situation, it is difficult to know
if these new ACL network routing changes are not just creating other issues. It makes you feel
like you are always putting out fires without ever discovering what started them.

We have entered a wonderful era of security breakthroughs. We have a better handle on our
networks because we are now able to control them with adaptive devices based on rules of our
own design. It has never been easier to acknowledge changes and verify that these changes
are not causing problems. Monitoring baselines for multi-vendor devices allows us to see our
environment’s big picture, and how these devices and their configurations are affecting traffic
and/or services. Gone are the days of being left in the dark, only able to monitor up/down status.
The goal of this eBook is to provide you with a deeper understanding of the amazing tools we
now have to work with, and learn why we should be demanding more from our monitoring and
managing software solutions.

page 3
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

SPECIFIC TOPICS THIS EBOOK WILL COVER

1. The importance of understanding VPNs

2. How to verify the placement of ACLs

3. Why ACLs sometimes cancel each other out

4. Cisco ASA’s multi-context configuration management

5. The importance of monitoring

IS THIS EBOOK FOR YOU?

I do not want to waste your time reading this if it is above or below your level of understanding.
To save you any further unnecessary reading, let me lay down some basic questions:

• If you do not manage or monitor ASAs, this book is probably not for you. How-
ever, if you plan on being more involved with security, this could be the book
you’ve been looking for.

• If you have a CCIE Security Certification, this book is probably not for you, my
friend. But it might be a great start for that junior colleague who just joined
your team.

• If you are monitoring and managing ASAs with third-party software, well, Gold-
ilocks, this book is probably just right for you. You will be able to identify with
(or at least understand) the frustrations I am describing and appreciate the
best practices I share.

page 4
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Virtual Private Networks

VPNs create a private connection between multiple sites, people, and devices
across the internet by producing a secure tunnel for your data to pass through.
VPNs shield and protect your data and transactions across the internet from
hackers, malware, and anyone whom you do not want to know what, how, or even
where you are surfing or browsing. VPNs are only as secure as their encryption
capabilities. If you have ever been to an IT conference, you understand the value
of excellent encryption to hide your browsing traffic. Unfortunately, there are some
people who would love to try out new sniffing capabilities to grab info from your
cell phone.

HISTORY OF VPNs
Ever since we started connecting with one another via the internet, security has been an issue.
Beginning October 29, 1969, the Advanced Research Projects Agency (ARPANET) of the United
States Department of Defense launched the world’s first successful packet-switched wide area
computer network. Years later, around 1996, a Microsoft® employee invented a peer-to-peer tun-
neling protocol (PPTP). That was technically the beginning of the VPNs we have today. However,
the security aspect was lost somewhere along the way. Security holes became more prevalent
once user-friendly options were introduced.

Data protection is a constant security nightmare. Encrypted data is set up with a private key that
can unlock and unencrypt your data when it reaches the correct destination. In data encryption,
algorithms are used to encode your data. This makes it harder for someone trying to steal your
data to understand it, though it is still a mathematical equation that can be breached.

page 5
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Thanks to the internet, managing new applications with accessibility and availability require-
ments has become a 24/7 job, with only scheduled maintenance and downtime allowed. If
you have hundreds of servers that need to be patched, or even 25 network devices that need
firmware upgrades, how are you going to implement all of this in only two hours of downtime
on a Friday night?

Now think about how this window has changed over time, due to modern innovations providing
faster and more reliable networks. If someone was stealing data a decade ago, your relatively
slow network speeds and small bandwidth actually worked to your advantage by throttling what
the thief could get and how fast he could get it. Now, the majority of data thieves have fantastic
data speeds (I have Google Fiber® at my house, and I live 25 miles away from a major city), and
can easily take ample amounts of data right out from under your nose. This also means that our
monitoring and security practices have to be able to access the environment and know what is
going on so we can stop data theft as quickly as possible. Today, someone could realistically
steal large amounts of data in less than 15 minutes, and data speeds are only going to get faster.

VPN concentrators initially were created by different companies. Altiga Networks got a lot of
attention around 1999 when Cisco® bought them out and turned their technology, as well as
their own, into the VPN 3000 Concentrator series. These devices were great at providing secure
VPN connections and data delivery between VPN nodes. They also extended the capabilities of
a VPN router by adding encryption and data transfer. This was accomplished by managing larger
amounts of VPN tunnels to shield the network traffic from site to site. At the time, there was a
significant need for a device to only handle these creations and the management of VPN tunnels.

VPNs are now combined with antivirus and intrusion detection devices, as well as data security
software to help businesses get a better handle on securing their data. This brief history of data
encryption emphasizes why it has always been (and should continue to be) an important secu-
rity measure. Since the number of users working on mobile devices from remote locations is
only increasing, so too is VPN deployment. This is fantastic, because it allows workers from all
over the world to work for your company, while also encrypting their data so they’re not a water
hose of proprietary information. VPNs have helped expand businesses opportunities without
compromising security.

Data breaches are now trending up at alarming rates. The need for strong VPNs (meaning im-
plementing strong encryption and authentication practices) to reduce security breaches and
help stop sensitive data from being seen or taken is at an all-time high. Luckily, the technology
to secure your data and help give you more authentications has grown with this need. The issue
is that this technology is not being implemented as often as it should.

VPNs have certainly come a long way, from peer-to-peer all the way to large networks filled with
services and applications. Users are now able to fully connect to what they need anywhere they
have a connection. VPNs are also great for personal use, because they encrypt the traffic that
is leaving your home.

page 6
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Smart home user tip

Just because we’re focusing on the corporate side of VPNs doesn’t mean the end-user version
I mention above doesn’t have any uses. In fact, VPNs are almost as important to have in your
home setup. Why? Well, for one thing, they can shield your traffic from being “sniffed” by the Big
Bad Wolf (i.e., your internet provider). BUT that also means that you better have a trustworthy
VPN solution. I use mobile encryption and have a home VPN solution for all my traffic, just in
case. The major difference here is that all my internet traffic is being encrypted for me, so any
music/movie streaming, chats, or shopping is protected. I’m not one who thinks too kindly about
people profiling my traffic.

PROBLEMS AND ISSUES WITH VPNS

When we expressed need and longing for these secure VPN tunnels to allow this great business
protection, who knew that designing, building, managing, and maintaining them would end up
being a constant burden? After speaking with thousands of other engineers at conventions, user
groups, and on the job, I know I’m not the only one who feels this way. So, I’m going to discuss
the common VPN issues that often force me to work a long night to find and resolve the issue
without breaking anything in the process.

VPN connection failure

VPN connection failures happen for lots of reasons, and tracking down the root cause can feel
like pulling teeth. There could be an incorrectly configured IP address in the pre-shared key or
crypto map, ACL overlaps with other tunnels, routing issues, mismatches in the ISAKMP policies,
and more. These are all very common and sometimes overlooked issues that can cause count-
less hours of stress. To avoid them, we need to focus on being able to see why the connection
failed and correlate that to how the device is configured. Being able to see the ACL order and
the way ACLs relate to each other saves time and frustration. Routing loops that could cause
an issue of not getting to the desired path should be seen and then resolved from one location.

Slow VPN connections

These are harder issues to troubleshoot when you can’t see the bigger picture. Someone who
tells you that the network is slow might just be used to the instant gratification of high speed
we’ve all grown accustomed to. However, sometimes we compare connection times between
the office and remote locations and actually do notice an unusual increase in slowness. This
opens up several windows (and even a few doors) to a whole new world of troubleshooting. If
we are not aware of the quality of our network, which can only be compared against a baseline,
we are stuck if we have to prove that we know there is a problem outside of our environment.

Remote issues

Let’s say a user contacts you and says they’re unable to connect to the corporate network. First,
you have to establish their network and situation. This allows you to know where, what, how,
and when they are connecting. Remote issues can be frustrating, since they’re outside your

page 7
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

control. We have to know, based on our monitoring abilities and baselines, whether everything
is normal. This allows us to move on to the next portion of troubleshooting outside of our zone
and finally find a solution to the user’s connectivity problem. If the user is within our environment
and monitoring view, we should have a true baseline available for us to compare their current
experience. Once an issue occurs, we can then correlate the events, changes, anomalies, and
logs to pinpoint the issue and implement any required solutions.

Firewalls blocking VPN concentrators

This is where you can block the connections between peers. You can set up a firewall rule to
block certain IP addresses, subnets, application types, and more that will immediately deny a
connection before the user/application has a chance to authenticate. These can be troublesome
to find, especially if you’re not aware of any recent changes that could have created an ACL block.

Network traffic that is protected by Internet Protocol Security (IPSec) usually has override black
rules so VPN traffic is able to pass through, even if an ACL would otherwise block this traffic.
This is something we can do, but it is not always implemented. These authenticated bypass
rules are in place to help prevent this type of issue.

These are just a few of the most common problems you may run into with VPNs, in general. Of
course, all kinds of other issues can arise, but I don’t want you hiding under your desk in terror.
The bottom line is that these issues signal the need for better ways to gather critical information
for faster troubleshooting and resolution. We need to be able to see the bigger picture without
any blind spots or unfinished brush strokes blocking our view.

BREAKTHROUGHS FOR MONITORING AND MANAGING VPNS

Engineers get really good at troubleshooting because of the problems listed above. Living with
those issues (and their manual solutions) makes us search for better ways to spot, resolve, and
even prevent instabilities in our IT environment.

I believe it is wise to have a healthy mix of mad troubleshooting skills and good tools to help
you avoid problems in the first place. If everything in our network went smoothly, we wouldn’t
need to support anything ever again. But realistically, that’s not how IT works, so we should
constantly look for better tools to help us. We also need to keep our hands-on troubleshooting
knowledge as up-to-date as possible.

When managing VPN tunnels, we need to know how to create a baseline, which is critical to
successful management and monitoring. This means having a clear understanding of what
normal is within your environment. Let’s say that since day one, Router A has had a CPU of 92%
and runs correctly without fail. This is normal. This is your baseline from which you compare
network health, performance, and configurations of similar devices and connections when you
are monitoring your network. Simply knowing the health of a device is not enough if you haven’t
established what constitutes normal operating behavior.

page 8
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

With this knowledge, you can alert on true anomalies. For example, if CPU utilization is rising, its
processing may be compromised. This may then lead you to look at changes in other metrics,
such as an increase of dropped, fragmented, or discarded packets. On the other hand, maybe
the CPU rising on that particular device at that time of day is 100% normal, even if it is way out
of line for other devices of its type. The trick is being able to see all of this information from one
solution. Remember: I’m in troubleshooting mode if I have to resolve even one of these VPN
issues. I do NOT want to be jumping around and losing context while trying to pinpoint an issue.

When managing VPNs, we should focus on where they are and what they are doing. I like to
segregate site-to-site VPNs, in contrast to remote logins that may be maintained on one device.
I feel like this is a great way to be able to go straight to focusing on the VPN issue that is being
presented. Little things like this can eliminate hours of troubleshooting and wasted time. It
sounds so simple, yet it has been missing from solutions for too long. Monitoring and managing
software can allow you to pick up on each VPN so that you are able to see this segmentation
and focus on how it is being used and maintained.

Remote access VPN tunnel visibility has raised the level of difficulty for troubleshooting remote
devices. More devices with VPN applications that are available to multiple devices enlarges your
environment from outside your company all the way to the end-user’s remote destination. (For
example, a coffee shop on Tuesday and then a hotel on Friday.) In this situation, you need to ask:

»» How many users are connected?

»» How many have connected over time?

»» What’s the type of tunnel?

»» What’s the version of the software client?

»» How many failed connections have there been?

While we are implementing and changing access for VPNs, these are the metrics we need to
be aware of (i.e., collecting data for) and proactively alerting on to stay aware of our VPNs’ per-
formance. When you have users who are not able to connect, you have skyrocketing costs due
to lost productivity and increased frustration. Sometimes when the issue is on your side, it can
be a small configuration error. Human error can and will happen. For example, Google® recently
experienced downtime due to one such error. Validating VPNs and having checks and balances
in place allows us to focus on bigger issues, rather than overlapping ACL configurations that
could be accidentally blocking or allowing the wrong users.

Correlating VPN changes to the platform’s health metrics is vital. What I mean is, if I’m rolling out
a change, or having to make one on the fly, I need to be able to quickly compare metrics before
and after that change to know if there is a problem so I can roll them back immediately. Prob-
lems with these devices can adversely impact several factors of your business. These include,
but certainly are not limited to, application access to potential clients, credit card transactions,
and even denying access to remote workers or offices. These can all result in increased costs

page 9
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

to your business. It is vital to have full network insight, including visibility into the inner work-
ings of the devices’ setup and the way configurations are being implemented. This is not only
to maintain stability, but also to help ensure that the security procedures within your security
team or business are being accurately utilized within your network.

If you manage these devices, you understand that VPNs allow a tunnel to encrypt data coming
in and decrypt that data at its destination. This creates a shield around your data and a network
environment that is not easily penetrated by malicious people or bots. But how do we monitor
VPN tunnels when they are encrypted and set up by devices like Cisco ASA (Adaptive Security
Appliance) next-generation firewalls? We need to be able to view the whole VPN and monitor
more in-depth metrics. This includes site-to-site and remote VPN users. Complete visibility,
beyond simple hardware health and performance, allows us to better manage and monitor the
tunnels we are responsible for.

Users and network/security engineers are equally frustrated by not having visibility into a VPN
tunnel. It is frustrating to the user because he doesn’t understand how the engineer is unable
to see an issue or have the ability to compare connections and resolve the issue. But more
importantly (to people reading this book, at least), it is frustrating to engineers because they
don’t have access to any tools that would give them a baseline of the network, or the ability
to manage and monitor its VPN connections. On top of all that, losing connectivity to remote
locations and analyzing ACL configurations have been a daunting task because there is no
visibility into what they are or aren’t doing with network traffic. Meanwhile, gathering logs and
creating queries usually involves having to use several different tools and software, which is
already frustrating. It is challenging to be in the middle of chasing down a problem, with users
and management screaming in the background, to have to suddenly switch screens or open
other tools. This usually ends up confusing the engineer, who can, at this point, lose his way in
the troubleshooting matrix.

But it doesn’t have to be that way. This is where your tools can shine and really help you avoid
some misery. Your monitoring software should help you to:

»» Verify VPN tunnels to help ensure connectivity between sites

»» Determine uptime

»» See packet encrypt/decrypt counts

»» Understand ingress and egress bandwidth

»» Find out which algorithms have been selected

»» See VPN tunnel history

»» Inspect how many connections have failed

»» See the session history for a particular username

page 10
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Once again, baselines are important because they give you insight into the health and perfor-
mance of your VPNs. Correlating NetFlow data and collecting layer-three device traffic to one
central location (with interface health statistics and the device’s overall health) are ways to help
ensure VPN reliability and uptime. By this point, we know how VPN tunnels are implemented.
However, visualization and alerting is where we can understand the flow of our VPN and user
functionality. Creating checks and balances with the VPN connections and session IDs—both
site-to-site and remote—allows you to separate the things that improve baselining, troubleshoot-
ing, and visualizing your VPN tunnels.

Tools that allow you to monitor your health statistic devices are great to have for the baselines
I’m talking about. There are also tools that allow you to mimic the traffic of an application and
gather statistics, which helps you verify that the TCP port for the service/application is available
and that the pathway is functioning as designed. For example, source-to-destination monitoring
of networked pathways—within the SolarWinds Network Performance Monitor platform, there
is a feature called NetPath™—this allows you to verify that applications are available for connec-
tions to or from your offices and/or users. Adding the ability to monitor the actual technology
of the VPN encryption reduces troubleshooting time, which leads to quicker responses. It also
allows you to implement resolutions to these possible interruptions in services or applications.
Knowing your baseline history and doing continuous monitoring gives you the opportunity to
implement upgrades to firmware and make changes to ACLs with accurate reporting of before
and after metrics.

Monitoring tools should allow you to use the metrics within your database as a fluid way to
correlate events. For example, if I just implemented a new ACL on my Cisco ASA, I would like
to verify whether I caused any anomalies in my environment or not. I can place the metrics
I’m monitoring into a report and have it correlate the times that I have picked and chosen from
my monitoring tool. This gives me immediate sight into any changes that may affect the VPN
tunnel’s stability, which is crucial to monitoring. Proving change with any impact via a report or
sending a monitoring link that would showcase the live correlated data allows you to validate
a need for an immediate rollback or change request approval to resolve. Currently, SolarWinds
provides this through its PerfStack™ dashboard.

In the next chapter, I will discuss the actual devices and how we can now do more to manage
them with the ability to react to the situations we discussed in this chapter.

page 11
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Cisco Adaptive Security Appliance (ASA)

ASAs are Cisco next-generation, all-in-one security devices that are used by
businesses all over the world. The need for enhanced security and data protection
across the enterprise has only made these devices more critical to protecting
businesses from security breaches.

HISTORY OF ASAs
Oh, the excitement we all had witnessing the beginnings of device protection, when Cisco PIX
Firewalls, Firewall modules, and Cisco IOS® Firewalls were introduced! While routers are able to
deny traffic, and many devices and applications allow us to provide encrypted data transfers
and transactions, there was (and still is) a need for more security with more complex technolo-
gy and capabilities. After all, a device focusing on security eliminates the burden of inspecting
packets on the router.

First-generation firewalls allowed for security policy enforcement for intranet and extranets,
while deploying secure internet connections remotely and with branch offices. These were
great for blocking traffic and allowing the corporate audit team to fill in the security check box.
However, more sophisticated attacks arose that were difficult for those early devices to detect.
In the early days of information security (INFOsec), we also began to realize that attacks could
come from within our networks. A user could bring malware to the office after working from
home on their company laptop. There could also be a compromised mobile device connected
within the network. As attacks have continued to grow in sophistication, so too have the tools
we use to protect our networks against them.

Intrusion detection systems (IDS) and intrusion prevention systems (IPS) were, and still are, need-
ed for their ability to alert unauthorized or malicious activity for host- and network-based needs.

page 12
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

An IDS works by detecting spikes or new occurrences that are happening on your network. They
either look for signatures of known attacks (like antivirus definitions) or unusual behavior (such
as large amounts of data flowing to a new IP address it has never seen).

Meanwhile, an IPS looks at the traffic as it is passed through the IPS device on the network in-
line, rather than directly like IDSs. The IPS will block or allow packets based on its policy. Vendors
have introduced a whole array of IDS sensors—entire IDS modules on a single chip—and modules
to help achieve more accurate and intelligent in-line prevention services without burdening the
host device with an additional set of operations to perform. Additionally, network anti-virus,
anti-spyware, and malware mitigation capabilities improve businesses’ threat defenses and
should be used in combination with software/devices like IDS/IPSs.

VPN concentrators (see the first chapter for more information), like the Cisco 3000 series, were
needed because we required devices and modules that could integrate within our networks and
provide encryption services on a larger scale. The VPN concentrator 3000 series could easily
manage hundreds of routes for businesses.

Which finally leads us to the great and wonderful ASA devices that enhance the capabilities of
these concentrators.

Cisco ASA is an all-in-one solution that integrates firewall, IDS, and VPN capabilities into a single
package. This should explain why managing and monitoring these devices seems so complicat-
ed! If you have ever sat down at an ASA and felt like it was three times harder, it is because it is
three types of devices rolled into one. It also explains why the product took off in terms of sales.
They cut out an abundance of equipment, reduced the management complexity of connected
devices, reduced the amount of human error and correlation of working platforms, and allowed
us to focus on one device.

The ASA also took each of those capabilities to the next level. The wealth of technological ad-
vances allows for increased security. Current generation devices support mobile IKEv2 (MobIKE)
and even Active/Backup High Availability for ASAv on Microsoft Azure®. More info on all features
of a Cisco ASA is available here.

The problem is that the learning curve for these enhanced capabilities is pretty steep. Most
of us have our hands full with the job we were doing before the ASA arrived in the data center,
and the result is that most organizations are not fully utilizing these devices. You hear this a
lot because it is true: security is only as good as the weakest link. Knowing how to configure
an ASA is one thing. Being able to use it to meet the needs of your unique business case is a
completely different story.

page 13
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

PROBLEMS AND ISSUES WITH ASAs

To really dig into this topic, I have need to break the device down to its components areas.
Managing these devices is like a crash course in security. A lot of these devices are implement-
ed by network engineers who have not fully jumped into the security realm. That’s not a huge
problem, but being aware of the issues in this chapter will help you to have a better grasp of
the bigger picture.

ASA firewall management

Packet filtering allows us to focus on protecting the inside network, demilitarized zones (DMZs),
and the outside network. The device inspects all the traffic that passes through it and uses ac-
cess control lists (ACLs) to drop unwanted or unknown traffic. Each ACL contains an element
that permits or denies statements, often known as an access control entry (ACE). The device
can classify packets through inspection all the way up to Layer 4 headers. The problem is that
these statements can sometimes never be hit, not allowing traffic to go to the right destination,
or worse, not blocking traffic to certain destinations. Hit counts are when we know the rule has
been represented and the traffic has been verified to the rule. If it is not being hit, that means
that the rules in front of the non-hit rule are not allowing traffic to continue. Firewalls are devices
that can have trillions of hits per second. Monitoring and managing these should be a priority
since they are evaluating your network traffic at a high rate of speed. We will dive more into
firewalls in the next chapter.

IP routing identifies which interface and gateway should be used to forward packets based
on their destinations. These routing decisions can be made via dynamic routing protocols like
routing information protocol (RIP), open shortest path first (OPSF), and static routes. Devices
like Cisco ASAs have the ability to configure up to three default routes for traffic load-balancing
and should all point to the same interface.

Authentication, authorization, and accounting (AAA) gives access control to network devices
and are vital for business infrastructures. They provide an extra layer of security support for
managing and monitoring network devices. Yes, now we are bringing in local or external servers
for authentication that use AAA authentication protocols like RADIUS and TACACS+.

High availability has moved from a “nice to have” to a “must have” with today’s e-commerce
reliance on LANs and WANs. Connectivity losses can have huge financial impacts. Large enter-
prises could potentially lose a million dollars in revenue per hour. Think about huge distribution
centers for businesses and public sectors. Take the Amazon® typo incident for a quick “uh oh”
moment. This mistake cost the company $150 million dollars for three hours of service down-
time. If you would like to calculate your own downtime for critical services and outages, use
the following formula:

page 14
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Cost of downtime = lost revenue + lost productivity + cost to recover + cost of intangibles (loss
in customers) = cost of downtime per day. Then you have to take into consideration whether you
have a 24/7 business or one that runs on a more traditional 8 a.m.-5 p.m. schedule, because
that will increase or decrease your hourly rate.

Luckily, ASAs have failover capability, which allows you to have a disaster recovery device in
place in case one fails or goes offline. This helps you stay online and continue business while
you attend to the downed or dysfunctional device. Failover is vital. You do not want an open
gateway to your network if the active device goes down.

ASA IDS/IPS management

Intrusion Protection Systems (IPS) can be provided that include deep packet inspection (hence
the “adaptive” portion of the ASA abbreviation), and IP auditing with basic signatures. This lets
you decide if you want to use in-line or promiscuous IPS mode. Inline mode is when the traffic
passes through the firewall policies before it is sent to the Cisco ASA FirePOWER™ module. This
helps by only inspecting permitted traffic based on the policies. Once in the FirePOWER module,
if it is determined to be malicious based on defined security policies, it sends back the malicious
verdict to the ASA, which will then block the traffic going forward. Promiscuous Mode is when
copied packet traffic, which is defined in the service policy, is sent to the Cisco ASA FirePOWER
module. However, if it is only in monitoring mode and if anything is deemed malicious, it can
alert the administrator but will not block the traffic. This is where monitoring health statistics
comes into play. You can decide the amount of traffic and the effects of this traffic being inline
or promiscuous for the increase in inspection.

ASA VPN concentrator management

VPN site-to-site uses the IPSec protocol. Since managing this protocol setup is within the ca-
pabilities of an ASA, it is important to note that IPSec can use Internet Key Exchange (IKE)
for key management and tunnel negotiation. Why is that in bold? It is simple: IKE uses dif-
ferent combinations of Phase 1 (IKE_SA phase are defined in the key exchange) and Phase 2
(CHILD_SA phase defined within the data policy) attributes that will be negotiated between the
peers. If ANY one of these attributes are misconfigured, the IPSec tunnel will fail.

Because of the complexity of these tunnels and the required precision of configuration, debug
commands, like “show crypto isakmp sa detail,” are used to verify these phases. We also have
to know where to find these details quickly, to be able to better manage and troubleshoot if and
when issues arise.

Remote Access VPNs open a whole new can of worms. Not only do we need to keep up with
our network, now we have to verify the tunnels’ stability via remote devices. A saving grace is
that the steps and attention to detail required are similar to the site-to-site.

Multiple security contexts are needed for both split tunneling and better traffic management.
Contexts give you the ability to partition or separate an ASA into multiple virtual devices. This

page 15
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

allows you to have multiple security contexts within your device, which means they will be
included in the configuration file. These are separate security plans managed through one
device. Split tunneling is a way of load- balancing traffic and preventing your ASA from being a
complete bottleneck by having an intensive bandwidth load. Split tunneling is the configuration
of only sending corporate encrypted traffic to the ASA and things like email and messaging out
through the internet.

All of these capabilities can present a lack of monitoring situations. There are multiple features
of ASA devices that we need to see and manage, but due to their complexity, it has been a bit
of a mess to gather all of this information in one location. However, simple network manage-
ment protocols and a show command to download a configuration file is not always the best
or fastest way to approach these complex devices and their uses. This next section will show
how we can untangle this mess of monitoring and management and finally allow us to focus
on performance and security.

BREAKTHROUGHS FOR MONITORING AND MANAGING ASAs

Before managing and monitoring network devices, you should back up configuration files. These
files normally contain the configuration for one device and how it should be used. However, there
are devices that have what they call multi-context or security context configuration files. Just
like it sounds, they have different configurations on one device that may tell the network traffic
to behave or be directed to different locations based on the context that traffic adheres to. That
being said, these multi-context configuration files have been an issue for recovery and compari-
sons. Recovery is when you have multiple contexts (segments) that have to be accurately placed,
which has been a challenge for network configuration managers. The network traffic needs to
follow its correct context of the configuration so it is routed, secured, or dropped according to
the context that has been created. Comparing these configurations between like devices has
always been a challenge when the order of placed ACEs have actual meaning, and security
contexts can be seen as a configuration itself. As you read earlier, there can be multiple security
contexts, and these are often found in companies like service providers, large enterprises, or
governments that want to keep departments completely separate. The reason they exist seems
obvious when you are managing traffic through a device to allow proper traffic in designated
areas. Due to various complexities, the ability to download, store, compare, and restore specific
context configs to a device for monitoring and management continues to be a challenge.

The good news is that visibility into this ecosystem has never been clearer than it is now, with
mature monitoring solutions providing the ability to monitor these multi-context configurations.
Combining monitoring elements like receiving real-time change notifications, compliance reports
(based on your security needs), analyzing hit counts so you can simplify your ACLs, and being
able to compare like devices for network standardization is a game changer. Think about the
detection, prevention, analysis, and response (DPAR) cycle and how the multiple abilities of the
Cisco ASA tap into every aspect of it. When we are faced with the burdens of maintaining the

page 16
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

goals of confidentiality, integrity, and availability of information security, we cannot afford to

monitor and manage these devices less than completely.

Combining network performance metrics, such as overall platform health, load-balancing sta-
tus, and failover, allows more time to troubleshoot. But you still need more to gain full visibility
into the vast multitude of capabilities that ASA devices deliver. The following items help allow
VPN tunnels with Cisco ASA monitoring and management to have a true baseline for better
troubleshooting:

»» NetFlow and NSEL ingress and egress traffic with CBQoS information correlated to your
interface bandwidth

»» General health statistics for capacity

»» VPN tunnel visibility (LAN-to-LAN)

»» Remote access VPN visibility

»» Failover monitoring

»» Connection counts

»» Hit counts

»» Interface monitoring with security levels

SolarWinds® Network Configuration Manager (NCM) allows you to see how your configurations
are working within your network’s devices. Being able to validate your VPN tunnels is great,
don’t get me wrong. However, being able to add the capability to store backups for network
standardization, while also being able to alert on any changes made to these configurations,
is security gold.

While troubleshooting VPN, firewall, or IDS/IPS issues, we should all be striving to find one solu-
tion that provides a view into these details. For example, I want to quickly validate a failovers
active and passive state within my ASAs while using a managing and monitoring device that
lets me see a connection session’s history.

I want the ability to drill down into site-to-site or even remote access VPNs to troubleshoot
connection issues. I also want to see the overall performance of the device via CPU, memory,
and traffic flow to any increases that may have occurred. This would let me forecast capacity
planning for expansions to my ASA needs.

Hit counts are vital to managing your ACL list, which can sometimes get out of control. Showing
relationships; analyzing ACL configs; and showcasing redundant, shadowed, and unused rules is
quick and easy, which is great because that means I don’t have to personally go to each device
and run show commands.

page 17
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

When you combine NetFlow collected data, network performance fault monitoring, configuration
management, and vulnerability detections with many other metrics, we finally have a bigger
picture that provides complete ASA infrastructure monitoring and management. Full integration
of these metrics gives you proactive awareness about your ASAs with fluid, active monitoring
of traffic flow and the way your network, device, and users are being affected.

Vulnerability reports automatically check for vulnerabilities against your platform’s firmware.
When you add firmware bulk upgrades, you save time and reduce your risk of exposure. Make
sure to monitor the full realm of the device you plan to update, from locations to labs to test,
with a complete validation of metrics for potential issues that may arise after the upgrade.
Forecasting time and impact during maintenance helps teams better prepare for and prevent
issues that may cause service availability to falter.

page 18
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

Access Control Lists (ACLs)

HISTORY OF ACLs
For a more technical definition for Access Control Lists, the image below provides a list of rules
that specify which traffic should be granted or denied access to objects and locations. Each rule
then provides an action for the traffic that matches it, to be either dropped or allowed. Traffic
is checked in the order of the rule placement. If traffic matches a rule, then the traffic is either
permitted or denied and you will not pass to the next rule. If the traffic does not match the first
rule, then the packets will continue to be processed against the subsequent rules until a match
is found. ASAs have an implicit deny at the end of each ACL; any traffic that is not matched is
dropped. This packet filtering allows you to determine the path a packet will take from then on.
Rules can be based upon IP or MAC ACLs so you have options to better secure and use filtering
for your business needs.

PROBLEMS AND ISSUES WITH ACL MANAGEMENT

First, just as a baseline for practical usage, ACLs allow you to be the pass/no pass general of
the IT department. If it takes me twice as long as it should to gather data about a problem, I
am wasting valuable time that would be better spent resolving the issue. Meanwhile, service
availability may be compromised. ACLs, and the access control entries ACEs that go along
with them, can become a daily chore of managing new applications and access level changes
happening weekly, if not daily. ACEs specify the decisions that the ACL must perform.

I will focus mainly on Cisco ASAs devices. These are the ones that I am more familiar with and
they generally have a bigger market share. As with most ASA types of devices, you will find
they can have limitations. When discussing Cisco ASAs, there are limits, such as 128 rules per
ACL, or no more than 10,000 ACLs across all the ACLs in one virtual Ethernet module (VEM).
You might think this is great because they limit the size of your configuration building. Not ex-

page 19
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

actly—these rules have a hierarchical ordering, and they work by traversing the rules until each
packet is either permitted or officially denied. To accurately configure these rules, you need to
understand your traffic flow and business needs.

For example, you may have your ACLs granting access to a mail server from only one subnet
and is a permit rule. Then you have all other traffic as a deny rule. Very basic, obviously, and
then you are told there is a new application that only certain IPs need to access and they will
be coming through this Cisco ASA. I now have to adjust my ACLs to allow specific traffic before
the subnetted traffic to access the application. I could easily mistype and place it below the
deny rule and have connectivity issues. So, there are multiple ways to cause issues with rules
and the way they are arranged and managed.

This presents the problem of users either having access to data they shouldn’t, or the inability
to access data they should be able to reach. Or both. Both is always a hoot. Of course, you will
always hear about the denied access that interferes with a user’s work, but when it comes to
accidental open access, that may be a sleeping giant, one that can be exposed with damaging
repercussions. The upshot is that it is absolutely critical to get your ACLs correctly placed. You
must constantly monitor the EFFECTIVE rules (how they actually work) to help ensure that you
are getting what you think you designed.

Troubleshooting issues can be a headache to resolve. You or your team will constantly gather
informational metrics from the device and then manually correlate the problems. I literally had
a security pocket book I used to keep with me that had my most-used commands for trouble-
shooting these devices. Here are a couple of entries.

show access-list (name)

hostname# show access-list outside_access_in
access-list outside_access_in; 3 elements; name hash: 0x6892a938
access-list outside_access_in line 1 extended permit ip 10.2.2.0 255.255.255.0 any
(hitcnt=0) 0xcc48b55c
access-list outside_access_in line 2 extended permit ip host
2001:DB8::0DB8:800:200C:417A any (hitcnt=0) 0x79797f94
access-list outside_access_in line 3 extended permit ip user-group
LOCAL\\usergroup any any (hitcnt=0) 0xb0f5b1e1

Show Blocks
testASA#show blocks
SIZE MAX LOW CNT
4 1600 1597 1600
80 400 399 400

As my familiarity with the commands grew, I found that being able to quickly pull massive
amounts of information and make updates led to longer troubleshooting. I had to gather all of
these complex rules and know which ones applied to the incoming traffic, and then correlate

page 20
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

these to find the correct logical to physical name of the ACLs to their interfaces. If you find
yourself in the same situation, you have probably thought—as I did—that it must be easier.

Spoiler alert: It is.

Constantly shifting business and user needs create security holes and misalignment. These
needs include things like users requiring access to different sites and areas within the internal
network, setting up new internal applications that need to be accessed by the outside world,
and demanding technical solutions for what are ultimately management issues (“I need you to
keep Bob in accounting off Netflix®, but I still want to watch my show during lunch”). ACEs can
stack up and be placed in numerical order so they can be added when needed, without being
hit (the rules are not being tested since the previous ones are blocking them from getting to the
new rule that’s been applied). Or worse, there could be an incorrect rule placed too high, allowing
the wrong traffic through instead of having it filtered, denied, or sent elsewhere. Our ACLs can
hinder our service availability and leave us scratching our heads if we do not have the whole
picture of how they are being used.

BREAKTHROUGHS FOR MONITORING AND MANAGING ACLs

Monitoring and managing firewalls and ASAs, up to this point, has been overly complex. However,
now we finally have solutions that provide correlated data so we can truly take control. Have
you ever wondered how your traffic is flowing and being directed? Do you ever want to see your
logical and physical names of interfaces with their ACLs assigned, all within a single screen and
from one monitoring solution? Yeah, me neither. (Note the sarcasm in my tone.)

SolarWinds has the ability to take proven products and integrate! Their intelligence into a bo-
hemoth of ACL monitoring/management capabilities. Brace yourselves for this: backing up
your ACLs while offering change notification is here! Now I can compare my ACLs to other like
devices to validate and standardize the network. Also, if my ASA device goes down, I can now
know if the standby has successfully changed to active and not worry because an alert let me
know that a failover occurred with my ASAs. In case a hardware failure caused the failover and
I need a new device replacement, I can restore the device configuration quickly and accurately.
All of my ACLs and hard work make it appear like the failure never happened.

While ACLs on their own present a lot of challenges, security contexts—which present multiple,
separate sets of ACLs that may or may not be working together—are like going from Tic Tac Toe
to 3-D chess. Service providers who offer services to different customers use these security
contexts. Multiple users in different companies need their own secure transactions that apply
only to the services being provided. For example, say one is a large enterprise or college that
keeps departments separate via different security contexts within one ASA.

That said, managing ACLs within their specific context is crucial because they need precision
to filter packets properly. If I can view the impact of an ACL change and troubleshoot complex
ACLs within and across ASAs, I am in control. SolarWinds NAM delivers one single-pane-of-

page 21
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

glass solution with the capabilities of NetFlow, configuration management, events, and platform
health, allowing us to optimize ACLs by eliminating redundancy and shadow rules. Being able
to snapshot and version ACL configs keeps these accurate and orderly for training, network
standardization, and bare bones replacement.

Differencing

Highlighting the differences between the same devices, such as Cisco ASAs, allows me to
better manage my devices and protect the network from possible mistakes. Using compliance
reports with my business’ security protocols allows users the ability to quickly audit their devices
and help ensure proper configuration in large environments. Auto-remediation allows for quick
placement of dedicated ACLs that have to be on all like ASA devices.

Enumeration

The translation of interfaces from a physical name to a logical name (“outside,” for example)
allow you to see the interfaces and the ACLs that are residing on them from one location. Com-
bine this with NetFlow traffic for the interfaces, and you have a baseline that allows you to see
possible intrusions or traffic changes within your network.

Hit counts

Analyzing hit counts can reduce the number of unnecessary rules. This can also be used to
troubleshoot rules that are not being used because traffic is being denied or sent elsewhere.
Improve operational efficiency and optimize ACLs by automatically identifying shadow rules.
When you have the ability to filter, search, and view ACLs through rule browsers, you can find
and resolve issues quickly.

The bigger picture is integration for the win. When we integrate these metrics from different
locations and fully understand devices like Cisco ASAs, we improve service availability, decrease
human error, and optimize security. Visualizing the inner workings of the devices provides greater
awareness in monitoring and managing ACLs, which, in turn, immediately increases our security.

Managing and monitoring ACLs doesn’t have to be a headache caused by multiple tools. Impact
awareness and the ability to see traffic flow helps you make better decisions while cleaning up
redundant rules. We have crossed over into a new and exciting era of monitoring. Integration
without barriers allows us to see technology and present it to us as a solid solution. I am im-
pressed by that. I have always white boarded these ideas to make them actually make sense,
and now we have monitoring software that brings these ideas to life. It makes you wonder what
the next big thing will be, doesn’t it?

page 22
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

SOLARWINDS® NETWORK INSIGHT™ FOR CISCO® ASA

Standard monitoring tools don’t provide enough visibility into key firewall metrics. That’s why TRY NPM FREE
SolarWinds has launched the Network Insight for Cisco ASA, a feature of Network Performance
30 days, full version
Monitor (NPM) and Network Configuration Manager (NCM).

SolarWinds Network Insight for Cisco® ASA goes beyond basic up/down status. It provides TRY NCM FREE
comprehensive firewall performance and access control list monitoring.
30 days, full version

page 23
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

ABOUT THE AUTHOR

Destiny Bertucci is a Head Geek™ at SolarWinds and a Cis-
co Certified Network Associate (CCNA), Master CIW Design-
er, and INFOSEC, MCITP SQL, and SolarWinds Certified Pro-
fessional®. In her 15 years as a network manager, she has
worked in healthcare and application engineering, and was a
SolarWinds Senior Application Engineer for over nine years.

She started her networking career in 2001 by earning

CCNA/Security+ certification and launching a networking
consultant business. After using SolarWinds tools for many
years, she joined the company and continued earning certi-
fications and degrees to expand her professional reach into
database development and INFOSEC. Customizing Solar-
Winds products while working on setups and performance
deepened her knowledge of the complete SolarWinds prod-
uct line. She is now skilled and experienced in network,
security, application, server, virtualization, and database
management.

Follow Destiny on Twitter® @Dez_Sayz

Follow Destiny on THWACK® @Dez

Head over to solarwinds.com for more detailed information on products and pricing. You can
also download a free, unlimited (meaning you can load up as many devices as you want), demo
of any (or all) of the SolarWinds’ modules.

Pro Tip: There are also about two dozen free tools you can download over at
solarwinds.com/free-tools/

TO MY HUSBAND, TIM
Anyone who knows me understands that I keep pretty crazy hours, have rapidly cascading
thoughts, love new challenges, and push my own expectations of myself every day, all day. If
not for my husband, who keeps me grounded (without clipping my wings), I would not have been
able to attend trainings, earn certifications, or write an eBook in addition to everything else I do.
So, I dedicate this to the man who appreciates my geeky self, day in and day out.

page 24
THOU SHALT NOT PASS… I THINK?!: HOW TO BE A CISCO® ASA ACE

ABOUT SOLARWINDS
SolarWinds provides powerful and affordable IT management software to customers world-
wide, from Fortune 500® enterprises to small businesses, managed service providers (MSPs),
government agencies, and educational institutions. We are committed to focusing exclusively
on IT, MSP, and DevOps professionals, and strive to eliminate the complexity that our customers
have been forced to accept from traditional enterprise software vendors. Regardless of where
the IT asset or user sits, SolarWinds delivers products that are easy to find, buy, use, maintain,
and scale while providing the power to address key areas of the infrastructure from on-premises
to the cloud. This focus and commitment to excellence in end-to-end hybrid IT performance
management has established SolarWinds as the worldwide leader in both network manage-
ment software and MSP solutions, and is driving similar growth across the full spectrum of
IT management software. Our solutions are rooted in our deep connection to our user base,
which interacts in our THWACK® online community to solve problems, share technology and
best practices, and directly participate in our product development process. Learn more today
at www.solarwinds.com.

© 2017 SolarWinds Worldwide, LLC. All rights reserved.

The SolarWinds, SolarWinds & Design, Orion, and THWACK trademarks are the exclusive property of SolarWinds Worldwide, LLC
or its affiliates, are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other
countries. All other SolarWinds trademarks, service marks, and logos may be common law marks or are registered or pending
registration. All other trademarks mentioned herein are used for identification purposes only and are trademarks of (and may
be registered trademarks) of their respective companies.

page 25