Troubleshooting network and TCP/UDP port

connectivity issues on ESX/ESXi (2020669)

Purpose
This article provides information on troubleshooting network and TCP/UDP port connectivity issues using these
troubleshooting tools:
ping/vmkping to troubleshoot network connectivity between two servers.
telnet to troubleshoot TCP port connectivity.
netcat (nc) to troubleshoot TCP port connectivity.
openssl to troubleshoot SSL port connectivity and verify SSL certificate information.
tcpdump and tcpdump-uw to collect packet traces to troubleshoot network issues.
netstat and esxcli network to view active TCP/UDP connections to the host.
Note: The telnet and nc tools help you to check if a TCP port is online or if there is a firewall blocking access to a
TCP port.

Resolution
Confirming network connectivity with ping and vmkping
To check if a remote host is online, you can use the ping and vmkping commands on ESX/ESXi host. The syntax of
these commands are:
# ping destination-ip
# vmkping destination-ip
You see an output similar to:
# vmkping 192.168.48.133
PING 192.168.48.133 (192.168.48.133): 56 data bytes
64 bytes from 192.168.48.133: icmp_seq=0 ttl=64 time=0.978 ms
64 bytes from 192.168.48.133: icmp_seq=1 ttl=64 time=1.009 ms
In this sample output, you can see that the ESX/ESXi host is able to communicate with the remote host with IP
address 192.168.48.133.
Note: On ESX hosts, the ping command is run from the network stack of the Service Console, while
the vmkping command is run from the vmkernel network stack, which is independent of the Service Console. On
ESXi, theping and vmkping are the same command and run from the vmkernel network stack because there is no
Service Console in ESXi.
For more information on using the ping command, see Testing network connectivity with the ping command
(1003486).

Confirming connectivity to a TCP port with telnet

Note: Telnet is available only on ESX hosts. For

ESXi 3.5, 4.x and 5.x, you will need to use the
netcat (nc). Please see the section below titled "Confirming connectivity to a TCP
port with netcat" for further information.
While the ping command confirms connectivity, it does not necessarily mean that all TCP ports on the remote host
can be reached. It is possible for a network firewall to allow or block access to certain ports on a host.
To check if specific TCP ports are running on the remote host, you can use the telnet command to confirm if a port
is online.
# telnet destination-ip destination-port
When trying to establish a telnet connection to TCP port 80, you see an output similar to:
# telnet 192.168.48.133 80
Trying 192.168.48.133...
Connected to 192.168.48.133.
Escape character is '^]'.
In this sample output, you can see that you are connected to port 80 (http) on the server with IP
address 192.168.48.133.
If you choose a port number for a service that is not running on the host, you see an output similar to:
# telnet 192.168.48.133 81
Trying 192.168.48.133...
telnet: Unable to connect to remote host: Connection timed out
In this case, you can see that there is no response when you attempt to connect to port 81 on the server
192.168.48.133.
Note: Telnet is an application that operates using the TCP protocol. UDP connectivity can not be tested using Telnet.

Confirming connectivity to a TCP port with netcat

The telnet command is not available in any versions of ESXi and, therefore, you must use
netcat (nc) to confirm connectivity to a TCP port on a remote host. The syntax of
the nc command is:
# nc -z <destination-ip> <destination-port>
When testing connectivity to TCP port 80, you will see an output similar to:
# nc -z 192.168.48.133 80
Connection to 192.168.48.133 80 port [tcp/http] succeeded!
In the sample output, you can see that you are able to establish a connection to TCP port 80 on the host
192.168.48.133.

Note: Netcat includes an option to test UDP connectivity with the -uz flag, but because
UDP is a connectionless protocol, it will always report as 'succeeded' even when ports
are closed or blocked. Instead, test bi-directional UDP connectivity using tcpdump or
tcpdump-uw.
The nc command can also be used to check the connectivity to a range of TCP ports on a remote host:
# nc -w 1 -z 192.168.48.133 20-81
Connection to 192.168.48.133 22 port [tcp/ssh] succeeded!

Connection to 192.168.48.133 80 port [tcp/http] succeeded!

The -w option specifies a timeout value.
Note: Port scanning is a very powerful troubleshooting tool, but may be against your company network or security
policies. Check with your network or security team to ensure that they are aware of this activity.

Testing SSL port connectivity and certificate information with openssl

To test SSL ports, you can use the openssl command to test connectivity and also to confirm the current SSL
information. This can be useful when confirming SSL certificates with vCenter Server. The syntax of
the opensslcommand is:
# openssl s_client -connect destination-ip:ssl-port
You see an output similar to:
# openssl s_client -connect 192.168.48.133:443
CONNECTED(00000003)
Where 443 is the default SSL port.
In this sample output, you can see that connection to the remote server 192.168.48.133 over the SSL port
was successful.
Note: The output may contain considerable information regarding the SSL certificates, which may be useful in
troubleshooting certificate issues.

Collecting packet traces using tcpdump and tcpdump-uw

ESX and ESXi hosts come with the packet tracing tools, tcpdump and tcpdump-uw, which can be used to collect
network traces. The network traces are useful in troubleshooting network issues.
tcpdump can be used on ESX hosts to capture packet traces from the Service Console (vswif) interface.
For more information on using tcpdump, see Verifying gateway IP connection using the tcpdump
command (1008017).
To configure a vSwitch to monitor a virtual machine virtual network card, see Capturing virtual switch
traffic with tcpdump and other utilities (1000880).
tcpdump-uw can be used on ESXi hosts to capture packet traces from a vmkernel (vmk) interface.
For more information on using tcpdump-uw, see Capturing a network trace in ESXi using Tech
Support Mode or ESXi Shell (1031186).

Viewing active TCP/UDP connections with netstat and esxcli network

When troubleshooting network connectivity issues, it may be helpful to see all the active incoming and outgoing
TCP/UDP connections on an ESX/ESXi host. ESX hosts can use the netstat command and ESXi 4.1 and later
hosts can use esxcli network to show the list of TCP/UDP connections. The commands are:
ESX 3.5/4.x # netstat -tnp
ESXi 4.1 # esxcli network
ESXi 5.0 # esxcli network
ESXi 5.1 # esxcli network
ESXi 5.5 # esxcli network

connection list
ip connection list
ip connection list
ip connection list

Sample output from an ESXi 4.1 host:

# esxcli network connection list

Proto Recv-Q Send-Q Local Address
World ID
tcp
0
52
192.168.48.136:22
tcp
0
0
127.0.0.1:62024
tcp
0
0
127.0.0.1:57867
tcp
0
0
127.0.0.1:62196
tcp
0
0
127.0.0.1:8307
5790
tcp
0
0
127.0.0.1:52943
5790
tcp
0
0
127.0.0.1:80
5785
tcp
0
0
127.0.0.1:55629
6613
tcp
0
0
127.0.0.1:8307
5785
tcp
0
0
127.0.0.1:56319
5785
tcp
0
0
127.0.0.1:80
5166
tcp
0
0
127.0.0.1:62782
6613
tcp
0
0
127.0.0.1:5988
tcp
0
0
127.0.0.1:53808
5166
tcp
0
0
127.0.0.1:8307
5788
tcp
0
0
127.0.0.1:56963
5785
tcp
0
0
127.0.0.1:8307
5031
tcp
0
0
127.0.0.1:8309
5031
tcp
0
0
127.0.0.1:5988
tcp
0
0
0.0.0.0:5989
tcp
0
0
0.0.0.0:80
5031
tcp
0
0
0.0.0.0:443
5031
tcp
0
0
127.0.0.1:12001
5031
tcp
0
0
127.0.0.1:8889
5331
tcp
0
0
192.168.48.136:427
tcp
0
0
127.0.0.1:427
tcp
0
0
0.0.0.0:22
tcp
0
0
0.0.0.0:902
tcp
0
0
0.0.0.0:8000
4801
tcp
0
0
0.0.0.0:8100
4795
udp
0
0
192.168.48.136:427
udp
0
0
0.0.0.0:427

Foreign Address

State

192.168.48.1:55169
127.0.0.1:5988
127.0.0.1:5988
127.0.0.1:5988
127.0.0.1:52943

ESTABLISHED
TIME_WAIT
TIME_WAIT
TIME_WAIT
ESTABLISHED

127.0.0.1:8307

ESTABLISHED

127.0.0.1:55629

ESTABLISHED

127.0.0.1:80

ESTABLISHED

127.0.0.1:56319

ESTABLISHED

127.0.0.1:8307

ESTABLISHED

127.0.0.1:62782

ESTABLISHED

127.0.0.1:80

ESTABLISHED

127.0.0.1:53808
127.0.0.1:5988

FIN_WAIT_2
CLOSE_WAIT

127.0.0.1:56963

CLOSE_WAIT

127.0.0.1:8307

FIN_WAIT_2

0.0.0.0:0

LISTEN

0.0.0.0:0

LISTEN

0.0.0.0:0
0.0.0.0:0
0.0.0.0:0

LISTEN
LISTEN
LISTEN

0.0.0.0:0

LISTEN

0.0.0.0:0

LISTEN

0.0.0.0:0

LISTEN

0.0.0.0:0
0.0.0.0:0
0.0.0.0:0
0.0.0.0:0
0.0.0.0:0

LISTEN
LISTEN
LISTEN
LISTEN
LISTEN

0.0.0.0:0

LISTEN

0.0.0.0:0
0.0.0.0:0

0
0
0
0

0
0

0
0
0
0

0
0

udp
4693
udp
4795
udp
4686
udp
4686

192.168.48.136:68

0.0.0.0:0

0.0.0.0:8200

0.0.0.0:0

0.0.0.0:8301

0.0.0.0:0

0.0.0.0:8302

0.0.0.0:0

To retrieve errors and statistics for a network adapter, run this command:

# esxcli network nic stats get -n <vmnicX>

Where <vmnicX> is the name of a NIC in your ESXi host.

netstat for ESXi

The title of the post is actually misleading - on purpose - because there is no netstat for ESXi. The reason that I
bring this up today is because of a Twitter conversation from today regarding SSH access and VMkernel interfaces. I
was looking to see which ports were open and what interfaces were listening.
But that is a different post.
What is netstat? according to Wikipedia:
netstat (network statistics) is a command-line tool that displays network connections (both incoming and outgoing),
routing tables, and a number of network interface statistics. It is available on Unix, Unix-like, and Windows NTbased operating systems.
Why would you use it? For one thing for example, to check if a host has an open connection on a certain port, if it is
listening on a certain port - for troubleshooting purposes would be the proper answer.
So how do you get that information on ESXi?
Trying netstat on an ESXi host does not work - because that command is not there - see the screenshot below.

Well that is not good - if the command is not in the busybox console then how would you go about getting that
information? Well of course the clever people at VMware have already thought about this and have exposed all this

information through esxcli. William Lam wrote a great set of posts on esxcli
esxcli Part1 - What is esxcli?, esxcli Part2 - Automating esxcli using vMA and esxcli Part3 - Automating esxcli
using PowerShell
This is how you would go about getting the information from esxcli. (Be aware the command differ according to the
different versions - 4.x is not the same as 5.x)
?

esxcli network ip connection list

That is fine and dandy - but to get that info you need to either:
1.
or
2.

have access to the DCUI (and have it enabled of course)

access remotely with SSH (and also have it enabled of course)

But what if you do not want to enable neither of the above - that means you have to do it remotelyand for that you
have two options, vCLI or PowerCLI.

The vCLI way

?

esxcli --server esx1.maishsk.local network ip connection list

But me being more of PowerCLI guy I would do it like this.

The PowerCLI way

?

1
2
3

$esxcli = get-esxcli -vmhost esx1.maishsk.local

$esxcli.network.ip.connection.list() | ft

Output is almost identical - just that in the case of PowerCLI the values are returned as a set of objects - a
VMware.VimAutomation.ViCore.Impl.V1.EsxCli.EsxCliObjectImpl object to be precise. Once these presented as
objects I can start to mold and dice my results to my liking.

For example - I would like to check if there is any connections open on port 80 (http) - with vCli - this is not so
simple - because you are working essentially in a DOS window - so filtering is not the easiest with findstr. Using the
console or SSH is easier - a simple grep will work as you can see below.
?

esxcli network ip connection list | grep :80

With PowerCLI
?

$esxcli.network.ip.connection.list() | where { $_.LocalAddress -like "*:80" } |

ft

I hope you can see that the options this way are pretty much endless - like filtering all connections to show only
those from a specific IP, or a complete subnet.
So that is how you netstat on ESXi.