A full YouTube video of this blog post can be found here: https://youtu.be/aMEPK0L0d0U
When there are network connectivity issues, the first thing to be blamed is the firewall.
40% of my job is proving that the issue is not with the firewall. The simplest of tasks is just showing the customer that traffic has ingressed and egressed successfully out of a dual or multi-homed firewall.
We manage the firewalls of Customer A. We do not manage the switches, routers, load balancers, or the servers for Customer A. We just deal with firewall issues.
Customer A added a new server to their environment on Sunday evening. Connectivity was working fine at this point.
Monday morning, connectivity is broken, and Priority 1 phone calls start rolling into the SOC. I get the call of a frustrated project manager who is demanding to check the firewall as to why traffic is suddenly dropping to the server.
I ask them for a source and destination IP address in order begin capturing packets and figuring out where things are going wrong.
Below is an example of a packet capture performed on a Cisco ASA 5525 firewall.
SNT001/sec/act# sh cap cap
22 packets captured
1: 13:17:52.560166 802.1Q vlan#202 P0 10.45.1.66.57799 > 10.2.1.179.111: S 3436187463:3436187463(0) win 5840 <mss 1460,sackOK,timestamp 2465474 0,nop,wscale 7>
2: 13:17:52.560517 802.1Q vlan#202 P0 10.2.1.179.111 > 10.45.1.66.57799: S 3852648546:3852648546(0) ack 3436187464 win 5840 <mss 1380,nop,nop,sackOK>
3: 13:17:52.560655 802.1Q vlan#202 P0 10.45.1.66.57799 > 10.2.1.179.111: . ack 3852648547 win 5840
4: 13:17:52.560685 802.1Q vlan#202 P0 10.45.1.66.57799 >
10.2.1.179.111: P 3436187464:3436187524(60) ack 3852648547 win 5840
5: 13:17:52.560792 802.1Q vlan#202 P0 10.2.1.179.111 > 10.45.1.66.57799: . ack 3436187524 win 5780
Remember how you're studying in Domain 4: Network Security about the TCP Three-way Handshake? Well the above output is a real-life example of it. We use this simple output to determine if a connection has been established or not. Or just if traffic is traversing the firewall. Traffic could be going through the firewall, but being dropped somewhere else.
Click here to watch "The TCP Handshake" video if you are not familiar with this concept right now.
In Line 1, a SYN packet is being sent from the source to the destination.
Source: 10.45.1.66
Destination: 10.2.1.179
S: SYN
In Line 2, a SYN/ACK is being sent back from the destination to the source.
Source: 10.2.1.179
Destination: 10.45.1.66
S...ack: SYN/ACK
In Line 3, an ACK is being sent from the original source back to the destination.
Source: 10.45.1.66
Destination: 10.2.1.179
ack: ACK
In Line 4 and 5, you start to see data flowing back and forth via the "P" flag. P stands for Push.
Source: 10.45.1.66
Destination: 10.2.1.179
P: Push (Data)
Because firewalls are a Layer 3 device, and because TCP is a Layer 4 protocol, we were safely able to tell the customer that this is not a networking issue on the firewall. They would need to either check the networking devices in between the source and destination, or they would need to check OSI Model Layers 5, 6, or 7.
We came to find out that the configuration on the load balances which resides right before the server was changed on Monday morning by another party without proper approval. That was the cause of their connectivity issues.
CISSP Concepts
Domain 4: Network Security
Transmission Control Protocol
Three-way Handshake (the cornerstone of all TCP connections, without it you have no TCP connection)
IP Networking
Your basic network troubleshooting using the TCP Threeway Handshake
MPLS
The IPs involved in this troubleshooting are private IP addresses connecting within an MPLS. They are not going out to the Internet.
Domain 7: Security Operations
Change Management
Load balancer team did not follow proper change protocol, otherwise all parties would have been involved, as well as interdependencies.
Other Stories of a CISSP