Life in security has been one of sporadic accomplishment and constant humility. As in, I can resolve most firewall related problems, but other times just completely (and shamefully) miss the mark. Luckily, I work with some brilliant network security engineers who are there to help.
It started out with a simple change inquiry from our network operations center (NOC) for a request to configure Simple Network Management Protocol (SNMP) to monitor one of our Checkpoint firewalls (R80.30) interfaces at the security operations center (SOC).
NOC Request
SOC,
Please enable SNMP monitoring (if not already enabled) on the following cloud-based firewalls:
CloudProdP191999 Primary Firewall
CloudProdM191999 Secondary Firewall
The community string should be "cloudprodcommunity".
Then please allow the following traffic:
Source: 1.1.1.1
Destination: 2.2.2.2, 3.3.3.3
Service: SNMP (UDP port 161).
Action: Accept
Thank you.
NOC Services
This was, so I thought, a simple everyday firewall rule to apply to the policy, especially since the community string was already created.
Just as a reference, here is a quick overview of SNMP:
A protocol to monitor the health and status of IP networking devices
Consists of an SNMP manager and an SNMP agent management information base (MIB)
The SNMP manager monitors the information sent back from all the SNMP agents
SNMP agents can be configured on the devices which require health monitoring (routers, firewalls, load balancers, servers, wireless access points)
I proceeded to 2-factor into the Checkpoint firewall and create my change design for it. It read pretty much how the NOC request was formatted:
Change Management Design
<verify firewall policy name>
"fw stat"
pol.CloudProdM191999
Login to SmartConsole and open the SmartDashboard for management server.
Create new rule: Source: 1.1.1.1
Destination: 2.2.2.2, 3.3.3.3
Service: SNMP (UDP port 161)
Action: Accept
Push policy
Rollback
Disable or revert policy
After implementation, NOC stated their SNMP monitoring was NOT working. Okay, no problem, we'll just get on a conference call and figure it out with some packet captures. So immediately I setup a tcpdump filter:
"tcpdump -ni eth0 host 1.1.1.1 and 2.2.2.2"
The good news was that I could see the source traffic at least hit the firewall, 1.1.1.1. My first instinct is always to look for a TCP three-way handshake. But that wasn't possible with SNMP because it uses UDP, there is no handshake.
All I saw were ARP requests and replies when capturing traffic on the primary firewall's interface:
17:58:02.155495 arp who-has 2.2.2.2 tell 10.2.1.253 17:58:02.317444 arp who-has 10.0.0.96 tell 10.0.0.253 17:58:02.370446 arp who-has 10.3.1.12 tell 10.3.1.61
Seeing the ARP requests and replies, I naturally thought it was a Layer-2 MAC address/switching issue. I asked the NOC to check the switch which is in between the firewall and the external router. But they reported the switch fabric was handled by a higher tier division and they would need to be contacted. At this point, I was getting anxiety. I didn't want to disturb a whole another team over an issue that I just guessed would be the issue. If I winded up being wrong and it was a firewall issue, that would not look good. I'd look like an idiot (which happens a lot!).
I'm Luke Ahmed, CISSP. Not Luke Ahmed, CCIE. The CISSP comes easy to me, but I have to work very hard to understand technical networking concepts. I'm consistently making a complete fool out of myself in front of other senior engineers. But you know, it's worth the price of the knowledge gained in return. People also like humility, instead of someone who pretends to know it all.
The thing is, the NOC has a nice graphical user interface (GUI) to monitor SNMP statistics, while we just have command line access. So, the SOC has better insight into the traffic and packets exchanged than the NOC, as that is the power of a command line interface (CLI). As security professionals, the security engineers at the SOC prefer the CLI as it provides more insight into SNMP packets and how it works. This way, we are better able to defend and prevent against any disclosure of information. After all, SNMP is polling the most intimate health and system details of our firewalls, don't want that falling into the wrong hands.
Anyway, the packet captures also said something to the effect that "UDP port was closed". At this point, I thought the SNMP port on the firewall was closed. But that couldn't be, there was another interface which already had port 161 open. And I also put a rule on the firewall to open that port up. So what was wrong?!
Before escalating to the NOC, I wanted to confer with my senior network security engineer. I went to his cubicle and asked if he had a quick moment, he responded with a comforting "What's up man?" I know when the senior engineer says "What's up man?" the issue will be resolved. It happens every time.
I proceeded to give him a quick synopsis of the situation, provided him the packet captures, the source and destination traffic, and the logs.
He said "Thank you" and did not respond for 5 minutes.
5 minutes later all he said was "Ask them to test traffic again."
I didn't need to know the result, I already saw the full packet captures going back and forth, the SNMP traffic was working!
First thing was first, I told the NOC to try again, and of course they said now it works. I asked if they needed anything else and to let us know if more troubleshooting is required, and politely hung up the phone. Then asked the senior what he did to resolve the issue.
"You have to enable the SNMP agent on the interface through the GAIA portal".
Of course! Just putting a firewall rule on the firewall wasn't going to work, all that was going to do was allow the traffic through. But given how SNMP works, I didn't enable the SNMP agent on the firewall interface! That's part of the SNMP process! Being the senior, he knew about the high-level operation of SNMP, and just did some regular checks on the firewall to see if everything was configured properly. He knew how SNMP worked and was able to resolve the issue. The senior is not a CISSP and does not have a college degree. He just knows his stuff. Being a CISSP doesn't mean you know everything, or anything, it's the experience in the field which proves your skills.
Needless to say, from that day forward I made sure to know everything about SNMP and how it works with network traffic so I would never be in that desperate situation again.
I went back to the senior engineer and said "thank you!" to him. He said "No problem, Luke. Keep up the good work".
As I left his office, I thought I saw a slight grin on his face. But no, it couldn't be. That simply couldn't be possible. The senior engineer was always one of seriousness and professionalism. He never dared to lose that personality trait.
But, had it not been a professional setting, the senior engineer may have thought "It is worth the risk of showing a slight smile to Luke, in order to keep morale active. It is a part of my duty and responsibility".
I walked away quietly, dismissing the idea.
CISSP Take-Away Concepts
Domain 1: Security and Risk Management
Documentation would help new engineers learn Checkpoint command line commands such as "fw stat"
Without documentation, if the older engineers left the company, the new engineer may be faced with the same problem, and the cycle would repeat
Escalation
Having a senior engineer on hand to resolve issues is essential in a timely closure of troubleshooting tickets
CISSP Code of Ethics
Canon 3 states "Provide diligent and competent service to principals". I did not know how to resolve the issue and did not pretend to either. The key is to get the job done, even if it's a shot to your dignity. Escalating to someone who knows what they are doing ultimately resolved the issue.
Domain 4: Network Security
SNMP Monitoring
With SNMP, this experience related to understanding it is on the Application layer of the OSI model which uses UDP port 161
TCP Three-Way Handshake
SNMP typically uses UDP and not TCP, so there is no three-way handshake
ARP
Although I was seeing ARP request and replies, this was not the issue. Still, this was a topic from Layer 2 of the OSI Model in our CISSP books
Enabling SNMP on an interface
Part of understanding the overall structure of SNMP and what all is required for it to fully function properly
Domain 7: Security Operations
Change Management Process
Documented request submission and the creation of a text-based change design are part of a change management process. This way I'm not just making changes to the firewall without any approval.
Causing an outage as a mistake with documentation is excusable and can be defended by your manager. Causing an outage without being sanctioned to make changes, that can't be easily defended by the manager.