There are times when the security professional no matter how much technical knowledge, certifications, college degrees, SAT scores, or job interview skills...has to rely on just instincts. Has to rely on their own "qualitative analysis" - a term to know for the CISSP exam.
The other day at work we received a ticket from the project manager of a medium-sized financial company to decommission their primary HA pair firewalls. This is normal, for project managers to send tickets when they would like to take a firewall out of their environment. It is also a proper way to uphold the disposal phase of the systems development life cycle - another CISSP concept to understand.
For decommission/termination tickets we verify with the project manager the IP address, hostname, and networks the firewall is protecting before we start powering them down and taking them out of the data center. Because we don't want to EVER shut down the wrong firewall and take them out of a production network! It's happened before, and it was not a pleasant day for us or the customer. People were written up, tickets were being escalated, and there was a lot of yelling involved. It was a Checkpoint firewall and we had to recover the policy and objects using the objects_5_0.C file. A complex process done by our senior technical engineer.
Working at the SOC for several years now and having engaged this particular customer multiple times on troubleshooting calls, it came as a slight shock that they wanted their primary firewalls terminated. I chalked it up to maybe they were getting them migrated from physical to virtual, or switching from Checkpoint firewalls to Palo Altos, or just leaving as one of our customers. Which sucks, because the way I think, if customers are terminating firewalls, then we're losing customers and I might be out of a job. Either way, nobody likes to see termination tickets.
The ticket came in at 5pm in the evening, and as team lead, I told the nightshift crew to verify with the project manager once again that they for sure wanted these firewalls deprovisioned. The night crew sent an email, but by then the PM had probably gone home and never got back to us.
The next day, the same customer had a troubleshooting ticket come in the ticketing queue. I thought maybe they want some last-minute traffic to work before powering down, so I assigned the ticket to myself and joined the conference bridge. They wanted to allow an external IP to access one of their database servers. Simple enough, just needed a policy created on the firewall.
While waiting for the policy to push from the management server to the firewalls, I just happened to ask the customer "Hey Casey, sorry to hear you are having the firewalls terminated, we saw the decommission ticket come in last night. Are you guys switching firewalls or something?"
There was a slight pause. Casey replied back with "What? We're not leaving? What ticket?"
A thousand things just rushed through my mind all at once:
"Is Casey unaware of his own infrastructure? Were we supposed to wait until terminating the firewall? Maybe the project manager got the order to terminate the firewalls from someone with more authority than Casey..." Either way, it's one of those uneasy feelings you get when you know something bad is about to happen.
"Yeah uhh...your project manager opened a ticket with us yesterday asking to terminate the perimeter firewalls...?"
"WHAT?" - Customer.
A few crucial seconds pass as both of us try to comprehend the effect of what is going on.
"What was the project manager name on the ticket?" - the customer asked hastily.
"It was <project manager name>" - I responded as fast I could.
"Oh my god....that PM is set to leave the company in 5 days....he left on bad terms...DO NOT TERMINATE THOSE FIREWALLS! IGNORE THAT TICKET!!"
Avoiding the sudden surge of anxiety, I composed myself and thought "Well, if I'm on the firewall, and on the Checkpoint SmartDashboard, these firewalls are still operational, nobody has taken them down."
Doing a "fw tab -t connections -s" on the command line showed all the connections, there were about 4,000. Normal production traffic. Phew!
Just to be sure, I checked the ticket the project manager submitted, and saw that it had not been updated since our night crew asked to verify the firewall IP addresses.
So for now, no action had been taken on the termination request.
The next day we found out the truth: the PM had quit his job in a disgruntled manner and was supposed to leave by the end of the week. However, since he still had the authority to open tickets with the SOC, he submitted one to take down the company firewalls to exact some sort of revenge!
And it would have almost worked because of the following reasons:
1. PM was authorized to initiate such requests
2. A proper change management process was followed
3. A ticket instead of a change request was created
Everything was done by the book for this malicious action to take place, and it almost did!
This situation was averted with pure luck, because 1) we asked to verify the termination, 2) we just happened to ask the customer on an unrelated conference call about the termination.
Two strokes of luck which saved a potentially catastrophic situation. If we had terminated those firewalls, it means we would have deleted everything. The firewall policy, the NAT rules, the objects, the CMA on Provider-1, and wiped the firewall clean and taken them out of the data center. It would have taken at least 24 hours of manpower to bring everything back up again as they were. Not to mention frustrated supervisors, engineers, executives and account managers. Luckily, it wouldn't have been our fault since it was a sabotage attempt by an employee internal to the customer. But still, it would have been a lot of work on our part to bring the firewall back up.
It was later found that that the project manager was asked to leave the company premises immediately after the conference call, and his laptop seized and access revoked. It was lucky that I happened to ask the customer about the termination, and I'm not trying to show I'm some super security professional, but the chain of events was just too coincidental.
Firewall termination? But customer still requesting policies? Weird.
I thank the years of experience and learning from people MUCH smarter than me in the security field in avoiding a difficult situation and to trust my instincts.
CISSP Take-Away Concepts
Domain 1: Security and Risk Management
Upholding Availability
The firewalls were an HA pair, high-availability. This means that if one firewall went down, the other would take over. But, the termination ticket was to take both the firewalls down, this would not have upheld the core CISSP concept of availability.
Domain 3: Security Engineering
Insider Threat
The PM may have been a normal employee months ago, but recently due to conflicts with the organization, he was an insider threat. This is an example of why the insider threat is the biggest threat to a company, they have intimate access and knowledge of the inner workings. No amount of due care or due diligence measures can stop a determined insider threat.
Domain 4: Network Security
Firewall Policy
On the conference call a simple security policy rule was required in order to enable access from the Internet to an internal server.
Domain 7: Security Operations
Proper Change Management
A proper change management is good for documentation, rollbacks, and maintaining operational effectiveness. In this case, it was used as a tool for harmful purposes: to take down firewalls.
Life in the information security sector is funny sometimes. Here are some more stories if you care to read it: