Somebody hacked one of our customers recently, and it was our fault.
While we did perform due care, we did not perform due diligence.
It started out with a failing hard drive on a firewall. We started to receive hard drive sector error alerts and decided it just best to swap out the firewall with another one before it failed completely.
We performed our due care by making sure the new firewall had the same routes, default gateway, and interface settings as the old failing firewall.
Then we made sure both the old and new firewalls matched in their firmware, hotfixes, and updates.
We even made sure to schedule the swap for a time when the customer would experience the least amount of traffic to their environment, so not during business hours.
During the maintenance window, we took the new firewall from the staging area, and swapped it out with the customer’s failing device in their production environment.
Here’s a diagram of the two environments:
We also saw this ACL that allowed ANY traffic to the customer servers.
Click to enlarge
If it were a Cisco ASA, the ACL would look something like this:
ciscoasa(config)# access-list outside_access_list extended permit tcp any object-group ALL_INTERNAL_SERVERS eq 3389
It is a high security concern to have a source of “ANY” on a firewall. This would mean anybody on the Internet would have access, and in this case, access through Remote Desktop Protocol.
In this case however, the customer had multiple perimeter edge firewalls in their production area to filter traffic before this “ANY” rule would take effect.
But then how was it hacked by someone from the Internet?!?
Because we didn’t do our due diligence.
We were so focused with due care on the firewall, we didn’t look at the bigger picture, which was the staging area network. We came to find out the staging area network environment, did not have perimeter firewalls like the customer’s production area.
Which means that anybody from the Internet could directly access the new firewall we were preparing, and they did this while the new firewall was being configured in the staging area.
Sometimes it’s so easy to forget the bigger picture at work when you’re trying to make sure the task at hand gets completed perfectly.
Due care is taking the immediate necessary steps, and due diligence and researching the bigger picture before practicing due care.
Mistakes like this will always happen.
We are humans after all, not machines.