Thank you for taking your precious time to read this article.
What is commercial-off-the-shelf software?
Commercial-off-the-shelf (COTS) software is a new 2021 CISSP exam topic. COTS means that you are buying a pre-made program, you are not developing it in-house. In this regard, most organizations have COTS in some form whether purchasing it from a third-party vendor, downloading it locally to your hard drive, or having it delivered to you as a Software-as-a-Service.
Think of it this way: if you want to wipe your hard drive, you can download open-source software like DBAN. If you want a remote VPN client to encrypt your traffic, you can purchase a license from a third-party vendor. Or if you wanted to subscribe to a CISSP course unlike any other, you can have it delivered to your browser using a software-as-a-service.
I mention all these topics because you need to know their differences for the CISSP exam per section "8.5 Assess security impact of acquired software", from the 2021 ISC2 CISSP Exam syllabus:
Open-source software, third-party software, and software-as-a-service can be considered commercial-off-the-shelf software, the differences are in how they delivered. Basically, if you're not developing software within your own organization, you have COTS.
Why do you have to know COTS for the new CISSP exam? After all, the CISSP syllabus focuses more on your ability to secure the development practices of in-house software. We are more consumed with security development approaches, software life cycles, change control, configuration management, and even database concepts and security issues. Now, with the 2021 exam, it is also putting emphasis on the security of software acquired external to your organization. The truth is that securing and examining the risk of commercial-off-the-shelf software was always there in the syllabus, it is now being brought to more attention with the increase in commercial applications in the form of SaaS or third-party vendors.
Core COTS CISSP Concepts
Always conduct a risk assessment and test commercial software for security vulnerabilities before integration into your environment. Testing can be conducted on the software itself by an in-house team or by a third-party testing company. The goal being to check if the COTS software will help the company with their business function or introduce new risk. With generic "one-size fits all" software like COTS, the risk of embedded malware, backdoors, coding flaws, or hurried development practices that leave behind vulnerabilities are high security concerns. COTS are well-known and widely available, which means even attackers have access to them. Attackers can obtain the same COTS software that is used by your organization. They can scan through the lines of code, learn more about it, find vulnerabilities, and then use them against your own organization.
You should try to make sure that your organization knows more about a COTS software than an attacker.
COTS can be used to perform the following attacks:
DDOS
Data leakage
Disclosure of private user or business information
System modifications
Reputation damage (if COTS software is modified to ruin or embarrass a business)
It is best to do your due diligence on commercial-off-the-shelf software, your Chief Information Security Officer will thank you later.
As you will see from my personal experience below, the security and business requirements of legacy systems that will use COTS software is just as important as integrating COTS with new systems.
I remember a particular experience with COTS software back when I was a junior systems administrator in a small Washington DC law firm.
Migrating from Microsoft Exchange Server to Lotus Notes
The law firm was a place to learn my IT skills from the very bottom of the corporate ladder - as a junior systems administrator. It was ideal by the standards of my harsh Bengali math teachers when I was a child. It was complex, tricky, and with a lot of high-pressure - it was at times mentally uncomfortable; and a mistake could cause anxiety that would be very hard to live down.
These were also not the days of YouTube explainer videos or Reddit forums dedicated to one extremely narrow topic of conversation. It was the early to mid-2000s and the Director of IT and I were trying to migrate from a legacy Microsoft Exchange mail server to IBM Lotus Notes. Gmail or Office365 were not around or integrated as a corporate email system at this time. We were still using the Outlook 2000 email client on a Windows XP operating system!
Looking back, those days as an "IT fix-it guy" with predictable Windows blue screen memory management errors or the "Low Memory" balloon appearing in the right-side of the XP taskbar...are dearly missed. Such simple technology problems back then, not the onslaught of packet-level network routing protocol troubleshooting and urgent client demands I have to resolve today. Some great days of my career. Although I am only remembering the happy times, and not some of the more grittier situations.
In order to migrate from Exchange to IBM Lotus Notes we had to convert Outlook .PST files to ones compatible with Lotus Notes, known as .NSF files. PST files are data files that contain all of a user's emails, folders, calendar, contacts, tasks, and reminders, and can be saved as an exportable file to another Microsoft Outlook client. But in order to migrate and convert this file to .NSF and make it work on Lotus Notes by IBM, we needed a third-party software, a "commercial-off-the-shelf" software.
We'll call this software tool "COTS #1" as I don't want to mention the real software name. I'm not one to bash a product, I'll mention it only if I have something nice to say.
.PST to .NSF Conversation Error
I used the company credit card to download and purchase the full version of COTS #1. It was a simple interface that required us to (1) upload a single or a group of .PST Outlook mail files, (2) choose .NSF as the converted format, and (3) run the conversion. Simple enough!
But even as a junior sys admin with just 2 years of IT experience, I knew to expect errors. I just held out hope that the errors were minor and not major. The only thing that should be major is my performance at work.
Early into the conversion process, and like clockwork, the first error message appeared after 3 minutes. This was the general text of the error message:
"Mail file too big! Please reduce the size of the file".
I can't remember exactly, but the .PST file was somewhere around 5GB in size, and COTS#1 couldn't handle that big of a file size. I had to reduce the size of the .PST file to continue. This was already becoming a big roadblock to the migration project because most of the user .PST files were rather large around the 5GB mark. If we had this issue with every .PST file, this migration was not going to happen - no way management wanted a disruption or delay to their daily emails - ask any lawyer!
I had to figure out a way to reduce the .PST file size!
Before we continue, you should know something about the lawyers at this firm: they considered their "Deleted Items" folder a secondary storage folder. As in, if they deleted an email from their Inbox, they considered it stored in the Deleted Items folder for later use. They wanted to refer to this email again if necessary, and didn't consider the Deleted folder for its intended purpose.
I was not aware of this secondary "storage" strategy by the lawyers. When I delete something, I consider it something I won't use again, CALL ME A WEIRDO.
Reducing the .PST file size
When I opened the .PST file to look and see if I could delete some items, I saw over 33,000 emails in one of their Deleted Items folder! Emptying this folder would for sure reduce the size of the .PST file and allow it to continue conversion process.
I used Outlook's "Mailbox Cleanup" tool as well as the "Empty Deleted Items Folder" option to remove some emails.
Checking the new .PST file size, it was now around 3GB! I had successfully reduced the .PST file size!
I ran the file conversation tool again. It converted to .NSF with no issues. Now for testing purposes, we decided to use just this one .PST file of 1 user (the mail file belonged to one of the Vice-Presidents of the company!) as a test for bringing up the email on the Lotus Notes application. If it worked with the VP's email, we would then continue the migration with the rest of the users.
"Where are all the emails in my deleted folder? I need those!"
Once the newly created .NSF file was uploaded to the Lotus mail platform, it brought everything up. We asked the VP to check out his new email setup. All the emails that were in the Vice-President's Outlook Inbox was still there, all the emails from all of his other folders were there. All his contacts, calendar appointments, and reminders were all there.
But then the Vice-President asked the best question I have ever heard in my time in the IT sector:
"Hey, where are all the emails in my deleted folder? I need those!"
My Director immediately hopped to the VP's computer and looked at the Trash folder, "Doesn't look like anything is there. Maybe it's in a different folder in Lotus, let me look around..." He said curiously squinting at the monitor.
But I knew it would be no use. I knew the Director would not find anything. I knew exactly where those deleted mail files were, they were DELETED! I deleted them to reduce the .PST file so I could run the .PST to .NSF file conversion on COTS#1.
Welp, time to admit it. You know, I wasn't even scared or nervous. I did what I had to do. I deleted emails in the Deleted folder, because that's what is supposed to happen.
I confidently looked at the Director, without looking at the VP "Oh, I deleted those during the migration process..." The Director craned his neck back to look at me, his hand now lightly taken off the mouse, realizing now that looking for the VP's emails no longer mattered.
I continued, "Remember, Director? I told you there was an error and we had to reduce the file size? I deleted everything in the Deleted folder and did some other general clean-up."
I then looked at the VP and asked him innocently "Did you...need all the emails in the Deleted Folder?" Looking back, I will freely admit this was a dumb question to ask him at this particular point of time. Not to mention, I just unintentionally blamed everything on the Director. A decade later, I still feel bad about it.
The high-powered VP didn't even look at me or acknowledge my existence in this vast Universe. "Yes! I need all those emails back!" His eyes shot arrows toward the Director. "I need those emails back as soon as possible. They have important client information. Get them back please."
My Director of IT was a good open-minded individual. But when decisions had to be made and action had to be taken, he would stare down the ranks of corporate executives and even the founder of the company. So it came to me as a small surprise when he stated the following ""Okay, we will do our best". It was so melancholy, sad, defeated - and I was the reason!
The VP stormed out of the room; his $1,200 Italian leather shoes heard stinging down the hallway.
The Director looked at me "Dude...why'd you delete all those emails!?"
Now, back then I was young and still new to this IT career, but c'mon! I only had one response, "Director, I deleted the emails in the DELETED FOLDER. They were in the Deleted folder! I didn't know that was his like...storage folder?"
The Director sighed. He knew I was right. "<big exhale> The people who work here are strange, Luke. I won't deny that. You're going to find certain quirks like that in every company (he was right). It's all in the way you deal with them. We have a job where people tell us what to do, we don't get to tell others what to do. They need something to do their job properly, we gotta make it happen for them. But, it's fine Luke. I'm going to see if the mail server itself has a backup of all his emails including the Deleted Items, maybe I can sync them back to his Outlook. Try to see what you can do while I work on that."
Did I ever tell you the IT Director at this job was my mentor and the biggest influence in my information security work ethic? He knew how to encourage without insult, share knowledge without any secrets, and lead by example. He was Director of IT, but once in a while others would still see him crawling underneath desks replacing faulty surge protectors.
The Corrupter
I found a random solution to recover the deleted files from the .PST file. I was desperately hoping it would work. No matter what my opinion was about the function of a Deleted Items folder, it was still my fault and I had to fix it. If you want to make it in the corporate world, you have to put your personal opinions and feelings aside, and play the game like a professional.
Back then Google was just starting to become the powerhouse search engine that it is today, so blogs articles, forums, and posts regarding specific information technology issues were scarce. But I did find this one obscure article that showed a way to restore items in the Deleted Folder by messing with the hex editor. If you don't know what a hex editor is then you probably shouldn't be messing with it. I had no idea what it was, but I had no choice but to mess with it.
Here's what I had to do in order to try and recover the items from the Deleted folder:
Backup the original .PST file
Purchase another COTS software that was a hex editor, let's call it COTS#2
Open the .PST file in COTS#2 hex editor
Edit the hex that comes up by deleting or rearranging certain characters and positions
Call me Chow Yun-Fat because I was now the corrupter! By editing the hex and deleting some numbers in the hex editor, I was intentionally "corrupting" the .PST. By purposely "corrupting" the .PST I could then run it through another COTS software included with Outlook known as SCANPST.EXE (COTS #3).
Once SCANPST.EXE scans the corrupted .PST, it should be able to repair it AND recover the deleted items
Why did I need to first corrupt the file in order to recover the emails from the Deleted folder?
What actually is happening in this whole process is that when you delete something, Outlook does not actually completely delete everything. What is actually removed is the index pointer that tells Outlook where the Deleted Items are located. When we "repair" the .PST using SCANPST.EXE, the COTS software is just "rebuilding" the original index pointers. Once this is done, the files should show back up in the Deleted Items folder. This technique also did not work for me using the Outlook Repair tool, so I had to look for a COTS#2.
After the corruption, we scanned the .PST file with the Microsoft Outlook Repair Tool aka SCANPST.EXE aka COTS#3. Once SCANPST.EXE finished its process, we loaded up the newly repaired .PST file onto the VP's Outlook client.
The seconds that were passing as the Microsoft application loaded up were one of those tense moments where if it worked, it would be a good day. If it didn't work, there would be many more hours of troubleshooting and frustration - not to mention an increasingly agitated Vice-President looking for his important emails (that were in his Deleted folder!).
The Director and I both let out unrehearsed yet synchronized sighs of relief as we saw that the VP's Deleted Folder once again had 33,000 emails back. PHEW! What a day!!
As you can see, my personal experience with COTS wasn't so much a security issue as it was a business disruptor (thinking like a manager, this is of greater concern). It wasn't a big problem but rather an annoyance that prevented a smooth migration from legacy to a new system. Unfortunately the project was put on hold, and by the time it was once again initiated, I had moved on to another company.
Here is a summary of what happened with each COTS software:
COTS #1 - .PST to NSF Conversion Tool
We used this tool to convert .PST files from Microsoft Outlook to .NSF files for IBM Lotus Domino.
COTS #2 - Hex Editor
After I deleted everything in the Deleted Items folder in Outlook, I used COTS#2 to edit the hex numbers in order to "corrupt" the .PST file so it could be repaired by COTS#3.
COTS #3 - SCANPST.EXE
COTS#3 repaired the corrupted .PST file and all the deleted emails from the Deleted Items folder came back.
Lessons Learned
Since COTS is made by another company, we can't always tailor it to our needs
We should perform our due diligence to reduce the risk of the unseen effects of COTS
Just because COTS is on the new CISSP exam syllabus doesn't mean it always has something to do with security, it also has to do with how it works with and impacts the business
COTS isn't going to work for all organizations. Take your time to research and test before downloading, buying the license, or deploying it into your network
How to protect your organization before using commercial-off-the-shelf software:
Identify the components of the commercial software
Investigate how COTS will impact your organization's user privacy, business processes, classified data, points of entry
Test how COTS will perform with your own organization's infrastructure
Secure your own infrastructure before introducing commercial software
Check search engines, reviews, forums, and other communities who have had success or concerns with the same software
Seek out certification or assurance for the software
Update the software immediately after installation
Continuously monitor, update, audit, and review the operation of COTS
Think "when" it will fail, and not "if" it will fail
CISSP Takeaway Concepts
Domain 8: Software Development Security
COTS - Three types of commercial-off-the-shelf software was used.
Domain 7: Security Operations
We were attempting to migrate a legacy mail server to a new platform with more features
Click here to read other Stories of a CISSP.