If you go back and read my entry for 28 May 2011 entitled A Security Hole Big Enough to Drive a School Bus Through you will see where I had made a serious mistake. I made major changes to a system just before going to another contracted computer site for a few days. I returned to face some very angry people.
I should have known better. At Energy Mines and Resources (now, Natural Resources Canada), we had a system called a SCAR. That was: System Change Authorization Request. Before making any changes to a system you were supposed to complete one of these forms and circulate it amongst all the department heads for their approval before you went ahead. The idea was that you were supposed to lay out, step by step, what you were planning to do; what benefit would be derived from the change; and, how to back out when things went wrong. Because this was a government site, we had lots of time to implement changes without directly impacting the user community seeing as they never worked after 5:00 pm, or before 7:00 am, nor on weekends and holidays. Generally, I'd plan my changes for Friday night or Saturday to give me the maximum amount of time to repair whatever I had messed up.
At the time of this story, I was managing two junior system managers. One of them was hopeless and I was trying to get rid of him--a very difficult job in the civil service (fortunately he joined the Department of Defense, so he became someone else's problem); the other had a lot of potential and was a very hard-working and dedicated young man. He did quite well later in his career. But, one Monday morning, just after my protege had left on a two-week vacation, my phone rang. (It was almost always something bad when my phone rang.)
A client complained that the system told him his account number was being rejected by the system and so he could not rent a scratch tape. The renting of scratch tapes used to be a complex paper operation that I had automated. All the user had to do was replace his "mount" command with my "scratch" command and the system took care of everything. He could use all the same parameters and qualifiers that the "mount" command could accept, along with some specific to renting tapes. The routine that got the client's account number was one that I had written a year or so before and was used in a couple of applications. It made things a lot more secure because now the user did not have to type his account number in his program or job control file. The system took care of it, hidden from prying eyes. The routine had always worked. The user was not new; his account had been fine since the beginning and heavily used.
It never occurred to me that the problem might be rooted in my program because it has always worked; I had not made any changes to it; and, when I tested it using my account, there was no problem. The user gave me his account number to try--and sure enough the system replied that it was invalid. Damn! Where was this coming from? I looked at the details of the user's account record, but could see nothing unusual. I was at a loss and told the user I would call him back later.
The question I would ask myself in similar situations was: What is different? What the "what" is is another good question. I printed out my account details and laid them beside a print-out of the user's account details; I compared them entry by entry. Nothing odd there. I tried the same commands on different machines: same results no matter what machine I used. The only thing left was my routine. So, I went through it line by line. Nothing unusual in it. So, I turned on my debugger and ran it. The program balked at one line when I ran it with the user's account number, but went sailing along smoothly with my account number. The line in my code where the system hiccuped was the line where the account number was verified.
So, why was it rejecting the client's account number and not mine? Both were five digit numerals. One of the verifications that the number was valid was to turn it into an integer (it was a string when retrieved). A string is a series of alpha-numeric letters, each represented as a character code by the system. An integer, on the other hand, was a working number that could be used in mathematical calculations. Its internal storage was completely different than its equivalent representation as a string. It was a commonly-used way of determining if there were illegal characters in a numeric. (In fact, I found that very useful during the Y2K crisis a few years later.)
I looked at it, my eyes going back and forth between the two numbers. Mine, being a "system" number (which meant I got any computer services I wishes for free) was a low number. The users was a fairly high number. For example (I don't recall the precise numbers), my account number might have been: 11002 while his might have been 51010. What's the difference, right? I verified that the user's number did not contain the letter "o" in place of the zeros. Nope.
I then it went back to my program and this time looked at the declarations at the beginning of the code. (This is where the programmer defines the variables he will use in the following code and defines what type of variables they are so that the system can set aside memory storage for them in the right format for the type of variable. As discussed above, the way a system stores a "string" is much different than the way it stores an "integer.")
And, then I looked again. Though I did not recall doing it, the variable where the integer form of the account number was to be stored during the run was indicated as type called "signed word." Odd. I was sure I had not declared it as such. A "signed word" was a form of number stored in 16 bits with one bit being used to indicate whether the number was positive or negative (that's the "signed" part). The range of numbers that that encompasses is − 32,768 to 32,767. Five digits, just like our account numbers. But our account numbers were never negative. Huh? I couldn't believe I had declared as such; I was usually very precise in my declarations. It explained why the user's account number was invalid while mine was okay. His "number" was higher than 32767 while mine was lower. I would not have used even an unsigned word because its range is only 0 to 65,535. It is possible that we could have an account number higher than 65535.
Shaking my head in disbelieve at my stupidity (while wondering how it had worked for the past year), I checked the date of the last compile. It had been the Friday before. There had been no reason to compile it then; as I said it had worked fine for more than a year. So, I compared the last two versions of the program: voila! what had originally been declared as an unsigned longword (32 bits, with a range of 0 to 4,294,967,295) had been changed to the signed word on Friday. There was only one way that could have happened. My protege and I were the only ones with access to the source code and I knew I would never have made such a change.
I corrected the declaration, recompiled, and tested. The user's number was now fine, as I knew it would be. It was the middle of the afternoon by this point and the user had been waiting since mid-morning to be able to rent a tape.
So, when my young friend returned from his vacation two weeks later I told him what had happened. "Yes," he admitted, he had changed it. Why? Because I was wasting memory with my declaration as a longword; we'd never have an account number with more than 5 digits and a longword could handle up to 10 digits. Where's the SCAR? I asked. He admitted he hadn't prepared one because it was such a trivial change. Do you know the range of a word? I asked. He admitted he did not, but he knew that it wouldn't waste as much memory as a longword. I told him that if he had prepared a SCAR like he was supposed to I would have rejected it--and then I told him why.
My friend learned a valuable lesson. As far as I know he never changed anything in a system without doing research first and then following whatever change control procedures were in place.
No comments:
Post a Comment