What Programmers and System Folks Should Never Do

If you go back and read my entry for 28 May 2011 entitled A Security Hole Big Enough to Drive a School Bus Through you will see where I had made a serious mistake. I made major changes to a system just before going to another contracted computer site for a few days. I returned to face some very angry people.

I should have known better. At Energy Mines and Resources (now, Natural Resources Canada), we had a system called a SCAR. That was: System Change Authorization Request. Before making any changes to a system you were supposed to complete one of these forms and circulate it amongst all the department heads for their approval before you went ahead. The idea was that you were supposed to lay out, step by step, what you were planning to do; what benefit would be derived from the change; and, how to back out when things went wrong. Because this was a government site, we had lots of time to implement changes without directly impacting the user community seeing as they never worked after 5:00 pm, or before 7:00 am, nor on weekends and holidays. Generally, I'd plan my changes for Friday night or Saturday to give me the maximum amount of time to repair whatever I had messed up.

At the time of this story, I was managing two junior system managers. One of them was hopeless and I was trying to get rid of him--a very difficult job in the civil service (fortunately he joined the Department of Defense, so he became someone else's problem); the other had a lot of potential and was a very hard-working and dedicated young man. He did quite well later in his career. But, one Monday morning, just after my protege had left on a two-week vacation, my phone rang. (It was almost always something bad when my phone rang.)

A client complained that the system told him his account number was being rejected by the system and so he could not rent a scratch tape. The renting of scratch tapes used to be a complex paper operation that I had automated. All the user had to do was replace his "mount" command with my "scratch" command and the system took care of everything. He could use all the same parameters and qualifiers that the "mount" command could accept, along with some specific to renting tapes. The routine that got the client's account number was one that I had written a year or so before and was used in a couple of applications. It made things a lot more secure because now the user did not have to type his account number in his program or job control file. The system took care of it, hidden from prying eyes. The routine had always worked. The user was not new; his account had been fine since the beginning and heavily used.

It never occurred to me that the problem might be rooted in my program because it has always worked; I had not made any changes to it; and, when I tested it using my account, there was no problem. The user gave me his account number to try--and sure enough the system replied that it was invalid. Damn! Where was this coming from? I looked at the details of the user's account record, but could see nothing unusual. I was at a loss and told the user I would call him back later.

The question I would ask myself in similar situations was: What is different? What the "what" is is another good question. I printed out my account details and laid them beside a print-out of the user's account details; I compared them entry by entry. Nothing odd there. I tried the same commands on different machines: same results no matter what machine I used. The only thing left was my routine. So, I went through it line by line. Nothing unusual in it. So, I turned on my debugger and ran it. The program balked at one line when I ran it with the user's account number, but went sailing along smoothly with my account number. The line in my code where the system hiccuped was the line where the account number was verified.

So, why was it rejecting the client's account number and not mine? Both were five digit numerals. One of the verifications that the number was valid was to turn it into an integer (it was a string when retrieved). A string is a series of alpha-numeric letters, each represented as a character code by the system. An integer, on the other hand, was a working number that could be used in mathematical calculations. Its internal storage was completely different than its equivalent representation as a string. It was a commonly-used way of determining if there were illegal characters in a numeric. (In fact, I found that very useful during the Y2K crisis a few years later.)

I looked at it, my eyes going back and forth between the two numbers. Mine, being a "system" number (which meant I got any computer services I wishes for free) was a low number. The users was a fairly high number. For example (I don't recall the precise numbers), my account number might have been: 11002 while his might have been 51010. What's the difference, right? I verified that the user's number did not contain the letter "o" in place of the zeros. Nope.

I then it went back to my program and this time looked at the declarations at the beginning of the code. (This is where the programmer defines the variables he will use in the following code and defines what type of variables they are so that the system can set aside memory storage for them in the right format for the type of variable. As discussed above, the way a system stores a "string" is much different than the way it stores an "integer.")

And, then I looked again. Though I did not recall doing it, the variable where the integer form of the account number was to be stored during the run was indicated as type called "signed word." Odd. I was sure I had not declared it as such. A "signed word" was a form of number stored in 16 bits with one bit being used to indicate whether the number was positive or negative (that's the "signed" part). The range of numbers that that encompasses is − 32,768 to 32,767. Five digits, just like our account numbers. But our account numbers were never negative. Huh? I couldn't believe I had declared as such; I was usually very precise in my declarations. It explained why the user's account number was invalid while mine was okay. His "number" was higher than 32767 while mine was lower. I would not have used even an unsigned word because its range is only 0 to 65,535. It is possible that we could have an account number higher than 65535.

Shaking my head in disbelieve at my stupidity (while wondering how it had worked for the past year), I checked the date of the last compile. It had been the Friday before. There had been no reason to compile it then; as I said it had worked fine for more than a year. So, I compared the last two versions of the program: voila! what had originally been declared as an unsigned longword (32 bits, with a range of 0 to 4,294,967,295) had been changed to the signed word on Friday. There was only one way that could have happened. My protege and I were the only ones with access to the source code and I knew I would never have made such a change.

I corrected the declaration, recompiled, and tested. The user's number was now fine, as I knew it would be. It was the middle of the afternoon by this point and the user had been waiting since mid-morning to be able to rent a tape.

So, when my young friend returned from his vacation two weeks later I told him what had happened. "Yes," he admitted, he had changed it. Why? Because I was wasting memory with my declaration as a longword; we'd never have an account number with more than 5 digits and a longword could handle up to 10 digits. Where's the SCAR? I asked. He admitted he hadn't prepared one because it was such a trivial change. Do you know the range of a word? I asked. He admitted he did not, but he knew that it wouldn't waste as much memory as a longword. I told him that if he had prepared a SCAR like he was supposed to I would have rejected it--and then I told him why.

My friend learned a valuable lesson. As far as I know he never changed anything in a system without doing research first and then following whatever change control procedures were in place.

Problem Solving With Other People (shudder)

I found, when working on a program or a computer problem, that the best way for me to approach it was to work alone. Other people distract me from the line of reasoning I'm following; sometimes we'd get side-tracked on irrelevant arguments while I waited impatiently to get back to working on the problem. In fact, sometimes when working with others, especially those with limited knowledge, I'd get so thrown off track that I would have to walk away so I could think about it later.

One of my managers was so impressed with my ability to get to the root of any problem that he told me he wanted to lock me in a room and just throw me some raw meat every now and then. I liked the imagery. I once told an interviewer that I worked best when in a room by myself with two slots in the door: one for problems in, the other for solutions out. I got the position.

The last few years that I worked in high tech were very frustrating and disappointing for me. I had gotten used to being the guy who everyone else looked to for solutions. Even when I didn't know the answer, I could usually point them in the right direction. However, I was on a contract where I was working in a team and often was called upon to solve a hardware problem at some remote location at 3:00 am with a hysterical operator giving me only part of the story, and a frustrated tech at the other end telling me something else altogether. I hated it. The work environment, other than the middle of the night pages, was great and the pay was very good. But I was bored out of my mind and disappointed that I sometimes could not get a handle on a problem when in situations like above.

The fact that there were unstated rules made it even more difficult for me. One time I had spent the entire night walking a tech through the rebuilding of a system from backups. The family had tickets to a ballgame the next day. I slept from about 6:00 am when the restore finally finished to about 10:00 when I got up to grab a cup of coffee and head out the door for the long drive to the ball park.

The entire family was in the car and I had just started the engine when my pager went off. Stomp back into the house and call the operator. He starts babbling something that, after being up all night, I had to keep asking him to slow down and repeat what he had just said. He eventually realized that I was in no condition to help him and asked me if he should call the backup on-call person. Go ahead, I told him, relieved. I slept through most of the game in my seat.

Monday my backup was furious with me. Apparently the operator was not supposed to call the backup, only the prime on-call was allowed to do that. She was so upset that she didn't speak with me for about 18 months--which made things awkward as we were both members of a small team and our desks were next to each other.

Early in my career I was managing a system for an office group. The machines were running All-in-One, a document-tracking application from Digital Equipment. I had just started the contract and was still getting the hang off All-in-One when a DEC support person upgraded the application. My phone started ringing. Everyone's old emails had disappeared. The package provided a simple interface for email; it kept track of where emails were in a small file in each individual's work-space. Knowing that much, at least, I checked, and, sure enough, when the expert applied the upgrade he had allowed a new empty copy of the tracking file to replace the working one in each person's account.

So, I figured the simple approach would be to restore the original files, which, fortunately, were still on disk--and so I did in short order. Then the phone started ringing again: everyone's old emails were back, but any new ones they had created earlier that day had disappeared. Of course, between the time that the techie had installed the upgrade and when I had restored the files, all the messages were being tracked by the new file. You know, this situation should not have happened at all. Application upgrades are supposed to be planned with all the steps laid out clearly and with a backout plan in place in case something went wrong. Also, you never did an upgrade on a live system while people were working. You did it off-hours with users locked out of the system. However, experts usually considered themselves above following basic routine procedures. And now we were stuck. My "solution" had actually been as poorly implemented as the original upgrade had been.

Now my manager was in a panic. She didn't know very much about computers and relied on me for all the technical stuff. I was trying to think of a way to get both files, the old one and the new one, back together again so that the users could access all their email. However, she kept insisting that I had to do a restore from backup tape. "Restore from what?" I asked. She had an idea in her head that if we restored the backup from the previous night somehow everything would get fixed. I tried to tell her how absurd that was: the backup had been run before today's emails had been created. I had already restored the old files from disk and there was no point in doing it again from tape. She hated it when I tried to deflect her from her set path.

She was the boss. So, I spent the next several hours spinning tapes. Of course it didn't work because that day's emails were not on any tape. She was insistent, telling me to backtrack to the last complete system backup and then restoring all the days in between one at a time. Nothing I said would penetrate her thick skull that files that had never been backed up to tape would not suddenly appear if you used tapes from different dates. It took hours to do each restore (systems and tape drives were not very fast in those days).

The clock crept around to 2:00 am. I told my boss I was too tired to spin any more tapes and needed a break. So, she braved the computer room to pursue her pointless exercise. Finally! I had a chance to look closely at the system and the files. I determined the structure of the tracking files and then looked up some file-management documentation. Bingo! I could merge the two files together using simple data processing commands. So, I did it. Voila!

I went to the computer room to tell my boss I had fixed the problem and we could go home. She was thunderstruck and asked me to show her what I did. I typed the command and the two file names, and, sure enough, all the user's emails were now available. If she had left me alone from the beginning I would have gotten there about 12 hours earlier and saved a lot of overtime.

She was really ticked off with me because by solving something so simple I had shown her how useless her approach was. When it came time for my annual review, she crucified me, citing how my arrogance had caused system down time (how I got blamed for all the downtime is beyond me). She quoted some similar incidents where problems had proven to be beyond her abilities to solve while I casually fixed them. That demonstrated all sorts of nasty things about me.

I never once, in my entire career, had received a negative annual review. In fact, our boss had her re-write that one and she left the department soon afterwards. (I took over her position and I was a lot more reasonable with those reporting to me than she had ever been. In fact, the boss had to ask me to tone down my reviews of those reporting to me; they were too positive.)

NAK-NAK, Who's There?

In the late 1980's I was working in the computer department of the Civic Hospital in Ottawa as they were rolling out a new hospital patient-management package. The major problems with the new package was: it was American, necessitating the education of American programmers and re-writing large sections of code; and, it was obsolete even before the hospital bought it. I've mentioned before that the source code that ran most of our banks, governments, and similar large institutions was written in the early to mid-1070's using the tools available at the time. That meant: COBOL (COmmon Business- Oriented Language).

COBOL was originally designed in 1959 and underwent major redesigns in 1960, 1974, and 1985. One of its main draw-backs, especially from today's vantage point, was that it was overly-verbose and lacked the ability to perform some common programming tasks such as recursion, handle memory allocation issues (eg., "local" variables versus "global" variables, as well as dynamic assignment of variables), and, in general, was not designed for structured programming. In other words, it, much like BASIC, another common language of the time, encouraged "spaghetti code." The term spaghetti code referred to programs that did not adhere to consistent construction and architecture. In other words, the logic of the flow of the code was missing, often resulting in a haphazard mess very difficult to maintain and debug. I ran across quite a few samples of this kind of writing in my early consulting years which resulted in my becoming somewhat anal about programming standards in my own work, and in the work of any employees that I supervised.

In any case, the new code that was being rolled out at the Civic Hospital was a disaster from the beginning and a number of senior information technology heads rolled once the extent of the disaster was clear. My observation was that the entire process was flawed. I mentioned before that decisions about computing were rarely based on logic and analysis. In this case, they bought this particular package over a Canadian-designed one because it was backed by a large American corporation (McDonnell-Douglas, to be exact). At least, that was the only reason I was ever given.

I think part of the problem was that when computers moved into the business world in a big way during the 1970's there simply were not enough people experienced in the field, especially at the management level. Prior to that computing was the reserve of scientists. I quickly learned during my first ten years or so in the field that I much preferred working with scientists over office management, because scientists, as a rule, no matter what their field, understood that computers were no more than tools and were as dependent on the skill of the user as any other tool. Consider the case of a hammer being used by a skilled construction worker versus a novice who has rarely touched one. The novices, in this case, were the office workers whose expectations of computing far exceeded the ability of the machines at that time to deliver. Emotional relationships with the machines became inevitable in the magical-thinking world of the technologically illiterate. Machines became stupid and willfully uncooperative when they failed to meet the expectations of the users. I still see those attitudes in the work-place today despite our 40-plus years experience at this point.

(A prime example of this from within the last two years: a fellow employee was trying to enter data in the wrong field on her screen. Instead of stopping to think about why the machine was rejecting her input, she kept typing the data over and over again, growing more and more frustrated. She got so emotionally wound up that I had to tell her, several times, more firmly and loudly each time, to stop what she was doing before I could point her to the correct area to enter her data.)

I don't recall all the highly embarrassing events that unfolded when the Civic's new program was rolled out into production mode. One that stood out vividly was that when visitors to the hospital inquired at an information desk about the whereabouts of their loved one, if the patient had been moved from one room to another they were reported to be deceased. Very fertile ground for serious lawsuits, indeed. But I was involved in the technical side and so that's where I saw most of the program's shortcomings.

To give a detailed example. Hospitals distribute plastic identity cards to their patients. These cards are almost invariably embossed (raised lettering) so that they can be run through simple carbon-pressure machines to take an image of them. This helps enormously in tracking and identifying patients. The machines that produce these cards are called embossers and they are usually placed strategically throughout a hospital at the various points where patients enter the system. An admitting clerk types the information into a computer screen and the program directs the required information to an embosser to produce the card. This is all very much like sending information to a printer to produce a form or a letter on paper.

There are standard ways to control the flow of information to a device like a printer, or, in this case, an embosser. The computer needs to know when the device is ready to start receiving data and when the computer should wait before sending more--or if the computer should resend the data because the receiving device missed or otherwise detected an error in the data already sent. At the heart of the control matter are two characters called ACK and NAK. When a device has received seemingly error-free data it returns an ASCII character called an ACK (short for acknowledge), indicating it is ready to receive more; the code for this character is 06. If the device needs data to be resent, it returns a NAK (short for not acknowledged), ASCII character code 15(Hex), or 21(Decimal). This all is a process called flow control. Think of an airplane pilot saying "Roger that" or "Say again" to the control tower. If the pilot instead said, "Okay," or "What?" chances are pretty good that he would be misunderstood. That's why we use code words, or code characters.

As soon as the new program went live and patients started being admitted to the hospital all the embossers failed. They produced garbled output if they produced anything at all. Desperate phone calls with the American support team finally lead to the person who developed the code. How did you manage flow control? he was asked--a fairly astute question considering the panic. Oh, I used ACK and NAK, he replied. So, if flow control is not the problem, what else could it be?

And then the network boys tapped into a line to trace what was actually going to and from the embossers. They did not see ASCII codes 06 and 15. What they did see were strings of 41-43-4B and 4E-41-4B. And what do those codes translate to? You probably guessed it: the alphabet characters A-C-K and N-A-K. And that was the sort of incompetence that lay at the heart of the shinning new management package.

The pressure, frustration, disappointment, and desperation throughout the computer department was starting to affect my health and, when an opportunity elsewhere presented itself, I jumped at it--just before the big axe fell on those responsible.