I Screw Up

As I tell these stories, I'm somewhat concerned you might think that I'm arrogant and I'm sneering at others. First: yes, I do have a problem walking the line between self-confidence and arrogance. When things come easily to one that others struggle with it's hard not to feel somewhat...elevated. Secondly, I came across a lot of good, hard-working, and intelligent people during my career, both inside and outside of the public service. It's just that the screw-ups make more interesting stories. So, in an attempt to balance things out, here's a story where I screwed up.

I told the story earlier of how I helped land a multi-million dollar contract  at SSC (Supply and Services Canada, now PWGSC). I had, at that time, been establishing myself as an expert in VMS performance and capacity planning. I was invited to and presented talks at three national conferences on the subject (in Calgary, Edmonton, and Toronto). So, I was getting pretty sure of myself. But this story doesn't involve detailed technical details: I think anyone can understand that spreading the work-load evenly over several devices is a more efficient way to operate. This is true for computers as well. The slowest part of computer performance is communication to and from disk drives. To give you an idea: to find something that is present in RAM (memory) is equivalent to a carpenter picking up his hammer and driving a nail. To find something that is on a disk drive is like being that same carpenter, but, when he reaches for the hammer, it isn't there. He goes home to get it. On the way he stops at a bar, gets arrested for drunkenness, spends the night in jail, gets home the next day to retrieve the hammer and goes back to the work-site to drive the nail. And, if a computer has to go to disk again, it's as if the poor carpenter had to go through that same routine for every nail that was to be hammered. As you can surmise, it is going to take that carpenter a long time to finish building a house. That should give you an idea of the relative speeds when we are talking about billions of operations a second.

Now, when I got to SSC one of the first things I discovered was that the policy was to keep adding to a disk until it was full, then starting in an another. Empty disks were left spinning uselessly waiting for the day when other disks filled up. You don't need an advanced degree in computer science to figure out that this is not only wasting resources, but is directing all the traffic along one or two paths. I set about reorganizing disk usage so that all the drives were used and with the load spread fairly evenly amongst them. I didn't make a secret of what I was doing--I kept the head of operations informed every step of the way and, as it was his job to make sure that the disk drives were backed up to tape every night I assumed he was adjusting the backup schedules accordingly.

The main part of assume is ass, but you already know that.

Disk drives failed fairly regularly in the 1980's. Swapping out a failing drive and replacing it with a new one was a common procedure carried out by a technician from the manufacturing company (in this case, DEC) with an operator to assist. The operator's part could be carried out by any junior operator; he had to make sure that the data was backed up before giving the technician the go-ahead, restore the data to the new disk, and run a few basic tests to make sure everything was okay. Inevitably one of our drives started to fail and the call went out to have it replaced. As I left that evening the operator who would be staying with the technician (such work was always done outside of regular working hours, unless it was an extreme emergency), asked me what was up. I told him that the tech was going to replace the HDA and told him which specific disk.

The next morning I arrived at the operations center to find a very glum group staring helpless at one of the VAXes. There was a handful of operators, the head operator, and the client project manager. They told me that the minister's email had gone missing. In the government, department ministers are political appointees to positions roughly equivalent to that of the president of a large corporation. They are powerful figures and everyone treads lightly around them. You don't mess with them. Missing email was a serous security problem as well as a prelude to several people losing their jobs.

A quick look at the minister's account told me that her files were on the disk that had been replaced the night before. So, get the backup tape and let's go. There was no backup tape. The operator did not realize first: that the disk was in active production; and secondly: that when I said HDA, I was using jargon for the "Head-Disk-Assembly"--in other words, the whole darn disk, except for the casing. Okay, let's get the regular backup tape from the previous night. There was no backup tape from then either. The drive had not been included as part of the regular backup schedule. The head operator assumed that I had been changing the backup schedules while I was moving files around; I assumed that he had been adjusting the backup schedule because it was his responsibility and I had kept him informed of what I was doing. The operator had not only not realized what the HDA was, but had also assumed that the drive was not in production and so no backup-restore operation was necessary.

With all these asses around you'd think there was blame to share, but, not really. When I took on the job of reorganizing the systems it was up to me to make sure that I didn't cause any harm. Despite all the apologies I heard, I knew it was my screw-up and no one else's.

The only option we had left was to call the technician who had replaced the drive and hope that he had not taken the old one to scrap processing yet. Whew! he had not. The drive was still in the trunk of his car. But, it had been very much below freezing overnight, so we had no idea if the disk was now even readable after being bounced around in the trunk and frozen. He brought it in--I should mention that in those days disk drives weighed abut 200 pounds and were about the size of a large file cabinet drawer. He swapped the drives while we all held our breaths. I mounted it, checked it out, and everything was still there and intact. We backed it up, swapped it out again, and restored the files to the new disk. Did I mention that all through this the phone was ringing every few minutes with people from the minister's office wanting to know what was going on and why couldn't the minister read her email? We simply could not tell the real story. Everyone of us would have been instantly escorted out of the building and told never to return. And, if the story got to the press we would be instantly fired again. We all knew the consequences which is why we all agreed that it was all just a temporary technical fluke that we had heroically managed to correct.