Back on 1984 there were very few tools for managing systems. As a result, most of us in the field wrote our own. We needed usage reports, backup procedures, monitoring, and clean-up tools. One thing that everyone needed then was an inactive terminal killer program. Resources were very limited. If a system had only 15 ports and 20 people needed access (a fairly common situation), then it was unfair of someone to be connected to a port while they took a coffee break. At the same time, security considerations made an untended terminal a worrisome feature.
When I required such a tool I went looking to see what others had written. I was disappointed by all that I looked at. Invariably they were written in DCL, an interpreted script language. What this means is that every time the system reads a statement it has to pause while it translates it into machine language. This is not a very desirable way to write an application, especially one that will be run frequently on a system. It is slow, cumbersome, and easy to break into. One such program, called WatchDog, appeared to be popular. However, the fact that it was written in DCL and not in a high-level language placed it on my reject list. Also, the way it went about determining if a terminal was inactive or not was to measure about a half-dozen different metrics, slowing it even more.
On thinking it over, I decided that such a program had to be written in Assembler. Assembler is as close to the language that machines can understand directly as a human can get. It is then compiled (translated to machine language) when developed and, so the resulting file is about as fast as one can get. In fact, it is several orders of magnitude faster to execute than anything written in a script language. Secondly, I did not want the program to waste time by measuring anything that was not needed to make the determination of whether a terminal was idle or not. Seeing as, no matter what it is doing, a process has to pass through the CPU in order to actually do it, it made sense to me that all I needed to measure was accumulated CPU time.
I set up a number of tests to actually measure how much time was spent in the CPU when a process was idle and compared that to how much time was required to process a carriage return. I came up with a number of .02 seconds as a reasonable threshold, and then doubled it to make sure. So, as long as a terminal consumed less than .04 seconds of CPU time within five minutes, it was idle. The program was intended to cycle once every five minutes and issue a warning at the 25 minute mark and, again, right before the process was terminated. I wrote the code, keeping it as simple and compact as I could.
It worked like a charm. It never logged anyone out while they were actually working and when that half-hour was up, that was it. No exceptions. The program itself required about .2 seconds of CPU time to execute on a system with 200 terminals. WatchDog, in contrast, required almost a full minute to execute on a similarly-sized system, a requirement that was far too expensive in terms of system resources. You want your monitoring and management programs to be as unobtrusive as possible.
Every year system managers contributed the tools they had developed to a users' community that put the programs on tape to be distributed to anyone interested. I was certain that my program would generate interest because it was so much more efficient than WatchDog and its relatives. What I hadn't counted on was that though system managers, as a rule, could read and program in DCL (most of them badly), very few could read and work with Assembler. My program might have been hundreds of times faster than anything in DCL, but, if they couldn't tinker with it, they weren't interested.
WatchDog hung around for years. It became even more complex, and slower, with each iterative. And, then, someone owned it. Now if you wanted it, you had to pay for it. The company that claimed ownership developed a suite of tools to assist in managing systems. Most of them were hopelessly inefficient at first but, given a choice between something that was free and something you had to pay for, most high tech managers opted to pay. I can't swear that WatchDog was Computer Associates' first successful product, but, it was there at the beginning of that corporation's existence. Computer Associates now makes a huge range of highly sophisticated (and much more efficient) products for IT management. Their revenue was $4.4 billion in fiscal year 2011.
Could I have parlayed my program into a similar-sized business? Could I have sold it for a nice profit? I'll never know. Maybe that did happen in a parallel universe, but, in this universe, better doesn't always mean success.
Friday, 24 June 2011
Sunday, 19 June 2011
Who's Minding the Store?
I had a contract at a section of the Secretary of State's department to supply system management services. This would have been about 1991 or 92. By this time I had a 9-track tape filled with all the system management tools I had created over the years, so it didn't take long for me to get set up. A key feature of my tools was that the system manager didn't have to do anything once everything was up and running. The system monitored itself. I was there for emergencies and software upgrades.
I had a great corner office overlooking Eddy Street in downtown Hull. Across the street was a store that sold "adult" toys and accessories. I could see all the comings and goings. Though I am sure that many people think such businesses attract perverts and other Riff-Raff, most of the clients I saw were well-dressed young women. Twenty minutes or so after entering the store they'd emerge with a small or a moderately-sized package. An interesting way to spend my time.
This section had two operators on staff. They had their own office on a different floor. In theory, they spent their days monitoring the system. In reality there was nothing for them to do, especially after I had set up the automated system management tools. Well, one of them would have to mount a tape before leaving for the evening and remove, label, and store it next morning after the backup had run. They played computer games all day. Not that I could blame them. They weren't responsible for creating their job descriptions. At least they weren't hurting anything.
One day as I dozed in front of my PC, my inner hacker alert went off. I saw a process running that had full system privileges (these are the keys to unlock various off-limits parts of the operating system, including things like creating and changing accounts, accessing all files, and destroying the system.) I immediately sent a message to the suspected hacker: You have 30 seconds to call this number: xxx-xxx-xxxx or I will blow away your process. My phone rang almost immediately.
Who are you? I asked.
Your system manager, was the reply.
I'm the system manager. Who do you work for?
The intruder explained that he worked for a large well-known company that had a contract to monitor and manage the system. He told me that all he did was check in every few months to see how things were going.
Off to the client I go to ask for an explanation. My liaison knew nothing about this other company with a system management contract. He had to get back to me. Apparently he asked around and discovered that they did, indeed, have such a contract.
So, now we have 3 1/2 people doing a job that could really be done by someone who spent ten minutes or so on the system every week. I proposed to my boss, back at my company, that we make a proposal to do just that. The two game-playing civil servants would be transferred elsewhere in the hopes that they would find something useful to do, the second contact would be cancelled, and we'd have a small contract for system management on an as-needed basis. That probably would save the government $300,000 or more a year. My idea was nixed because my company didn't want to risk the $150,000/year the client was paying for me (out of which I made about $70,000) in order to replace it with a contract for $50,000 or less per year.
Sigh! Big sigh!
Multiply that $300,000 by all the tiny sections of huge departments where people are so bored with their jobs they read novels all day or cruise the Internet and I'm sure the government deficit would disappear.
Oh, sorry, what a ridiculous idea. The government would be forced to find something else to waste money on. Maybe they could build more prisons even though the crime-rate has been steadily falling for decades or buy some expensive military hardware designed to operate on Mars. After all, that appears to be one of the government's main duties.
(PS. Yes, I know there are many dedicated hard-working people in the civil service. I met and worked with many of them.)
I had a great corner office overlooking Eddy Street in downtown Hull. Across the street was a store that sold "adult" toys and accessories. I could see all the comings and goings. Though I am sure that many people think such businesses attract perverts and other Riff-Raff, most of the clients I saw were well-dressed young women. Twenty minutes or so after entering the store they'd emerge with a small or a moderately-sized package. An interesting way to spend my time.
This section had two operators on staff. They had their own office on a different floor. In theory, they spent their days monitoring the system. In reality there was nothing for them to do, especially after I had set up the automated system management tools. Well, one of them would have to mount a tape before leaving for the evening and remove, label, and store it next morning after the backup had run. They played computer games all day. Not that I could blame them. They weren't responsible for creating their job descriptions. At least they weren't hurting anything.
One day as I dozed in front of my PC, my inner hacker alert went off. I saw a process running that had full system privileges (these are the keys to unlock various off-limits parts of the operating system, including things like creating and changing accounts, accessing all files, and destroying the system.) I immediately sent a message to the suspected hacker: You have 30 seconds to call this number: xxx-xxx-xxxx or I will blow away your process. My phone rang almost immediately.
Who are you? I asked.
Your system manager, was the reply.
I'm the system manager. Who do you work for?
The intruder explained that he worked for a large well-known company that had a contract to monitor and manage the system. He told me that all he did was check in every few months to see how things were going.
Off to the client I go to ask for an explanation. My liaison knew nothing about this other company with a system management contract. He had to get back to me. Apparently he asked around and discovered that they did, indeed, have such a contract.
So, now we have 3 1/2 people doing a job that could really be done by someone who spent ten minutes or so on the system every week. I proposed to my boss, back at my company, that we make a proposal to do just that. The two game-playing civil servants would be transferred elsewhere in the hopes that they would find something useful to do, the second contact would be cancelled, and we'd have a small contract for system management on an as-needed basis. That probably would save the government $300,000 or more a year. My idea was nixed because my company didn't want to risk the $150,000/year the client was paying for me (out of which I made about $70,000) in order to replace it with a contract for $50,000 or less per year.
Sigh! Big sigh!
Multiply that $300,000 by all the tiny sections of huge departments where people are so bored with their jobs they read novels all day or cruise the Internet and I'm sure the government deficit would disappear.
Oh, sorry, what a ridiculous idea. The government would be forced to find something else to waste money on. Maybe they could build more prisons even though the crime-rate has been steadily falling for decades or buy some expensive military hardware designed to operate on Mars. After all, that appears to be one of the government's main duties.
(PS. Yes, I know there are many dedicated hard-working people in the civil service. I met and worked with many of them.)
Saturday, 18 June 2011
High Tech and Literacy
I don't know what it is, but those who work in the high tech field generally have a very poor grasp of written English. Well, when I think about it, so do most people I've run across. But, you'd think (hope?) that those who've gone beyond secondary school would be able to put together a coherent sentence. Alas, that appears to be the exception, not the rule.
Documentation is vital in the computer world. Without detailed instructions to accompany them, many programs would remain underutilized and difficult to master. That goes for all stages of computer development. Programmers need to be able to understand what programmers before them were attempting to do; operators need to know what steps, exactly, they are to take when an event occurs; those supporting the application's clients need to have a clue how it works; and those who are going to use it need to know how to use it.
So, starting at the bottom: most programmers do not put descriptive notes in their programs to explain what sections of code are supposed to do. Many of those who do, write non-helpful comments (and this is a true example) like: This code is fucked up. Thanks guys, I really appreciate that helpful advice. So, right away you have a problem.
Joe writes a function that takes the system time in its internal format and extracts and formats the fields the program requires. Of the 128 bits that many computers use to track time, we want to know only the name of the day of the week, for example. Joe leaves the company and Mary discovers that the program needs only the first three letters of the name of the day of the week, so she goes into Joe's function and tinkers with it. Two years later someone thinks that the time of day would be a good thing for the function to fetch as well, so they add their fragments of code to what was already there. Unless these things are done correctly, the program will break--and how do you know how to approach it if there are no comments in the code to guide you? When Mary shortened the name of the day of the week, (probably by simply truncating it), she had left some bytes unused in the calling program (the ones that displayed the rest of the name originally). When the time of day is added, you are now returning much more complicated information to the calling program--and does that program have enough memory reserved to be able to contain the added information? Not likely. And, the way large programs are constructed, the function could be called from anywhere in the program, sometimes layered so deeply in other functions and subroutines that its existence is unknown until the program crashes and someone manages to step through the source code, one line at a time, to locate the problem area.
That should explain to you why programs sometimes fail. Another reason is that no one anticipated the uses to which the program could be put. When I was working in the '80s and '90s banks, and other large institutions, were relying on code sometimes written 15 or 20 years earlier. Over the years as business requirements changed, bits and pieces were spliced into the original code. At some point the limitations of the old code were going to have a problem digesting some of the new data and data formats. Banking networks crash and the economy is put on hold while techies locate the backup tapes so they can restore the system and program to a time when it did work. That is a normal part of the procedure and recovery time and procedures are (should be) included in all plans to upgrade or otherwise change something in the program.
A lot of wasted time and effort could be avoided if the folks writing the original code made an attempt to make it decipherable to the programmers that follow them. But, that would entail writing actual sentences in English and real programmers never RTFM. (That's Read the frigging manual for you non-techies reading this).
I sincerely hope that by now large institutions have bitten the bullet and spent the millions necessary to replace the ancient COBOL programs that were written during the 1970's, and that, as part of the process, they required all those involved in the process to master enough basic English so that they could include helpful information explaining just what the heck they were trying to achieve. (I wouldn't count on an MBA (Master of Business Administration) to have mastered that art yet. They were generally too busy spouting nonsense and spending their inflated salaries to bother with something like learning the basics.)
Of course, I was on the inside of the inside of the system and so didn't always appreciate the 10,000 foot view that MBA's and Vice Presidents of Computer Operations and Development worked in. From that altitude they didn't want to know about a function that could fail and endanger the entire project; technical details like that were too far below the tips of their noses to be able to see and appreciate. (Remember that poor programmer I told you about who was working on the External Affairs COSICS project? Hundreds of million of dollars were riding on his shoulders, but no one was interested enough in the details of what he was doing or knowledgeable enough to be able to rectify the situation.)
BTW, if you are operating under the illusion that decisions involving potential millions of dollars are based on careful analysis, detailed planning, and the expert application of technology, I am sorry, but I will have to burst your bubble. It just doesn't happen that way.
Documentation is vital in the computer world. Without detailed instructions to accompany them, many programs would remain underutilized and difficult to master. That goes for all stages of computer development. Programmers need to be able to understand what programmers before them were attempting to do; operators need to know what steps, exactly, they are to take when an event occurs; those supporting the application's clients need to have a clue how it works; and those who are going to use it need to know how to use it.
So, starting at the bottom: most programmers do not put descriptive notes in their programs to explain what sections of code are supposed to do. Many of those who do, write non-helpful comments (and this is a true example) like: This code is fucked up. Thanks guys, I really appreciate that helpful advice. So, right away you have a problem.
Joe writes a function that takes the system time in its internal format and extracts and formats the fields the program requires. Of the 128 bits that many computers use to track time, we want to know only the name of the day of the week, for example. Joe leaves the company and Mary discovers that the program needs only the first three letters of the name of the day of the week, so she goes into Joe's function and tinkers with it. Two years later someone thinks that the time of day would be a good thing for the function to fetch as well, so they add their fragments of code to what was already there. Unless these things are done correctly, the program will break--and how do you know how to approach it if there are no comments in the code to guide you? When Mary shortened the name of the day of the week, (probably by simply truncating it), she had left some bytes unused in the calling program (the ones that displayed the rest of the name originally). When the time of day is added, you are now returning much more complicated information to the calling program--and does that program have enough memory reserved to be able to contain the added information? Not likely. And, the way large programs are constructed, the function could be called from anywhere in the program, sometimes layered so deeply in other functions and subroutines that its existence is unknown until the program crashes and someone manages to step through the source code, one line at a time, to locate the problem area.
That should explain to you why programs sometimes fail. Another reason is that no one anticipated the uses to which the program could be put. When I was working in the '80s and '90s banks, and other large institutions, were relying on code sometimes written 15 or 20 years earlier. Over the years as business requirements changed, bits and pieces were spliced into the original code. At some point the limitations of the old code were going to have a problem digesting some of the new data and data formats. Banking networks crash and the economy is put on hold while techies locate the backup tapes so they can restore the system and program to a time when it did work. That is a normal part of the procedure and recovery time and procedures are (should be) included in all plans to upgrade or otherwise change something in the program.
A lot of wasted time and effort could be avoided if the folks writing the original code made an attempt to make it decipherable to the programmers that follow them. But, that would entail writing actual sentences in English and real programmers never RTFM. (That's Read the frigging manual for you non-techies reading this).
I sincerely hope that by now large institutions have bitten the bullet and spent the millions necessary to replace the ancient COBOL programs that were written during the 1970's, and that, as part of the process, they required all those involved in the process to master enough basic English so that they could include helpful information explaining just what the heck they were trying to achieve. (I wouldn't count on an MBA (Master of Business Administration) to have mastered that art yet. They were generally too busy spouting nonsense and spending their inflated salaries to bother with something like learning the basics.)
Of course, I was on the inside of the inside of the system and so didn't always appreciate the 10,000 foot view that MBA's and Vice Presidents of Computer Operations and Development worked in. From that altitude they didn't want to know about a function that could fail and endanger the entire project; technical details like that were too far below the tips of their noses to be able to see and appreciate. (Remember that poor programmer I told you about who was working on the External Affairs COSICS project? Hundreds of million of dollars were riding on his shoulders, but no one was interested enough in the details of what he was doing or knowledgeable enough to be able to rectify the situation.)
BTW, if you are operating under the illusion that decisions involving potential millions of dollars are based on careful analysis, detailed planning, and the expert application of technology, I am sorry, but I will have to burst your bubble. It just doesn't happen that way.
Thursday, 9 June 2011
It's All in How You Look at It
You should have picked up by now that solving problems in the computer world takes a certain kind of mind set. You have to be able to see problems the way that computers do. They are incredibly stupid machines. They can't do anything more than what humans have told them to do. When a computer gets uppity and gives us a hard time, remember, it's nothing but bits of silicon, copper, and plastic dumbly following the instructions that people have given it. At bottom, there is nothing difficult or magical about it. It is straight and simple logic: if this event occurs, then do this; if it doesn't, then do something else and do this until that becomes true. Pull the plug (reboot) and most machines will go back to behaving themselves. (And the first question tech support usually asks is: have you rebooted?)
In my first high tech job after six months of working swing shifts in operations, mounting tapes and clearing printers, I had had enough. A temp position opened up in user support and I jumped at it. Now I got to deal with the people side of computing, not just the machines. It didn't take long to figure out that if people would leave computers alone they would perform much more efficiently. There were two aspects to my job that over-lapped: supporting users and evaluating credit requests.
In those days, people did not own computers: corporations and governments did. It's not many people who had a few million dollars to spare just to add columns of figures and issue invoices or to perform some arcane mathematical calculation. So computer time was sold in a system called time sharing. (Sounds suspiciously like a certain condo ownership arrangement at a holiday resort.) People as individuals could not afford to pay for time on these monolithic machines, but, their employers and departments could.
So, the first thing most users did when their programs failed was call tech support (namely, me) and try to blame a mainframe error for their problem. Despite their experience, most never did figure out that machines don't make mistakes. My first question always was: what have you changed since the last successful run? They always swore they hadn't changed a thing, at which point I would ask to see their code. It would come to me in the form of fan-fold paper print-outs, some of them several inches thick. I would have to step though the code, line by line, until I came across something absurd. I'd telephone the user and tell him to look at line such and such: it didn't make sense. That's when I would get the truth: Well, I didn't think that one little change would cause the whole program to crash. Goren, the TV detective on Law and Order: Criminal Intent, has expressed the belief that all people lie. He must have worked in tech support at some time.
The other aspect of my job was to evaluate credit requests. When users thought that their computer time was wasted because of an operations error (such as mounting the wrong tape or unintentionally aborted a job prematurely), they would fill out a form requesting that the cost of that run be refunded. We never gave them any real money, just credit against future runs. I'd have to decide if a botched job was the fault of the user or of operations staff, and then fill out another form with my findings. That form would then circulate through department heads, each initialing it until it was either approved or disapproved. I had to ensure that the forms kept moving through the system which I did mainly by pestering. I kept two copies of the forms: one of the original request and one of the final approve decision. No one before me in the job had tried to create any sort of filing system other than dumping all copies of the forms into a drawer.
This, I thought, was ridiculous. I would never be able to find an old credit-request form if, for some reason, I needed it. And why were we using so much paper? It didn't make sense to me in a large computer centre that we were using carbon copied forms.
So, I proposed and won approval to computerize the process. I wrote the code. Now that users could apply for credit requests online, I could review their requests, tick an "approved" or "disapproved" box, and write up my findings on the computer form. An email would then be automatically sent to the various department heads to let them know that there was a credit request awaiting their decision. If they didn't tick off their box on the computerized form within 24 hours, they'd get another email--and so on until they finally ticked Yes or No beside their name. Once all the required approvals were complete then accounting would get an email advising them that there was an approved credit request awaiting their action. They too would get automatic reminders until they dealt with it. Whichever way the decision went, the user would be informed by email.
All done by machine with minimal human interaction. Now no credit requests would get mislaid and there was a history of all credit requests permanently available to anyone who needed it. I no longer had to pester department heads to move on the credit requests on their desks. Computers are very good at pestering. And, I got to shred a drawer-full of useless paper.
It's all in how you look at it.
In my first high tech job after six months of working swing shifts in operations, mounting tapes and clearing printers, I had had enough. A temp position opened up in user support and I jumped at it. Now I got to deal with the people side of computing, not just the machines. It didn't take long to figure out that if people would leave computers alone they would perform much more efficiently. There were two aspects to my job that over-lapped: supporting users and evaluating credit requests.
In those days, people did not own computers: corporations and governments did. It's not many people who had a few million dollars to spare just to add columns of figures and issue invoices or to perform some arcane mathematical calculation. So computer time was sold in a system called time sharing. (Sounds suspiciously like a certain condo ownership arrangement at a holiday resort.) People as individuals could not afford to pay for time on these monolithic machines, but, their employers and departments could.
So, the first thing most users did when their programs failed was call tech support (namely, me) and try to blame a mainframe error for their problem. Despite their experience, most never did figure out that machines don't make mistakes. My first question always was: what have you changed since the last successful run? They always swore they hadn't changed a thing, at which point I would ask to see their code. It would come to me in the form of fan-fold paper print-outs, some of them several inches thick. I would have to step though the code, line by line, until I came across something absurd. I'd telephone the user and tell him to look at line such and such: it didn't make sense. That's when I would get the truth: Well, I didn't think that one little change would cause the whole program to crash. Goren, the TV detective on Law and Order: Criminal Intent, has expressed the belief that all people lie. He must have worked in tech support at some time.
The other aspect of my job was to evaluate credit requests. When users thought that their computer time was wasted because of an operations error (such as mounting the wrong tape or unintentionally aborted a job prematurely), they would fill out a form requesting that the cost of that run be refunded. We never gave them any real money, just credit against future runs. I'd have to decide if a botched job was the fault of the user or of operations staff, and then fill out another form with my findings. That form would then circulate through department heads, each initialing it until it was either approved or disapproved. I had to ensure that the forms kept moving through the system which I did mainly by pestering. I kept two copies of the forms: one of the original request and one of the final approve decision. No one before me in the job had tried to create any sort of filing system other than dumping all copies of the forms into a drawer.
This, I thought, was ridiculous. I would never be able to find an old credit-request form if, for some reason, I needed it. And why were we using so much paper? It didn't make sense to me in a large computer centre that we were using carbon copied forms.
So, I proposed and won approval to computerize the process. I wrote the code. Now that users could apply for credit requests online, I could review their requests, tick an "approved" or "disapproved" box, and write up my findings on the computer form. An email would then be automatically sent to the various department heads to let them know that there was a credit request awaiting their decision. If they didn't tick off their box on the computerized form within 24 hours, they'd get another email--and so on until they finally ticked Yes or No beside their name. Once all the required approvals were complete then accounting would get an email advising them that there was an approved credit request awaiting their action. They too would get automatic reminders until they dealt with it. Whichever way the decision went, the user would be informed by email.
All done by machine with minimal human interaction. Now no credit requests would get mislaid and there was a history of all credit requests permanently available to anyone who needed it. I no longer had to pester department heads to move on the credit requests on their desks. Computers are very good at pestering. And, I got to shred a drawer-full of useless paper.
It's all in how you look at it.
Monday, 6 June 2011
I Make Life a Bit Easier for Federal Prisoners
Writing computer code is an art form. Once you become familiar with someone's style you can recognize it the same way that you would recognize the writing style of a favorite author, or the musical style of a composer. Everyone has his little habits and personal quirks. One of my quirks was that I insisted on absolute neatness. My columns would line up precisely; comments would appear in consistent places. And, code was as brief as possible. If I could combine two or three steps in one line of code, I would. If an operation was going to be repeated more than once, it would be written as a subroutine or function--even if the data types were different. I'd handle that in the subroutine.
In other words, my code was as compact and efficient as I could make it--as well as looking good.
Another feature of my code was that it worked. You might think that of course it should work, shouldn't all computer code work? Yes and no. I've seen a lot of code during my time that not only did not work, but had potential nasty side-effects. As much as possible, I ensured that my code would do only what it was supposed to do and nothing else. A manager once gave me an assignment to write a function for a complex application. When we tried it as a test, the program failed. The boss was not surprised. Everyone's code failed on the first run-through. But, in debugging I found the problem: it was not in my code, but the data was not in the format it was supposed to be. In other words, the database manager had screwed up and my code worked perfectly. I'll never forget how amazed they were that my code actually did what it was supposed to on the first run, once you discount the data error. Those sorts of events confused me somewhat. If code does not do what it is supposed to do, then what is its purpose?
As I moved from computer site to computer site I took my programs with me. There wasn't much point in re-writing something I had already done. But, I noticed a curious thing when I returned to a site after having been somewhere else for a month or longer. My code would have someone else's name attached to it. To add insult to injury, there were changes that made the code not as clean and efficient--and would sometimes make the entire program, or significant parts of it, fail. The true professionals who were familiar with my coding style recognized that my code had been tinkered with by someone else. Lines of code would suddenly not be lined up, spelling errors would appear, inconsistencies would pop up, and code was suddenly not as concise and carefully delimited as it could be. Another thing: I simply didn't do some of the idiotic things that they did under my name.
Ironic situations would arise. For example, I once worked at a few sites with another consultant. I had a good reputation and stayed in touch with people I had worked for, sometimes running into them again and again in different circumstances. I was told that this joker would take my code and, as he put it, change it to a standard format. The only changes he ever made were to erase my name and put his in its place. He had the gall to telephone me once to complain about a program I had written some five years before. He admitted that, yes, it had run every day without fail for five years. The obvious question is: What changed? He insisted that nothing had changed and he wanted me to drop everything and rush to where he was working to fix my code. I wasn't buying any of that bulls**t. A few days later one of my friends who was working with this clown called to tell me about what really happened. The slimy bastard had modified my code to make it "more efficient," and, now that it no longer worked, he was trying to blame me. Eventually they restored my original code from backup tapes and the problems went away.
Talking about bad code, when I arrived on contract at Corrections Canada the first assignment I was given was to determine if there were ways to speed up the backup process. A full backup of all of Correction Canada's regional computers was started at midnight every Saturday. The entire process typically took from ten to twelve hours, sometimes longer. The problem was that while the backup was running the computers were inaccessible--and while the computers were inaccessible no visitors were allowed into or out of any federal prison anywhere in the country. Relatives and friends would line up Sunday mornings to visit the government's guests, hoping that the computers would be back online before visiting hours expired.
The code I looked at was a bad joke. The full backup ran Saturday while incremental backups (backup of only the data that had changed since the last full backup) were run Mondays through Thursdays. There was no backup run Fridays. (Don't ask: that wasn't my problem.) So, the first thing such a program needs to know is: what day of the week it is. I read through the code, block after block of it marked off with If today .eq. "Friday" then... skip the following. And nested inside the check for Friday was a block like this: If today .eq. "Saturday" then... execute this code, else execute this other code. To make things worse, the program never did figure out what day of the week it was until it came to the final section of code, and then it would recall itself (run the entire thing again), this time with the day of the week information.
Common sense says that the first thing you do is find out the day of the week. If it is Friday then exit, quit, don't do anything else, just stop right there, we're done. If it is Saturday, then execute the program that contains the code for a full backup. If it is not one of those days, then execute the program that contains the code for incremental backups. And, in the event that they ever do decide to run a backup on Friday, make it simple to change the program so that it would run on Friday--or perform a full backup on a different day of the week. I cannot understand why anyone would want to approach it differently.
Then I addressed the problem of getting those backups done more efficiently, and hence faster. I looked at my resources: I had four high-speed tape drives available. It made sense to keep them all busy simultaneously (unlike the earlier code which accessed them one at a time). The solution to me was to create a queue with a list of all the backups to be run. This queue would release the jobs, one at a time, to whichever tape drive was free. So, we start with four tape drives busy, but, how do we make sure that they will all be busy until the entire process is finished? I doodled a lot with pictures of a tube breaking into four channels and it hit me. The most efficient way to arrange the backup jobs was order of size, processing them from largest to smallest. Think about it. Drive number four should finish the first round of backups first because it is processing the smallest of the four jobs, so it will immediately start processing the next largest backup. And so on. As each drive is freed up, it takes the next largest job. This could mean that one tape drive could spend the entire time on one huge job while the others race through the smaller jobs, but so what: in the end they do roughly the same amount of work in total.
With my code, the entire full backup took two hours to complete. That's down from 10 to 12, or more hours. At 2:00 am Sunday, Canada's prisons, could, in theory, start processing visitors.
You can guess what happened next. Nine months later after I left Corrections Canada to move on, my name disappeared from the programs and they suddenly started failing and having other problems. If only people would get over the idea that they have to fix things that are not broken, remove the name of the original programmer, and then insert their own name.
Have I ever mentioned that there were (and probably still are) a lot of phonies and fakes posing as "computer experts" getting paid large salaries? That sort of thing is easier to get away with in the government than in the private section where you are expected to produce results. In the government accountability is a very rare commodity and it is always easy to blame someone else. I'm proud of the work I did, even if I did make the occasional mistake. I do know that years afterwards some of my code with my name still attached was still running, as reliably as it should be.
In other words, my code was as compact and efficient as I could make it--as well as looking good.
Another feature of my code was that it worked. You might think that of course it should work, shouldn't all computer code work? Yes and no. I've seen a lot of code during my time that not only did not work, but had potential nasty side-effects. As much as possible, I ensured that my code would do only what it was supposed to do and nothing else. A manager once gave me an assignment to write a function for a complex application. When we tried it as a test, the program failed. The boss was not surprised. Everyone's code failed on the first run-through. But, in debugging I found the problem: it was not in my code, but the data was not in the format it was supposed to be. In other words, the database manager had screwed up and my code worked perfectly. I'll never forget how amazed they were that my code actually did what it was supposed to on the first run, once you discount the data error. Those sorts of events confused me somewhat. If code does not do what it is supposed to do, then what is its purpose?
As I moved from computer site to computer site I took my programs with me. There wasn't much point in re-writing something I had already done. But, I noticed a curious thing when I returned to a site after having been somewhere else for a month or longer. My code would have someone else's name attached to it. To add insult to injury, there were changes that made the code not as clean and efficient--and would sometimes make the entire program, or significant parts of it, fail. The true professionals who were familiar with my coding style recognized that my code had been tinkered with by someone else. Lines of code would suddenly not be lined up, spelling errors would appear, inconsistencies would pop up, and code was suddenly not as concise and carefully delimited as it could be. Another thing: I simply didn't do some of the idiotic things that they did under my name.
Ironic situations would arise. For example, I once worked at a few sites with another consultant. I had a good reputation and stayed in touch with people I had worked for, sometimes running into them again and again in different circumstances. I was told that this joker would take my code and, as he put it, change it to a standard format. The only changes he ever made were to erase my name and put his in its place. He had the gall to telephone me once to complain about a program I had written some five years before. He admitted that, yes, it had run every day without fail for five years. The obvious question is: What changed? He insisted that nothing had changed and he wanted me to drop everything and rush to where he was working to fix my code. I wasn't buying any of that bulls**t. A few days later one of my friends who was working with this clown called to tell me about what really happened. The slimy bastard had modified my code to make it "more efficient," and, now that it no longer worked, he was trying to blame me. Eventually they restored my original code from backup tapes and the problems went away.
Talking about bad code, when I arrived on contract at Corrections Canada the first assignment I was given was to determine if there were ways to speed up the backup process. A full backup of all of Correction Canada's regional computers was started at midnight every Saturday. The entire process typically took from ten to twelve hours, sometimes longer. The problem was that while the backup was running the computers were inaccessible--and while the computers were inaccessible no visitors were allowed into or out of any federal prison anywhere in the country. Relatives and friends would line up Sunday mornings to visit the government's guests, hoping that the computers would be back online before visiting hours expired.
The code I looked at was a bad joke. The full backup ran Saturday while incremental backups (backup of only the data that had changed since the last full backup) were run Mondays through Thursdays. There was no backup run Fridays. (Don't ask: that wasn't my problem.) So, the first thing such a program needs to know is: what day of the week it is. I read through the code, block after block of it marked off with If today .eq. "Friday" then... skip the following. And nested inside the check for Friday was a block like this: If today .eq. "Saturday" then... execute this code, else execute this other code. To make things worse, the program never did figure out what day of the week it was until it came to the final section of code, and then it would recall itself (run the entire thing again), this time with the day of the week information.
Common sense says that the first thing you do is find out the day of the week. If it is Friday then exit, quit, don't do anything else, just stop right there, we're done. If it is Saturday, then execute the program that contains the code for a full backup. If it is not one of those days, then execute the program that contains the code for incremental backups. And, in the event that they ever do decide to run a backup on Friday, make it simple to change the program so that it would run on Friday--or perform a full backup on a different day of the week. I cannot understand why anyone would want to approach it differently.
Then I addressed the problem of getting those backups done more efficiently, and hence faster. I looked at my resources: I had four high-speed tape drives available. It made sense to keep them all busy simultaneously (unlike the earlier code which accessed them one at a time). The solution to me was to create a queue with a list of all the backups to be run. This queue would release the jobs, one at a time, to whichever tape drive was free. So, we start with four tape drives busy, but, how do we make sure that they will all be busy until the entire process is finished? I doodled a lot with pictures of a tube breaking into four channels and it hit me. The most efficient way to arrange the backup jobs was order of size, processing them from largest to smallest. Think about it. Drive number four should finish the first round of backups first because it is processing the smallest of the four jobs, so it will immediately start processing the next largest backup. And so on. As each drive is freed up, it takes the next largest job. This could mean that one tape drive could spend the entire time on one huge job while the others race through the smaller jobs, but so what: in the end they do roughly the same amount of work in total.
With my code, the entire full backup took two hours to complete. That's down from 10 to 12, or more hours. At 2:00 am Sunday, Canada's prisons, could, in theory, start processing visitors.
You can guess what happened next. Nine months later after I left Corrections Canada to move on, my name disappeared from the programs and they suddenly started failing and having other problems. If only people would get over the idea that they have to fix things that are not broken, remove the name of the original programmer, and then insert their own name.
Have I ever mentioned that there were (and probably still are) a lot of phonies and fakes posing as "computer experts" getting paid large salaries? That sort of thing is easier to get away with in the government than in the private section where you are expected to produce results. In the government accountability is a very rare commodity and it is always easy to blame someone else. I'm proud of the work I did, even if I did make the occasional mistake. I do know that years afterwards some of my code with my name still attached was still running, as reliably as it should be.
Sunday, 5 June 2011
Supressing the Truth
During my five years at Energy, Mines, and Resources there were two separate system support groups. One was called System Support (oddly enough) and the other one was me and one or two other people (it varied over time). We were called System Managers. The System Support team was responsible for the smooth running and maintenance of the CDC Cyber mainframes and an IBM mid-range onsite. We System Managers were responsible for the smooth running and maintenance of all the DEC VAX computers. System Support did not like or trust System Managers.
System Support required a staff of eight, plus a manager. System Management required, well, me. I reported to the head of operations. The Support team looked after a total of three machines. I looked after (at its peak) five machines. I also wrote all the programs required for maintenance and reporting, tape rentals, etc. Support did not write code; there was a separate department for that (of another eight or so individuals). I did all of the front line user support; the Support team did not do that either; there was a separate group (of six to eight under a different manager) who looked after that. In fact, I don't really know what they did other than have meetings.
So, we are talking of about 24 people to look after three machines (plus an operations staff of another eight for a total of 32). It took one person, plus an operator to mount tapes during non-office hours, for a total of two to look after five machines. The other one or two system managers I mentioned were not really necessary. They were more of a contingency plan in case I got hit by a bus or wanted to take some time off. The one junior system manager who stuck with me took over my position after I left the department.
I once asked a member of the support team what was so special about their backgrounds and he answered, "We can read Assembler." Well, so could I; in fact I wrote a few applications in Assembler. (Assembler is as close to machine language as humans can operate. The language itself is made up of instructions that are hard-wired into the CPU. You are talking directly to the machine in its language. Instructions use words like this: MOVC5, PUSHAQ, and CMPL. (The first few letters describe the nature of the operation, in other words which built-in circuits to use, and the latter letters describe the characteristics of the bytes to operate on. So, CMPL means, in English, "Compare two integer longwords (32 bits).") Did I mention that the support group did not write applications, in Assembler or otherwise?
Another time I asked a member of this team to define the word mainframe (it was a word they used frequently), and he told me that a mainframe was any machine that required water-cooling. A rather odd definition if you ask me because many mainframes at the time were air-cooled, just like my VAXes, and water-cooling was becoming obsolete.
To the point: it was decided to test the CDC Cybers, my VAXes, and the IBM mid-range to compare their computational speeds. The test required both text manipulation and mathematics operations. We all agreed. After the tests were run, the results were tabulated and printed up in the in-house newsletter. As expected, the Cybers solidly out-performed the other machines when it came to mathematics. But, the VAX was faster than the IBM system. The IBM came out ahead when it came to manipulating text, with the Cybers a very distant third. I was not surprised; VAXes stood between the two extremes, doing a decent-enough job in both text and mathematics. So, it was published. The Support team as ecstatic because their machine had out-performed the VAX when you took all of the factors into consideration.
...and then, someone discovered an error in their calculations. A decimal point was two places out of position. When corrected, it turned out that the VAX had actually performed 100 times faster than the IBM system.
There was no retraction published and the Support team put the Cone of Silence over the entire project. After a while they denied that any such test had taken place.
This story is important because similar events were happening around the world as Computer Experts desperately fought a losing battle to keep their mainframes a going concern. There were more myths going around about VAXes and their operating system, VMS, than there were stories in all the children's books in existence. Unfortunate, because a great company was eventually driven out of business. The descendants of the VAXes and their operating system are now owned by HP. They are still the most reliable computers available. I read that VAXes survived the collapse of the World Trade Center towers and were still operational after being dug out of the ruble.
System Support required a staff of eight, plus a manager. System Management required, well, me. I reported to the head of operations. The Support team looked after a total of three machines. I looked after (at its peak) five machines. I also wrote all the programs required for maintenance and reporting, tape rentals, etc. Support did not write code; there was a separate department for that (of another eight or so individuals). I did all of the front line user support; the Support team did not do that either; there was a separate group (of six to eight under a different manager) who looked after that. In fact, I don't really know what they did other than have meetings.
So, we are talking of about 24 people to look after three machines (plus an operations staff of another eight for a total of 32). It took one person, plus an operator to mount tapes during non-office hours, for a total of two to look after five machines. The other one or two system managers I mentioned were not really necessary. They were more of a contingency plan in case I got hit by a bus or wanted to take some time off. The one junior system manager who stuck with me took over my position after I left the department.
I once asked a member of the support team what was so special about their backgrounds and he answered, "We can read Assembler." Well, so could I; in fact I wrote a few applications in Assembler. (Assembler is as close to machine language as humans can operate. The language itself is made up of instructions that are hard-wired into the CPU. You are talking directly to the machine in its language. Instructions use words like this: MOVC5, PUSHAQ, and CMPL. (The first few letters describe the nature of the operation, in other words which built-in circuits to use, and the latter letters describe the characteristics of the bytes to operate on. So, CMPL means, in English, "Compare two integer longwords (32 bits).") Did I mention that the support group did not write applications, in Assembler or otherwise?
Another time I asked a member of this team to define the word mainframe (it was a word they used frequently), and he told me that a mainframe was any machine that required water-cooling. A rather odd definition if you ask me because many mainframes at the time were air-cooled, just like my VAXes, and water-cooling was becoming obsolete.
To the point: it was decided to test the CDC Cybers, my VAXes, and the IBM mid-range to compare their computational speeds. The test required both text manipulation and mathematics operations. We all agreed. After the tests were run, the results were tabulated and printed up in the in-house newsletter. As expected, the Cybers solidly out-performed the other machines when it came to mathematics. But, the VAX was faster than the IBM system. The IBM came out ahead when it came to manipulating text, with the Cybers a very distant third. I was not surprised; VAXes stood between the two extremes, doing a decent-enough job in both text and mathematics. So, it was published. The Support team as ecstatic because their machine had out-performed the VAX when you took all of the factors into consideration.
...and then, someone discovered an error in their calculations. A decimal point was two places out of position. When corrected, it turned out that the VAX had actually performed 100 times faster than the IBM system.
There was no retraction published and the Support team put the Cone of Silence over the entire project. After a while they denied that any such test had taken place.
This story is important because similar events were happening around the world as Computer Experts desperately fought a losing battle to keep their mainframes a going concern. There were more myths going around about VAXes and their operating system, VMS, than there were stories in all the children's books in existence. Unfortunate, because a great company was eventually driven out of business. The descendants of the VAXes and their operating system are now owned by HP. They are still the most reliable computers available. I read that VAXes survived the collapse of the World Trade Center towers and were still operational after being dug out of the ruble.
Wednesday, 1 June 2011
Thinking Backwards Cost Taxpayers $300,000,000
Did that title catch your attention? I hope so.
Writing computer code requires a certain perspective, a way of looking at problems that most are not used to. I taught a programming course evenings at a local community college and the biggest problem I faced was getting students to look at problems from a computer's point of view. It is, as I have pointed out earlier, a very elementary system of breaking problems down into component parts. To have a computer print the numbers from 1 to 10 involves several steps. You need to set aside a memory location, you need to initialize the value at that memory location, you need to loop in order to increment the value at that location, you need to tell the computer to print out the result of that incrementation, you need a condition to tell the computer when to stop looping.
In 1993 I was assigned to a two-day project at the Department of External Affairs. I had been reading about a developing scandal in the department for the past few weeks, but hadn't paid particular attention. Numbers of between $200,000,000 and $300,000,000 were being tossed around.The COSICS project at External Affairs was supposed to be a world-wide network designed to link all of Canada's foreign embassies and consulates into one secure communications system. I was asked to take a look at the state of the acceptance testing and to make recommendations for improving the performance of the system.
Security at External Affairs is serious business. I've worked in a Department of Defense Research Facility and the Prime Minister's Office, but had never experienced anything like this. I was met by a security officer who accompanied me throughout the day. At no time was I allowed to get close enough to a computer keyboard to be able to reach out and press a few keys. Instead I had to dictate what to type to a security officer and have him read me back the results. An interesting way to analyze a system. Still it took only a few minutes to identify a problem program. I don't recall the name of the program but its purpose was to scan all mailboxes on the system looking for the keyword; ***Urgent***. On finding it, it was supposed to send an alert to the intended recipient of the message, using a technique that would flash the message on the user's screen. So, let's call it "Sentry." It took almost five minutes to complete its task and then it would start over again--and while it ran the system slowed to a crawl. We've seen this before. It had a raised priority so that it had almost exclusive use of the CPU. Users got whatever free cycles the program didn't need. Interesting....
The security agent couldn't answer any questions about the program. It was his job to type and read back the results.
I was then led to a room where I was given a pile of computer printouts. This represented all of the acceptance testing to date. Acceptance testing is something like a game. The buyer of the system or program has a number of specified targets the product was supposed to meet, and both the buyer and the vendor agree before-hand on specific tests to be run to demonstrate that the system meets those requirements. I have seen this system abused and manipulated, but, such did not appear to be the case here. The thing that jumped out at me as I read through pages after pages of boring tables was that part way through the testing cycle the vendor had made significant changes to the core program. I looked back. Sure enough, those changes would impact earlier tests and change the results. There was no doubt in my mind that the testing process had to be started over.
An External Affairs contractor then took me to the offices of the engineering company who had designed and was implementing the system. You can read all about this company in StevieCamerons' book: On the take: crime, corruption, and greed in the Mulroney years (Random House Canada). I was not at all interested in the sales and analysts explanations that the money spent so far was intended only for half the project, not the entire thing as the press was claiming. It would take another $200,000,000 - $250,000,000 to actually finish the project, but this had been the plan all along. (You might want to go back and read my Amicus article on contract completion and acceptance.) That was all irrelevant to me. I wanted to talk to whoever was responsible for that "Sentry" program. Finally, I was introduced to a very nervous young man, still in his 20's, wearing blue jeans. That's what I wanted.
First of all, I asked him why his program was running at an elevated priority. Did he not realize that this was killing the overall system's performance? His stammered answer was that the contract said that no more than five minutes could pass between the reception of the urgent message and the warning issued to the target's computer screen. Okay, how does the program work?
He had taken the approach of opening each account's email directory and reading through it, one character at a time, looking for ***Urgent***. All email accounts? I asked. "Yes." "Why?" In case there's an urgent message. What if the account has not been used for months? Are you still opening it? He blushed at this point and said he had not taken that into account.
I gave him a hard long look. He was working for a competitor. How much help should I give him? I offered a token. Why did you write this in DCL? (DCL was a scripting interpretive language--which meant that every line of code had to be translated into machine language by the CPU before it could process it; compiled languages, on the other hand, were translated to machine language at the time of their creation, significantly speeding up their execution. Generally speaking, production applications are always written in compiled languages.)
He said he was in the process of re-writing the program, hoping to speed it up.
I wrote up my report the next day and handed it to the project head. First, I told him that the acceptance testing had to be restarted because the core program had been changed. I like the way this guy thinks, he said to a colleague at the meeting. What about performance? he asked. I handed him a page long list of bullet points indicating methods for getting better performance. Uppermost was getting rid of that Sentry program. As long as it was run in the form it was in, they could never achieve a satisfactory level of performance.
What about the requirement for urgent email handling? I was asked. It can be done, I answered, but not with the approach that the programmer is using. I was thanked, hand-shakes all-round and a security agent escorted me from the building. A few months later I read that the entire project had been scrapped.
What I didn't tell the young programmer was that he was thinking backwards. Look at the problem: you want to know when a message with the string ***Urgent*** in it has been received by the system. Then why on earth would you go off looking at messages that have already been received? Especially, why read the same messages, character by character, on each cycle of the program? You already know that they are of no interest. Are you that insecure that you need to verify that no old messages contain that string every five minutes? That's the sort of clarity of thinking that programmers need--and which is very difficult to teach. You have to understand what the computer is doing.
If |I want to know when an email that contains a certain string of characters arrives at the system, then I want to look at each message at the time of its arrival. There is absolutely no requirement for the monitoring program to be cycling every five minutes. It should simply wait--and when a message arrives, it wakes and does its job. I wrote that type of code fairly often so I know that it works; the process has a name: Asynchronous System Trap (AST.)The Sentry program if written in C (and I would have preferred Assembler for this task) would wake, check for the magic string, split off another process to send the message to the user. A dozen lines of code executed in microseconds and only when needed.
Maybe the overall system would have worked in the end if there were experienced analysts and programmers working on it, or maybe it never would have worked. It is hard to know at this stage. International communications, especially secure ones, were just being developed in the early 1990's--this after a decade of out-standing scandals involving computer hackers waltzing through supposedly secure military systems. But thinking backwards will bring you down every time.
Subscribe to:
Posts (Atom)