I remember seeing, in the 1970’s, computer tape drives in store front windows. Operators would have to go into the window display area to change tapes, making the entire thing a show (and they were wearing white shirts and ties, which is not a particularly good idea when working with equipment that has rapidly spinning parts). That’s what most people thought of when they thought of computers in those days. In fact, a drawing of an old-fashioned tape drive is still used to represent computers in cartoons today. Eventually IBM figured out that sunlight is very hard on plastic tape—in fact, leave it in the sun long enough and it will disintegrate—and so they moved the tape drives into air conditioned computer rooms where they belonged.
But those computer rooms, though nominally clean, were not the best place for disk drives and other highly sensitive equipment. The main problem with the old computer rooms is that they contained people—and people are, and were, the dirtiest things in such rooms. In the early days it took fairly large teams of people to keep computer systems functioning. Contrary to popular belief, most of the people in the computer rooms were not scientists or mathematicians; they were more likely to be high school grads with no particular skills other than a willingness to follow instructions and work through the night and on weekends. Being out of public view they dressed pretty much as they pleased under the lab coats that they wore (in order to protect them from ink in the printers and plotters, not because they were scientists).
When I began there were employees in the computer room who had been there for several years. They were full-fledged government employees protected by unions and a hierarchical system that rated seniority above ability. But, new employees did not receive that benefit; instead, they were hired as “term” employees, meaning they were not protected by the unions and their term of employment was limited (usually to six months, though such terms were regularly renewed.) It took me about a year to win full-time permanent status, but, most employees continued as terms for many years. I once remarked to my boss that I thought the reason that new employees were permanently on term (a slight contradiction in terms) was to prepare for the day when computers would not need so many humans attending on them. He agreed.
I was one of the ones working on the process to get people out of the noisy air-conditioned rooms, to leave the computers to look after themselves. I wrote a series of programs that did some of the routine monitoring that people used to do. Where the huge CDC Cybers and the midrange IBM system required dozens of people in constant attendance, my VAX VMS machines required only an operator to change tapes now and then. Everything else I set up as automated self-regulating systems. My purpose was never to put people out of work, but to free them from boring repetitive tasks that computers could handle on their own at far greater accuracy. People, I figured, would get more interesting jobs; for one, there was a shortage of programmers and software engineers.
Contracting-out expanded rapidly throughout the Mulroney years. Permanent employees are expensive. Besides medical benefits, long term disability, employer taxes, most employers then paid for pension plans. Unionized employees were even more expensive because you had to give them a severance package far better than what the law required—and you had to lay people off in order of seniority leaving you with a skeleton crew of the highest paid group of employees. By contracting-out services you avoid all that overhead. Companies bid to supply the services and you pay a predictable amount for the contract. Who the people are who are doing the actual work and what they are paid does not matter to you as long as they don’t screw up.
Another benefit was that the government could claim it had slashed so many jobs from the public service (always a popular move in the rest of the country). The fact that often those very same people would be back at the same jobs under a contracting arrangement was not mentioned. That accounts for the contradiction of fewer public employees but higher government expenditures. Every political party claims it intends to, or has, let thousands of people go, but, the numbers actually working in the public service keep growing. It’s just that most of them are not “employees;” they are “contractors”—and we don’t count those.
When the axe fell and almost two hundred people employed at the centre and in its support systems were let go, I moved on to a consulting firm. Because the firm I joined had won the contract to supply personnel to keep the computer-room going, I was asked for recommendations by my new private-business employers. For each of the former employees at the centre who I recommended they hire, I was paid $1,000. Not a bad deal from my point of view. (I think that my total was $7,000 in such “bonuses.”)
Because privately-owned companies are in the business of making money, the fewer people they have to pay to meet the terms of a contract the better. And so the squeeze began in earnest. Where once you needed two people: one monitoring the system and the other standing by waiting for a computer to ask for a tape to be mounted, now you could ask the person monitoring the system to do that. In fact, I was once asked to design a system so that the monitoring equipment and tape drives could be kept at the contractor’s site, just leaving the computer and main communications gear at the client site. That way many systems could be monitored and attended to by a much smaller crew. Remote communication simply was not fast enough and reliable enough at the time to set up such a system though it is certainly feasible today. The point is that people were becoming redundant.
“Lights out” technology became a buzz-word for a while, especially as the huge old people-dependant machines were phased out. The advantages of having no humans at all in computer rooms were obvious. At the time, people still had to enter the rooms to change tapes, but, otherwise, all monitoring and software support tasks could be done from outside the room. As communications improved, those monitoring and otherwise supporting did not have to be in the same building that the systems were located. They could be anywhere in the country (or the world, as it is today). In fact, I did a lot of remote support of systems from one coast to the other, often from my office at home. I could nurse a sick server in Kamloops, BC, oversee a disk-drive replacement in Halifax, NS, and talk to someone in New York about a software licence, and then “attend” a conference call with my country-wide clients in the morning, all without leaving the house. So, instead of one of me in every city across the country, there was just one living in a remote forest. And all those computers were running smoothly in cool darkened and people-free rooms.
Tuesday, 11 October 2011
Monday, 3 October 2011
Data and Disaster
In my first year or two in the computing world I saw a lot of card decks. No, not the playing kind—the computer kind. Hollerith cards were originally created by Herman Hollerith (1860-1929) as tools to be used in analyzing statistics. By punching holes in standard-size cards at predetermined locations, one could use mechanical methods to sort data quickly. His technique was used to tabulate the 1890 American census results. Eventually his company, through mergers, became one of the foundations of IBM (founded in 1924).
The standard cards in use in the computing industry throughout the later part of the 20ieth century had 80 columns of 12 rows each. The rows were numbered 0 through 9, with two more rows reserved, originally for plus and minus signs, but later used to expand the vocabulary of a card to allow for upper and lower case letters and other special marks. Sometimes the holes were used to represent binary numbers (which, you will recall, are symbolized by digits of only 0 and 1). As computing spread through the business world in the 1970’s and early 1980’s key-punch operators were in demand. Financial records, medical records, inventory-tracking systems, scientific data were all being recorded on punch cards.
I often handled card decks in my early years. In fact, my first programs were written on punch cards. I would write the program out by hand, then sit at a huge card punch machine to enter the code that would be punched onto cards. The cards were then taken to the computer room where they were fed through another huge machine to be read and executed by the computer. Output was printed onto fan-fold paper. Cards in-paper out was the way we operated then. In my role as operator at the computer centre where I first worked, users would bring me boxes of cards that I would accept through a submission window. I’d put the cards through the card reader when I had time. The job wouldn’t actually execute until a senior operator gave it the go-ahead—and that might not be until 2:00 am the following day, depending on how busy the system was. After it did run, I’d fetch the paper output from the printer that was about the size of a small car then wrap the paper around the original card box and secure it with an elastic band. The package was kept in pigeon holes to await the return of the user.
I’ve gone into some detail here in order to stress the fact that “data” was something tangible that users could hold in their hands. I sometimes tried to convince users that their data was safe if stored on disk—and much more convenient to access. Some wouldn’t buy it. They had spent a career gathering the data vital to their research and didn’t trust it being turned into electronic signals. Cards they could keep secure in their offices—and if the computer centre burned to the ground, their data was safe and could be processed on another computer. Eventually, of course, card decks disappeared except as curiosity items.
But, what about the data they represented? First, while 12 rows of 80 columns can hold a tremendous amount of data, sometimes that simply wasn’t enough space. Many simple computer code instructions, for example, could not be fit into 80 character spaces—to give a mundane example. One of the science groups I worked with recorded data that had to be precise to 29 decimal places. That doesn’t leave much room for anything else. You might recall, if your name was longer than, say, 12 characters, receiving mail with your name truncated. That’s because a programmer was trying to fit your name and full address, along with other relevant information into 80 spaces. So, cards were already unreliable because of the limitations on how much data each card could contain. It is one thing to truncate a name, but quite another to truncate medical information.
Secondly, cards were not as physically secure as many users believed. They were made of very light cardboard and so could get caught in card readers and torn to shreds. They could get dropped and slide unnoticed under a file cabinet. They burned readily—and sometimes they showered out of a university’s computer room to the streets below. Computer disks, which at that time were the size of a standard home dryer, didn’t suffer from those limitations—though a particle of cigarette smoke could get between the disk surface and the read-write head, dislodging the head causing it to crash onto the disk’s surface. Hence: a head-crash. Usually unrecoverable because, even with a new head assembly, the disk surface, where the data was stored, was scratched and computers could no longer make sense of the signals they were reading.
Because of the limitations of physical devices computer rooms everywhere established procedures to protect the data in the event of a head crash. The standard method was to copy the data, in a compressed format, onto tapes. At first the tapes were stored in the computer room but eventually it occurred to someone that this, perhaps, wasn’t the best idea. The tapes should be stored at some other location so that, even if the computer room did burn down, the vital information was secure. At the same time, the data needed to be readily accessible, so compromises were made, like one copy on-site, one off-site.
But that’s not the end of the story. Tapes, made of ferrous oxide on a plastic strip, had limited lifetimes. If the tape was left unused for long enough, eventually data would “bleed” from one location on the tape to an adjacent strip pressed firmly against it. Oxide eventually disintegrated and fell off the surface. Plastic would get brittle given enough time and break when it was stretched by a tape drive. The half-life (the time at which 50% of the tapes became unusable) was usually about ten years. Not good enough. Businesses, government offices, research facilities, medical centres, etc. could not afford to lose half of their data every ten years.
To preserve the data on tapes for as long as possible, tape librarians—if they knew what they were doing—would “exercise” the tapes at regular intervals by mounting, unreeling them, then rewinding. Tape formats changed over the years in order to get more data into physically smaller space, but, the basic limitations of tape never changed. Still, large computer centres relied on tape, sometimes in robotic-controlled “silos” capable of storing thousands of tapes, usually in cartridge format.
In the mid 1980’s to early 1990’s a new approach was developed. Disk drives were now fast enough that data could be written to two disks simultaneously—thus providing a real-time very accessible backup. RAID sets, as groups of linked disks were called, came in different configurations depending on how vital immediate access to backed up data was (and the pocket book of the owner). A popular configuration was a three-disk unit where the data was written to three disks, and could be reconstructed, based on the parity information on the surviving two disks, in the event that one of them crashed. This doesn’t get around the limitation of storing data in one place, so it was often combined with off-site tape storage. An even better solution, when immediate access to data no matter what disaster occurred, was the development of remote clusters.
A system cluster was a group of computers linked together in such a way that they could perform, as far as the user was concerned, as one system much larger than any individual one. Not only was performance, as a whole, enhanced, but immediate access to redundant data was possible. Once it became possible to have a cluster that functioned as a unit even though components were at different locations, live off-site backup was a reality. Such setups are expensive, but absolutely necessary in some situations (like banking information). In these setups usually both sites are accessed as if they were one unit, but the data is kept at both locations. If one site goes down, the other can continue to function, though at a reduced performance rate. Users could possibly notice no more than a temporary glitch in their sessions.
Not every company can afford such a setup. Depending on the urgency of access to data (and the amount of money available) there are different strategies in place to enable a business to recover from the disaster of a computer-room fire. There are companies that specialize in disaster recovery who offer different levels of service. One method that I worked on with two different clients involved a contract commitment to a 24-hour recovery. This meant that the recovery company guaranteed that the business could be functional within 24 hours of a disaster. It involved having equipment available that could mirror the setup of the target system quickly and provide the delivery of off-site backup tapes. As well, rented equipment, such as personal computers, servers, desks, telephones, could be delivered to a pre-arranged location. The tapes would restore the data to the mirror system while the new “office” was being set up.
You might think, on reading this, that every business is now protected by data redundancy. Such, however, is not the case. After I left the field and worked for non-computer companies I was surprised to find no backup strategies in place at all. At one small company, I started a system of backing-up the key data every night to a writable CD and then doing a full backup once a week. The full backup I would hand to the company owner and tell him, “This is your business. Keep it at home in a safe place.”
At another company, we experienced a disk crash. No one at the company seemed to have a clue what was going on as the disk failed over the course of a few hours. A general feeling panic developed as people with no background or experience guessed at what was happening and how long recovery would take. When the disk finally failed completely and a new disk installed by a technician, I was stunned to find out that the company had no backups of the data. When I asked why not, I was told that they had never experienced a problem with that software package before. No one understood when I tried to explain that the failure had absolutely nothing to do with the software application. It was a hardware failure, pure and simple, and, eventually, all hardware will fail.
Fortunately, I was able to rebuild the lost data using information stored on another disk and in the paper files. It took several weeks at 4-8 hours a day—and the only reason I could do it was because I was intimately familiar with the two software packages involved, as well as general computing basics. No one else at that office could have accomplished what I did. They now backup their data nightly.
The standard cards in use in the computing industry throughout the later part of the 20ieth century had 80 columns of 12 rows each. The rows were numbered 0 through 9, with two more rows reserved, originally for plus and minus signs, but later used to expand the vocabulary of a card to allow for upper and lower case letters and other special marks. Sometimes the holes were used to represent binary numbers (which, you will recall, are symbolized by digits of only 0 and 1). As computing spread through the business world in the 1970’s and early 1980’s key-punch operators were in demand. Financial records, medical records, inventory-tracking systems, scientific data were all being recorded on punch cards.
I often handled card decks in my early years. In fact, my first programs were written on punch cards. I would write the program out by hand, then sit at a huge card punch machine to enter the code that would be punched onto cards. The cards were then taken to the computer room where they were fed through another huge machine to be read and executed by the computer. Output was printed onto fan-fold paper. Cards in-paper out was the way we operated then. In my role as operator at the computer centre where I first worked, users would bring me boxes of cards that I would accept through a submission window. I’d put the cards through the card reader when I had time. The job wouldn’t actually execute until a senior operator gave it the go-ahead—and that might not be until 2:00 am the following day, depending on how busy the system was. After it did run, I’d fetch the paper output from the printer that was about the size of a small car then wrap the paper around the original card box and secure it with an elastic band. The package was kept in pigeon holes to await the return of the user.
I’ve gone into some detail here in order to stress the fact that “data” was something tangible that users could hold in their hands. I sometimes tried to convince users that their data was safe if stored on disk—and much more convenient to access. Some wouldn’t buy it. They had spent a career gathering the data vital to their research and didn’t trust it being turned into electronic signals. Cards they could keep secure in their offices—and if the computer centre burned to the ground, their data was safe and could be processed on another computer. Eventually, of course, card decks disappeared except as curiosity items.
But, what about the data they represented? First, while 12 rows of 80 columns can hold a tremendous amount of data, sometimes that simply wasn’t enough space. Many simple computer code instructions, for example, could not be fit into 80 character spaces—to give a mundane example. One of the science groups I worked with recorded data that had to be precise to 29 decimal places. That doesn’t leave much room for anything else. You might recall, if your name was longer than, say, 12 characters, receiving mail with your name truncated. That’s because a programmer was trying to fit your name and full address, along with other relevant information into 80 spaces. So, cards were already unreliable because of the limitations on how much data each card could contain. It is one thing to truncate a name, but quite another to truncate medical information.
Secondly, cards were not as physically secure as many users believed. They were made of very light cardboard and so could get caught in card readers and torn to shreds. They could get dropped and slide unnoticed under a file cabinet. They burned readily—and sometimes they showered out of a university’s computer room to the streets below. Computer disks, which at that time were the size of a standard home dryer, didn’t suffer from those limitations—though a particle of cigarette smoke could get between the disk surface and the read-write head, dislodging the head causing it to crash onto the disk’s surface. Hence: a head-crash. Usually unrecoverable because, even with a new head assembly, the disk surface, where the data was stored, was scratched and computers could no longer make sense of the signals they were reading.
Because of the limitations of physical devices computer rooms everywhere established procedures to protect the data in the event of a head crash. The standard method was to copy the data, in a compressed format, onto tapes. At first the tapes were stored in the computer room but eventually it occurred to someone that this, perhaps, wasn’t the best idea. The tapes should be stored at some other location so that, even if the computer room did burn down, the vital information was secure. At the same time, the data needed to be readily accessible, so compromises were made, like one copy on-site, one off-site.
But that’s not the end of the story. Tapes, made of ferrous oxide on a plastic strip, had limited lifetimes. If the tape was left unused for long enough, eventually data would “bleed” from one location on the tape to an adjacent strip pressed firmly against it. Oxide eventually disintegrated and fell off the surface. Plastic would get brittle given enough time and break when it was stretched by a tape drive. The half-life (the time at which 50% of the tapes became unusable) was usually about ten years. Not good enough. Businesses, government offices, research facilities, medical centres, etc. could not afford to lose half of their data every ten years.
To preserve the data on tapes for as long as possible, tape librarians—if they knew what they were doing—would “exercise” the tapes at regular intervals by mounting, unreeling them, then rewinding. Tape formats changed over the years in order to get more data into physically smaller space, but, the basic limitations of tape never changed. Still, large computer centres relied on tape, sometimes in robotic-controlled “silos” capable of storing thousands of tapes, usually in cartridge format.
In the mid 1980’s to early 1990’s a new approach was developed. Disk drives were now fast enough that data could be written to two disks simultaneously—thus providing a real-time very accessible backup. RAID sets, as groups of linked disks were called, came in different configurations depending on how vital immediate access to backed up data was (and the pocket book of the owner). A popular configuration was a three-disk unit where the data was written to three disks, and could be reconstructed, based on the parity information on the surviving two disks, in the event that one of them crashed. This doesn’t get around the limitation of storing data in one place, so it was often combined with off-site tape storage. An even better solution, when immediate access to data no matter what disaster occurred, was the development of remote clusters.
A system cluster was a group of computers linked together in such a way that they could perform, as far as the user was concerned, as one system much larger than any individual one. Not only was performance, as a whole, enhanced, but immediate access to redundant data was possible. Once it became possible to have a cluster that functioned as a unit even though components were at different locations, live off-site backup was a reality. Such setups are expensive, but absolutely necessary in some situations (like banking information). In these setups usually both sites are accessed as if they were one unit, but the data is kept at both locations. If one site goes down, the other can continue to function, though at a reduced performance rate. Users could possibly notice no more than a temporary glitch in their sessions.
Not every company can afford such a setup. Depending on the urgency of access to data (and the amount of money available) there are different strategies in place to enable a business to recover from the disaster of a computer-room fire. There are companies that specialize in disaster recovery who offer different levels of service. One method that I worked on with two different clients involved a contract commitment to a 24-hour recovery. This meant that the recovery company guaranteed that the business could be functional within 24 hours of a disaster. It involved having equipment available that could mirror the setup of the target system quickly and provide the delivery of off-site backup tapes. As well, rented equipment, such as personal computers, servers, desks, telephones, could be delivered to a pre-arranged location. The tapes would restore the data to the mirror system while the new “office” was being set up.
You might think, on reading this, that every business is now protected by data redundancy. Such, however, is not the case. After I left the field and worked for non-computer companies I was surprised to find no backup strategies in place at all. At one small company, I started a system of backing-up the key data every night to a writable CD and then doing a full backup once a week. The full backup I would hand to the company owner and tell him, “This is your business. Keep it at home in a safe place.”
At another company, we experienced a disk crash. No one at the company seemed to have a clue what was going on as the disk failed over the course of a few hours. A general feeling panic developed as people with no background or experience guessed at what was happening and how long recovery would take. When the disk finally failed completely and a new disk installed by a technician, I was stunned to find out that the company had no backups of the data. When I asked why not, I was told that they had never experienced a problem with that software package before. No one understood when I tried to explain that the failure had absolutely nothing to do with the software application. It was a hardware failure, pure and simple, and, eventually, all hardware will fail.
Fortunately, I was able to rebuild the lost data using information stored on another disk and in the paper files. It took several weeks at 4-8 hours a day—and the only reason I could do it was because I was intimately familiar with the two software packages involved, as well as general computing basics. No one else at that office could have accomplished what I did. They now backup their data nightly.
Subscribe to:
Posts (Atom)