Data and Disaster

In my first year or two in the computing world I saw a lot of card decks. No, not the playing kind—the computer kind. Hollerith cards were originally created by Herman Hollerith (1860-1929) as tools to be used in analyzing statistics. By punching holes in standard-size cards at predetermined locations, one could use mechanical methods to sort data quickly. His technique was used to tabulate the 1890 American census results. Eventually his company, through mergers, became one of the foundations of IBM (founded in 1924).

The standard cards in use in the computing industry throughout the later part of the 20ieth century had 80 columns of 12 rows each. The rows were numbered 0 through 9, with two more rows reserved, originally for plus and minus signs, but later used to expand the vocabulary of a card to allow for upper and lower case letters and other special marks. Sometimes the holes were used to represent binary numbers (which, you will recall, are symbolized by digits of only 0 and 1). As computing spread through the business world in the 1970’s and early 1980’s key-punch operators were in demand. Financial records, medical records, inventory-tracking systems, scientific data were all being recorded on punch cards.

I often handled card decks in my early years. In fact, my first programs were written on punch cards. I would write the program out by hand, then sit at a huge card punch machine to enter the code that would be punched onto cards. The cards were then taken to the computer room where they were fed through another huge machine to be read and executed by the computer. Output was printed onto fan-fold paper. Cards in-paper out was the way we operated then. In my role as operator at the computer centre where I first worked, users would bring me boxes of cards that I would accept through a submission window. I’d put the cards through the card reader when I had time. The job wouldn’t actually execute until a senior operator gave it the go-ahead—and that might not be until 2:00 am the following day, depending on how busy the system was. After it did run, I’d fetch the paper output from the printer that was about the size of a small car then wrap the paper around the original card box and secure it with an elastic band. The package was kept in pigeon holes to await the return of the user.

I’ve gone into some detail here in order to stress the fact that “data” was something tangible that users could hold in their hands. I sometimes tried to convince users that their data was safe if stored on disk—and much more convenient to access. Some wouldn’t buy it. They had spent a career gathering the data vital to their research and didn’t trust it being turned into electronic signals. Cards they could keep secure in their offices—and if the computer centre burned to the ground, their data was safe and could be processed on another computer. Eventually, of course, card decks disappeared except as curiosity items.

But, what about the data they represented? First, while 12 rows of 80 columns can hold a tremendous amount of data, sometimes that simply wasn’t enough space. Many simple computer code instructions, for example, could not be fit into 80 character spaces—to give a mundane example. One of the science groups I worked with recorded data that had to be precise to 29 decimal places. That doesn’t leave much room for anything else. You might recall, if your name was longer than, say, 12 characters, receiving mail with your name truncated. That’s because a programmer was trying to fit your name and full address, along with other relevant information into 80 spaces. So, cards were already unreliable because of the limitations on how much data each card could contain. It is one thing to truncate a name, but quite another to truncate medical information.

Secondly, cards were not as physically secure as many users believed. They were made of very light cardboard and so could get caught in card readers and torn to shreds. They could get dropped and slide unnoticed under a file cabinet. They burned readily—and sometimes they showered out of a university’s computer room to the streets below. Computer disks, which at that time were the size of a standard home dryer, didn’t suffer from those limitations—though a particle of cigarette smoke could get between the disk surface and the read-write head, dislodging the head causing it to crash onto the disk’s surface. Hence: a head-crash. Usually unrecoverable because, even with a new head assembly, the disk surface, where the data was stored, was scratched and computers could no longer make sense of the signals they were reading.

Because of the limitations of physical devices computer rooms everywhere established procedures to protect the data in the event of a head crash. The standard method was to copy the data, in a compressed format, onto tapes. At first the tapes were stored in the computer room but eventually it occurred to someone that this, perhaps, wasn’t the best idea. The tapes should be stored at some other location so that, even if the computer room did burn down, the vital information was secure. At the same time, the data needed to be readily accessible, so compromises were made, like one copy on-site, one off-site.

But that’s not the end of the story. Tapes, made of ferrous oxide on a plastic strip, had limited lifetimes. If the tape was left unused for long enough, eventually data would “bleed” from one location on the tape to an adjacent strip pressed firmly against it. Oxide eventually disintegrated and fell off the surface. Plastic would get brittle given enough time and break when it was stretched by a tape drive. The half-life (the time at which 50% of the tapes became unusable) was usually about ten years. Not good enough. Businesses, government offices, research facilities, medical centres, etc. could not afford to lose half of their data every ten years.

To preserve the data on tapes for as long as possible, tape librarians—if they knew what they were doing—would “exercise” the tapes at regular intervals by mounting, unreeling them, then rewinding. Tape formats changed over the years in order to get more data into physically smaller space, but, the basic limitations of tape never changed. Still, large computer centres relied on tape, sometimes in robotic-controlled “silos” capable of storing thousands of tapes, usually in cartridge format.

In the mid 1980’s to early 1990’s a new approach was developed. Disk drives were now fast enough that data could be written to two disks simultaneously—thus providing a real-time very accessible backup. RAID sets, as groups of linked disks were called, came in different configurations depending on how vital immediate access to backed up data was (and the pocket book of the owner). A popular configuration was a three-disk unit where the data was written to three disks, and could be reconstructed, based on the parity information on the surviving two disks, in the event that one of them crashed. This doesn’t get around the limitation of storing data in one place, so it was often combined with off-site tape storage. An even better solution, when immediate access to data no matter what disaster occurred, was the development of remote clusters.

A system cluster was a group of computers linked together in such a way that they could perform, as far as the user was concerned, as one system much larger than any individual one. Not only was performance, as a whole, enhanced, but immediate access to redundant data was possible. Once it became possible to have a cluster that functioned as a unit even though components were at different locations, live off-site backup was a reality. Such setups are expensive, but absolutely necessary in some situations (like banking information). In these setups usually both sites are accessed as if they were one unit, but the data is kept at both locations. If one site goes down, the other can continue to function, though at a reduced performance rate. Users could possibly notice no more than a temporary glitch in their sessions.

Not every company can afford such a setup. Depending on the urgency of access to data (and the amount of money available) there are different strategies in place to enable a business to recover from the disaster of a computer-room fire. There are companies that specialize in disaster recovery who offer different levels of service. One method that I worked on with two different clients involved a contract commitment to a 24-hour recovery. This meant that the recovery company guaranteed that the business could be functional within 24 hours of a disaster. It involved having equipment available that could mirror the setup of the target system quickly and provide the delivery of off-site backup tapes. As well, rented equipment, such as personal computers, servers, desks, telephones, could be delivered to a pre-arranged location. The tapes would restore the data to the mirror system while the new “office” was being set up.

You might think, on reading this, that every business is now protected by data redundancy. Such, however, is not the case. After I left the field and worked for non-computer companies I was surprised to find no backup strategies in place at all. At one small company, I started a system of backing-up the key data every night to a writable CD and then doing a full backup once a week. The full backup I would hand to the company owner and tell him, “This is your business. Keep it at home in a safe place.”

At another company, we experienced a disk crash. No one at the company seemed to have a clue what was going on as the disk failed over the course of a few hours. A general feeling panic developed as people with no background or experience guessed at what was happening and how long recovery would take. When the disk finally failed completely and a new disk installed by a technician, I was stunned to find out that the company had no backups of the data. When I asked why not, I was told that they had never experienced a problem with that software package before. No one understood when I tried to explain that the failure had absolutely nothing to do with the software application. It was a hardware failure, pure and simple, and, eventually, all hardware will fail.

Fortunately, I was able to rebuild the lost data using information stored on another disk and in the paper files. It took several weeks at 4-8 hours a day—and the only reason I could do it was because I was intimately familiar with the two software packages involved, as well as general computing basics. No one else at that office could have accomplished what I did. They now backup their data nightly.