Back to White Paper List »
"Using Technology as a Surrogate for Managing and Capturing
Vital Paper Based Records"
What are vital records?
You have heard it many times but it is worth
repeating. The term 'vital records' refers to that collection of information which is
essential for the ongoing operation of your business, without which your business
may fail. A more formal, text book definition is:
"Records that are essential for the continuous operation
of a business and must be specially protected against possible disasters, both
natural and man-made." operation
of a business and must be specially protected against possible disasters, both
natural and man-made."
Vital records aren't just "secret" things like formulae,
patents, specifications and source code. Vital records are also records of company
registration, tax records, contracts, agreements, etc.
Whereas it isn't the purpose of this paper to explain the
processes necessary for the identification of vital records, it is important that you have
some idea of the types of information it is necessary to capture using technology because
different types of information may require different technology.
One more very important point to consider. There isn't one set of
'static' vital records which you can put away once and then forget about. Most organizations
create vital records each and every day as part of the business process.
Vital records are dynamic.
This fact is a major consideration when choosing appropriate
technologies.
What Format?
We tend to think of vital records as paper,
file folders, letters, reports, contracts etc. However, this is 1996 and in 1996 records
are in many more formats than paper. Whereas we will look at the use of say, imaging
technologies to capture vital records, we also need to realize that some of our vital
records may already be in an 'image' or electronic format, for example:
- Microsoft word format on a hard disk drive
- WinFax Pro format on a hard disk drive
- Lotus notes format on a Unix server
- Microsoft mail format on a Novell server
- Voice mail format on a NT server
- Video image on VHS or Beta tape
- Transaction records in our corporate databases on the IBM
mainframe.
This 'difference' of format is an extremely important
consideration. It also makes the choosing of an appropriate technology even more
difficult. In many cases, it dictates the need to use several different technologies to
capture your vital records.
What do we mean by
"technology"
The New Collins Concise English Dictionary
defines technology as:
"the application of practical or mechanical sciences
to industry or commerce" and "the methods, theory and practices governing
such application", or even more broadly, "the total knowledge and
skills available to any human society". sciences
to industry or commerce" and "the methods, theory and practices governing
such application", or even more broadly, "the total knowledge and
skills available to any human society".
These definitions give me, and you, enormous scope. I am now able
to use whatever aspect of technology that is appropriate to the problem at hand. Most
importantly, I am not limited to computers, optical drives, DAT tape or microfilm. If
appropriate, I could use a thumb nail dipped in tar or even paper, (heaven forbid!).
What are our objectives?
Let's not get confused. The objective is not
to have fun spending the boss's money using the latest, greatest, whiz-bang technology.
The real objective is:
"to store an original record or facsimile, (an exact copy
or reproduction), in such a manner that it is protected from harm or deterioration and can
be retrieved and viewed or read at some indeterminate time in the future".
The technologies available to us are limitless and range from
biscuit tins sealed with solder and wrapped in sewn hessian bags, as used in the First
World War, to High Density ROM writing technology using UltraHigh Focused Ion Beams
encoding up to 23GB per square inch of disk surface at the rate of a terabyte a day.
What are our alternatives?
Magnetic
When we speak about magnetic media we generally mean conventional
hard disk drives and floppies utilizing either a SCSI or IDE interfaces.
When considering the use of magnetic media, two things are
certain:
- we are nowhere near the limits of conventional 'hard' disk
technology; and
- the cost of 'hard' disks will continue to fall at a rate of 50% a
year or greater.
So, we can be assured that the hard disk vendors will continue to
be able to cram more and more data per square inch and that the cost per byte stored will
continue to fall at a very rapid rate.
This means that conventional 'hard' disks will always be a most
cost-effective way to store information, significantly more cost effective and ‘user
friendly’ than optical disks. In today's market, $1,000 buys at least two Gigabytes
of reliable, high speed, easy to install, easy to use, standard interface, SCSI
storage.
However, is this an appropriate medium for storing vital records?
Absolutely, given that the facility is ‘off-site’,
regularly updated, regularly backed up and that the back up is regularly transferred to
the latest hardware and software technology. Have you recently tried to read an
ESDI hard disk written on a PC XT in 1983? (I bet that you, like me,
can't even find the interface cables.)
Tape Units
Magnetic tape is a reasonably durable medium.
I have several 'twenty-four-hundred-foot' tape reels written at
1600 bpi on an IBM mainframe about eight years ago that I can still read on any ‘open
reel’ tape drive. I don't need the original tape drive or computer or operating
system because all vendors wrote their tapes in an industry standard manner. I can use
practically any similar tape drive today on virtually any operating system and have an
excellent chance of reading these tapes. This is one of the strengths of this type of
media.
However, I have great difficulty reading a 40MB tape cartridge
from a proprietary drive installed in one of my 286s about the same time, 1988 or so,
because the format used to encode the data was not an industry standard.
The 'cartridge' situation, unfortunately, has not improved much
since 1988. Let me give you an up-to-date example of 'non-standards'. You can go out today
and buy DAT tape drives from several different vendors at several different capacities
(1GB, 2GB, 4GB, 8GB and 16GB) and I guarantee that you will have major problems with tape
compatibility because of the compression technology used in the hardware of each tape
drive and the differences in the backup software used by each vendor. This is a major
weakness of the current generation of DAT tape drives.
So, is magnetic tape a good medium to use for vital records?
It is fine as long as each backup is verified, regularly tested
and regularly transferred to your most current technology, e.g., 2GB drive.
Optical
In the past, 'imaging' has been incorrectly linked to
optical disks. I have as recently as a few months ago heard an IT Manager talk about
"optical disking" all of his company's data. When I said, "I presume you
mean to capture images of your data", he replied, "yes, optical
disk it!"
You don't need optical disks to process images. Images can be
written to conventional hard disks, floppies, magnetic tape and even microfilm. Optical
disk is only one of many storage options. Imaging and optical disk are not synonyms!
So why do we see so many organizations using optical disk for
information storage? The reason is normally associated with cost and media 'life', (the
remainder or extent of its life.)
Whereas the initial capital cost of an optical device is usually
higher than that of a conventional 'hard' disk drive, the use of removable media,
(i.e.,
optical cartridges and platters), can reduce the average cost per byte stored to a
level below that of conventional 'hard' disk.
Most optical disk vendors claim between 10 and 100 years as the
'life' of their devices. This means that you will be able to read them again when
required. However, the "100 years" certainly hasn't been put to the test and we
severely doubt, without major improvements in medical technology, that the sales person
guaranteeing this performance today will be around to sue in 99 years if it doesn't work!
Why isn't optical disk in more sites? Why is the current use of
optical technology far less than predicted by the industry five years ago? Largely,
because in the past:
- it was too costly;
- the I/O (Input/Output) technology was 'non standard'. The optical
drives weren't supported under the standard operating systems of the day and 'special'
software, drivers and interface code were required often putting it into the 'too hard'
and 'too expensive' category; and
- there was no standard for encoding, an optical disk on one vendor's
drive couldn't be read on another vendor's drive.
What is the situation today?
- Imaging is becoming an essential part of any application. We talk
of ‘image enabled’ applications. Imaging as a feature is becoming ubiquitous.
Every user now expects an application to be able to handle images.
- Optical drives are now low cost and largely supported under
standard operating systems, (though there are still many propriety, 'non standard'
examples). Recordable CD ROM drives for example can be bought for around $1,500 whereas
only a few years ago it would have cost $250,000 to buy a device capable of mastering CD
ROMS.
- These new recordable drives attach to your PC in the same manner as
a normal CD ROM and work as drive "D" or "E". They are cheap to buy,
easy to install and use.
- They support industry standard formats so that CD ROMs produced in
this manner can be read by any other CD ROM drive the same way a 1.4MB floppy can be read
on anyone's PC.
Is optical technology suitable for vital records? Absolutely yes.
However, if you plan to use optical disk for the storage of vital records try to ensure
that they write data according to a common industry standard. Don't get locked into
proprietary technology. Also ensure that, just as with magnetic media, you regularly
transfer your vital information to the latest technology in use.
Microfilm
Are you aware that John Benjamin Dancer first invented microfilm
in 1839? Or that it was used by the French during the Franco-Prussian war of 1870?
With Paris completely surrounded by the Prussian armies, the
French postal authorities used pigeons and microfilms to communicate with the guerrilla
forces fighting in the provinces. Is this the first example of P-Mail with a FLAPI
interface? (This is an in joke for the aware IT.)
The use of microfilm in the commercial world began with a New York
banker called George McCarthy in the 1920's who used microfilm to make a positive record
of cheques that were returned to depositors or forwarded to other banks.
Did you realize that a four by six-inch microfiche, utilizing
a
75X-reduction technique, can store more than 1,600 pages? That the new 'Ultrafiche'
technology using 150X and above reduction technology can store up to 3,300 pages on this
same microfiche card?
And finally, did you know that using a ‘ScreenScan’
microfilm scanner, you can convert microfilm into images (automatic retrieval of blip
encoded microfilm images) and transfer them directly onto your PC's disk drive in TIFF
Group 4 format?
Microfilm is an entirely appropriate technology for the storage of
vital records. It is durable, low cost, requires little storage space and is not dependent
upon current technology so you can be assured that it will be 'readable' ten, twenty,
fifty or even hundred years in the future.
Paper
I can still read a document I created over 13 years ago
when I first set up my company. I can go to the library and read books, which are hundreds
of years old. However, I am no longer able to read a WORM optical cartridge I created six
years ago and I have no chance of reading backups I took on my Burroughs ET2000 PC eight
years ago.
- Paper is not bad.
- Paper is one of the greatest inventions of all time!
- Paper lasts for centuries, modern 'archive' quality paper will
probably outlast you, your children, your grandchildren and your great grandchildren and
your great great grandchildren. Modern, specially designed 'archive' quality paper is the
result of advanced technology. It is a perfectly acceptable medium for vital records.
Please don't disregard paper out of hand. It is always one of the
mediums you should consider when building your vital records program.
Other
Technology is changing at an enormous rate. For example, today
most CD ROM cartridges hold around 600MB of information (say 10,000 A4 pages) yet the
technology now exists to store more than ten thousand (10,000) times this amount
(100,000,000 A4 pages) on a single CD ROM cartridge.
An example of this new technology is called ‘Focussed Ion
Beams’ and was developed by the Los Alamos National Laboratory (LANL) in the US. As
mentioned earlier, it can encode up to 25GB (Gigabytes) of information per square inch of
disk surface.
The first examples of this technology are now being marketed by a
US company called Norsam. They will release a CD ROM writer in August this year called the
'Rosetta Analogue Data Storage System'. It can store one million (1,000,000) A4 pages on a
five and a quarter inch CD. Furthermore, any data thus encoded is permanent and not
subject to abrasion, electromagnetic pulses, temperature or atmospheric changes. The only
way to destroy the data is to destroy the disk!
And finally, this same optical disk can also be read by-any
conventional CD-ROM drive!
The down side to this technology is cost. A mastering system will
cost around one point six million dollars Australian, ($1,600,000). Just a smidgin more
than most of us are prepared to pay. However, I am prepared to bet that the cost drops
significantly over the next few years. I am also prepared to bet that we will also see
service bureaus offering this technology within twelve months.
In my opinion, this particular technology represents the first
serious threat to microfilm technology for large volume archival procedures.
Dispelling some myths
Paper is an inappropriate medium for vital
records storage.
Not true. Modern archive quality paper is a durable medium and may
well be an appropriate choice for some of your vital records. Even old, non-archival
quality paper has proven to be a remarkably durable medium. Try visiting some of the great
libraries of the world, read books and manuscripts, which are many hundreds of years old.
However, when contemplating using paper as the storage medium for
vital records remember that the environment is all-important. Paper should always be
stored in a climate-controlled environment and this cost should be taken into account when
deciding on the most appropriate medium for your vital records.
Optical disk is an ideal medium for vital records storage.
Earlier, I gave you an example where I was no longer able to read
a WORM cartridge I had created some years previously. The reason I couldn't read it were:
- the drive was no longer installed;
- the software drivers used only worked on the then, 1988 version of
SCO Unix; and
- the format used was proprietary.
The data is probably still intact on the cartridge, (of this I
have little doubt), but to read it I would have to resurrect the old hardware drivers and
vendor's software, hire a programmer to upgrade them to the latest version of SCO Unix,
(or install an old 1988 version of SCO Unix if I could find it and the activation key),
find the cables and the interface card, install the original Maxtor drive (hoping that it
still works), and then pray, (we had plug and pray long before Microsoft coined the
phrase).
In fairness to the optical industry, this isn't just a problem
with optical drives. I have also old backups in the following formats:
- Apple 2E diskettes (128KB format)
- Burroughs ET2000 diskettes (810KB format)
- IBM PC XT diskettes (1 MB format)
- Burroughs BTOS diskettes (1MB format)
- 40MB DC2000 half inch cartridge tapes
I can't read any of these without enormous effort and the
resurrection of museum type hardware, (most of which I actually do have stored away). Even
then, the possibility is that at least 50% of the diskettes and cartridge tapes will be
unreadable.
This is not to say that optical drives aren't an appropriate
medium for vital records. It is just that, in my humble opinion, they aren't as good as
paper or microfilm; (now I am going to have the optical disk industry up in arms or after
my head!)
If you write to any kind of magnetic or optical media you must
regularly transfer the records to then current technology. Don't expect to be able to read
a twelve-inch optical platter written in 1995 in 2005 because you won't have the physical
drive or any of the required software and the optical drives of 2005 will not be
able to read it!
The use of any kind of magnetic or optical media means a
commitment to regularly transfer the records to new technology. The fact that an optical
cartridge may have a guaranteed life of 100 years is totally irrelevant because of the
need to transfer the information every five years or so.
Microfilm is out-of-date technology
Hardly! Microfilm is in fact far better than
magnetic or optical media when it comes to 'safe' technology, (the only kind we are
interested in for storing vital records). I can still read microfiche I used twenty years
ago when I worked for Sperry Univac. I don't need the original reader, any reader or even
a good magnifying glass will do!
This same twenty-year-old fiche is in perfectly good condition
even though it has never been climate protected and has shared a drawer with a variety of
substances including stale pizza, (I used to be a programmer).
You will recall that I spoke earlier about the microfilm produced
by Rene Dagron during the Franco Prussian war. Fortunately some of these earlier
microfilms have survived and they are still perfectly legible after more than one hundred
and forty years!
The current technology of microfilm and the ongoing improvements
to this technology make it an entirely appropriate, convenient, low cost and 'safe'
technology for the storage of vital records.
A final warning - the data explosion
When we installed our first Novell file
server in 1988 or thereabouts, it was a NEC Powermate 286 with 80MB of disk space. Our
workstations were also 286's with either 10MB or 20MB hard disks.
My current file servers hold 2OGB of data and our latest Pentium
workstations have 800MB hard drives, (the smallest now available).
That's a growth of 250 times or 25,000% in just ten years!
Have you looked at the growth of your records lately?
Whereas traditional vital records may not have experienced this
same explosive growth, they have certainly grown at a significant rate. Is this not
the era of the 'data warehouse'? What proportion of your corporate database could be
considered to be 'vital'? How many of the millions of records stored in your corporate
file servers and mainframes should be considered 'vital'?
This explosive growth factor is a critical ingredient when
deciding what technology to use to store the various types of vital records in your
organization.
Conclusions
Fifty years ago, the problem was much more
straightforward. You identified your vital records, which were all paper based, you bought
a fire proof safe or you held them off site, probably in a bank vault or safe deposit box.
The more frugal stored them either in the attic or basement of their house.
Today (and tomorrow) the vital records problem is
both much larger and much more complex. Your vital records are in multiple formats and are
much more dynamic. Just identifying your vital records today is a hundred times more
complex task than it was fifty years ago.
You have no choice but to use a variety of technologies to
safeguard them
The good news is that there is any number of appropriate
technologies available including:
- magnetic media
- optical media
- microfilm
- paper
Recommendations
- Hire an expert to identify your vital records
and prepare a vital records program and retention schedule.
- Formally review the vital records program and schedule at least
once a year.
- Ensure that your expert and your IT department work together to
implement an acceptable, cost efficient solution using proven technology that is
consistent with your IT direction and business goals.
- You don't have to spend a fortune, but you do have to spend enough!
It's no wonder that the end users are confused when
the vendors' definitions for the above software applications are often confusing,
overlapping and sometimes downright misleading. It is patently obvious that some software
vendors write their documentation long before they actually produce the product. Comparing
the functionality of the "glossy" to the actual product can sometimes be more
than disheartening.
1996 - Frank McKenna
Back
to White Paper List »
