Taking it to the streets: why Australia's National Archives has embraced digitisation on demand

 

 

by Ted Ling
Director, Legislative and Accessibility Projects
National Archives of Australia

 

Abstract

The National Archives of Australia (N.A.A.) launched its digitisation on demand programme on April 11, 2001, to provide a growing client base with better access to its holdings of historic records without requiring people to travel sometimes great distances across that vast continent to visit the repositories in Canberra or the State capital.  Within six months, the programme had half a million document images on line and was being overwhelmed by clients clamouring for more.   The author reviews the massive project in a case study paper to the second International Conference on Cultural Policy Research (ICCPR) held at the National Museum of New Zealand Te Papa Tongarewa in Wellington, New Zealand, in January 2002.   He explains how the initiative developed, was tested and implemented.  He discusses some of the lessons learned and muses on the project's potential for future researches.

 

---

 

Introduction

Cultural institutions today are faced with the challenge of how to promote wider access to, and greater use of, their collections.  For the National Archives of Australia, this challenge is complicated by the:

·       Size of our collection – about 270 kilometres of records;

·       Value, unique nature and, in some cases, the fragility of the collection; and

·       Wide geographical distribution of Australia’s population which prevents many people gaining direct access to the collection.

This paper describes the Archives’ attempts to meet these challenges through an initiative known as digitisation on demand.  I will explain how this initiative was first developed, trialled and implemented and the lessons learned since implementation.  I will also speak about how we see this initiative proceeding in the future and the potential it gives us to develop digital packages of records from the collection to meet the needs of our researchers wherever they may be.[1]

 

The tyranny of distance and the needs of researchers

The National Archives of Australia has a head office in Canberra and offices with reading rooms in each State and Territorial capital city.  There are eight public facilities throughout the country. 

Such a network is of little use to researchers who are unable to visit our reading rooms.  It is important to remember that the Archives does not move records from one city to another.  Researchers must go to where the records are located where they can view them free of charge.  Alternatively, researchers can have search agents examine the records on their behalf for a fee, or they can have photocopies made and sent to them, also for a fee.

However, why should researchers be penalised because they are unable to visit a reading room, while other researchers who are able to visit a reading room can access records at no cost? The Archives could not adequately address this inequity in the traditional reference service environment.

Computer technology has enabled the Archives to provide access to information about its publications, standards and policies through the Internet to anyone who needs it regardless of where they live or work.  Importantly, for those who require access to the collection, it has provided the Archives with means of presenting information about the collection, and the government agencies that created these records.  This has been achieved through RecordSearch, our online database.  RecordSearch has given our dispersed researcher audience the ability to identify records that may be relevant to their research through a keyword search facility.  However, until digitisation on demand, the Archives was unable to fulfil all the informational needs of researchers because they could not access the actual records online.

 

Evolution of digitisation on demand service

The Archives could not ignore the issues the expansion of Internet access raised.  Researchers increasingly expect to meet most of their information needs online, and at the place and time of their choosing.  The Archives felt that it had to meet this demand or, over time, risk becoming irrelevant.

In 2000, the Archives began a series of trials to investigate digitisation and Internet access using a number of different methods.  The starting point for the project was provision of access for remote researchers that, as far as possible, replicated the reading room experience.  The essential criteria for the trials were speed, minimisation of cost for image capture and adequate legibility of the finished product. 

A number of digitisation methods were trialled – overhead, bookeye and flatbed scanners, as well as digital cameras.  Also tested were digital conversions from microfilm, proprietary commercial scanning systems and high-speed bureau imaging services.

Opening screen in a N.A.A digital archive search of World War I Australian Army dossiers for service papers of legendary Australian Gallipole campaign commander General Sir John Monash, the soldier commemorated in the name of Melbourne's Monash University (detail).

N.A.A. planners also used a variety of record formats to test each method’s ability to capture all types of documents.  Considerable time was spent using World War I Army dossiers – of which there are 420,000 – as test models.  These dossiers had already presented a particular problem with photocopying, because of the variety of ink and pencil colours, handwritten and typed entries, stamps and background colours on the forms.  Therefore, a selection of service dossiers was an obvious choice for inclusion in the trials because if digitising all these formats was successful, other less difficult records should be relatively easy. 

An important facet of the project was the development of a cost-effective and suitable method for making the digitised images available through the Archives’ website.  It became obvious very early on that all images should be attached to their respective record entry on RecordSearch on the Archives website in order to:

Gen. Sir John Monash

·       Remove the need to create or replicate metadata – a significant factor in keeping capture costs low;

·       Maximise the chance of the long-term preservation of the digital data;

·       Avoid the duplication of digitised images on the website;

·       Maximise researchers’ chances of learning about images of records they might wish to consult; and

·       Link images on RecordSearch to other parts of the website if required.

 

Issues emerging from the trials

The clear result of the trials was that overhead digital cameras met the image capture criteria better than the alternatives.  The cameras provide low-resolution images with adequate legibility and the greatest processing speed.  They also provided the simplest system for image capture that best replicated the reading room experience with respect to the appearance of the images.

General Sir John Monash service papers folder noting his post World War I return to Australia "with wife and one child", some of his numerous honours and his "Death after discharge (8.10.1931)". Detail.

The Archives has continued using digital cameras for image capture.  While the planners know that their approach is at odds with much of the literature about digitising archival records, there are two reasons for using the method.[2]  Most of the experience leading to the formulation of high-resolution, best image models has-been based on pictorial material or rare and attractive records, a significant proportion of which will probably be required for illustrative and publication purposes or research. 

Secondly, the literature and digitisation discourse in Australia stress the digital preservation of objects.  The primary aim in this instance is accessibility, not preservation or publication, so the digital quality can be lower.  Paying to copy a record to a higher standard when required for other purposes costs less than capturing all images at a higher standard at the outset. 

In addition, the overhead scanning techniques adopted markedly increase the speed and reduce the cost of image capture.  A significantly greater number of images can be captured and made available to researchers than with more conventional methods.

 

Costs

The speed of processing digitisation requests was considered just as important as cost.  Keeping costs per image as low as possible depends primarily on the number of images the operators can capture per shift.  During the trials, the average number of images captured and processed was 3.26 per minute.  At this rate, and at the salary level of the operators, the cost per image was between $0.11 and $0.14 depending on the type of material being processed, with an average cost of $0.13.[3]  To this had to be added the cost of equipment and the cost of developing the software, so the total cost of capturing an image was $0.14.  These costs are less than any alternative method the Archives examined or knew about.

Further cost savings can be made by separating image capture from image processing and online loading and using staff with different classification levels for the differing levels of capability required for each task.  It was estimated that temporary staff working relatively short shifts could capture more images per minute than the results obtained during the trial.  For example, at four images per minute, the cost for image capture (including the processing and loading time) is reduced to an average of $0.11 per image. 

There are, of course, other costs involved.  The full costs include retrieving records from storage and subsequently returning them (calculated at approximately $0.09 per image), as well as storing the digital images (calculated at $0.02 per image).  However, these costs are incurred irrespective of the image capture system used.

The costs of retrieving a record and returning it to storage already apply if that record is used for remote researcher photocopying, or when it is made available in our reading rooms.  By digitising the record, the Archives is saved the expense of further retrieval and return to storage costs.  The advantage of the digital approach is that these costs are incurred only once, when the request is actioned.  The Archives’ philosophy is that digital capture once equals many uses.

 

The final outcome

At the end of the trials it was clear that low-resolution digitising by overhead camera was the most efficient way to proceed.  The Archives decided to initiate a digitisation on demand service that would allow researchers to request digital copies of records to be loaded onto RecordSearch.  The Archives also decided to identify high use records for digitising and loading onto RecordSearch. 

 

Privacy

However, there was one final issue to be considered before introduction of the online digital service – privacy.

Australian legislation regulates public access to the collection and requires that at the N.A.A. we withhold sensitive personal information from every form of access.  It is important to note that we only digitise records suitable for public release with an access status of ”open” or ”open with exception”.  We sought legal advice to determine if there was a distinction between releasing records to the public in a reading room, or in photocopy form, and loading digital copies onto a website where they can be viewed by anyone with Internet access.  We were advised that there is no difference, so we were able to proceed.

 

Digitisation on demand service – how it works

The Archives’ process of creating digital copies for RecordSearch has three components – capturing images using digital cameras, processing them and loading them into RecordSearch using software developed in-house called ImageStore and ImageLoader. 

 

Capture

Capturing the digital image is a simple task for the operators.  The procedure requires them to:

·       Be aware of preservation considerations at every step of the process;

·       Log the record’s barcode;

·       Place the record under the camera, aligned in a pre-set position;

·       Capture the image by releasing the camera shutter; and

·       Turn to the next folio and continue until the whole record is digitised.

The operator is required to digitise from the top of the record down and to avoid dismantling the record unless it is necessary for legibility.

The hardware used consists of a digital camera (Canon Powershot Pro70) mounted on an adjustable stand for overhead alignment and a computer for uploading the captured digital images.

Operators work in four-hour shifts, with short breaks totalling 20 minutes.  Capture rates have been averaging 1,100 pages per operator per shift.  The average capture rate is easily achievable for regularly formatted records (i.e. where no dismantling of records, removal of pins, plastic sleeves, unfolding of maps, etc is required).

 

Processing

ImageStore rotates, crops, sharpens and adds contrast to the captured images without human intervention.  It allows an on-screen review of documents copied, and the replacement or redoing of single pages if necessary.  The program saves a large and a small copy of each raw image produced during the capture stage.  The small image is the default image and is loaded for viewing.  Persons using RecordSearch can select the larger image for print purposes.

Image processing is about 4,400 pages per operator per four-hour shift.  This is based on a processing rate of 20 images per minute.  However, in practice, the processing rate is constrained by the rate of capture. 

 

Loading

ImageLoader is the conduit for loading the digital images onto RecordSearch.  This program will also load images that have been captured in processes other than the digital camera/ImageStore mechanisms.  It has the facility to replace and delete pages or whole records.

A summary of the Archives’ specifications is at Appendix 1.

 

Commencing the digitisation on demand service

"Time to refine procedures, gauge volume of requests & establish appropriate infrastructure for a national service." Ted Ling

The Archives began its digitisation on demand service on 11 April 2001.  Researchers could request records in our collection to be digitised and loaded onto RecordSearch.  This new service was not publicised widely, as we did not know how the processes that were trialled in an artificial environment might translate to an actual service environment.  Nor did we have an appreciation of the volume of requests that would be handled by an initiative that was very much in an embryonic stage.

Before introducing the service we decided that it would be offered for records located in Canberra only.  This, we thought, would give us time to refine procedures, gauge the volume of requests and establish the appropriate infrastructure needed to provide a national service.  When the service is fully functional it will be extended to our State offices. 

 

How our researchers request online digital records

To request an online digital copy, a researcher selects one of the two icons that appear on the record description screen on RecordSearch.  The icons prompt requests for either a photocopy or an online digital copy.  The researcher lodges an online request for a digital copy and in return receives an electronic acknowledgment. 

When the digital copy has been made and is available for viewing online, an icon appears on the record description screen.  We do not contact researchers and advise them when a record is available but rely on them checking the website from time to time. 

When researchers open the digital copy they see a navigational tool at the top of the page.  It allows them to advance through the record, page by page, or jump ahead to any page they require.  There are also version selection icons that appear at the top left side of the screen.  By default, the ‘small’ digital image (eg 52KB) will appear, which is adequate for on-screen viewing.  However, if required, researchers can select the ‘large’ digital image for printing purposes.  In practice, we have found that the ‘small’ image usually provides a very legible printed copy.

As part of our digitisation on demand service we undertake to provide our researchers with:

·       Legible copies;

·       Each page copied in its entirety (i.e., no information is missing because of poor framing, etc);

·       A copy of the entire record (including masks and removal advices) or, if not, the researcher is told why a full copy cannot be provided.

We do not promise total quality control, as we generally do not check the images.  If we are advised that an image is poor we will simply re-scan it.  Nor do we promise high quality images as if they had been scanned on, say, a flatbed scanner.  There will be some pixellation.  We use standard fluorescent lighting, not studio lighting, so some glossy surfaces do present problems with reflection and the lighting of pages is not always evenly distributed.  Some of these deficiencies can be resolved quite easily but this requires more individual attention and is thus more time consuming and reduces output.  Digitisation is also limited to formats of A3 size or smaller.

In essence, we believe that the primary measure of the success of our digitisation on demand service is legibility, not the cosmetic appearance of the images.

 

Digitisation on demand – six months’ experience

Our digitisation on demand service has now been in operation for six months and our researchers are delighted with the service.  This is what two of them had to say:

“I feel that this service has the potential to revolutionise the study of history for those of us undertaking postgraduate study at regional universities (in my case a PhD in history at the University of Newcastle).”

and

“Sincere thanks to you and your staff for a great job.  You have provided us with detailed information on our family war heroes, information that was previously very difficult to access.  In our case, we were able to establish details, including photos, which were a great joy to a sister of those heroes.”

We have received many similar bouquets.  The service is outstandingly popular … almost too popular!

 

Managing the demand

We have been overwhelmed by the interest generated by this initiative.  Even though there was little publicity the demand was instantaneous and it has shown no sign of abating.  A few statistics will help to illustrate this point:

·       Between 11 April 2001, when the service began, and 30 June, we received 5,548 requests for digital records and in response 177,301 pages were digitised.

·       Between 1 July 2001 and 30 September we received 7,325 requests for digital records and in response 265,605 pages were digitised.

·       By 30 September 2001 we had digitised a total of 442,906 pages.

We initially promised our researchers a 30-day turnaround time.  However, the high volume of requests has meant delays of over 80 days.  We now simply tell researchers at what date requests currently being digitised were received.

To help manage the demand we have introduced night shifts, three nights a week.  We have a team of 10 operators, with four operators working shifts between 9.00 am and 10.00 pm.

However, the demand is still rising.  So we have now limited the number of records a researcher can request to five each year.  However, this has not stemmed the flow.

The service is currently free.  We could introduce a fee but why should someone have to pay for a digital copy that is then loaded onto our website for the entire world to see for free?  Furthermore, the service we are now providing is intended to assist those researchers who cannot visit our reading rooms, where they could access the records at no charge.

We could adopt the same policy as the National Archives of Canada and consult with various groups to ascertain which are our most valued records and then digitise them en masse, rather than digitise individual records on request.  But if we followed the Canadian model we would probably be digitising some records that are of no interest to many researchers.

The reality is that through our digitisation on demand service we are giving our researchers exactly what they want.  Our researchers are telling us precisely which records are of value to them and we are doing our best to meet that demand.

"New Military Commander's Plans for Portuguese Timor": 17th July 1961 campaign plan for Australian forces in Timor, from Australian Army dossiers in N.A.A. digital archive.

It is Archives’ policy to develop a combination of proactive and reactive digitisation services.  Proactively, like the Canadians, we will identify certain high demand records and have them digitised by external contractors.  Half of the requests we currently receive are for World War I Army dossiers, so it is an ideal group of records to digitise in its entirety.  Reactively, we will continue to digitise records on demand in-house.  The delays are likely to continue and we will advise our researchers accordingly.  If they are prepared to wait we will digitise the records they want at no charge, if they cannot wait they have the option of obtaining a photocopy (for a fee) or visiting our reading rooms to see the records personally (at no cost).  So far, the evidence is that most researchers appreciate the service and are prepared to wait.

 

Administering the initiative

The service we offer is, in a sense, a microcosm of any other record-related operation.  It requires administration and is akin to running a reading room – the researcher is not present but a similar range of records-related problems are encountered for which experienced staff are needed, such as:

·       Ordering and returning records to the repository;

·       Retrieving records on issue to someone else;

·       Writing to individuals to explain delays or difficulties (eg large format), and dealing with special requests.

Our early experience has shown that a larger than anticipated number of records has required dismantling or special preparation prior to capture (eg sorting pages that were out of order) and this has slowed the capture rate.  It also reflects part of the ‘unknown’ facet of operating in an actual work environment as opposed a trial environment.

However, we are confident that productivity rates can improve, in light of further refinements we are making to the software that will allow a number of processes during the data capture stage to be undertaken automatically, rather than manually as is presently the case.

Once the records have been loaded onto the website, if a researcher wishes to download and print a copy they must currently do so one page at a time.  This can be quite frustrating when a record consists of several hundred pages.  We are contemplating modifications to the software that will give researchers the choice of downloading an entire record, or particular pages only.

 

Wear and tear on the digital cameras

We have discovered that the cameras we use, which are designed for the domestic market, are not coping with constant high usage.  We have found that after six months they are starting to wear out and it is unlikely that they will last more than a year.  This is not surprising when you consider that each camera is processing over 22,000 images a month. 

While we are currently examining the suitability of other cameras, it should be noted that the ones we are currently using cost just over $2,000 and we do not consider this to be expensive.

 

Extending accessibility into the future

Despite the overwhelming demand and the technical difficulties we have encountered it is clear that we have introduced a service that our researchers want and that this service will only continue to grow.  We know that many institutions are watching with interest to see how we manage the service.  In this, the final part of my paper, I would like to talk about where I see digitisation taking us.

In the past year we have worked with a number of organisations to increase accessibility to our collection through the Internet.  The digital system that we have established allows external sites to link to digital records in RecordSearch.  This has a multiplier effect in that some of the researchers who come to RecordSearch from other sites may not have had access to these records if it had not been for the link provided from their original search site.  A few examples will illustrate this point.

Alliances and links

We have developed an alliance with the Hellenic Studies Centre at La Trobe University in Victoria, to help them gather together records that document Greek migration and other aspects of life in Australia for Hellenic people.  Rather than requesting photocopies of relevant records, the Centre now selects records and the Archives digitises them.  The Centre then provides links from their online collection to the records on RecordSearch.  The result is that a significant group of records are available through the websites of both organisations. 

The John Curtin Prime Ministerial Library in Western Australia maintains a website that deals with the life and career of Australia’s wartime Prime Minister John Curtin.  The website contains digital copies of a large array of records, many from our collection.  The Archives and the Library have worked together to make these digital images available.  There are now direct links between the Library’s website and digital copies of our collection.  So again we have an example of a single digital capture resulting in many uses.

 

Digital packaging

We can, however, take these developments much further.  We now have the capability of taking digital copies of records and ‘wrapping’ them into a digital package.  The researcher can access the information they need for a particular subject or topic, together with pertinent information about those records.  Here are a few examples.

At the beginning of each year, Australian Cabinet records which are 30 years old are publicly released.  A media launch takes places in early December before the public release.  At the moment we provide journalists with a bound volume of selected highlights (which we call a ‘brick’).  The journalists take the volume away with them and use it to write their stories.  In the future we can package these records in a digital form, so journalists can access a digital ‘brick’ from their home or office.

During the digitisation trials, we were approached by the Foreign Relations Department at the University of Newcastle, north of Sydney.  They wanted to make digital copies of archival documents available to their students for research course work.  A number of records were digitised and have subsequently been made available online, both for students and for anyone else interested in foreign relations.  This group of records covers aspects of Australia’s foreign relations with Japan, Indonesia, Portuguese Timor and China.  We have since developed a number of subject-based icons on our website so that researchers have the option of locating records grouped by subjects such as Foreign Relations.  Researchers can access records by their control numbers, or they can simply search the Foreign Relations icon.  While there is only one digital copy of each record, each can be accessed through different points on our website. 

We are now approaching other Australian universities that teach courses that could make use of our collection.  An example is the Northern Territory University which has a course entitled Commonwealth Administration of the Northern Territory 1911–78.  The Archives has a wealth of material from government agencies relevant to this course.  Again, we can ‘wrap’ these records into a digital package, so those students undertaking the course have access to the records they need from their home or library. 

 

Committees of inquiry

In recent years there have been a number of committees of inquiry, e.g. Aboriginal deaths in custody, the separation of Aboriginal and Torres Strait Islander children from their families, and child migration from the United Kingdom and Malta.  Such committees have often indicated how important records are to people’s lives and their identities.  We now have the potential to provide online copies of key records identified by these committees and referred to in their reports.  We can actually link the records to the committees’ online reports.

 

Fact sheets and reference guides

Like many archival institutions we produce an array of fact sheets and detailed subject-based reference guides.  These products are located on our website.  We can now link digital copies of records to the fact sheet or guide in which they are listed.  This provides researchers with an opportunity to view not only the information about a record, but a digital copy of the record as well.

 

Government agencies

Digital accessibility is just as important for government agencies as it is for public researchers.  Agencies often require access to records for their current work, even when those records are more than 30 years old and are thus publicly available.  The difficulty in the past has been the need for these records to be in archival custody and publicly accessible (both of which are requirements under our legislation), while still available for reference by agency staff.  In some cases their staff have had to consult these records in our reading rooms or have photocopies made.  In the future, digitisation will facilitate agencies’ access to our collection.

In the case of the Office of Australian War Graves (OAWG), the agency had a microfilm set of grave registration sheets produced to enable them to continue with their core work following the transfer of the registration sheets to our custody.  They have used the microfilm for a number of years but now it is deteriorating badly and recopying will be required.  Fortunately they have a new copying option available in the form of digitisation.  The Archives is working with OAWG as a joint project expected to result in digital copies being available through RecordSearch.  This will fill the agency’s need for access to the information contained in the registration sheets and, in addition, will make these documents available for others.

 

Conclusion

Over the past five years we have witnessed how new and emerging technology has changed people’s lives.  The Internet has now become a central part of our communication, business and entertainment.  According to the Australian Bureau of Statistics, in 1997, 7.5% of households had access to the Internet.  The following year access increased to 19%, followed by 25% in 1999 and 37% in 2000.  To give a more pertinent reading, in the 12 months leading to November 2000, 50% of all Australian adults accessed the Internet.  I am sure that Australia is not unique in this regard.

In 1995 the Archives grasped the opportunity that the Internet provided to make our services and research tools more widely accessible.  It was this technological foundation that enabled the transition to an online digital service that began in April 2001.

If we are to continue to provide accessibility to our collection and services that are relevant to our ever-changing environment, we cannot afford to ignore new technologies or the wants and needs of our researchers.  Our digitisation on demand service is simply the beginning.  There is much more that we can do and the only limitations are technology and the resources available to us.

 

---

 

 

Appendix 1: Image capture output specifications and statistics

 

Digital camera:  Canon PowerShot Pro70

Image resolution: 180 dpi

Image format:  Progressive jpeg, slightly compressed

 

Document size

 

Pixel dimensions

 

Average image file sizes

 

Capture rate – 1,100 pages per operator per four-hour shift.  This is based on a capture rate of five images per minute.  This is easily achievable for records in regular formats (i.e. where no dismantling of records, removal of pins, plastic sleeves, unfolding of maps, etc is required).  The capture rate can quickly fall to as little as 2 to 2.5 pages per minute if this sort of manual preparation is needed. 

Processing time – 4,400 pages per operator per four-hour shift.  This is based on a processing rate of 20 images per minute.  In practice, the processing rate is constrained by the rate of capture. 

Output rate – In ideal conditions, an average of 960 images can be captured and processed, per operator, per four-hour shift.  This is a rate of 240 images per hour, comprising 48 minutes per hour of capture and 12 minutes per hour of processing.  Each shift involves approximately 48 minutes (12 minutes x 4 hours) of processing by ImageStore.  Breaks are usually taken while processing is occurring.  Processing time is also used for ordering records, returning them to the repository, reassembling records that have to be taken apart for capturing, searching for records that cannot be located, and other administrative tasks associated with the process.

Storage of captured data – Captured data is housed on a single server and is presently 207GB in size.  It is growing at the rate of 9.6GB – 16GB per month.  The server has about 43GB of free space left.  An expansion module has been added providing another 372GB of storage.

 

---

Author

Ted Ling is the National Archives of Australia's Director, Legislative and Accessibility Projects. He has written a number of publications on the subject of archival buildings including Solid, Safe, Secure: Building Archives Repositories in Australia and Guidelines for Mobile Shelving for Archives, Libraries and Museums. He is a member of the International Council on Archives Committee on Archival Buildings and Equipment. Currently, he is responsible for reviewing the Commonwealth Archives Act and preparing ministerial proposals for amended archival legislation and for managing a number of the Archives' outreach activities including digitisation on demand, Summer Scholarships Scheme, Frederick Watson Fellowship and Community Heritage Grants.

 

---

For further study

The Caldeson Consultancy Indexing Concepts Guide gives a new slant to thinking on the classification of images, particularly photographs, for digitised archive applications. The Guide discusses methods of indexing facial expressions and body language, images of moods, feelings and actions with novel approaches valued by picture editors and other image users.

 

---