I caught up with Adam Jansen, Digital Archivist, from the Washington State Digital Archives last week.  Adam is heading up a drive to support a community of individuals that are interested in the issues of the long term storage of digital records - enter stage left, the Digital Preservation Network.

"The Digital Preservation Network is dedicated to forging a community of practitioners who are focused on the issues of preserving the digital records and publications of government.  This online forum will be a repository for the exchange and discussion of ideas, research, strategy and documents that can be used by other practitioners in their organization.  Membership into this community will be open to any practitioner employed in a government funded institution that is currently researching or participating in appraisal, acquisition, preservation or access of government records or publications."

Unfortunately the answer does not come in the form of buying a few more disk drives to solve problems; even big ones!  All sorts of things need to be considered including ensuring the integrity of data for the long term, agreement on standards for the classification of digital records, file formats - and their futures, indexing, retrieval, presentation, and so on.  This is a big debate, and not one that can happen in a vacuum at a state, or even US level.  Adam has currently got over 250 members on the site that are made up from the primary libraries in the US, and more recently international members from the likes of the British Library - an organization we have also been working with to assist them in scanning their 25,000,000 pages of content.

What is interesting about this particular community is the speed at which it has become established, and the amount of activity in it despite the fact that it must appear a niche area to the casual observer...far from it.  The site launched earlier this year, and the rate of new membership growth has exceeded Adam's expectations.  Digital Preservation is something that is increasingly on the agenda of many public sector organizations across the world.  So much of what we do and governments produce now is captured on some form of digital medium; whether it is a document, a picture, a video, an audio recording or something else.  In connection with that, so many new systems that are procured now to service the needs of countries will create even more data, so the issue needs a healthy dose of shared thinking

According to a study that Berkeley did in 2003:

Print, film, magnetic, and optical storage media produced about 5 exabytes of new information in 2002. Ninety-two percent of the new information was stored on magnetic media, mostly in hard disks.

• How big is five exabytes? If digitized with full formatting, the seventeen million books in the Library of Congress contain about 136 terabytes of information; five exabytes of information is equivalent in size to the information contained in 37,000 new libraries the size of the Library of Congress book collections.
• Hard disks store most new information. Ninety-two percent of new information is stored on magnetic media, primarily hard disks. Film represents 7% of the total, paper 0.01%, and optical media 0.002%.

This is a mind-boggling amount, but you can appreciate the challenge.  Berkeley concluded their report by saying they had started to train a new breed of professional called 'information managers'...perhaps some of them are even now contributing to the DPN...

Pay DPN a visit.