Explore the Handbook

Procurement and third party services

Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark

Introduction

 

This section provides an overview of key issues and guidance in selecting and using and third-party services for digital preservation. The ways in which a service may be procured often vary according to sector or country. Individual organisations must identify and follow their statutory and regulatory purchasing policies to ensure that services are purchased using the correct procedures. Failure to purchase under the specific guidelines could lead to a serious issue possibly involving compensation to other potential contractors disadvantaged by incorrect purchasing processes.

Three tables are provided as part of the guidance: Staff resources for procurement tasks; Issues and potential advantages and disadvantages of using third party services in digital preservation activities; and a Checklist for assessing storage readiness for digital preservation as procurement is often a major component of implementing archival storage (see Storage and Cloud services). The final Resources section provides additional pointers to and summary description of further guidance and case studies.

Cost will clearly be a key consideration when deciding whether or not to contract out digital preservation but there are also other factors to consider and the advantages and disadvantages of each will need to be balanced against the overall mission of the institution. These include the contract, service level agreement, functionality and quality of the services provided, integration with the institution's processes and environment, disaster recovery and business continuity plans, ability to exit the service if needed, and how the service can be monitored and measured. For example, legal requirements for data privacy or confidentiality may influence whether outsourcing is appropriate or not given the jurisdiction of the service provider and where the service is physically located.

Outsourcing specific tasks or services is by no means a new phenomenon. Repositories have contracted out some of their operations for decades. This is an area in which lessons learned from outsourcing in other services can be of value. A major learning experience which is directly applicable to the digital environment is the critical importance of having sufficient staff resources and knowledge of the technology to be able to prepare effective specifications.

 

Staff resources for procurement tasks

 

The extent to which the potential advantages of using third party services can be maximised and the potential disadvantages minimised will be heavily dependent on dedicating staff resources to the following activities:

Staff resources for procurement tasks

Establishing the organizational remit and appropriate governance when selecting third-party services:

  • Advocate the digital preservation concept.
  • Involve the appropriate internal stakeholders early in your thinking.
  • Develop a communications strategy not just for your procurement team but for broader stakeholders.
  • Maintain an up to date risk register for the procurement.
  • Use the expertise within your organization: for example do you have a procurement section?

Establishing clear and realistic requirements:

  • Align business case with identified needs within organisation. This may take some time but is vital to achieving a good outcome.
  • Create an environment in which your stakeholders can contribute to the discourse and feel they have had input.
  • Learn from experience of others in digital preservation community as a whole: this could include reference site visits and/or sharing documentation and viewpoints.
  • Define which preservation functions are to be included, e.g. ingest; storage, preservation planning, curation and management activities. Are all activities to be outsourced to third party provider or just part of the digital preservation framework?
  • Distinguish between essential functionality and desirable ‘added value’ functionality: using a particular requirements methodology for example the MoSCoW template provides important discipline not just in the early procurement stages but for any project plan going forwards.
  • Have unambiguous and measurable requirements that you can use to clearly show the contractor if they are meeting them or underperforming.

Clarifying legal requirements:

  • Follow institutional and regulatory procurement requirements and processes.
  • Data Protection, FOI, other sensitive content and copyright will all need to be considered.
  • It is worth spending time upfront on negotiating your contracts and agreements with third party providers. Misunderstandings in the future are time consuming.
  • If you require changes to the contract or agreement offered it is vital to secure them before signing any binding contract in law. Build in review points to the contract and understand what levers you have at your disposal.
  • Ensure that you are not obligated to award a contract to provide you with flexibility up to that point.
  • Ensure that you can terminate the contract in a minimally disruptive way if the contractor is not meeting the requirements of the contract. Where possible only pay for the goods and services you have received and not those that you may receive in the future.
  • Clearly understand what services and products are being offered within the baseline costs of the contract and which will incur additional costs.
  • Make all legal requirements, including the legal jurisdiction/governing law, available to potential contractors as early as possible in the procurement as this may greatly affect if they take part in the process.

Maintaining good communication between the contractor and the institution:

  • Service Level Agreement to identify roles and responsibilities of each party.
  • Access to technological infrastructure only or external staff time / development support also?
  • The softer vendor relationship building skills are also important in this context.
  • Is there an active user community for your chosen system that provides feedback and good interaction with the contractor?

Undertaking quality assurance checks:

  • Establish responsibility for functions such as integrity checking.
  • Match quality assurance checks with the measurable requirements you specified in the contract and ensure the supplier is meeting requirements or changing /correcting their practices to meet them.
  • Audit / compliance with legal responsibilities.

Developing and monitoring the contract:

  • This may seem premature but an exit strategy should be identified upfront. Digital preservation function will outlast commercial service provider and current technological infrastructure.
  • Understand your rights regarding your data. Are there costs of retrieving data required for access or transfer to another provider?
  • Be mindful of the market and financial models used by vendors. You may need to think outside the box as these models might not match the financial model prevalent in your organisation (capital expenditure vs revenue expenditure is one frequent dilemma).
  • Awareness of any changes to technological environment for third party provider.
  • Keep up to date with the market after you have concluded your procurement. You need to know how commercially robust your vendor is. Update your due diligence checking periodically.

 

These costs will need to be added to the overall contract costs when calculating the cost benefit of using third party services for digital preservation, bearing in mind that most of these costs will be or should be incurred even if preservation is not outsourced.

 

Issues and potential advantages and disadvantages of using third party services in digital preservation activities

 

Issue

Potential advantage of using 3rd party services

Potential disadvantage of using 3rd party services

Limited staff, skills and experience

  • Provides specialist skills and experience which may not be available within the institution.
  • Without some practical experience and expertise, it will be difficult to develop and monitor effective contracts. Without practical experience it will also be difficult to understand and communicate effectively the requirements of the organisation (or to assess whether they are technically feasible or not).
  • Without practical experience it will also be difficult to understand and communicate effectively the requirements of the organisation (or to assess whether they are technically feasible or not)

Costs
  • Avoids the need to develop costly infrastructure (particularly important for small institutions).
  • If there are economies of scale, outsourcing may well be cost effective.
  • There is very little established bench marking. It is still too new an area.
  • Risk of business failure.
  • Until the market increases there may be an over-dependence on one contractor.

Speed of deployment
  • Allows action to be taken in the short to medium term, pending development of infrastructure.
  • Unless there are adequate exit strategies, may be locked into an outsourcing contract longer than intended.

Core competencies
  • Allows the institution to focus on other aspects of service provision.
  • Danger of either not developing or losing specialist skills base. Still need ability to make informed decisions.

Access considerations
  • Monitoring usage may be more efficient (assuming the contractor has a demonstrated ability to deliver meaningful usage statistics).
  • There may be synergies and cost savings in outsourcing access and preservation together.
  • • Difficult to control response times which may be unacceptably low and/or more costly, especially for high-use items.
  • May be difficult to forecast future needs in this area.

Rights Management
  • Avoids what is often a resource intensive activity for the institution.
  • May significantly increase the cost of the contract and/or complicate negotiations with third party rights holders.

Security
  • Contract can guarantee security arrangements required by the institution.
  • Lack of control, especially for sensitive material.

Quality control
  • A watertight contract will build in stringent quality control requirements.
  • Risk of loss or distortion may still be unacceptably high for highly significant and/or sensitive material.

Storage
  • Access to professionally managed and experienced storage arrangements with easy replication of content and integrity checking.
  • Issues of trust and legal considerations when storing sensitive data.
  • Difficult to anticipate the actual costs of some services e.g. cloud storage and computing because the organisation often does not know exactly how much service it will need. This is uncertainty can be reduced with experience.

 

Checklists for selecting and comparing service providers

 

Checklists and standards can be valuable starting points when considering or evaluating the use of third-party services as they are ready made lists that you can easily adopt or adapt to fit your needs. In particular, checklists help you identify things that you might otherwise forget to consider as well as helping you to express issues and requirements clearly.

Checklists work well when coupled to a maturity model. For example, the NDSA preservation levels allow a checklist to be constructed to see how well a service provider delivers to each level. An organisation identifies what level of maturity they need both now and in the future and then looks for service providers with matching levels.

Checklists and standards for repository services are valuable starting points because you can pick and choose the parts of the checklist that would apply to the specific services you seek. Examples of relevant checklists and standards are available in Resources and are also discussed in more detail in the Audit and certification section of the Handbook.

A Handbook checklist for assessing storage readiness for digital preservation is provided below:

Checklist: questions for your preservation storage service provider
checkbox What level of redundancy does the storage system provide? How many physical locations is digital material held in? What is the geographical distance between them?
checkbox Are different types of storage technology employed to mitigate/spread risk? For example online and off-line storage.
checkbox If a file has become corrupted or unintentionally altered, how does this get detected and when does detection happen? Are audit trails or other forms of logging available to show that data integrity checks have been done and to show the result?
checkbox What is the disaster recovery strategy, for example if a storage system fails or there is a natural disaster at a storage site then how are digital materials recovered? When was the last time this DR strategy was tested?
checkbox What is the storage migration strategy to address technical obsolescence? What happens when the system is at the end of its life and content needs to be migrated to a new system? Is the content still accessible during this process?
checkbox What is the exit strategy when using a given type of storage (e.g. onsite, cloud) for example what happens if the vendor of the storage system goes out of business?
checkbox What measures are in place to contain corrupted or altered files, for example quarantining files to prevent them from being replicated?
checkbox What security and auditing measures are in place to prevent unwanted access and/or modification of the digital materials?
checkbox Who is responsible for monitoring and managing the storage system to ensure it is functioning correctly? Is there continuity of staff in cases of holiday, sickness or departures?
checkbox What contracts, warranties or guarantees come with the storage solution or service that commit the vendor or supplier to support, recovery or replacement if there are any problems?
checkbox What approach or support is in place for storage technology watch and risk assessment so that migrations, refreshes, upgrades or maintenance can be planned and executed in a timely way?
checkbox Are the costs and risks clear so that a trade-off can be assessed and made between number of copies, type of storage, ease of access, and safety of the digital materials?
checkbox What standards does the provider aim to comply with? (e.g. OAIS, Information Security Standards) Does it aim to achieve recognition as a trusted digital repository?
checkbox How can the provider demonstrate they are doing what you have agreed?

 

Resources

OAIS: Open Archival Information Systems: Reference Model for an Open Archival Information System. Recommended practice

https://public.ccsds.org/pubs/650x0m2.pdf

Provides a useful shared terminology and functional model when identifying requirements for procuring third party digital preservation services. (135 pages).

Core Trust Seal

https://www.coretrustseal.org/

The Core Trust Seal is the first step in the global framework for repository certification. This repository assessment includes a 16 point checklist and can be used for self-assessment or peer review.

ISO16363: 2012 Audit and certification of trustworthy digital repositories

http://www.iso16363.org/

ISO 16363 is an evidence-based audit framework for digital preservation consisting of more than 80 criteria that can be used for self-audit or external audit. The criteria used in the standard look across the entire organisation and not just the technical system in which collection content is stored. The CCSDS Magenta Book pre-print version of the standard is freely available at http://public.ccsds.org/publications/archive/652x0m1.pdf.

DIN 31644 Information and documentation - Criteria for trustworthy digital archives

http://files.dnb.de/nestor/materialien/nestor_mat_17_eng.pdf

The extended self-assessment process for digital archives is a helpful checklist developed by nestor on the basis of the DIN 31644 Information and documentation - Criteria for trustworthy digital archives standard.(44 pages).

The NDSA Levels of Digital Preservation: An Explanation and Uses

http://www.digitalpreservation.gov/ndsa/working_groups/documents/NDSA_Levels_Archiving_2013.pdf

The US National Digital Stewardship Alliance (NDSA) Preservation Levels are used widely throughout the Handbook and are helpful in thinking about many areas of digital preservation. There are also Mappings of NDSA preservation levels to cloud storage vendor profiles by AVPreserve.(7 pages).

Where to keep research data DCC Checklist for Evaluating Data Repositories

http://www.dcc.ac.uk/sites/default/files/documents/publications/Where%20to%20keep%20research%20data.pdf

A useful Digital Curation Centre checklist on where to keep research data safe that includes Service Level Agreement maturity levels. It is mainly concerned with external third-party repositories that offer a managed service to the UK research community.(20 pages).

The National Archives Cloud Storage Guidance

http://www.nationalarchives.gov.uk/archives-sector/digital-collections.htm

Provides information about procurement in the context of cloud computing services for preservation purposes, including case studies from several institutions (see below). It is particularly notable for its consideration of the legal issues.

DPC procuring preservation event

http://www.dpconline.org/events/previous-events/1150-procuring-preservation-writing-and-understanding-requirements-in-digital-preservation

For an overview of some of the elements of scoping requirements see the individual presentations listed. Presentations on Requirements analysis, and Procuring Preservation: hoops, hurdles and processes are particularly relevant.

 

Case studies

Archives & Records Council Wales Digital Preservation Working Group

http://www.nationalarchives.gov.uk/documents/Cloud-Storage-casestudy_Wales_2015.pdf

This case study discusses the experience of a cross-sectoral working group of Welsh archives cooperating to test a range of systems and service deployments in a proof of concept for cloud archiving. It explains the organisational context, the varied nature of their digital preservation requirements and approaches, and their experience with selecting, deploying and testing digital preservation in the cloud. The case study examined the open source Archivematica software with Microsoft's Windows Azure; Archivematica with CloudSigma; Preservica Cloud Edition and has begun testing Archivematica with Arkivum 100. January 2015 (10 pages).

Tate Gallery

http://www.nationalarchives.gov.uk/documents/Cloud-Storage-casestudy_Tate_Gallery_2015.pdf

This case study discusses the experience of developing a shared digital archive for the Tate's four physical locations powered by a commercial storage system from Arkivum. It explains the organisational context, the nature of their digital preservation requirements and approaches, and their rationale for selecting Arkivum's on-premise solution, "Arkivum/OnSite" in preference to any cloud-based offerings. It concludes with the key lessons learned, and discusses plans for future development. January 2015 (7 pages).

Dorset History Centre

http://www.nationalarchives.gov.uk/documents/Cloud-Storage-case-study_Dorset_2015_%281%29.pdf

This case study covers the Dorset History Centre, a local government archive service. It explains the organisational context of the archive, the nature of its digital preservation requirements and approaches, its two year pilot project using Preservica Cloud Edition (a cloud-based digital preservation service), the archive's technical infrastructure, and the business case and funding for the pilot. It concludes with the key lessons they have learnt and future plans. January 2015 (9 pages).

Parliamentary Archives

http://www.nationalarchives.gov.uk/documents/Cloud-Storage-casestudy_Parliament_2015.pdf

This case study covers the Parliamentary Archives and their experience of procuring via the G-Cloud framework. For extra resilience/an exit strategy they have selected two cloud service providers with different underlying storage infrastructures. This is an example of an archive using a hybrid set of storage solutions (part-public cloud and part-locally installed) for digital preservation as the archive has a locally installed preservation system (Preservica Enterprise Edition) which is integrated with cloud and local storage and is storing sensitive material locally, not in the cloud. January 2015 (6 pages).

Partnering with IT to Identify a Commercial Tool for Capturing Archival E-mail of University Executives at the University of Michigan

http://files.archivists.org/pubs/CampusCaseStudies/CASE-14-FINAL.pdf

Aprille Cooke McKay, Bentley Historical Library, University of Michigan, examines the challenges and opportunities of partnering with IT to issue a Request for Proposal (RFP) for commercial e-mail archiving software. 2013 (53 pages).

University of Sheffield Procurement Case Study

https://www.sheffield.ac.uk/library/digitalpreservation/casestudy

A summary of the process of procuring a digital preservation system at the University of Sheffield.(2 pages).