Collaboration in Digital Preservation

In May 2018 I was invited to participate in a workshop entitled ‘The Challenges of Managing Digital Information for Measuring the Sustainable Development Goals’ with a brief to discuss ‘Fostering National and International Collaboration in Preserving and Reusing Digital Information’.  This blog post is the manuscript for that presentation. The workshop organisers intend to publish the outcomes of the workshop so the text is presented here as an early draft to encourage comment, criticism or addition.

In this short paper I want to talk about the opportunities and the challenges of collaboration in digital preservation. In particular I will draw on the experience of the Digital Preservation Coalition (DPC), an agency which will be familiar to some of you. I want to reflect on some of the successes we have achieved and barriers we have faced over the years and how we have overcome them, or not as the case may be.  I will also project a little to the future, recognising that the collaboration possible through organizations like the DPC cannot be static.  I also want to question in a slightly more light-hearted but hopefully insightful way why the DPC is still here and why we have not all been able to get back to our day jobs.

I should add, that there are many other types of collaboration possible and many other examples of good practice from which we can learn.  I am happy to reflect on these in discussion and want to start by noting and paying tribute to them. The fact that I am not going to provide much detail about them represents nothing more than my sense that I am not the right person to provide a deep reflection on them.  So, this is not an overview of every collaboration that has existed, nor will it be an abstract consideration of organizational structure.  But I hope that the DPC experience will give you some insights on how to structure a sustainable collaboration for specific purposes in digital preservation, and perhaps also how not to.

Also, my purpose in this paper is not to make a rallying call for more collaboration or to praise collaboration as a worthwhile goal.  For the avoidance of doubt, I do indeed wish to encourage fuller and more productive collaboration; I firmly believe that the challenges of digital preservation cannot be addressed without a commitment to collaboration; and moreover, I consider the DPC to be well-suited to support both of these outcomes.  But I will simply take these as read.  The plain fact is that a large, growing and diverse community around the world seeks opportunities to work together in the preservation of digital content.  I don’t question here the origins of that aspiration, but simply present the strengths and weaknesses of one collaborative model.

The DPC

The Digital Preservation Coalition is probably already known to many of you, not least because we have been in existence for sixteen years.  I am told that first meeting of the DPC Board was impressively optimistic about the prospects and impact of the new agency, more or less implying that work would be complete within a year or so, and therefore that they might think to wind it up in around two years.  I cannot vouch for the truth of that anecdote but there’s no doubt that the DPC’s continuing existence and unbroken expansion would have surprised the founders.  It’s not as if digital preservation has lacked strong advocates and wise developers over the years.  In some sense the DPC’s continuity has been both cause and effect of the growing digital preservation community.

Some basics are worth describing because I think the configuration of the DPC has contributed significantly to its success and stability over the years.  DPC is a not for profit company limited by guarantee and registered in England.  As a membership organization its members are its shareholders who thus exercise all the prerogatives of an independent legal entity through a board of directors drawn from the members. More recently this has been supplemented by a representative council comprised of our full members who set strategy.  The DPC has no independent funding source outside of its annual member subscriptions so in every practical sense it is the members who guarantee its existence. 

The DPC has a simple business model recognisable to any club or association: members pay subscriptions to access the DPC and get various benefits as a result.  It obtains small amounts of funding from events and from occasional consultancy and research grants, but the purpose is such that DPC tries to avoid placing itself in competition with its own members for funds. Consequently, ambitions for additional funds are sensibly inscribed and subject to approval. The DPC’s planning assumption is that the majority of income derives from subscriptions each year because otherwise the organization risks losing focus on member needs.   A number of large external grants have been received over the years to extend the member programme. By and large the energy of the expanded programme has proven a draw to those outside the DPC and has raised the organization’s profile. Thus, by the time the grants have come to an end, the concomitant growth in membership has funded the continuance of the expanded programme.

The membership has been robust over the years suggesting that by and large agencies get value from their subscription.  It is striking that around 2010 when the financial crisis began to effect public funding that a group of DPC members dropped out: not because they were unhappy but because the government grants on which they survived were being wound down.  So, it is not true to say that the DPC lost members during that period, so much as those members simply no longer existed. 

The experience of other cognate agencies around the same time is also telling. Agencies like UKOLN, MLA, RIN, BLPAC or CDLR which relied on one or two large grants from public sources seemed much more generously funded than DPC: but with a single point of failure were exposed to dramatic drops in income when one or two funders pulled out.  DPC with a very large number of small funders from multiple different sectors turned out to be more robust.

The DPC’s mandate is derived from its members, expressed in a strategic plan which has been renewed every three years since 2009. This sets out the values, instruments and scope of the Coalition and provides an elementary guide for all the collaborations that follow.  For example, the DPC remains neutral with respect to technology and vendors.  Because the DPC has no tie to any given tools or products its advice and training are impartial.   This is important when it comes to building confidence, but it also limits what the DPC can do in terms of providing tools or services and it means vendors cannot become members.

The strategic plan also outlines the areas in which the DPC works.  The current plan, endorsed at the AGM in November 2017, outlines six strategic objectives which will define our programmes between now and July 2022:

• Community Engagement: enabling a growing number of agencies and individuals in all sectors and in all countries to participate in a dynamic and mutually supportive digital preservation community.
• Advocacy: campaigning for a political and institutional climate more responsive and better informed about the digital preservation challenge; raising awareness about the new opportunities that resilient digital assets create.
• Workforce Development: providing opportunities for our members to acquire, develop and retain competent and responsive workforces that are ready to address the challenges of digital preservation.
• Capacity Building: supporting and assuring our members in the delivery and maintenance of high quality and sustainable digital preservation services through knowledge exchange, technology watch, research and development.
• Good Practice and Standards: identifying and developing good practice and standards that make digital preservation achievable, supporting efforts to ensure services are tightly matched to shifting requirements.
• Management and Governance: ensuring the DPC is a sustainable, competent organization focussed on member needs, providing a robust and trusted platform for collaboration within and beyond the Coalition.

This is a very ambitious programme and considerably larger both in scope and impact than could have been imagined in 2002. It is also recognizably consistent with our founding documents however. The benefit of a renewing mandate from a known group is that the DPC can remake itself every few years and in a strange way this act of renewal is also an act of continuity.

The first five of the DPC’s strategic objectives speak for themselves and their practical realisation can be visited through the Coalition's history: the Digital Preservation Awards; the BitList of Digitally Endangered Species; the World Digital Preservation Day; the Leadership Programme; the Digital Preservation Handbook; the ‘Getting Started…’ and ‘Making Progress…’ Training Workshops; the Technology Watch Reports; the Briefing Days and the DPC member support work.  However, the sixth strategic objective is less about content but structure.  So, to understand how the Coalition views and supports collaboration it is worth expanding on this otherwise somewhat opaque theme.

As a member-led agency the DPC is committed to understanding and responding to members: a task which becomes marginally harder with each new member that joins.  The DPC was founded with 16 members in 2002, and by the beginning of 2018 we had reached 80 members. In early years the relatively limited programme could be reported to a relatively small membership with ease.  But as the programme expanded, so the Board grew, and the reports associated with each meeting became somewhat unwieldy.  Therefore, the DPC has had to work hard to innovate new lines of communication that ensure responsiveness and transparency in spite of growth: an annual prospectus of activities, an annual unconference to shape the programme of work, a series of specialist sub-committees to scrutinise the detail of our work, a large representative council to capture member needs, periodic consultations and surveys on areas of activity. It’s hard to be precise but around two thirds of the DPC’s effort is spent communicating with members, and something more than half of that is spent listening or testing proposals from them.

The responsiveness and transparency to members connects back to the financial planning aspect of the DPC.  For the DPC, where there is no partnership there is no sustainability.  That could be a slogan the whole digital preservation community could adopt.

The Nature of the Challenge

I would like to pick out eight themes which characterize or influence the DPC at this point in its development and which I think have to be understood as constraints that configure the Coalition’s work and its processes:

• Broad but shallow;
• Voluntary mandate;
• Dynamic diverse community;
• Dynamic diverse challenge;
• Multiple geographies cultures and languages;
• Time and money;
• Stable commitments; and
• The political context of the UK.

Broad but Shallow

The DPC is open to all.  It was initially inscribed by the borders of the UK and Ireland, and practically focussed entirely on the public sector but we have moved well beyond these constraints in recent years. The breadth of the membership has become one of the DPC’s most significant strengths over the years, generating opportunities for inter-sectoral and inter-professional dialogue which was unimaginable at the outset.  A recent example of this was a workshop on authenticity and significance of digital objects co-ordinated by the DPC and comparing the experience of Tate and the Mechanism for the International Criminal Tribunal (UN MICT). Although radically different in purpose, the intersection of interests between these agencies generated an incredibly useful dialogue which was in turn shared with the rest of the membership.  Breadth like this is a real strength, but it means across the Coalition, collaboration is relatively shallow: deeper collaboration on shared resources does happen between members but typically between relatively small groups working beyond the DPC’s own mandate. So, the DPC’s role might be described as a facilitator for deeper collaboration, helping members to identify themes and the opportunities and amplifying lessons learned:

Voluntary Mandate

Agencies join the DPC because they choose to, not because they have to. There is no policy or regulatory imperative behind membership and nor is there a binding contractual obligation to renew.  To this extent the DPC cannot give instruction to its members, nor would it be wise to do so. Typically, members are already highly structured and professional agencies which exist within complex matrices of accountability audit and control.  They seek support in their work but cannot readily set these aside simply to meet some requirement imposed from the outside.  On the contrary, the DPC typically assumes that members are the best judges of their own contexts.  In some senses, therefore, the Coalition helps members understand their needs whilst not being prescriptive about how or when these are met.  Bluntly, the Coalition has no authority to require anything of anyone therefore depends on leadership beyond authority.

Dynamic, Diverse Community

By whatever measure one uses, the DPC membership is growing in number and in context.  Whereas much of the early running in digital preservation was delivered by memory institutions and ‘big science’, the topic in fact encompasses any agency or individual that presents a use case for digital materials which reaches beyond the limits of technological change media obsolescence or organizational function.  When viewed from this perspective the need for digital preservation includes almost everyone who has saved a file and attempted to use it again at a later date. The surprise is not the growth of agencies like the DPC but why they have not grown more quickly. 

Dynamic, Diverse Challenge

The drivers for digital preservation are typically outwith the control of DPC members: deep-seated economic structures of the IT industry, emerging challenges of information and cyber-security, and changing legal or regulatory requirements.  This fluidity is underpinned by general trends to ever-greater proliferation in data, growing complexity in data and growing expectation of what digital materials can offer.  The boundaries of data and application are unclear, and the economics of data storage and retrieval are unstable.  It is a paradox that preservation solutions are subject to the same obsolescence that they exist to address and are at constant risk of inundation: they are a contingent solution to an emergent problem.  From the perspective of the Coalition which promotes standards and good practice and supports workforce development, established programmes also need constant renewal and new opportunities for collaboration present themselves continuously.

Multiple Geographies, Cultures, Languages, Time Zones  

The DPC’s membership has always been to some extent distributed but that is increasing, introducing significant new challenges in demonstrating member value.  The Qatar National Library, the Internet Archive and the United Nations Office in Geneva have all joined the DPC within a month of this presentation, and the strategic plan speaks of the Coalition making a pivot from being a local agency with a global offering to becoming, by summer 2022, a truly global organization. This expansion in terms of language, culture, geography and time zones has necessitated two significant additions to the DPC’s toolset in the last year.  For several years the DPC has been experimenting with webcasting and recording face to face events, but since Summer 2017 this has become standard practice.  In addition, the DPC has always valued open and constructive dialogue between members, offering a warm welcome to anyone interested in participating in our work.  But recognising that diversity and growth create the conditions for avoidable misunderstanding the DPCs’ values statement has been expanded to a fuller statement about our expectations for inclusion and accessibility. These are early steps, and much more, will ultimately be needed to achieve the purpose of transformation.

Constrained Time, Money, Capacity

The DPC is relatively well funded and carries a small but necessary reserve and contingency to ensure we are stable.  Even so, membership organizations face a moral hazard when they invite comments from members on what they should do: that members ask for more than can be delivered; and perhaps worse that more than can be delivered is promised to them.  Adding new items to the agenda for a membership organization with a finite resource requires at least one of three things to happen: an existing activity is dropped; that members’ subscriptions increase; or that the membership base increases.  The DPC has been fortunate that the last of these has been the dominant trend in recent years and so the strategic plan published in January 2018 includes two new promises: expanding our output on standards and on campaigning.  However, adding capacity to deliver these tasks is demanding.  Direct recruitment of new staff is risky and expensive, temporary contracts can be disruptive, and short secondments can be difficult to manage.  

Stable, Sustainable Commitment

The annual membership round is a curious phenomenon which is not obvious at first.  When first established, the DPC had a rigid membership year and an expectation that the member fund would be completely spent each year.  Therefore, the Coalition would start each financial year with no cash in hand and the urgent necessity of persuading members to renew.  This created a dilemma: it was impossible to draft a programme of activity until it was clear how many members would renew; and in desperate need of a programme so that they might be convinced to do so. Since then a three-year subscription has been introduced which creates a longer planning horizon and allows the publication of an annual plan well ahead of the production of invoices.  This organizational resilience is one of the key lessons of the DPC’s experience as it means members have a stable and sustainable platform for collaboration.

UK’s poor reputation wrt EU

As noted the DPC seeks to be an international agency and benefits from many relationships around the world: although based in the UK there is no UK route to digital preservation.  The DPC has benefitted directly from EU funds, and members have at times subsidised or justified their membership on the basis of EU research funds.  DPC has recruited staff internationally and has learned from participation and informal exchanges over the years.  It is hard to know the damage to the DPC from the UK’s impending departure from the EU, but from almost any perspective it is likely to be damaging and, in more extreme forms, an existential threat.  At the time of writing the ultimate shape of the UK’s relationship with the rest of the world remains unclear and even when agreed there is likely to be some turbulence as these new arrangements take effect. It will require judicious and prompt action to ensure that the DPC is not put at risk in the short term.  If our base in the UK means the DPC is excluded from many of the subtle and informal discussions around research funds, for example, it may be that the DPC needs to establish a presence outside the UK, if not leave altogether.

Renewing the Social infrastructure of Digital Preservation

It is not my purpose in this paper to extol collaboration, merely to observe in the experience of the DPC a platform for collaboration that has been effective and to consider the challenges we face.  But it is worth noting that this collaboration cannot remain static: so, if what I report in 2018 has not changed significantly by the end of 2022 then the digital preservation community will be poorly served.  In some sense collaboration is a critical proposition for the digital preservation community.  The community has long recognised the need to renew the technical infrastructure that supports access and exploitation of digital materials, in the same way it is essential to renew the social and intellectual infrastructure of the community.

The digital preservation community, which as noted, is dynamic and growing, is in some senses formed around a series of standards.  While tools and approaches may come and go, the standards are a codification of good practice in an activity in which success is always deferred.  I have argued before that standards should matter to us in particular:

‘Facing a large and complicated task, standards offer a beginners’ guide to requirements; working on behalf of heterogenous and risk-averse agencies standards enable meaningful comparison between incommensurate processes. Standards hold out the promise of interoperability and a keystone of succession planning. They are our best working solution to the paradox of obsolescence and a technology that cannot guarantee itself.’

But the problem which we have so far addressed only poorly is the extent to which the technology paradox is infectious: that we can ask technology to abide by standards, but standards become obsolete too.  The only practical defence against the obsolescence of standards is the community: so, when asked what to do about it we can only answer ‘use standards’ for as long as the community offers a warranty to support them. 

That task is hard enough in a stable population: but placed in the context of a rapidly growing and diverse community we have an urgent task to renew and upgrade the social and intellectual infrastructure from which standards emerge.  That’s an urgent argument for collaboration, and an easy invitation to participate in the future of the DPC.