Amy Rudersdorf is a Senior Consultant with AVP.
Introduction
2020 hasn’t been the year most of us expected. At the top of all the stuff that this year has thrown our way is the global pandemic. For some organizations it has also meant reductions in resources and staffing. This has left many cash-strapped organizations with tighter budgets than ever before, even as the costs of technology and human resources continue to rise. At the same time, many organizations are struggling to get a handle on the mountains of digitized and born-digital content in their collections to ensure it is safe now, and managed over time to ensure it is accessible and renderable into the future.
So what’s a digital preservationist do when it comes to selecting technology for their program? Is it cheaper, better, and faster to build from scratch or buy an out-of-the-box (OOTB) solution? This post describes the benefits and drawbacks of do-it-yourself (“DIY”) digital preservation that may help decide in which direction to go.
Before we get into it, though, I want to be clear in saying technology ≠ digital preservation. It is one of three important guiding principles; the other two are “organization” and “resources.”
Figure 1. Organization, technology, and resources are all equally important to shoring up your digital preservation program.
While you can’t talk about a program without mentioning all three, the remaining paragraphs in this blog post focus mainly on supporting pre-ingest, ingest, and management tools and processes necessary to do the dirty work of digital preservation: fixity, redundant storage, characterization, migration, etc. Governance, legal, financial, strategy, and policy are all topics for another day.
When we talk about DIY digital preservation, we are really talking about taking a “microservices” approach to the management of your content. This means that the different tasks required (fixity, storage, characterization, migration, etc.) can/may be performed with standalone tools or activities. All of these individual microservices are combined into a single workflow, through which the content is run in either a manual or automated fashion. The workflows might look something like Figure 2, below, where “Pre-Ingest Functions” are linear, and are likely only applied for each — let’s call them files — once, and “Active Storage,” where the workflows are repeated over time. The arrow with the dotted line indicates that either migration or emulation may also take place one or more times.
Figure 2. Sample workflow for pre-ingest functions and active storage.
All files will require most or all of these processes in a fully implemented digital preservation program. We don’t have space to define each, but there are loads of resources available on the open web you can turn to to learn more about them. I recommend you start on the Digital Preservation Coalition’s website, and go from there.
OK, now that we know what needs to be done, let’s talk about how to do it. I’m approaching this topic from a rather black-and-white perspective (either all DIY or all OOTB), but there are definitely shades of gray where some microservices are performed using standalone tools, and others are performed by an OOTB solution. Let’s start by talking about the benefits of doing it yourself.
Benefits of DIY
Here are a few of the benefits to building your own DIY microservices workflow for digital preservation. I’m sure there are others — please leave a comment below with your thoughts!
Total control
If you build your own microservices workflow, you are, ultimately, in control of ensuring that what you need to get done is part of the workflow, and the stuff you don’t need to worry about, isn’t.
No black boxes
A “black box” is any complex piece of equipment where the inner workings are mysterious to or unknowable by a user. Another way to describe this type of technology is “opaque.” If you are constructing the workflow yourself using a series of microservices, there will be very little mystery as to how your files are processed. In fact, as the builder (and/or manager) of the workflow, you’ll need to have a pretty deep understanding of these technologies! OOTB systems may be a bit more opaque — they perform the processes you need, but you may not know how or how to adjust them.
No long-term contracts
Many of the microservices tools you would employ in your digital preservation workflow would likely be open source (e.g., Exactly, PRONOM) or low-cost subscriptions (e.g., Fixity Pro). It is unlikely you would be required to sign expensive, multi-year contracts. (If you do, talk to me. Something might not be quite right.)
Open source tools
As mentioned above, if you are building your own workflow, there are loads of open-source tools available to you. Some are command-line driven, others are programs with GUI interfaces. Keep in mind — open source doesn’t necessarily mean free, but generally these tools don’t cost much to purchase or to buy a subscription.
Just the essentials
If you build the workflow, you only need to incorporate what you need. No extra overhead or bloated systems, here, folks!
Costs are lower. . . ?
The financial burden for the microservices tools in your DIY workflow is likely going to be lower than the license, subscription, or other annual costs for out-of-the-box digital preservation management software. This can be misleading, however, as I’ll talk about later. (Sneak peek: you have to pay for folks to develop, implement, and/or manage those DIY workflow tools. Also, storage ain’t free.)
Drawbacks of DIY
The list of benefits above is pretty long, but definitely keep in mind the drawbacks are equally myriad. Ultimately, it comes down to what makes sense for your organization. Here are three big drawbacks.
Gotta know your tech
As I mentioned before, some of the DIY microservices tools you’ll need to run may be command-line driven or have very little associated documentation. While this may be easy breezy for some, this can pose a real challenge for organizations with little to no internal technology knowledge or support. Some tools may have intuitive GUI interfaces, but you’ll still need to have a handle on what each tool does and what you want to get out of it. It should be mentioned, though, that OOTB systems will perform most of the microservices in Figure 2 behind the scenes, so you don’t have to worry about them. But, you’ll still need to learn the system, ingest protocols, and more.
Costs are higher. . . ?
Things get a little murky here, as many costs are dependent on each organization’s unique set up. Do you have technology staff in place who can help you out (i.e. can you DIY. . .yourself)? Maybe DIY will be cheaper, then, than an OOTB solution. Will you need to hire staff to run your DIY shop? Human resources can be expensive! But the reality is, you probably need to hire or dedicate staff to an OOTB system, too. So maybe human resources costs zero out in this scenario? It’s hard to say. Just keep in mind that you are not only paying for systems; you are paying for people, systems, and storage and you should budget accordingly.
Don’t be a DIY fail!
Without sufficient knowledge and support, you could end up with a cobbled together and unstable workflow. Also (and this is really important), systems built in-house can end up being dependent on a single person for support. If that person leaves your organization, the whole house of cards could come tumbling down. If that’s a possibility, DON’T DIY! Consider putting in place an OOTB digital preservation management system that will be the responsibility of that company to maintain. This’ll give you an opportunity to focus on the work, rather than duct taping an unreliable system together.
Final thoughts
As I mentioned at the start, there are a lot of variables to consider when choosing between DIY or OOTB (or a combination!). Step one for any organization is to take a serious look at what your specific needs are, what you are able to support in-house, and what you can gain from the support you might get from the purchase of a full-feature OOTB system. Document those requirements, get a handle on the financial and resource implications, and decide what you want to do — be a DIY tinkerer or manage an OOTB system. In either case, good luck!
Do you have thoughts or suggestions for additions to the lists I’ve provided? I’d love to hear them — reach out to me at amy@weareavp.com or twitter at @weareavp.