In 2014 the Irish Film Institute (IFI) Irish Film Archive faced the need to preserve digital formats alongside its traditional analogue collections. The switch to digital formats within the film and broadcasting sector in Ireland was sudden, encouraged by a government sponsored scheme enabling cinemas to change from analogue to digital exhibition. This left the IFI Archive with no option but to take in digital material as part of our preservation agreements with the two main funders of moving image production in Ireland, the Broadcasting Authority of Ireland and the Irish Film Board.
After securing funding from the State to upgrade our digital infrastructure so that we could deal with the processing and storage of high resolution files we realised that much of the software and equipment we had purchased was made for the broadcast sector and had not been created with long -term preservation in mind. Frustrated when a vendor issue wasn’t being addressed, one of our team Kieran O’Leary learned to write python and created a script that solved the problem that was restricting our development. This led to the development of a suite of tools which not only can be used by the IFI into the future but have proven beneficial to a range of international organisations. These scripts are used by the Irish Film Archive staff’s daily for activities such as fixity, movement and transcoding. Adopting a microservices architecture has enabled the archive to create discrete modular services and applications that can work independently or as part of a larger workflow structure. Being able to create bespoke open source tools has transformed the way the Archive works, automating many of the preservation and quality control tasks within our digital work flows.
The IFIscripts were born out of necessity, IFI Archive staff were both restricted by funding limitations and influenced and encouraged by the work of other archives that had adopted Open Source Software solutions. These organisations shared their tools freely which in turn encouraged the IFI to also share our in-house tools. Our engagement with a dynamic community of likeminded archivists and developers has been fundamental, where possible, contributed to the Open Source projects that it uses, which leads to a sustainable, collaborative engagement with software development and use.
The microservices architecture the IFI has developed furthers the principles of Flexibility, Scalability, Sustainability that were outlined as the core of our Digital Preservation and Access Strategy, https://ifi.ie/2015/06/digital-preservation-and-access-strategy/ which was published in 2014. We hope that our experience can serve as a positive, innovative, cost effective model for other archives of our size.
Application of IFIscripts
The Loopline Conservation Project is the first end to end project we have undertaken using IFIscripts and finishes on 29th June 2018. This is a large scale conservation, cataloguing and preservation project focusing on the output of Loopline one of Ireland’s most important independent production companies, run by filmmaker Se Merry Doyle. The collection provides an unparalleled record of key areas of Irish history and arts from the 1990s to the present day, documenting a transition in Irish society pre- Celtic tiger and post economic crash. In addition to material chronicling the rapid social and economic change in Ireland during the late 20th century it includes interviews with international cultural figures such as Margaret Atwell, Martin Scorsese & Maureen O’Hara.
In 2017 Loopline closed its studios and transferred its holdings to the IFI Irish Film Archive. The aim of this project was to assess, catalogue, digitise, preserve, and make accessible the Loopline Collection begins in 1992 and contains over 900 hours of material. The collection comprises broadcast tapes, film and most importantly a significant amount of born digital content held on hard drives and camera cards representing the first large scale digital preservation project attempted by IFI Archive staff. Please see letter outlining the importance of this collection by leading Irish poet and documentary maker Theo Dorgan https://en.wikipedia.org/wiki/Theo_Dorgan
Born digital files
Within the Loopline project we encountered project born digital material from a production source which is more complex than the type of digital files we have received. This was a first for us as we had never worked with this type of material before and necessitated the writing of a large number of custom computer scripts in house.
The IFI Archive staff devised a plan to assess arrange describe and preserve born digital material from Loopline hard drives. We discovered 390,000 files across the hard drives which was a huge amount of material to process. To deal with this vast quantity of material IFI Archive staff wrote a bespoke computer programme that allowed us to create inventories to decide how to arrange the material and what to keep. This allowed us to identify where there were multiples of the same file, partial files and fragments not worth keeping. As it was impossible to watch everything we had to
devise an automated way to appraise. Staff researched how we could carry out arrangements on these complex files within the context of OAIS and Spectrum and existing IFI processes. We drew up guidelines that allow us to identify corrupt files and dupes which are eliminated. A verification script was written to allow us to decide what to keep. Findings written up for future projects.
Arrangement and description were also a challenge as digital files ranged from individual objects to object comprised of multiple files held in complex folder structures that are all related. For example camera cards. The key pieces of software were sipcreator.py, which allowed us to package both simple and complex objects into consistent packages, and concat.py, which created simplified, normalised, versions of complex objects. We then wrote further tools that completed the ingest procedure, and they automated the AIP finalisation, technical cataloguing, accessioning database ingest and movement to our preservation storage system. It also creates a low res proxy file for cataloguing. 3 copies of archival packages are created in accordance with best practice.
These scripts not only help us with future work but also have made public and shared with other archives around the world. We are also in the process of publishing all of the detailed documentation we created around our processes, tools and workflows as part of this project.