From Archiveteam
Revision as of 20:05, 16 June 2015 by Jscott (talk | contribs) (Git-annex won, for now)
Jump to: navigation, search
Backing up the Internet Archive

The wonder of the Internet Archive's petabytes of stored data is that they're a world treasure, providing access to a mass of information and stored culture, gathered from decades of history (in some case, centuries), and available in a (relatively) easy-to-find fashion. And as media and popular websites begin to cover the Archive's mission in earnest, the audience is growing notably.

In each wave of interest, questions come forward out of the accolades:

  • Why is the Internet Archive only in one (actually, limited) physical space?
  • What is the disaster recovery plan?
  • What steps are being taken to ensure integrity and access?

Some of the answers are essays in themselves, but regarding the protection of the Internet Archive's storehouses of data, a project has been launched to add some real-world information to the theoretical consideration of data protection.

The INTERNETARCHIVE.BAK project (also known as IA.BAK or IABAK) is a combined experiment and research project to back up the Internet Archive's data stores, utilizing zero infrastructure of the Archive itself (save for bandwidth used in download) and, along the way, gain real-world knowledge of what issues and considerations are involved with such a project. Started in April of 2015, the project already has dozens of contributors and partners, and has resulted in a fairly robust environment backing up terabytes of the Archive in multiple locations around the world.

To see the current status of the project's data storage, click here.


The project's main goals include:

  • Maintaining multiple, verified, geographically-disparate copies of Internet Archive data
  • No use of Internet Archive infrastructure, that is, machines or drives (bandwidth just for download)
  • Keep at least 3 external copies in three different physical locations from the Archive.
  • Conduct bi-weekly verification that the data is secure in the external locations.
  • Expire or remove data that has not been verified after 30 days, replacing it with functional space.

The project's secondary goals include:

  • Add ease of use to the end-user clients so the maximum amount of drive space contributors can participate.
  • Offer optional peer-to-peer data provision for both the main site's items and to speed synchronization.
  • Issue useful conclusions as to how the project is conducted.
  • Ultimately provide encrypted versions of some data so that internal/system data can be backed up too.

The project rests on several assumptions:

  • Given an opportunity and good feedback, there are petabytes of volunteer disk space available
  • The Internet Archive will gain more awareness by the general public by this project
  • A non-homogenous environment for the data makes the data that much stronger and disaster-resistant
  • The project is an educational workshop as well as a good-faith effort, providing hard answers to theoretical questions.


To understand what might be involved in backing up a large set of data, information about that dataset needed to be researched.

The project, known as the Internet Archive Census resulted in some solid numbers, as well as a set of facts that helped nail down the scope of the project. They include:

  • The Internet Archive has roughly 21 petabytes of unique data at this juncture. (It grows daily.)
  • Some is not available for direct download by the general public, so IA.BAK is not addressing it (yet).
  • Some data is, by its nature, more "critical" or "important" than others. Unique acquired data versus, say, universally available datasets or mirrors of prominent and accessible-from-many-locations music videos.
  • A lot of it, the vast, vast majority, is extremely static, and meant to live forever.
  • A lot of it are "derives", and marked as such - files that were derived from other files.

The census classified the total store of public-accessible, unique data as very roughly 14 petabytes. Of that data, there is some level of redundancy and there are collections that might be called "crown jewels" or treasures (the Prelinger Archives, the Bali Culture collection, collections of scanned books from partner libraries) while others are more "nice to have" (universally accessible podcasts or video blogs, multiple revisions of the same website backup).


There is now a channel, #internetarchive.bak, on EFNet, for discussion about the project as well as bot-driven updates about code changes and new contributors. It is occasionally very active and mostly quiet in between new issues being handled. It is not required to use as a contributor of disk space, but is an excellent resource for questions.

After multiple discussions, the back-end for this phase of this project is using the git-annex suite of tools. Reasons include the maturity of the project and the willing contribution of time by the git-annex code designer and maintainer, Joey Hess (also a co-founder of Archive Team). Other possible software suites are listed at the end of this Wiki entry and might play a part in future revisions of the project.

To read the git-annex overview and proposal, read here: INTERNETARCHIVE.BAK/git-annex_implementation

To read the git-annex implementation documentation, read it here: [1]


As space is relatively limited to provide to IA.BAK, the project has had a second task line of choosing which public-facing data at the Internet Archive to push into IA.BAK. Generally, the group is looking for:

  • Data unique enough that it is predominantly at the Internet Archive
  • Data that does not flood the capacity of the project, i.e. hundreds of Terabytes
  • Data that does not generally limit which countries can host it
  • Data that is generally smaller in size, yet still unique

There is now a page to nominate sets of data at the Internet Archive that should be added to future shards:

The project status page has methods for seeing what collections are now in the project, although it lacks simple search (yet).


More people can add concerns, but my main one is preparing against Bad Actors, where someone might mess with their copy of the sector of The Drive that they have. Protections and checks will have to be put in to make sure the given backups are in good shape. There will always be continued risk, however, and hence the "high availability" items where there will be lots of copies to "vote". NOTE: Lots of thoughts on bad actors are on the discussion page.

There is also a thought about recovery - we want to be able to have the data pulled back, and that will mean a recovery system of some sort.


We are soliciting proposals for ways to implement this using your favorite distributed system. So far, we have:

Started! Try this one!

See Also

Alternate proposed software suites for this project besides git-annex:

  • Valhalla - Valhalla was a discussion about the "ultimate home" for uploaded Archive Team data, be it the Internet Archive, elsewhere, or both. IA.BAK is an example of a potential shared home, although Archive Team projects have not generally been backed up using IA.BAK, yet.