The WARC Ecosystem

From Archiveteam
Jump to navigation Jump to search

Everything about the WARC format and the tools that support it.

Information

Tools

name

 1 license
 2 programming language
 3 test suite
 4 has documentation
 5 # of authors
 6 description

wget v1.14+

 * GPL v3+
 * C
 * Has a test suite but does not test any warc functionality
 * Man pages, website, blog posts all over the net
 * 2+ according to the changelog
 * A non-interactive network downloader. wget also generates duplicate record ids in warc files.

More information about flags can be found on the Wget with WARC output page.

warc python library

 * GPL v2
 * Python
 * looks to have a test suite - https://github.com/internetarchive/warc/blob/master/warc/tests/test_warc.py
 * A readme with examples online at http://warc.readthedocs.org/en/latest/
 * 3 commiters on github
 * library to work with WARC files

WarcProxy

 * BSD
 * python
 * NO TEST SUITE
 * A readme file.
 * 1 author
 * a simple HTTP proxy that saves all HTTP traffic to a file

warc-tools

 * MIT License
 * python 2.6
 * NO TEST SUITE
 * A readme file
 * 4 commiters
 * warc validator, dump, search, index, convert arc to warc

The previous version can be found at https://code.google.com/p/warc-tools/

http://code.hanzoarchives.com/warc-tools/src/6e1d36297688/hanzo/warcextract.py

WARC viewer

 * no license information
 * python
 * NO TEST SUITE
 * A readme file
 * 1 author
 * WARC viewer for browsing the contents of a WARC file.
 - needs a firefox addon installed to work

Megawarc

 * no license information
 * python
 * NO TEST SUITE
 * A readme file
 * 1 author
 * Merge many small warcs into a large one

Checks if WARC files can be un-gzipped before adding them to the megawarc. Does not check anything else.

warc to zip

 * no license information
 * python
 * NO TEST SUITE
 * A readme file
 * 1 author
 * An HTTP-based warc-to-zip converter

warcat

 * GPL v3
 * Python 3
 * yes
 * A readme file.
 * 1 author
 * warcat concat, extract, list, pass, split, verify warc files

Install: pip-3 install warcat
Run: python3 -m warcat verify mysite.warc.gz

https://github.com/internetarchive/ia-web-commons

https://github.com/internetarchive/ia-hadoop-tools

Archive Team megawarc factory

 * no license information
 * Bash shell scripting
 * NO TEST SUITE
 * A readme file.
 * 1 author
 * Generates 50gb warc files from existing warc files

Uploads to archive.org

CDX Writer

 * no license information
 * python
 * Has a test suite
 * A readme file.
 * 1 author
 * Create CDX index files from WARC files.

Heritrix

 * Apache v2.0
 * java
 * Has a test suite
 * javadoc, website
 * many authors
 * Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Heritrix-Cassandra A library for writing Heritrix 3 output directly to Cassandra as records.

DeDuplicator (Heritrix add-on) The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.

python-heritrix A simple wrapper around the Heritrix 3.x API. Developed in April 2012 against Heritrix 3.1.0 at GWU Libraries in Washington, DC, USA.

Chrome/Chromium plugin WARCreate

 * no license information
 * javascript
 * ???
 * none
 * 1 author
 * WARCreate is a Google Chrome extension that allows a user to create a Web ARChive (WARC) file from any browseable webpage.

Deprecated

The WARC format

  • A .warc file is usually a group of one or more WARC records.
  • The first record usually describes the records to follow.
  • compression is optional
  • each record is compressed via gzip. A gzip file supports multiple "members"
  • compressed warcs end in .warc.gz
  • According to the guidelines warc files should top out at 1gb


WARC record

  • header
  • content block
  • two newlines

WARC record header

The beginning of a WARC record, consisting of one first line declaring the record to be in the WARC format with a given version number, followed by lines of named fields up to a blank line. The WARC record header format largely follows the tradition of HTTP/1.1 [RFC2616] and [RFC2822] headers, with one major exception, allowing UTF-8 [RFC3629].


Example of a 'request' record header:

 WARC/1.0
 WARC-Type: request
 WARC-Target-URI: http://xbox.gamespy.com/
 Content-Type: application/http;msgtype=request
 WARC-Date: 2013-04-02T16:12:40Z
 WARC-Record-ID: <urn:uuid:08d9edb9-0ab8-4352-ba56-6cbbd590f34f>
 WARC-IP-Address: 213.248.112.146
 WARC-Warcinfo-ID: <urn:uuid:2b6ad3f1-efab-4e37-8faa-fc8ad112692f>
 WARC-Block-Digest: sha1:T6PJSZTTP7HBNA6OFZACXAFK25GGLVT4
 Content-Length: 150

WARC named fields

  • A set of elements consisting of a name, a colon, and a value, with long values continued on indented lines.
  • Named fields may appear in any order.
  • Field values may contain any UTF-8 character.
  • The 'encoded-word' mechanism of [RFC2047] may also be used when writing WARC fields and shall also be understood by WARC reading software.

WARC content block

Part (zero or more octets) of a WARC record that follows the header and that forms the main body of a WARC record.


CDX File Format