Difference between revisions of "Working with ARCHIVE.ORG"

From Archiveteam
Jump to navigation Jump to search
m (Reverted edits by Megalanya1 (talk) to last revision by Jscott)
 
(8 intermediate revisions by 7 users not shown)
Line 1: Line 1:
[[Image:Archivesneedlove.jpg.jpg]]
[[Image:Archivesneedlove.jpg.jpg]]


The [[Internet Archive]] has enormous resources at its disposal, and shares many parallel goals with Archive Team. There are some advantages to making as much of Archive Team's data saves available to ARCHIVE.ORG, especially as regards the [http://wayback.archive.org/web/ Wayback Machine], a mechanism to be able to browse older web materials going back years. Where possible, Archive Team works with ARCHIVE.ORG, and get the love going.


The Internet Archive has enormous resources at its disposal, and shares many parallel goals as Archive Team. There's some advantages to making as much of Archive Team's data saves available to ARCHIVE.ORG, especially as regards the [http://wayback.archive.org/web/ Wayback Machine], a mechanism to be able to browse older web materials going back years. Where possible, Archive Team should try to work with ARCHIVE.ORG, and get the love going.
__TOC__
__TOC__
== Headers and Logs ==
== Headers and Logs ==


The most striking difference between Archive Team and ARCHIVE.ORG is that while Archiveteam traditionally considers header and logging information to be a nice hat trick if you have the time, ARCHIVE.ORG absolutely needs it to import into their Wayback Machine. With WGET, the best way to do this is to save off the log files, and use ''--save-headers'' to include the headers in the file.
The most striking difference between Archive Team and ARCHIVE.ORG is that while Archiveteam traditionally considers header and logging information to be a nice hat trick if you have the time, ARCHIVE.ORG absolutely needs it to import into their Wayback Machine.


(More research needs to be done to ensure these options are in scripts.)
How? See [[The WARC Ecosystem]].


Unfortunately, this produces files that are not, initially, usable for just dumping into a new location, so there is a chance either two copies should be made, or a script written that strips the headers out of the files for later use.
== The Archive.org "Archiveteam" Collection ==
 
== Stripping the Headers ==
As an example, let's download this page:
<pre>wget -O headers.raw.html --save-headers "http://www.archiveteam.org/index.php?title=Working_with_ARCHIVE.ORG"</pre>
 
If we want to strip the headers, then we run:
<pre>cat headers.raw.html|grep -vP ".*?\r" > noheaders.raw.html</pre>


Note that this requires a grep with PCRE built in (most greps come with this compiled in)
Archive.org currently has an [http://www.archive.org/details/archiveteam Archive Team Collection], which consists of site-grabs, older archives, and various rips/mass downloads from websites and Archive Team activities. Right now, that collection is primarily administered by Jason Scott &mdash; sending him e-mail at jason@textfiles.com with something you think that collection should have is probably the best way to go about things.


== The Archive.org "Archiveteam" Collection ==
[[Category:Tools]]


Archive.org currently has an [http://www.archive.org/details/archiveteam Archive Team Collection], which consists of site-grabs, older archives, and various rips/mass downloads from websites and Archive Team activities. Right now, that collection is primarily administered by Jason Scott - sending him e-mail at jason@textfiles.com with something you think that collection should have is probably the best way to go about things.
{{Navigation box}}

Latest revision as of 16:13, 17 January 2017

Archivesneedlove.jpg.jpg

The Internet Archive has enormous resources at its disposal, and shares many parallel goals with Archive Team. There are some advantages to making as much of Archive Team's data saves available to ARCHIVE.ORG, especially as regards the Wayback Machine, a mechanism to be able to browse older web materials going back years. Where possible, Archive Team works with ARCHIVE.ORG, and get the love going.

Headers and Logs

The most striking difference between Archive Team and ARCHIVE.ORG is that while Archiveteam traditionally considers header and logging information to be a nice hat trick if you have the time, ARCHIVE.ORG absolutely needs it to import into their Wayback Machine.

How? See The WARC Ecosystem.

The Archive.org "Archiveteam" Collection

Archive.org currently has an Archive Team Collection, which consists of site-grabs, older archives, and various rips/mass downloads from websites and Archive Team activities. Right now, that collection is primarily administered by Jason Scott — sending him e-mail at jason@textfiles.com with something you think that collection should have is probably the best way to go about things.


Retrieved from "https://wiki.archiveteam.org/index.php?title=Working_with_ARCHIVE.ORG&oldid=28631"