Difference between revisions of "Tracker"

From Archiveteam
Jump to navigation Jump to search
Line 1: Line 1:
The [http://tracker.archiveteam.org/ Tracker] is the center-pivot of Archiveteam's distributed archiving efforts.  The [[ArchiveTeam Warrior|Warrior]] is the yang to the Tracker's yin. The warriors get the list of current projects from [http://warriorhq.archiveteam.org/].
= General Overview =


For most projects, the [http://tracker.archiveteam.org/ tracker] provides seed points such as usernames, and keeps track of who retrieved what resources and where they're stashed.
The [https://github.com/ArchiveTeam/universal-tracker Tracker] software is the center-pivot of Archiveteam's distributed archiving efforts. It hands out items to be downloaded and keeps track of what is completed. Items can be usernames, subdomains, full urls, basically any unit we can use to break the site into manageable chunks. The progress of each project can be viewed via the leader board interface on http://tracker.archiveteam.org .


= Hardware =
The [[ArchiveTeam Warrior|Warrior]] is the yang to the Tracker's yin. The warriors get the list of current projects from the project file on http://warriorhq.archiveteam.org/ .
 
This is a sample from the project file:
 
<pre>
{
    "name": "streetfiles",
    "title": "Streetfiles",
    "description": "Streetfiles is closing April, 30th, 2013.",
    "repository": "https://github.com/ArchiveTeam/streetfiles-grab.git",
    "logo": "http://archiveteam.org/images/7/7b/Streetfiles-logo.png",
    "marker_html": "<a href='http://tracker.archiveteam.org/streetfiles/'><img src='http://archiveteam.org/images/7/7b/Streetfiles-logo.png' alt='Streetfiles' width='235' height='50' /></a>",
    "deadline": "2013-04-30T23:59:59Z",
    "host": "streetfiles.org",
    "leaderboard": "http://tracker.archiveteam.org/streetfiles/",
    "lat_lng": [
        51,
        9
    ]
},
</pre>
 
It shows where to get the grab code and other project information.
 
== Hardware ==
The tracker runs on a [http://www.archiveteam.org/index.php?title=Clown_hosting#linode Linode 1 GB] instance operated by [[User:Chronomex|chronomex]].  The system is tracked by [http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/index.html Munin].
The tracker runs on a [http://www.archiveteam.org/index.php?title=Clown_hosting#linode Linode 1 GB] instance operated by [[User:Chronomex|chronomex]].  The system is tracked by [http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/index.html Munin].


= Software =
== Software ==
[[User:Alard|alard]] takes care of making the tracker run happily.  The custom software is a python based HTTP application that sends and receives JSON payloads and uses Redis for the data store; all on top of Debian Linux. The code is hosted [https://github.com/ArchiveTeam/universal-tracker on GitHub].
[[User:Alard|alard]] takes care of making the tracker run happily.  The custom software is a python based HTTP application that sends and receives JSON payloads and uses Redis for the data store; all on top of Debian Linux. The code is hosted [https://github.com/ArchiveTeam/universal-tracker on GitHub].


The warriors get the list of current projects from the Warrior HQ. The code is in the [https://github.com/ArchiveTeam/warrior-hq warrior-hq] repository.
The warriors get the list of current projects from the Warrior HQ. The code is in the [https://github.com/ArchiveTeam/warrior-hq warrior-hq] repository.

Revision as of 18:54, 1 May 2013

General Overview

The Tracker software is the center-pivot of Archiveteam's distributed archiving efforts. It hands out items to be downloaded and keeps track of what is completed. Items can be usernames, subdomains, full urls, basically any unit we can use to break the site into manageable chunks. The progress of each project can be viewed via the leader board interface on http://tracker.archiveteam.org .

The Warrior is the yang to the Tracker's yin. The warriors get the list of current projects from the project file on http://warriorhq.archiveteam.org/ .

This is a sample from the project file:

{
    "name": "streetfiles",
    "title": "Streetfiles",
    "description": "Streetfiles is closing April, 30th, 2013.",
    "repository": "https://github.com/ArchiveTeam/streetfiles-grab.git",
    "logo": "http://archiveteam.org/images/7/7b/Streetfiles-logo.png",
    "marker_html": "<a href='http://tracker.archiveteam.org/streetfiles/'><img src='http://archiveteam.org/images/7/7b/Streetfiles-logo.png' alt='Streetfiles' width='235' height='50' /></a>",
    "deadline": "2013-04-30T23:59:59Z",
    "host": "streetfiles.org",
    "leaderboard": "http://tracker.archiveteam.org/streetfiles/",
    "lat_lng": [
        51,
        9
    ]
},

It shows where to get the grab code and other project information.

Hardware

The tracker runs on a Linode 1 GB instance operated by chronomex. The system is tracked by Munin.

Software

alard takes care of making the tracker run happily. The custom software is a python based HTTP application that sends and receives JSON payloads and uses Redis for the data store; all on top of Debian Linux. The code is hosted on GitHub.

The warriors get the list of current projects from the Warrior HQ. The code is in the warrior-hq repository.