Difference between revisions of "Tracker"

From Archiveteam
Jump to navigation Jump to search
(→‎People: removed out-of-date table that shouldn't be on this page anyways, updated list of active users to contact for tracker issues)
(38 intermediate revisions by 10 users not shown)
Line 1: Line 1:
The [http://tracker.archiveteam.org/ Tracker] is the center-pivot of Archiveteam's distributed archiving efforts.  The [[ArchiveTeam Warrior|Warrior]] is the yang to the Tracker's yin. The warriors get the list of current projects from [http://warriorhq.archiveteam.org/].
== General Overview ==


For most projects, the [http://tracker.archiveteam.org/ tracker] provides seed points such as usernames, and keeps track of who retrieved what resources and where they're stashed.
[[File:Tracker_test_project_overview_screenshot.png|right|thumb|Project admin overview]]


= Hardware =
The [https://github.com/ArchiveTeam/universal-tracker Tracker] software is the center-pivot of Archiveteam's distributed archiving efforts. It hands out items to be downloaded and keeps track of what is completed. Items can be usernames, subdomains, full urls, basically any unit we can use to break the site into manageable chunks. The progress of each project can be viewed via the leader board interface on http://tracker.archiveteam.org .
The tracker runs on a [http://www.archiveteam.org/index.php?title=Clown_hosting#linode Linode 1024] instance operated by [[User:Chronomex|chronomex]]. The system is tracked by [http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/index.html Munin].


= Software =
[[File:Xanga_leaderboard.png|right|thumb|A leaderboard]]
[[User:Alard|alard]] takes care of making the tracker run happily.  The custom software is a python based HTTP application that sends and receives JSON payloads and uses Redis for the data store; all on top of Debian Linux. The code is hosted [https://github.com/ArchiveTeam/universal-tracker on GitHub].


The warriors get the list of current projects from the Warrior HQ. The code is in the [https://github.com/ArchiveTeam/warrior-hq warrior-hq] repository.
The [[ArchiveTeam Warrior|Warrior]] is the yang to the Tracker's yin. The warriors get the list of current projects from the project file on http://warriorhq.archiveteam.org/ .
 
__TOC__
 
== API ==
 
This is a sample project snippet from the projects.json file (line breaks included for readability):
 
<pre>
{
    "name": "streetfiles",
    "title": "Streetfiles",
    "description": "Streetfiles is closing April, 30th, 2013.",
    "repository": "https://github.com/ArchiveTeam/streetfiles-grab.git",
    "logo": "http://archiveteam.org/images/7/7b/Streetfiles-logo.png",
    "marker_html":
        "<a href='http://tracker.archiveteam.org/streetfiles/'>
        <img src='http://archiveteam.org/images/7/7b/Streetfiles-logo.png'
        alt='Streetfiles' width='235' height='50' /></a>",
    "deadline": "2013-04-30T23:59:59Z",
    "host": "streetfiles.org",
    "leaderboard": "http://tracker.archiveteam.org/streetfiles/",
    "lat_lng": [
        51,
        9
    ]
},
</pre>
 
It shows where to get the grab code and other project information.
 
Here is an example root of the file:
 
<pre>
{
    "auto_project": "projectslug",
    "broadcast_message": "<p>This message is shown only
        in the warrior VM web UI at time of writing.</p>",
    "tracker_banner_html": "This is shown on the tracker
        front page. <em>Wow!</em>",
    "warrior": {"seesaw_version":"0.7.0"},
    "projects": [],
}
</pre>
 
== Hardware ==
The tracker runs on a [http://www.archiveteam.org/index.php?title=Clown_hosting#linode Linode] instance operated by [[User:Chronomex|chronomex]].
 
== Monitoring ==
 
http://tracker.archiveteam.org has a Munin instance located at http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/.
 
== Software used: ==
 
* [https://github.com/ArchiveTeam/universal-tracker Universal Tracker] is a Ruby HTTP application that sends and receives JSON payloads and uses Redis for the data store.
* Redis A memory based key value store
* [http://debian.org/ Debian] is the Linux distribution the stack is built upon.
* [https://github.com/ArchiveTeam/warrior-hq warrior-hq] a small Sinatra web app to manage the Warriors and display the geo-location world map.
 
You can also [[Tracker_Setup|set up your own tracker]].
 
== People ==
In case of tracker maintenance emergency, please contact these active users (as of 2014-08-29). Tracker emergencies only!:
 
* [[user:chfoo]]
* arkiver
 
{{Navigation box}}

Revision as of 13:34, 25 April 2019

General Overview

Project admin overview

The Tracker software is the center-pivot of Archiveteam's distributed archiving efforts. It hands out items to be downloaded and keeps track of what is completed. Items can be usernames, subdomains, full urls, basically any unit we can use to break the site into manageable chunks. The progress of each project can be viewed via the leader board interface on http://tracker.archiveteam.org .

A leaderboard

The Warrior is the yang to the Tracker's yin. The warriors get the list of current projects from the project file on http://warriorhq.archiveteam.org/ .

API

This is a sample project snippet from the projects.json file (line breaks included for readability):

{
    "name": "streetfiles",
    "title": "Streetfiles",
    "description": "Streetfiles is closing April, 30th, 2013.",
    "repository": "https://github.com/ArchiveTeam/streetfiles-grab.git",
    "logo": "http://archiveteam.org/images/7/7b/Streetfiles-logo.png",
    "marker_html": 
        "<a href='http://tracker.archiveteam.org/streetfiles/'>
        <img src='http://archiveteam.org/images/7/7b/Streetfiles-logo.png'
        alt='Streetfiles' width='235' height='50' /></a>",
    "deadline": "2013-04-30T23:59:59Z",
    "host": "streetfiles.org",
    "leaderboard": "http://tracker.archiveteam.org/streetfiles/",
    "lat_lng": [
        51,
        9
    ]
},

It shows where to get the grab code and other project information.

Here is an example root of the file:

{
    "auto_project": "projectslug",
    "broadcast_message": "<p>This message is shown only 
        in the warrior VM web UI at time of writing.</p>",
    "tracker_banner_html": "This is shown on the tracker 
        front page. <em>Wow!</em>",
    "warrior": {"seesaw_version":"0.7.0"},
    "projects": [],
}

Hardware

The tracker runs on a Linode instance operated by chronomex.

Monitoring

http://tracker.archiveteam.org has a Munin instance located at http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/.

Software used:

  • Universal Tracker is a Ruby HTTP application that sends and receives JSON payloads and uses Redis for the data store.
  • Redis A memory based key value store
  • Debian is the Linux distribution the stack is built upon.
  • warrior-hq a small Sinatra web app to manage the Warriors and display the geo-location world map.

You can also set up your own tracker.

People

In case of tracker maintenance emergency, please contact these active users (as of 2014-08-29). Tracker emergencies only!: