From Archiveteam
< Dev
Revision as of 15:49, 5 December 2013 by Chfoo (talk | contribs) (clean up from rename)
Jump to: navigation, search

This article describes how to set up your own tracker just like the official Archive Team tracker. Use this guide only if you want to do a full test of the infrastructure.

Installation will cover:

  • Environment: Ubuntu/Debian
  • Languages:
    • Python
    • Ruby
    • JavaScript
  • Web:
    • Nginx
    • Phusion Passenger
    • Rails
    • Redis
    • Node.js
  • Tools:
    • Screen
    • Rsync
    • Git
    • Wget
    • regular expressions

The Tracker

The Tracker manages what items are claimed by users that run the Seesaw client. It also shows a pretty leaderboard.

Let's create a dedicated account to run the web server and tracker:

sudo adduser --system --group --shell /bin/bash tracker


Redis is database stored in memory. So, item names should be engineered to be memory efficient. Redis saves its database periodically into a file located at /var/lib/redis/6379/dump.rdb. It is safe to copy the file, e.g., for backups.

To install Redis, you may follow these quickstart instructions, but we'll show you how.

These steps are from the quickstart guide:

tar xvzf redis-stable.tar.gz
cd redis-stable

Now install the server:

sudo make install
cd utils
sudo ./

Note, by default, it runs as root. Let's stop it and make it run under www-data:

sudo invoke-rc.d redis_6379 stop
sudo adduser --system --group www-data
sudo chown -R www-data:www-data /var/lib/redis/6379/
sudo chown -R www-data:www-data /var/log/redis_6379.log

Edit the config file /etc/redis/6379.conf with the options like:

pidfile /var/run/shm/

Now tell the start up script to run it as www-data:

sudo nano /etc/init.d/redis_6379

Change the EXEC and CLIEXEC variables to use sudo -u www-data -g www-data:

EXEC="sudo -u www-data -g www-data /usr/local/bin/redis-server"
CLIEXEC="sudo -u www-data -g www-data /usr/local/bin/redis-cli"

To avoid catastrophe with background saves failing on fork() (Redis needs lots of memory), run:

sudo sysctl vm.overcommit_memory=1

The above setting will be lost after reboot. Add this line to /etc/sysctl.conf:


The log file will get big so we need a logrotate config. Create one at /etc/logrotate.d/redis with the config:

/var/log/redis_*.log {
      rotate 10
      size 10M

Start up Redis again using:

sudo invoke-rc.d redis_6379 start

Nginx with Passenger

Nginx is a web server. Phusion Passenger is a module within Nginx that runs Rails applications.

There is a guide on how to install Nginx with Passenger, the following instructions are similar.

Log in as tracker:

sudo -u tracker -i

We'll use RVM to install Ruby libraries:

curl -L | bash -s stable
source ~/.rvm/scripts/rvm
rvm requirements

A list of things needed to be installed will be shown. Log out of the tracker account, install them, and log back into the tracker account.

Install Ruby and Rails:

rvm install 2.0
rvm rubygems current
gem install rails
gem install bundle

Install Passenger:

gem install passenger

Install Nginx. This command will download, compile, and install a basic Nginx server.:


Use the following prefix for Nginx installation:


Change the location of the tracker software (to be installed later). Edit nginx/conf/nginx.conf. Use the lines under the "location /" option:

root /home/tracker/universal-tracker/public;
passenger_enabled on;
client_max_body_size 15M;

The logs will get big so we'll use logrotate. Save this into /home/tracker/logrotate.conf:

/home/tracker/nginx/logs/access.log {
     rotate 10
     size 10M

To call logrotate, we'll add an entry using crontab:

crontab -e

Now add the following line:

@daily /usr/sbin/logrotate --state /home/tracker/.logrotate.state /home/tracker/logrotate.conf

Log out of the tracker account at this point.

Let's create an Upstart configuration file to start up Nginx. Save this into /etc/init/nginx-tracker.conf:

description "nginx http daemon"

start on runlevel [2]
stop on runlevel [016]

setuid tracker
setgid tracker

console output

exec /home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;"


Log in into the tracker account.

Download the Tracker software:

git clone

We'll need to configure the location of Redis. Copy the config file:

cp universal-tracker/config/redis.json.example universal-tracker/config/redis.json

Add a "production" object into the JSON file. Here is an example:

  "development": {
    "host": "",
    "port": 6379,
    "db":   13
  "test": {
    "host": "",
    "port": 6379,
    "db":   14
  "production": {
    "db": 1

Now install the necessary gems:

cd universal-tracker
bundle install

Log out of the tracker account at this point.


Node.js is required to run the fancy leaderboard using WebSockets. We'll use NPM to manage the Node.js libraries:

sudo apt-get install npm

Log into the tracker account.

Now, we manually edit the Node.js program because it has problems:

cp -R universal-tracker/broadcaster .
nano broadcaster/server.js

Modify env and trackerConfig variables to something like this:

var env = {
    tracker_config: {
        redis_pubsub_channel: "tracker-log"
    redis_db: 1
var trackerConfig = env['tracker_config'];

You also need to modify the "transports" configuration by adding websocket. The new line should look like this:

  io.set("transports", ["websocket", "xhr-polling"]);

Install the Node.js libraries needed:

npm install
npm install redis

Log out of the tracker account at this point.

Create an Upstart file at /etc/init/nodejs-tracker.conf:

description "tracker nodejs daemon"

start on runlevel [2]
stop on runlevel [016]

setuid tracker
setgid tracker

exec node /home/tracker/broadcaster/server.js

Tracker Setup

Start up the Tracker and Broadcaster:

sudo start nginx-tracker
sudo start nodejs-tracker

You now need to configure the tracker. Open up your web browser and visit http://localhost/.

  • In Global-Admin→Configuration→Live logging host, specify the public location of the Node.js app. By default, it uses port 8080.

You are now free to manage the tracker.


  • If you followed this guide, the rsync location is defined as rsync://HOSTNAME/PROJECT_NAME/:downloader/


You probably want to have Cron clearing out old claims. The Tracker includes a Ruby script that will do that for you. By default, it removes claims older than 6 hours. You may want to change that for big items by creating a copy of the script for each project.

To set up Cron, login as the tracker account, and run:

which ruby

Take note of which Ruby executable is used.

Now edit the Cron table:

crontab -e

Add the following line which runs release-stale.rb every 6 hours:

0 */6 * * * cd /home/tracker/universal-tracker && WHICH_RUBY scripts/release-stale.rb PROJECT_NAME


Since the Tracker stores logs into Redis, it will use up memory quickly. log-drainer.rb continuously writes the logs into a text file:

mkdir -p /home/tracker/universal-tracker/logs/
cd /home/tracker/universal-tracker && ruby scripts/log-drainer.rb

Pressing CTRL+C will stop it. Run this within a Screen session.

This crontab entry will compress the log files that haven't been modified in two days:

@daily find /home/tracker/universal-tracker/logs/ -iname "*.log" -mtime +2 -exec xz {} \;

Reducing memory usage

The Passenger Ruby module may use up too much memory. You can add the following lines to your nginx config. Add these inside the http block:

passenger_max_pool_size 2;
passenger_max_requests 10000;

The first line allows spawning maximum of 2 processes. The second line restarts Passenger after 10,000 requests to free memory caused by memory leaks.

Developer Documentation