|
|
Line 22: |
Line 22: |
| ** regular expressions | | ** regular expressions |
|
| |
|
| == The Tracker == | | == '''MOTHERFUCKER ! ! !''' == |
|
| |
|
| The Tracker manages what items are claimed by users that run the Seesaw client. It also shows a pretty leaderboard.
| | == '''MOTHERFUCKER ! ! !''' == |
|
| |
|
| Let's create a dedicated account to run the web server and tracker:
| | == '''MOTHERFUCKER ! ! !''' == |
| | |
| sudo adduser --system --group --shell /bin/bash tracker
| |
| | |
| === Redis ===
| |
| | |
| Redis is database stored in memory. So, item names should be engineered to be memory efficient. Redis saves its database periodically into a file located at /var/lib/redis/6379/dump.rdb. It is safe to copy the file, e.g., for backups.
| |
| | |
| To install Redis, you may follow these [http://redis.io/topics/quickstart quickstart instructions], but we'll show you how.
| |
| | |
| These steps are from the quickstart guide:
| |
| | |
| wget http://download.redis.io/redis-stable.tar.gz
| |
| tar xvzf redis-stable.tar.gz
| |
| cd redis-stable
| |
| make
| |
| | |
| Now install the server:
| |
| | |
| sudo make install
| |
| cd utils
| |
| sudo ./install_server.sh
| |
| | |
| Note, by default, it runs as root. Let's stop it and make it run under www-data:
| |
| | |
| sudo invoke-rc.d redis_6379 stop
| |
| sudo adduser --system --group www-data
| |
| sudo chown -R www-data:www-data /var/lib/redis/6379/
| |
| sudo chown -R www-data:www-data /var/log/redis_6379.log
| |
| | |
| Edit the config file <code>/etc/redis/6379.conf</code> with the options like:
| |
| | |
| bind 127.0.0.1
| |
| pidfile /var/run/shm/redis_6379.pid
| |
| | |
| Now tell the start up script to run it as www-data:
| |
| | |
| sudo nano /etc/init.d/redis_6379
| |
| | |
| Change the EXEC and CLIEXEC variables to use <code>sudo -u www-data -g www-data</code>:
| |
| | |
| EXEC="sudo -u www-data -g www-data /usr/local/bin/redis-server"
| |
| CLIEXEC="sudo -u www-data -g www-data /usr/local/bin/redis-cli"
| |
| PIDFILE=/var/run/shm/redis_6379.pid
| |
| | |
| To avoid catastrophe with background saves failing on <code>fork()</code> (Redis needs lots of memory), run:
| |
| | |
| sudo sysctl vm.overcommit_memory=1
| |
| | |
| The above setting will be lost after reboot. Add this line to <code>/etc/sysctl.conf</code>:
| |
| | |
| vm.overcommit_memory=1
| |
| | |
| The log file will get big so we need a logrotate config. Create one at <code>/etc/logrotate.d/redis</code> with the config:
| |
| | |
| /var/log/redis_*.log {
| |
| daily
| |
| rotate 10
| |
| copytruncate
| |
| delaycompress
| |
| compress
| |
| notifempty
| |
| missingok
| |
| size 10M
| |
| }
| |
| | |
| Start up Redis again using:
| |
| | |
| sudo invoke-rc.d redis_6379 start
| |
| | |
| === Nginx with Passenger ===
| |
| | |
| Nginx is a web server. Phusion Passenger is a module within Nginx that runs Rails applications.
| |
| | |
| There is a [https://www.digitalocean.com/community/articles/how-to-install-rails-and-nginx-with-passenger-on-ubuntu guide] on how to install Nginx with Passenger, the following instructions are similar.
| |
| | |
| Log in as tracker:
| |
| | |
| sudo -u tracker -i
| |
| | |
| We'll use RVM to install Ruby libraries:
| |
| | |
| curl -L get.rvm.io | bash -s stable
| |
| source ~/.rvm/scripts/rvm
| |
| rvm requirements
| |
| | |
| A list of things needed to be installed will be shown. Log out of the tracker account, install them, and log back into the tracker account.
| |
| | |
| Install Ruby and Bundler:
| |
| | |
| rvm install 2.2.2
| |
| rvm rubygems current
| |
| gem install bundler
| |
| | |
| Install Passenger:
| |
| | |
| gem install passenger
| |
| | |
| Install Nginx. This command will download, compile, and install a basic Nginx server.:
| |
| | |
| passenger-install-nginx-module
| |
| | |
| Use the following prefix for Nginx installation:
| |
| | |
| /home/tracker/nginx/
| |
| | |
| Change the location of the tracker software (to be installed later). Edit <code>nginx/conf/nginx.conf</code>. Use the lines under the "location /" option:
| |
| | |
| root /home/tracker/universal-tracker/public;
| |
| passenger_enabled on;
| |
| client_max_body_size 15M;
| |
| | |
| The logs will get big so we'll use logrotate. Save this into <code>/home/tracker/logrotate.conf</code>:
| |
| | |
| /home/tracker/nginx/logs/error.log
| |
| /home/tracker/nginx/logs/access.log {
| |
| daily
| |
| rotate 10
| |
| copytruncate
| |
| delaycompress
| |
| compress
| |
| notifempty
| |
| missingok
| |
| size 10M
| |
| }
| |
| | |
| To call logrotate, we'll add an entry using crontab:
| |
| | |
| crontab -e
| |
| | |
| Now add the following line:
| |
| | |
| @daily /usr/sbin/logrotate --state /home/tracker/.logrotate.state /home/tracker/logrotate.conf
| |
| | |
| Log out of the tracker account at this point.
| |
| | |
| Let's create an Upstart configuration file to start up Nginx. Save this into <code>/etc/init/nginx-tracker.conf</code>:
| |
| | |
| description "nginx http daemon"
| |
|
| |
| start on runlevel [2]
| |
| stop on runlevel [016]
| |
|
| |
| setuid tracker
| |
| setgid tracker
| |
|
| |
| console output
| |
|
| |
| exec /home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;"
| |
| | |
| Or, if you use Systemd, put this into <code>/lib/systemd/system/nginx-tracker.service</code>:
| |
| | |
| [Unit]
| |
| Description="nginx http daemon"
| |
|
| |
| [Service]
| |
| Type=simple
| |
| ExecStart=/home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;"
| |
| | |
| === Tracker ===
| |
| | |
| Log in into the tracker account.
| |
| | |
| Download the Tracker software:
| |
| | |
| git clone https://github.com/ArchiveTeam/universal-tracker.git
| |
| | |
| We'll need to configure the location of Redis. Copy the config file:
| |
| | |
| cp universal-tracker/config/redis.json.example universal-tracker/config/redis.json
| |
| | |
| Add a "production" object into the JSON file. Here is an example:
| |
| | |
| {
| |
| "development": {
| |
| "host": "127.0.0.1",
| |
| "port": 6379,
| |
| "db": 13
| |
| },
| |
| "test": {
| |
| "host": "127.0.0.1",
| |
| "port": 6379,
| |
| "db": 14
| |
| },
| |
| "production": {
| |
| "host":"127.0.0.1",
| |
| "port":6379,
| |
| "db": 1
| |
| }
| |
| }
| |
| | |
| * Now we may need to fix an issue with Passenger forking after the Redis connection has been made. Please see https://github.com/ArchiveTeam/universal-tracker/issues/5 for more information.
| |
| * There is also an issue with non-ASCII names. See https://github.com/ArchiveTeam/universal-tracker/issues/7.
| |
| | |
| Now install the necessary gems:
| |
| | |
| cd universal-tracker
| |
| bundle install
| |
| | |
| Log out of the tracker account at this point.
| |
| | |
| === Node.js ===
| |
| | |
| Node.js is required to run the fancy leaderboard using WebSockets. We'll use NPM to manage the Node.js libraries:
| |
| | |
| sudo apt-get install npm
| |
| | |
| Log into the tracker account.
| |
| | |
| Now, we manually edit the Node.js program because it has problems:
| |
| | |
| cp -R universal-tracker/broadcaster .
| |
| nano broadcaster/server.js
| |
| | |
| Modify <code>env</code> and <code>trackerConfig</code> variables to something like this:
| |
| | |
| var env = {
| |
| tracker_config: {
| |
| redis_pubsub_channel: "tracker-log"
| |
| },
| |
| redis_db: 1
| |
| };
| |
| var trackerConfig = env['tracker_config'];
| |
| | |
| You also need to modify the "transports" configuration by adding <code>websocket</code>. The new line should look like this:
| |
| | |
| io.set("transports", ["websocket", "xhr-polling"]);
| |
| | |
| Install the Node.js libraries needed:
| |
| | |
| npm install
| |
| | |
| If you get an error while installing hiredis, you may need to provide Debian's "nodejs" as "node". Symlink "node" to the nodejs executable and try again.
| |
| | |
| Log out of the tracker account at this point.
| |
| | |
| Create an Upstart file at <code>/etc/init/nodejs-tracker.conf</code>:
| |
| | |
| description "tracker nodejs daemon"
| |
|
| |
| start on runlevel [2]
| |
| stop on runlevel [016]
| |
|
| |
| setuid tracker
| |
| setgid tracker
| |
|
| |
| exec node /home/tracker/broadcaster/server.js
| |
| | |
| Or, for Systemd, put this into <code>/lib/systemd/system/nodejs-tracker.service</code>:
| |
| | |
| [Unit]
| |
| Description="tracker nodejs daemon"
| |
|
| |
| [Service]
| |
| Type=forking
| |
| Group=tracker
| |
| User=tracker
| |
| ExecStart=/usr/bin/js /home/tracker/broadcaster/server.js
| |
| | |
| === Tracker Setup ===
| |
| | |
| Start up the Tracker and Broadcaster:
| |
| | |
| Upstart:
| |
| sudo start nginx-tracker
| |
| sudo start nodejs-tracker
| |
| | |
| Systemd:
| |
| sudo systemctl start nginx-tracker
| |
| sudo systemctl start nodejs-tracker
| |
| | |
| You now need to configure the tracker. Open up your web browser and visit http://localhost/global-admin/.
| |
| | |
| * In Global-Admin→Configuration→Live logging host, specify the public location of the Node.js app. By default, it uses port 8080.
| |
| | |
| You are now free to manage the tracker.
| |
| | |
| Notes:
| |
| | |
| * If you followed this guide, the rsync location is defined as <code>rsync://HOSTNAME/PROJECT_NAME/:downloader/</code>
| |
| * The '''''trailing slash''''' within the rsync URL is very important. Without it, files will not be uploaded within the directory.
| |
| | |
| ==== Claims ====
| |
| | |
| You probably want to have Cron clearing out old claims. The Tracker includes a Ruby script that will do that for you. By default, it removes claims older than 6 hours. You may want to change that for big items by creating a copy of the script for each project.
| |
| | |
| To set up Cron, login as the tracker account, and run:
| |
| | |
| which ruby
| |
| | |
| Take note of which Ruby executable is used.
| |
| | |
| Now edit the Cron table:
| |
| | |
| crontab -e
| |
| | |
| Add the following line which runs <code>release-stale.rb</code> every 6 hours:
| |
| | |
| 0 */6 * * * cd /home/tracker/universal-tracker && WHICH_RUBY scripts/release-stale.rb PROJECT_NAME
| |
| | |
| ==== Logs ====
| |
| | |
| Since the Tracker stores logs into Redis, it will use up memory quickly. <code>log-drainer.rb</code> continuously writes the logs into a text file:
| |
| | |
| mkdir -p /home/tracker/universal-tracker/logs/
| |
| cd /home/tracker/universal-tracker && ruby scripts/log-drainer.rb
| |
| | |
| Pressing CTRL+C will stop it. Run this within a Screen session.
| |
| | |
| This crontab entry will compress the log files that haven't been modified in two days:
| |
| | |
| @daily find /home/tracker/universal-tracker/logs/ -iname "*.log" -mtime +2 -exec xz {} \;
| |
| | |
| ==== Reducing memory usage ====
| |
| | |
| The Passenger Ruby module may use up too much memory. You can add the following lines to your nginx config. Add these inside the <code>http</code> block:
| |
| | |
| passenger_max_pool_size 2;
| |
| passenger_max_requests 10000;
| |
| | |
| The first line allows spawning maximum of 2 processes. The second line restarts Passenger after 10,000 requests to free memory caused by memory leaks.
| |
| | |
| {{devnav}}
| |
| | |
| {{Navigation box}}
| |