Difference between revisions of "Twitch.tv"

From Archiveteam
Jump to: navigation, search
(By Archive Team: add link to index)
(add information about software commonly used to download twitch vods and chat)
 
(36 intermediate revisions by 8 users not shown)
Line 4: Line 4:
 
| image = Twitch_homepage_screenshot.png
 
| image = Twitch_homepage_screenshot.png
 
| logo = Twitch_Logo.png
 
| logo = Twitch_Logo.png
| project_status = '''Special Case''' (Archives to be deleted)
+
| project_status = {{specialcase}} (archives of streams actively purged after an amount of time)
| archiving_status = {{saved}} (popular videos only)
+
| archiving_status = {{partiallysaved}} (popular videos only)
 
| irc = burnthetwitch
 
| irc = burnthetwitch
| source = [https://github.com/ArchiveTeam/twitchtv-discovery-grab Phase 1],[https://github.com/ArchiveTeam/twitchtv-grab Phase 2], [https://github.com/ArchiveTeam/twitchtv-items Items]
+
| source = [https://github.com/ArchiveTeam/twitchtv-discovery-grab Phase 1],[https://github.com/ArchiveTeam/twitchtv-grab Phase 2], [https://github.com/ArchiveTeam/twitchtv-items Items], [https://github.com/ArchiveTeam/twitchtv-items Index]
 
| tracker = [http://tracker.archiveteam.org/twitchdisco/ Phase 1], [http://tracker.archiveteam.org/twitchtv/ Phase 2]
 
| tracker = [http://tracker.archiveteam.org/twitchdisco/ Phase 1], [http://tracker.archiveteam.org/twitchtv/ Phase 2]
 
}}
 
}}
Line 13: Line 13:
 
Justin.tv—sorry, ''cough'', I mean to say—'''Twitch.tv''' is a live video streaming service.
 
Justin.tv—sorry, ''cough'', I mean to say—'''Twitch.tv''' is a live video streaming service.
  
Twitch was supposedly acquired by [[YouTube]]/[[Google]] but [[Amazon]] was the final buyer.<ref>http://www.twitch.tv/p/thankyou</ref>
+
Twitch was rumored to have been acquired by [[YouTube]]/[[Google]] but [[Amazon]] was the final buyer.<ref>http://www.twitch.tv/p/thankyou</ref>
  
== Shutdown ==
+
== Broadcast retention changes ==
 +
 
 +
After Twitch's acquisition by Amazon, changes were made to how long broadcasts (sometimes called VODs) were retained for viewing on the site. Previously, all partnered accounts had indefinite storage, while standard accounts had storage for a few days. This was then cut down to two months.
  
 
<blockquote>
 
<blockquote>
Line 44: Line 46:
  
 
</blockquote>
 
</blockquote>
 +
 +
Thus a mission was started to archive as much of Twitch as we reasonably could. Our efforts can be found in [[Twitch.tv/Warroom|the War Room]] and on the [[Talk:Twitch.tv|Talk page]].
 +
 +
== Known exceptions ==
 +
 +
Although most partner accounts have broadcasts deleted after 60 days, there are some exceptions. Most of these are esports tournament channels, but other channels may be excluded for reasons such as being culturally significant. Below is an incomplete list of these exceptions.
 +
 +
{| class="wikitable"
 +
|-
 +
! Channel !! Indefinite since !! Probable reason
 +
|-
 +
| [https://www.twitch.tv/dota2ti Dota 2 The International] || Policy Inception || Esports tournament
 +
|-
 +
| [https://www.twitch.tv/riotgames Riot Games], [https://www.twitch.tv/lcs LCS], [https://www.twitch.tv/lec LEC] || Policy Inception || Game developer & esports tournaments
 +
|-
 +
| [https://www.twitch.tv/beyondthesummit/ Beyond The Summit] || Policy Inception || Esports broadcaster
 +
|-
 +
| [https://www.twitch.tv/twitchplayspokemon Twitch Plays Pokemon] || Policy Inception || Culturally significant
 +
|-
 +
| [https://www.twitch.tv/joindotared/ joinDOTA Red], [https://www.twitch.tv/joindotablue/ Blue], [https://www.twitch.tv/joindotacommunity/ Community] || Policy Inception || Esports broadcaster
 +
|-
 +
| [https://www.twitch.tv/gamesdonequick/ Games Done Quick] || Policy Inception || Culturally significant (charity fundraiser)
 +
|-
 +
| [https://www.twitch.tv/tipofthehats/ Tip of the Hats] || Policy Inception || Culturally significant (charity fundraiser)
 +
|-
 +
| [https://www.twitch.tv/rocketleague/ Rocket League] || April 2016 || Game channel & esports tournaments
 +
|-
 +
| [https://www.twitch.tv/Evo Evo] || Policy Inception || Esports tournament
 +
|-
 +
| [https://www.twitch.tv/esl_csgo/ ESL CS:GO] || Policy Inception || Esports broadcaster
 +
|-
 +
| [https://www.twitch.tv/dreamhackcs/ DreamHack Counter-Strike] || Policy Inception || Esports & LAN event broadcaster
 +
|-
 +
| [https://www.twitch.tv/shroud/ shroud] || Policy Inception? || Notable streamer & esports personality
 +
|-
 +
| [https://www.twitch.tv/ninja/ Ninja] || January 2018 || Notable streamer
 +
|-
 +
| [https://www.twitch.tv/pokimane/ pokimane] || Policy Inception || Notable streamer
 +
|-
 +
| [https://www.twitch.tv/starcraft/ StarCraft] || Policy Inception || Game channel & esports tournaments
 +
|-
 +
| [https://www.twitch.tv/playoverwatch/ PlayOverwatch] || August 2017? || Game channel & esports tournaments
 +
|-
 +
| [https://www.twitch.tv/playhearthstone/ PlayHearthstone] || Policy Inception || Game channel & esports tournaments
 +
|-
 +
| [https://www.twitch.tv/fortnite Fortnite] || Channel Creation || Game channel & esports tournaments
 +
|-
 +
| [https://www.twitch.tv/faceittv/ FACEIT TV] || Policy Inception || Esports service & tournaments
 +
|-
 +
| [https://www.twitch.tv/teamfortresstv/ teamfortress.tv] || Policy Inception || Esports broadcaster
 +
|-
 +
| [https://www.twitch.tv/thegdstudio/ The GD Studio] || Policy Inception || Esports broadcaster
 +
|-
 +
| [https://www.twitch.tv/roomonfire/ Room On Fire] || Policy Inception || Esports broadcaster
 +
|-
 +
| [https://www.twitch.tv/nipgamingtv/ Ninjas In Pyjamas] || Policy Inception || Esports team/broadcaster
 +
|-
 +
| [https://www.twitch.tv/bobross/ Bob Ross] || June 2016 || TV show & culturally significant
 +
|-
 +
| [https://www.twitch.tv/pgl/ PGL] || Policy Inception || Esports tournament organizer
 +
|-
 +
| [https://www.twitch.tv/notch/ Notch] || Policy Inception || Minecraft creator
 +
|-
 +
| [https://www.twitch.tv/deadmau5 deadmau5] || Policy Inception || Musician
 +
|-
 +
| [https://www.twitch.tv/porterrobinson Porter Robinson] || Policy Inception? || Musician
 +
|-
 +
| [https://www.twitch.tv/mogra MOGRA] || Policy Inception? || Music club in Akihabara, JP
 +
|-
 +
| [https://www.twitch.tv/mojang Mojang] || Policy Inception || Game developers, including Minecraft
 +
|-
 +
| [https://www.twitch.tv/yogscast Yogscast] || Policy Inception || Culturally significant
 +
|-
 +
| [https://www.twitch.tv/twitch Twitch] || Policy Inception || The site (duh)
 +
|-
 +
| [https://www.twitch.tv/twitchpresents Twitch Presents] || March 2017 || Also the site, special event streams
 +
|-
 +
| [https://www.twitch.tv/thegameawards The Game Awards] || 2016? || Awards show
 +
|-
 +
| [https://www.twitch.tv/esportsawards The Esports Awards] || June 2017 || Awards show
 +
|-
 +
| [https://www.twitch.tv/amd AMD] || Policy Inception || CPU/GPU manufacturer
 +
|-
 +
| [https://www.twitch.tv/reddit reddit] || 2017 || Website
 +
|-
 +
| [https://www.twitch.tv/xbox Xbox] || Policy Inception || Console platform
 +
|-
 +
| [https://www.twitch.tv/playstation PlayStation] || Policy Inception || Console platform
 +
|-
 +
| [https://www.twitch.tv/nintendo Nintendo] || Policy Inception || Game developer & console manufacturer
 +
|-
 +
| [https://www.twitch.tv/pax PAX] || Policy Inception || Gaming convention
 +
|-
 +
| [https://www.twitch.tv/ign IGN] || Policy Inception || Magazine & website
 +
|-
 +
| [https://www.twitch.tv/pokerstars PokerStars] || Policy Inception || Poker website & tournaments
 +
|-
 +
| [https://www.twitch.tv/team/first FIRST Robotics Competition] || January 2017 || Partnership
 +
|}
  
 
== Site structure ==
 
== Site structure ==
Line 58: Line 159:
 
* Is there any irregularities? Differences between highlights and past broadcasts?
 
* Is there any irregularities? Differences between highlights and past broadcasts?
  
=== Storage Issues ===
+
== Software ==  
 
 
* How to decide which are important? 10+ views again? Do a discovery crawl first?
 
* Tahoe-LAFS? Grab ''all'' the videos into temp storage?
 
* Compress all the unwatched videos into postage stamp sized videos?
 
 
 
=== Discovery ===
 
 
 
All items discovered are located at [https://github.com/ArchiveTeam/twitchtv-items twitchtv-items]. A collated [https://dl.dropboxusercontent.com/u/672132/archiveteam/twitchtv_dump_json_2014-09-05.txt.xz JSON dump is available.]
 
 
 
== How can I help? ==
 
 
 
Download and fire up your [[warrior]]! Then select Twitch Phase 2. Better yet, select Archive Team's Choice.
 
 
 
Alternatively for advanced users, you can run the scripts manually see below.
 
 
 
Don't forget to '''[https://archive.org/donate/ donate to the Internet Archive]''' who will be hosting these files. Disk space is cheap but maintaining them is not!
 
 
 
=== For those not using the Warrior ===
 
 
 
''Advanced User Quick Start''
 
 
 
Please run these sysctl tweaks to optimize uploads:
 
 
 
<pre>
 
# Add to /etc/sysctl.conf and run "sysctl -p"
 
# increase TCP max buffer size settable using setsockopt()
 
net.core.rmem_max = 16777216
 
net.core.wmem_max = 16777216
 
# increase Linux autotuning TCP buffer limit
 
net.ipv4.tcp_rmem = 4096 87380 16777216
 
net.ipv4.tcp_wmem = 4096 65536 16777216
 
</pre>
 
 
 
You can also issue them without modifying /etc/sysctl.conf by running e.g. <pre>sysctl net.core.rmem_max=16777216 net.core.wmem_max=16777216</pre>, but be aware that those won't stick around across reboots.
 
 
 
<pre>apt-get install git git-core libgnutls-dev lua5.1 liblua5.1-0 liblua5.1-0-dev screen python-dev python-pip bzip2 zlib1g-dev</pre>
 
 
 
<pre>git clone https://github.com/ArchiveTeam/twitchtv-grab
 
cd ./twitchtv-grab
 
pip install seesaw
 
./get-wget-lua.sh
 
 
 
...
 
 
 
pip install requests</pre>
 
 
 
  
wget-lua may of failed earlier, if so then:
+
=== Twitch Chat Downloader ===
  
<pre>cd get-wget-lua.tmp
+
Download link: ttps://github.com/PetterKraabol/Twitch-Chat-Downloader
mv src/wget ../wget-lua
 
cd ..</pre>
 
  
And finally to actually run
+
Software to download the twitch chat of a given twitch VOD, or a selection of VODs by channel. Note that you will want to (at least) use the JSON output to preserve the most information about the chat.
  
<pre>run-pipeline pipeline.py --concurrent 2 YOURNICKHERE --disable-web-server</pre>
+
Example command to download the chat:
  
For troubleshooting and the details please see [https://github.com/ArchiveTeam/twitchtv-grab/blob/master/README.md README].
+
    tcd --channel "AOC" -o "." --format "default,irc,json,srt,ssa" --timezone "UTC" --verbose --debug --log
  
=== What we are saving ===
+
The output files will be files named after the twitch video ID with the given file extensions.
  
Currently:
+
=== TwitchLeecher ===
  
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/01_twitchplayspokemon.txt twitchplayspokemon]: "test" run, estimated 3 TB ($6000)
+
Download link: https://github.com/Franiac/TwitchLeecher
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/02_suggestions_19553_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/01_suggestions_wiki-19553.txt these channels] with 100 or more views: estimated 23 TB ($46000)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/03_suggestions_19553_100views_2.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/01_suggestions_wiki-19553.txt these channels] with 100 or more views: estimated 0.7 TB ($1400)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/04_suggestions_19616_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/02_suggestions_wiki-19616.txt these channels] with 100 or more views: estimated 4 TB ($8000)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/05_top_videos_10000views.txt These top videos] which have 10000 or more views: estimated 20 TB ($40000)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/06_suggestions_19795_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/03_suggestions_wiki-19795.txt these channels] with 100 or more views: estimated 8 TB ($16000)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/07_socialblade_top_5000views.txt These videos] selected from SocialBlade's top Twitch channels with 5000 or more views: estimated 10 TB ($20000)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/08_suggestions_19851_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/05_suggestions_wiki-19851.txt these channels] with 100 or more views: estimated 15 TB ($30000)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/09_top_vid_per_suggestion_19553-19851.txt These videos], from previous suggestions, that are most viewed per channel: estimated 0.1TB ($200)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/10_suggestions_19874_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/06_suggestions_wiki-19874.txt these channels] with 100 or more views: estimated 7 TB ($14000)
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/11_suggestions_19911_100views_and_top1.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/07_suggestions_wiki-19911.txt these channels] with 100 or more views or most viewed video per channel: estimated 1.6 TB ($3200)
 
  
Next:
+
Software to download the raw .ts segments of a twitch VOD (of a given quality) and combine them, with an optional conversion to MP4.
  
* Sorry, no more suggestions! More suggestions may be considered if you [https://archive.org/donate/ donate to the Internet Archive].
+
Example filename template (set in settings) to include all of the relevant information of the VOD in the filename:
  
Dollar figures shown to illustrate cost of permanent archives. These are not actual values but are meant to represent simplified values and act as a sane budget. Dollars in USD at $2000 per TB estimate (not per TB of disk space alone).
+
    {date}_{time24}_{channel}_{id}_{game}_{title}_{res}_{fps}fps_{start}_{end}
  
Channels not included:
 
 
* speeddemosarchivesda: already in IA
 
* vinesauce: avoiding duplication, see below
 
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_blacklists/user_blacklist.txt and others]
 
 
Anything culturally significant to add? Comment on [[Talk:Twitch.tv]]. Don't forget to sign your comments with <code><nowiki>~~~~</nowiki></code>
 
.
 
  
 
== Archives ==
 
== Archives ==
Line 155: Line 190:
 
Archives will be made available later as [[WARC]] files in the [https://archive.org/details/archiveteam_twitchtv archiveteam_twitchtv] collection at the Internet Archive. You can access them by the Wayback Machine, but you'll need search an index to find the media files.  
 
Archives will be made available later as [[WARC]] files in the [https://archive.org/details/archiveteam_twitchtv archiveteam_twitchtv] collection at the Internet Archive. You can access them by the Wayback Machine, but you'll need search an index to find the media files.  
  
A work-in-progress [http://chfoo-cn.mooo.com/~archiveteam/twitchtv-index/html/ '''searchable index''' is now available!]
+
A work-in-progress [http://archive.fart.website/archiveteam/twitchtv-index/html/ '''searchable index''' is now available!]
  
 
=== Renegade Stream Archives ===
 
=== Renegade Stream Archives ===
Line 164: Line 199:
 
** [http://vinesauce.com/vinetalk/index.php?topic=4321.msg81672 Vinesauce Forum link]
 
** [http://vinesauce.com/vinetalk/index.php?topic=4321.msg81672 Vinesauce Forum link]
 
* [http://archive.klaxa.eu Klaxa.eu's Archive of The 4chan Cup] - An existing, complete archive of The 4chan Cup, starting from the 2014 Autumn Games up till today.
 
* [http://archive.klaxa.eu Klaxa.eu's Archive of The 4chan Cup] - An existing, complete archive of The 4chan Cup, starting from the 2014 Autumn Games up till today.
 +
 +
=== Other ===
 +
 +
* https://archive.org/details/archiveteam_twitchtv_espesgrab
 +
 +
== Butt Controversy ==
 +
In June 2016, Twitch deleted a bunch of custom emoticons on the grounds of obscenity. (See {{url|http://www.dailydot.com/esports/twitch-butt-emotes/}} for details.)
 +
 +
The emotes can be found in the backend at: <nowiki>https://static-cdn.jtvnw.net/emoticons/v1/<id number>/<size></nowiki>
 +
where id number ranges from 1 to 103667(as of 20160622), with no leading zeroes, and size is 1.0, 2.0 or 3.0.
 +
 +
Note: sizes 0.5, 1.5, 2.5, 3.5, 4.0, 4.5 and 5.0 are valid as well, but return the same data for most(all?) emotes as the next highest available 'whole number' or the largest below that one, i.e. for 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, will match 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0 respectively)
 +
 +
All emote graphics and sizes (but not their associated chat 'shortnames' i.e. "<3" for emote #9, which cannot be easily determined) still existing in the backend system up to emote id 103667 were archived through [[Archivebot]] [http://archive.fart.website/archivebot/viewer/job/b8fnv here]; the resulting WARCs can be downloaded through the viewer.
  
 
== See Also ==  
 
== See Also ==  
Line 173: Line 222:
 
* [http://www.reddit.com/r/twitchplayspokemon/comments/2d2vvb/tpps_vods_will_be_saved_in_their_entirety/ "TPP'S VODS WILL BE SAVED IN THEIR ENTIRETY"]
 
* [http://www.reddit.com/r/twitchplayspokemon/comments/2d2vvb/tpps_vods_will_be_saved_in_their_entirety/ "TPP'S VODS WILL BE SAVED IN THEIR ENTIRETY"]
 
* [http://www.businessinsider.com/amazon-in-talks-to-buy-twitch-2014-8 "Amazon Might Buy Video Game Streaming Service Twitch Before Google Can"]
 
* [http://www.businessinsider.com/amazon-in-talks-to-buy-twitch-2014-8 "Amazon Might Buy Video Game Streaming Service Twitch Before Google Can"]
 +
* [http://www.reddit.com/r/twitchplayspokemon/comments/2fksq1/twitch_is_trying_to_restore_deleted_vods/ "Twitch is Trying to Restore Deleted VODs"]
  
 
== References ==
 
== References ==
Line 180: Line 230:
 
{{navigation box}}
 
{{navigation box}}
  
[[Category:Video hosting services]]
+
[[Category:Video hosting]]

Latest revision as of 10:16, 21 October 2020

Twitch.tv
Twitch.tv logo
Twitch homepage screenshot.png
URL http://twitch.tv
Project status Special case (archives of streams actively purged after an amount of time)
Archiving status Partially saved (popular videos only)
Project source Phase 1,Phase 2, Items, Index
Project tracker Phase 1, Phase 2
IRC channel #burnthetwitch (on EFnet)
Project lead Unknown

Justin.tv—sorry, cough, I mean to say—Twitch.tv is a live video streaming service.

Twitch was rumored to have been acquired by YouTube/Google but Amazon was the final buyer.[1]

Broadcast retention changes

After Twitch's acquisition by Amazon, changes were made to how long broadcasts (sometimes called VODs) were retained for viewing on the site. Previously, all partnered accounts had indefinite storage, while standard accounts had storage for a few days. This was then cut down to two months.

Changes To VODs On Twitch

Aug 06 2014 · Engineering, Tech

Our goal at Twitch is straightforward: deliver the highest quality video. This includes the ability to watch video on demand (VOD) on all of our platforms, not just the website.

In order to create a system that supports live and VOD across the globe and on multiple platforms, we need to make significant changes to the way we’re currently storing video. Today, we’d like to discuss what these changes are, why they’re necessary, and how they benefit the entire Twitch community now and in the future.

[...]

Looking at Viewership Data

We found that the vast majority of past broadcast views happen within the first two weeks after they’re created. On the days following, viewership reduces exponentially.

We also discovered that 80% of our storage capacity is filled with past broadcasts that are never watched. That’s multiple petabytes for video that no one has ever viewed.

Highlights, on the other hand, have much more value and longevity. Over their lifetime, highlights get 9x as many views as past broadcasts.

[...]

As for existing past broadcasts, beginning three weeks from today, we will begin removing them from Twitch servers. If you would like to keep your past broadcasts, we encourage you to begin exporting or making highlights of your best moments so that they’re saved for posterity.

[...][2]

Thus a mission was started to archive as much of Twitch as we reasonably could. Our efforts can be found in the War Room and on the Talk page.

Known exceptions

Although most partner accounts have broadcasts deleted after 60 days, there are some exceptions. Most of these are esports tournament channels, but other channels may be excluded for reasons such as being culturally significant. Below is an incomplete list of these exceptions.

Channel Indefinite since Probable reason
Dota 2 The International Policy Inception Esports tournament
Riot Games, LCS, LEC Policy Inception Game developer & esports tournaments
Beyond The Summit Policy Inception Esports broadcaster
Twitch Plays Pokemon Policy Inception Culturally significant
joinDOTA Red, Blue, Community Policy Inception Esports broadcaster
Games Done Quick Policy Inception Culturally significant (charity fundraiser)
Tip of the Hats Policy Inception Culturally significant (charity fundraiser)
Rocket League April 2016 Game channel & esports tournaments
Evo Policy Inception Esports tournament
ESL CS:GO Policy Inception Esports broadcaster
DreamHack Counter-Strike Policy Inception Esports & LAN event broadcaster
shroud Policy Inception? Notable streamer & esports personality
Ninja January 2018 Notable streamer
pokimane Policy Inception Notable streamer
StarCraft Policy Inception Game channel & esports tournaments
PlayOverwatch August 2017? Game channel & esports tournaments
PlayHearthstone Policy Inception Game channel & esports tournaments
Fortnite Channel Creation Game channel & esports tournaments
FACEIT TV Policy Inception Esports service & tournaments
teamfortress.tv Policy Inception Esports broadcaster
The GD Studio Policy Inception Esports broadcaster
Room On Fire Policy Inception Esports broadcaster
Ninjas In Pyjamas Policy Inception Esports team/broadcaster
Bob Ross June 2016 TV show & culturally significant
PGL Policy Inception Esports tournament organizer
Notch Policy Inception Minecraft creator
deadmau5 Policy Inception Musician
Porter Robinson Policy Inception? Musician
MOGRA Policy Inception? Music club in Akihabara, JP
Mojang Policy Inception Game developers, including Minecraft
Yogscast Policy Inception Culturally significant
Twitch Policy Inception The site (duh)
Twitch Presents March 2017 Also the site, special event streams
The Game Awards 2016? Awards show
The Esports Awards June 2017 Awards show
AMD Policy Inception CPU/GPU manufacturer
reddit 2017 Website
Xbox Policy Inception Console platform
PlayStation Policy Inception Console platform
Nintendo Policy Inception Game developer & console manufacturer
PAX Policy Inception Gaming convention
IGN Policy Inception Magazine & website
PokerStars Policy Inception Poker website & tournaments
FIRST Robotics Competition January 2017 Partnership

Site structure

Software

Twitch Chat Downloader

Download link: ttps://github.com/PetterKraabol/Twitch-Chat-Downloader

Software to download the twitch chat of a given twitch VOD, or a selection of VODs by channel. Note that you will want to (at least) use the JSON output to preserve the most information about the chat.

Example command to download the chat:

   tcd --channel "AOC" -o "." --format "default,irc,json,srt,ssa" --timezone "UTC" --verbose --debug --log

The output files will be files named after the twitch video ID with the given file extensions.

TwitchLeecher

Download link: https://github.com/Franiac/TwitchLeecher

Software to download the raw .ts segments of a twitch VOD (of a given quality) and combine them, with an optional conversion to MP4.

Example filename template (set in settings) to include all of the relevant information of the VOD in the filename:

   {date}_{time24}_{channel}_{id}_{game}_{title}_{res}_{fps}fps_{start}_{end}


Archives

By Archive Team

Archives will be made available later as WARC files in the archiveteam_twitchtv collection at the Internet Archive. You can access them by the Wayback Machine, but you'll need search an index to find the media files.

A work-in-progress searchable index is now available!

Renegade Stream Archives

These archives are made in a manual fashion through the efforts of streaming communities. Feel free to expand this list.

Other

Butt Controversy

In June 2016, Twitch deleted a bunch of custom emoticons on the grounds of obscenity. (See http://www.dailydot.com/esports/twitch-butt-emotes/[IAWcite.todayMemWeb] for details.)

The emotes can be found in the backend at: https://static-cdn.jtvnw.net/emoticons/v1/<id number>/<size> where id number ranges from 1 to 103667(as of 20160622), with no leading zeroes, and size is 1.0, 2.0 or 3.0.

Note: sizes 0.5, 1.5, 2.5, 3.5, 4.0, 4.5 and 5.0 are valid as well, but return the same data for most(all?) emotes as the next highest available 'whole number' or the largest below that one, i.e. for 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, will match 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0 respectively)

All emote graphics and sizes (but not their associated chat 'shortnames' i.e. "<3" for emote #9, which cannot be easily determined) still existing in the backend system up to emote id 103667 were archived through Archivebot here; the resulting WARCs can be downloaded through the viewer.

See Also

External links

References


v · t · e         Archive Team
Current events

Alive... OR ARE THEY · Deathwatch · Projects

Archiveteam.jpg
Archiving projects

APKMirror · Archive.is · BetaArchive · Government Backup (#datarefuge · ftp-gov· Gmane · Internet Archive · It Died · Megalodon.jp · OldApps.com · OldVersion.com · OSBetaArchive · TEXTFILES.COM · The Dead, the Dying & The Damned · The Mail Archive · UK Web Archive · WebCite · Vaporwave.me

Blogging

Blog.pl · Blogger · Blogster · Blogter.hu · Freeblog.hu · Fuelmyblog · Jux · LiveJournal · My Opera · Nolblog.hu · Open Diary · ownlog.com · Posterous · Powerblogs · Proust · Roon · Splinder · Tumblr · Vox · Weblog.nl · Windows Live Spaces · Wordpress.com · Xanga · Yahoo! Blog · Zapd

Cloud hosting/file sharing

aDrive · AnyHub · Box · Dropbox · Docstoc · Fast.io · Google Drive · Google Groups Files · iCloud · Fileplanet · LayerVault · MediaCrush · MediaFire · Mega · MegaUpload · MobileMe · OneDrive · Pomf.se · RapidShare · Ubuntu One · Yahoo! Briefcase

Corporations

Apple · IBM · Google · Loblaw · Lycos Europe · Microsoft · Yahoo!

Events

Arab Spring · Great Ape-Snake War · Spanish Revolution

Font Repos

DaFont · Google Web Fonts · GNU FreeFont · Fontspace

Forums/Message boards

4chan · Captain Luffy Forums · College Confidential · DSLReports · ESPN Forums · Facepunch Forums · forums.starwars.com · HeavenGames · JamiiForums · Invisionfree · NeoGAF · Textream · The Classic Horror Film Board · Yahoo! Messages · Yahoo! Neighbors · Yuku.com · Zetaboards

Gaming

Atomicgamer · Bazaar.tf · City of Heroes · Club Nintendo · Clutch · Counter-Strike: Global Offensive · CS:GO Lounge · Desura · Dota 2 · Dota 2 Lounge · Emulation Zone · ESEA · GameBanana · GameMaker Sandbox · GameTrailers · Halo · HLTV.org · HQ Trivia · Infinite Crisis · joinDOTA · League of Legends · Liquipedia · Minecraft.net · Player.me · Playfire · Raptr · SingStar · Steam · SteamDB · SteamGridDB · Team Fortress 2 · TF2 Outpost · Warhammer · Xfire

Image hosting

500px · AOL Pictures · Blipfoto · Blingee · Canv.as · Camera+ · Cameroid · DailyBooth · Degree Confluence Project · DeviantART · Demotivalo.net · Flickr · Fotoalbum.hu · Fotolog.com · Fotopedia · Frontback · Geograph Britain and Ireland · Giphy · GTF Képhost · ImageShack · Imgh.us · Imgur · Inkblazers · Instagram · Kepfeltoltes.hu · Kephost.com · Kephost.hu · Kepkezelo.com · Keptarad.hu · Madden GIFERATOR · MLKSHK · Microsoft Clip Art · Microsoft Photosynth · Nokia Memories · noob.hu · Odysee · Panoramio · Photobucket · Picasa · Picplz · Pixiv · Portalgraphics.net · PSharing · Ptch · puu.sh · Rawporter · Relay.im · ScreenshotsDatabase.com · Sketch · Smack Jeeves · Snapjoy · Streetfiles · Tabblo · Tinypic · Trovebox · TwitPic · Wallbase · Wallhaven · Webshots · Wikimedia Commons

Knowledge/Wikis

arXiv · Citizendium · Clipboard.com · Deletionpedia · EditThis · Encyclopedia Dramatica · Etherpad · Everything2 · infoAnarchy · GeoNames · GNUPedia · Google Books (Google Books Ngram· Horror Movie Database · Insurgency Wiki · Knol · Lost Media Wiki · Neoseeker.com · Notepad.cc · Nupedia · OpenCourseWare · OpenStreetMap · Orain · Pastebin · Patch.com · Project Gutenberg · Puella Magi · Referata · Resedagboken · SongMeanings · ShoutWiki · The Internet Movie Database · TropicalWikis · Uncyclopedia · Urban Dictionary · Urban Exploration Resource · Webmonkey · Wikia · Wikidot · WikiHow · Wikkii · WikiLeaks · Wikipedia (Simple English Wikipedia· Wikispaces · Wikispot · Wik.is · Wiki-Site · WikiTravel · Word Count Journal

Magazines/Blogs/News

Cyberpunkreview.com · Game Developer Magazine · Gigaom · Hardware Canucks · Helium · JPG Magazine · Make Magazine · The Escapist · Polygamia.pl · San Fransisco Bay Guardian · Scoop · Regretsy · Yahoo! Voices

Microblogging

Heello · Identi.ca · Jaiku · Mommo.hu · Plurk · Sina Weibo · Tencent Weibo · Twitter · TwitLonger

Music/Audio

8tracks · AOL Music · Audimated.com · Cinch · digCCmixter · Dogmazic.net · Earbits · exfm · Free Music Archive · Gogoyoko · Indaba Music · Instacast · Instaudio · Jamendo · Last.fm · Music Unlimited · MOG · PureVolume · Reverbnation · ShareTheMusic · SoundCloud · Soundpedia · Spotify · This Is My Jam · TuneWiki · Twaud.io · WinAmp

People

Aaron Swartz · Michael S. Hart · Steve Jobs · Mark Pilgrim · Dennis Ritchie · Len Sassaman Project

Protocols/Infrastructure

FTP · Gopher · IRC · Usenet · World Wide Web
BitTorrent DHT

Q&A

Askville · Answerbag · Answers.com · Ask.com · Askalo · Baidu Knows · Blurtit · ChaCha · Experts Exchange · Formspring · GirlsAskGuys · Google Answers · Google Baraza · JustAnswer · MetaFilter · Quora · Retrospring · StackExchange · The AnswerBank · The Internet Oracle · Uclue · WikiAnswers · Yahoo! Answers

Recipes/Food

Allrecipes · Epicurious · Food.com · Foodily · Food Network · Punchfork · ZipList

Social bookmarking

Addinto · Backflip · Balatarin · BibSonomy · Bkmrx · Blinklist · BlogMarks · BookmarkSync · CiteULike · Connotea · Delicious · Designer News · Digg · Diigo · Dir.eccion.es · Evernote · Excite Bookmark · Faves · Favilous · folkd · Freelish · Getboo · GiveALink.org · Gnolia · Google Bookmarks · Hacker News · HeyStaks · IndianPad · Kippt · Knowledge Plaza · Licorize · Linkwad · Menéame · Microsoft Developer Network · myVIP · Mister Wong · My Web · Mylink Vault · Newsvine · Oneview · Pearltrees · Pinboard · Pocket · Propeller.com · Reddit · sabros.us · Scloog · Scuttle · Simpy · SiteBar · Slashdot · Squidoo · StumbleUpon · Twine · Voat · Vizited · Yummymarks · Xmarks · Yahoo! Buzz · Zootool · Zotero

Social networks

Bebo · BlackPlanet · Classmates.com · Cyworld · Dogster · Dopplr · douban · Ello · Facebook · Flixster · FriendFeed · Friendster · Friends Reunited · Gaia Online · Google+ · Habbo · hi5 · Hyves · iWiW · LinkedIn · Miiverse · mixi · MyHeritage · MyLife · Myspace · myVIP · Netlog · Odnoklassniki · Orkut · Plaxo · Qzone · Renren · Skyrock · Sonico.com · Storylane · Tagged · tvtag · Upcoming · Viadeo · Vine · Vkontakte · WeeWorld · Weibo · Wretch · Yahoo! Groups · Yahoo! Stars India · Yahoo! Upcoming · more sites...

Shopping/Retail

Alibaba · AliExpress · Amazon · Apple Store · Barnes & Noble · DirectCanada · eBay · Kmart · NCIX · Printfection · RadioShack · Sears · Sears Canada · Target · The Book Depository · ThinkGeek · Toys "R" Us · Walmart

Software/code hosting

Android Development · Alioth · Assembla · BerliOS · Betavine · Bitbucket · BountySource · Codecademy · CodePlex · Freepository · Free Software Foundation · GNU Savannah · GitHost  · GitHub · GitHub Downloads · Gitorious · Gna! · Google Code · ibiblio · java.net · JavaForge · KnowledgeForge · Launchpad · LuaForge · Maemo · mozdev · OSOR.eu · OW2 Consortium · Openmoko · OpenSolaris · Ourproject.org · Ovi Store · Project Kenai · RubyForge · SEUL.org · SourceForge · Stypi · TestFlight · tigris.org · Transifex · TuxFamily · Yahoo! Downloads

Television/Radio

ABC · Austin City Limits · BBC · CBC · CBS · Computer Chronicles · CTV · Fox · G4 · Global TV · Jeopardy! · NBC · NHK · PBS · Penn & Teller: Bullshit! · The Howard Stern Show · TV News Archive (Understanding 9/11)

Torrenting/Piracy

ExtraTorrent · EZTV · isoHunt · KickassTorrents · The Pirate Bay · Torrentz · Library Genesis

Video hosting

Academic Earth · Bambuser · Blip.tv · Epic · Freshlive · Google Video · Justin.tv · Mixer · Niconico · Nokia Trailers · Oddshot.tv · Periscope · Plays.tv · Qwiki · Skillfeed · Stickam · TED Talks · Ticker.tv · Twitch.tv · Ustream · Videoplayer.hu · Viddler · Viddy · Vidme · Vimeo · Vine · Vstreamers · Yahoo! Video · YouTube · Famous Internet videos (Me at the zoo)

Web hosting

Angelfire · Brace.io · BT Internet · CableAmerica Personal Web Space · Claranet Netherlands Personal Web Pages · Comcast Personal Web Pages · Extra.hu · FortuneCity · Free ProHosting · GeoCities (patch· Google Business Sitebuilder · Google Sites · Internet Centrum · MBinternet · MSN TV · Nifty · Nwnyet · Parodius Networking · Prodigy.net · Saunalahti Iso G · Swipnet · Telenor · Tripod · University of Michigan personal webpages · Verizon Mysite · Verizon Personal Web Space · Webs · Webzdarma · Virgin Media

Web applications

Mailman · MediaWiki · phpBB · Simple Machines Forum · vBulletin

Information

A Million Ways to Die on the Web · Backup Tips · Cheap storage · Collecting items randomly · Data compression algorithms and tools · Dev · Discovery Data · DOS Floppies · Fortress of Solitude · Keywords · Naughty List · Nightmare Projects · Rescuing floppy disks · Rescuing optical media · Site exploration · The WARC Ecosystem · Working with ARCHIVE.ORG

Projects

ArchiveCorps · Audit2014 · Emularity · Faceoff · FlickrFckr · Froogle · INTERNETARCHIVE.BAK (Internet Archive Census· IRC Quotes · JSMESS · JSVLC · Just Solve the Problem · NewsGrabber · Project Newsletter · Valhalla · Web Roasting (ISP Hosting · University Web Hosting· Woohoo

Tools

ArchiveBot · ArchiveTeam Warrior (Tracker· Google Takeout · HTTrack · Video downloaders · Wget (Lua · WARC)

Teams

Bibliotheca Anonoma · LibreTeam · URLTeam · Yahoo Video Warroom · WikiTeam

Other

800notes · AOL · Akoha · Ancestry.com · April Fools' Day · Amplicate · AutoAdmit · Bre.ad · Circavie · Cobook · Co.mments · Countdown · Discourse · Distill · Dmoz · Easel · Eircode · Electronic Frontier Foundation · FanFiction.Net · Feedly · Ficlets · Forrst · FunnyExam.com · FurAffinity · Google Helpouts · Google Moderator · Google Poly · Google Reader · ICQmail · IFTTT · Jajah · JuniorNet · Lulu Poetry · Mobile Phone Applications · Mochi Media · Mozilla Firefox · MyBlogLog · NBII · Newgrounds · Neopets · Quantcast · Quizilla · Salon Table Talk · Shutdownify · Slidecast · Stack Overflow · SOPA blackout pages · starwars.yahoo.com · TechNet · Toshiba Support · USA-Gov · Volán · Widgetbox · Windows Technical Preview · Wunderlist · YTMND · Zoocasa

About Archive Team

Introduction · Philosophy · Who We Are · Our stance on robots.txt · Why Back Up? · Software · Formats · Storage Media · Recommended Reading · Films and documentaries about archiving · Talks · In The Media · FAQ