Difference between revisions of "Twitch.tv"

From Archiveteam
Jump to navigation Jump to search
m (fix)
(19 intermediate revisions by 6 users not shown)
Line 15: Line 15:
Twitch was rumored to have been acquired by [[YouTube]]/[[Google]] but [[Amazon]] was the final buyer.<ref>http://www.twitch.tv/p/thankyou</ref>
Twitch was rumored to have been acquired by [[YouTube]]/[[Google]] but [[Amazon]] was the final buyer.<ref>http://www.twitch.tv/p/thankyou</ref>


== Shutdown ==
== Broadcast retention changes ==
 
After Twitch's acquisition by Amazon, changes were made to how long broadcasts (sometimes called VODs) were retained for viewing on the site. Previously, all partnered accounts had indefinite storage, while standard accounts had storage for a few days. This was then cut down to two months.


<blockquote>
<blockquote>
Line 44: Line 46:


</blockquote>
</blockquote>
Thus a mission was started to archive as much of Twitch as we reasonably could. Our efforts can be found in [[Twitch.tv/Warroom|the War Room]] and on the [[Talk:Twitch.tv|Talk page]].
== Known exceptions ==
Although most partner accounts have broadcasts deleted after 60 days, there are some exceptions. Most of these are esports tournament channels, but other channels may be excluded for reasons such as being culturally significant. Below is an incomplete list of these exceptions.
{| class="wikitable"
|-
! Channel !! Indefinite since !! Probable reason
|-
| [https://www.twitch.tv/dota2ti Dota 2 The International] || Policy Inception || Esports tournament
|-
| [https://www.twitch.tv/riotgames Riot Games] || Policy Inception || Esports tournaments
|-
| [https://www.twitch.tv/beyondthesummit/ Beyond The Summit] || Policy Inception || Esports broadcaster
|-
| [https://www.twitch.tv/twitchplayspokemon Twitch Plays Pokemon] || Policy Inception || Culturally significant
|-
| [https://www.twitch.tv/joindotared/ joinDOTA Red], [https://www.twitch.tv/joindotablue/ Blue], [https://www.twitch.tv/joindotacommunity/ Community] || Policy Inception || Esports broadcaster
|-
| [https://www.twitch.tv/gamesdonequick/ Games Done Quick] || Policy Inception || Culturally significant (charity fundraiser)
|-
| [https://www.twitch.tv/tipofthehats/ Tip of the Hats] || Policy Inception || Culturally significant (charity fundraiser)
|-
| [https://www.twitch.tv/rocketleague/ Rocket League] || April 2016 || Game channel & esports tournaments
|-
| [https://www.twitch.tv/Evo Evo] || Policy Inception || Esports tournament
|-
| [https://www.twitch.tv/esl_csgo/ ESL CS:GO] || Policy Inception || Esports broadcaster
|-
| [https://www.twitch.tv/dreamhackcs/ DreamHack Counter-Strike] || Policy Inception || Esports & LAN event broadcaster
|-
| [https://www.twitch.tv/shroud/ shroud] || Policy Inception? || Notable streamer & esports personality
|-
| [https://www.twitch.tv/ninja/ Ninja] || January 2018 || Notable streamer
|-
| [https://www.twitch.tv/starcraft/ StarCraft] || Policy Inception || Game channel & esports tournaments
|-
| [https://www.twitch.tv/playoverwatch/ PlayOverwatch] || August 2017? || Game channel & esports tournaments
|-
| [https://www.twitch.tv/playhearthstone/ PlayHearthstone] || Policy Inception || Game channel & esports tournaments
|-
| [https://www.twitch.tv/faceittv/ FACEIT TV] || Policy Inception || Esports service & tournaments
|-
| [https://www.twitch.tv/teamfortresstv/ teamfortress.tv] || Policy Inception || Esports broadcaster
|-
| [https://www.twitch.tv/bobross/ Bob Ross] || June 2016 || TV show & culturally significant
|-
| [https://www.twitch.tv/pgl/ PGL] || Policy Inception || Esports tournament organizer
|-
| [https://www.twitch.tv/notch/ Notch] || Policy Inception || Minecraft creator
|-
| [https://www.twitch.tv/deadmau5 deadmau5] || Policy Inception || Musician
|-
| [https://www.twitch.tv/mojang Mojang] || Policy Inception || Game developers, including Minecraft
|-
| [https://www.twitch.tv/yogscast Yogscast] || Policy Inception || Culturally significant
|-
| [https://www.twitch.tv/twitch Twitch] || Policy Inception || The site (duh)
|-
| [https://www.twitch.tv/twitchpresents Twitch Presents] || March 2017 || Also the site, special event streams
|-
| [https://www.twitch.tv/thegameawards The Game Awards] || 2016? || Awards show
|-
| [https://www.twitch.tv/reddit reddit] || 2017 || Website
|-
| [https://www.twitch.tv/xbox Xbox] || Policy Inception || Console platform
|-
| [https://www.twitch.tv/playstation PlayStation] || Policy Inception || Console platform
|-
| [https://www.twitch.tv/nintendo Nintendo] || Policy Inception || Game developer & console manufacturer
|-
| [https://www.twitch.tv/pax PAX] || Policy Inception || Gaming convention
|-
| [https://www.twitch.tv/ign IGN] || Policy Inception || Magazine & website
|-
| [https://www.twitch.tv/pokerstars PokerStars] || Policy Inception || Poker website & tournaments
|}


== Site structure ==
== Site structure ==
Line 57: Line 138:
* Scraping: https://api.twitch.tv/kraken/videos/top?limit=20&offset=0&period=all
* Scraping: https://api.twitch.tv/kraken/videos/top?limit=20&offset=0&period=all
* Is there any irregularities? Differences between highlights and past broadcasts?
* Is there any irregularities? Differences between highlights and past broadcasts?
=== Storage Issues ===
* How to decide which are important? 10+ views again? Do a discovery crawl first?
* Tahoe-LAFS? Grab ''all'' the videos into temp storage?
* Compress all the unwatched videos into postage stamp sized videos?
=== Discovery ===
All items discovered are located at [https://github.com/ArchiveTeam/twitchtv-items twitchtv-items]. A collated [https://archive.org/details/twitchtv_scrape_dataset_2014-09-05 JSON dump is available.]
== How can I help? ==
Download and fire up your [[warrior]]! Then select Twitch Phase 2. Better yet, select Archive Team's Choice.
Alternatively for advanced users, you can run the scripts manually see below.
Don't forget to '''[https://archive.org/donate/ donate to the Internet Archive]''' who will be hosting these files. Disk space is cheap but maintaining them is not!
=== For those not using the Warrior ===
''Advanced User Quick Start''
Please run these sysctl tweaks to optimize uploads:
<pre>
# Add to /etc/sysctl.conf and run "sysctl -p"
# increase TCP max buffer size settable using setsockopt()
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limit
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
</pre>
You can also issue them without modifying /etc/sysctl.conf by running e.g. <pre>sysctl net.core.rmem_max=16777216 net.core.wmem_max=16777216</pre>, but be aware that those won't stick around across reboots.
<pre>apt-get install git git-core libgnutls-dev lua5.1 liblua5.1-0 liblua5.1-0-dev screen python-dev python-pip bzip2 zlib1g-dev</pre>
<pre>git clone https://github.com/ArchiveTeam/twitchtv-grab
cd ./twitchtv-grab
pip install seesaw
./get-wget-lua.sh
...
pip install requests</pre>
wget-lua may of failed earlier, if so then:
<pre>cd get-wget-lua.tmp
mv src/wget ../wget-lua
cd ..</pre>
And finally to actually run
<pre>run-pipeline pipeline.py --concurrent 2 YOURNICKHERE --disable-web-server</pre>
For troubleshooting and the details please see [https://github.com/ArchiveTeam/twitchtv-grab/blob/master/README.md README].
=== What we are saving ===
Currently:
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/01_twitchplayspokemon.txt twitchplayspokemon]: "test" run, estimated 3 TB ($6000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/02_suggestions_19553_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/01_suggestions_wiki-19553.txt these channels] with 100 or more views: estimated 23 TB ($46000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/03_suggestions_19553_100views_2.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/01_suggestions_wiki-19553.txt these channels] with 100 or more views: estimated 0.7 TB ($1400)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/04_suggestions_19616_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/02_suggestions_wiki-19616.txt these channels] with 100 or more views: estimated 4 TB ($8000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/05_top_videos_10000views.txt These top videos] which have 10000 or more views: estimated 20 TB ($40000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/06_suggestions_19795_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/03_suggestions_wiki-19795.txt these channels] with 100 or more views: estimated 8 TB ($16000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/07_socialblade_top_5000views.txt These videos] selected from SocialBlade's top Twitch channels with 5000 or more views: estimated 10 TB ($20000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/08_suggestions_19851_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/05_suggestions_wiki-19851.txt these channels] with 100 or more views: estimated 15 TB ($30000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/09_top_vid_per_suggestion_19553-19851.txt These videos], from previous suggestions, that are most viewed per channel: estimated 0.1TB ($200)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/10_suggestions_19874_100views.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/06_suggestions_wiki-19874.txt these channels] with 100 or more views: estimated 7 TB ($14000)
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/items/video_pages/11_suggestions_19911_100views_and_top1.txt These videos] selected from [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_suggestions/07_suggestions_wiki-19911.txt these channels] with 100 or more views or most viewed video per channel: estimated 1.6 TB ($3200)
Next:
* Sorry, no more suggestions! More suggestions may be considered if you [https://archive.org/donate/ donate to the Internet Archive].
Dollar figures shown to illustrate cost of permanent archives. These are not actual values but are meant to represent simplified values and act as a sane budget. Dollars in USD at $2000 per TB estimate (not per TB of disk space alone).
Channels not included:
* speeddemosarchivesda: already in IA
* vinesauce: avoiding duplication, see below
* [https://github.com/ArchiveTeam/twitchtv-items/blob/master/user_blacklists/user_blacklist.txt and others]
Anything culturally significant to add? Comment on [[Talk:Twitch.tv]]. Don't forget to sign your comments with <code><nowiki>~~~~</nowiki></code>
.


== Archives ==
== Archives ==
Line 170: Line 160:


== Butt Controversy ==
== Butt Controversy ==
* http://www.dailydot.com/esports/twitch-butt-emotes/
In June 2016, Twitch deleted a bunch of custom emoticons on the grounds of obscenity. (See {{url|http://www.dailydot.com/esports/twitch-butt-emotes/}} for details.)
<p>
 
Emotes can be found in the backend at:
The emotes can be found in the backend at: <nowiki>https://static-cdn.jtvnw.net/emoticons/v1/<id number>/<size></nowiki>
<pre>
where id number ranges from 1 to 103667(as of 20160622), with no leading zeroes, and size is 1.0, 2.0 or 3.0.
https://static-cdn.jtvnw.net/emoticons/v1/<id number>/<size>
 
</pre>
where id number ranges from 1 to 103598(as of 20160622) and size is 1.0, 2.0 or 3.0.
</p>
<p>
Note: sizes 0.5, 1.5, 2.5, 3.5, 4.0, 4.5 and 5.0 are valid as well, but return the same data for most(all?) emotes as the next highest available 'whole number' or the largest below that one, i.e. for 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, will match 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0 respectively)
Note: sizes 0.5, 1.5, 2.5, 3.5, 4.0, 4.5 and 5.0 are valid as well, but return the same data for most(all?) emotes as the next highest available 'whole number' or the largest below that one, i.e. for 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, will match 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0 respectively)
</p>
 
<p>
All emote graphics and sizes (but not their associated chat 'shortnames' i.e. "<3" for emote #9, which cannot be easily determined) still existing in the backend system up to emote id 103667 were archived through [[Archivebot]] [http://archive.fart.website/archivebot/viewer/job/b8fnv here]; the resulting WARCs can be downloaded through the viewer.
Archived all emote graphics and sizes (but not their associated chat 'shortnames' i.e. "<3" for emote #9, which cannot be easily determined) still existing in the backend system up to emote id 103598 as job ids dmginny6vayrkxs90ed2kgs5t and b953b4vdqo8m2yjp3wtunl8on
</p>


== See Also ==  
== See Also ==  

Revision as of 21:18, 13 October 2019

Twitch.tv
Twitch.tv logo
Twitch homepage screenshot.png
URL http://twitch.tv
Status Special case (archives of streams actively purged after an amount of time)
Archiving status Partially saved (popular videos only)
Archiving type Unknown
Project source Phase 1,Phase 2, Items, Index
Project tracker Phase 1, Phase 2
IRC channel #burnthetwitch (on hackint)

Justin.tv—sorry, cough, I mean to say—Twitch.tv is a live video streaming service.

Twitch was rumored to have been acquired by YouTube/Google but Amazon was the final buyer.[1]

Broadcast retention changes

After Twitch's acquisition by Amazon, changes were made to how long broadcasts (sometimes called VODs) were retained for viewing on the site. Previously, all partnered accounts had indefinite storage, while standard accounts had storage for a few days. This was then cut down to two months.

Changes To VODs On Twitch

Aug 06 2014 · Engineering, Tech

Our goal at Twitch is straightforward: deliver the highest quality video. This includes the ability to watch video on demand (VOD) on all of our platforms, not just the website.

In order to create a system that supports live and VOD across the globe and on multiple platforms, we need to make significant changes to the way we’re currently storing video. Today, we’d like to discuss what these changes are, why they’re necessary, and how they benefit the entire Twitch community now and in the future.

[...]

Looking at Viewership Data

We found that the vast majority of past broadcast views happen within the first two weeks after they’re created. On the days following, viewership reduces exponentially.

We also discovered that 80% of our storage capacity is filled with past broadcasts that are never watched. That’s multiple petabytes for video that no one has ever viewed.

Highlights, on the other hand, have much more value and longevity. Over their lifetime, highlights get 9x as many views as past broadcasts.

[...]

As for existing past broadcasts, beginning three weeks from today, we will begin removing them from Twitch servers. If you would like to keep your past broadcasts, we encourage you to begin exporting or making highlights of your best moments so that they’re saved for posterity.

[...][2]

Thus a mission was started to archive as much of Twitch as we reasonably could. Our efforts can be found in the War Room and on the Talk page.

Known exceptions

Although most partner accounts have broadcasts deleted after 60 days, there are some exceptions. Most of these are esports tournament channels, but other channels may be excluded for reasons such as being culturally significant. Below is an incomplete list of these exceptions.

Channel Indefinite since Probable reason
Dota 2 The International Policy Inception Esports tournament
Riot Games Policy Inception Esports tournaments
Beyond The Summit Policy Inception Esports broadcaster
Twitch Plays Pokemon Policy Inception Culturally significant
joinDOTA Red, Blue, Community Policy Inception Esports broadcaster
Games Done Quick Policy Inception Culturally significant (charity fundraiser)
Tip of the Hats Policy Inception Culturally significant (charity fundraiser)
Rocket League April 2016 Game channel & esports tournaments
Evo Policy Inception Esports tournament
ESL CS:GO Policy Inception Esports broadcaster
DreamHack Counter-Strike Policy Inception Esports & LAN event broadcaster
shroud Policy Inception? Notable streamer & esports personality
Ninja January 2018 Notable streamer
StarCraft Policy Inception Game channel & esports tournaments
PlayOverwatch August 2017? Game channel & esports tournaments
PlayHearthstone Policy Inception Game channel & esports tournaments
FACEIT TV Policy Inception Esports service & tournaments
teamfortress.tv Policy Inception Esports broadcaster
Bob Ross June 2016 TV show & culturally significant
PGL Policy Inception Esports tournament organizer
Notch Policy Inception Minecraft creator
deadmau5 Policy Inception Musician
Mojang Policy Inception Game developers, including Minecraft
Yogscast Policy Inception Culturally significant
Twitch Policy Inception The site (duh)
Twitch Presents March 2017 Also the site, special event streams
The Game Awards 2016? Awards show
reddit 2017 Website
Xbox Policy Inception Console platform
PlayStation Policy Inception Console platform
Nintendo Policy Inception Game developer & console manufacturer
PAX Policy Inception Gaming convention
IGN Policy Inception Magazine & website
PokerStars Policy Inception Poker website & tournaments

Site structure

Archives

By Archive Team

Archives will be made available later as WARC files in the archiveteam_twitchtv collection at the Internet Archive. You can access them by the Wayback Machine, but you'll need search an index to find the media files.

A work-in-progress searchable index is now available!

Renegade Stream Archives

These archives are made in a manual fashion through the efforts of streaming communities. Feel free to expand this list.

Other

Butt Controversy

In June 2016, Twitch deleted a bunch of custom emoticons on the grounds of obscenity. (See http://www.dailydot.com/esports/twitch-butt-emotes/[IAWcite.todayMemWeb] for details.)

The emotes can be found in the backend at: https://static-cdn.jtvnw.net/emoticons/v1/<id number>/<size> where id number ranges from 1 to 103667(as of 20160622), with no leading zeroes, and size is 1.0, 2.0 or 3.0.

Note: sizes 0.5, 1.5, 2.5, 3.5, 4.0, 4.5 and 5.0 are valid as well, but return the same data for most(all?) emotes as the next highest available 'whole number' or the largest below that one, i.e. for 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, will match 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0 respectively)

All emote graphics and sizes (but not their associated chat 'shortnames' i.e. "<3" for emote #9, which cannot be easily determined) still existing in the backend system up to emote id 103667 were archived through Archivebot here; the resulting WARCs can be downloaded through the viewer.

See Also

External links

References