Yahoo! Video Warroom

From Archiveteam
Revision as of 21:59, 23 February 2011 by Chronomex (talk | contribs) (Phase Two: Video Acquisition: Completed a block of videos.)
Jump to: navigation, search

Planning room for Yahoo! Video.


Yahoovideo.jpg

Phase One: Username Acquisition

"You people are working so hard!"

Acquiring the usernames and which have videos is done by the script yvscan. Ask Jason for a copy. The process is somewhat slow, as going too fast gets the dreaded 999. As a result, we need a large number of people and machines doing the work, so we have to put them here to keep track of who's doing what. The scraper pulls in userpics, a copy of the user page, and generates a userlist.txt of all the accounts it has found.

There are roughly 9,360,000 user accounts on Yahoo Video. (Let us not fool ourselves as to thinking these are real people.) Some are "Networks", i.e. collections of other videos (but not many, maybe 1 out of every 50,000, or less.) Most have no videos at all, but to make a hitlist of videos to download, we need those.

When You Finish

In these instructions, 0_000_000 is the beginning number of your block.

Zip up the files you downloaded:

tar cjvf 0_000_000-to-0_050_000.tar.bz2 userpics userfiles

Name the userlist.txt and networklist.txt (if it exists) to something unique, then compress:

mv userlist.txt userlist_0_000_000.txt
mv networklist.txt networklist_0_000_000.txt
bzip2 userlist_0_000_000.txt
bzip2 networklist_0_000_000.txt

Email the user and network lists to { Duncan at xrtc.net }. They should take up just a few megabytes in total. If you have a website with enough space, please put the .tar.bz2 there and include the url in the mail. Otherwise, ask for assistance.

Summary

  • 0,000,000 - 4,500,000 Jason Scott, Chronomex, Joey, Underscor, Soultcer
  • 4,750,000 - 5,300,000 Jason Scott, Underscor
  • 5,500,000 - 6,000,000 Ymgve, Underscor
  • 7,500,000 - 9,360,000 Underscor, Jason Scott, Spirit, Sungo

Finished

  • 0,000,000 - 0,750,000 Jason Scott, Chronomex
  • 0,750,000 - 1,000,000 Chronomex
  • 1,000,000 - 1,750,000 Joey
  • 1,750,000 - 2,000,000 Underscor
  • 2,000,000 - 2,250,000 Joey
  • 2,250,000 - 2,500,000 Underscor/Chronomex (Chronomex is finishing)
  • 2,500,000 - 2,750,000 Soultcer
  • 2,750,000 - 3,000,000 Underscor
  • 3,000,000 - 3,250,000 Soultcer
  • 3,250,000 - 3,500,000 Underscor
  • 3,500,000 - 4,000,000 Chronomex
  • 4,000,000 - 4,250,000 Chronomex (gir)
  • 4,250,000 - 4,500,000 Chronomex (dedi)
  • 4,750,000 - 5,000,000 Underscor
  • 5,000,000 - 5,300,000 Jason Scott
  • 5,500,000 - 5,750,000 Ymgve
  • 5,750,000 - 6,000,000 Underscor
  • 7,500,000 - 8,000,000 Jason Scott
  • 8,000,000 - 8,250,000 Spirit
  • 8,250,000 - 8,500,000 Sungo
  • 8,500,000 - 8,750,000 Sungo
  • 8,750,000 - 9,000,000 Spirit
  • 9,000,000 - 9,360,000 Jason Scott/Underscor/Ymgve

Continuing

  • 4,500,000 - 4,625,000 Underscor (dns1)
  • 4,625,000 - 4,750,000 Underscor (bozoc6)
  • 5,250,000 - 5,500,000 Jason Scott/Spirit (Spirit is finishing)
  • 6,000,000 - 6,050,000 Underscor (server down)
  • 6,050,000 - 6,100,000 robzy
  • 6,100,000 - 6,150,000 robzy
  • 6,150,000 - 6,200,000 phuz
  • 6,200,000 - 6,250,000 cb43569
  • 6,250,000 - 6,300,000 phuz
  • 6,300,000 - 6,350,000 Chronomex
  • 6,350,000 - 6,400,000 NovaKing
  • 6,400,000 - 6,450,000 NovaKing
  • 6,450,000 - 6,500,000 NovaKing
  • 6,500,000 - 6,550,000 phuz
  • 6,550,000 - 6,600,000 phuz
  • 6,600,000 - 6,650,000 phuz
  • 6,650,000 - 6,700,000 NovaKing
  • 6,700,000 - 6,750,000 NovaKing
  • 6,750,000 - 7,000,000 chronomex (dante01)
  • 7,000,000 - 7,250,000 chronomex (dante02)
  • 7,250,000 - 7,500,000 chronomex (dante03)

Phase Two: Video Acquisition

"Oh my god, some of these videos!"

Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called grapefruit (the g and fruit are silent, ræp) that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.

We have no idea where we're going to put this. We're going to need a bigger boat.

Finished

  • 0,100,000 - 0,200,000 (6,247 Users, x videos, 159gb) Underscor
  • 0,200,000 - 0,300,000 (6,461 Users, x videos, 141gb) Underscor
  • 0,400,000 - 0,500,000 (6,231 Users, x videos, 159gb) Underscor
  • 0,500,000 - 0,600,000 (6,543 Users, 13,849 videos, 115gb) NovaKing
  • 0,750,000 - 0,800,000 (3,735 Users, 1,097 videos, 7gb) atg (Use as test)
  • 1,100,000 - 1,200,000 (7,613 Users, 12,892 videos, 177gb ) NovaKing
  • 2,100,000 - 2,200,000 (4,519 Users) Joey (rsyncing)
  • 2,500,000 - 2,600,000 (3,949 Users, 7,849 videos, 82gb) Zumthing
  • 3,000,000 - 3,100,000 (6,297 Users, 18,418 videos, 163gb) Guan
  • 3,100,000 - 3,200,000 (5,760 Users, 15,661 videos, 186gb) Masterono
  • 3,600,000 - 3,700,000 (4,494 Users) chronomex: gir
  • 5,150,000 - 5,200,000 (3,313 Users, 18,127 videos, 235gb) Guan
  • 7,500,000 - 7,600,000 (4,347 Users, 20,418 videos, 240gb) Guan
  • 7,600,000 - 7,700,000 (4,200 Users, 17,885 videos, 189gb) NovaKing
  • 7,700,000 - 7,800,000 (4,028 Users, 15,037 videos, 171gb) K4K
  • 7,900,000 - 8,000,000 (3,517 Users, 10,392 videos, 111gb) matzie
  • 9,000,000 - 9,050,000 (1,745 Users) atg
  • 9,050,000 - 9,100,000 (1,796 Users, 6,142 videos, 79gb) Masterono
  • 9,150,000 - 9,200,000 (5,173 Users, 22,655 videos, 261gb) Underscor
  • 9,200,000 - 9,300,000 (8,941 Users, 30,179 videos, 341gb) Zumthing (overlaps with drspangle)

In-Process

LEGEND.

  • "???" = unassigned
  • "OPEN" = unassigned, hotlist available

And now the list.

  • 0,000,000 - 0,100,000 (72,176 Users) Underscor 0-10k Done
  • 0,300,000 - 0,400,000 (6,909 Users) Underscor Rebuilding
  • 0,600,000 - 0,700,000 (7,833 Users) Underscor Rebuilding
  • 0,700,000 - 0,750,000 (3,213 Users) Underscor Running
  • 0,800,000 - 0,850,000 (3,140 Users) Crust
  • 0,850,000 - 0,900,000 (3,479 Users) filer
  • 0,900,000 - 1,000,000 (6,453 Users) K4K
  • 1,000,000 - 1,100,000 (6,331 Users) Zumthing
  • 1,200,000 - 1,300,000 (7,130 Users) drspangle
  • 1,300,000 - 1,400,000 (7,598 Users) sep332
  • 1,400,000 - 1,500,000 (5,194 Users) beardicus
  • 1,500,000 - 1,600,000 (5,252 Users) NovaKing
  • 1,600,000 - 1,700,000 (4,933 Users) Masterono
  • 1,700,000 - 1,800,000 (4,368 Users) atg
  • 1,800,000 - 1,900,000 (4,744 Users) bluknight (sandra)
  • 1,900,000 - 2,000,000 (4,481 Users) ajclement
  • 2,000,000 - 2,100,000 (4,408 Users) dashcloud
  • 2,200,000 - 2,300,000 (4,216 Users) OPEN
  • 2,300,000 - 2,400,000 (4,615 Users) OPEN
  • 2,400,000 - 2,500,000 (2,802 Users) OPEN
  • 2,600,000 - 2,700,000 (4,208 Users) Frigolit
  • 2,700,000 - 2,800,000 (4,754 Users) ersi
  • 2,900,000 - 3,000,000 (6,221 Users) OPEN
  • 3,200,000 - 3,250,000 (2,603 Users) Soultcer
  • 3,250,000 - 3,300,000 (2,439 Users) OPEN
  • 3,300,000 - 3,400,000 (4,252 Users) OPEN
  • 3,400,000 - 3,500,000 (4,560 Users) OPEN
  • 3,700,000 - 3,800,000 (4,262 Users) OPEN
  • 3,800,000 - 3,900,000 (3,936 Users) dysproseu
  • 3,900,000 - 4,000,000 (4,055 Users) HA
  • 4,000,000 - 4,100,000 ???
  • 4,100,000 - 4,200,000 ???
  • 4,200,000 - 4,300,000 ???
  • 4,300,000 - 4,400,000 ???
  • 4,400,000 - 4,500,000 ???
  • 4,600,000 - 4,700,000 ???
  • 4,700,000 - 4,800,000 ???
  • 4,800,000 - 4,900,000 (5,764 Users) monst3r_
  • 4,900,000 - 5,000,000 (6,738 Users) jch
  • 5,000,000 - 5,100,000 (6,545 Users) monst3r_
  • 5,100,000 - 5,150,000 (3,254 Users) Spirit
  • 5,200,000 - 5,300,000 ???
  • 5,300,000 - 5,400,000 ???
  • 5,400,000 - 5,500,000 ???
  • 5,600,000 - 5,700,000 (5,903 Users) _kp
  • 5,700,000 - 5,800,000 ???
  • 5,800,000 - 5,900,000 ???
  • 5,900,000 - 6,000,000 ???
  • 6,000,000 - 6,100,000 ???
  • 6,100,000 - 6,200,000 ???
  • 6,200,000 - 6,300,000 ???
  • 6,300,000 - 6,400,000 ???
  • 6,400,000 - 6,500,000 ???
  • 6,500,000 - 6,600,000 ???
  • 6,600,000 - 6,700,000 ???
  • 6,700,000 - 6,800,000 ???
  • 6,800,000 - 6,900,000 ???
  • 6,900,000 - 7,000,000 ???
  • 7,000,000 - 7,100,000 ???
  • 7,100,000 - 7,200,000 ???
  • 7,200,000 - 7,300,000 ???
  • 7,300,000 - 7,400,000 ???
  • 7,400,000 - 7,500,000 ???
  • 7,500,000 - 7,600,000 (4,347 Users) sadcarrot
  • 7,700,000 - 7,800,000 (4,028 Users) OPEN
  • 7,800,000 - 7,900,000 (3,720 Users) OPEN
  • 8,000,000 - 8,100,000 (3,416 Users) bluknight (tara)
  • 8,100,000 - 8,200,000 (3,380 Users) Primus104
  • 8,200,000 - 8,300,000 (3,335 Users) OPEN
  • 8,300,000 - 8,400,000 (2,920 Users) OPEN
  • 8,400,000 - 8,500,000 (3,535 Users) yoyoguy2
  • 8,500,000 - 8,600,000 (3,516 Users) red_cell
  • 8,600,000 - 8,700,000 (7,726 Users) Joey (slowbox)
  • 8,700,000 - 8,800,000 (3,730 Users) t0phux
  • 8,800,000 - 8,850,000 (2,170 Users) sdboyd
  • 8,850,000 - 9,000,000 (1,770 Users) Gandalfar
  • 8,900,000 - 9,000,000 (3,793 Users) khaoohs
  • 9,100,000 - 9,150,000 (1,989 Users) Killface
  • 9,250,000 - 9,300,000 (2,835 Users) drspangle (overlaps with Zumthing)
  • 9,300,000 - 9,360,000 (2,179 Users) Robertcu

Phase Three: Final Resting Place

In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our very rough estimate is that a download of Yahoo! Video is going to be 25 terabytes. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".

Jason and a few others have locations that video downloaders can upload a copy of their acquired data to; these are being placed on a stack of hard drives. At the same time, Archive.org is providing a location for the data to be uploaded to as well. Other entities are being contacted at the present about providing space for copies of the full archive, as well.

The priority, however, is to get all the data in the first place.

Statistics

"Why didn't more people use it! It was awesome!"

People like easy to quote statistics from a project, so here are some, with others coming soon.

Yahoo! Video officially started as a user content system in June of 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.

However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.

It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on Youtube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.