Difference between revisions of "Ispygames"

From Archiveteam
Jump to navigation Jump to search
Line 88: Line 88:
* http://apassey.dev.m.ign.com
* http://apassey.dev.m.ign.com
* http://au.bestof.ign.com
* http://au.bestof.ign.com
* http://au.media.ign.com
* http://au.microsites.ign.com
* http://au.microsites.ign.com
* http://au.retro.ign.com
* http://au.retro.ign.com
Line 100: Line 99:
* http://broadband.ign.com
* http://broadband.ign.com
* http://browserthemes.ign.com
* http://browserthemes.ign.com
* http://carsmedia.ign.com
* http://carsmovies.ign.com
* http://ces2009.ign.com
* http://ces2009.ign.com
* http://c.ffmovies.ign.com
* http://championshipgamingseries.ign.com
* http://championshipgamingseries.ign.com
* http://championsonline.ign.com
* http://championsonline.ign.com
* http://code.ign.com
* http://code.ign.com
* http://codesmedia.ign.com
* http://codesmedia.ign.com
* http://cohvault.ign.com
* http://cohvault.ign.com
* http://comiccon.ign.com
* http://comiccon.ign.com
Line 116: Line 110:
* http://crose.dev.m.ign.com
* http://crose.dev.m.ign.com
* http://crossassault.ign.com
* http://crossassault.ign.com
* http://cubemedia.ign.com
* http://dcmedia.ign.com
* http://design.ign.com
* http://design.ign.com
* http://download.ign.com
* http://download.ign.com
Line 125: Line 117:
* http://dsvault.ign.com
* http://dsvault.ign.com
* http://emailpreferences.ign.com
* http://emailpreferences.ign.com
* http://entertainmentmedia.ign.com
* http://entertainmentmedia.ign.com
* http://eq2vault.ign.com
* http://eq2vault.ign.com
* http://eqvault.ign.com
* http://eqvault.ign.com
* http://esports.ign.com
* http://esports.ign.com
* http://evevault.ign.com
* http://evevault.ign.com
* http://faqsmedia.ign.com
* http://faqsmedia.ign.com
* http://faqsmovies.ign.com
* http://faqsmovies.ign.com
* http://feeds.ign.com
* http://feeds.ign.com
* http://ffmedia.ign.com
* http://ffvault.ign.com
* http://ffvault.ign.com
* http://findit.ign.com
* http://findit.ign.com
* http://formenmedia.ign.com
* http://gamechanger.ign.com
* http://gamechanger.ign.com
* http://gamesites.ign.com
* http://gamesites.ign.com
Line 154: Line 138:
* http://gta.ign.com
* http://gta.ign.com
* http://guidesarchive.ign.com
* http://guidesarchive.ign.com
* http://guidesmedia.ign.com
* http://guitarhero3.ign.com
* http://guitarhero3.ign.com
* http://gwvault.ign.com
* http://gwvault.ign.com
Line 162: Line 145:
* http://horizonsvault.ign.com
* http://horizonsvault.ign.com
* http://ie.bestof.ign.com
* http://ie.bestof.ign.com
* http://ie.media.ign.com
* http://ie.top100.ign.com
* http://ie.top100.ign.com
* http://insdermedia.ign.com
* http://insiderdownloads.ign.com
* http://insidermedia.ign.com
* http://ipl.ign.com
* http://ipl.ign.com
* http://iplstore.ign.com
* http://iplstore.ign.com
Line 181: Line 160:
* http://lotrovault.ign.com
* http://lotrovault.ign.com
* http://mac.ign.com  
* http://mac.ign.com  
* http://macmedia.ign.com
* http://macmovies.ign.com
* http://mag.ign.com
* http://mag.ign.com
* http://mail2.ign.com
* http://mail2.ign.com
Line 188: Line 165:
* http://m.ca.ign.com
* http://m.ca.ign.com
* http://mcraft.ign.com
* http://mcraft.ign.com
* http://media.ign.com
* http://memoviedia.ign.com
* http://media.ign.com
* http://memoviedia.ign.com  
* http://mevault.ign.com
* http://mevault.ign.com
* http://m.ie.ign.com
* http://m.ie.ign.com
Line 197: Line 172:
* http://mobile.ign.com
* http://mobile.ign.com
* http://mobile.ign.com  
* http://mobile.ign.com  
* http://moviemedia.ign.com
* http://moviesmovies.ign.com
* http://m.uk.ign.com
* http://m.uk.ign.com
* http://musichub.ign.com
* http://musichub.ign.com
Line 209: Line 182:
* http://niboppub.ign.com
* http://niboppub.ign.com
* http://nwvault.ign.com
* http://nwvault.ign.com
* http://o.faqsmedia.ign.com
* http://o.faqsmovies.ign.com
* http://o.ffmovies.ign.com
* http://o.guidesarchive.ign.com
* http://o.guidesarchive.ign.com
* http://o.guidesmedia.ign.com
* http://o.insidermedia.ign.com
* http://o.media.ign.com
* http://o.mobile.ign.com
* http://o.mobile.ign.com
* http://o.moviesmovies.ign.com
* http://o.pcmedia.ign.com
* http://open.em.ign.com
* http://open.em.ign.com
* http://o.pocketmedia.ign.com
* http://o.ps2media.ign.com
* http://opt-out.emailpreferences.ign.com
* http://opt-out.emailpreferences.ign.com
* http://o.rpgvaultarchive.ign.com
* http://o.rpgvaultarchive.ign.com
* http://overlord.ign.com
* http://overlord.ign.com
* http://o.xboxmedia.ign.com
* http://pawong.dev.www.ign.com
* http://pawong.dev.www.ign.com
* http://pcmedia.ign.com
* http://pcmedia.ign.com
* http://planetelderscrolls.ign.com
* http://planetelderscrolls.ign.com
* http://play.ign.com
* http://play.ign.com
* http://pocketmedia.ign.com
* http://pocketmedia.ign.com
* http://podcast.ign.com
* http://podcast.ign.com
* http://podcasts.ign.com
* http://podcasts.ign.com
Line 239: Line 197:
* http://promotions.ign.com
* http://promotions.ign.com
* http://promotools.ign.com
* http://promotools.ign.com
* http://ps2media.ign.com
* http://ps2media.ign.com
* http://psxmedia.ign.com
* http://psxmedia.ign.com
* http://psxmovies.ign.com
* http://psxmovies.ign.com
* http://publish.ign.com
* http://publish.ign.com
* http://rift.ign.com
* http://rift.ign.com
Line 257: Line 209:
* http://ryzomvault.ign.com
* http://ryzomvault.ign.com
* http://sbvault.ign.com
* http://sbvault.ign.com
* http://scifimedia.ign.com
* http://scifimedia.ign.com
* http://s.faqsmovies.ign.com
* http://s.ffmovies.ign.com
* http://share.affiliation.com  
* http://share.affiliation.com  
* http://share.ign.com
* http://share.ign.com
Line 268: Line 216:
* http://smashbros.ign.com
* http://smashbros.ign.com
* http://smcnabb.dev.www.ign.com
* http://smcnabb.dev.www.ign.com
* http://s.moviesmovies.ign.com
* http://sovault.ign.com
* http://sovault.ign.com
* http://speedtv.ign.com
* http://speedtv.ign.com
* http://s.psxmovies.ign.com
* http://sslvpn.ign.com
* http://sslvpn.ign.com
* http://staging-api.ign.com
* http://staging-api.ign.com
Line 284: Line 230:
* http://swgvault.ign.com
* http://swgvault.ign.com
* http://swvault.ign.com
* http://swvault.ign.com
* http://s.xboxmovies.ign.com
* http://tabularasavault.ign.com
* http://tabularasavault.ign.com
* http://tdu.ign.com
* http://tdu.ign.com
Line 305: Line 250:
* http://uk.bestof.ign.com
* http://uk.bestof.ign.com
* http://uk.corp.ign.com
* http://uk.corp.ign.com
* http://uk.media.ign.com
* http://uk.retro.ign.com
* http://uk.retro.ign.com
* http://uk.sports.ign.com
* http://uk.sports.ign.com
Line 316: Line 260:
* http://v3-api.stg.www.ign.com
* http://v3-api.stg.www.ign.com
* http://vanguardvault.ign.com
* http://vanguardvault.ign.com
* http://vaultmedia.ign.com
* http://vaultmedia.ign.com
* http://ve3d.ign.com - grabbing
* http://ve3d.ign.com - grabbing
* http://vgu.stg.www.ign.com
* http://vgu.stg.www.ign.com
Line 329: Line 271:
* http://wikihub.stg.www.ign.com
* http://wikihub.stg.www.ign.com
* http://wiki.stg.www.ign.com
* http://wiki.stg.www.ign.com
* http://wiremedia.ign.com
* http://wiremedia.ign.com
* http://wiremovies.ign.com
* http://wiremovies.ign.com
* http://wishvault.ign.com
* http://wishvault.ign.com
* http://witchervault.ign.com
* http://witchervault.ign.com
Line 343: Line 281:
* http://www.supersmashbros.ign.com
* http://www.supersmashbros.ign.com
* http://xboxlive.ign.com
* http://xboxlive.ign.com
=== These might be asset only hosting sites ===
* http://au.media.ign.com
* http://carsmedia.ign.com
* http://carsmovies.ign.com
* http://codesmedia.ign.com
* http://cubemedia.ign.com
* http://dcmedia.ign.com
* http://entertainmentmedia.ign.com
* http://faqsmedia.ign.com
* http://faqsmovies.ign.com
* http://ffmedia.ign.com
* http://formenmedia.ign.com
* http://guidesmedia.ign.com
* http://ie.media.ign.com
* http://insdermedia.ign.com
* http://insiderdownloads.ign.com
* http://insidermedia.ign.com
* http://macmedia.ign.com
* http://macmovies.ign.com
* http://media.ign.com
* http://moviemedia.ign.com
* http://moviesmovies.ign.com
* http://o.faqsmedia.ign.com
* http://o.faqsmovies.ign.com
* http://o.ffmovies.ign.com
* http://o.guidesmedia.ign.com
* http://o.insidermedia.ign.com
* http://o.media.ign.com
* http://o.moviesmovies.ign.com
* http://o.pcmedia.ign.com
* http://o.pocketmedia.ign.com
* http://o.ps2media.ign.com
* http://o.xboxmedia.ign.com
* http://xboxlivemedia.ign.com
* http://xboxlivemedia.ign.com
* http://xboxlivemovies.ign.com
* http://xboxlivemovies.ign.com
* http://xboxmedia.ign.com
* http://xboxmedia.ign.com
* http://xboxmovies.ign.com
* http://xboxmovies.ign.com
* http://pcmedia.ign.com
* http://pocketmedia.ign.com
* http://ps2media.ign.com
* http://psxmedia.ign.com
* http://psxmovies.ign.com
* http://scifimedia.ign.com
* http://uk.media.ign.com
* http://vaultmedia.ign.com
* http://wiremedia.ign.com
* http://wiremovies.ign.com
* http://c.ffmovies.ign.com
* http://s.faqsmovies.ign.com
* http://s.ffmovies.ign.com
* http://s.moviesmovies.ign.com
* http://s.psxmovies.ign.com
* http://s.xboxmovies.ign.com


=== Redirects ===
=== Redirects ===

Revision as of 12:05, 2 April 2013

The News

IGN hit with layoffs, 1UP, UGO and GameSpy shutting down
1UP, UGO and GameSpy to be shut down

The Problems

  • Once you start digging around these sites you find it to be a mess of inconsistent url schemes and content everywhere.
  • Some files are being hosted on MediaFire.
  • Based on tests the larger and older a site is the more that is missed by a wget crawl due to the url scheme.

What we know

  • We already have a list of almost all the domains involved
  • A clean list with dups and bad domains is already being process and will be posted here when complete.
  • Most of the sites are not that big, but a few are huge.

The plan

  • Save the sites and related content
  • Backup the twitter feeds for any associated accounts. All my tweets just takes a username and returns the max tweets possible.


wget test command

This if for the gamespy sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://planetdoom.gamespy.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST" \
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com

Try this for the ign, ugo sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://ve3d.ign.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST"

IGN domains

Ready to grab

untested

These might be asset only hosting sites

Redirects

Gamespy Domains

Ready to grab

In Progress