Ispygames

The News

IGN hit with layoffs, 1UP, UGO and GameSpy shutting down
1UP, UGO and GameSpy to be shut down

The Problems

• Once you start digging around these sites you find it to be a mess of inconsistent url schemes and content everywhere.
• Some files are being hosted on MediaFire.
• Based on tests the larger and older a site is the more that is missed by a wget crawl due to the url scheme.

What we know

• We already have a list of almost all the domains involved
• A clean list with dups and bad domains is already being process and will be posted here when complete.
• Most of the sites are not that big, but a few are huge.

The plan

• Save the sites and related content
• Backup the twitter feeds for any associated accounts. All my tweets just takes a username and returns the max tweets possible.

wget test command

This if for the gamespy sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://planetdoom.gamespy.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST" \
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com

Try this for the ign, ugo sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://ve3d.ign.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST"

IGN domains

• http://aavault.ign.com - grabbed, checking for completeness
• http://autoassaultvault.ign.com/ - grabbed, checking for completeness
• http://beacon.snowball.com - already closed
• http://bombergirl.ign.com - closed contest page in flash
• http://o.cubemedia.ign.com - dead end
• http://actionunleashed.ign.com - grabbed, checking for completeness
• http://dndvault.ign.com - grabbed, checking for completeness
• http://vault.ign.com - grabbing
• http://ac2vault.ign.com - Smiley - done
• http://acvault.ign.com - Smiley - done
• http://aion.ign.com - Smiley Grabbing
• http://wowvault.ign.com - Smiley grabbing
• http://atvault.ign.com - Smiley grabbing
• http://alli.ign.com - Smiley done
• http://antis.ign.com - Smiley grabbing
• http://doa.ign.com - Smiley grabbing
• http://aovault.ign.com - siliconvalleypark - grabbed
• http://aevans.dev.m.au.ign.com - dead
• http://aevans.dev.m.ca.ign.com - dead
• http://aevans.dev.m.ie.ign.com - dead
• http://aevans.dev.m.ign.com - dead
• http://aevans.dev.m.uk.ign.com - dead
• http://cdyi.dev.m.ign.com - dead
• http://ve3d.ign.com - grabbed, checking for completeness
• http://flashpoint.ign.com - siliconvalleypark - grabbed
• http://beaterator.ign.com - siliconvalleypark - grabbed
• http://aivlev.dev.m.au.ign.com - password protected dev site
• http://aivlev.dev.m.ie.ign.com - password protected dev site
• http://aivlev.dev.m.ign.com - password protected dev site
• http://aivlev.dev.m.uk.ign.com - password protected dev site
• http://apassey.dev.m.ign.com - password protected dev site
• http://bestof.ign.com - Tracer grabbing
• http://www.blockbuster.ign.com - already dead
• http://kaneandlynch.ign.com/ - flash based
• http://wishvault.ign.com - siliconvalleypark grabbing
• http://witchervault.ign.com - siliconvalleypark - grabbed
• http://www.supersmashbros.ign.com - Smiley grabbing
• http://www.championshipgamingseries.com - Smiley grabbing
• http://code.ign.com - Smiley grabbing
• http://mevault.ign.com - siliconvalleypark grabbing
• http://beacon.ign.com - Smiley, done
• http://blockbuster.ign.com - 404 not found
• http://broadband.ign.com - Smiley
• http://browserthemes.ign.com - Smiley
• http://ces2009.ign.com - Smiley, all broken links
• http://championshipgamingseries.ign.com - Smiley
• http://championsonline.ign.com - Smiley
• http://cohvault.ign.com - Smiley
• http://comiccon.ign.com - dead
• http://corp.ign.com - Smiley
• http://covvault.ign.com - Smiley, redirects to cohvault
• http://adtools.ign.com - blank
• http://crossassault.ign.com - Smiley, done
• http://design.ign.com - Smiley, done
• http://dragonica.ign.com - Smiley, done
• http://eq2vault.ign.com - Smiley, done with some timeouts
• http://eqvault.ign.com - Smiley, finished with some timeouts
• http://esports.ign.com - Smiley, done
• http://evevault.ign.com - Smiley
• http://feeds.ign.com - Smiley, done
• http://ffvault.ign.com - Smiley, finished with some timeouts
• http://findit.ign.com - Smiley, done
• http://gamechanger.ign.com - Smiley, done
• http://gamesites.ign.com - Smiley, done
• http://gamestore.ign.com - Smiley, done
• http://gamingworldrecord.ign.com - Smiley, done
• http://gbartone.dev.m.au.ign.com - not found
• http://gbartone.dev.m.ca.ign.com - not found
• http://gbartone.dev.m.ie.ign.com - not found
• http://gbartone.dev.m.ign.com - not found
• http://gbartone.dev.m.uk.ign.com - not found
• http://grandtheftautohood.ign.com -> grandtheftauto.ign.com
• http://grandtheftauto.ign.com - Smiley
• http://gtahood.ign.com -> grandtheftauto.ign.com
• http://gta.ign.com - Smiley, done
• http://www.blockbuster.ign.com - already dead
• http://guitarhero3.ign.com - Smiley, done
• http://gwvault.ign.com - Smiley, finished with some timeouts
• http://halo.ign.com - Smiley, done
• http://hls.gbartone.dev.m.ign.com -> hls.gbartone.dev.m.uk.ign.com, done, smiley
• http://horizonsvault.ign.com - Smiley, done
• http://ie.bestof.ign.com -> uk.bestof.ign.com (localized version of sites?
• http://ie.top100.ign.com - Smiley, (localized version redirected to .uk.)
• http://uk.bestof.ign.com - Smiley grabbing
• http://uk.corp.ign.com - Smiley grabbing
• http://uk.retro.ign.com - Smiley grabbing -> broken by + in url
• http://uk.sports.ign.com - Smiley grabbing -> uk.ign.com
• http://uk.top100.ign.com - Done
• http://iplstore.ign.com - dead, uploaded, Smiley
• http://jloijens.dev.m.ign.com - Smiley, redirected to .uk. version, done
• http://jtai.dev.m.ign.com - Smiley, dead
• http://kaneandlynch.ign.com - Smiley, done
• http://kjaniak.dev.m.ign.com - blocked by auth, Smiley
• http://l2vault.ign.com - Smiley
• http://labs.ign.com - Smiley, done
• http://lanoire.ign.com - Smiley, done
• http://links.em.ign.com - Smiley, done
• http://littlebigplanet.ign.com - Smiley, done
• http://live.ign.com - Smiley, done
• http://lotrovault.ign.com - Smiley
• http://mag.ign.com - Smiley, done
• http://mail2.ign.com - Smiley, done
• http://mcraft.ign.com - Smiley, done
• http://memoviedia.ign.com - Smiley, dead
• http://mevault.ign.com - Smiley, done
• http://m.ign.com -> m.uk.ign.com, Smiley
• http://minecraft.ign.com - Smiley
• http://uk.video.ign.com - Smiley grabbing -> broken by redirect
• http://niboppub.ign.com - Smiley, done
• http://nwvault.ign.com - Smiley
• http://m.uk.ign.com - Smiley
• http://musichub.ign.com - Smiley
• http://mxovault.ign.com - Smiley
• http://nchandra.dev.m.uk.ign.com - Smiley, done
• http://o.rpgvaultarchive.ign.com - Smiley, done
• http://potbsvault.ign.com - Smiley, done with some timeouts
• http://rotavault.ign.com - Smiley
• http://rpgvaultarchive.ign.com - Smiley
• http://ryzomvault.ign.com - Smiley, done, some timeouts
• http://sbvault.ign.com - Smiley, dead, done
• http://starcraft2.ign.com - Smiley done

Ready to grab

• http://au.bestof.ign.com
• http://au.retro.ign.com
• http://au.sports.ign.com
• http://wiki.stg.www.ign.com

untested

• http://au.microsites.ign.com
• http://au.top100.ign.com
• http://au.video.ign.com
• http://crose.dev.m.ca.ign.com
• http://crose.dev.m.ign.com
• http://m.au.ign.com
• http://m.ca.ign.com
• http://m.ie.ign.com
• http://mobile.ign.com
• http://mobile.ign.com
• http://nchandra.dev.m.au.ign.com
• http://nchandra.dev.m.ie.ign.com
• http://nchandra.dev.m.ign.com
• http://nchandra.dev.www.ign.com
• http://o.mobile.ign.com
• http://open.em.ign.com
• http://opt-out.emailpreferences.ign.com
• http://overlord.ign.com
• http://pawong.dev.www.ign.com
• http://planetelderscrolls.ign.com
• http://play.ign.com
• http://podcast.ign.com
• http://podcasts.ign.com
• http://primeblog.ign.com
• http://promotions.ign.com
• http://promotools.ign.com
• http://publish.ign.com
• http://rift.ign.com
• http://rmcadams.dev.m.ca.ign.com
• http://rmcadams.dev.m.ign.com
• http://rmcadams.dev.www.ign.com
• http://rsullivan.dev.m.ca.ign.com
• http://rsullivan.dev.m.ign.com
• http://rsullivan.dev.www.ign.com
• http://share.affiliation.com
• http://share.ign.com
• http://shootmania.ign.com
• http://s.insiderdownloads.ign.com
• http://skate2.ign.com
• http://smashbros.ign.com
• http://smcnabb.dev.www.ign.com
• http://sovault.ign.com
• http://speedtv.ign.com
• http://sslvpn.ign.com
• http://staging-api.ign.com
• http://store.ign.com
• http://strangleholdcentral.ign.com
• http://strategyvault.ign.com
• http://sts.ign.com
• http://studio.ign.com
• http://supersmashbros.ign.com
• http://swgvault.ign.com
• http://swvault.ign.com
• http://tabularasavault.ign.com
• http://tdu.ign.com
• http://tford.dev.m.ign.com
• http://tford.dev.www.ign.com
• http://thhe.ign.com
• http://ticket.ign.com
• http://tickets.ign.com
• http://titanquestvault.ign.com
• http://tjohnson.dev.au.ign.com
• http://tjohnson.dev.ca.ign.com
• http://tjohnson.dev.ie.ign.com
• http://tjohnson.dev.uk.ign.com
• http://tjohnson.dev.www.ign.com
• http://top100.ign.com
• http://touch.ign.com
• http://tqvault.ign.com
• http://trvault.ign.com
• http://twoworldsvault.ign.com
• http://uovault.ign.com
• http://v3-api.stg.ie.ign.com
• http://v3-api.stg.m.au.ign.com
• http://v3-api.stg.m.ie.ign.com
• http://v3-api.stg.www.ign.com
• http://vanguardvault.ign.com
• http://vgu.stg.www.ign.com
• http://viashoka.dev.m.ign.com
• http://viashoka.dev.www.ign.com
• http://videopatch1.atio.dev.uk.ign.com
• http://videoplayer.phantom.stg.www.ign.com
• http://video.stg.www.ign.com
• http://vnboards.ign.com
• http://warhammervault.ign.com
• http://wikihub.stg.www.ign.com
• http://wiki.stg.www.ign.com
• http://wishvault.ign.com
• http://witchervault.ign.com
• http://www.antis.ign.com
• http://www.championshipgamingseries.com
• http://www.ipl.ign.com
• http://www.kaneandlynch.ign.com
• http://www.mevault.ign.com
• http://www.supersmashbros.ign.com
• http://xboxlive.ign.com

These might be asset only hosting sites

• http://au.media.ign.com
• http://carsmedia.ign.com
• http://carsmovies.ign.com
• http://codesmedia.ign.com
• http://cubemedia.ign.com
• http://dcmedia.ign.com
• http://entertainmentmedia.ign.com
• http://faqsmedia.ign.com
• http://faqsmovies.ign.com
• http://ffmedia.ign.com
• http://formenmedia.ign.com
• http://guidesmedia.ign.com
• http://ie.media.ign.com
• http://insdermedia.ign.com
• http://insiderdownloads.ign.com
• http://insidermedia.ign.com
• http://macmedia.ign.com
• http://macmovies.ign.com
• http://media.ign.com
• http://moviemedia.ign.com
• http://moviesmovies.ign.com
• http://o.faqsmedia.ign.com
• http://o.faqsmovies.ign.com
• http://o.ffmovies.ign.com
• http://o.guidesmedia.ign.com
• http://o.insidermedia.ign.com
• http://o.media.ign.com
• http://o.moviesmovies.ign.com
• http://o.pcmedia.ign.com
• http://o.pocketmedia.ign.com
• http://o.ps2media.ign.com
• http://o.xboxmedia.ign.com
• http://xboxlivemedia.ign.com
• http://xboxlivemovies.ign.com
• http://xboxmedia.ign.com
• http://xboxmovies.ign.com
• http://pcmedia.ign.com
• http://pocketmedia.ign.com
• http://ps2media.ign.com
• http://psxmedia.ign.com
• http://psxmovies.ign.com
• http://scifimedia.ign.com
• http://uk.media.ign.com
• http://vaultmedia.ign.com
• http://wiremedia.ign.com
• http://wiremovies.ign.com
• http://c.ffmovies.ign.com
• http://s.faqsmovies.ign.com
• http://s.ffmovies.ign.com
• http://s.moviesmovies.ign.com
• http://s.psxmovies.ign.com
• http://s.xboxmovies.ign.com

Redirects

• http://cars.ign.com -> http://www.ign.com/
• http://guides.ign.com -> http://www.ign.com/wikis
• http://answers.ign.com -> http://www.ign.com/boards/
• http://ve3dboards.ign.com -> http://www.ign.com/boards/
• http://ffmovies.ign.com -> http://ffmedia.ign.com
• http://ddovault.ign.com -> http://dndvault.ign.com/
• http://bigworldvault.ign.com -> http://vault.ign.com
• http://911.ign.com -> http://tickets.ign-inc.com/
• http://bestofe3.ign.com -> http://games.ign.com/bestofe3.html
• http://dsi.ign.com -> http://ds.ign.com/dsi/
• http://www.ipl.ign.com -> http://www.ign.com/ipl/
• http://xboxlive.ign.com -> http://www.ign.com/xbox-live
• http://download.ign.com - redirect to fileplanet
• http://downloads.ign.com - redirect to fileplanet
• http://dsvault.ign.com - redirect to planetdungeonsiege.com
• http://emailpreferences.ign.com - redirect to mail.ign.com
• http://guidesarchive.ign.com -> http://uk.ign.com/wikis
• http://hjvault.ign.com - redirects to vault.ign.com
• http://ipl.ign.com -> ign.com/ipl
• http://mac.ign.com -> http://uk.ign.com/games/reviews?platformSlug=mac
• http://o.guidesarchive.ign.com -> uk.ign.com/wikis
• http://starcraft.ign.com - Smiley done - redirect to starcraft2.ign.com

Gamespy Domains

Ready to grab

• http://planetthemovies.gamespy.com
• http://planetelderscrolls.gamespy.com
• http://sslvpn.gamespy.com
• http://lanoirepc.d2gstore.gamespy.com

In Progress

• http://planetcnc.gamespy.com - grabbed, checking for completeness
• http://planetthesims.gamespy.com - grabbed, checking for completeness
• http://planetfrontlines.gamespy.com - grabbed, checking for completeness
• http://planetcivilization.gamespy.com - grabbed, checking for completeness
• http://planethalflife.gamespy.com - grabbed, checking for completeness
• http://planettransformers.gamespy.com - grabbed, checking for completeness
• http://planetcoh.gamespy.com - grabbed, checking for completeness
• http://planetbattlefield.gamespy.com - grabbed, checking for completeness
• http://planetresidentevil.gamespy.com - grabbed, checking for completeness
• http://planetxmen.gamespy.com - grabbed, checking for completeness
• http://planetquake.gamespy.com - grabbed, checking for completeness, grabbing again.
• http://planetgrandtheftauto.gamespy.com - grabbed, checking for completeness
• http://planettonyhawk.gamespy.com - grabbed, checking for completeness
• http://planetunreal.gamespy.com - grabbed, checking for completeness
• http://planetfallout.gamespy.com - grabbed, checking for completeness
• http://planetageofempires.gamespy.com - grabbed, checking for completeness
• http://planetgearsofwar.gamespy.com - grabbed, checking for completeness
• http://planetcallofduty.gamespy.com - grabbed, checking for completeness
• http://classicgaming.gamespy.com - grabbed, checking for completeness
• http://planetdoom.gamespy.com - grabbed, checking for completeness
• http://planetwwe.gamespy.com - grabbed, checking for completeness
• http://pc.gamespy.com - grabbed, checking for completeness
• http://psp.gamespy.com - grabbed, checking for completeness
• http://ds.gamespy.com - grabbed, checking for completeness
• http://xbox360.gamespy.com - grabbed, checking for completeness
• http://planetfarcry.gamespy.com - grabbed, checking for completeness
• http://planetcrysis.gamespy.com - grabbing omf_
• http://planetmedalofhonor.gamespy.com - grabbing omf_
• http://xbox.gamespy.com - grabbing Smiley
• http://norad.gamespy.com - grabbing Smiley
• http://bf2142portalservices.gamespy.com - grabbing Smiley
• http://arena.gamespy.com - grabbing Smiley
• http://www.gamespy.com - grabbing Smiley
• http://bugsubmit.gamespy.com - grabbing Smiley
• http://ps3.gamespy.com - Smiley grabbing

Redirects

• http://forumplanet.gamespy.com -> ign.com/boards
• http://forums.gamespy.com -> ign.com/boards/categories/gamespy