Difference between revisions of "Ispygames"

From Archiveteam
Jump to navigation Jump to search
(47 intermediate revisions by 15 users not shown)
Line 11: Line 11:
| width=125px | '''Project status''' || {{{project_status|{{Closing}}}}}
| width=125px | '''Project status''' || {{{project_status|{{Closing}}}}}
|-
|-
| width=125px | '''Archiving status''' || {{{archiving_status|{{inprogress}}}}}
| width=125px | '''Archiving status''' || {{partiallysaved}}
|-
|-
| width=125px | '''Project source''' || {{{source|{{Unknown}}}}}
| width=125px | '''Project source''' || {{{source|{{Unknown}}}}}
Line 23: Line 23:


[http://www.polygon.com/2013/2/21/4014196/ign-layoffs-1up-ugo-and-gamespy-shutting-down IGN hit with layoffs, 1UP, UGO and GameSpy shutting down]<br />
[http://www.polygon.com/2013/2/21/4014196/ign-layoffs-1up-ugo-and-gamespy-shutting-down IGN hit with layoffs, 1UP, UGO and GameSpy shutting down]<br />
[http://www.examiner.com/article/1up-ugo-and-gamespy-to-be-shut-down 1UP, UGO and GameSpy to be shut down]
[http://www.examiner.com/article/1up-ugo-and-gamespy-to-be-shut-down 1UP, UGO and GameSpy to be shut down]<br />
 
[http://pc.gamespy.com/articles/122/1227460p1.html Goodbye, And Thank You From The GameSpy Team]
== The Problems ==
== The Problems ==


Line 40: Line 40:


* Save the sites and related content
* Save the sites and related content
* Backup the twitter feeds for any associated accounts. [http://www.allmytweets.net/ All my tweets] just takes a username and returns the max tweets possible.
* Backup the Twitter feeds for any associated accounts. [http://www.allmytweets.net/ All my tweets] just takes a username and returns the max tweets possible.




Line 107: Line 107:
* http://www.blockbuster.ign.com - already dead
* http://www.blockbuster.ign.com - already dead
* http://kaneandlynch.ign.com/ - flash based
* http://kaneandlynch.ign.com/ - flash based
* http://wishvault.ign.com - siliconvalleypark grabbing
* http://wishvault.ign.com - siliconvalleypark grabbing, Smiley done
* http://witchervault.ign.com - siliconvalleypark - grabbed
* http://witchervault.ign.com - siliconvalleypark, Smiley - grabbed, done
* http://www.supersmashbros.ign.com - Smiley grabbing
* http://www.supersmashbros.ign.com - Smiley grabbing
* http://www.championshipgamingseries.com - Smiley grabbing
* http://www.championshipgamingseries.com - Smiley grabbing
Line 145: Line 145:
* http://gbartone.dev.m.uk.ign.com - not found
* http://gbartone.dev.m.uk.ign.com - not found
* http://grandtheftautohood.ign.com -> grandtheftauto.ign.com
* http://grandtheftautohood.ign.com -> grandtheftauto.ign.com
* http://grandtheftauto.ign.com - Smiley
* http://grandtheftauto.ign.com - Smiley, done
* http://gtahood.ign.com -> grandtheftauto.ign.com
* http://gtahood.ign.com -> grandtheftauto.ign.com
* http://gta.ign.com - Smiley, done
* http://gta.ign.com - Smiley, done
Line 195: Line 195:
* http://starcraft2.ign.com - Smiley done
* http://starcraft2.ign.com - Smiley done
* http://opt-out.emailpreferences.ign.com -> mail.ign.com, Smiley, done
* http://opt-out.emailpreferences.ign.com -> mail.ign.com, Smiley, done
* http://overlord.ign.com - Smiley
* http://overlord.ign.com - Smiley, done
* http://pawong.dev.www.ign.com - Smiley, auth failed
* http://pawong.dev.www.ign.com - Smiley, auth failed
* http://planetelderscrolls.ign.com - Smiley, done
* http://planetelderscrolls.ign.com - Smiley, done
Line 206: Line 206:
* http://promotools.ign.com - Smiley
* http://promotools.ign.com - Smiley
* http://publish.ign.com - Smiley, done
* http://publish.ign.com - Smiley, done
* http://rift.ign.com - Smiley
* http://rift.ign.com - Smiley, done
* http://rmcadams.dev.m.ca.ign.com - Smiley, auth failed
* http://rmcadams.dev.m.ca.ign.com - Smiley, auth failed
* http://rmcadams.dev.m.ign.com - Smiley, auth failed
* http://rmcadams.dev.m.ign.com - Smiley, auth failed
Line 219: Line 219:
* http://skate2.ign.com - Smiley, done
* http://skate2.ign.com - Smiley, done
* http://smashbros.ign.com -> supersmashbros.ign.com, Smiley, done
* http://smashbros.ign.com -> supersmashbros.ign.com, Smiley, done
* http://supersmashbros.ign.com - Smiley
* http://supersmashbros.ign.com - Breaks wget
* http://smcnabb.dev.www.ign.com - Smiley, auth failed
* http://smcnabb.dev.www.ign.com - Smiley, auth failed
* http://sovault.ign.com -> vault.ign.com, Smiley, done
* http://sovault.ign.com -> vault.ign.com, Smiley, done
Line 228: Line 228:
* http://swgvault.ign.com - Smiley
* http://swgvault.ign.com - Smiley
* http://swvault.ign.com - Smiley, done with errors
* http://swvault.ign.com - Smiley, done with errors
* http://tabularasavault.ign.com - Smiley
* http://tabularasavault.ign.com - Smiley, done
* http://tdu.ign.com - Smiley, done, broken
* http://tdu.ign.com - Smiley, done, broken
* http://tford.dev.m.ign.com - not found, Smiley
* http://tford.dev.m.ign.com - not found, Smiley
Line 235: Line 235:
* http://ticket.ign.com - Smiley, done
* http://ticket.ign.com - Smiley, done
* http://tickets.ign.com - Smiley, done
* http://tickets.ign.com - Smiley, done
* http://titanquestvault.ign.com - Smiley
* http://titanquestvault.ign.com - Smiley, done
* http://tjohnson.dev.uk.ign.com - 404, Smiley
* http://tjohnson.dev.uk.ign.com - 404, Smiley
* http://touch.ign.com - Smiley, done
* http://touch.ign.com - Smiley, done
Line 242: Line 242:
* http://twoworldsvault.ign.com - Smiley
* http://twoworldsvault.ign.com - Smiley
* http://uovault.ign.com -> http://vault.ign.com/uovault.html, Smiley, done
* http://uovault.ign.com -> http://vault.ign.com/uovault.html, Smiley, done
* http://vanguardvault.ign.com - Smiley, done
* http://warhammervault.ign.com - Smiley, done
* http://wikihub.stg.www.ign.com - 404
* http://wiki.stg.www.ign.com - 404
* http://wishvault.ign.com - Smiley
* http://witchervault.ign.com - Smiley
* http://www.antis.ign.com -> http://entertainment.ign.com/antis.html
* http://www.championshipgamingseries.com - Smiley
* http://www.ipl.ign.com -> http://www.ign.com/ipl/
* http://www.kaneandlynch.ign.com -> kaneandlynch.ign.com, Smiley, done
* http://www.mevault.ign.com - Smiley, done
* http://xboxlive.ign.com -> uk.ign.com/xbox-live


=== Ready to grab ===
=== Ready to grab ===
Line 279: Line 291:
* http://v3-api.stg.m.ie.ign.com
* http://v3-api.stg.m.ie.ign.com
* http://v3-api.stg.www.ign.com
* http://v3-api.stg.www.ign.com
* http://vanguardvault.ign.com - Smiley
* http://vgu.stg.www.ign.com
* http://vgu.stg.www.ign.com
* http://viashoka.dev.m.ign.com
* http://viashoka.dev.m.ign.com
Line 287: Line 298:
* http://video.stg.www.ign.com
* http://video.stg.www.ign.com
* http://vnboards.ign.com
* http://vnboards.ign.com
* http://warhammervault.ign.com
 
* http://wikihub.stg.www.ign.com
* http://wiki.stg.www.ign.com
* http://wishvault.ign.com
* http://witchervault.ign.com
* http://www.antis.ign.com
* http://www.championshipgamingseries.com
* http://www.ipl.ign.com
* http://www.kaneandlynch.ign.com
* http://www.mevault.ign.com
* http://xboxlive.ign.com


=== These might be asset only hosting sites ===
=== These might be asset only hosting sites ===
Line 381: Line 382:


=== Ready to grab ===
=== Ready to grab ===
* http://planetthemovies.gamespy.com
* http://planetelderscrolls.gamespy.com
* http://sslvpn.gamespy.com
* http://sslvpn.gamespy.com
* http://lanoirepc.d2gstore.gamespy.com
* http://gamespyarcade.com - Smiley


=== In Progress ===
=== In Progress ===


* http://lanoirepc.d2gstore.gamespy.com - Smiley, done
* http://gamespyarcade.com - Smiley, done
* http://planetthemovies.gamespy.com - Smiley, done
* http://planetelderscrolls.gamespy.com - Smiley
* http://planetcnc.gamespy.com - grabbed, checking for completeness
* http://planetcnc.gamespy.com - grabbed, checking for completeness
* http://planetthesims.gamespy.com - grabbed, checking for completeness
* http://planetthesims.gamespy.com - grabbed, checking for completeness
Line 423: Line 424:
* http://www.gamespy.com - grabbing Smiley
* http://www.gamespy.com - grabbing Smiley
* http://bugsubmit.gamespy.com - grabbing Smiley
* http://bugsubmit.gamespy.com - grabbing Smiley
* http://ps3.gamespy.com - Smiley grabbing
* http://ps3.gamespy.com - Smiley, done with 503's at the end.


=== Redirects ===
=== Redirects ===
Line 429: Line 430:
* http://forumplanet.gamespy.com -> ign.com/boards
* http://forumplanet.gamespy.com -> ign.com/boards
* http://forums.gamespy.com -> ign.com/boards/categories/gamespy
* http://forums.gamespy.com -> ign.com/boards/categories/gamespy
* http://planetdeusex.gamespy.com -> gamespy.com (actual site's at planetdeusex.com)
* http://planetelderscrolls.gamespy.com -> planetelderscrolls.ign.com
== 1up.com ==
On 2016-05-24, http://www.1up.com has been thrown into [[ArchiveBot]] with {{Job|35fcc4zofjl5kg52fkbcskgus}}.
{{Navigation box}}

Revision as of 16:04, 13 April 2018

Gamespy, IGN, 1up, ugo
Ispygames logo
Gamespy.jpg
URL http://www.gamespy.com & many others
Project status Closing
Archiving status Partially saved
Project source Unknown
Project tracker Unknown
IRC channel #ispygames

The News

IGN hit with layoffs, 1UP, UGO and GameSpy shutting down
1UP, UGO and GameSpy to be shut down
Goodbye, And Thank You From The GameSpy Team

The Problems

  • Once you start digging around these sites you find it to be a mess of inconsistent url schemes and content everywhere.
  • Some files are being hosted on MediaFire.
  • Based on tests the larger and older a site is the more that is missed by a wget crawl due to the url scheme.

What we know

  • We already have a list of almost all the domains involved
  • A clean list with dups and bad domains is already being process and will be posted here when complete.
  • Most of the sites are not that big, but a few are huge.

The plan

  • Save the sites and related content
  • Backup the Twitter feeds for any associated accounts. All my tweets just takes a username and returns the max tweets possible.


wget test command

This if for the gamespy sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://planetdoom.gamespy.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST" \
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com

Try this for the ign, ugo sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://ve3d.ign.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST"

IGN domains

In progress

Ready to grab


untested


These might be asset only hosting sites

Redirects

Gamespy Domains

Ready to grab

In Progress

Redirects

1up.com

On 2016-05-24, http://www.1up.com has been thrown into ArchiveBot with job:35fcc4zofjl5kg52fkbcskgus.