Difference between revisions of "Ispygames"

From Archiveteam
Jump to: navigation, search
(IGN domains)
(wget test command)
Line 23: Line 23:
  
 
== wget test command ==
 
== wget test command ==
 +
This if for the gamespy sites.
 
<pre>
 
<pre>
 
USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
 
USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="planetdoom.gamespy.com"
+
SAVE_HOST="http://planetdoom.gamespy.com"
 +
WARC_NAME="warc_name"
  
wget -e robots=off --mirror --page-requisites --waitretry 5 --timeout 60 \
+
wget -e robots=off --mirror --page-requisites \
--tries 10 --warc-header "operator: Archive Team" --warc-cdx \
+
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
"$SAVE_HOST" --warc-file="$SAVE_HOST" --wait 2 -U "$USER_AGENT" \
+
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$SAVE_HOST" \
 +
-U "$USER_AGENT" "$SAVE_HOST" \
 
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com
 
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com
 +
</pre>
 +
 +
Try this for the ign, ugo sites.
 +
 +
<pre>
 +
USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
 +
SAVE_HOST="http://ve3d.ign.com"
 +
WARC_NAME="warc_name"
 +
 +
wget -e robots=off --mirror --page-requisites \
 +
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
 +
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$SAVE_HOST" \
 +
-U "$USER_AGENT" "$SAVE_HOST"
 
</pre>
 
</pre>
  

Revision as of 00:55, 24 March 2013

The News

IGN hit with layoffs, 1UP, UGO and GameSpy shutting down
1UP, UGO and GameSpy to be shut down

The Problems

  • Once you start digging around these sites you find it to be a mess of inconsistent url schemes and content everywhere.
  • Some files are being hosted on MediaFire.
  • Based on tests the larger and older a site is the more that is missed by a wget crawl due to the url scheme.

What we know

  • We already have a list of almost all the domains involved
  • A clean list with dups and bad domains is already being process and will be posted here when complete.
  • Most of the sites are not that big, but a few are huge.

The plan

  • Save the sites and related content
  • Backup the twitter feeds for any associated accounts. All my tweets just takes a username and returns the max tweets possible.


wget test command

This if for the gamespy sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://planetdoom.gamespy.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$SAVE_HOST" \
-U "$USER_AGENT" "$SAVE_HOST" \
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com

Try this for the ign, ugo sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://ve3d.ign.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$SAVE_HOST" \
-U "$USER_AGENT" "$SAVE_HOST"

IGN domains

Gamespy Domains