Fileplanet

From Archiveteam
Revision as of 03:40, 13 June 2012 by Underscor (talk | contribs)
Jump to navigation Jump to search
FilePlanet
Fileplanet logo
Website host of game content, 1999-2012
Website host of game content, 1999-2012
URL http://www.fileplanet.com
Status Closing
Archiving status In progress...
Archiving type Unknown
IRC channel #fireplanet (on hackint)

FilePlanet is no longer hosting new content, and "is in the process of being archived [by IGN]."

FilePlanet hosts 87,190 download pages of game-related material (demos, patches, mods, promo stuff, etc.), which needs to be archived. These tend to be larger files, ranging from 10MB patches to 3GB clients. We'll want all the arms we can for this one, since it gets harder the farther the archiving goes (files are numbered chronologically, and Skyrim mods are bigger than Doom ones).

What We Need

How to help

  • Have bash, wget, grep, rev, cut
  • >100 gigabytes of space, just to be safe
  • Put https://raw.github.com/SpiritQuaddicted/fileplanet-file-download/master/download_pages_and_files_from_fileplanet.sh somewhere (I'd suggest ~/somepath/fileplanetdownload/ ) and "chmod +x" it
  • Pick a free increment (eg 110000-114999) and tell people about it (#fireplanet in EFnet or post it here). Be careful. In lower ranges a 5k range might work, but they get HUGE later. In the 220k range and probably lower too, we better use 100 IDs per chunk.
  • * Keep the chunk sizes small. <30G would be nice. The less the better.
  • Run the script with your start and end IDs as arguments. Eg "./download_pages_and_files_from_fileplanet.sh 110000 114999"
  • Take a walk for half a day.
  • You can tail the .log files if you are curious. See right below.
  • Once you are done with your chunk, you will have a directory named after your range, eg 110000-114999/ . Inside that pages_xx000-xx999.log and files_xx000-xx999.log plus the www.fileplanet.com/ directory.
  • Done! GOTO 10

In the end we'll upload all the parts to archive.org. If you have an account, you can use eg s3cmd.

s3cmd --add-header x-archive-auto-make-bucket:1 --add-header "x-archive-meta-description:Files from Fileplanet (www.fileplanet.com), all files from the ID range 110000 to 114999." put 110000-114999/*.log 110000-114999.tar s3://FileplanetFiles_110000-114999/

The log files are important! Make sure they are saved!

Notes

  • For planning a good range to download, check http://www.quaddicted.com/stuff/temp/file_IDs_from_sitemaps.txt but be aware that apparently that does not cover all IDs we can get by simply incrementing by 1. Schbirid downloaded eg the file 75059 which is not listed in the sitemaps. So you can not trust that ID list.
  • The range 175000-177761 (weird end number since that's when the server ran out of space...) had ~1100 files and 69G. We will need to use 1k ID increments for those ranges.
  • Schbirid mailed to FPOps@IGN.com on the 3rd of May, no reply.

Status

  1. kumquat don't remove this, it makes it easier to find please
Range Status Number of files Size in gigabytes Downloader
00000-09999 Done, archived 1991 1G Schbirid
10000-19999 Done, archived 3159 9G Schbirid
20000-29999 Done, archived 6453 7G Schbirid
30000-39999 Done, archived 4085 9G Schbirid
40000-49999 Done, archived 5704 18G Schbirid
50000-54999 Done, archived 2707 24G Schbirid
55000-59999 Done, archived (bad URL) 2390 24G Schbirid
60000-64999 Done, archived 2349 24G Schbirid
65000-69999 Done, archived 305 4G Schbirid
70000-79999 Done, archived 59 0.2G Schbirid
80000-84999 Done, archived 2822 31G Debianer
85000-89999 Done, archived 1869 29G Schbirid
90000-109999 Done, empty 0 0 Schbirid
110000-114999 Done, archived 2139 35G Schbirid
115000-115999 Done, archived 932 1.9G codebear
116000-116999 Done, archived 694 11G codebear
117000-117999 Done, archived 752 16G codebear
118000-118999 Done, archived 726 16G codebear
119000-119999 Done, locally 718 28G codebear
120000-124999 Done, locally 3463 68G codebear
125000-129999 Done, archived [1] [2] [3] [4] [5] (bad URL) 3384 78G S[h]O[r]T
130000-130999 Done, archived 603 24G codebear
131000-131999 Done, archived 640 22G codebear
132000-132999 Done, archived 626 17G codebear
133000-133999 Done, archived 602 25G codebear
134000-134999 Done, archived 551 19G codebear
135000-135999 Done, archived 763 21G codebear
136000-136999 Done, archived 728 27G codebear
137000-137999 Done, archived 601 18G codebear
138000-138999 Done, archived 689 26G codebear
139000-139999 Done, archived 705 18G codebear
140000-140999 Done, archived 750 26G S[h]O[r]T
141000-141999 Done, archived 586 30G S[h]O[r]T
142000-142999 Done, archived 337 19G S[h]O[r]T
143000-143999 Done, archived 292 14G S[h]O[r]T
144000-144999 Done, archived 328 20G S[h]O[r]T
145000-145999 Done, archived 216 25G Schbirid
146000-146999 Done, archived 383 30G Schbirid
147000-147499 Done, archived 279 20G Schbirid
147500-147999 Done, archived 309 17G Schbirid
148000-148499 Done, archived 311 15G Schbirid
148500-148999 Done, archived 229 14G Schbirid
149000-149500 Done, archived 202 8G Schbirid
149500-149999 Done, archived 221 9G Schbirid
150000-150499 Done, archived 216 15G S[h]O[r]T
150500-150999 Done, archived 270 13G S[h]O[r]T
151000-151999 Done, archived 310 19G S[h]O[r]T
151500-151999 Done, archived 244 17G S[h]O[r]T
152000-152499 Done, archived 234 19G S[h]O[r]T
152500-152999 Done, archived 255 13G S[h]O[r]T
153000-153499 Done, archived 287 19G S[h]O[r]T
153500-153999 Done, archived 269 17G S[h]O[r]T
154000-154499 Done, archived 248 18G S[h]O[r]T
154500-154999 Done, archived 173 8.7G S[h]O[r]T
155000-155499 Done, archived 199 11G S[h]O[r]T
155500-155999 Done, archived 179 11G S[h]O[r]T
156000-156499 Done, archived 238 13G S[h]O[r]T
156500-156999 Done, archived 185 15G S[h]O[r]T
157000-157499 Done, archived 247 18G S[h]O[r]T
157500-157999 Done, archived 267 17G S[h]O[r]T
158000-158499 Done, archived 252 16G S[h]O[r]T
158500-158999 Done, archived 278 30G S[h]O[r]T
159000-159499 Done, archived 260 11G S[h]O[r]T
159500-159999 Done, archived 220 13G S[h]O[r]T
160000-160499 Done, archived 214 15G NotGLaDOS
160500-160999 Done, archived 154 17G NotGLaDOS
161000-161499 Done, archived 232 22G NotGLaDOS
161500-161999 Done, archived 277 38G NotGLaDOS
162000-162499 Done, archived 268 34G NotGLaDOS
162500-162999 Done, archived 294 29G NotGLaDOS
163000-163499 Done, archived 340 21G NotGLaDOS
163500-163999 Done, archived 279 24G NotGLaDOS
164000-164499 Done, archived 309 19G NotGLaDOS
164500-164999 Done, archived 230 27G NotGLaDOS
165000-165499 Done, archived 266 20G NotGLaDOS
165500-165999 Done, archived 174 9.2G NotGLaDOS
166000-166499 Done, locally 104 12G NotGLaDOS
166500-166999 Done, locally 119 22G NotGLaDOS
167000-167499 Done, locally 167 27G NotGLaDOS
167500-167999 Done, locally 99 13G NotGLaDOS
168000-168499 Done, locally 149 14G NotGLaDOS
168500-168999 Done, locally 195 30G NotGLaDOS
169000-169499 Done, locally 190 19G NotGLaDOS
169500-169999 Done, locally 290 34G NotGLaDOS
170000-170499 Done, archived 286 50G S[h]O[r]T
170500-170999 Done, archived 231 16G S[h]O[r]T
171000-171499 Done, archived 187 21G S[h]O[r]T
171500-171999 Done, archived 113 9.8G S[h]O[r]T
172000-172499 Done, archived 227 23G S[h]O[r]T
172500-172999 Done, archived 221 13G S[h]O[r]T
173000-173499 Done, archived 216 11G S[h]O[r]T
173500-173999 Done, archived 274 24G S[h]O[r]T
174000-174499 Done, archived 231 7.9G S[h]O[r]T
174500-174999 Done, archived 248 23G S[h]O[r]T
175000-175499 Done, archived 294 18G S[h]O[r]T
175500-175999 Done, archived 244 18G S[h]O[r]T
176000-176499 Done, archived 354 34G S[h]O[r]T
176500-176999 Done, archived 390 23G S[h]O[r]T
177000-177499 Done, archived 359 27G S[h]O[r]T
177500-177999 Done, archived 283 24G S[h]O[r]T
178000-178499 Done, archived 174 12G S[h]O[r]T
178500-178999 Done, archived 315 31G S[h]O[r]T
179000-179499 Done, archived 249 32G S[h]O[r]T
179500-179999 Done, archived 211 22G S[h]O[r]T
180000-180499 Done, archived 179 37G Schbirid
180500-180999 Done, archived 174 23G Schbirid
181000-181499 Done, archived. 1 2 3 4 5 (trash) 218 71G Schbirid
181500-181599 Done, archived 49 5G Schbirid
181600-181699 Done, archived 15 2G Schbirid
181700-181799 Done, archived 53 14G Schbirid
181800-181899 Done, archived 32 5G Schbirid
181900-181999 Done, archived 40 8G Schbirid
182000-182099 Done, archived (trash) 63 22G Schbirid
182100-182199 Done, archived 52 18G Schbirid
182200-182299 Done, archived 63 25G Schbirid
182300-182399 Done, archived 44 25G Schbirid
182400-182499 Done, archived 44 10G Schbirid
182500-182599 Done, archived (trash) 50 21G Schbirid
182600-182699 Done, archived 60 21G Schbirid
182700-182799 Done, archived 54 23G Schbirid
182800-182899 Done, archived 52 22G Schbirid
182900-182999 Done, archived 38 10G Schbirid
183000-183099 Done, archived 45 16G Schbirid
183100-183199 Done, archived 68 50G Schbirid
183200-183299 Done, archived 39 7.8G Schbirid
183300-183399 Done, archived 23 2.4G Schbirid
183400-183499 Done, archived 41 3.2G Schbirid
183500-183999 Done, archived 213 27G Schbirid
184000-184099 Done, archived 54 7.7G Schbirid
184100-184199 Done, archived 66 2.7G Schbirid
184200-184299 Done, archived 58 6.3G Schbirid
184300-184399 Done, archived 63 19G Schbirid
184400-184499 Done, archived 63 24G Schbirid
184500-184599 Done, archived 41 1017M Schbirid
184600-184699 Done, archived 53 19G Schbirid
184700-184799 Done, archived 63 25G Schbirid
184800-184899 Done, archived 54 9.0G Schbirid
184900-184999 Done, archived 55 6.2G Schbirid
185000-185500 Done, archived 286 64G underscor
185500-185999 Done, archived 169 57G underscor
186000-186499 Done, archived 136 47G underscor
186500-186999 Done, locally 126 28G underscor
187000-187499 Done, locally 155 42G underscor
185000-189999 In progress underscor
190000-190099 Done, locally 24 14G Sui
190100-190199 Done, locally 24 4.7G Sui
190200-190299 Done, locally 28 5.9G Sui
190300-190399 Done, locally 19 2.6G Sui
190400-190499 Done, locally 21 13G Sui
190500-190999 In progress Sui
191000-191099 Done, archived 28 28G Schbirid
191100-191199 Done, archived 27 8.4G Schbirid
191200-191299 Done, archived 34 22G Schbirid
191300-191399 Done, archived 14 26G Schbirid
191400-191499 Done, archived 41 17G Schbirid
191500-191599 Done, archived 30 21G Schbirid
191600-191699 Done, archived 37 25G Schbirid
191700-191799 Done, archived 20 34G Schbirid
191800-191899 Done, archived 15 3.3G Schbirid
191900-191999 Done, archived 31 32G Schbirid
192000-192099 Done, archived 20 8.2G Schbirid
192100-192199 Done, archived 22 4.4G Schbirid
192200-192299 Done, archived 45 29G Schbirid
192300-192399 Done, archived 43 18G Schbirid
192400-192499 Done, archived 33 8.8G Schbirid
192500-192599 Done, archived 30 11G Schbirid
192600-192699 Done, archived 36 5.6G Schbirid
192700-192799 Done, archived 17 888M Schbirid
192800-192899 Done, archived 38 3.2G Schbirid
192900-192999 Done, archived 23 9.2G Schbirid
193000-193099 Done, archived 45 13G Schbirid
193100-193199 Done, archived 44 9.0G Schbirid
193200-193299 Done, archived 43 5.1G Schbirid
193300-193399 Done, archived 44 2.0G Schbirid
193400-193499 Done, archived 25 2.9G Schbirid
193500-193599 Done, archived 45 32G Schbirid
193600-193699 Done, archived 33 15G Schbirid
193700-193799 Done, archived 41 15G Schbirid
193800-193899 Done, archived 41 47G Schbirid
193900-193999 Done, archived 34 5.1G Schbirid
194000-194099 Done, archived 52 22G Schbirid
194100-194199 Done, archived 40 25G Schbirid
194200-194299 Done, archived 35 2.9G Schbirid
194300-194399 Done, archived 56 3.0G Schbirid
194400-194499 Done, archived 55 30G Schbirid
194500-194599 Done, archived 34 7.2G Schbirid
194600-194699 Done, archived 25 45G Schbirid
194700-194799 Done, archived 31 25G Schbirid
194800-194899 Done, archived 37 14G Schbirid
194900-194999 Done, archived 32 15G Schbirid
196000-196099 Done, archived 55 46G Schbirid
196100-196199 Done, archived 64 25G Schbirid
196200-196299 Done, archived 57 15G Schbirid
196300-196399 Done, archived 67 14G Schbirid
196400-196499 Done, archived 74 24G Schbirid
196500-196599 Done, archived 55 27G Schbirid
196600-196699 Done, archived 58 72G Schbirid
196700-196799 Done, archived 26 3.4G Schbirid
196800-196899 Done, archived 52 19G Schbirid
196900-196999 Done, archived 50 15G Schbirid
197000-197099 Done, archived 53 50G Schbirid
197100-197199 Done, archived 40 15G Schbirid
197200-197299 Done, archived 53 30G Schbirid
197300-197399 Done, archived 48 5.5G Schbirid
197400-197499 Done, archived 46 18G Schbirid
197500-197599 Done, archived 49 11G Schbirid
197500-197599 Done, archived 49 11G Schbirid
197600-197699 Done, archived 59 48G Schbirid
197700-197799 Done, archived 35 6.0G Schbirid
197800-197899 Done, archived 27 30G Schbirid
197900-197999 Done, archived 35 3.8G Schbirid
198000-198099 Done, archived 33 3.4G Schbirid
198100-198199 Done, archived 48 36G Schbirid
198100-198199 Done, archived 48 36G Schbirid
198200-198299 Done, archived 32 6.5G Schbirid
198300-198399 Done, archived 48 50G Schbirid
198400-198499 Done, archived 50 43G Schbirid
198500-198599 Done, archived 37 17G Schbirid
198600-198699 Done, archived 26 17G Schbirid
198700-198799 Done, archived 16 4.7G Schbirid
198800-198899 Done, archived 48 21G Schbirid
198900-198999 Done, archived 51 19G Schbirid
199000-199099 Done, archived 44 7.1G Schbirid
199100-199199 Done, archived 42 11G Schbirid
199200-199299 Done, archived 49 4.3G Schbirid
199300-199399 Done, archived 55 14G Schbirid
199400-199499 Done, archived 39 2.6G Schbirid
199500-199599 Done, archived 53 34G Schbirid
199600-199699 Done, archived 31 4.3G Schbirid
199700-199799 Done, archived 42 2.6G Schbirid
199800-199899 Done, archived 49 4.5G Schbirid
199900-199999 Done, archived 42 13G Schbirid
200000-200999 Done, archived (bad URL) 247 41G Schbirid
201000-201099 Done, locally 61 7.6G NotGLaDOS
201100-201199 Done, locally 71 7.6G NotGLaDOS
201200-201299 Done, locally 39 7.5G NotGLaDOS
201300-201399 Done, locally 44 27G NotGLaDOS
201400-201499 Done, locally 46 7.9G NotGLaDOS
201500-201599 Done, locally 54 33G NotGLaDOS
201600-201699 Done, locally 53 21G NotGLaDOS
201700-201799 Done, locally 49 24G NotGLaDOS
201800-201899 Done, locally 58 47G NotGLaDOS
201900-201999 Done, locally 45 33G NotGLaDOS
202000-202099 Done, locally 50 15G NotGLaDOS
202100-202199 Done, locally 35 18G NotGLaDOS
202200-202299 Done, locally 56 14G NotGLaDOS
202300-202399 Done, locally 42 8.8G NotGLaDOS
202400-202499 Done, locally 59 34G NotGLaDOS
202500-202599 Done, locally 48 6.7G NotGLaDOS
202500-202599 Done, locally 48 6.7G NotGLaDOS
202600-202699 Done, locally 51 13G NotGLaDOS
202700-202799 Done, locally 56 14G NotGLaDOS
202800-209999 In Progress NotGLaDOS
210000-210099 Done, archived 39 3.6G Schbirid
210100-210199 Done, archived 43 9.1G Schbirid
210200-210299 Done, archived 52 7.0G Schbirid
210300-210399 Done, archived 57 7.6G Schbirid
210400-210499 Done, archived 35 6.8G Schbirid
210500-210599 Done, archived 46 8.1G Schbirid
210600-210699 Done, archived 66 5.5G Schbirid
210700-210799 Done, archived 55 6.8G Schbirid
210800-210899 Done, archived 60 13G Schbirid
210900-210999 Done, archived 47 13G Schbirid
210100-210199 Done, archived 43 9.1G Schbirid
211100-211199 Done, archived 22 2.5G Schbirid
211200-211299 Done, archived 52 7.5G Schbirid
211300-211399 Done, archived 56 5.5G Schbirid
211300-211399 Done, archived 56 5.5G Schbirid
211400-211499 Done, archived 49 7.9G Schbirid
211500-211599 Done, archived 28 4.7G Schbirid
211600-211699 Done, archived 47 7.3G Schbirid
211700-211799 Done, archived 46 22G Schbirid
211800-211899 Done, archived 50 4.3G Schbirid
211900-211999 Done, archived 72 8.8G Schbirid
212000-212099 Done, archived 70 9.5G Schbirid
212100-212199 Done, archived 53 16G Schbirid
212200-212299 Done, archived 68 11G Schbirid
212300-212399 Done, archived 73 13G Schbirid
212400-212499 Done, archived 67 11G Schbirid
212500-212599 Done, archived 77 12G Schbirid
212600-212699 Done, archived 43 2.8G Schbirid
212700-212799 Done, archived 60 6.6G Schbirid
212800-212899 Done, archived 52 5.5G Schbirid
212900-212999 Done, archived 78 16G Schbirid
213000-213099 Done, archived 64 16G Schbirid
213100-213199 Done, archived 62 3.8G Schbirid
213200-213299 Done, archived 72 20G Schbirid
213300-213399 Done, archived 85 11G Schbirid
213400-213499 Done, archived 78 11G Schbirid
213500-213599 Done, archived 68 11G Schbirid
213600-213699 Done, archived 71 9.6G Schbirid
213700-213799 Done, archived 79 8.8G Schbirid
213800-213899 Done, archived 79 13G Schbirid
213900-213999 Done, archived 62 11G Schbirid
214000-219999 In progress Schbirid
220000-220499 Done, archived (bad URL) 250 35G Schbirid
220500-220999 Done, locally (500KB/s up, huge file, someone redo in 100st once rest is done) 322 54G Debianer
221000-221999 Done, locally (500KB/s up, huge file, someone redo in 100st once rest is done) 676 80G Debianer
222000-222099 Done, locally 75 15G underscor
222100-222199 Done, locally 75 11G underscor
222200-222299 Done, locally 70 24G underscor
222300-222399 Done, locally 65 16G underscor
222400-222499 Done, locally 78 12G underscor
222500-222599 Done, locally 58 3.2G underscor
222600-222699 Done, locally 91 7.0G underscor
222700-222799 Done, locally 77 19G underscor
222800-222899 Done, locally 61 9.6G underscor
222900-222999 Done, locally 75 8.2G underscor
223000-223099 Done, locally 80 11G underscor
223100-223199 Done, locally 60 8.2G underscor
223200-223299 Done, locally 65 13G underscor
223300-223399 Done, locally 70 30G underscor
223400-223499 Done, locally 84 11G underscor
223500-223599 Done, locally 74 3.3G underscor
223600-223699 Done, locally 38 1.4G underscor
223700-223799 Done, locally 61 3.4G underscor
223800-223899 Done, locally 68 7.9G underscor
223900-223999 Done, locally 71 13G underscor
224000-224099 Done, locally 47 480M underscor
224100-224199 Done, locally 50 4.4G underscor
224200-224299 Done, locally 41 6.7G underscor
224300-224399 Done, locally 71 2.1G underscor
224400-224499 Done, locally 65 7.7G underscor
224500-224599 Done, locally 62 9.4G underscor
224600-224699 Done, locally 56 958M underscor
224700-224799 Done, locally 58 4.0G underscor
224800-224899 Done, locally 57 5.0G underscor
224900-225000 Done, locally 51 351M underscor

Graphs

Fileplanet number of IDs from the sitemaps per 1k range.png