Difference between revisions of "FurAffinity"
| Line 10: | Line 10: | ||
** Comments | ** Comments | ||
** Metadata (Author, submission date, category, theme, species, gender, tags, maturity rating, photographic meta data) | ** Metadata (Author, submission date, category, theme, species, gender, tags, maturity rating, photographic meta data) | ||
Keep in mind that many FurAffinity submissions are marked "Mature" which means they are only visible to registered users. | |||
* User profiles | * User profiles | ||
| Line 18: | Line 20: | ||
* Forums | * Forums | ||
== Things to figure out. == | |||
Exactly how much space is this going to take? A shot in the dark estimate is around 10 TB. | |||
What's the average file size? The file size limit is 10 MB, but the average is certain to be far lower. This will help us be sure of how much space is needed. | |||
There are also users that have elected to not make their profile and/or gallery visible to non registered users. Should we exclude them? Is there an easy way to? | There are also users that have elected to not make their profile and/or gallery visible to non registered users. Should we exclude them? Is there an easy way to? When logged in, there doesn't seem to be any way to distinguish users that have elected to be invisible to unregistered views compared to those that have not. | ||
Revision as of 07:36, 19 November 2014
FurAffinity is the world's largest furry community. Founded in 2005, it hosts 15,000,000+ images, stories, songs, videos, poems, and other files (mostly images).
When planning to crawl FA, please bear in mind that it recently suffered a major DDoS, and the admins are likely still a bit jumpy about seeing unusually large amounts of traffic.
Things to be saved
- Media files (including jpg, gif, png, jpeg, txt, swf, doc, docx, rtf, txt, pdf, odt, mid, wav, mp3, mpeg, and others?)
- Descriptions
- Thumbnails for files (Even for image files, as they are often different than a simple crop/resize)
- Comments
- Metadata (Author, submission date, category, theme, species, gender, tags, maturity rating, photographic meta data)
Keep in mind that many FurAffinity submissions are marked "Mature" which means they are only visible to registered users.
- User profiles
- Blogs
- Page comments
- Favorites
- Forums
Things to figure out.
Exactly how much space is this going to take? A shot in the dark estimate is around 10 TB.
What's the average file size? The file size limit is 10 MB, but the average is certain to be far lower. This will help us be sure of how much space is needed.
There are also users that have elected to not make their profile and/or gallery visible to non registered users. Should we exclude them? Is there an easy way to? When logged in, there doesn't seem to be any way to distinguish users that have elected to be invisible to unregistered views compared to those that have not.