Difference between revisions of "Chromebot"

From Archiveteam
Jump to navigation Jump to search
m (No references anymore)
m (→‎People: add devops scripts)
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''chromebot''' aka. '''crocoite''' is an [[IRC]] bot parallel to [[ArchiveBot]] that uses Google Chrome and thus is able to archive JavaScript-heavy and bottomless websites. Both, [https://github.com/PromyLOPh/crocoite software] and bot, are maintained by [[User:PurpleSymphony]]. [[WARC]]s are uploaded daily to the [https://archive.org/details/archiveteam_chromebot?sort=-publicdate chromebot collection] on archive.org.
'''chromebot''' aka. '''crocoite''' is an [[IRC]] bot parallel to [[ArchiveBot]] that uses Google Chrome and thus is able to archive JavaScript-heavy and bottomless websites. [[WARC]]s are uploaded twice a day to the [https://archive.org/details/archiveteam_chromebot?sort=-publicdate chromebot collection] on archive.org.


By default the bot only grabs a single URL. However it supports recursion, which is rather slow, since every single page needs to be loaded and rendered by a browser. A [https://6xq.net/chromebot/ dashboard] is available for watching the progress of such jobs.
By default the bot only grabs a single URL. However it supports recursion, which is rather slow, since every single page needs to be loaded and rendered by a browser. A [http://chromebot.6xq.net/ dashboard] is available for watching the progress of such jobs.


== Usage ==
== Usage ==
Line 13: Line 13:
|-  
|-  
| white-space: nowrap |
| white-space: nowrap |
<code>chromebot: a <url></code><br />
<code>chromebot: a <url> -r <policy> -j <concurrency></code>
<code>chromebot: a <url> <concurrency></code><br />
<code>chromebot: a <url> <concurrency> <policy></code><br />
<code>chromebot a <url></code><br />
<code>chromebot a <url> <concurrency></code><br />
<code>chromebot a <url> <concurrency> <policy></code><br /> 
 
|| Archive ''<url>'' with ''<concurrency>'' processes according to recursion ''<policy>''.
|| Archive ''<url>'' with ''<concurrency>'' processes according to recursion ''<policy>''.
|-
|-
| <code>chromebot: s <uuid></code><br /><code>chromebot s <uuid></code> || Get job status for ''<uuid>''.
| <code>chromebot: s <uuid></code></code> || Get job status for ''<uuid>''.
|-
|-
| <code>chromebot: r <uuid></code><br /><code>chromebot r <uuid></code> || Revoke or abort running job with ''<uuid>''.
| <code>chromebot: r <uuid></code></code> || Revoke or abort running job with ''<uuid>''.
|}
|}


Please note that the commands are case-sensitive.
Please note that the commands are case-sensitive.
URL lists can be archived using recursion, for example:
<code>chromebot: a https://transfer.notkiska.pw/inline/UpfR/HollyConrad-tweets -r 1 -j 4</code>
chromebot will assume all lines starting with http(s):// are valid links. Note that the list itself must be retured by the server as an *inline* document, not as a download (attachment).


== Restrictions ==
== Restrictions ==
=== Instagram.com ===
=== Instagram ===
chromebot has been blacklisted by [[Instagram]]. When trying to archive any Instagram.com website, chromebot responds with the following error:
chromebot has been blacklisted by [[Instagram]]. When trying to archive any Instagram.com website, chromebot responds with the following error:
  ''<Instagram.com URL> cannot be queued: Banned by Instagram''
  ''<Instagram.com URL> cannot be queued: Banned by Instagram''
Line 36: Line 36:
=== Cloudflare DDoS protection ===
=== Cloudflare DDoS protection ===
chromebot should be able to circumvent Cloudflare's DDoS protection, but scrolling and other behaviour may be disabled after the reload ([https://github.com/PromyLOPh/crocoite/issues/13 issue #13 on GitHub]).
chromebot should be able to circumvent Cloudflare's DDoS protection, but scrolling and other behaviour may be disabled after the reload ([https://github.com/PromyLOPh/crocoite/issues/13 issue #13 on GitHub]).
== People ==
[[User:PurpleSymphony|PurpleSym]] maintains [https://github.com/PromyLOPh/crocoite software], [https://github.com/PromyLOPh/chromebot scripts], pays the server bills and has administrative access. katocala is a server administrator.


[[Category:Bots]]
[[Category:Bots]]

Revision as of 18:53, 30 December 2019

chromebot aka. crocoite is an IRC bot parallel to ArchiveBot that uses Google Chrome and thus is able to archive JavaScript-heavy and bottomless websites. WARCs are uploaded twice a day to the chromebot collection on archive.org.

By default the bot only grabs a single URL. However it supports recursion, which is rather slow, since every single page needs to be loaded and rendered by a browser. A dashboard is available for watching the progress of such jobs.

Usage

crocoite usage documentation on GitHub

You can call chromebot on the #archivebot (on hackint) IRC channel, which chromebot shares with ArchiveBot. Both “chromebot” and “chromebot:” work, with or without the colon.

Command Description

chromebot: a <url> -r <policy> -j <concurrency>

Archive <url> with <concurrency> processes according to recursion <policy>.
chromebot: s <uuid> Get job status for <uuid>.
chromebot: r <uuid> Revoke or abort running job with <uuid>.

Please note that the commands are case-sensitive.

URL lists can be archived using recursion, for example:

chromebot: a https://transfer.notkiska.pw/inline/UpfR/HollyConrad-tweets -r 1 -j 4

chromebot will assume all lines starting with http(s):// are valid links. Note that the list itself must be retured by the server as an *inline* document, not as a download (attachment).

Restrictions

Instagram

chromebot has been blacklisted by Instagram. When trying to archive any Instagram.com website, chromebot responds with the following error:

<Instagram.com URL> cannot be queued: Banned by Instagram

Cloudflare DDoS protection

chromebot should be able to circumvent Cloudflare's DDoS protection, but scrolling and other behaviour may be disabled after the reload (issue #13 on GitHub).

People

PurpleSym maintains software, scripts, pays the server bills and has administrative access. katocala is a server administrator.