AOL

From Archiveteam
Jump to navigation Jump to search
AOL
AOL Screen Shot 2013-01-27 at 8.42.32 PM.png
Status Offline as of April 2018
Archiving status Partially saved
Archiving type Unknown
IRC channel #aohell (on hackint)

This is about archiving the original AOL, not AOL's current website. The AOL system is currently in major disrepair. It is as if they left the machines sitting in the datacenter, and as they die, they do not fix any issues. There is much broken infrastructure.

Shutdown and Current Status

As of September 2018, AOL has closed down all methods of getting the old data. Back in March 2018, Matan2001 pointed out that AOL for Mac OS X works, but only on PPC's or lower. In August 2018, I managed to get a Tiger VM fully working and attempted to connect using the notes Matan2001 put on the wiki, but kept getting an error as seen here: https://www.youtube.com/watch?v=_1oFcnEkBWI

Upon closer inspection and running a Man-in-the-middle attack on the application, I found that the program was querying americaonline.aol.com which is no longer registered as a valid subdomain

User:Adinbied tried finding old DNS records and started digging into the system a bit more, and found that the old system ran on the AOL Transit Data Network, which seems to be completely dead. He also went ahead and grabbed all of the data we did manage to grab from the AOL grab from here: https://archive.org/details/archiveteam_aol and started to look at what exactly the files consist of. Here's a WinDirStat overview of the files, with the largest percentage of the files being .zip, .exe, .wav, and .sit : https://i.imgur.com/0bOLxF1.jpg

Getting Started

You'll need to sign up for a user account here: https://new.aol.com. Not every field is required- phone definitely isn't. Make sure there are no special chars in the username or password, and the username and password is short (8 chars or less).

Client software

As of the middle of April 2018, the AOL network was finally taken off the air and you are not able to log into any client program.

AOL has discontinued and disabled by January 2018 all but one of their clients. All of their Windows clients aren't working, and if you try to log in, the program forces you out.

The only functioning client for a few months was "AOL for Mac OS X" (not to be confused with "AOL Desktop for Mac"). It came up with the "discontinued" message but you could just click "Learn More" and get into the program regularly.

Setup

The program unfortunately works on Mac OS X 10.5 Leopard and lower (as it is a PPC application).

Windows users can try and download an ISO of 10.5 and install it on VirtualBox, although this takes quite a long time and requires fiddling with settings.

Protocol

Diagrams

Quantumlink Packet
+---+---+---+---+---+---+---+---+---+======+---+
| Z |      CRC      |  Seq  |ProgID | Data |CR | 
+---+---+---+---+---+---+---+---+---+======+---+
P3 Packet (old style)
+---+---+---+---+---+---+---+======+---+
| Z |  CRC  |  Seq  | Token | Data |CR | 
+---+---+---+---+---+---+---+======+---+
P3 Packet (new style)
+---+---+---+---+---+---+---+---+---+======+---+
| Z |      CRC      |  Seq  | Token | Data |CR | 
+---+---+---+---+---+---+---+---+---+======+---+

Definitions

P3

Communication protocol used to communicate over lossy channels (but can be used over TCP).

There's two types of the packet:

  1. the old one with a plain CRC
  2. the new one with a CRC where it is encoded redundantly as defined in the Q-Link protocol.

It consists of packets containing magic start byte, CRC, sequence number, packet type, data, and the magic stop byte.

Form Definition Operator / Form Display Operation (FDO91 or FDO)

Form display convention or protocol. It consists of a Token and an Atom Stream. It goes into the data portion of a P3 packet. Officially, it's a programming language—but in reality—it's like if someone mixed a scripting language, a state machine, X11 display protocol, tree structure, database, GUI toolkit, and RPC protocol into one big ugly mess.

FDO91 is for Windows and Macintosh. FDO88 is for Apple II.

Token

A Token is a 2 byte value used to dispatch handlers to handle the Atom Stream.

Atom Stream

An Atom Stream consists of an Atom Stream ID and Atoms. Atom Streams are serialized (assembled) and unserialized (disassembled).

Atom

An Atom consists of a 2 byte Atom ID. The Atom ID is processed as two values. The first byte describes the category of the Atom and the second byte describes a specific command or Turing machine operation. Next is zero or more arguments of various types.

Care is needed to assemble/dissemble the Atom Streams because a framing error will cause the rest of the Atom Stream to read out like garbage.

Star Tool

The Star Tool is additional blobs that is appended or patched into an existing install of an AOL client.

Typically you find a zip file of the tools and you copy it into the install folder with the appropriate files into the appropriate subfolder (i.e., TOL files go into the folder with TOL files). When installed, appears as a * in the application's menu bar. Things such as "Invoke Database Record" are located in this menu.

Atomic Debugger

The Atomic Debugger disassembles the Atom Streams as it passes through the client.

Remote Area Information Manager (Rainman)

Protocol for displaying information (Pages) in a window.

Visual Publisher

Designs Pages to create AMP files.

Database Form ID

A 32 bit unsigned integer represented as two unsigned 16 bit integers in decimal format used to retrieve forms. For example: the ID "123-4567" is 8065495 in base 10 or 0x007B11D7.

Tool On Demand (TOD)

AOL software update.

Links

P3/FDO/Tokens/Atoms

Lots of sources: Documents covering how to make AOL forms and various such things:

Samples of custom forms:

Some FDO lessons:

About the class names:

Here is an early version of aol-files.com:

Atoms list:

More internal docs:

General

PengAOL / Penggy

PlayNet / Quantum Link (Q-Link)

Instant AOL (Linux) "Gamera"

AOL Chat

Reverse Engineering

The trunk version of Wireshark includes a dissassembler for the AOL protocol that breaks out the basic header information, such as the packet type and the token. It doesn't go into any detail about the contents of the packet, but this is a good start. This isn't available for download yet, so you'll have to build it yourself, from the svn trunk; once built wireshark reports itself as 1.9.0.

http://db48x.net/temp/Screenshot%20-%2001292013%20-%2008:28:31%20PM.png

Packet Types

INIT (x’23’)
Client sends this to the server to begin comminucation.
ACK (x’24’)
Acknowledge a packet as received, for instance an INIT or heartbeat.
SS (x’21’)
An SS requests the other end of the connection to send an SSR.
SSR (x’22’)
An SSR is a response to an SS.
NAK (x’25’)
Negative acknowledgement of a packet, when the packet was received incorrectly.
DATA (x’20’)
A packet containing data, identified by a token.
HEARTBEAT (x’26’)
The other side suspects that the line has dropped; respond with an ACK

Tokens

Each packet has a token that determines what is in the data field of the packet. Documentation for these tokens is very sparse; it's likely that AOL never had a comprehensive document listing all of them. Instead, the documentation merely tells the reader to view the list of tokens while logged into the server.

Downloading a file

  1. ← mD – client requests a file (by id?)
  2. → uJ – unicode file name
  3. → tf – start of a download; includes file name (non-unicode?); requests immediate xG ack
  4. ← xG – client acks download
  5. → FF – packet containing file data, no ack requested
  6. → F7 – packet containing file data, no ack requested
  7. ← xG – periodic acks
  8. → F9 – packet containing file data, last in sequence
  9. ← eX – mail download complete (unrelated?)
[21:24:10] <db48x> there's a packet coming from the server with a token tf
[21:24:16] <db48x> the data has a filename in it
[21:24:59] <db48x> the data is in a series of packets with token FF and F7 (no explanation of the difference is available)
[21:25:24] <balrog_> but like when you view a file library 
[21:25:34] <balrog_> how does it tell the server which library to display?
[21:25:36] <db48x> the last packet of the file has token F9
21:25:42] <db48x> haven't figured that out yet
[21:25:56] <balrog_> ah
[21:26:01] <db48x> before this file in the capture there are packets with tokens EB and uJ going from the client to the server
[21:26:03] <balrog_> none of the documentation covers this?
[21:26:09] <balrog_> aaah
[21:26:44] <db48x> and mD
[21:26:51] <db48x> and tokens AT and tD coming back
[21:29:29] <db48x> looks like the tD coming back has the metadata in it
[21:30:50] <balrog_> http://sicexcels.tripod.com/~SicExcels/rm-vpd_info/TokenTypes_Basic.txt
[21:31:12] <balrog_> http://sicexcels.tripod.com/~SicExcels/rm-vpd_info/TokenTypes_Plus.txt
[21:31:16] <balrog_> quite incomplete 
[21:33:26] <db48x> mD = download now, then
[21:34:31] <db48x> and an mF, file description
[21:34:41] <db48x> followed by an AT with a bunch of data
[21:35:35] <db48x> looks like labels for buttons like 'download now', 'download later', 'ask the staff', 'related files'
[21:35:56] <db48x> packet 538
[21:37:19] <db48x> continues in the next AT packet, 540, which looks like it has the description in it
[21:37:29] <db48x> talks about using ShrinkIt to unpack the file

Retransmissions

In normal transmission, messages are being passed in both directions. Each message sent carries the number of the last message correctly received, which is an implicit acknowledgement of all messages up to and including that one. When a message is received correctly, it is passed up to the application level. Then the response number of the message is examined. If it acknowledges any messages currently in the buffer, they are dropped from the buffer. If the receiver of the message has received a certain number of messages without acknowledging, it will send an ACK to keep the sender’s window from closing. (A window is closed when the buffer of sent messages is full, preventing any more transmissions.)

If a single message gets mangled, the receiver will get a bad checksum, and send a NAK (assuming its window is open) requesting re-transmission of all messages starting at the mangled one. It will then ignore out of sequence messages until it gets the mangled message correctly. If its window is closed, and there is no NAK queued, it will queue the NAK for transmission when the window opens. If there is a NAK queued already, it will ignore the new one.

The same NAK logic would apply to messages received out of sequence, assuming that a NAK had not already been sent.

In all cases, where a numbered message is sent, the window is checked. If it is closed, an SS is sent to try to re-open the wondow. When an SS has been sent, and no SSR has been received, all NAKs are accepted, but they are ignored, instead of being processed.

When a SSR is received, any messages that were not received are queued for transmission. When there is a message to send, and the window is open, it is sent and put into buffer. If the window is closed, the message is queued for transmission. This is separate from NAK queue.

URLs

From http://www.applefritter.com/aol: The url for the Apple II New Files library is aol://4400:8287, and here is the URL for the UnForkIt file, contained in that library: aol://4401:8287:636250. The first value identifies the resource type. In this case, either 4400 for a library, or 4401 for a file. The second number, 8287, is the library ID. 636250 is the file ID. The file IDs are not consecutive within libraries.

aol://nnnn

  • 1722: Keywords
  • 2719: Chatrooms (Private room through keyword: aol://2719:2-2-room name)
  • 3548: User profiles
  • 4344: Interactive page
  • 4400: File libraries
  • 4401: Files
  • 586x: ???
  • 9293: IM: aol://9293:[sn] (from http://justinakapaste.com/category/aolaim-tutorials/)

Examples

  • aol://4344:1264.a2main.10029531.514525857
  • aol://4400:8287
  • aol://4344:1264.a2abt.10037404
  • aol://4344:117.mtv.591130
  • aol://4344:226.llll.2755674.520114429 (Access code: 3675)

Sources

List of aol:// URLs. See Links section above for HTTP links about AOL.

Structure

        <balrog_> yes, but aol://4344:nnnn doesn't work without the extra
[19:52] <balrog_> aol://4344:1264.a2main.10029531 also works
        <balrog_> simply aol://4344:1264.a2main does not work.
[20:17] <DrainLbry> so to summarize we've got aol://4400:ID (from
                    spreadsheet), for file libraries, and
                    aol://4344:uniqueidentifier for interactive content
[20:18] <balrog_> aol://4344:uniqueidentifier:ID
        <balrog_> as per
                  http://web.archive.org/web/20060207004722/http://daol.aol.com/aolatoz
                  keywords used to be aol://1722:keyword
        <balrog_> but that's no longer working

Software

<raylee> given #aohell seems dead,
<raylee> i'll just say it here
<raylee> I just found startools aol / master aol / the debugging tools for AOL
<raylee> http://www.aciddr0p.net/aolunorgd/
<raylee> maol*.zip
<raylee> the master.tol / master.aol file goes into the tools dir.. then dbinvokes work PERFECTLY...
  • Regarding archival of file libraries: I (slipstream/raylee) made an autoit script to drive the AOL client (only works perfectly on 9.7) to get everything (metadata, and files). Here's the script. Updated 7-Sep-2014. The script only fails on connection loss or AOL client crash. (By the way, the reason it doesn't work on 7.0 and below, is because I already tried that, and random lag (if you've used these old clients you'll know what I'm talking about) basically kills the script.)

Goals

save forums/files/etc

AOL has a large number of forums on every topic, file libraries containing art, shareware, game mods, etc., etc. These should be fairly easy to enumerate, and once found it should be fairly easy to download all of the forum messages and files. Archives of these would be worth saving.

save everything

Every window that you can click on in AOL was created by a 'producer' at AOL. Many of them are essentially identical, file libraries for instance, but many are also one-offs created for a specific purpose. We ought to save these as well. Going this route will take a more thorough understanding of both the AOL protocol and the FDO scripts.

Plans

There are several ways to go about this, with tradeoffs that are only lightly explored.

custom scraper

Write a scraper in python that understands the AOL protocol and FDO scripts, and writes everything to warc files. Warc save us much of the trouble of figuring out how to organize everything on disk. they also make it much easier to create a server than can pretend to be the AOL server, or that can translate into http/html to allow anyone with a web browser to see what AOL was like.

wget-aol

Modify wget to support the AOL protocol. Very ambitious, but it would let us reuse wget's infrastructure, which may make the task easier; we'd be able to concentrate on just implementing the protocol and FDO parsing and leave the rest to wget. Would that reuse save us time, or would dealing with wget's internals drive us mad? Hard to say. This method would also allow us to create warc files.

script the client

Drive the real AOL client, perhaps with debugging tools installed, in order to capture both the FDO sources and screenshots of the rendering. Probably more fragile, but we wouldn't have to understand the actual protocol. Wouldn't be able to create warc files.

Complete client clone

An attempt to write a client library, client interface, client recording suite is located at https://github.com/chfoo/notaol/. It's far from complete; currently stuck on atom serialize/unserialize.

Archives

Archives are being uploaded to IA by godane: https://archive.org/search.php?query=creator%3A%22AOL+Files%22