Datastar Common Lisp SDK

================================================================

       |             
      / \            DATASTAR HYPERMEDIA FRAMEWORK
     /___\          
    |  D  |          
    | C*L |                '(DATASTAR-CL)    
    |     |                 
   /|  |  |\
  | |_ |__| |              COMMON LISP SDK
  |/  /.\  \|            

================================================================

What is it

This is a Common Lisp implementation of the Datastar SDK, following the Datastar Architecture Decision Record (ADR) as closely as possible, making the necessary adjustments for Common Lisp.

Focus has been on implementing the core SDK functionality, with additional utilities planned.

How to use it

Clone the repository somewhere ASDF2 can find it (~~/src/lisp~ is commonly already configured), and use Quicklisp to load it:

(ql:quickload "datastar-cl")

Approach

The SDK uses CLOS and is based on a sse-generator class, that currently has two subclasses:

1. hunchentoot-sse-generator: used with the Hunchentoot web server.
2. clack-sse-generator: used with the Clack web application environment.

Both approaches use a stream, although in different ways and initialized differently. For Clack, the work done is at the Lack level, and has been tested with both Hunchentoot and Woo.

We are currently using JZON as the JSON parser in two places: read-signals and patch-signals. This is not a hard requirement to remove, but for now it allows immediate use and testing of the library.

SSE and Common Lisp webservers

When using Clack with Woo, SSE connections are limited by Woo’s worker thread count when using the same approach as Hunchentoot: each SSE connection blocks one worker for its entire duration. This is a fundamental architectural constraint - see SSE-AND-WOO.org for details and practical solutions.

This was initially dealt with by pointing to using polling, essentially giving up on using Woo with long-running SSE. It was recently improved by adding a timer-based solution that uses the Woo event loop and libev.

Examples

The test/ directory has some basic examples. Additionally, the following projects use it

• Data SPICE: uses Datastar-CL for a 2D simulation of the solar system and Cassini-Huygens probe; written in parallel with the SDK to showcase it and also make sure it worked.
• Horizons JPL API explorer: in the same theme, but a vastly simplified app that queries the Horizons API for solar system data.

The Horizons JPL app is very simple – one single file – and should show the essentials. The Data SPICE application shows SSE streaming.

Compression

Support for compression was recently added, currently covering all backends but limited to zstd: read COMPRESSION.org for details.

Testing

The test/ directory contains the code to setup and start the supported servers, Hunchentoot and Clack (with Woo and Hunchentoot backends, others can be added). It’s also a good place to use as an example of how the SDK can be used, especially since there are some differences depending on the backend (Hunchentoot or Clack). SSE has a specific test file that can also serve as an example, and some important edge cases (like what happens when connections are interrupted) are tested as well.

License

The code follows Datastar’s licence choice, the MIT license.