Synapse

A synapse is ‘smart’ sync database that will automatically synchronise data between two or more connected synapses. It is a more powerful version of a sync database that can be used to synchronise an entire sync database across multiple Volt instances.

When two or more synapses are connected together, the Volt will monitor the underlying sync database for changes and automatically synchronise such that all connected synapses will have the same document content.

This is achieved by requiring that all documents in a given synapse have a schema defined for each of the top-level shared types it contains.

As a reminder, every YJS document can have multiple shared types defined at the root level, for example, YMap and YArray and YText. These are accessed using the relevant YJS API method, for example:

const map = ydoc.getMap("my-map");
const array = ydoc.getArray("my-array");
const text = ydoc.getText("my-text");

By defining a schema for each of these top-level shared types, the Volt is able to deterministically detect when a document is modified and apply the changes to all other connected synapses.

Example

The following example demonstrates how to use a synapse to synchronise AIS vessel data across multiple Volt instances.

Setup

Note that the following example assumes that you have already created two Volts and have acquired connection credentials for them stored in ./volt-1.config.json and ./volt-2.config.json (see the Volt auth CLI command).

For the purposes of this example, we will assume that the Volt DIDs are did:volt:1 and did:volt:2 respectively.

To start, we need to create two synapses using the create-synapse command. Note that we pass the connection credentials for Volt 1 and Volt 2 to the command using the -c switch.

Terminal window
# Create a synapse with the alias @ais-synapse-1 on Volt 1
volt create-synapse "AIS vessel data" --alias ais-synapse-1 -c ./volt-1.config.json
Terminal window
# Create a synapse with the alias @ais-synapse-2 on Volt 2
volt create-synapse "AIS vessel data" --alias ais-synapse-2 -c ./volt-2.config.json

You can also use the fusebox to create synapses.

We now have two synapses, @ais-synapse-1 and @ais-synapse-2. We can connect them together using the create-synapse-connection command.

Below we connect synapse @ais-synapse-1 on Volt 1 to synapse @ais-synapse-2 on Volt 2. Note that we pass the connection credentials for Volt 1 to the command using the -c switch. In this case, the connection is created on Volt 1 and is owned by the identity in ./volt-1.config.json. It will attempt to authenticate on Volt 2 and establish a bi-directional synchronisation.

Terminal window
# Connect synapse @ais-synapse-1 on Volt 1 to synapse @ais-synapse-2 on Volt 2
volt create-synapse-connection @ais-synapse-1 did:volt:2 @ais-synapse-2 -c ./volt-1.config.json

You will need to ensure that Volt 2 has granted volt:database-write permission for @ais-synapse-2 to the Volt 1 identity.

In the event of any connection errors, the synapse will periodically retry to establish the connection until it succeeds.

Schema definition

Next, we need to define the schema for the shared types that will be used to store the AIS vessel data.

For example, the following JSON schema defines a property schema that can be used to store AIS vessel data. Copy this to a file called ais-trace-schema.json for use in the next step.

{
  "type": "object",
  "additionalProperties": {
    "type": "object",
    "x-yrs-storage": "shared",
    "properties": {
      "MMSI": { "type": "number" },
      "TSTAMP": { "type": "string" },
      "LATITUDE": { "type": "number" },
      "LONGITUDE": { "type": "number" },
      "COG": { "type": "number" },
      "SOG": { "type": "number" },
      "HEADING": { "type": "number" },
      "NAVSTAT": { "type": "number" },
      "IMO": { "type": "number" },
      "NAME": { "type": "string" },
      "CALLSIGN": { "type": "string" },
      "TYPE": { "type": "number" },
      "DRAUGHT": { "type": "number" },
      "DEST": { "type": "string" },
      "ETA": { "type": "string" }
    }
  }
}

x-yrs-storage

Note the x-yrs-storage custom JSON schema property in the above schema. This tells the Volt that the data is stored as a YJS shared type rather than a plain JSON object. The default storage type is json. This property can be used at any level in the schema, but only if the parent object is also a shared type, i.e. a plain JSON object cannot have a sub-property that is a shared type. This is useful for optimising the synchronisation deltas. For example, if the schema above omitted the x-yrs-storage property, then every time the LATITUDE property is updated, the entire Southern Cross object would be synchronised to the other synapse, even if only the LATITUDE property changed.

A document that conforms to this schema would look like this:

{
  "Southern Cross": {
    "MMSI": 203035273,
    "TSTAMP": "2024-11-19 20:10:00 GMT",
    "LATITUDE": 39.92121,
    "LONGITUDE": 7.60231,
    "COG": 144.9,
    "SOG": 4.7,
    "HEADING": 149,
    "NAVSTAT": 7,
    "IMO": 9666865,
    "NAME": "Southern Cross",
    "CALLSIGN": "KMUJ",
    "TYPE": 0,
    "DRAUGHT": 0,
    "DEST": "STARNMEER",
    "ETA": "11-26 20:10"
  },
  "Harbor Master": {
    "MMSI": 209083743,
    "TSTAMP": "2024-11-19 20:05:00 GMT",
    "LATITUDE": 44.6323,
    "LONGITUDE": 16.52741,
    "COG": 226,
    "SOG": 15.6,
    "HEADING": 231,
    "NAVSTAT": 0,
    "IMO": 9908828,
    "NAME": "Harbor Master",
    "CALLSIGN": "IMOP2",
    "TYPE": 70,
    "DRAUGHT": 0,
    "DEST": "ROTTERDAM",
    "ETA": "11-26 20:05"
  },
  ...
}

We now use the synapse-set-metadata CLI command to set the metadata for the synapse document that will contain the AIS vessel data.

Terminal window
# Set the metadata for the synapse document that will contain the AIS vessel data on Volt 1
volt synapse-set-metadata @ais-synapse-1 00000000-0000-0000-0000-000000000123 aisTrace map /path/to/ais-trace-schema.json -c ./volt-1.config.json

Note we only need to set the metadata on one of the synapses. The metadata will be synchronised to the other synapse automatically.

Writing data to a synapse

Let’s write some AIS vessel data to the synapse document, using the synapse-write-path CLI command.

Terminal window
# Write some AIS vessel data to the synapse document on Volt 1
volt synapse-write-path @ais-synapse-1 00000000-0000-0000-0000-000000000123 $.aisTrace.SouthernCross '{"MMSI": 203035273, "TSTAMP": "2024-11-19 20:10:00 GMT", "LATITUDE": 39.92121, "LONGITUDE": 7.60231}' -c ./volt-1.config.json

The data will be synchronised to the other synapse automatically.

We could also write data to the other synapse, note here we write to the LATITUDE property of the Southern Cross vessel, and we use the Volt 2 configuration file.

Terminal window
# Write some AIS vessel data to the synapse document on Volt 2
volt synapse-write-path @ais-synapse-2 00000000-0000-0000-0000-000000000123 $.aisTrace.SouthernCross.LATITUDE 39.9377 -c ./volt-2.config.json

Watching for changes to a synapse

We can also watch for changes to the synapse document using the synapse-watch-path CLI command. This will set up a long-lived connection to the synapse and print out the changes as they happen.

Terminal window
# Watch for changes to the synapse document on Volt 1
volt synapse-watch-path @ais-synapse-1 00000000-0000-0000-0000-000000000123 $.aisTrace.SouthernCross -c ./volt-1.config.json

We can watch for changes to specific vessels or any vessel using the wildcard *.

Terminal window
# Watch for changes to any vessel's NAVSTAT property
volt synapse-watch-path @ais-synapse-1 00000000-0000-0000-0000-000000000123 $.aisTrace.*.NAVSTAT -c ./volt-1.config.json

The output will indicate the path that was updated and the new value.

This will print out the changes as they happen.

Using the YJS API

So far we have been using the CLI to interact with the synapse. However, we can also use the standard Volt API to interact with the synapse via any of the client libraries.

When using the client libraries, we usually treat the synapse as a regular sync database and use the SyncProvider class to synchronise the data. The synapse running on the Volt will automatically synchronise the data to all connected synapses.

import grpc from "@grpc/grpc-js";
import { VoltClient } from "@tdxvolt/volt-client-grpc";
import { SyncProvider } from "@tdxvolt/volt-client-grpc";
import * as Y from "yjs";


const configPath = "./volt.config.json";


const voltClient = new VoltClient(grpc);
await voltClient.initialise(configPath);


const ydoc = new Y.Doc({ guid: "00000000-0000-0000-0000-000000000123" });


const syncProvider = new SyncProvider(Y, voltClient, "@ais-synapse-1", ydoc);


let aisTraceMap;


syncProvider.on("status", ({ status }) => {
  if (status === "connected") {
    console.log("connected");
    aisTraceMap = ydoc.getMap("aisTrace");


    aisTraceMap.observe((event) => {
      // Handle change event.
    });


    aisTraceMap.set("Southern Cross", {
      MMSI: 203035273,
      TSTAMP: "2024-11-19 20:10:00 GMT",
      LATITUDE: 39.92121,
      LONGITUDE: 7.60231,
    });
  } else {
    console.log("disconnected");
  }
});


syncProvider.startSync();

When writing data via the YJS API, as long as the data you write to the document conforms to the schema defined for the shared type, it will be synchronised to all connected synapses automatically. When using the YJS API, the synapse won't be able to prevent you from writing data that doesn't conform to the schema, but it might not be synchronised to other synapses.

Using the synapse API

The section above utilised the YJS API to read and write data to the synapse.You can also use any of the synapse API methods via the client libraries.

To set the metadata for a shared type use the SetSynapseDocumentMetadata API.

const request = {
  database_id: "@ais-synapse-1",
  document_id: "00000000-0000-0000-0000-000000000123",
  metadata: [
    {
      name: "aisTrace",
      type: "map",
      json_schema:
        '{"type":"object","additionalProperties":{"type":"object","x-yrs-storage":"shared","properties":{"MMSI":{"type":"number"},"TSTAMP":{"type":"string"},"LATITUDE":{"type":"number"},"LONGITUDE":{"type":"number"},"COG":{"type":"number"},"SOG":{"type":"number"},"HEADING":{"type":"number"},"NAVSTAT":{"type":"number"},"IMO":{"type":"number"},"NAME":{"type":"string"},"CALLSIGN":{"type":"string"},"TYPE":{"type":"number"},"DRAUGHT":{"type":"number"},"DEST":{"type":"string"},"ETA":{"type":"string"}}}}',
    },
  ],
};
const response = await voltClient.SetSynapseDocumentMetadata(request);
console.log(response);

To write data to a synapse use the WriteSynapsePath API.

const request = {
  database_id: "@ais-synapse-1",
  document_id: "00000000-0000-0000-0000-000000000123",
  path: "$.aisTrace.SouthernCross.LATITUDE",
  json: 39.9377,
};
const response = await voltClient.WriteSynapsePath(request);
console.log(response);

To watch for changes to a synapse use the WatchSynapsePath API.

const request = {
  start: {
    database_id: "@ais-synapse-1",
    document_id: "00000000-0000-0000-0000-000000000123",
    path: ["$.aisTrace.*.LATITUDE", "$.aisTrace.*.LONGITUDE"],
  },
};
const watchStream = await voltClient.WatchSynapsePath(request);


watchStream.on("data", (response) => {
  console.log(response);
});


watchStream.on("end", () => {
  console.log("watch ended");
});


watchStream.on("error", (err) => {
  console.log(err);
});

The synapse API methods will enforce the schema defined for the shared type and will fail if the data you write doesn't conform to the schema.