Spec - Source
Sources are used as a generic interface for reading, observing, and saving data. It is usually backed by a database like PostgreSQL or the file system. Sources are often made from other Sources- for example the client will use a network Source which connects to a Source on the server.
Source Properties/Methods
A Source is any object with the following properties:
isConnected - an observable behavior subject of a boolean that represents connectivity to the upstream Source. May falter due to network issues, may be false during startup, and should be false after closing.
close - function for you to call when done using the source. It will release resources such as network connections or memory.
observeDoc - function that returns an observable behavior subject of a doc.
observeDocChildren - function that returns an observable subject of add/remove events to the list of children.
dispatch - async function where actions can be passed, and a JSON response will be returned
id - source id string
Concept - Docs and Blocks
Blocks are immutable chunks of JSON data. All data is saved in blocks, which are identified by a checksum so that any party holding the data can be sure they agree, without transmiting the actual data.
Blocks can contain references to other blocks by their id. There is a special "BlockReference" format for this. Blocks can be uploaded in bulk by providing values alongside the references. The source is responsible for recursively finding references, uploading their values, and embedding the Block id in the BlockReference object instead of the value.
Docs are named, lightweight references to Blocks. They can be created with a name via PutDoc(Value), and they can be created with an automatically unique name via PostDoc. They can renamed/moved with MoveDoc, and deleted with DestroyDoc.
No Blocks are ever renamed or destroyed when working with Docs. They will be cleaned up with a garbage collection process, which is still a work in progress..
Concept - Domains
A Domain is a container for a set of Docs, (and, by reference, Blocks)
All Doc actions should be provided by a domain.
Concept - Doc Heirarchy
Docs are all saved under the context of a domain, but each doc can have children. A Doc named "person/profilePhoto", has a local name of "profilePhoto" and a parent Doc named "person".
Concept - Doc Listing
To know all the current Docs in a domain, the process is a bit intricate:
Start by subscribing to the list of docs via observeDocChildren
Call ListDocs to see the names of child docs.
If ListDocs returns hasMore, repeatedly call ListDocs again with afterName to get the full list.
Add or remove any listed Docs according to the events from observeDocChildren
At this point, you may know the full list of top-level Docs for a given Domain, but there might be children Docs. You can recursively repeat the above process to see the entire doc heirarchy of a Domain.
API - Dispatch Actions
The following actions can all be dispatched asyncronously with the `source.dispatch` function.
const result = await source.dispatch({
type: 'GetStatus',
// result may be { isConnected: true }
Set a Doc's current Block id reference. The specified Block must already be known by the Source for PutDoc to succeed
const result = await source.dispatch({
type: 'PutDoc',
domain: 'main',
name: 'TodoEvents',
id: 'xyz',
// no return value. If the action succeeds, the ID of "main" has been set.
This is a low-level API for setting the reference of a doc. If you want transaction safety, use PutDocValue or PutTransactionValue instead.
Create a new Block, saving a chunk of JSON in the Source.
This is a low-level API for saving data to the Source. It is disallowed in a Protected Source because the ownership of all data should known. Generally you should use PutDocValue to write data.
A block is a chunk of data that is not necessarily associated with any single domain or doc. As such, a garbage collection mechanism should be deleting unreferenced blocks beyond a certain age. So, you should generally not expect a block to stick around inside a source unless you quickly refer a doc to it.
const result = await source.dispatch({
type: 'PutBlock',
value: { foo: 'bar' }
// returns { id: 'abc' }, where abc is the id(checksum) of the saved Block
This is a primary mechanism for writing data to the Source. Effectively it performs a PutBlock for a chunk of data, then does a PutDoc to refer to the new Block.
You can also upload embedded blocks that will be de-referenced by the source and uploaded seperately:
const result = await source.dispatch({
type: 'PutDocValue',
domain: 'main',
name: 'People',
value: {
personA: { type: 'BlockReference', value: { firstName: 'A' }},
personB: { type: 'BlockReference', value: { firstName: 'B' }},
// The following blocks will now be uploaded:
// Block#A = { firstName: 'A' }
// Block#B = { firstName: 'B' }
// Block#C = {
// personA: { type: 'BlockReference', id: 'Block#A' },
// personB: { type: 'BlockReference', id: 'Block#B' },
// }
// And, the "People" doc has been updated to point to Block#C
PutDocValue is also able to enforce the order of doc writes, by uploading a special block that refers to the doc's currently active block:
const result = await source.dispatch({
type: 'PutDocValue',
domain: 'main',
name: 'Message',
value: {
type: 'TransactionValue',
on: { type: 'BlockReference', id: '#lastBlockId' },
value: "NewValue",
// The above will only succeed if the "Message" doc currently refers to the #lastBlockId block
This action is similar to PutDocValue, because it saves a new Block and changes a Doc to refer to it as the current value. But in this case, the actual saved value is a "TransactionValue", whose ID refers to the previous active block.
// Pretend the 'TodoEvents' doc currently refers to the #lastAction block..
const result = await source.dispatch({
type: 'PutTransactionValue',
domain: 'main',
name: 'TodoEvents',
value: { type: 'TaskCompleted', id: 2 }
// A block like this will be uploaded and referred to by "TodoEvents":
// Block#newAction = {
// type: 'TransactionValue',
// on: { type: 'BlockReference', id: '#lastAction' },
// value: { type: 'TaskCompleted', id: 2 }
// }
// The result may be { id: '#newAction' }
This action is reccomended for clients who want to defer transaction merge to the server. Clients who want to perform manual merges on the data can write "TransactionValue" objects to "PutDocValue"
Create a child doc with a new unique name
// Pretend the 'TodoEvents' doc currently refers to the #lastAction block..
const result = await source.dispatch({
type: 'PostDoc',
domain: 'main',
name: 'Products',
value: { name: 'Tablet' }
// a new Doc will be uploaded at "Products/ABC" where ABC is a unique id provided by the server
// return value:
// {
// name: 'Products/ABC',
// id: '#checksum'
// }
Get a chunk of data
Get the current ID of the doc. This is a lightweight and low-level API that allows you to check the value of a Doc without bothering to get the value.
Get the current id of a Doc, and fetches the corresponding Block value as well.
Get an object describing the connection status and public configuration of a source.
List the available domains
List doc names, or specify a parent name to list children doc names. There is no recursive listing.
Provide 'parentName' to list children docs under a parent doc
Provide 'afterName' to list all children after a particular name. This is how children are paginated.
Destroy a doc and any children docs
Blocks will not be deleted as an immediate result, but they should be garbage collected later
Request the Source to forget about a Block.
Rename a doc and children docs, retaining all values. Destination name must be available
Children of a moved doc will follow. If "a" moves to "x/y", then "a/b" would get moved to "x/y/b".
Note: Complex deep moves like a>a/b/c should ideally be supported, but is not covered in tests yet, and is probably broken.
Clean up unreferenced blocks? Not yet implemented..