@fluencelabs/registry

Search…
🌊
Aqua Book
Introduction
Getting Started
Language
Libraries
@fluencelabs/aqua-lib
@fluencelabs/aqua-ipfs
@fluencelabs/registry
Aqua CLI
Changelog
@fluencelabs/registry
Aqua implementation of Fluence Registry and ResourcesAPI
Fluence Registry is an essential part of the Fluence network protocol. It provides a Resources API that can be used for service advertisement and discovery. For more details check out our community call.
Releases
You can find the latest registry release on NPM and the changelogs are in the GitHub repo.
API
For the API implementation, take a look at resources-api.aqua in the registry repo.
Terminology
Registry - a service that provides low-level API. resources-api.aqua is built on top of it.
@fluencelabs/registry - an Aqua library on NPM that provides high-level and low-level APIs to develop custom registry scripts.
Resource/Provider - a pattern for peer/service advertisement and discovery. Providers register for a Resource that can be discovered by its resource_id.
Kademlia - an algorithm for structuring a peer-to-peer network so that peers can find each other efficiently, i.e., in no more than O(logN) hops where N is the total number of peers in the network.
Resource – a string label with associated owner_peer_id and a list of providers. A resource should be understood as a group of services or a group of peers united by some common feature. In low-level API Resource is a Registry key.
Registry Key - a structure, signed by owner_peer_id, which holds information about Resource:
1
data Key:
2
  id: string
3
  label: string
4
  owner_peer_id: string
5
  timestamp_created: u64
6
  challenge: []u8
7
  challenge_type: string
8
  signature: []u8
Copied!
resource_id - a stable identifier created from the hash of label and owner_peer_id used to identify any resource. id field of Registry Key
Resource owner - the owner_peer_id that created the resource. Other users can create resources with the same label but the identifier will be different because of the owner_peer_id.
challenge/challenge_type – dummy fields which will be used for permission management in the next Registry iterations.
Provider – a peer which is registered as a resource provider, optionally with an associated relay_id and service_id. Each provider is associated with a Registry record.
Registry record - a structure, signed by set_by peer_id, which holds information about Provider:
1
data Record:
2
  key_id: string
3
  value: string
4
  peer_id: string
5
  set_by: string
6
  relay_id: []string
7
  service_id: []string
8
  timestamp_created: u64
9
  solution: []u8
10
  signature: []u8
Copied!
provider's value – any string which can be defined and used in accordance with protocol requirements.
provider's peer_id – peer id of provider of this resource.
provider's relay_id - optional, set if provider is available on the network through this relay. 
When a provider doesn't have a publicly accessible IP address, e.g. the client peer is a browser, it connects to the network through a relay node. That means that other peers only can connect to this provider through a relay. In that case,registerProvider implicitly set relay_id toHOST_PEER_ID. 
provider's service_id - optional, id of the service of that provider.
solution – dummy field, will be used for permission checking in the next Registry iterations.
provider limit - a resource can have at most 32 providers. Each new provider added after the provider limit has been reached results in removing an old provider following the FIFO principle. Soon provider's prioritization will be handled by TrustGraph.
host provider's record - a Registry record with peer_idof a node. When a node is registered as a provider via registerNodeProvider or createResourceAndRegisterNodeProvider, the record is a host record. Host records live through garbage collection, unlike other Registry records. See Register As A Provider for details.
resource and provider lifetime - a resource and provider record are republished every  1 hour and evicted, i.e. removed, 24 hours after being unused.
So there are two types of providers. First is a node provider which lifetime controlled by this node. Second is a JS peer provider and should be renewed periodically by this peer.
script caller - a peer that executes a script by sending it to the network. In Aqua it's INIT_PEER_ID
node - usually a Fluence node hosted by the community or Fluence Team. Nodes are long-lived, can host WebAssembly services and participate in the Kademlia network.
How To Use Registry
There are several simple examples in the fluencelabs/registry repo. Give them a look.
Create A Resource
Before registering as a provider is possible, resource must be created. That's exactly what createResource does. 
Here's a simple Aqua example:
1
import "@fluencelabs/registry/resources-api.aqua"
2
import "@fluencelabs/aqua-lib/builtin.aqua"
3
​
4
func my_function(label: string) -> ?string, *string:
5
    resource_id, errors <- createResource(resource)
6
    if resource_id != nil:
7
        -- resource created successfully
8
        Op.noop()
9
    else:
10
        -- resource creation failed
11
        Op.noop()
12
        
13
    <- resource_id, errors
Copied!
Register As A Provider
There are four functions that register providers. Let's review them.
These you would use for most of your needs:
registerProvider - registersINIT_PEER_ID as a provider for existent resource.
createResourceAndRegisterProvider - creates a resource first and then registers  INIT_PEER_ID as a provider for it.
And these are needed to register a node provider for a resource:
registerNodeProvider - registers the given node as a provider for an existing resource.
createResourceAndRegisterNodeProvider - creates a resource first and then registers the given node as a provider.
Now, let's review them in more detail.
createResourceAndRegisterProvider & registerProvider
These functions register the caller of a script as a provider:
createResourceAndRegisterProvider creates a resource prior to registration
registerProvider simply adds a registration as a provider for existing resource.
createResourceAndRegisterNodeProvider & registerNodeProvider
These two functions work almost the same as their non-Node counterparts, except that they register a node instead of a caller. This is useful when you want to register a service hosted on a node.
Records created by these two functions live through garbage collection unlike records created  by registerProvider.
Here's how you could use it in TypeScript:
You first need to have export.aqua file and compile it to TypeScript, see here​
1
import {Fluence, KeyPair} from "@fluencelabs/fluence";
2
import { krasnodar } from "@fluencelabs/fluence-network-environment";
3
import {registerNodeProvider, createResource, registerProvider, resolveProviders} from "./generated/export";
4
import assert from "assert";
5
​
6
async function main() {
7
    // create the first peer and connect it to the network
8
    await Fluence.start({ connectTo: krasnodar[1] });
9
    console.log(
10
        "📗 created a fluence peer %s with relay %s",
11
        Fluence.getStatus().peerId,
12
        Fluence.getStatus().relayPeerId
13
    );
14
​
15
    let label = "myLabel";
16
    console.log("Will create resource with label:", label);
17
    let [resource_id, create_error] = await createResource(label);
18
    assert(resource_id !== null, create_error.toString());
19
    console.log("resource %s created successfully", resource_id);
20
    
21
    let value = "myValue";
22
    let node_provider = krasnodar[4].peerId;
23
    let service_id = "identity";
24
    let [node_success, reg_node_error] = await registerNodeProvider(node_provider, resource_id, value, service_id);
25
    assert(node_success, reg_node_error.toString());
26
    console.log("node %s registered as provider successfully", node_provider);
27
​
28
    let [success, reg_error] = await registerProvider(resource_id, value, service_id);
29
    console.log("peer %s registered as provider successfully", Fluence.getStatus().peerId);
30
    assert(success, reg_error.toString());
31
​
32
    let [providers, error] = await resolveProviders(resource_id, 2);
33
    // as a result we will see two providers records
34
    console.log("resource providers:", providers);
35
}
36
​
37
main().then(() => process.exit(0))
38
    .catch(error => {
39
    console.error(error);
40
    process.exit(1);
41
  });
Copied!
Renew Record Periodically
After a non-host record is created, it must be used at least once an hour to keep it from being marked stale and deleted. Also, peers must renew themselves at least once per 24 hours to prevent record expiration and deletion.
While this collection schedule may seem aggressive, it keeps the Registry up-to-date and performant as short-lived client-peers, such as browsers, can go offline at any time or periodically change their relay nodes.
Call A Function On Resource Providers
executeOnProviders
registry provides a function to callback on every Record associated with a resource:
1
func executeOnProviders(resource_id: string, ack: i16, call: Record -> ())
Copied!
It reduces boilerplate when writing an Aqua script that calls a (common) function on each provider. For example:
1
import "@fluencelabs/registry/resources-api.aqua"
2
​
3
-- API that every subscriber must adhere to
4
-- You can think of it as an application protocol
5
service ProviderAPI:
6
    do_smth(value: string)
7
​
8
func call_provider(p: Record):
9
    -- topological move to provider via relay
10
    on p.peer_id via p.relay_id:
11
        -- resolve service on a provider
12
        ProviderAPI p.service_id
13
        -- call function
14
        ProviderApi.do_smth(p.value)
15
​
16
-- call ProviderApi.do_smth() on every provider
17
func call_everyone(resource_id: String, ack: i16):
18
    executeOnProviders(resource_id, ack, call_provider)
Copied!
Passing Data To Providers
Due to the limitations in callbacks, executeOnProviders doesn't allow us to send dynamic data to providers. However, this limitation is easily overcome by using a for loop:
Consider this Aqua code:
1
import "@fluencelabs/registry/resources-api.aqua"
2
​
3
-- Application event
4
data Event:
5
    value: string
6
​
7
-- API that every provider must adhere to
8
-- You can think of it as an application protocol
9
service ProviderAPI:
10
    receive_event(event: Event)
11
​
12
func call_provider(p: Record, event: Event):
13
    -- topological move to provider via relay
14
    on p.peer_id via p.relay_id:
15
        -- resolve service on a provider
16
        ProviderAPI p.service_id
17
        -- call function
18
        ProviderAPI.receive_event(event)
19
​
20
-- send event to every provider
21
func send_everyone(resource_id: String, ack: i16, event: Event):
22
    -- retrieve all providers of a resource
23
    providers <- resolveProviders(resource_id, ack)
24
    -- iterate through them
25
    for p <- providers par:
26
        call_provider(p, event)
Copied!
Handling Function Calls
​Fluence JS SDK allows JS/TS peers to define their API through services and functions. 
Let's take the ProviderApi from the previous example and extend it a little:
1
data Event:
2
    value: string
3
    
4
service ProviderAPI:
5
    -- receive an event
6
    receive_event(event: Event)
7
    -- do something and return data
8
    do_something(value: string) -> u64
9
​
Copied!
Let's save this file to provider_api.aqua and compile it
1
aqua -i . -o src/generated
Copied!
1
import { Fluence } from "@fluencelabs/fluence";
2
import { krasnodar } from "@fluencelabs/fluence-network-environment";
3
import { registerProviderAPI, ProviderAPIDef } from "./generated/provider_api"
4
    
5
async function main() {
6
    await Fluence.start({ connectTo: krasnodar[2] });
7
    
8
    let service_id = 'api';
9
    let counter = 0;
10
    
11
    await registerProviderAPI(service_id, {
12
        receive_event: (event: any) => {
13
            console.log("event received!", event);
14
        },
15
        do_something: (value: any) => {
16
            counter += 1;
17
            console.log("doing logging!", value, counter);
18
            return counter;
19
        }
20
    });
21
}
22
​
23
main().then(() => process.exit(0))
24
    .catch(error => {
25
        console.error(error);
26
        process.exit(1);
27
    });
Copied!
Overcoming The Record Limit
If your app requires more than 32 providers for a single resource, then it's time to think about a custom WebAssembly service that stores all these records. Basically a simple "records directory" service.
With such a service implemented and deployed, you can use resources-api.aqua to register that "records directory" service and host as provider. Depending on your app's architecture, you might want to have several instances of "records directory" service. 
The code to get all records from "directory" services might look something like this in Aqua:
1
import "@fluencelabs/registry/resources-api.aqua"
2
​
3
service RecDir:
4
    get_records(resource_id: string) -> []Record
5
​
6
func dir_subscribers(resource_id: String, ack: i16) -> [][]Record:
7
    -- this stream will hold all records
8
    allRecs: *[]Record
9
    -- retrieve RecDir records from Registry
10
    providers <- resolveProviders(resource_id, ack)
11
    -- iterate through all RecDir services
12
    for dir <- providers:
13
        on dir.peer_id:
14
            RecDir dir.service_id
15
            -- get all records from RecDir and write to allRecs
16
            allRecs <- SubDir.get_records(resource_id)
17
    <- allRecs
Copied!
Concepts
Kademlia Neighborhood
Fluence nodes participate in the Kademlia network. Kademlia organizes peers in such a way that given any key, you can find a set of peers that are "responsible" for that key. That set contains up to 20 nodes.
That set is called "neighborhood" or "K-closest nodes" (K=20). In Aqua, it is accessible in aqua-lib via the Kademlia.neighbourhood function.
The two most important properties of the Kademlia neighborhood are: 
1) it exists for any key 
2) it is more or less stable
Data Replication
On write
When a registration as a provider for a resource is done, it is written to the Kademlia neighborhood of that resource_id. Here's a registerProvider implementation in Aqua:
1
-- Register for a resource as provider
2
-- Note: resource must be already created
3
func registerProvider(resource_id: ResourceId, value: string, service_id: ?string) -> bool, *Error:
4
  success: *bool
5
  error: *Error
6
  relay_id: ?string
7
  relay_id <<- HOST_PEER_ID
8
​
9
  t <- Peer.timestamp_sec()
10
​
11
  on HOST_PEER_ID:
12
    record_sig_result <- getRecordSignature(resource_id, value, relay_id, service_id, t)
13
​
14
    if record_sig_result.success == false:
15
      error <<- record_sig_result.error!
16
      success <<- false
17
    else:
18
      record_signature = record_sig_result.signature!
19
      -- find resource
20
      key, error_get <- getResource(resource_id)
21
      if key == nil:
22
        appendErrors(error, error_get)
23
        success <<- false
24
      else:
25
        successful: *bool
26
        -- get neighbourhood for the resource_id
27
        nodes <- getNeighbours(resource_id)
28
        -- iterate through each node in the neighbourhood
29
        for n <- nodes par:
30
          error <<- n
31
          on n:
32
            try:
33
              -- republish key/resource
34
              republish_res <- republishKey(key!)
35
              if republish_res.success == false:
36
                error <<- republish_res.error
37
              else:
38
                -- register as a provider on each node in the neighbourhood
39
                put_res <- putRecord(resource_id, value, relay_id, service_id, t, record_signature)
40
                if put_res.success:
41
                  successful <<- true
42
                else:
43
                  error <<- put_res.error
44
​
45
        timeout: ?string
46
        -- at least one successful write should be performed
47
        join successful[0]
48
        par timeout <- Peer.timeout(5000, "provider hasn't registered: timeout exceeded")
49
​
50
        if timeout == nil:
51
          success <<- true
52
        else:
53
          success <<- false
54
          error <<- timeout!
55
​
56
  <- success!, error
Copied!
This ensures that data is replicated across several peers.
At rest
Resource Keys and Provider records are also replicated "at rest". That is, once per hour all stale keys and records are removed and replicated to all nodes in the neighborhood, once per day all expired keys and records are removed.  
This ensures that even if a neighborhood for a resource_id has changed due to some peers go offline and others join the network, data will be replicated to all nodes in the neighborhood.
...
For advanced users accustomed to Aqua scripts:
There's an implementation of "at rest" replication for Registry on GitHub​