← Writing

Light Client in Polkadot Network

Light Client - Substrate Connect provides a secure, minimalistic approach for Decentralized Applications on Polkadot, bypassing the need for full node installations.

Polkadot · Light Client · Smoldot · Substrate · Web3

Substrate Connect

Introduction

Blockchain interactions require communication with a Peer-to-Peer (P2P) network to validate transactions and generate new blocks. Polkadot, like other blockchains, relies on a network of nodes that powers its consensus mechanism. align=center&caption=Peer-to-Peer Network&sourceTitle= Polkadot's Architecture&sourceURL=https://softtech-blog.weebly.com/outline-of-polkadot-architecture To communicate between the decentralized application and a network node in Polkadot is through a JSON RPC protocol. Typically, there are 2 main ways to do this:

  • User-Controlled Nodes: The Dapp (decentralized application) connects to a node client which the users have installed and run on their machine. These nodes are extremely secure because the users fully control it, but to install and maintain them are quietly complicated as they must be active 24/7 and connect with network rapidly.
  • Publicly-Accessible Nodes: There are several third-parties or platforms providing a publicly-accessible node client. These nodes are available every time when users need so they are more convenient to interact. However, their drawbacks are insecure and centralized because they are controlled by their owner. Substrate Connect, on the other hand, is introduced for decentralized application to minimally connect to Polkadot network instead of utilizing a centralized RPC node but still securely.

Substate Connect & The perfect alternative for Substrate: Smoldot

Firstly, Substrate was ever one of the core repositories in the Polkadot SDK to help everyone to build the unique Blockchain on the Polkadot. However, it was merged with two repositories Cumulus and Polkadot into a new repository: the Polkadot SDK. From here, Smoldot is introduced as an alternative client of Substrate-based chain. Smoldot validates the submitted transactions (from users) before sending to full nodes (centralized nodes) by itself to add those transactions to the transaction pool. Substrate Connect provides the infrastructure necessary to run light clients directly in the browsers. By default, Substrate Connect runs in the browser independently with each browser tab, install the browser extension to enable resource sharing among browser tabs. With Substrate Connect, there’s no need for a centralized RPC node; it operates like a bridge, running Smoldot in the browser extension making it possible for every tabs or website to sync with the chain. Abstractly, Smoldot is used to interact with the relay chain or a parachain, it always needs to sync with the relay chain. What about for a parachain? The parachain's finalized state is guaranteed on the relay chain. To verify whether a relay chain block is finalized, the light client needs to know the elected validators from the GRANDPA protocol [7]. GRANDPA** (GHOST-based Recursive ANcestor Deriving Prefix Agreement)** is the finality gadget that is implemented for the relay chain [9]. The Polkadot Host uses the GRANDPA finality protocol to finality blocks. The condition for Smoldot to set the block finalized is that more than 2/3 of the validators nodes voted FOR on the block and the signatures are correct. Polkadot uses NPoS (Nominated Proof-of-Stake) as its mechanism for selecting the validator set. Changes authority set or validator set are crucial to track for a node who tries to sync with the relay chain in order to verify justifications. The speed up way to sync which its proposed is warp sync. Instead of downloading the entire chain as the full sync protocol or downloading the block header history and validating the authority set changes like fast sync , it only downloads the block headers where authority set changes occurred and this things are called fragments. These fragments form the arrays and concat with the boolean value indicating whether the warp sync has been completed as the below format: P = (f_x...f_y, c) [9]

  • f_x...f_y: an array consisting of warp sync fragments.
  • c: a boolean indicating whether the warp sync has been completed. When it is up to date with the latest GRANDPA, Smoldot will start syncing with the chain similar to a full node. Smoldot only retrieve new block headers and verify by verifying the authenticity of those blocks [7]. In addition, Polkadot also uses the BABE protocol to produce the new blocks. BABE (Blind Assignmnent for Blockchain Extension) is the block production mechanism that runs between the validator nodes and determines the author or the owner of new blocks. After Smoldot verifies th block header, whether the author of the block was selected by BABE protocol. Practically, BABE separate into many epochs, with each epochs being divided into slots. All slots in each epoch are sequentially indexed starting from 0. At the beginning of each epoch, the BABE needs to run the Block Production Lottery to find out which slots it should produce a block and gossip to the other block producers. Finally, Smoldot verifies the consensus and the finality by tracking consecutively the authority set. The authority set will be obtained through the runtime when it updates the chain it acquires subsequent changes from the block headers. [7]

What does Smoldot need to run ?

In order to start syncing with a chain, the Smoldot requires the list of information called chainspec. A chain specification is the description of everything related the chain being desired connected and required for the client to successfully interact with that chain. The chainspec only consists of clientspec that look like as the following structure:

...
#[derive(Serialize, Deserialize, Clone, Debug)]
#[serde(rename_all = "camelCase")]
#[serde(deny_unknown_fields)]
pub(super) struct ClientSpec {
    pub(super) name: String,
    pub(super) id: String,

    #[serde(default)]
    pub(super) chain_type: ChainType,

    #[serde(default)]
    pub(super) code_substitutes: HashMap<u64, HexString, fnv::FnvBuildHasher>,
    pub(super) boot_nodes: Vec<String>,
    pub(super) telemetry_endpoints: Option<Vec<(String, u8)>>,
    pub(super) protocol_id: Option<String>,
    
    #[serde(default = "Default::default", skip_serializing_if = "Option::is_none")]
    pub(super) fork_id: Option<String>,
    
    #[serde(default = "Default::default", skip_serializing_if = "Option::is_none")]
    pub(super) block_number_bytes: Option<u8>,
    pub(super) properties: Option<Box<serde_json::value::RawValue>>,
    pub(super) fork_blocks: Option<Vec<(u64, HashHexString)>>,
    pub(super) bad_blocks: Option<HashSet<HashHexString, FnvBuildHasher>>,
    
    #[serde(default, skip_serializing)]
    #[allow(unused)]
    pub(super) consensus_engine: (),
    pub(super) genesis: Genesis,
    pub(super) light_sync_state: Option<LightSyncState>,
    
    #[serde(flatten)]
    pub(super) parachain: Option<ChainSpecParachain>,
}
...

From the chainspec, the chain information is built. As we can see in the code, the chain information is all the crucial information Smoldot requires to verify the consensus and finality [7]. It will be constructed from either:

  • The genesis state
  • The checkpoint
  • From the database As a result, the chain information provides the starting authority set for the warp sync process [7].
#[derive(Debug, Clone)]
pub struct ChainInformation {
    /// Header of the highest known finalized block.
    pub finalized_block_header: header::Header,

    /// Extra items that depend on the consensus engine.
    pub consensus: ChainInformationConsensus,

    /// Extra items that depend on the finality engine.
    pub finality: ChainInformationFinality,
}

However, to sync with a chain it needs much more information like that. It requires the information from other peers and therefore needs to join the peer-to-peer network.

Networking

size=medium&caption=Networking Service&alt=Networking Service&sourceTitle=Smoldot: Hello world!&sourceURL=https://hackmd.io/@s_iGZLIITG6WjSgnFX0pcg/rkmmcvBno First of all, in order to connect to other nodes Smoldot needs the networking service. It's responsible to try stay connected as much as possible to the nodes of the peer-to-peer network of the chain and maintain open substreams with them in order to send out requests and notifications [7]. The network service will create on background task for each active connection. There are 3 background tasks, the network service needs to start:

(config.tasks_executor)(
  "network-service".into(),
  Box::pin({
      let shared = shared.clone();
      let future = background_task(shared, event_senders);

      let (abortable, abort_handle) = future::abortable(future);
      abort_handles.push(abort_handle);
      abortable.map(|_| ())
  }),
);
  • Network discovery (done through a separate task). Source code
(config.tasks_executor)(
  "network-discovery".into(),
  Box::pin({
      let shared = shared.clone();
      let future = async move {
          let mut next_discovery = Duration::from_secs(5);

          loop {
              TPlat::sleep(next_discovery).await;
              next_discovery = cmp::min(next_discovery * 2, Duration::from_secs(120));

              let mut guarded = shared.guarded.lock().await;
              for chain_index in 0..shared.log_chain_names.len() {
                  let operation_id = guarded
                      .network
                      .start_kademlia_discovery_round(TPlat::now(), chain_index);

                  let _prev_value = guarded
                      .kademlia_discovery_operations
                      .insert(operation_id, chain_index);
                  debug_assert!(_prev_value.is_none());
              }
              shared.wake_up_main_background_task.notify(1);
          }
      };

      let (abortable, abort_handle) = future::abortable(future);
      abort_handles.push(abort_handle);
      abortable.map(|_| ())
  }),
);
(config.tasks_executor)(
  "connections".into(),
  Box::pin({
      let future = async move {
          let mut connections = stream::FuturesUnordered::new();
          loop {
              futures::select! {
                  new_connec = new_tasks_rx.select_next_some() => {
                      connections.push(new_connec);
                  },
                  () = connections.select_next_some() => {},
              }
          }
      };

      let (abortable, abort_handle) = future::abortable(future);
      abort_handles.push(abort_handle);
      abortable.map(|_| ())
  }),
);

Processing the networking service [7]

This task is the main task in execution task of Smoldot. It ensures the objective of the Networking Service as we mentioned that Smoldot is always connected to the peer-to-peer network. The amount of in-slots and out-slots refer, to the maximum amount of peers that can connect to Smoldot and the maximum amount of peers Smoldot connect to [7]. In the full node, the number of in_slots and out_slots is 25 for each. However, the light node only has 3 for in_slots and the out_slots is 4. Smoldot uses the libp2p protocol to coordinates the requests and responses from the connections to Smoldot and vice versa.

  • Assign out_slots: [7] Only the maximum number of slots is not reached and that peers have not had a slot yet, the combinations of notifications protocol and peers will be marked as "desired". In addition, it inserts that peer to out_peers in ChainConfig. (The assign_out_slot occurs in update_round process)
pub fn assign_out_slot(&mut self, chain_index: usize, peer_id: PeerId) {
      let chain = &mut self.chains[chain_index];

      // Check if maximum number of slots is reached.
      if chain.out_peers.len()
          >= usize::try_from(chain.chain_config.out_slots).unwrap_or(usize::max_value())
      {
          return; 
      }

      // Don't assign slots to peers that already have a slot.
      if chain.out_peers.contains(&peer_id) || chain.in_peers.contains(&peer_id) {
          return; 
      }

      self.inner.set_peer_notifications_out_desired(
          &peer_id,
          chain_index * NOTIFICATIONS_PROTOCOLS_PER_CHAIN,
          peers::DesiredState::DesiredReset, 
      );

      chain.out_peers.insert(peer_id);
  }
  • Start new connections with desired peers: [7] After all peer assignments, the list of connections should be opened.
loop {
    let start_connect = match guarded.network.next_start_connect(|| TPlat::now()) {
        Some(sc) => sc,
        None => break,
    };

    let is_important = guarded
        .important_nodes
        .contains(&start_connect.expected_peer_id);

    // Perform the connection process in a separate task.
    let task = tasks::connection_task(
        start_connect,
        shared.clone(),
        guarded.messages_from_connections_tx.clone(),
        is_important,
    );

    // Sending the new task might fail in case a shutdown is happening, in which case
    // we don't really care about the state of anything anymore.
    // The sending here is normally very quick.
    let _ = guarded.new_tasks_tx.send(Box::pin(task)).await;
}

During starting connections, the API user muse give that they use single stream or multi stream to report how connection attempt went.

pub fn next_start_connect(&mut self, now: impl FnOnce() -> TNow) -> Option<StartConnect<TNow>> {
    // Ask the underlying state machine which nodes are desired but don't have any
    // associated connection attempt yet.
    // Since the underlying state machine is only made aware of connections when
    // `pending_outcome_ok` is reached, we must filter out nodes that already have an
    // associated `PendingId`.
    let unfulfilled_desired_peers = self.inner.unfulfilled_desired_peers();

    for peer_id in unfulfilled_desired_peers {
        // TODO: allow more than one simultaneous dial per peer, and distribute the dials so that we don't just return the same peer multiple times in a row while there are other peers waiting
        let entry = match self.num_pending_per_peer.entry(peer_id.clone()) {
            hashbrown::hash_map::Entry::Occupied(_) => continue,
            hashbrown::hash_map::Entry::Vacant(entry) => entry,
        };

        let multiaddr: multiaddr::Multiaddr = {
            let potential = self
                .chains
                .iter_mut()
                .flat_map(|chain| chain.kbuckets.iter_mut_ordered())
                .find(|(p, _)| **p == *entry.key())
                .and_then(|(peer_id, _)| {
                    self.kbuckets_peers
                        .get_mut(peer_id)
                        .unwrap()
                        .addresses
                        .addr_to_pending()
                });
            match potential {
                Some(a) => a.clone(),
                None => continue,
            }
        };

        let now = now();
        let pending_id = PendingId(self.pending_ids.insert((
            entry.key().clone(),
            multiaddr.clone(),
            now.clone(),
        )));

        let start_connect = StartConnect {
            expected_peer_id: entry.key().clone(),
            id: pending_id,
            multiaddr,
            timeout: now + self.handshake_timeout,
        };

        entry.insert(NonZeroUsize::new(1).unwrap());

        return Some(start_connect);
    }

    None
}

After creating connections, Smoldot perform the connection process in a separate task

let task = tasks::connection_task(
    start_connect,
    shared.clone(),
    guarded.messages_from_connections_tx.clone(),
    is_important,
);
let _ = guarded.new_tasks_tx.send(Box::pin(task)).await;
  • The coordinator is updated about the connectivity status with a give peer. It's responsible for distributing it to the right service. [7]
  • Process requests and updates from other services: [7] Smoldot pulls the messages that the coordinator has generated in destination to the various connections and sends that messages to the active connections.
loop {
    let (connection_id, message) = match guarded.network.pull_message_to_connection() {
        Some(m) => m,
        None => break,
    };

    // Note that it is critical for the sending to not take too long here, in order to not
    // block the process of the network service.
    // In particular, if sending the message to the connection is blocked due to sending
    // a message on the connection-to-coordinator channel, this will result in a deadlock.
    // For this reason, the connection task is always ready to immediately accept a message
    // on the coordinator-to-connection channel.
    guarded
        .active_connections
        .get_mut(&connection_id)
        .unwrap()
        .send(message)
        .await
        .unwrap();
}

Network discovery

In a peer-to-peer network, it's necessary to have the information of the connection to connect to any given peer, available at any given time. To make this information decentralized, Smoldot need to divide it over all the peers through the Kademlia Distributed Hash Tables [7]. It's a distributed system for mapping keys to values in IPFS. Each information or data in IPFS would be hashed to get its digest and with each data changes the hash digest would be changed as well. In the Distributed hash table, each node was called k-buckets, which make a partial view of the complete list of all the peers in the network. [7]

pub struct KBuckets<K, V, TNow, const ENTRIES_PER_BUCKET: usize> {
    /// Key of the "local" node, that holds the buckets.
    local_key: (K, Key),
    /// List of buckets, ordered by increasing distance. In other words, the first elements of
    /// this field are the ones that are the closest to [`KBuckets::local_key`].
    buckets: Vec<Bucket<K, V, TNow, ENTRIES_PER_BUCKET>>,
    /// Duration after which the last entry of each bucket will expired if it is disconnected.
    pending_timeout: Duration,
}

The objective of Network discovery is that sending a Kademlia "find node" request to single peer. Moreover, Smoldot based on the Kademlia request-response protocol to build a wire message to get the nodes closest to the parameters [7]. The following steps is to decide which peer it sends the request message:

  • Creating a Sha256 hash digest from the a random created peerID (local_key), the target hash. [7]
pub fn new(local_key: K, pending_timeout: Duration) -> Self {
    let local_key_hashed = Key::new(local_key.as_ref());

    KBuckets {
        local_key: (local_key, local_key_hashed),
        buckets: (0..256)
            .map(|_| Bucket {
                entries: arrayvec::ArrayVec::new(),
                num_connected_entries: 0,
                pending_entry: None,
            })
            .collect(),
        pending_timeout,
    }
}
  • Finding out the k-buckets for the closest peers [7]
pub fn get(&self, key: &K) -> Option<&V> {
    let key_hashed = Key::new(key.as_ref());
    let distance = match distance_log2(&self.local_key.1, &key_hashed) {
        Some(d) => d,
        None => return None,
    };

    self.buckets[usize::from(distance)].get(key)
}

pub fn get_mut(&mut self, key: &K) -> Option<&mut V> {
    let key_hashed = Key::new(key.as_ref());
    let distance = match distance_log2(&self.local_key.1, &key_hashed) {
        Some(d) => d,
        None => return None,
    };

    self.buckets[usize::from(distance)].get_mut(key)
}
  • Hashing the peerIDs of closest peers and sorting its list with ascending order of the log2 distance between the target hash and the k-bucket peer's hash. [7]
pub fn closest_entries(&self, target: &K) -> impl Iterator<Item= (&K, &V)> {
    // TODO: this is extremely unoptimized
    let target_hashed = Key::new(target.as_ref());
    let mut list = self.iter_ordered().collect::<Vec<_>>();
    list.sort_by_key(|(key, _)| {
        let key_hashed = Key::new(key.as_ref());
        distance_log2(&key_hashed, &target_hashed).map_or(0, |d| u16::from(d) + 1)
    });
    list.into_iter()
}
  • It obtains the peer with the shortest distance and where it already has an established connection. [7]

Processing existing connections [7]

Smoldot has 2 types of connections such as TCP connection and WebRTC connection.

  • TCP is the Single Stream connection where the browser just sends or receives data and Smoldot implements substreams on top of that.
  • In the Multi stream WebRTC connection the browser deals with the substreams. There are two types of substreams: [7]
  1. Notifications:
    1. Block announcements protocol
    2. GRANDPA announcements protocol (GRANDPA commits)
    3. Transaction announcements protocol (not implemented for Smoldot)
  2. Request-response:
    1. IPFS id protocol
    2. Sync protocol
    3. Light protocol
    4. Kademlia protocol
    5. Sync warp protocol
    6. State protocol

Synchronization service [7]

After, The networking service has been set up it starts a new background task, the synchronization service. The objective of the sync service is to do whatever necessary to get and stay up to date with the best and finalized blocks of the chain. Generally, Smoldot will track a list of sources, which represent peers that Smoldot receives new block headers and GRANDPA commits from. With the received information from a given peer, it will know which blocks this peer is aware of.

Runtime service [7]

After synchronization service, the next task is the runtime service. Typically, this service wants to have the latest finalized downloaded runtime to provide to other services. Therefore, it needs to stay up to date with the chain.

Transaction service

Everything which relate to transactions will be handled by transaction service. As we know, when the transaction sent from the user to network would be injected to the pool called as transaction pool. This transactions will hold the pending status and wait until mining or validating by active nodes to add to the new blocks. What's about in Light Client? The transaction service and transaction pool are most of the time idle. Because of this, it is only operating when the user submits a transaction.

JSON-RPC service

Finally, as the last asynchronous task, the JSON-RPC service will be spawned. It holds a state machine which consists of a list of client, list of request sent by a client but not yet pulled by, and the request subscriptions

Why use Substrate Connect ?

Polkadot is designed with decentralization at its core, following the Blockchain Trilemma (Decentralization, Security, and Scalability). However, most user interfaces connect to the network through centralized nodes, making decentralization challenging. Substrate Connect provides a lightweight, decentralized option for DApps to interact securely with Polkadot, lowering complexity for end-users.

Integrating the Light Client by Smoldot in your User Interface !!!

This guide walks you through integrating a Light Client with Smoldot** , Substrate Connect **and Dedot (an alternative to PolkadotJS API ) in our demo application. Our repository is using Next.Js framework to build the User-Interface Application for Decentralized Application - Demo

Prepare the Chain Specification JSON

Firstly, we must prepare the chain specification json file for the Relaychain or Parachain. We can find the chain specification of our expected network in Substrate Connect repository.

https://github.com/paritytech/substrate-connect/tree/main/packages/connect-known-chains/specs

Install Dependencies

Install necessary packages such as smoldot , @dedot/chaintypes and a compatible version of Dedot:

npm install smoldot dedot@0.5.0 @dedot/chaintypes

Implement Substrate Connect Provider and Client

Set up the Substrate Connect Provider and Client in a custom hook for use in your Next.js or React project:

https://github.com/0xharryriddle/Voting-Escrow-Light-Client/blob/main/src/hooks/useApi.ts

align=center

Configure API Context for Project Layout

Integrate the Light Client provider across your application with an API Context setup:

https://github.com/0xharryriddle/Voting-Escrow-Light-Client/blob/main/src/providers/ApiProvider.tsx

align=center

https://github.com/0xharryriddle/Voting-Escrow-Light-Client/tree/main/src/app

align=center

Test the Light Client Connection

Run our application with:

npm run dev

If successful, you’ll see confirmation in the browser console 🎉. This confirms your Light Client is now operational within your Dapp! align=center In this context, we used the Edge Browser to access to our application.

Interact with Smart Contracts

Using the “api” client from the API Context, interact with the Relaychain or Parachain via Dedot SDK or PolkadotJS API to call smart contracts.

Challenges 💪🏻

The Smoldot SDK supports WASM-compatible browsers, so ensure your Dapp runs on a browser optimized for WASM.

Acknowledgement

This article is the part of Open Contribution Bounty initiative by the OpenGuild Community. A heartfelt thank you to OpenGuild for being a pioneering community that empowers and supports emerging developers in the Blockchain space, particularly within the Polkadot ecosystem.

References

About the Author 👋🏻

Bio : Harry Riddle, an undergraduate Web3 Developer focused on Blockchain and AI, is a member of the OpenGuild community in Vietnam. Contact info :