Introduction

We announced the Zephyr VM about a couple of weeks ago. If you haven't already, we recommend giving a quick read at the Zephyr announcement before reading this blog post.

Today, we have no announcements, just a preview of what we have developed so far: we will write, compile and deploy to Mercury the first public Zephyr application!

The contents of this article are written based on code, SDKs and functionalities that are not public yet, thus you won't be able to emulate what I'm doing here. This post serves the purpose of introducing interested users and developers to the anatomy of a Zephyr program and what the current progress is.

Also, stay tuned on X as we will publish a video walkthrough of what's going on here where I'll be explaining some stuff about Zephyr and try out live the deployment and result of this Zephyr program.

We are also thinking of making live stream challenges for writing Zephyr programs

Creating the Application

We want to write a Zephyr program that is simple for demonstrative purposes, but that also showcases one of the use cases of Zephyr: custom data manipulation.

Idea

Imagine for a minute that we are a liquidity pools protocol; we would like a way of querying prices for every ledger (assuming that they changed) and also the volume of our pools. This requires us to put our hands on the data (i.e manipulate) before it gets saved on the database as we need to calculate prices and volume. That is because we want to get prices and volume directly, so we don't want to subscribe to instance, query and decode it, map the keys and perform the calculations client side.

Additionally, we want to be able to form queries along the lines of:

query Prices {
  allPrices {
    nodes {
      a,
      b,
      volume,
      ledger
    }
  }
}

Where a and b is the price of the respective token. This would allow us to have a convenient query that is consistent across all clients. We can easily achieve this behaviour by writing a Zephyr program and deploying it on Mercury.

Note that this is a very simple application of Zephyr, which also doesn't use all the fuctionalities that this experimental version provides such as reading from the database for multi-step workflows.

Theory

We know what we want, and that's great! Now it's time to understand how to go about it in our Zephyr application.

First, we need to understand what data we want. To calculate prices and the volume we can easily rely on the two reserves (a, b) of a Soroban liquidity pool:

• price of a is $\frac{b}{a}$
• price of b is $\frac{a}{b}$
• volume is a + b

Okay so we know that when a swap, deposit or withdrawal occurs we want to capture the new reserves for a and b, calculate prices and volume, and then finally store them in the database.

Additionally, by looking at the liquidity pool code, we know that reserve is stored on contract instance.

This leaves us with three main questions:

• how do we obtain changes in the contract instance from a ledger close?
• how do we obtain changes to the new reserves from the instance?
• how do we store data on the database?

Contract instance updates given a ledger close.

Ledger close metas contain all the information regarding what happened in the network within the ledger, so it will surely contain the updates that happened to a certain entry.

Changes to ledger entries are structured in a ledger close meta as an array of LedgerEntryChange:

pub enum LedgerEntryChange {
    Created(LedgerEntry),
    Updated(LedgerEntry),
    Removed(LedgerKey),
    State(LedgerEntry),
}

For our liquidity pool, we want to monitor for updates, so we need to pay attention to LedgerEntryChange::Updated. Luckily, the rs-zephyr-sdk contains some helpers for dealing with ledger close meta, more specifically the SDK exports a MetaReader object.

For example, given a certain meta, we could get updates in that ledger as follows:

let reader = MetaReader::new(&meta);
let EntryChanges { updated, .. } = reader.v2_ledger_entries();

To conclude this small paragraph, knowing where the data you need lies is crucial in a Zephyr application. The SDK can help in the retrieval, but you need to know what is the data you need and where it's located.

From instance storage to reserves

Instance storage is shared state of a contract, and it's an "all-in-one" package, meaning that it contains all entries associated with the instance as a map. In order to retrieve the reserve from the instance, we should iterate over each element of the instance map, and if the key of the element matches the key of the reserves, then we know that the value of the element will the reserve.

And how do we get the keys for reserve a and reserve b? We just look at the code:

#[derive(Clone, Copy)]
#[repr(u32)]
pub enum DataKey {
    TokenA = 0,
    TokenB = 1,
    TokenShare = 2,
    TotalShares = 3,
    ReserveA = 4,
    ReserveB = 5,
}

So, they key for reserve a will be a U32(4) and the key for reserve b will be a U32(5).

Storing into the database

In our situation we want to store:

• the ledger in which the change occurred
• the price of token a
• the price of token b
• the volume of the pool

The zephyr SDK provides a clean way of storing stuff in the database under the table name "prices":

Database::write_table(
	"prices",
	&["ledger", "a", "b", "volume"],
	&[
		&sequence.to_be_bytes(),
	    a_on_b_price.to_be_bytes().as_slice(),
	    b_on_a_price.to_be_bytes().as_slice(),
	    volume.to_be_bytes().as_slice(),
	],
)

Note: as you can see, we are converting the numbers to byte arrays to load them into the database. That is because in this experimental version of Mercury + Zephyr only byte arrays are allowed for Mercury DB & Zephyr interoperability. This will change in the future where there will be some basic types (like integers) besides byte arrays.

Practice

To sum up what we said in theory, here is the workflow of our program:

1. get the ledger metadata.
2. get ledger entry from the meta.
3. get updated entries from the meta.
4. iterate over updated entries:
   • check that the updated entry is from our liquidity pool contract + that the updated entry is the contract instance.
     • get reserves by iterating on the instance map.
     • calculate prices, volume and store on the database.

Imports

Before starting, let's import the needed modules and structures from the SDK:

use rs_zephyr_sdk::stellar_xdr::{self, LedgerEntry, LedgerEntryData, ScVal};
use rs_zephyr_sdk::{scval_utils, Database, EntryChanges, EnvClient, MetaReader};

Entry point

We start by defining our entry point function: on_close(). This function will have to adhere to C calling conventions.

#[no_mangle]
pub extern "C" fn on_close() {
    let meta = EnvClient::get_last_ledger_meta();
    let reader = MetaReader::new(&meta);

    let sequence = reader.ledger_sequence();
    let EntryChanges { updated, .. } = reader.v2_ledger_entries();

    for updated_entry in updated {
        if let (Some(a), Some(b)) = reserves(updated_entry) {
            let volume = a + b;
            let a = a as f64;
            let b = b as f64;
            let a_on_b_price = b / a;
            let b_on_a_price = a / b;

            Database::write_table(
                "prices",
                &["ledger", "a", "b", "volume"],
                &[
                    &sequence.to_be_bytes(),
                    a_on_b_price.to_be_bytes().as_slice(),
                    b_on_a_price.to_be_bytes().as_slice(),
                    volume.to_be_bytes().as_slice(),
                ],
            )
        }
    }
}

Here we are getting the ledger sequence and updated entries. Then we iterator over the entries, and if a certain function reserves(updated_entry: LedgerEntry) -> (Option<i128>, Option<i128>) (which we still have to define) returns us with (Some(a), Some(b)) (reserve a and b), then we calculate prices and volume and finally write a new row to the prices table.

The reserves function

The function reserves() takes an entry and returns (Some(reserve_a), Some(reserve_b)) if the entry is the liquidity pool's instance. To enforce this check we first hardcode the id of the liquidity pool in our program:

const LP_CONTRACT: [u8; 32] = [
    113, 57, 185, 164, 248, 21, 164, 17, 241, 133, 16, 240, 36, 190, 54, 250, 162, 120, 240, 16,
    55, 18, 230, 202, 63, 15, 152, 25, 21, 128, 27, 70,
];

Note: in this example we need to hardcode this value. However, we plan on Zephyr applications to have some (small) store attached to the binary itself. This store will support being dynamically changed and will be a better fit for values like contract addresses to monitor.

next, we proceed with matching the contract address and the keys:

fn reserves(updated_entry: LedgerEntry) -> (Option<i128>, Option<i128>) {
    let reserve_a_key = scval_utils::to_datakey_u32(4);
    let reserve_b_key = scval_utils::to_datakey_u32(5);

    let mut reserve_a = None;
    let mut reserve_b = None;

    if let LedgerEntryData::ContractData(data) = &updated_entry.data {
        let contract = match &data.contract {
            stellar_xdr::ScAddress::Contract(id) => id.0,
            stellar_xdr::ScAddress::Account(_) => {
                unreachable!()
            }
        };

        if contract == LP_CONTRACT && data.key == ScVal::LedgerKeyContractInstance {
            let val = &data.val;

            if let Some(entries) = scval_utils::instance_entries(val) {
                for entry in entries {
                    if let ScVal::I128(parts) = &entry.val {
                        let reserve = ((parts.hi as i128) << 64) | (parts.lo as i128);

                        if entry.key == reserve_a_key {
                            reserve_a = Some(reserve);
                        } else if entry.key == reserve_b_key {
                            reserve_b = Some(reserve);
                        }
                    }
                }
            }
        };
    }

    (reserve_a, reserve_b)
}

Wrapping it up

This is the final code for our Zephyr program:

use rs_zephyr_sdk::stellar_xdr::{self, LedgerEntry, LedgerEntryData, ScVal};
use rs_zephyr_sdk::{scval_utils, Database, EntryChanges, EnvClient, MetaReader};

const LP_CONTRACT: [u8; 32] = [
    113, 57, 185, 164, 248, 21, 164, 17, 241, 133, 16, 240, 36, 190, 54, 250, 162, 120, 240, 16,
    55, 18, 230, 202, 63, 15, 152, 25, 21, 128, 27, 70,
];

fn reserves(updated_entry: LedgerEntry) -> (Option<i128>, Option<i128>) {
    let reserve_a_key = scval_utils::to_datakey_u32(4);
    let reserve_b_key = scval_utils::to_datakey_u32(5);

    let mut reserve_a = None;
    let mut reserve_b = None;

    if let LedgerEntryData::ContractData(data) = &updated_entry.data {
        let contract = match &data.contract {
            stellar_xdr::ScAddress::Contract(id) => id.0,
            stellar_xdr::ScAddress::Account(_) => {
                unreachable!()
            }
        };

        if contract == LP_CONTRACT && data.key == ScVal::LedgerKeyContractInstance {
            let val = &data.val;

            if let Some(entries) = scval_utils::instance_entries(val) {
                for entry in entries {
                    if let ScVal::I128(parts) = &entry.val {
                        let reserve = ((parts.hi as i128) << 64) | (parts.lo as i128);

                        if entry.key == reserve_a_key {
                            reserve_a = Some(reserve);
                        } else if entry.key == reserve_b_key {
                            reserve_b = Some(reserve);
                        }
                    }
                }
            }
        };
    }

    (reserve_a, reserve_b)
}

#[no_mangle]
pub extern "C" fn on_close() {
    let meta = EnvClient::get_last_ledger_meta();
    let reader = MetaReader::new(&meta);

    let sequence = reader.ledger_sequence();
    let EntryChanges { updated, .. } = reader.v2_ledger_entries();

    for updated_entry in updated {
        if let (Some(a), Some(b)) = reserves(updated_entry) {
            let volume = a + b;
            let a = a as f64;
            let b = b as f64;
            let a_on_b_price = b / a;
            let b_on_a_price = a / b;

            Database::write_table(
                "prices",
                &["ledger", "a", "b", "volume"],
                &[
                    &sequence.to_be_bytes(),
                    a_on_b_price.to_be_bytes().as_slice(),
                    b_on_a_price.to_be_bytes().as_slice(),
                    volume.to_be_bytes().as_slice(),
                ],
            )
        }
    }
}

Deployment

We now have our program and are only three steps away from having this code run along with the Stellar Testnet, manipulating and storing our customized data:

• compilation
• declaring the "prices" table through the CLI.
• deploying the zephyr application through the CLI.

Compilation

We must compile our library to WebAssembly using some special RUSTFLAGS, specifically we are enabling WASM's multivalue post-MVP feature:

RUSTFLAGS="-Ctarget-feature=+multivalue -Cpanic=abort" cargo +nightly build --target wasm32-unknown-unknown --release -Zbuild-std=panic_abort,std

Declaring the prices table

Our program writes to the "prices" table, however there is no such thing in Mercury (and even if there was, the guest environment woulnd't be authorized to write to it).

This means that we need to create (or declare) our table. Our current Zephyr-CLI implementation currently allows to declare table as follows:

zephyr --jwt $MY_MERCURY_JWT new-table --name "prices" --columns 'ledger' 'a' 'b' 'volume'

Which yields:

[+] Table "zephyr_f358aba3fc0d866e252ddd29541e55f2s" created successfully 

This means that we have created table f358aba3fc0d866e252ddd29541e55f2s in the database, which is the MD5 hash for the table name and the user id associated with the Mercury JWT. In fact, zephyr-created tables are user-specific and can only be modified and written by the user owning the table.

Deploying

We can finally deploy our WASM binary using:

zephyr --jwt $MY_MERCURY_JWT deploy --wasm ./target/wasm32-unknown-unknown/release/zephyr_lp.wasm 

Our Zephyr program is now alredy operational on Mercury (actually, on my development branch of Mercury)!

Results

If we now try to invoke swaps or deposits for the pool, we can see in real time our Zephyr program writing prices to the database. For instance, I've tried doing a swap, a deposit and then a swap again and this is what we get on the Mercury API:

query Prices {
  allZephyrF358Aba3Fc0D866E252Ddd29541E55F2S {
    nodes {
      a,
      b,
      volume,
      ledger
    }
  }
}

{
  "data": {
    "allZephyrF358Aba3Fc0D866E252Ddd29541E55F2S": {
      "nodes": [
        {
          "a": "\\x400060b6ec610606",
          "b": "\\x3fdf430865468abf",
          "volume": "\\x0000000000000000000000000001d95e",
          "ledger": "\\x001fe44c"
        },
        {
          "a": "\\x400060be02fcad54",
          "b": "\\x3fdf42fadd956752",
          "volume": "\\x00000000000000000000000000024da7",
          "ledger": "\\x001fe454"
        },
        {
          "a": "\\x400071c1cefb6d74",
          "b": "\\x3fdf22a24184e1d7",
          "volume": "\\x00000000000000000000000000024e11",
          "ledger": "\\x001fe457"
        }
      ]
    }
  }
}

Note: the above results are byte arrays represented as strings as in this experimental preview of the Zephyr + Mercury integration only byte arrays are supported for the Zephyr-DB interaction. This is likely going to change in the future as we will allow types like integers to also be interchangeable.

Zephyr is still very early, but its fundamentals are already working and development is going fast. Soon you'll also be able to start playing with it.

I hope this was a nice read and that you are excited about trying Zephyr yourself!