VM Integration

Our indexing integrations use the Substreams library to transform raw blockchain data into higher level data streams. This is done by implementing a Rust module that is compiled into a SPKG file and then loaded by the Substreams server.

Example

We have integrated the Balancer protocol as a reference, see /substreams/ethereum-balancer for more information.

Step by step

Install Rust, you can do so with the following command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Install Substreams CLI, you can either use brew:

brew install streamingfast/tap/substreams

use precompiled binaries

# Use correct binary for your platform
LINK=$(curl -s https://api.github.com/repos/streamingfast/substreams/releases/latest | awk '/download.url.*linux/ {print $2}' | sed 's/"//g')
curl -L  $LINK  | tar zxf -

or compile from source:

git clone https://github.com/streamingfast/substreams
cd substreams
go install -v ./cmd/substreams

Start by making a local copy of the Propeller Protocol Lib repository:
```
git clone https://github.com/propeller-heads/tycho-protocol-sdk
```

Understanding the Substreams VM integration

Substreams is a new indexing technology, which uses Rust modules to compose raw blockchain data streams into higher level data streams, in our case specific to the protocol. These modules together with the protobuf definitions and manifest are then wrapped into SPKG packages (more info here) that are then run remotely on the Substreams server.

For more information, read the quick explanation of Substreams or jump into the Substreams documentation. It describes the functions that need to be implemented as well as the manifest file.

ProtoBuf files

Generally these describe the raw blockchain data that we get on the input stream and the output data that we want to produce using the Rust module.

If you are unfamiliar with ProtoBuf at all, you can start with the official documentation.

First get familiar with the raw ProtoBuf definitions provided by us:

common.proto - Common types used by all integration types

You can also create your own intermediate ProtoBufs. These files should reside in your own substreams package, e.g. ./substreams/ethereum-template/proto/custom-messages.proto. You have to link these files in the substreams.yaml file, see the manifest docs for more information or you can look at the official substreams example integration of UniswapV2.

Note: Internally we are referring to our indexing library as Tycho, which is why our protobuf files are under the proto/tycho directory.

Rust module

The goal of the rust module is to implement the logic that will transform the raw blockchain data into the desired output data.

This is the actual integration code that you will be writing!

The module is a Rust library that is compiled into a SPKG (.spkg) file using the Substreams CLI and then loaded by the Substreams server. It is defined by the lib.rs file (see the Balancer reference example).

Read our Substreams README.md for more information on how to write the Rust module.

How to implement the integration

Create a new directory for your integration by cloning the template, rename all the references to ethereum-template to [CHAIN]-[PROTOCOL_SYSTEM]:
```
cp -r ./substreams/ethereum-template ./substreams/[CHAIN]-[PROTOCOL_SYSTEM]
```
Implement the logic in the Rust module lib.rs. The main function to implement is the map_protocol_changes function, which is called for every block.
```
#[substreams::handlers::map]
fn map_protocol_changes(
    block: eth::v2::Block,
) -> Result<tycho::BlockChanges, substreams::errors::Error> {}
```
The map_protocol_changes function takes a raw block as input and returns a BlockChanges struct, which is derived from the BlockChanges protobuf message in common.proto.
The BlockChanges is a list of TransactionChanges, which includes these main fields:
- list of ContractChange - All storage slots that have changed in the transaction for every contract tracked by any ProtocolComponent
- list of EntityChanges - All the attribute changes in the transaction
- list of ProtocolComponent - All the protocol component changes in the transaction
- list of BalanceChange - All the token balances changes in the transaction
See the Balancer reference example for more information.
If you are more advanced with Substreams, you can define more steps than a single "map" step, including defining your own protobuf files. Add these protobuf files in your pb folder and update the manifest accordingly. This allows for better parallelization of the indexing process. See the official documentation of modules.

Testing

Read the Substreams testing docs for more information on how to test your integration.

5.8 KiB Raw Blame History