Substreams Integration

Example

We have integrated the Ambient protocol as a reference, see /substreams/ethereum-ambient for more information.

Step by step

Install Rust, you can do so with the following command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Install Substreams CLI, you can either use brew:

brew install streamingfast/tap/substreams

use precompiled binaries

# Use correct binary for your platform
LINK=$(curl -s https://api.github.com/repos/streamingfast/substreams/releases/latest | awk '/download.url.*linux/ {print $2}' | sed 's/"//g')
curl -L  $LINK  | tar zxf -

or compile from source:

git clone https://github.com/streamingfast/substreams
cd substreams
go install -v ./cmd/substreams

Start by making a local copy of the Propeller Protocol Lib repository:
```
git clone https://github.com/propeller-heads/propeller-protocol-lib
```

Understanding the Substreams integration

Substreams is a new indexing technology, which uses Rust modules to compose raw blockchain data streams into higher level data streams, in out case specific to the protocol. These modules together with the protobuf definitions and manifest are then wrapped into SPKG packages (more info here) that is then run remotely on the Substreams server.

For more information, read the quick explanation of Substreams or jump into the Substreams documentation. It describes the functions that need to be implemented as well as the manifest file.

ProtoBuf files

Generally these describe the raw blockchain data that we get on the input stream and the output data that we want to produce using the Rust module.

If you are unfamiliar with ProtoBuf at all, you can start with the official documentation.

First get familiar with the raw ProtoBuf definitions provided by us:

common.proto - Common types used by all integration types
vm.proto - Types specific to the VM integration

You can also create your own intermediate ProtoBufs. These files should reside in your own substreams package, e.g. ./substreams/ethereum-template/proto/custom-messages.proto. You have to link these files in the substreams.yaml file, see the manifest docs for more information or you can look at the official substreams example integration of UniswapV2.

Note: Internally we are referring to the substreams integration as Tycho, which is why our protobuf files are under the proto/tycho directory.

Rust module

The goal of the rust module is to implement the logic that will transform the raw blockchain data into the desired output data.

This is the actual integration code that you will be writing!

The module is a Rust library that is compiled into a SPKG (.spkg) file using the Substreams CLI and then loaded by the Substreams server. It is defined by the lib.rs file (see the Ambient reference example).

Read our Substreams README.md for more information on how to write the Rust module.

Testing

Read the Substreams testing docs for more information on how to test your integration.

3.8 KiB Raw Blame History