Substreams Integration

Example

We have integrated the Ambient protocol as a reference, see /substreams/ethereum-ambient for more information.

Step by step

Install Rust, you can do so with the following command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Install Substreams CLI, you can either use brew:

brew install streamingfast/tap/substreams

use precompiled binaries

# Use correct binary for your platform
LINK=$(curl -s https://api.github.com/repos/streamingfast/substreams/releases/latest | awk '/download.url.*linux/ {print $2}' | sed 's/"//g')
curl -L  $LINK  | tar zxf -

or compile from source:

git clone https://github.com/streamingfast/substreams
cd substreams
go install -v ./cmd/substreams

Start by making a local copy of the Propeller Protocol Lib repository:
```
git clone https://github.com/propeller-heads/propeller-protocol-lib
```

Understanding the Substreams integration

Substreams is a new indexing technology, which uses Rust modules to compose raw blockchain data streams into higher level data streams, in out case specific to the protocol. These modules together with the protobuf definitions and manifest are then wrapped into SPKG packages (more info here) that is then run remotely on the Substreams server.

For more information, read the quick explanation of Substreams or jump into the Substreams documentation. It describes the functions that need to be implemented as well as the manifest file.

ProtoBuf files

Generally these describe the raw blockchain data that we get on the input stream and the output data that we want to produce using the Rust module.

If you are unfamiliar with ProtoBuf at all, you can start with the official documentation.

First get familiar with the raw ProtoBuf definitions provided by us:

common.proto - Common types used by all integration types
vm.proto - Types specific to the VM integration

You can also create your own intermediate ProtoBufs. These files should reside in your own substreams package, e.g. ./substreams/ethereum-template/proto/custom-messages.proto. You have to link these files in the substreams.yaml file, see the manifest docs for more information or you can look at the official substreams example integration of UniswapV2.

Note: Internally we are referring to the substreams integration as Tycho, which is why our protobuf files are under the proto/tycho directory.

Rust module

The goal of the rust module is to implement the logic that will transform the raw blockchain data into the desired output data.

This is the actual integration code that you will be writing!

The module is a Rust library that is compiled into a SPKG (.spkg) file using the Substreams CLI and then loaded by the Substreams server. It is defined by the lib.rs file (see the Ambient reference example).

Read our Substreams README.md for more information on how to write the Rust module.

How to implement the integration

Create a new directory for your integration by cloning the template, rename all the references to ethereum-template to [CHAIN]-[PROTOCOL_SYSTEM]:
```
cp -r ./substreams/ethereum-template ./substreams/[CHAIN]-[PROTOCOL_SYSTEM]
```
Implement the logic in the Rust module lib.rs. The main function to implement is the map_changes function, which is called for every block.
```
#[substreams::handlers::map]
fn map_changes(
    block: eth::v2::Block,
) -> Result<tycho::BlockContractChanges, substreams::errors::Error> {}
```
The map_changes function takes a raw block as input and returns a BlockContractChanges struct, which is derived from the BlockContractChanges protobuf message in vm.proto.
The BlockContractChanges is a list of TransactionContractChanges, which includes these main fields:
- list of ContractChange - All storage slots that have changed in the transaction for every contract tracked by any ProtocolComponent
- list of ProtocolComponent - All the protocol component changes in the transaction
- list of BalanceChange - All the contract component changes in the transaction
See the Ambient reference example for more information.

Testing

Read the Substreams testing docs for more information on how to test your integration.