Subtreams packages

This directory contains all substream packages that are used to index integrated protocols across different blockchains.

Adding a new package

To add a new package add a folder. The naming convention is [CHAIN]-[PROTOCOL_SYSTEM].

Manifest

In this new folder add a manifest file substreams.yaml. You can use the template below to get started:

specVersion: v0.1.0
package:
  name: 'substreams_[CHAIN]_[PROTOCOL_SYSTEM]'
  version: v0.1.0

protobuf:
  files:
    - vm.proto
    - common.proto
    # You can specify any internal proto files here
  importPaths:
    - ../../proto/tycho/evm/v1/
    # Any private message types only used in internal modules 
    # can remain local to the folder.
    - ./proto

binaries:
  default:
    type: wasm/rust-v1
    # this points to the workspace target directory we use a special 
    # substreams build profile to optimise wasm binaries
    file: ../../target/wasm32-unknown-unknown/substreams/substreams_[CHAIN]_[PROTOCOL_SYSTEM].wasm

modules:
  - name: map_changes
    kind: map
    inputs:
      - source: sf.ethereum.type.v2.Block
    output:
      type: proto:tycho.evm.state.v1.BlockContractChanges

Substreams packages are Rust crates so we also need a cargo.toml. The example from the official docs will serve us just well:

[package]
name = "substreams-[CHAIN]-[PROTOCOL_SYSTEM]"
version = "0.1.0"
edition = "2021"

[lib]
name = "substreams_[CHAIN]_[PROTOCOL_SYSTEM]"
crate-type = ["cdylib"]

[dependencies]
substreams = "0.5"
substreams-ethereum = "0.9"
prost = "0.11"

There are already some generated rust files in the src/pb directory. These are generated from the protobuf files in the /proto/tycho/evm/v1 directory. They specify the output protobuf messages we want to generate. The input Block is specified by the subtreams crate, specifically the sf.ethereum.type.v2.Block message.

You can define your own protobuf messages, make a new directory /substreams/[CHAIN]-[PROTOCOL]/proto for them.

Now we can generate the Rust protobuf code:

substreams protogen substreams.yaml --exclude-paths="sf/substreams,google"

The command above should put the generate rust files under /src/pb. You can start using these now in your module handlers: See the official substreams documentation on how to implement module handlers.

You can also look into already existing substreams packages to see how it is done. E.g. ethereum-ambient provides a pretty good example of how to get access to raw contract storage.

Tests

To create a block test asset for ethereum do the following:

Follow this tutorial. Make sure you set up the substreams-explorer repo in the same directory as this repo.
- Comment out image: ./ethereum.png in ethereum-explorer/substreams.yaml
- Add prost-types = "0.11.0" to ethereum-explorer/Cargo.toml
Make sure you set up your key env vars.
Run sh scripts/download-ethereum-block-to-s3 BLOCK_NUMBER

Do not commit the block files (they are quite big).