Add substreams integration template, readmes

This commit is contained in:
pistomat
2023-12-20 18:12:41 +01:00
parent 5f87b1d0af
commit 0f3a5b819c
14 changed files with 1549 additions and 53 deletions

View File

@@ -2,4 +2,4 @@
Protocol lib is a library used by Propellerheads.xyz solvers to integrate decentralized protocols. Currently, only swap/exchange protocols are supported.
Please refer to the [README.md](docs/README.md) in the [docs](docs) folder for more information.
Please refer to the [README.md](docs/README.md) for more information.

View File

@@ -22,7 +22,7 @@ While VM integration is certainly the quickest and probably most accessible one
### Indexing
For indexing purposes, it is required that you provide a [substreams](https://thegraph.com/docs/en/substreams/) package that emits a specified set of messages. If your protocol already has a [substreams](https://thegraph.com/docs/en/substreams/) package for indexing implemented, you can adjust it to emit the required messages.
_Specifications coming soon._
For indexing purposes, it is required that you provide a [substreams](https://substreams.streamingfast.io/) package that emits a specified set of messages. If your protocol already has a [substreams package](https://github.com/messari/substreams) package for indexing implemented, you can adjust it to emit the required messages.
**VM Integration** Currently the only supported integration is for EVM protocols in order to complement the Solidity protocol logic. **[Read more here.](indexing/vm-integration/README.md)**
**Custom Entity Integration** Coming soon, this integration will complement the upcoming native Rust protocol logic.

View File

@@ -11,4 +11,4 @@
## Indexing
* [Substreams Integration](indexing/substreams-integration/README.md)
* [Tutorial: UniswapV2](indexing/substreams-integration/tutorial-uniswapv2.md)
* [Tutorial: Ambient](indexing/substreams-integration/tutorial-ambient.md)

View File

@@ -1,3 +1,70 @@
# Substreams Integration
Coming Soon
## Example
We have integrated the **Ambient** protocol as a reference, see `/substreams/ethereum-ambient` for more information.
## Step by step
1. Install [Rust](https://www.rust-lang.org/tools/install), you can do so with the following command:
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
1. Install [Substreams CLI](https://substreams.streamingfast.io/getting-started/installing-the-cli), you can either use brew:
```bash
brew install streamingfast/tap/substreams
```
use precompiled binaries
```bash
# Use correct binary for your platform
LINK=$(curl -s https://api.github.com/repos/streamingfast/substreams/releases/latest | awk '/download.url.*linux/ {print $2}' | sed 's/"//g')
curl -L $LINK | tar zxf -
```
or compile from source:
```bash
git clone https://github.com/streamingfast/substreams
cd substreams
go install -v ./cmd/substreams
```
1. Start by making a local copy of the Propeller Protocol Lib repository:
```bash
git clone https://github.com/propeller-heads/propeller-protocol-lib
```
## Understanding the Substreams integration
Substreams is a new indexing technology, which uses Rust modules to compose raw blockchain data streams into higher level data streams, in out case specific to the protocol. These modules together with the protobuf definitions and manifest are then wrapped into SPKG packages (more info [here](https://substreams.streamingfast.io/quick-access/glossary#spkg-.spkg)) that is then run remotely on the Substreams server.
For more information, read the [quick explanation of Substreams](https://thegraph.com/docs/en/substreams/) or jump into the [Substreams documentation](https://substreams.streamingfast.io/). It describes the functions that need to be implemented as well as the manifest file.
### ProtoBuf files
Generally these describe the raw blockchain data that we get on the input stream and the output data that we want to produce using the Rust module.
If you are unfamiliar with ProtoBuf at all, you can start with the [official documentation](https://protobuf.dev/overview/).
First get familiar with the raw ProtoBuf definitions provided by us:
- [common.proto](../../../proto/tycho/evm/v1/common.proto) - Common types used by all integration types
- [vm.proto](../../../proto/tycho/evm/v1/vm.proto) - Types specific to the VM integration
You can also create your own intermediate ProtoBufs. These files should reside in your own substreams package, e.g. `./substreams/ethereum-template/proto/custom-messages.proto`. You have to link these files in the `substreams.yaml` file, see the [manifest docs](https://substreams.streamingfast.io/developers-guide/creating-your-manifest) for more information or you can look at the official substreams example integration of [UniswapV2](https://github.com/messari/substreams/blob/master/uniswap-v2/substreams.yaml#L20-L22).
*Note: Internally we are referring to the substreams integration as `Tycho`, which is why our protobuf files are under the `proto/tycho` directory.*
### Rust module
The goal of the rust module is to implement the logic that will transform the raw blockchain data into the desired output data.
*This is the actual integration code that you will be writing!*
The module is a Rust library that is compiled into a SPKG (`.spkg`) file using the Substreams CLI and then loaded by the Substreams server. It is defined by the `lib.rs` file (see the [Ambient reference example](../../../substreams/ethereum-ambient/src/lib.rs)).
Read our [Substreams README.md](../../../substreams/README.md) for more information on how to write the Rust module.
### Testing
Read the [Substreams testing docs](../../../substreams/README.md#testing-your-implementation) for more information on how to test your integration.

View File

@@ -24,12 +24,12 @@ Following exchanges have been integrated using VM approach:
foundryup
```
2. Start by making a local copy of the Propeller Protocol Lib repository:
1. Start by making a local copy of the Propeller Protocol Lib repository:
```bash
git clone https://github.com/propeller-heads/propeller-protocol-lib
```
3. Install forge dependencies:
1. Install forge dependencies:
```bash
cd ./propeller-protocol-lib/evm/
forge install

View File

@@ -5,8 +5,10 @@ blockchains.
## Adding a new package
To add a new package add folder. The naming convention is `[CHAIN]-[PROTOCOL_SYSTEM]`. In this new folder add a manifest
file `substreams.yaml`. You can use the template below to get started:
To add a new package add folder. The naming convention is `[CHAIN]-[PROTOCOL_SYSTEM]`.
### Manifest
In this new folder add a manifest file `substreams.yaml`. You can use the template below to get started:
```yaml
specVersion: v0.1.0
@@ -36,13 +38,12 @@ binaries:
file: ../../target/wasm32-unknown-unknown/substreams/substreams_[CHAIN]_[PROTOCOL_SYSTEM].wasm
modules:
# sample module provides access to blocks.
- name: map_block
- name: map_changes
kind: map
inputs:
- source: sf.ethereum.type.v2.Block
output:
type: proto:acme.block_meta.v1.BlockMeta
type: proto:tycho.evm.state.v1.BlockContractChanges
```
Substreams packages are Rust crates so we also need a `cargo.toml`.
@@ -65,6 +66,9 @@ prost = "0.11"
```
There are already some generated rust files in the `src/pb` directory. These are generated from the protobuf files in the
Now we can generate the Rust protobuf code:
```

View File

@@ -180,7 +180,24 @@ version = "17.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e4966fba78396ff92db3b817ee71143eccd98acf0f876b8d600e585a670c5d1b"
dependencies = [
"ethereum-types",
"ethereum-types 0.13.1",
"hex",
"once_cell",
"regex",
"serde",
"serde_json",
"sha3",
"thiserror",
"uint",
]
[[package]]
name = "ethabi"
version = "18.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7413c5f74cc903ea37386a8965a936cbeb334bd270862fdece542c1b2dcbc898"
dependencies = [
"ethereum-types 0.14.1",
"hex",
"once_cell",
"regex",
@@ -198,9 +215,22 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11da94e443c60508eb62cf256243a64da87304c2802ac2528847f79d750007ef"
dependencies = [
"crunchy",
"fixed-hash",
"fixed-hash 0.7.0",
"impl-rlp",
"impl-serde",
"impl-serde 0.3.2",
"tiny-keccak",
]
[[package]]
name = "ethbloom"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c22d4b5885b6aa2fe5e8b9329fb8d232bf739e434e6b87347c63bdd00c120f60"
dependencies = [
"crunchy",
"fixed-hash 0.8.0",
"impl-rlp",
"impl-serde 0.4.0",
"tiny-keccak",
]
@@ -210,11 +240,25 @@ version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2827b94c556145446fcce834ca86b7abf0c39a805883fe20e72c5bfdb5a0dc6"
dependencies = [
"ethbloom",
"fixed-hash",
"ethbloom 0.12.1",
"fixed-hash 0.7.0",
"impl-rlp",
"impl-serde",
"primitive-types",
"impl-serde 0.3.2",
"primitive-types 0.11.1",
"uint",
]
[[package]]
name = "ethereum-types"
version = "0.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02d215cbf040552efcbe99a38372fe80ab9d00268e20012b79fcd0f073edd8ee"
dependencies = [
"ethbloom 0.13.0",
"fixed-hash 0.8.0",
"impl-rlp",
"impl-serde 0.4.0",
"primitive-types 0.12.2",
"uint",
]
@@ -236,6 +280,18 @@ dependencies = [
"static_assertions",
]
[[package]]
name = "fixed-hash"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "835c052cb0c08c1acf6ffd71c022172e18723949c8282f2b9f27efbc51e64534"
dependencies = [
"byteorder",
"rand",
"rustc-hex",
"static_assertions",
]
[[package]]
name = "fixedbitset"
version = "0.4.2"
@@ -335,6 +391,15 @@ dependencies = [
"serde",
]
[[package]]
name = "impl-serde"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc88fc67028ae3db0c853baa36269d398d5f45b6982f95549ff5def78c935cd"
dependencies = [
"serde",
]
[[package]]
name = "impl-trait-for-tuples"
version = "0.2.2"
@@ -519,10 +584,23 @@ version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e28720988bff275df1f51b171e1b2a18c30d194c4d2b61defdacecd625a5d94a"
dependencies = [
"fixed-hash",
"fixed-hash 0.7.0",
"impl-codec",
"impl-rlp",
"impl-serde",
"impl-serde 0.3.2",
"uint",
]
[[package]]
name = "primitive-types"
version = "0.12.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b34d9fd68ae0b74a41b21c03c2f62847aa0ffea044eee893b4c140b37e244e2"
dependencies = [
"fixed-hash 0.8.0",
"impl-codec",
"impl-rlp",
"impl-serde 0.4.0",
"uint",
]
@@ -805,7 +883,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "97a176f39a6e09553c17a287edacd1854e5686fd20ffea3c9655dfc44d94b35e"
dependencies = [
"anyhow",
"ethabi",
"ethabi 17.2.0",
"heck",
"hex",
"prettyplease",
@@ -819,6 +897,10 @@ dependencies = [
name = "substreams-ethereum-ambient"
version = "0.3.0"
dependencies = [
"anyhow",
"bytes",
"ethabi 18.0.0",
"hex",
"hex-literal 0.4.1",
"prost",
"substreams",
@@ -832,7 +914,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db4700cfe408b75634a3c6b3a0caf7bddba4879601d2085c811485ea54cbde2d"
dependencies = [
"bigdecimal",
"ethabi",
"ethabi 17.2.0",
"getrandom",
"num-bigint",
"prost",
@@ -847,7 +929,7 @@ version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40d6d278d926fe3f0775d996ee2b5e1dc822c1b4bf4f7bf07c7fbb5bce6c79a9"
dependencies = [
"ethabi",
"ethabi 17.2.0",
"heck",
"hex",
"num-bigint",

View File

@@ -3,9 +3,9 @@ use std::collections::{hash_map::Entry, HashMap};
use anyhow::{anyhow, bail};
use ethabi::{decode, ParamType};
use hex_literal::hex;
use substreams_ethereum::pb::eth::{self};
use substreams_ethereum::pb::eth;
use pb::tycho::evm::v1::{self as tycho, ChangeType};
use pb::tycho::evm::v1::{self as tycho};
mod pb;
@@ -52,24 +52,6 @@ impl From<InterimContractChange> for tycho::ContractChange {
}
/// Extracts all contract changes relevant to vm simulations
///
/// This implementation has currently two major limitations:
/// 1. It is hardwired to only care about changes to the ambient main contract, this is ok for this
/// particular use case but for a more general purpose implementation this is not ideal
/// 2. Changes are processed separately, this means that if there are any side effects between each
/// other (e.g. if account is deleted and then created again in ethereum all the storage is set
/// to 0. So there is a side effect between account creation and contract storage.) these might
/// not be properly accounted for. Most of the time this should not be a major issue but may lead
/// to wrong results so consume this implementation with care. See example below for a concrete
/// case where this is problematic.
///
/// ## A very contrived example:
/// 1. Some existing contract receives a transaction that changes it state, the state is updated
/// 2. Next, this contract has self destruct called on itself
/// 3. The contract is created again using CREATE2 at the same address
/// 4. The contract receives a transaction that changes it state
/// 5. We would emit this as as contract creation with slots set from 1 and from 4, although we
/// should only emit the slots changed from 4.
#[substreams::handlers::map]
fn map_changes(
block: eth::v2::Block,
@@ -179,7 +161,7 @@ fn map_changes(
let static_attribute = tycho::Attribute {
name: String::from("pool_index"),
value: pool_index.to_be_bytes().to_vec(),
change: ChangeType::Creation.into(),
change: tycho::ChangeType::Creation.into(),
};
let mut tokens: Vec<Vec<u8>> = vec![base.clone(), quote.clone()];
@@ -195,7 +177,7 @@ fn map_changes(
tokens,
contracts: vec![hex::encode(AMBIENT_CONTRACT)],
static_att: vec![static_attribute],
change: ChangeType::Creation.into(),
change: tycho::ChangeType::Creation.into(),
};
tx_change
.component_changes
@@ -257,9 +239,9 @@ fn map_changes(
code: Vec::new(),
slots,
change: if created_accounts.contains_key(&storage_change.address) {
ChangeType::Creation
tycho::ChangeType::Creation
} else {
ChangeType::Update
tycho::ChangeType::Update
},
});
}
@@ -298,9 +280,9 @@ fn map_changes(
code: Vec::new(),
slots: HashMap::new(),
change: if created_accounts.contains_key(&balance_change.address) {
ChangeType::Creation
tycho::ChangeType::Creation
} else {
ChangeType::Update
tycho::ChangeType::Update
},
});
}
@@ -337,9 +319,9 @@ fn map_changes(
code: code_change.new_code.clone(),
slots: HashMap::new(),
change: if created_accounts.contains_key(&code_change.address) {
ChangeType::Creation
tycho::ChangeType::Creation
} else {
ChangeType::Update
tycho::ChangeType::Update
},
});
}

1135
substreams/ethereum-template/Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,13 @@
[package]
name = "substreams-ethereum-template"
version = "0.1.0"
edition = "2021"
[lib]
name = "substreams_ethereum_template"
crate-type = ["cdylib"]
[dependencies]
substreams = "0.5"
substreams-ethereum = "0.9"
prost = "0.11"

View File

@@ -0,0 +1,13 @@
use substreams_ethereum::pb::eth;
use pb::tycho::evm::v1::{self as tycho};
mod pb;
#[substreams::handlers::map]
fn map_changes(
block: eth::v2::Block,
) -> Result<tycho::BlockContractChanges, substreams::errors::Error> {
todo!("Not implemented")
}

View File

@@ -0,0 +1,10 @@
// @generated
pub mod tycho {
pub mod evm {
// @@protoc_insertion_point(attribute:tycho.evm.v1)
pub mod v1 {
include!("tycho.evm.v1.rs");
// @@protoc_insertion_point(tycho.evm.v1)
}
}
}

View File

@@ -0,0 +1,166 @@
// @generated
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct Block {
#[prost(bytes="vec", tag="1")]
pub hash: ::prost::alloc::vec::Vec<u8>,
#[prost(bytes="vec", tag="2")]
pub parent_hash: ::prost::alloc::vec::Vec<u8>,
#[prost(uint64, tag="3")]
pub number: u64,
#[prost(uint64, tag="4")]
pub ts: u64,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct Transaction {
#[prost(bytes="vec", tag="1")]
pub hash: ::prost::alloc::vec::Vec<u8>,
#[prost(bytes="vec", tag="2")]
pub from: ::prost::alloc::vec::Vec<u8>,
#[prost(bytes="vec", tag="3")]
pub to: ::prost::alloc::vec::Vec<u8>,
#[prost(uint64, tag="4")]
pub index: u64,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct Attribute {
#[prost(string, tag="1")]
pub name: ::prost::alloc::string::String,
#[prost(bytes="vec", tag="2")]
pub value: ::prost::alloc::vec::Vec<u8>,
#[prost(enumeration="ChangeType", tag="3")]
pub change: i32,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ProtocolComponent {
#[prost(string, tag="1")]
pub id: ::prost::alloc::string::String,
#[prost(bytes="vec", repeated, tag="2")]
pub tokens: ::prost::alloc::vec::Vec<::prost::alloc::vec::Vec<u8>>,
#[prost(string, repeated, tag="3")]
pub contracts: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
#[prost(message, repeated, tag="4")]
pub static_att: ::prost::alloc::vec::Vec<Attribute>,
#[prost(enumeration="ChangeType", tag="5")]
pub change: i32,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct BalanceChange {
#[prost(bytes="vec", tag="1")]
pub token: ::prost::alloc::vec::Vec<u8>,
#[prost(bytes="vec", tag="2")]
pub balance: ::prost::alloc::vec::Vec<u8>,
#[prost(bytes="vec", tag="3")]
pub component_id: ::prost::alloc::vec::Vec<u8>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)]
#[repr(i32)]
pub enum ChangeType {
Unspecified = 0,
Update = 1,
Creation = 2,
Deletion = 3,
}
impl ChangeType {
/// String value of the enum field names used in the ProtoBuf definition.
///
/// The values are not transformed in any way and thus are considered stable
/// (if the ProtoBuf definition does not change) and safe for programmatic use.
pub fn as_str_name(&self) -> &'static str {
match self {
ChangeType::Unspecified => "CHANGE_TYPE_UNSPECIFIED",
ChangeType::Update => "CHANGE_TYPE_UPDATE",
ChangeType::Creation => "CHANGE_TYPE_CREATION",
ChangeType::Deletion => "CHANGE_TYPE_DELETION",
}
}
/// Creates an enum from field names used in the ProtoBuf definition.
pub fn from_str_name(value: &str) -> ::core::option::Option<Self> {
match value {
"CHANGE_TYPE_UNSPECIFIED" => Some(Self::Unspecified),
"CHANGE_TYPE_UPDATE" => Some(Self::Update),
"CHANGE_TYPE_CREATION" => Some(Self::Creation),
"CHANGE_TYPE_DELETION" => Some(Self::Deletion),
_ => None,
}
}
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct EntityChanges {
#[prost(string, tag="1")]
pub component_id: ::prost::alloc::string::String,
#[prost(message, repeated, tag="2")]
pub attributes: ::prost::alloc::vec::Vec<Attribute>,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct TransactionEntityChanges {
#[prost(message, optional, tag="1")]
pub tx: ::core::option::Option<Transaction>,
#[prost(message, repeated, tag="2")]
pub entity_changes: ::prost::alloc::vec::Vec<EntityChanges>,
#[prost(message, repeated, tag="3")]
pub component_changes: ::prost::alloc::vec::Vec<ProtocolComponent>,
#[prost(message, repeated, tag="4")]
pub balance_changes: ::prost::alloc::vec::Vec<BalanceChange>,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct BlockEntityChanges {
#[prost(message, optional, tag="1")]
pub block: ::core::option::Option<Block>,
#[prost(message, repeated, tag="2")]
pub changes: ::prost::alloc::vec::Vec<TransactionEntityChanges>,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ContractSlot {
#[prost(bytes="vec", tag="2")]
pub slot: ::prost::alloc::vec::Vec<u8>,
#[prost(bytes="vec", tag="3")]
pub value: ::prost::alloc::vec::Vec<u8>,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ContractChange {
#[prost(bytes="vec", tag="1")]
pub address: ::prost::alloc::vec::Vec<u8>,
/// empty bytes indicates no change
#[prost(bytes="vec", tag="2")]
pub balance: ::prost::alloc::vec::Vec<u8>,
/// empty bytes indicates no change
#[prost(bytes="vec", tag="3")]
pub code: ::prost::alloc::vec::Vec<u8>,
/// empty sequence indicates no change
#[prost(message, repeated, tag="4")]
pub slots: ::prost::alloc::vec::Vec<ContractSlot>,
/// Whether this is an update, creation or deletion
#[prost(enumeration="ChangeType", tag="5")]
pub change: i32,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct TransactionContractChanges {
#[prost(message, optional, tag="1")]
pub tx: ::core::option::Option<Transaction>,
#[prost(message, repeated, tag="2")]
pub contract_changes: ::prost::alloc::vec::Vec<ContractChange>,
#[prost(message, repeated, tag="3")]
pub component_changes: ::prost::alloc::vec::Vec<ProtocolComponent>,
#[prost(message, repeated, tag="4")]
pub balance_changes: ::prost::alloc::vec::Vec<BalanceChange>,
}
#[allow(clippy::derive_partial_eq_without_eq)]
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct BlockContractChanges {
#[prost(message, optional, tag="1")]
pub block: ::core::option::Option<Block>,
#[prost(message, repeated, tag="2")]
pub changes: ::prost::alloc::vec::Vec<TransactionContractChanges>,
}
// @@protoc_insertion_point(module)

View File

@@ -0,0 +1,24 @@
specVersion: v0.1.0
package:
name: "substreams_ethereum_template"
version: v0.1.0
protobuf:
files:
- vm.proto
- common.proto
importPaths:
- ../../proto/tycho/evm/v1/
binaries:
default:
type: wasm/rust-v1
file: ../../target/wasm32-unknown-unknown/substreams/substreams_ethereum_template.wasm
modules:
- name: map_changes
kind: map
inputs:
- source: sf.ethereum.type.v2.Block
output:
type: proto:tycho.evm.state.v1.BlockContractChanges