Liquidity.Party/tycho-protocol-sdk
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(substreams): miscellaneous docs improvements and update

Browse Source
This commit is contained in:
Florian Pellissier
2024-08-16 15:22:27 +02:00
parent bc2cd6bab2
commit 37f1fbfe04
6 changed files with 137 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

82
docs/indexing/substreams-integration.md
View File

@@ -18,29 +18,15 @@ So basically when processing a block we need to emit the block itself, all trans
**The data model that encodes changes, transaction and blocks in messages, can be found** [**here**](https://github.com/propeller-heads/propeller-protocol-lib/tree/main/proto/tycho/evm/v1)**.** 
#### Common Models
#### Models
The following models are shared for both vm and native integrations.
The models below are used for communication between Substreams and our indexer, as well as between Substreams modules.
{% @github-files/github-code-block url="https://github.com/propeller-heads/propeller-protocol-lib/blob/main/proto/tycho/evm/v1/common.proto" %}
Our indexer expects to receive a `BlockChanges` output from your Substreams package.
#### VM Specific Models
{% @github-files/github-code-block url="https://github.com/propeller-heads/propeller-protocol-lib/blob/main/proto/tycho/evm/v1/" %}
The models shown below are specific to vm integrations:
{% @github-files/github-code-block url="https://github.com/propeller-heads/propeller-protocol-lib/blob/main/proto/tycho/evm/v1/vm.proto" %}
Please be aware that changes need to be aggregated on the transaction level, it is considered an error to emit `BlockContractChanges` with duplicated transactions present in the `changes` attributes.
All attributes are expected to be set in the final message unless the docs (in the comments) indicate otherwise.
#### Native Integration Models
The models below are very similar to the vm integration models but have a few modifications necessary to support native integrations.
{% @github-files/github-code-block url="https://github.com/propeller-heads/propeller-protocol-lib/blob/main/proto/tycho/evm/v1/entity.proto" %}
Once again changes must be aggregated on a transaction level, emitting these models with duplicated transaction as the final output would be considered an error.
Please be aware that changes need to be aggregated on the transaction level, it is considered an error to emit `BlockChanges` with duplicated transactions present in the `changes` attributes.
#### Integer Byte encoding
@@ -50,17 +36,15 @@ Many of the types above are variable length bytes. This allows for flexibility a
**Strings**: If you need to store strings, please use utf-8 encoding to store them as bytes.
**Attributes:** the value encoding for attributes in the native implementation messages is variable. It depends on the use case. Since the attributes are highly dynamic they are only used by the corresponding logic components, so the encoding can be tailored to the logic implementation: E.g. since Rust uses little endian one may choose to use little endian encoding for integers if the native logic module is written in Rust.
**Attributes:** the value encoding for attributes is variable. It depends on the use case. Since the attributes are highly dynamic they are only used by the corresponding logic components, so the encoding can be tailored to the logic implementation: E.g. since Rust uses little endian one may choose to use little endian encoding for integers if the native logic module is written in Rust.
### Changes of interest
PropellerHeads integration should at least communicate the following changes:
* Any changes to the protocol state, for VM integrations that usually means contract storage changes of all contracts whose state may be accessed during a swap operation.
* Any newly added protocol component such as a pool, pair, market, etc. Basically anything that signifies that a new operation can be executed now using the protocol.
* ERC20 Balances, whenever the balances of one contracts involved with the protocol change, this change should be communicated in terms of absolute balances.
- Any changes to the protocol state, for VM integrations that usually means contract storage changes of all contracts whose state may be accessed during a swap operation.
- Any newly added protocol component such as a pool, pair, market, etc. Basically anything that signifies that a new operation can be executed now using the protocol.
- ERC20 Balances, whenever the balances of one contracts involved with the protocol change, this change should be communicated in terms of absolute balances.
In the next section we will show a few common techniques that can be leveraged to quickly implement an integration.
@@ -70,9 +54,9 @@ Before starting, it is important to be aware of the protocol we are aiming to in
It is especially important to know:
* Which contracts are involved in the protocol and what functions do they serve. How do they affect the behaviour of the component being integrated?
* What conditions (e.g. oracle update) or what kind of method calls can lead to a relevant state change on the protocol, which ultimately changes the protocols behaviour if observed externally.
* Are there components added or removed, and how are they added. Most protocols use either a factory contract, which can be used to deploy new components, or they use a method call that provisiona a new component within the overall system.
- Which contracts are involved in the protocol and what functions do they serve. How do they affect the behaviour of the component being integrated?
- What conditions (e.g. oracle update) or what kind of method calls can lead to a relevant state change on the protocol, which ultimately changes the protocols behaviour if observed externally.
- Are there components added or removed, and how are they added. Most protocols use either a factory contract, which can be used to deploy new components, or they use a method call that provisiona a new component within the overall system.
Once the workings of the protocol are clear the implementation can start.
@@ -94,44 +78,19 @@ You are ready to start coding. Please refer to the substreams documentation for
Usually the first step consists in detecting the creation of new components and store their contract addresses in a store, so they can be properly tracked further downstream.
Later we'll have to emit balance and state changes based on the set of currently tracked components.
Later we'll have to emit balance and state changes based on the set of currently tracked components.
{% hint style="info" %}
Note that emitting state changes of components that have not been previously announced is considered an error.
Note that emitting state changes of components that have not been previously announced is considered an error.
{% endhint %}
Newly created components are detected by mapping over the `sf.ethereum.type.v2.Block model`. 
The output message should usually contain as much information about the component available at that time as well as the transaction that created the protocol component.
We have found that using the final model prefilled with only component changes is usually good enough since it holds all the information that will be necessary at the end. 
We have found that using the final model (`BlockChanges`) prefilled with only component changes is usually good enough since it holds all the information that will be necessary at the end. 
For VM Integrations the final model is `BlockContractChanges`:
```protobuf
// A set of changes aggregated by transaction.
message TransactionContractChanges {
// The transaction instance that results in the changes.
Transaction tx = 1;
// Contains the changes induced by the above transaction, aggregated on a per-contract basis.
// Must include changes to every contract that is tracked by all ProtocolComponents.
repeated ContractChange contract_changes = 2;
// An array of any component changes.
repeated ProtocolComponent component_changes = 3;
// An array of balance changes to components.
repeated BalanceChange balance_changes = 4;
}
// A set of transaction changes within a single block.
message BlockContractChanges {
// The block for which these changes are collectively computed.
Block block = 1;
// The set of transaction changes observed in the specified block.
repeated TransactionContractChanges changes = 2;
}
```
Note that a single transaction may emit multiple newly created components. In this case it is expected that the `TransactionContractChanges.component_changes`, contains multiple `ProtocolComponents`.
Note that a single transaction may emit multiple newly created components. In this case it is expected that the `TransactionChanges.component_changes`, contains multiple `ProtocolComponents`.
Once emitted, the protocol components should be stored in a Store, since we will later have to use this store to decide whether a contract is interesting to us or not.
@@ -143,11 +102,12 @@ This means the relative values have to be aggregated by component, to arrive at
Since this is challenging the following approach is recommended:
* Use a handler to process a block and emit the `BalanceDeltas` struct. Make sure to sort the balance deltas by `component_id, token_address`
* Aggregate the BalanceDelta messages using a `BigIntAddStore`.
* In a final handler, use as inputs: A `DeltaStore` input from step 2 and the `BalanceDeltas` from step 1. You can now zip the deltas from the store with the balance deltas from step 1. The store deltas contains the aggregated (absolute) balance at each version and the balance deltas contain the corresponding transaction.
- Use a handler to process a block and emit the `BlockBalanceDeltas` struct. Make sure to sort the balance deltas by `component_id, token_address`
- Aggregate the BalanceDelta messages using a `BigIntAddStore`.
- In a final handler, use as inputs: A `DeltaStore` input from step 2 and the `BlockBalanceDeltas` from step 1. You can now zip the deltas from the store with the balance deltas from step 1. The store deltas contains the aggregated (absolute) balance at each version and the balance deltas contain the corresponding transaction.
Our Substreams SDK provide the `extract_balance_deltas_from_tx` function that extracts all relevant `BalanceDelta` from ERC20 `Transfer` events in a given transaction (see Curve implementation).
#### Tracking State Changes
To track contract changes, you can simply use the `extract_contract_changes` function (see balancer implementation). This function will extract all relevant contract storage changes given the full block model and a store that flags contract addresses as relevant.

22
docs/indexing/vm-integration/README.md
View File

@@ -4,7 +4,7 @@ Our indexing integrations use the Substreams library to transform raw blockchain
## Example
We have integrated the **Ambient** protocol as a reference, see `/substreams/ethereum-ambient` for more information.
We have integrated the **Balancer** protocol as a reference, see `/substreams/ethereum-balancer` for more information.
## Step by step
@@ -51,7 +51,6 @@ If you are unfamiliar with ProtoBuf at all, you can start with the [official doc
First get familiar with the raw ProtoBuf definitions provided by us:
- [common.proto](../../../proto/tycho/evm/v1/common.proto) - Common types used by all integration types
- [vm.proto](../../../proto/tycho/evm/v1/vm.proto) - Types specific to the VM integration
You can also create your own intermediate ProtoBufs. These files should reside in your own substreams package, e.g. `./substreams/ethereum-template/proto/custom-messages.proto`. You have to link these files in the `substreams.yaml` file, see the [manifest docs](https://substreams.streamingfast.io/developers-guide/creating-your-manifest) for more information or you can look at the official substreams example integration of [UniswapV2](https://github.com/messari/substreams/blob/master/uniswap-v2/substreams.yaml#L20-L22).
@@ -63,7 +62,7 @@ The goal of the rust module is to implement the logic that will transform the ra
*This is the actual integration code that you will be writing!*
The module is a Rust library that is compiled into a SPKG (`.spkg`) file using the Substreams CLI and then loaded by the Substreams server. It is defined by the `lib.rs` file (see the [Ambient reference example](../../../substreams/ethereum-ambient/src/lib.rs)).
The module is a Rust library that is compiled into a SPKG (`.spkg`) file using the Substreams CLI and then loaded by the Substreams server. It is defined by the `lib.rs` file (see the [Balancer reference example](../../../substreams/ethereum-balancer/src/lib.rs)).
Read our [Substreams README.md](../../../substreams/README.md) for more information on how to write the Rust module.
@@ -74,26 +73,27 @@ Read our [Substreams README.md](../../../substreams/README.md) for more informat
```bash
cp -r ./substreams/ethereum-template ./substreams/[CHAIN]-[PROTOCOL_SYSTEM]
```
1. Implement the logic in the Rust module `lib.rs`. The main function to implement is the `map_changes` function, which is called for every block.
1. Implement the logic in the Rust module `lib.rs`. The main function to implement is the `map_protocol_changes` function, which is called for every block.
```rust
#[substreams::handlers::map]
fn map_changes(
fn map_protocol_changes(
block: eth::v2::Block,
) -> Result<tycho::BlockContractChanges, substreams::errors::Error> {}
) -> Result<tycho::BlockChanges, substreams::errors::Error> {}
```
The `map_changes` function takes a raw block as input and returns a `BlockContractChanges` struct, which is derived from the `BlockContractChanges` protobuf message in [vm.proto](../../../proto/tycho/evm/v1/vm.proto).
The `map_protocol_changes` function takes a raw block as input and returns a `BlockChanges` struct, which is derived from the `BlockChanges` protobuf message in [vm.proto](../../../proto/tycho/evm/v1/vm.proto).
1. The `BlockContractChanges` is a list of `TransactionContractChanges`, which includes these main fields:
1. The `BlockChanges` is a list of `TransactionChanges`, which includes these main fields:
- list of `ContractChange` - All storage slots that have changed in the transaction for every contract tracked by any ProtocolComponent
- list of `EntityChanges` - All the attribute changes in the transaction
- list of `ProtocolComponent` - All the protocol component changes in the transaction
- list of `BalanceChange` - All the contract component changes in the transaction
- list of `BalanceChange` - All the token balances changes in the transaction
See the [Ambient reference example](../../../substreams/ethereum-ambient/src/lib.rs) for more information.
See the [Balancer reference example](../../../substreams/ethereum-balancer/src/lib.rs) for more information.
1. If you are more advanced with Substreams, you can define more steps than a single "map" step, including defining your own protobuf files. Add these protobuf files in your `pb` folder and update the manifest accordingly. This allows for better parallelization of the indexing process. See the official documentation of [modules](https://substreams.streamingfast.io/concepts-and-fundamentals/modules#modules-basics-overview).
### Testing
Read the [Substreams testing docs](../../../substreams/README.md#testing-your-implementation) for more information on how to test your integration.
Read the [Substreams testing docs](../../../substreams/README.md#test-your-implementation) for more information on how to test your integration.