Merge branch 'docs' into zz/update-docs-summary

docs/README.md

@@ -15,7 +15,7 @@ To propose an integration, create a pull request in this repository with the abo
 
 PropellerHeads currently exposes two integration modes to specify the protocols' underlying logic:
 
-* **VM Integration:** This integration type requires implementing an adapter interface in any language that compiles to the respective vm byte code. This SDK provides the interface only in Solidity. **[Read more here.](logic/vm-integration/README.md)**
+* **VM Integration:** This integration type requires implementing an adapter interface in any language that compiles to the respective vm byte code. This SDK provides the interface only in Solidity. [**Read more here.**](logic/vm-integration/)
 * **Native Rust Integration:** Coming soon, this integration type requires implementing a Rust trait that describes the protocol logic.
 
 While VM integration is certainly the quickest and probably most accessible one for protocol developers, native implementations are much faster and allow us to consider the protocol for more time-sensitive use cases - e.g. quoting.
@@ -24,5 +24,4 @@ While VM integration is certainly the quickest and probably most accessible one
 
 For indexing purposes, it is required that you provide a [substreams](https://substreams.streamingfast.io/) package that emits a specified set of messages. If your protocol already has a [substreams package](https://github.com/messari/substreams) for indexing implemented, you can adjust it to emit the required messages.
 
-**VM Integration** Currently the only supported integration is for EVM protocols in order to complement the Solidity protocol logic. **[Read more here.](indexing/vm-integration/README.md)**
+**VM Integration** Currently the only supported integration is for EVM protocols in order to complement the Solidity protocol logic. [**Read more here.**](https://github.com/propeller-heads/propeller-venue-lib/blob/main/docs/indexing/vm-integration/README.md) **Custom Entity Integration** Coming soon, this integration will complement the upcoming native Rust protocol logic.
-**Custom Entity Integration** Coming soon, this integration will complement the upcoming native Rust protocol logic.

docs/SUMMARY.md

@@ -1,16 +1,17 @@
 # Table of contents
 
-* [Propeller Protocol Lib](README.md)
+- [Propeller Protocol Lib](README.md)
 
 ## Logic
 
-* [VM Integration](logic/vm-integration/README.md)
+- [VM Integration](logic/vm-integration/README.md)
-  * [Ethereum: Solidity](logic/vm-integration/ethereum-solidity.md)
+  - [Ethereum: Solidity](logic/vm-integration/ethereum-solidity.md)
-* [Native Integration](logic/native-integration.md)
+- [Native Integration](logic/native-integration.md)
 
 ## Indexing
 
-* [Substreams Integration](indexing/substreams-integration.md)
+- [Overview](indexing/overview.md)
-  * [VM Integration](indexing/vm-integration/README.md)
+- [Getting Started](indexing/getting-started.md)
-  * [Native Integration](indexing/native-integration.md)
+- [VM Integration](indexing/vm-integration/README.md)
-  * [Reserved Attributes](indexing/reserved-attributes.md)
+- [Native Integration](indexing/native-integration.md)
+- [Reserved Attributes](indexing/reserved-attributes.md)

docs/indexing/getting-started.md

@@ -0,0 +1,204 @@
+# Getting Started
+
+## How to Integrate
+
+Before starting, it is important to have a good understanding of the protocol we are aiming to integrate.
+
+It is essential to understand:
+
+- Which contracts are involved in the protocol and what functions do they serve. How do they affect the behaviour of the component being integrated?
+- What conditions (e.g. oracle update) or what kind of method calls can lead to a relevant state change on the protocol, which ultimately changes the protocols behaviour if observed externally.
+- Are there components added or removed, and how are they added. Most protocols use either a factory contract, which can be used to deploy new components, or they use a method call that provisiona a new component within the overall system.
+
+Once the workings of the protocol are clear the implementation can start.
+
+## Setup
+
+PropellerHeads indexing integrations are provided as [substreams](https://substreams.streamingfast.io/) skpg files. If you do not know substreams yet, make sure to go check them out and set up their [cli](https://substreams.streamingfast.io/documentation/consume/installing-the-cli) before continuing.
+
+Please start a new package for your integration, by copying the `ethereum-template` to a new name. The convention is: `[CHAIN]-[PROTOCOL_SYSTEM]` please make sure to also adjust: `cargo.toml` as well as `substreams.yaml` accordingly.
+
+It should be possible now to generate the necessary protobuf code:
+
+```bash
+substreams protogen substreams.yaml --exclude-paths="sf/substreams,google"
+```
+
+Next please register the new package with the workspace by registering it as a workspace member. This is simply done by adding the package's name to the members list under `substreams/Cargo.toml`.
+
+You are ready to start coding. The template you just copied already contains the `tycho-substreams` package as a dependency. This package contains all the necessary output types that tycho expects, as well as some interim helper types and functions for common tasks.
+
+Before continuing though, it is important to understand the concept of map modules and store modules. You can read about those [here](https://substreams.streamingfast.io/documentation/develop/manifest-modules/types).
+
+## Overview
+
+In the following section, we outline the typical procedure for structuring an integration for the Virtual Machine (VM) implementation type:
+
+Commonly, protocols employ factory contracts to deploy a contract for each swap component (also known as a pool or pair). We will explain how to efficiently index this system of contracts. It's important to note, however, that the techniques described below are broadly applicable, even if a protocol deviates from this specific pattern it should be possible to index it and emit the required messages.
+
+{% hint style="info" %}
+The following examples and code snippets have been taken from the ethereum-balancer substream. The complete code is available [here](https://github.com/propeller-heads/propeller-protocol-lib/tree/main/substreams/ethereum-balancer).
+{% endhint %}
+
+Usually an integration consists of the following modules:
+
+- `map_components(block)`
+  - This map module extracts any newly created components by inspecting the block model (e.g factory contract logs). The recommended output model for this module is `BlockTransactionProtocolComponents.`
+- `store_components(components, components_store)`
+  - This store module takes the detected components and stores any necessary information about the component for downstream modules. For vm integrations the address is most likely enough.
+- `map_relative_balances(block, components_store)`:
+  - This map module is necessary for protocols that do not emit absolute balances changes (of ERC20 tokens and/or the native token). Since no absolute balances values are available the block model most likely will only provide deltas. Our sdk provides helpers to convert these into absolute balances as required by our data model. The responsibility of the module is to extract these relative changes and communicate them. The recommended output models for this module is `BlockBalanceDeltas`
+- `store_balances(balance_deltas, balance_store)`:
+  - This module stores the relative balances deltas in an additive store, which essentially converts them into absolute balances.
+- `map_protocol_changes(balance_deltas, balance_store, components_store, ...)`:
+  - The module that pulls everything together and build the final output model: `BlockChanges`.
+
+The DAG formed by this structure can be seen below:
+
+```mermaid
+graph TD;
+  map_components[map: map_components];
+  sf.ethereum.type.v2.Block[source: sf.ethereum.type.v2.Block] --> map_components;
+  store_components[store: store_components];
+  map_components --> store_components;
+  map_relative_balances[map: map_relative_balances];
+  sf.ethereum.type.v2.Block[source: sf.ethereum.type.v2.Block] --> map_relative_balances;
+  store_components --> map_relative_balances;
+  store_balances[store: store_balances];
+  map_relative_balances --> store_balances;
+  map_changes[map: map_changes];
+  sf.ethereum.type.v2.Block[source: sf.ethereum.type.v2.Block] --> map_changes;
+  map_components --> map_changes;
+  map_relative_balances --> map_changes;
+  store_components --> map_changes;
+  store_balances -- deltas --> map_changes;
+```
+
+### Tracking Components
+
+Usually the first step consists of detecting the creation of new components and storing their contract addresses in a store so they can be properly tracked further downstream.
+
+Later we'll have to emit balance and state changes based on the set of currently tracked components.
+
+{% hint style="info" %}
+Emitting state changes of components that have not been previously announced is considered an error.
+{% endhint %}
+
+Usually you want to start by implementing a map module that emits any newly created components. These newly created components are detected by inspecting the `sf.ethereum.type.v2.Block` model.
+
+The output message should then contain as much information about the component available at that time, as well as the transaction that created the protocol component.
+
+The recommended action here is to implement a `factory.rs` module that will help with detecting newly deployed components. Then use that module within a map handler to detect any newly emitted protocol components. The recommended output model for this first handler is `BlockTransactionProtocolComponents`:
+
+```protobuf
+// A message containing protocol components that were created by a single tx.
+message TransactionProtocolComponents {
+  Transaction tx = 1;
+  repeated ProtocolComponent components = 2;
+}
+
+// All protocol components that were created within a block with their corresponding tx.
+message BlockTransactionProtocolComponents {
+  repeated TransactionProtocolComponents tx_components = 1;
+}
+```
+
+Note that a single transaction may emit multiple newly created components. In this case it is expected that the `TransactionProtocolComponents.components` contains multiple `ProtocolComponents`.
+
+Once emitted, the protocol components should be stored in a Store since we will later have to use this store to decide whether a contract is interesting to us or not.
+
+### Tracking Absolute Balances
+
+Tracking balances can be tricky since often balance information is only available in relative values. 
+
+This means the relative values have to be aggregated by component and token to arrive at an absolute value. Additionally, throughout this aggregation we need to track the balance changes per transaction within a block.
+
+Since this is challenging the following approach is recommended:
+
+#### 1. Index relative balance changes
+
+To accurately process a block and report the respective balance changes, implement a handler that utilises the `BlockBalanceDeltas` struct. It is crucial to ensure that each `BalanceDelta` within a component token pair is assigned a strictly increasing ordinal. This specificity is key to maintaining the integrity of aggregated values at the transaction level. Incorrect ordinal sequencing could lead to inaccurate balance reporting.
+
+Below is an example of an interface for a handler. This handler interfaces with a store that employs an integer as an indicator to denote whether a specific address (identified by the keys) is a component.
+
+```rust
+#[substreams::handlers::map]
+pub fn map_relative_balances(
+    block: eth::v2::Block,
+    components_store: StoreGetInt64,
+) -> Result<BlockBalanceDeltas, anyhow::Error> {
+    todo!()
+}
+```
+
+Our Substreams SDK provides the `tycho_substream::balances::extract_balance_deltas_from_tx` function that extracts all relevant `BalanceDelta` from ERC20 `Transfer` events for a given transaction (see Curve implementation).
+
+#### 2. Aggregate balances with an additive store
+
+To aggregate `BlockBalanceDeltas` messages into absolute values efficiently while maintaining transaction level granularity, we can leverage the additive `StoreAddBigInt` type with a store module.
+
+The `tycho_substream::balances::store_balance_changes` helper function is available for this purpose, streamlining the implementation significantly.&#x20;
+
+Thus, the typical use case can be addressed with the provided snippet:
+
+```rust
+#[substreams::handlers::store]
+pub fn store_balances(deltas: BlockBalanceDeltas, store: StoreAddBigInt) {
+    tycho_substreams::balances::store_balance_changes(deltas, store);
+}
+```
+
+#### 3. Combine absolute values with component and address
+
+Last but not least, we have to associate the absolute balances with their respective transaction, component and token.
+
+To simplify the final step of aggregating balance changes, utilise the `tycho_substream::balances::aggregate_balances_changes` helper function. Detailed instructions are available in the function's docstring. Essentially, it outputs aggregated `BalanceChange` structs per transaction. These aggregates can then be seamlessly integrated into `map_protocol_changes` for retrieving absolute balance changes associated with each transaction.
+
+Below is a snip on how it is usually used:
+
+```rust
+#[substreams::handlers::map]
+pub fn map_protocol_changes(
+    block: eth::v2::Block,
+    grouped_components: BlockTransactionProtocolComponents,
+    deltas: BlockBalanceDeltas,
+    components_store: StoreGetInt64,
+    balance_store: StoreDeltas,
+) -> Result<BlockChanges> {
+    let mut transaction_contract_changes: HashMap<_, TransactionChanges> = HashMap::new();
+
+    aggregate_balances_changes(balance_store, deltas)
+        .into_iter()
+        .for_each(|(_, (tx, balances))| {
+            transaction_contract_changes
+                .entry(tx.index)
+                .or_insert_with(|| TransactionChanges::new(&tx))
+                .balance_changes
+                .extend(balances.into_values());
+        });
+}
+```
+
+### Tracking State Changes
+
+In vm implementations, it's crucial to accurately identify and extract all relevant contract changes. Typically, there's a one-to-one mapping between contracts and components which allows us to follow the convention of utilising the hex-encoded address as the component's ID.
+
+To facilitate the extraction of pertinent changes from the expanded block model, we recommend using the `tycho_substreams::contract::extract_contract_changes` helper function. This function significantly simplifies the process.
+
+Below, we illustrate how to leverage a component store to define a predicate. This predicate serves to pinpoint the contract addresses that are of particular interest:
+
+```rust
+use tycho_substreams::contract::extract_contract_changes;
+
+let mut transaction_contract_changes: HashMap<_, TransactionChanges> = HashMap::new();
+
+extract_contract_changes(
+    &block,
+    |addr| {
+        components_store
+            .get_last(format!("pool:{0}", hex::encode(addr)))
+            .is_some()
+    },
+    &mut transaction_contract_changes,
+);
+```

docs/indexing/overview.md

@@ -0,0 +1,53 @@
+# Overview
+
+This page gives an overview over the data model required to ingest protocol state into the PropellerHeads solver. 
+
+To integrate a protocol PropellerHeads rely on either native or vm logic. Most integration will likely choose to use the VM, as this is usually less effort, the guide will focus mostly on providing state for vm integrations. 
+
+Native integration should operate following exactly the same pattern, just that they should emit changed attributes instead of changes contract storage slots. 
+
+### Understanding the Data Model
+
+PropellerHeads ingest all data versioned by block and transaction. This helps maintain a low latency feed and deal correctly with chains that can experience reverts.
+
+This means each state change that is communicated must be communicated with its respective transaction that caused the change.
+
+Next, for each emitted transactions that carries state changes, the corresponding block must be provided as well.
+
+So basically when processing a block we need to emit the block itself, all transactions that introduced protocol state changes and last but not least the state changes themselves, associated to their corresponding transaction.
+
+**The data model that encodes changes, transaction and blocks in messages, can be found** [**here**](https://github.com/propeller-heads/propeller-protocol-lib/tree/main/proto/tycho/evm/v1)**.** 
+
+#### Models
+
+The models below are used for communication between Substreams and Tycho indexer, as well as between Substreams modules.
+
+Our indexer expects to receive a `BlockChanges` output from your Substreams package.
+
+{% @github-files/github-code-block url="https://github.com/propeller-heads/propeller-protocol-lib/blob/main/proto/tycho/evm/v1/" %}
+
+Please be aware that changes need to be aggregated on the transaction level, it is considered an error to emit `BlockChanges` with duplicated transactions present in the `changes` attributes.
+
+#### Integer Byte encoding
+
+Many of the types above are variable length bytes. This allows for flexibility across blockchains but require agreeing on an informal interface, so later applications know how to interpret these bytes.
+
+**Integers:** especially integers used to communicate balances, should always be encoded as unsigned big-endian integer. This is simply because balances serve multiple purposes within the system and need to be decoded in multiple location of the messages journey.
+
+**Strings**: If you need to store strings, please use utf-8 encoding to store them as bytes.
+
+**Attributes:** the value encoding for attributes is variable. It depends on the use case. Since the attributes are highly dynamic they are only used by the corresponding logic components, so the encoding can be tailored to the logic implementation: E.g. since Rust uses little endian one may choose to use little endian encoding for integers if the native logic module is written in Rust.
+
+#### Special attribute names
+
+Certain attribute names are reserved exclusively for specific purposes in our simulation process. Please use them only for their intended functions. See the [list of reserved attributes](./reserved-attributes.md)
+
+### Changes of interest
+
+PropellerHeads integrations should at least communicate the following changes:
+
+- Any changes to the protocol state, for VM integrations that usually means contract storage changes of all contracts whose state may be accessed during a swap operation.
+- Any newly added protocol component such as a pool, pair, market, etc. Basically anything that signifies that a new operation can be executed now using the protocol.
+- ERC20 Balances, whenever the balances of one contracts involved with the protocol change, this change should be communicated in terms of absolute balances.
+
+Please see the getting started page to see how to actually implement an integration.

docs/logic/vm-integration/README.md

@@ -8,28 +8,31 @@ To create a VM integration, it is required to provide a manifest file as well as
 
 Following exchanges have been integrated using VM approach:
 
-- Uniswap V2 (see `/evm/src/uniswap-v2`)
+* Uniswap V2 (see `/evm/src/uniswap-v2`)
-- Balancer V2 (see `/evm/src/balancer-v2`)
+* Balancer V2 (see `/evm/src/balancer-v2`)
 
 ## Step by step
 
 ### Prerequisites
 
 1.  Install [Foundry](https://book.getfoundry.sh/getting-started/installation#using-foundryup), start by downloading and installing the Foundry installer:
+
     ```bash
     curl -L https://foundry.paradigm.xyz | bash
     ```
+
     then start a new terminal session and run
+
     ```bash
     foundryup
     ```
+2.  Start by making a local copy of the Propeller Protocol Lib repository:
 
-1. Start by making a local copy of the Propeller Protocol Lib repository:
     ```bash
     git clone https://github.com/propeller-heads/propeller-protocol-lib
     ```
+3.  Install forge dependencies:
 
-1. Install forge dependencies:
     ```bash
     cd ./propeller-protocol-lib/evm/
     forge install
@@ -37,26 +40,31 @@ Following exchanges have been integrated using VM approach:
 
 ### Understanding the ISwapAdapter
 
-Read the the documentation of the [Ethereum Solidity interface](ethereum-solidity.md). It describes the functions that need to be implemented as well as the manifest file.
+Read the the documentation of the [Ethereum Solidity interface](ethereum-solidity.md). It describes the functions that need to be implemented as well as the manifest file. Additionally read through the docstring of the [ISwapAdapter.sol](https://github.com/propeller-heads/propeller-venue-lib/blob/main/evm/src/interfaces/ISwapAdapter.sol) interface and the [ISwapAdapterTypes.sol](https://github.com/propeller-heads/propeller-venue-lib/blob/main/evm/src/interfaces/ISwapAdapterTypes.sol) interface which defines the data types and errors used by the adapter interface. You can also generate the documentation locally and the look at the generated documentation in the `./docs` folder:
-Additionally read through the docstring of the [ISwapAdapter.sol](../../../evm/src/interfaces/ISwapAdapter.sol) interface and the [ISwapAdapterTypes.sol](../../../evm/src/interfaces/ISwapAdapterTypes.sol) interface which defines the data types and errors used by the adapter interface.
+
-You can also generate the documentation locally and the look at the generated documentation in the `./docs` folder:
 ```bash
 cd ./evm/
 forge doc
 ```
+
 ### Implementing the ISwapAdapter interface
+
 Your integration should be in a separate directory in the `evm/src` folder. Start by cloning the template directory:
+
 ```bash
 cp ./evm/src/template ./evm/src/<your-adapter-name>
 ```
+
 Implement the `ISwapAdapter` interface in the `./evm/src/<your-adapter-name>.sol` file. There are two reference implementations, one for Uniswap V2 and the other for Balancer V2.
 
 ### Testing your implementation
+
 Clone the `evm/test/TemplateSwapAdapter.t.sol` file and rename it to `<your-adapter-name>.t.sol`. Implement the tests for your adapter, make sure all implemented functions are tested and working correctly. Look at the examples of `UniswapV2SwapAdapter.t.sol` and `BalancerV2SwapAdapter.t.sol` for reference. The [Foundry test guide](https://book.getfoundry.sh/forge/tests) is a good reference, especially the chapter for [Fuzz testing](https://book.getfoundry.sh/forge/fuzz-testing), which is used in both the Uniswap and Balancer tests.
 
 We are using fork testing, i.e. we are running a local Ethereum node and fork the mainnet state. This allows us to test the integration against the real contracts and real data. To run the tests, you need to set the `ETH_RPC_URL` environment variable to the URL of an ethereum RPC. It can be your own node or a public one, like [Alchemy](https://www.alchemy.com/) or [Infura](https://infura.io/).
 
 Finally, run the tests with:
+
 ```bash
 cd ./evm
 forge test