98c63c685f
* refactor: stop using private pypi This was required because our repos were not public. Now that they are, people can directly access our Python packages and therefore they don't need access to our private codeartifact anymore * docs: add a list of dependencies in the readme and early check in the setup env script --------- Co-authored-by: zizou <111426680+flopell@users.noreply.github.com>
118 lines
4.3 KiB
Markdown
118 lines
4.3 KiB
Markdown
# Substreams Testing
|
||
|
||
This package provides a comprehensive testing suite for Substreams modules. The testing suite is designed to facilitate
|
||
end-to-end testing, ensuring that your Substreams modules function as expected.
|
||
|
||
## Overview
|
||
|
||
The testing suite builds the `.spkg` for your Substreams module, indexes a specified block range, and verifies that the
|
||
expected state has been correctly indexed in PostgreSQL.
|
||
Additionally, it will also try to simulate some transactions using the `SwapAdapter` interface.
|
||
|
||
## Prerequisites
|
||
|
||
- Latest version of our indexer, Tycho. Please contact us to obtain the latest version. Once acquired, place it in a directory that is included in your system’s PATH.
|
||
- Access to PropellerHeads' private PyPI repository. Please contact us to obtain access.
|
||
- Docker installed on your machine.
|
||
- [Conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html)
|
||
and [AWS cli](https://aws.amazon.com/cli/) installed
|
||
|
||
## Test Configuration
|
||
|
||
Tests are defined in a `yaml` file. A documented template can be found at
|
||
`substreams/ethereum-template/integration_test.tycho.yaml`. The configuration file should include:
|
||
|
||
- The target Substreams config file.
|
||
- The corresponding SwapAdapter and args to build it.
|
||
- The expected protocol types.
|
||
- The tests to be run.
|
||
|
||
Each test will index all blocks between `start-block` and `stop-block`, verify that the indexed state matches the
|
||
expected state and optionally simulate transactions using `SwapAdapter` interface.
|
||
|
||
You will also need the VM Runtime file for the adapter contract.
|
||
Our testing script should be able to build it using your test config.
|
||
The script to generate this file manually is available under `evm/scripts/buildRuntime.sh`.
|
||
|
||
## Setup testing environment
|
||
|
||
## Prerequisites
|
||
|
||
Before setting up the Python environment, ensure the following tools and libraries are installed on your system:
|
||
|
||
- **Git**: Version control tool (https://git-scm.com/)
|
||
- **Rust**: Programming language and toolchain (https://www.rust-lang.org/)
|
||
- **GCC**: GNU Compiler Collection (https://gcc.gnu.org/)
|
||
- **libpq**: PostgreSQL client library (https://www.postgresql.org/docs/9.5/libpq.html)
|
||
- **OpenSSL (libssl)**: OpenSSL development library (https://github.com/openssl/openssl)
|
||
- **pkg-config**: Helper tool for managing compiler flags (https://www.freedesktop.org/wiki/Software/pkg-config/)
|
||
- **Conda**: Python package manager (https://docs.conda.io/en/latest/)
|
||
- **pip**: Python package installer (https://pip.pypa.io/)
|
||
|
||
Run the setup env script. It will create a conda virtual env and install all dependencies.
|
||
This script must be run from within the `tycho-protocol-sdk/testing` directory.
|
||
|
||
```
|
||
setup_env.sh
|
||
```
|
||
|
||
## Running Tests
|
||
|
||
### Prerequisites
|
||
|
||
This section requires a testing environment setup. If you don’t have it yet, please refer to the [setup testing
|
||
environment section](#setup-testing-environment)
|
||
|
||
### Step 1: Export Environment Variables
|
||
|
||
Export the required environment variables for the execution. You can find the available environment variables in the
|
||
`.env.default` file.
|
||
Please create a `.env` file in the `testing` directory and set the required environment variables.
|
||
|
||
#### Environment Variables
|
||
|
||
**RPC_URL**
|
||
|
||
- **Description**: The URL for the Ethereum RPC endpoint. This is used to fetch the storage data. The node needs to be
|
||
an archive node, and support [debug_storageRangeAt](https://www.quicknode.com/docs/ethereum/debug_storageRangeAt).
|
||
- **Example**: `export RPC_URL="https://ethereum-mainnet.core.chainstack.com/123123123123"`
|
||
|
||
**SUBSTREAMS_API_TOKEN**
|
||
|
||
- **Description**: The API token for accessing Substreams services. This token is required for authentication.
|
||
- **Example**: `export SUBSTREAMS_API_TOKEN=eyJhbGci...`
|
||
|
||
### Step 2: Run tests
|
||
|
||
Run local postgres database using docker compose
|
||
|
||
```bash
|
||
docker compose up -d db
|
||
```
|
||
|
||
Run tests for your package.
|
||
|
||
```bash
|
||
python ./testing/src/runner/cli.py --package "your-package-name"
|
||
```
|
||
|
||
#### Example
|
||
|
||
If you want to run tests for `ethereum-balancer-v2`, use:
|
||
|
||
```bash
|
||
conda activate tycho-protocol-sdk-testing
|
||
export RPC_URL="https://ethereum-mainnet.core.chainstack.com/123123123123"
|
||
export SUBSTREAMS_API_TOKEN=eyJhbGci...
|
||
docker compose up -d db
|
||
python ./testing/src/runner/cli.py --package "ethereum-balancer-v2"
|
||
```
|
||
|
||
#### Testing CLI args
|
||
|
||
A list and description of all available CLI args can be found using:
|
||
|
||
```
|
||
python ./testing/src/runner/cli.py --help
|
||
```
|