chore: Add observability to docker-compose (#113)

* Add observability

* Update

* Update

* Fix indent

* Fix again

* Fix againt

* fix

* Add external address

* Fix links

* Fixes

* Fix

* Add IPFS

---------

Co-authored-by: folex <0xdxdy@gmail.com>

README.md

@@ -1,6 +1,7 @@
 # Rust Peer Distro
 
-The distributive and packaging of the [Rust peer](https://github.com/fluencelabs/rust-peer).
+The distributive and packaging of the
+[Rust peer](https://github.com/fluencelabs/rust-peer).
 
 ## Installation and usage
 
@@ -14,27 +15,37 @@ To get a list of commands that can be passed to rust-peer run:
 docker run --rm --name rust-peer fluencelabs/rust-peer:latest --help
 ```
 
-See deployment instructions and tips at [deploy](deploy).
+See deployment instructions and tips at
+[deploy](https://github.com/fluencelabs/rust-peer-distro/tree/master/deploy).
 
 ## Documentation
 
-- Rust peer distro [image flavours](docs/flavours.md)
+- Rust peer distro
-- [Builtin services](docs/builtins.md)
+  [image flavours](https://github.com/fluencelabs/rust-peer-distro/tree/master/docs/flavours.md)
-
+- [Builtin services](https://github.com/fluencelabs/rust-peer-distro/tree/master/docs/builtins.md)
-Comprehensive documentation on everything related to Fluence can be found [here](https://fluence.dev/). Check also our [YouTube channel](https://www.youtube.com/@fluencelabs).
 
+Comprehensive documentation on everything related to Fluence can be found
+[here](https://fluence.dev/). Check also our
+[YouTube channel](https://www.youtube.com/@fluencelabs).
 
 ## Support
 
-Please, file an [issue](https://github.com/fluencelabs/rust-peer-distro/issues) if you find a bug. You can also contact us at [Discord](https://discord.com/invite/5qSnPZKh7u) or [Telegram](https://t.me/fluence_project).  We will do our best to resolve the issue ASAP.
+Please, file an [issue](https://github.com/fluencelabs/rust-peer-distro/issues)
-
+if you find a bug. You can also contact us at
+[Discord](https://discord.com/invite/5qSnPZKh7u) or
+[Telegram](https://t.me/fluence_project). We will do our best to resolve the
+issue ASAP.
 
 ## Contributing
 
-Any interested person is welcome to contribute to the project. Please, make sure you read and follow some basic [rules](./CONTRIBUTING.md). The Contributor License Agreement can be found [here](./FluenceCLA).
+Any interested person is welcome to contribute to the project. Please, make sure
-
+you read and follow some basic
+[rules](https://github.com/fluencelabs/rust-peer-distro/tree/master/CONTRIBUTING.md).
+The Contributor License Agreement can be found
+[here](https://github.com/fluencelabs/rust-peer-distro/tree/master/FluenceCLA).
 
 ## License
 
-All software code is copyright (c) Fluence Labs, Inc. under the [Apache-2.0](./LICENSE) license.
+All software code is copyright (c) Fluence Labs, Inc. under the
-
+[Apache-2.0](https://github.com/fluencelabs/rust-peer-distro/tree/master/LICENSE)
+license.

deploy/README.md

@@ -3,8 +3,7 @@
 Currently there is only one way to try out the Rust peer. It is by running it with [docker-compose](docker-compose/). An example configuration for popular orchestration platforms like [HasiCorp Nomad](https://www.nomadproject.io/) and K8S are coming soon! Meanwhile, you can adapt [this docker-compose file](docker-compose/docker-compose.yml) for the platform of your choice.
 
 
-## Configuring Rust Peer Distro Image
+## Configuring rust-peer distro image
 
-Checkout [this doc](../docs/flavours.md) to learn more about Rust peer image
+Checkout [this doc](../docs/flavours.md) to learn more about rust-peer image
 flavours and environment variables used to configure the images.
-

deploy/docker-compose/README.md

@@ -1,40 +1,121 @@
 # Run rust-peer with docker-compose
 
-This docker-compose file starts a local network of three rust-peers.
+This guide explains how to use docker-compose to start a local network of three
+[rust-peer](https://github.com/fluencelabs/rust-peer) nodes.
 
-## Installing docker and docker-compose
+## Introduction
 
-Follow official instruction for
+The rust-peer network is a set of peer nodes that can communicate with each
-[docker](https://docs.docker.com/engine/install/) and
+other to share data and execute code plus local IPFS node used by
-[docker-compose](https://docs.docker.com/compose/install/linux/#install-using-the-repository).
+[aqua-ipfs builtin](../../docs/builtins.md##aqua-ipfs). By running a local
+rust-peer network, you can test your applications in a controlled environment
+without relying on external networks.
 
-## Running local rust-peer network
+## Prerequisites
 
-1. Either `git clone` this repository locally and run `cd deploy/docker-compose`
+Before you can run the rust-peer network, you need to have Docker and
-   or download [`docker-compose.yml`](docker-compose.yml) directly.
+docker-compose installed on your system. You can follow the official
+instructions for installing Docker and installing docker-compose on your
+operating system:
 
-2. Ensure you have the most up-to-date container images:
+- [docker](https://docs.docker.com/engine/install/)
+- [docker-compose](https://docs.docker.com/compose/install/linux/#install-using-the-repository)
 
+## Starting local rust-peer network
+
+1. `git clone` this repository locally and run `cd deploy/docker-compose`.
+
+2. Pull the latest container images by running the following command:
    ```bash
    docker-compose pull
    ```
 
-3. Run the network:
+3. Start the rust-peer network by running the following command:
-
    ```bash
    docker-compose up -d
    ```
 
+This will start three rust-peer nodes, each listening on a different port.
+
 ## Accessing local rust-peer network
 
-Using fluence-cli follow the
+To interact with the rust-peer network, you can use the
-[example workflow](https://github.com/fluencelabs/fluence-cli/blob/main/docs/EXAMPLE.md#currently-supported-workflow-example)
+[fluence-cli](https://github.com/fluencelabs/fluence-cli) tool.
-appending `--relay <peer-multiaddr>` to `run` commands.
 
-Local network multiaddresses:
+1. Run `fluence init` and chose `minimal` project template.
+2. Change `hosts` key in `fluence.yaml` to:
+   ```yml
+   hosts:
+     defaultWorker:
+       peerIds:
+         - 12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR
+   ```
+
+3. Change `relays` key in `fluence.yaml` to:
+   ```yml
+   relays:
+     - /ip4/127.0.0.1/tcp/9991/ws/p2p/12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR
+     - /ip4/127.0.0.1/tcp/9992/ws/p2p/12D3KooWQdpukY3p2DhDfUfDgphAqsGu5ZUrmQ4mcHSGrRag6gQK
+     - /ip4/127.0.0.1/tcp/9993/ws/p2p/12D3KooWRT8V5awYdEZm6aAV9HWweCEbhWd7df4wehqHZXAB7yMZ
+   ```
+
+4. Run
+   ```bash
+   fluence run -f 'helloWorld("Fluence")'
+   fluence run -f 'getInfo()'
+   ```
+
+## Using local rust-peer network in your project
+
+You must make changes to `fluence.yaml` to use a local rust-peer network:
+
+- changing `hosts` key in `fluence.yaml` to:
+  ```yml
+  hosts:
+    defaultWorker:
+      peerIds:
+        - 12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR
+  ```
+- changing `relays` key in `fluence.yaml` to:
+  ```yml
+  relays:
+    - /ip4/127.0.0.1/tcp/9991/ws/p2p/12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR
+    - /ip4/127.0.0.1/tcp/9992/ws/p2p/12D3KooWQdpukY3p2DhDfUfDgphAqsGu5ZUrmQ4mcHSGrRag6gQK
+    - /ip4/127.0.0.1/tcp/9993/ws/p2p/12D3KooWRT8V5awYdEZm6aAV9HWweCEbhWd7df4wehqHZXAB7yMZ
+  ```
+
+You can try following the example
+[workflow](https://github.com/fluencelabs/fluence-cli/blob/main/docs/EXAMPLE.md)
+provided by Fluence Labs making these changes.
+
+Here is a table with multiaddress for each node:
 
 | container | multiaddress                                                                        |
 | --------- | ----------------------------------------------------------------------------------- |
 | peer-1    | /ip4/127.0.0.1/tcp/9991/ws/p2p/12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR |
 | peer-2    | /ip4/127.0.0.1/tcp/9992/ws/p2p/12D3KooWQdpukY3p2DhDfUfDgphAqsGu5ZUrmQ4mcHSGrRag6gQK |
 | peer-3    | /ip4/127.0.0.1/tcp/9993/ws/p2p/12D3KooWRT8V5awYdEZm6aAV9HWweCEbhWd7df4wehqHZXAB7yMZ |
+
+## Running with observability stack
+
+Stack consists of:
+
+- [Prometheus](https://prometheus.io/) - TSDB that collects and stores metrics
+- [Loki](https://grafana.com/logs/) - lightweight centralized logging solution
+- [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) - log
+  collection agent
+- [Grafana](https://grafana.com/grafana/) - data visualization tool
+
+To set it up run:
+
+```bash
+docker-compose -f docker-compose.yml -f docker-compose.observability.yml up -d
+```
+
+Grafana will have automatically preprovisioned dashboards:
+
+- rust-peer stats - overview of rust-peer network
+- Service metrics - detailed stats on deployed services
+
+You can find Grafana at http://localhost:3000. To access rust-peer logs use
+`Explore` tab and chose `Loki` datasource.

deploy/docker-compose/configs/loki.yml

@@ -0,0 +1,40 @@
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+
+ingester:
+  lifecycler:
+    address: 127.0.0.1
+    ring:
+      kvstore:
+        store: inmemory
+      replication_factor: 1
+    final_sleep: 0s
+  chunk_idle_period: 5m
+  chunk_retain_period: 30s
+  wal:
+    dir: /data/wal
+
+
+schema_config:
+  configs:
+  - from: 2023-01-01
+    store: boltdb
+    object_store: filesystem
+    schema: v12
+    index:
+      prefix: index_
+      period: 168h
+
+storage_config:
+  boltdb:
+    directory: /data/index
+
+  filesystem:
+    directory: /data/chunks
+
+limits_config:
+  enforce_metric_name: false
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h

deploy/docker-compose/configs/prometheus.yml

@@ -0,0 +1,22 @@
+global:
+  scrape_interval: "15s"
+
+scrape_configs:
+  - job_name: "rust-peer"
+    scrape_interval: "15s"
+    static_configs:
+      - targets:
+          - "peer-1:18080"
+        labels:
+          instance: "peer-1"
+          env: "demo"
+      - targets:
+          - "peer-2:18080"
+        labels:
+          instance: "peer-2"
+          env: "demo"
+      - targets:
+          - "peer-3:18080"
+        labels:
+          instance: "peer-3"
+          env: "demo"

deploy/docker-compose/configs/promtail.yml

@@ -0,0 +1,29 @@
+positions:
+  filename: "/data/positions.yml"
+
+clients:
+  - url: "http://loki:3100/loki/api/v1/push"
+
+scrape_configs:
+  - job_name: "rust-peer"
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        filters:
+          - name: label
+            values: ["scrape=true"]
+    pipeline_stages:
+      - multiline:
+          firstline: '^[(?P<time>\S+)\s+\S+\s+\S+]'
+      - regex:
+          expression: '^[(?P<time>\S+)\s+\S+\s+\S+].*'
+      - timestamp:
+          source: time
+          format: "RFC3339Nano"
+      - drop:
+          expression: "tide::log::middleware"
+          drop_counter_reason: "spam"
+    relabel_configs:
+      - source_labels: ["__meta_docker_container_label_name"]
+        target_label: "instance"
+      - replacement: "demo"
+        target_label: env

deploy/docker-compose/docker-compose.observability.yml

@@ -0,0 +1,54 @@
+services:
+  grafana:
+    image: grafana/grafana
+    ports:
+      - 3000:3000
+    environment:
+      GF_AUTH_DISABLE_LOGIN_FORM: true
+      GF_AUTH_ANONYMOUS_ENABLED: true
+      GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin"
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning
+      - ./grafana/dashboards:/var/lib/grafana/dashboards
+      - grafana:/var/lib/grafana
+
+  prometheus:
+    image: prom/prometheus
+    user: root
+    command:
+      - "--config.file=/config.yml"
+      - "--storage.tsdb.retention.time=168h"
+      - "--storage.tsdb.path=/data"
+      - "--web.enable-lifecycle"
+    ports:
+      - 9090:9090
+    volumes:
+      - ./configs/prometheus.yml:/config.yml
+      - prometheus:/data
+
+  loki:
+    image: grafana/loki:2.7.4
+    user: root
+    expose:
+      - 3100
+    volumes:
+      - ./configs/loki.yml:/config.yml
+      - loki:/data
+    command:
+      - "-config.file=/config.yml"
+
+  promtail:
+    image: grafana/promtail:2.7.4
+    volumes:
+      - ./configs/promtail.yml:/config.yml
+      - promtail:/data
+      - /var/lib/docker/containers:/var/lib/docker/containers
+      - /var/run/docker.sock:/var/run/docker.sock
+    command:
+      - "-config.file=/config.yml"
+
+volumes:
+  grafana:
+  prometheus:
+  loki:
+  promtail:

deploy/docker-compose/docker-compose.yml

@@ -1,39 +1,93 @@
+name: rust-peer-network
+
 services:
   peer-1:
-    image: fluencelabs/rust-peer:latest
+    image: fluencelabs/rust-peer:minimal
+    container_name: "peer-1-12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR"
     ports:
       - 7771:7771
       - 9991:9991
+    expose:
+      - 18080
+    environment:
+      RUST_LOG: "info"
+      FLUENCE_ENV_AQUA_IPFS_EXTERNAL_API_MULTIADDR: /dns4/ipfs/tcp/5001
+      FLUENCE_ENV_AQUA_IPFS_LOCAL_API_MULTIADDR: /dns4/ipfs/tcp/5001
+    labels:
+      name: peer-1
+      scrape: true
+      peer_id: 12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR
     command:
       - --aqua-pool-size=2
       - -t=7771
       - -w=9991
+      - --allow-private-ips
+      - --external-maddrs
+      - /dns4/peer-1/tcp/7771
+      - /dns4/peer-1/tcp/9991/ws
       - --local
       # 12D3KooWBM3SdXWqGaawQDGQ6JprtwswEg3FWGvGhmgmMez1vRbR
       - -k=hK62afickoeP2uZbmSkAYXxxqP8ozq16VRN7qfTP719EHC5V5tjrtW57BSjUr8GvsEXmJRbtejUWyPZ2rZMyQdq
 
   peer-2:
-    image: fluencelabs/rust-peer:latest
+    image: fluencelabs/rust-peer:minimal
+    container_name: "peer-2-12D3KooWQdpukY3p2DhDfUfDgphAqsGu5ZUrmQ4mcHSGrRag6gQK"
     ports:
       - 7772:7772
       - 9992:9992
+    expose:
+      - 18080
+    environment:
+      RUST_LOG: "info"
+      FLUENCE_ENV_AQUA_IPFS_EXTERNAL_API_MULTIADDR: /dns4/ipfs/tcp/5001
+      FLUENCE_ENV_AQUA_IPFS_LOCAL_API_MULTIADDR: /dns4/ipfs/tcp/5001
+    labels:
+      name: peer-2
+      scrape: true
+      peer_id: 12D3KooWQdpukY3p2DhDfUfDgphAqsGu5ZUrmQ4mcHSGrRag6gQK
     command:
       - --aqua-pool-size=2
       - -t=7772
       - -w=9992
+      - --allow-private-ips
+      - --external-maddrs
+      - /dns4/peer-2/tcp/7772
+      - /dns4/peer-2/tcp/9992/ws
       - --bootstraps=/dns/peer-1/tcp/7771
       # 12D3KooWQdpukY3p2DhDfUfDgphAqsGu5ZUrmQ4mcHSGrRag6gQK
       - -k=2WijTVdhVRzyZamWjqPx4V4iNMrajegNMwNa2PmvPSZV6RRpo5M2fsPWdQr22HVRubuJhhSw8BrWiGt6FPhFAuXy
 
   peer-3:
-    image: fluencelabs/rust-peer:latest
+    image: fluencelabs/rust-peer:minimal
+    container_name: "peer-3-12D3KooWRT8V5awYdEZm6aAV9HWweCEbhWd7df4wehqHZXAB7yMZ"
     ports:
       - 7773:7773
       - 9993:9993
+    expose:
+      - 18080
+    environment:
+      RUST_LOG: "info"
+      FLUENCE_ENV_AQUA_IPFS_EXTERNAL_API_MULTIADDR: /dns4/ipfs/tcp/5001
+      FLUENCE_ENV_AQUA_IPFS_LOCAL_API_MULTIADDR: /dns4/ipfs/tcp/5001
+    labels:
+      name: peer-3
+      scrape: true
+      peer_id: 12D3KooWRT8V5awYdEZm6aAV9HWweCEbhWd7df4wehqHZXAB7yMZ
     command:
       - --aqua-pool-size=2
       - -t=7773
       - -w=9993
+      - --allow-private-ips
+      - --external-maddrs
+      - /dns4/peer-3/tcp/7773
+      - /dns4/peer-3/tcp/9993/ws
       - --bootstraps=/dns/peer-1/tcp/7771
       # 12D3KooWRT8V5awYdEZm6aAV9HWweCEbhWd7df4wehqHZXAB7yMZ
       - -k=2n2wBVanBeu2GWtvKBdrYK9DJAocgG3PrTUXMharq6TTfxqTL4sLdXL9BF23n6rsnkAY5pR9vBtx2uWYDQAiZdrX
+
+  ipfs:
+    image: ipfs/kubo
+    ports:
+      - 5001:5001
+    volumes:
+      - ./ipfs.sh:/container-init.d/01-ipfs.sh

deploy/docker-compose/grafana/provisioning/dashboards/dashboards.yml

@@ -0,0 +1,12 @@
+apiVersion: 1
+
+providers:
+  - name: "Dashboard provider"
+    orgId: 1
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: true

deploy/docker-compose/grafana/provisioning/datasources/datasources.yml

@@ -0,0 +1,17 @@
+apiVersion: 1
+deleteDatasources: []
+datasources:
+  - name: Prometheus
+    type: prometheus
+    url: http://prometheus:9090
+    basicAuth: false
+    isDefault: true
+    uid: PAE45454D0EDB9216
+    jsonData:
+      timeInterval: 15s
+
+  - name: Loki
+    type: loki
+    url: http://loki:3100
+    jsonData:
+      maxLines: 1000

deploy/docker-compose/ipfs.sh

@@ -0,0 +1,6 @@
+#! /usr/bin/env sh
+
+ipfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["*"]'
+ipfs config --json API.HTTPHeaders.Access-Control-Allow-Methods '["PUT", "POST"]'
+ipfs config --json Addresses.API '"/ip4/0.0.0.0/tcp/5001"'
+ipfs bootstrap rm --all