gitbook-docs/development_development/development_reward_block_app/development_sqlite.md

SQLite Service

All our work so far has been about gathering block reward information for the latest block:

// Block reward info on Wednesday, March 17, at 2021 7:42:57 PM GMT
// for block 12058131:
"{\"status\":\"1\",\"message\":\"OK\",\"result\":{\"blockNumber\":\"12058131\",
\"timeStamp\":\"1616010177\",\"blockMiner\":\"0x829bd824b016326a401d083b33d092293333a830\",
\"blockReward\":\"6159144598411626490\",\"uncles\":[
{\"miner\":\"0xe72f79190bc8f92067c6a62008656c6a9077f6aa\",\"unclePosition\":\"0\",
\"blockreward\":\"500000000000000000\"}],
\"uncleInclusionReward\":\"62500000000000000\"}}"

which happens about every 13 seconds or so on mainnet and every four seconds on Kovan. Rather than stashing the block reward results in a frontend-based storage solution, we deploy an SQLite service as our peer-to-peer hosted Ethqlite service. Please see the ethqlite repo for the code.

To get SQLite as a service, we build our service from two modules: the ethqlite repo and the Fluence sqlite Wasm module, which we can build or pickup as a wasm files from the releases. This largely, but not entirely, mirrors what we did with the cUrl service: build the service by providing an adapter to the binary. Unlike the cUrl binary, we are bringing our own sqlite binary, i.e., sqlite3.wasm, with us.

This leaves us to code our ethqlite module with respect to desired CRUD interfaces and security. As previously discussed, we want writes to the sqlite services to be privileged, which implies that we need to own the service and have the client seed to manage authentication and ambient authorization. Specifically, we can implement a rudimentary authorization system where authentication implies authorization to write. The is_owner function in the ethqlite repo does exactly that: if the caller can prove ownership by providing a valid client seed, than we have a true condition equating write-privileged ownership with the caller identity:

// auth.rs
use fluence::{fce, CallParameters};
use::fluence;
use crate::get_connection;

pub fn is_owner() -> bool {
    let meta = fluence::get_call_parameters();
    let caller = meta.init_peer_id;
    let owner = meta.service_creator_peer_id;

    caller == owner
}

#[fce]
pub fn am_i_owner() -> bool {
    is_owner()
}

where the fluence::get_call_parameters is a FCE function returning the populated CallParameter struct defined in the Fluence Rust SDK provides us with the salient creator and caller parameters at runtime.

While the majority of the CRUD operations in crud.rs are standard fare except, the auth & auth check appears in update_reward_blocks:

// crud.rs
#[fce]
pub fn update_reward_blocks(data_string: String) -> UpdateResult {   
    if !is_owner() {                                // <=  auth & auth check !!
        return UpdateResult { success:false, err_str: "You are not the owner".into()};
    }

    let obj:serde_json::Value = serde_json::from_str(&data_string).unwrap();
    let obj = obj["result"].clone();

    if obj["blockNumber"] == serde_json::Value::Null {
        return UpdateResult { success:false, err_str: "Empty reward block string".into()};
    }

    let conn = get_connection();

    let insert = "insert or ignore into reward_blocks values(?, ?, ?, ?)";
    let mut ins_cur = conn.prepare(insert).unwrap().cursor();
<snip>

That is, any non-permissioned call is prevented from write operations and an error message is returned. Please note that in main.rs we have a few admin convenience functions that are also protected by the is_owner guard.

Building and Deploying Ethqlite

Our build.sh script should look quite familiar with the possible exception of downloading the already built sqlite3.wasm file:

# build.sh
#!/bin/sh

fce build --release

rm artifacts/*
cp target/wasm32-wasi/release/ethqlite.wasm artifacts/
wget https://github.com/fluencelabs/sqlite/releases/download/v0.10.0_w/sqlite3.wasm
mv sqlite3.wasm artifacts/

Run ./build.sh and check the artifacts for the expected wasm files

Like all Fluence services, Ethqlite needs a service configuration file, which looks a little more involved than what we have seen so far.

modules_dir = "artifacts/"

[[module]]
    name = "sqlite3"
    mem_pages_count = 100
    logger_enabled = false

    [module.wasi]
    preopened_files = ["/tmp"]
    mapped_dirs = { "tmp" = "/tmp" }



[[module]]
name = "ethqlite"
    mem_pages_count = 1
    logger_enabled = false

    [module.wasi]
    preopened_files = ["/tmp"]
    mapped_dirs = { "tmp" = "/tmp" }

Let's break it down:

• the first module section
  • specifies the sqlite3.wasm module we pulled from the repo,
  • allocates memory, where each page is about 64KB, and
  • permissions and maps node file access
• the second section is for our business logic CRUD adapter module where, again, we allocate the memory and permission and map file access.

We can now fire up fce-repl:

fce-repl Config.toml
Welcome to the FCE REPL (version 0.5.2)
app service was created with service id = 9b923db7-3747-41ab-b1fd-66bd0ccd9f68
elapsed time 916.210305ms

1> interface
Loaded modules interface:
UpdateResult {
  success: I32
  err_str: String
}
RewardBlock {
  block_number: S64
  timestamp: S64
  block_miner: String
  block_reward: String
}
InitResult {
  success: I32
  err_msg: String
}
MinerRewards {
  miner_address: String
  rewards: Array<String>
}
DBOpenDescriptor {
  ret_code: S32
  db_handle: U32
}
DBPrepareDescriptor {
  ret_code: S32
  stmt_handle: U32
  tail: U32
}
DBExecDescriptor {
  ret_code: S32
  err_msg: String
}

ethqlite:
  fn init_service() -> InitResult
  fn get_miner_rewards(miner_address: String) -> MinerRewards
  fn owner_nuclear_reset() -> I32
  fn get_reward_block(block_number: U32) -> RewardBlock
  fn update_reward_blocks(data_string: String) -> UpdateResult
  fn get_latest_reward_block() -> RewardBlock
  fn am_i_owner() -> I32

sqlite3:
  fn sqlite3_reset(stmt_handle: U32) -> S32
  <snip>
  fn sqlite3_column_blob(stmt_handle: U32, icol: S32) -> Array<U8>

and see all the public Fluence interfaces including the ones from the sqlite3.wasm module. Let's upload the service to the local network:

fldist --node-id 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17  --node-addr /ip4/127.0.0.1/tcp/9999/ws/p2p/12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 new_service --ms ethqlite/artifacts/sqlite3.wasm:ethqlite/sqlite3_cfg.json ethqlite/artifacts/ethqlite.wasm:ethqlite/ethqlite_cfg.json --name EthQlite
client seed: 7VqRt2kXWZ15HABKh1hS4kvGfRcBA69cYuzV1Rwm3kHv
client peerId: 12D3KooWCzWm4xBv7nApuK8vNLSbKKYV36kvkz3ywqj5xcjscnz9
relay peerId: 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17
uploading blueprint EthQlite to node 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 via client 12D3KooWCzWm4xBv7nApuK8vNLSbKKYV36kvkz3ywqj5xcjscnz9
service id: fb9ba691-c0fc-4500-88cc-b74f3b281088
service created successfully

Now that we crated the service on our local node, let's make sure that we have the necessary owner privileges. First, we create a little AIR script that calls the am_i_owner function from thee ethqlite service:

; am_i_owner.clj
(xor
    (seq
        (seq
            (call relay ("op" "identity") [])
            (call node_1 (service "am_i_owner") [] result)
        )
        (seq
            (call relay ("op" "identity") [])
            (call %init_peer_id% (returnService "run") [result])
        )
    )
    (seq
        (call relay ("op" "identity") [])
        (call %init_peer_id% (returnService "run") ["XOR FAILED" %last_error%])
    )
)

and run it with the fldist tool:

 fldist --node-id 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17  --node-addr /ip4/127.0.0.1/tcp/9999/ws/p2p/12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 run_air -p air-scripts/am_i_owner.clj -d '{"service":"fb9ba691-c0fc-4500-88cc-b74f3b281088", "node":"12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17"}'
client seed: 3J8BqpGTQ1Ujbr8dvnpTxfr5EUneHf9ZwW84ru9sNmj7
client peerId: 12D3KooW9z5hBDY6cXnkEGraiPFn6hJ3VstqAkVaAM7oThTiWVjL
relay peerId: 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17
Particle id: efa37779-e3aa-4353-b63d-12b444b6366b. Waiting for results... Press Ctrl+C to stop the script.
===================
[
  0
]
[
  [
    {
      peer_pk: '12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17',
      service_id: 'fb9ba691-c0fc-4500-88cc-b74f3b281088',
      function_name: 'am_i_owner',
      json_path: ''
    }
  ]
]
===================

As discussed earlier, the service needs some proof that we have owner privileges, which we can provide by adding the client seed, -s, to our call parameters:

fldist --node-id 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17  --node-addr /ip4/127.0.0.1/tcp/9999/ws/p2p/12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 run_air -p air-scripts/am_i_owner.clj -d '{"service":"fb9ba691-c0fc-4500-88cc-b74f3b281088", "node":"12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17"}' -s 7VqRt2kXWZ15HABKh1hS4kvGfRcBA69cYuzV1Rwm3kHv
client seed: 7VqRt2kXWZ15HABKh1hS4kvGfRcBA69cYuzV1Rwm3kHv
client peerId: 12D3KooWCzWm4xBv7nApuK8vNLSbKKYV36kvkz3ywqj5xcjscnz9
relay peerId: 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17
Particle id: f0371615-7d75-4971-84a9-3111b8263de7. Waiting for results... Press Ctrl+C to stop the script.
===================
[
  1
]
[
  [
    {
      peer_pk: '12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17',
      service_id: 'fb9ba691-c0fc-4500-88cc-b74f3b281088',
      function_name: 'am_i_owner',
      json_path: ''
    }
  ]
]
===================

and all is well. So where does that client seed 7VqRt2kXWZ15HABKh1hS4kvGfRcBA69cYuzV1Rwm3kHv come from ? The easy answer is that we copied it from the service creation return values -- line 2 above. But that doesn't really answer the question. The more involved answer is that every developer should have one or more cryptographic key pairs from which the client seed is derived. Moreover, creating a new service, the client seed should be specified but if not, the system creates one instead as above.

The easiest way to get a keypair and seed is from the fldist tool:

fldist create_keypair
client seed: 8LKYUmsWkMSiHBxo8deXyNJD3wXutq265TSTcmmtgQTJ
client peerId: 12D3KooWRtrFyYjis4qQpC4kHcJWbtpM4mZgLYBoDn93eXJEGtVH
relay peerId: 12D3KooWBUJifCTgaxAUrcM9JysqCcS4CS8tiYH5hExbdWCAoNwb
{
  id: '12D3KooWKphxxaXofYzC2TsN79RHZVubjmutKVdPUxVMHY3ZsVww',
  privKey: 'CAESQO/TcX2DkTukK6XxJUc/2U6gqOLVza5PRWM2FhXfJ1qilKtA6qsHx0Rdibwxsg4Vh7JjTfRfMXSlLJphGCOb7zI=',
  pubKey: 'CAESIJSrQOqrB8dEXYm8MbIOFYeyY030XzF0pSyaYRgjm+8y',
  seed: 'H9BSbZwKmFs93462xbAyfEdGdMXb5LZuXL7GSA4uPK4V'
}

So let's re-deploy the Ethqlite service and specify the client seed at creation time:

fldist --node-id 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17  --node-addr /ip4/127.0.0.1/tcp/9999/ws/p2p/12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 new_service --ms ethqlite/artifacts/sqlite3.wasm:ethqlite/sqlite3_cfg.json ethqlite/artifacts/ethqlite.wasm:ethqlite/ethqlite_cfg.json --name EthQliteSecure -s H9BSbZwKmFs93462xbAyfEdGdMXb5LZuXL7GSA4uPK4V
client seed: H9BSbZwKmFs93462xbAyfEdGdMXb5LZuXL7GSA4uPK4V
client peerId: 12D3KooWKphxxaXofYzC2TsN79RHZVubjmutKVdPUxVMHY3ZsVww
relay peerId: 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17
uploading blueprint EthQliteSecure to node 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 via client 12D3KooWKphxxaXofYzC2TsN79RHZVubjmutKVdPUxVMHY3ZsVww
service id: 470fcaba-6834-4ccf-ac0c-4f6494e9e77b
service created successfully

Updating the call parameters to reflect the new service id and client seed confirms our ownership over the service:

fldist --node-id 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17  --node-addr /ip4/127.0.0.1/tcp/9999/ws/p2p/12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 run_air -p air-scripts/am_i_owner.clj -d '{"service":"470fcaba-6834-4ccf-ac0c-4f6494e9e77b", "node":"12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17"}' -s H9BSbZwKmFs93462xbAyfEdGdMXb5LZuXL7GSA4uPK4V
client seed: H9BSbZwKmFs93462xbAyfEdGdMXb5LZuXL7GSA4uPK4V
client peerId: 12D3KooWKphxxaXofYzC2TsN79RHZVubjmutKVdPUxVMHY3ZsVww
relay peerId: 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17
Particle id: 6d8c158b-d998-44ca-9d4c-255ce4b9cd21. Waiting for results... Press Ctrl+C to stop the script.
===================
[
  1
]
[
  [
    {
      peer_pk: '12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17',
      service_id: '470fcaba-6834-4ccf-ac0c-4f6494e9e77b',
      function_name: 'am_i_owner',
      json_path: ''
    }
  ]
]
===================

Back to our task at hand: persisting reward block data to our sqlite as a service. Looking over the source code, we know that in order to accomplish persistence, we need to:

• init the database: pub fn init_service() -> InitResult
• provide reward data : pub fn update_reward_blocks(data_string: String) -> UpdateResult

Initializing Ethqlite for the most part is a one time event, so we'll do it right now and outside of our recurring block discovery and commit workflow with another small AIR script:

; ethqlite_init.clj
(xor
    (seq
        (seq
            (call relay ("op" "identity") [])
            (call node_1 (service "init_service") [] result)
        )
        (seq
            (call relay ("op" "identity") [])
            (call %init_peer_id% (returnService "run") [result])
        )
    )
    (seq
        (call relay ("op" "identity") [])
        (call %init_peer_id% (returnService "run") ["XOR FAILED" %last_error%])
    )   
)

which we deploy to the node with the fldist tool:

fldist --node-id 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17  --node-addr /ip4/127.0.0.1/tcp/9999/ws/p2p/12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17 run_air -p air-scripts/ethqlite_init.clj -d '{"service":"470fcaba-6834-4ccf-ac0c-4f6494e9e77b", "node":"12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17"}' -s H9BSbZwKmFs93462xbAyfEdGdMXb5LZuXL7GSA4uPK4V
client seed: H9BSbZwKmFs93462xbAyfEdGdMXb5LZuXL7GSA4uPK4V
client peerId: 12D3KooWKphxxaXofYzC2TsN79RHZVubjmutKVdPUxVMHY3ZsVww
relay peerId: 12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17
Particle id: 2fb4a366-6f40-46c1-9329-d77c6d03dfad. Waiting for results... Press Ctrl+C to stop the script.
===================
[
  {
    "err_msg": "",
    "success": 1
  }
]
[
  [
    {
      peer_pk: '12D3KooWQQYXh78acqBNuL5p1J5tmH4XCKLCHM21tMb8pcxqGL17',
      service_id: '470fcaba-6834-4ccf-ac0c-4f6494e9e77b',
      function_name: 'init_service',
      json_path: ''
    }
  ]
]
===================

If you run the init script again, you will receive an error "Service already initiated", so we can be reasonably confident our code is working and it looks like our Ethqlite service is up and running on the local node.

Due to the security concerns for our database, it is not advisable, or even possible, to use an already deployed Sqlite service from the Fluence Dashboard. Instead, we deploy our own instance with our own secret client seed. To determine which network nodes are available, run:

fldist --env testnet env
client seed: Cj4Wpy5y955o2N3T8Hs5myRoFGhBaBhytCdsYeyFLQPw
client peerId: 12D3KooWQg8cyj4z8Bv4rGq1PeXL1XKEQd6Z2CCFguy9D4NnLaKm
relay peerId: 12D3KooWBUJifCTgaxAUrcM9JysqCcS4CS8tiYH5hExbdWCAoNwb
/dns4/net01.fluence.dev/tcp/19001/wss/p2p/12D3KooWEXNUbCXooUwHrHBbrmjsrpHXoEphPwbjQXEGyzbqKnE9
/dns4/net01.fluence.dev/tcp/19990/wss/p2p/12D3KooWMhVpgfQxBLkQkJed8VFNvgN4iE6MD7xCybb1ZYWW2Gtz
/dns4/net02.fluence.dev/tcp/19001/wss/p2p/12D3KooWHk9BjDQBUqnavciRPhAYFvqKBe4ZiPPvde7vDaqgn5er
/dns4/net03.fluence.dev/tcp/19001/wss/p2p/12D3KooWBUJifCTgaxAUrcM9JysqCcS4CS8tiYH5hExbdWCAoNwb
/dns4/net04.fluence.dev/tcp/19001/wss/p2p/12D3KooWJbJFaZ3k5sNd8DjQgg3aERoKtBAnirEvPV8yp76kEXHB
/dns4/net05.fluence.dev/tcp/19001/wss/p2p/12D3KooWCKCeqLPSgMnDjyFsJuWqREDtKNHx1JEBiwaMXhCLNTRb
/dns4/net06.fluence.dev/tcp/19001/wss/p2p/12D3KooWKnRcsTpYx9axkJ6d69LPfpPXrkVLe96skuPTAo76LLVH
/dns4/net07.fluence.dev/tcp/19001/wss/p2p/12D3KooWBSdm6TkqnEFrgBuSkpVE3dR1kr6952DsWQRNwJZjFZBv
/dns4/net08.fluence.dev/tcp/19001/wss/p2p/12D3KooWGzNvhSDsgFoHwpWHAyPf1kcTYCGeRBPfznL8J6qdyu2H
/dns4/net09.fluence.dev/tcp/19001/wss/p2p/12D3KooWF7gjXhQ4LaKj6j7ntxsPpGk34psdQicN2KNfBi9bFKXg
/dns4/net10.fluence.dev/tcp/19001/wss/p2p/12D3KooWB9P1xmV3c7ZPpBemovbwCiRRTKd3Kq2jsVPQN4ZukDfy

which lists the available testnet peers. Pick one, update the node-id parameter and drop the node-addr parameter in your deployment command-line, upload the new ethqlite service and initiate it. Congrat's, you are now the proud maker of a Fluence testnet Ehqlite service!

Now it is time to get block data into the database.