Encifher
CoProcessor Crates

Submitter

Submitter service responsibilities, endpoints, data formats, and integrations

Submitter

Overview

The submitter is a lightweight HTTP service responsible for:

  • Compressing and publishing batch results to a Data Availability layer (Turbo DA).
  • Submitting a compact commitment (accumulator root, TEE signature, recovery id) to the blockchain. The DA reference is passed to the chain client but is not persisted on-chain in the current implementation.
  • Forwarding withdraw requests to the on-chain program.

It exposes two POST endpoints and delegates DA and chain interactions to dedicated clients. Configuration is read via the shared configuration loader and mapped to environment variables for the chain client.

Batch Submission Workflow

The following diagram shows the complete flow from Processor to DA layer and blockchain:

Submission Steps:

  1. Processor sends BatchTreeSubmission with accumulator root, TEE signature, recovery_id, and batch trees
  2. Submitter compresses batch_trees (JSON → zlib → base64)
  3. Submitter publishes compressed data to Turbo DA with Content-Type: application/octet-stream
  4. DA returns submission_id for retrieval reference
  5. Submitter sets Solana environment variables and submits commitment to blockchain
  6. Chain client submits accumulator root (32B), signature (64B), recovery_id (1B) on-chain
  7. Return success with DA submission_id

Input: BatchTreeSubmission (from Processor accumulator job) Output: DA submission_id + on-chain transaction confirmation

Endpoints

POST /v1/submit_batch

  • Request body: BatchTreeSubmission (see Data Formats).
  • Processing steps:
    1. Serialize batch_trees to JSON using serde_json.
    2. Compress with zlib (default compression level via flate2).
    3. Base64-encode the compressed data (standard alphabet).
    4. Submit the base64 string to Turbo DA with Content-Type: application/octet-stream header and receive submission_id.
    5. Set da_submission_id in the BatchTreeSubmission object.
    6. Set Solana environment variables (SOLANA_RPC_URL, SOLANA_SUBMITTER_KEYPAIR_PATH, SOLANA_CLUSTER, SOLANA_CORE_MANAGER_PROGRAM_ID) from config and remove MOCK_SOLANA_TX.
    7. Submit accumulator root (32 bytes), signature (64 bytes), and recovery id (1 byte) to blockchain via BatchSubmitter::submit_batch_with_retry. The submission_id is passed to the chain client but not included in the on-chain instruction data.
  • Response (200): 
    {
  "status": "success",
  "turbo_da_submission_id": "<string>",
  "message": "Batch submitted to Turbo DA and blockchain"
}
  • Mock mode: Set AVAIL_DA_MOCK=true to skip DA submission and generate synthetic ID mock_da_<timestamp>. Blockchain submission still executes.

POST /v1/withdraw_request

  • Request body: WithdrawlRequest (from chain_client::types).
  • Processing steps:
    1. Create BatchSubmitter instance (reads Solana config from environment variables).
    2. Forward request to BatchSubmitter::submit_withdrawl_request_with_retry (up to 3 retry attempts).
    3. Return the original request payload as JSON upon success.
  • Note: This endpoint does not modify MOCK_SOLANA_TX environment variable. The batch submitter checks for MOCK_SOLANA_TX during execution to determine whether to send real transactions or return mock transaction IDs.

Data Availability Integration

Turbo DA Client

The submitter uses TurboDAClient to interact with Turbo DA for off-chain data storage.

Endpoints:

  • Submit data: POST {base_url}/v1/submit_raw_data
    • Headers: x-api-key, Content-Type: application/octet-stream
    • Body: String data converted to bytes via .into_bytes()
    • Returns: { "submission_id": "<string>" }
  • Retrieve data: GET {base_url}/v1/get_pre_image?submission_id=<id>
    • Headers: x-api-key
    • Returns: Raw bytes
  • Get submission info: GET {base_url}/v1/get_submission_info?submission_id=<id>
    • Headers: x-api-key
    • Returns: { "block_hash": "<string>", "block_number": <u64>, "tx_hash": "<string>", "state": "<string>" }

Configuration (from config.toml via turbo_da section):

  • api_key (required): Authentication key for Turbo DA API
  • app_id (required): Application identifier
  • api_url (optional): Base URL, defaults to https://staging.turbo-api.availproject.org

Error handling: Non-2xx responses return detailed errors with HTTP status code and response text.

Implementation:

// Trait definition [da_client.rs:15-25]
#[async_trait::async_trait]
pub trait DAClient: Send + Sync {
    async fn submit_data(&self, data: String) -> anyhow::Result<String>;
    async fn get_data(&self, submission_id: &str) -> anyhow::Result<Vec<u8>>;
}

// Client struct [da_client.rs:41-46]
pub struct TurboDAClient {
    client: Client,
    api_key: String,
    app_id: String,
    base_url: String,
}

// Additional method [da_client.rs:75-112]
impl TurboDAClient {
    pub async fn new() -> anyhow::Result<Self> { /* loads config */ }
    pub async fn get_submission_info(&self, submission_id: &str)
        -> anyhow::Result<TurboDASubmissionInfo> { /* ... */ }
}

// Response types [da_client.rs:28-39]
pub struct TurboDASubmissionResponse {
    pub submission_id: String,
}

pub struct TurboDASubmissionInfo {
    pub block_hash: String,
    pub block_number: u64,
    pub tx_hash: String,
    pub state: String,
}

Blockchain Submission

The submitter delegates blockchain transactions to chain_client::BatchSubmitter:

  • Batch commitment: Calls submit_batch_with_retry(signature, recovery_id, acc, Some(submission_id)) with up to 3 retry attempts.
  • Withdraw request: Calls submit_withdrawl_request_with_retry(withdrawl_request) with up to 3 retry attempts.

Implementation:

// Struct [batch_submitter.rs:29-36]
pub struct BatchSubmitter {
    client: RpcClient,
    submitter_keypair: Keypair,
    cluster: String,
    core_manager_program_id: Pubkey,
    ewrapper_program_id: Pubkey,
}

// Methods [batch_submitter.rs:38-527]
impl BatchSubmitter {
    pub fn new() -> Result<Self, Box<dyn std::error::Error>> { /* reads env vars */ }

    pub async fn submit_batch_with_retry(
        &self,
        signature: Vec<u8>,
        recovery_id: u8,
        acc: String,
        turbo_da_submission_id: Option<&str>,
    ) -> Result<(), Box<dyn std::error::Error>>;

    pub async fn submit_withdrawl_request_with_retry(
        &self,
        withdrawl_request: WithdrawlRequest,
    ) -> Result<(), Box<dyn std::error::Error>>;
}

Environment Variables

The batch submission endpoint sets these environment variables from config.toml [solana] section before creating BatchSubmitter:

  • SOLANA_RPC_URL - Solana RPC endpoint
  • SOLANA_SUBMITTER_KEYPAIR_PATH - Path to submitter keypair file
  • SOLANA_CLUSTER - Cluster name (e.g., "devnet")
  • SOLANA_CORE_MANAGER_PROGRAM_ID - Core manager program public key
  • MOCK_SOLANA_TX - Explicitly removed via env::remove_var() to ensure real transactions

The withdraw endpoint does not set environment variables; it relies on existing values.

BatchSubmitter::new() reads these additional variables:

  • SOLANA_EWORAPPER_PROGRAM_ID - Ewrapper program ID (defaults to CRx1GzdrrivpkxqasdMFXM4A5eQ9p3ktXTmrwLuhW4Rm if unset)
  • MOCK_SOLANA_TX - If present, returns mock transaction IDs instead of submitting to chain

Current Behavior

  • Retry logic: MAX_RETRIES = 3 attempts for both batch and withdraw submissions [batch_submitter.rs:26]
  • DA reference handling: The submission_id is passed to submit_batch_with_retry but marked as unused (_turbo_da_submission_id) in the instruction builder. Only accumulator root (32 bytes), signature (64 bytes), and recovery id (1 byte) are included in on-chain instruction data [batch_submitter.rs:262, 480-490]
  • Signature validation: The instruction builder validates signature length is exactly 64 bytes [batch_submitter.rs:482-488]
  • Mocking: Batch submission path removes MOCK_SOLANA_TX; withdraw path leaves it unchanged but still checks the variable during transaction building

Server

  • Framework: warp HTTP framework [main.rs:8]
  • Binding: Listens on 127.0.0.1:<port> where port is read from config.submitter.port [main.rs:27-36]
  • Routes:
    • POST /v1/submit_batch - Batch submission handler [main.rs:15-19]
    • POST /v1/withdraw_request - Withdraw request handler [main.rs:21-25]
  • Logging: Uses warp::log("submitter") middleware [main.rs:34]

Data Formats

BatchTreeSubmission

Defined in encifher_core::types [core/types.rs:1019-1025]:

pub struct BatchTreeSubmission {
    pub acc: String,                     // Accumulator root (hex string, 0x-prefixed)
    pub signature: Vec<u8>,              // TEE ECDSA signature (64 bytes)
    pub recovery_id: u8,                 // secp256k1 recovery ID
    pub batch_trees: Vec<BatchTree>,     // Merkle trees serialized to DA
    pub da_submission_id: Option<String>, // Set by submitter after DA publish
}

BatchTree

Defined in encifher_core::types [core/types.rs:1001-1004]:

pub struct BatchTree {
    pub tree: SimpleMerkleTree<Keccak256>,        // From chain_client::types
    pub request_and_cipher: Vec<RequestAndCiphertext>, // Request params + encrypted results
}

WithdrawlRequest

Defined in chain_client::types [chain_client/types.rs:25-31]:

pub struct WithdrawlRequest {
    pub amount: u128,
    pub signature: Vec<u8>,
    pub plaintext_amount: u64,
    pub mint: Vec<u8>,
    pub to_token_account: Vec<u8>,
}

Data Processing Pipeline

  1. Serialization: JSON via serde_json::to_vec() [submit_batch.rs:38]
  2. Compression: zlib using flate2::write::ZlibEncoder with Compression::default() [submit_batch.rs:41]
  3. Encoding: Base64 (standard alphabet) via base64::engine::general_purpose::STANDARD [submit_batch.rs:2, 27]
  4. Transmission: String data converted to bytes with Content-Type: application/octet-stream [da_client.rs:133-134]

Public Interfaces

HTTP API

  • POST /v1/submit_batch: Accepts JSON BatchTreeSubmission, returns submission result
  • POST /v1/withdraw_request: Accepts JSON WithdrawlRequest, returns original request on success

DA Client Interface

trait DAClient: Send + Sync {
    async fn submit_data(&self, data: String) -> anyhow::Result<String>;
    async fn get_data(&self, submission_id: &str) -> anyhow::Result<Vec<u8>>;
}

Implementation: TurboDAClient with config-based initialization and API key authentication

Chain Client Interface

  • BatchSubmitter::submit_batch_with_retry() - Batch root submission to Solana
  • BatchSubmitter::submit_withdrawl_request_with_retry() - Withdraw callback to ewrapper program
  • Configuration via environment variables (set by batch endpoint, read by BatchSubmitter)

Configuration

config.toml

[submitter]
port = 8082  # HTTP server port

[turbo_da]
api_key = "your_api_key"           # Required
app_id = "your_app_id"             # Required
api_url = "https://..."            # Optional (defaults to staging URL)

[solana]
rpc_url = "https://api.devnet.solana.com"
submitter_keypair_path = "./solana-keys/submitter-keypair.json"
cluster = "devnet"
core_manager_program_id = "FCZY1nTfhH8V8aaZpZ5YpxBb2JGcUvtxGRAcZxkUYKNr"
# Note: ewrapper_program_id is optional (has default value)

Environment Variables

Set by submitter (batch endpoint only) [submit_batch.rs:87-98]:

  • SOLANA_RPC_URL
  • SOLANA_SUBMITTER_KEYPAIR_PATH
  • SOLANA_CLUSTER
  • SOLANA_CORE_MANAGER_PROGRAM_ID

Read by BatchSubmitter [batch_submitter.rs:42-50]:

  • All of the above (required)
  • SOLANA_EWORAPPER_PROGRAM_ID (optional, defaults to CRx1GzdrrivpkxqasdMFXM4A5eQ9p3ktXTmrwLuhW4Rm)

Mock/Test modes:

  • AVAIL_DA_MOCK=true - Skip actual DA submission, return mock ID [submit_batch.rs:53]
  • MOCK_SOLANA_TX=<any> - Return mock transaction signatures instead of submitting [batch_submitter.rs:289, 375]

Error Handling

HTTP layer [submit_batch.rs:9-12, submit_decryption.rs:8]:

  • Errors wrapped in warp::reject::custom(WarpError) with descriptive messages
  • Serialization errors: "Serialization error: {e}"
  • Compression errors: "Compression error: {e}"
  • DA client errors: "DA client error: {e}" or "DA submission failed: {e}"
  • Chain client errors: "Batch submitter error: {e}" or "Blockchain submission failed: {e}"

DA client [da_client.rs:93-100, 138-145, 174-181]:

  • Non-2xx responses return detailed errors with HTTP status code and response text
  • Example: "Turbo DA submission failed (404): Not found"

Chain client [batch_submitter.rs:132-163, 324-355]:

  • Retry logic logs each failure with attempt number
  • Final error after MAX_RETRIES: "Max retries (3) reached. Last error: {e}"

Tests

Turbo DA Integration [turbo_da_integration_test.rs]:

  1. test_turbo_da_integration() - Round-trip test (submit → retrieve → verify)

    • Skipped if TURBO_DA_API_KEY environment variable not set [line 7-9]
    • Tests data integrity by comparing submitted and retrieved data

  2. test_turbo_da_client_creation() - Client initialization

    • Creates TurboDAClient using config.toml [turbo_da] section
    • Note: Client reads from config file, NOT from TURBO_DA_API_KEY env var [da_client.rs:50-55]
    • Test may fail if config file missing or incomplete

Indexer

About Indexer and how does it works

Decryption

Flow for decryption of value onchain and offchain