32 KiB

Raw Blame History

Fused Engine — Development Plan

1. Objective

Fuse the fh_ob (Feed Handler + Order Book) and oe_em (Opportunity Engine + Emission) Python components into a single C binary (fused_engine) that eliminates:

Two-process IPC overhead (Unix socket serialization/deserialization between fh_ob → oe_em)
Three-way JSON serialization (WS parse → socket serialize → socket deserialize → signal serialize)
Python interpreter overhead on the hot path (object allocation, GIL, asyncio dispatch)
GC pressure from transient OrderBookTop5, BookLevel, and dict allocations

The design maxima: from the moment an order book update arrives, triangle re-evaluation and signal dispatch must be immediate. No blocking I/O on the hot path.

2. Current Architecture (for reference)

2.1 Process Layout

[KuCoin WS] → [fh_ob] --Unix socket (JSON book snapshots)--> [oe_em] --Unix socket (JSON signals)--> [executor]

Three processes, two IPC hops. The critical path is:

WS frame arrives at fh_ob
  → json.loads()                    (fh_ob/ws_client.py:239-240)
  → BookStore.update()              (fh_ob/book_store.py:42-82)  [allocates OrderBookTop5, BookLevel objects]
  → json.dumps(book)                (fh_ob/book_store.py:86-91)  [serializes to JSON]
  → write to Unix socket            (fh_ob/socket_server.py:107-114)
  → oe_em reads from socket         (oe_em/book_consumer.py:77-93)
  → json.loads()                    (oe_em/book_consumer.py:84-91)  [second parse]
  → evaluate_triangles_for_pair()   (oe_em/opportunity.py:152-230)  [Python loop, dict lookups]
  → json.dumps(signal)              (oe_em/opportunity.py:290-310)  [third serialization]
  → write to executor socket        (oe_em/socket_client.py:33-53)

2.2 Key Python Files and Line References

File	Lines	Purpose
`fh_ob/ws_client.py`	1-148	KuCoin WS connection, multi-worker model, token fetch, subscribe/unsubscribe
`fh_ob/ws_client.py`	149-168	Dynamic subscription loop (REST-triggered add/remove symbols)
`fh_ob/ws_client.py`	199-206	Public token fetch via `/api/v1/bullet-public`
`fh_ob/ws_client.py`	208-227	Batch subscribe (100 symbols per message, topic `/spotMarket/level2Depth5:...`)
`fh_ob/ws_client.py`	238-270	Message handler: welcome/pong/ack/disconnect/message routing
`fh_ob/book_store.py`	18-25	`BookLevel` dataclass (price, size as `Decimal`)
`fh_ob/book_store.py`	27-40	`OrderBookTop5` dataclass (symbol, ts, sequence, bids/asks as lists)
`fh_ob/book_store.py`	42-82	`BookStore.update()` — parse WS message, allocate objects, store in dict
`fh_ob/book_store.py`	84-91	`to_json()` — serialize book snapshot for IPC
`fh_ob/socket_server.py`	1-114	Unix socket server, broadcast to all connected oe_em clients
`fh_ob/rest_server.py`	1-189	FastAPI REST: `/health`, `/book/{symbol}`, `/symbols` CRUD
`fh_ob/__main__.py`	1-196	Startup: config load, WS client init, dynamic subscription setup, REST server
`oe_em/book_consumer.py`	1-93	Unix socket client, connect to fh_ob, JSON parse, callback dispatch
`oe_em/opportunity.py`	47-109	`FeeTable` — fetch from KuCoin `/api/v1/base-fee`, apply KCS discount
`oe_em/opportunity.py`	111-149	`TriangleEnumerator` — enumerate triangles from pairs, filter by hold currencies
`oe_em/opportunity.py`	152-230	`evaluate_triangles_for_pair()` — hot path: cumulative rate, fee factor, max_volume
`oe_em/opportunity.py`	232-288	`create_signal()` — build signal dict with legs, books, metadata
`oe_em/opportunity.py`	290-310	`format_signal_json()` — serialize signal for executor
`oe_em/opportunity.py`	312-330	`check_cooldown()` — per-triangle cooldown enforcement
`oe_em/triangle_enum.py`	1-203	Standalone triangle enumeration utility (not used at runtime)
`oe_em/kucoin_api.py`	1-107	REST client for fee table, trading pairs fetch
`oe_em/socket_client.py`	1-53	Unix socket client to executor, reconnect loop
`oe_em/__main__.py`	1-281	Startup: config, fee table, triangle enum, book consumer, signal sender
`executor/executor.py`	204-227	`handle_signal()` — entry point, exception safety
`executor/executor.py`	229-305	`_handle_signal_impl()` — pause check, validation, stale rejection, slot check
`executor/executor.py`	307-398	`_precheck_volume()` — backward-propagation minimum calculation
`executor/executor.py`	464-852	`_execute_triangle()` — sequential 3-leg execution, paper/live mode
`executor/executor.py`	474-483	Timing markers: `t-2_book_snapshot`, `t-1_signal_created`, `signal_received`
`executor/executor.py`	854-890	`_emit_report()` — execution report with timing log
`executor/config.py`	1-81	Executor settings: paper mode, concurrency, isolation, initial capital
`executor/socket_server.py`	1-90	Unix socket server receiving signals from oe_em
`common/config.py`	1-59	fh_ob settings (YAML schema): symbols, WS URL, heartbeat, reconnect
`oe_em/config.py`	1-84	oe_em settings (YAML schema): threshold, cooldown, hold currencies, KCS discount

2.3 KuCoin API References

Endpoint / Feature	Doc URL	Notes
Public Token (Classic Spot)	https://www.kucoin.com/docs-new/websocket-api/base-info/get-public-token-spot-margin.md	POST `/api/v1/bullet-public`, returns token + `instanceServers` with `pingInterval`/`pingTimeout`
WS Connection	https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md	`wss://ws-api-spot.kucoin.com?token=...&connectId=...`, welcome message on connect
WS Heartbeat	https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md	Ping every `pingInterval` (18s), timeout after `pingTimeout` (10s). Any outgoing message resets timeout.
WS Subscribe	https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md	`{"type":"subscribe","topic":"/spotMarket/level2Depth5:BTC-USDT,...","response":true}`
WS Unsubscribe	https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md	`{"type":"unsubscribe","topic":"/spotMarket/level2Depth5:BTC-USDT","response":true}`
WS Message Format	https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md	`{"type":"message","subject":"matchLevel2","topic":"/spotMarket/level2Depth5:SYMBOL","data":{"time":...,"sequence":...,"bids":[...],"asks":[...]}}`
WS Topic Limit	https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md	400 topics per connection. Multiple connections needed for >400 symbols.
WS Token TTL	https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md	Token valid 24 hours, connection disconnected after 24h.
Get Symbols	https://www.kucoin.com/docs-new/rest/spot-trading/market-data/get-all-symbols.md	GET `/api/v1/symbols`, returns pair metadata (base, quote, min sizes, increments)
Get Fee Table	https://www.kucoin.com/docs-new/rest/account-info/trade-fee/get-basic-fee-spot-margin.md	GET `/api/v1/base-fee`, returns taker/maker fees per currency
Order Test	https://www.kucoin.com/docs-new/rest/spot-trading/orders/add-order-test.md	POST `/api/v1/orders/test`, validates order without placing
Place Order	https://www.kucoin.com/docs-new/rest/spot-trading/orders/add-order.md	POST `/api/v1/orders`, places market/limit order

2.4 WS Message Format (Classic API)

{
  "type": "message",
  "subject": "matchLevel2",
  "topic": "/spotMarket/level2Depth5:BTC-USDT",
  "data": {
    "time": 1746789012345,
    "sequence": 123456789,
    "sequenceNum": 123456789,
    "bids": [
      ["90701.1", "0.13918404"],
      ["90700.0", "1.00000000"],
      ["90699.5", "2.50000000"],
      ["90698.0", "0.75000000"],
      ["90697.5", "3.20000000"]
    ],
    "asks": [
      ["90701.2", "0.57715830"],
      ["90702.0", "0.25000000"],
      ["90703.5", "1.10000000"],
      ["90704.0", "0.80000000"],
      ["90705.0", "2.00000000"]
    ]
  }
}

2.5 Signal Format (oe_em → executor)

From oe_em/opportunity.py:232-310:

{
  "type": "signal",
  "correlation_id": "abc123",
  "triangle_key": ["USDT", "BTC", "ETH"],
  "primary_quote": "USDT",
  "legs": [
    {
      "pair": "BTC-USDT",
      "input_currency": "USDT",
      "output_currency": "BTC",
      "fee_currency": "USDT",
      "fee_rate": 0.001,
      "exchange_rate": 0.00001102,
      "side": "buy"
    },
    {
      "pair": "ETH-BTC",
      "input_currency": "BTC",
      "output_currency": "ETH",
      "fee_currency": "BTC",
      "fee_rate": 0.001,
      "exchange_rate": 16.5,
      "side": "buy"
    },
    {
      "pair": "ETH-USDT",
      "input_currency": "ETH",
      "output_currency": "USDT",
      "fee_currency": "USDT",
      "fee_rate": 0.001,
      "exchange_rate": 90701.2,
      "side": "sell"
    }
  ],
  "predicted_bps": 1.50,
  "max_volume": "100.00",
  "ts_ms": 1746789012349,
  "book_ts_ms": 1746789012345,
  "books": [
    {
      "symbol": "BTC-USDT",
      "bids": [{"price": "90701.1", "size": "0.139"}, ...],
      "asks": [{"price": "90701.2", "size": "0.577"}, ...],
      "ts_ms": 1746789012345
    },
    ...
  ]
}

The executor uses books[i].asks[0].price or books[i].bids[0].price for price lookup in _precheck_volume() (executor/executor.py:344-348) and _execute_triangle() (executor/executor.py:539-545).

3. Target Architecture

3.1 High-Level Diagram

[KuCoin WS] ──▶  [  fused_engine (C binary)  ]  ──Unix socket──▶  [executor (Python)]
                    │                     │
                    │  N WS connections   │  triangle_index (precomputed)
                    │  each ≤400 topics   │  book[] (fixed array)
                    │  single epoll loop  │
                    └── REST API (port 8000) ──┘

Single binary replaces fh_ob + oe_em. Executor is unchanged except for reading two new timing fields from the signal.

3.2 Threading Model

Two threads, separated by hot/cold concern:

Thread 1 (HOT — evaluation only)              Thread 2 (COLD — I/O)
─────────────────────────────────             ─────────────────────
epoll:                                         epoll:
  [WS fd #1, WS fd #2, ..., WS fd #N]          [unix_socket_server_fd,
  [timerfd for WS pings]                        rest_server_fd,
                                                eventfd_wakeup]

Incoming WS frame:                              Drains SPSC ring buffer:
  1. Decode WS frame (stack-only)                → JSON write signal to executor socket
  2. jsmn parse (0 alloc)                       → Handle REST API requests
  3. Update book[sym_idx] in-place              → HTTP token refresh, pair fetch
  4. clock_gettime(CLOCK_MONOTONIC) → t_arrive  → WS reconnect I/O
  5. clock_gettime(CLOCK_MONOTONIC) → t_eval    → Dynamic subscribe/unsubscribe
  6. for tri in tri_index[sym_idx]:
       compute net return (6 muls, 2 subs)
       if profitable && cooldown_ok:
         clock_gettime() → t_signal
         format signal JSON into fixed buffer
         push to SPSC ring buffer
         log latency line to stderr
  7. Return immediately (never blocks)

Guarantees:

WS read + triangle evaluation is never blocked by signal dispatch or REST traffic
Signal enters SPSC queue within microseconds of detection
Thread 2 stalls → Thread 1 keeps processing WS messages
SPSC push is lock-free: single atomic increment, < 100ns

3.3 SPSC Ring Buffer

Thread 1 pushes:  atomic increment head → copy signal blob → atomic publish
Thread 2 drains:  atomic read tail → copy signal blob → atomic increment tail

Bounded ring buffer of fixed-size entries (~4KB each, max 1024 entries). eventfd used to wake Thread 2 from epoll when buffer transitions from empty to non-empty.

3.4 File Structure

src/
  CMakeLists.txt          — build configuration
  main.c                  — startup, config parse, thread spawn, signal handling
  config.c/h              — YAML config parser (libyaml), reads fh_ob + oe_em sections
  hash.c/h                — FNV-1a string hash, symbol table (sorted array + bsearch)
  http_client.c/h         — raw HTTP GET/POST over TCP (token, pairs, symbols, fees)
  http_server.c/h         — minimal HTTP/1.1 server (REST API endpoints)
  hmac.c/h                — HMAC-SHA256 via OpenSSL EVP (for private endpoints if needed)
  queue.c/h               — lock-free SPSC ring buffer + eventfd wakeup
  triangle.c/h            — triangle enumeration, index builder, fee table
  ws_client.c/h           — WebSocket frame parser, OpenSSL BIO TLS, subscribe/unsubscribe
  book.c/h                — order book array (fixed-size, 0 alloc updates)
  evaluate.c/h            — triangle evaluation loop (HOT PATH, all inline)
  signal.c/h              — JSON signal formatter, pushes to SPSC queue
  events.c/h              — epoll loop, timerfd, eventfd, signal handling
  jsmn.h                  — jsmn library (header-only, dropped in)
config.json.example       — example config (same schema as config.yaml)

3.5 Dependencies

Library	Purpose	Notes
OpenSSL	TLS (WS connections), HMAC-SHA256	System-installed, `find_package(OpenSSL REQUIRED)`
libyaml	Parse `config.yaml` at startup	Startup-only, not on hot path
jsmn	JSON parsing (WS messages, token response, REST requests)	Header-only, 0 allocation, drop `jsmn.h` into `src/`
libc	epoll, timerfd, eventfd, Unix sockets, pthreads, clock_gettime	Linux 6.12+ only

3.6 Config

config.yaml is unchanged — read by both the C binary and the executor. The C binary uses libyaml to parse the fh_ob and oe_em sections at startup.

Relevant settings from common/config.py:9-45 (fh_ob section):

symbols — initial symbol list (can be empty, dynamic add via REST)
log_level — logging verbosity
socket_path — Unix socket path for executor (still /tmp/fh_ob.sock for fh_ob→oe_em internal, but the C binary writes directly to executor socket at /tmp/executor.sock)
rest_host, rest_port — REST API bind address/port
ws_url — KuCoin WS endpoint (from token response, not hardcoded)
token_url — KuCoin public token endpoint
reconnect_base_delay, reconnect_max_delay — exponential backoff
heartbeat_interval — WS ping interval (from token response, default 18s)

Relevant settings from oe_em/config.py:16-68 (oe_em section):

signal_threshold_bps — minimum net return in bps to fire signal
hold_currencies — base currencies for triangle enumeration (default ["USDT"])
excluded_currencies — currencies to exclude from enumeration
kcs_discount_active — multiply taker fees by 0.8
executor_socket_path — Unix socket path for executor signals (/tmp/executor.sock)
send_signals — whether to emit signals to executor

4. Data Structures

4.1 Symbol Table

// Sorted array of symbol entries, indexed by FNV-1a hash → bsearch
typedef struct {
    char name[16];        // e.g. "BTC-USDT\0"
    uint16_t index;       // index into book[] array
} symbol_entry_t;

typedef struct {
    symbol_entry_t *entries;
    uint32_t count;
} symbol_table_t;

// O(log N) lookup: hash string → bsearch in sorted array → return book index
uint16_t symbol_table_lookup(symbol_table_t *table, const char *name);

4.2 Order Book

typedef struct {
    uint16_t symbol_idx;
    int64_t  ts_ms;       // KuCoin's "time" field from WS message
    int64_t  sequence;    // sequence number for ordering
    double   bids[5][2];  // [level][0] = price, [level][1] = size
    double   asks[5][2];
    uint8_t  bid_count;   // actual number of bid levels (0-5)
    uint8_t  ask_count;   // actual number of ask levels (0-5)
} order_book_t;

// Fixed-size array, indexed by symbol index
// book[symbol_idx] gives O(1) access, cache-friendly
#define MAX_SYMBOLS 2048
order_book_t books[MAX_SYMBOLS];

4.3 Triangle

typedef struct {
    uint16_t symbol_idx[3];      // book[] indices for each leg
    uint8_t  use_bid[3];         // 1 = read bid (sell), 0 = read ask (buy)
    double   fee_factor[3];      // precomputed: (1.0 - taker_fee * kcs_discount)
    uint16_t id;                 // unique triangle ID (for cooldown tracking)
    uint8_t  currency_ids[3];    // compact currency identifiers
    uint8_t  primary_quote_id;   // index into hold_currencies[]
    char     symbol_names[3][16];// symbol strings for signal JSON
    char     base[16];           // base currency (e.g., "USDT")
    char     mid[16];            // mid currency (e.g., "BTC")
    char     quote[16];          // quote currency (e.g., "ETH")
} triangle_t;

4.4 Triangle Index

// For each symbol, offset/count into the flat triangle[] array
// Triangles are sorted by their first symbol index, then second, then third
// so all triangles containing a given symbol are contiguous
typedef struct {
    uint32_t offset;
    uint32_t count;
} tri_index_entry_t;

tri_index_entry_t tri_index[MAX_SYMBOLS];

4.5 Signal (SPSC Queue Entry)

#define SIGNAL_MAX_SIZE 4096

typedef struct {
    char data[SIGNAL_MAX_SIZE];  // pre-formatted JSON
    uint32_t len;
    int64_t ts_ms;               // signal creation timestamp
} signal_entry_t;

#define SPSC_CAPACITY 1024
signal_entry_t ring[SPSC_CAPACITY];
atomic_uint head;  // writer (Thread 1)
atomic_uint tail;  // reader (Thread 2)

4.6 Cooldown State

typedef struct {
    int64_t last_signal_ms;  // timestamp of last fired signal (ms)
} cooldown_t;

cooldown_t cooldowns[MAX_TRIANGLES];

5. Hot Path (evaluate.c)

The hot path runs inline in the WS message handler on Thread 1. Zero heap allocation. All data accessed by index into fixed arrays.

// Called from ws_client.c message handler, immediately after book update
// sym_idx is the index of the symbol that just updated
void evaluate_and_signal(uint16_t sym_idx) {
    // Timestamp: arrival (after WS decode + book update)
    struct timespec ts_arrive;
    clock_gettime(CLOCK_MONOTONIC, &ts_arrive);
    int64_t t_arrive_ms = ts_arrive.tv_sec * 1000 + ts_arrive.tv_nsec / 1e6;

    tri_index_entry_t *idx = &tri_index[sym_idx];
    int64_t book_ts_ms = books[sym_idx].ts_ms;

    for (uint32_t i = 0; i < idx->count; i++) {
        triangle_t *tri = &triangles[idx->offset + i];

        // Timestamp: evaluation start
        struct timespec ts_eval;
        clock_gettime(CLOCK_MONOTONIC, &ts_eval);
        int64_t t_eval_ms = ts_eval.tv_sec * 1000 + ts_eval.tv_nsec / 1e6;

        // Compute cumulative rate through all 3 legs
        double cum = 1.0;
        for (int leg = 0; leg < 3; leg++) {
            order_book_t *b = &books[tri->symbol_idx[leg]];
            double rate = tri->use_bid[leg]
                ? b->bids[0][0]      // sell leg: use best bid
                : 1.0 / b->asks[0][0]; // buy leg: use best ask (invert)
            cum *= rate * tri->fee_factor[leg];
        }

        double net_bps = (cum - 1.0) * 10000.0;

        if (net_bps > signal_threshold_bps
            && cooldown_ok(tri->id)
            && max_volume_ok(tri)) {

            // Compute max_volume (chained minimum across 3 legs)
            double max_vol = compute_max_volume(tri);

            // Timestamp: signal creation
            struct timespec ts_signal;
            clock_gettime(CLOCK_MONOTONIC, &ts_signal);
            int64_t t_signal_ms = ts_signal.tv_sec * 1000 + ts_signal.tv_nsec / 1e6;

            // Format signal JSON into fixed buffer
            char buf[SIGNAL_MAX_SIZE];
            int len = format_signal(buf, sizeof(buf), tri, net_bps, max_vol,
                                    t_arrive_ms, t_eval_ms, t_signal_ms, book_ts_ms);

            // Push to SPSC queue (lock-free, never blocks)
            spsc_push(buf, len, t_signal_ms);

            // Log latency line (to stderr, not disk)
            fprintf(stderr,
                "SIGNAL corr=%s sym=%s tri=%s/%s/%s bps=%.2f "
                "t_exchange=%ld t_arrive=%ld t_eval=%ld t_signal=%ld\n",
                correlation_id, tri->symbol_names[0],
                tri->base, tri->mid, tri->quote,
                net_bps,
                (long)book_ts_ms, (long)t_arrive_ms,
                (long)t_eval_ms, (long)t_signal_ms);

            set_cooldown(tri->id, t_signal_ms);
        }
    }
}

Performance characteristics:

Per triangle: 6 double multiplications, 2 subtractions, 3 array lookups
No heap allocation, no function call overhead (all static inline)
clock_gettime(CLOCK_MONOTONIC) is ~10ns on modern Linux
Worst case for a symbol in 50 triangles: ~50 × 1µs = 50µs total eval time

6. Signal JSON Format

Same as current oe_em → executor format, with two new timing fields. The executor reads t_arrive_ms and t_eval_ms (new) alongside existing ts_ms and book_ts_ms.

{
  "type": "signal",
  "correlation_id": "abc123",
  "triangle_key": ["USDT", "BTC", "ETH"],
  "primary_quote": "USDT",
  "legs": [
    {
      "pair": "BTC-USDT",
      "input_currency": "USDT",
      "output_currency": "BTC",
      "fee_currency": "USDT",
      "fee_rate": 0.001,
      "exchange_rate": 0.00001102,
      "side": "buy"
    },
    {
      "pair": "ETH-BTC",
      "input_currency": "BTC",
      "output_currency": "ETH",
      "fee_currency": "BTC",
      "fee_rate": 0.001,
      "exchange_rate": 16.5,
      "side": "buy"
    },
    {
      "pair": "ETH-USDT",
      "input_currency": "ETH",
      "output_currency": "USDT",
      "fee_currency": "USDT",
      "fee_rate": 0.001,
      "exchange_rate": 90701.2,
      "side": "sell"
    }
  ],
  "predicted_bps": 1.50,
  "max_volume": "100.00",
  "book_ts_ms": 1746789012345,
  "t_arrive_ms": 1746789012347,
  "t_eval_ms": 1746789012347,
  "ts_ms": 1746789012349,
  "books": [
    {
      "symbol": "BTC-USDT",
      "bids": [{"price": "90701.1", "size": "0.13918404"}, ...],
      "asks": [{"price": "90701.2", "size": "0.57715830"}, ...],
      "ts_ms": 1746789012345
    },
    ...
  ]
}

6.1 Executor Timing Derivation

The executor (executor/executor.py:474-483) currently derives:

timings.append({"step": "t-2_book_snapshot", "elapsed_ms": -(executor_receive_ts_ms - book_ts_ms)})
timings.append({"step": "t-1_signal_created", "elapsed_ms": -(executor_receive_ts_ms - signal_ts_ms)})

With the two new fields, the executor derives:

Metric	Formula	Meaning
Network latency	`t_arrive_ms - book_ts_ms`	KuCoin server → our doorstep
Dispatch overhead	`t_eval_ms - t_arrive_ms`	Book update → eval start
Eval time	`ts_ms - t_eval_ms`	Triangle evaluation loop
Socket + queue latency	`executor_receive_ts_ms - ts_ms`	SPSC push → executor receive
Total end-to-end	`executor_receive_ts_ms - book_ts_ms`	Full pipeline

Executor change required: In executor/executor.py:474-483, read the two new fields and add derived timing entries. Approximately 5 lines of code.

7. WS Connection Model

7.1 Multi-Connection Architecture

KuCoin limits each WS connection to 400 topics (KuCoin API docs: https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md). With ~100 symbols, a single connection suffices. But the design supports dynamic symbol addition that could exceed 400.

typedef struct {
    int socket_fd;              // epoll'd file descriptor
    BIO *bio;                   // OpenSSL BIO for TLS
    SSL *ssl;                   // OpenSSL SSL context
    char buffer[65536];         // WS frame receive buffer
    int buffer_len;
    uint32_t topic_count;       // current subscription count
    uint32_t max_topics;        // typically 400
    int64_t last_ping_ms;       // last ping timestamp
    int64_t ping_interval_ms;   // from token response
    int64_t ping_timeout_ms;    // from token response
    char token[512];            // public WS token
    char connect_id[64];        // unique connection ID
    uint16_t symbol_indices[400]; // subscribed symbol indices
    uint32_t symbol_count;
} ws_connection_t;

#define MAX_WS_CONNECTIONS 8
ws_connection_t ws_connections[MAX_WS_CONNECTIONS];
uint32_t ws_connection_count;

7.2 WS Frame Protocol

Classic KuCoin WS uses standard WebSocket framing over TLS. The C implementation handles framing manually (no libwebsockets dependency):

Outgoing:

Construct WS frame: FIN=1, opcode=1 (text), masked, payload = JSON string
Send via BIO_write(bio, frame, frame_len)

Incoming:

Read from BIO_read(bio, buf, sizeof(buf))
Parse frame header (2-14 bytes): FIN, RSV, opcode, mask, payload length
Unmask payload (XOR with 4-byte masking key)
Pass to jsmn parser

Frame parsing is ~50 lines of C. See rfc6455 Section 5.2 for frame format.

7.3 Subscribe/Unsubscribe

From fh_ob/ws_client.py:208-227: subscribe in batches of 100 symbols. Topic format: /spotMarket/level2Depth5:SYMBOL1,SYMBOL2,...

// Subscribe symbols to a WS connection
int ws_subscribe(ws_connection_t *conn, uint16_t *symbol_indices, uint32_t count);

// Unsubscribe symbols from a WS connection
int ws_unsubscribe(ws_connection_t *conn, uint16_t *symbol_indices, uint32_t count);

Dynamic subscribe/unsubscribe is triggered by REST API calls (POST /symbols and DELETE /symbols/{symbol}).

8. REST API

Kept from fh_ob for operational use. Same endpoints as fh_ob/rest_server.py:1-189.

Endpoint	Method	Purpose	Ref
`/health`	GET	Status: WS connected, book count, symbol count, uptime	`fh_ob/rest_server.py:35-48`
`/book/{symbol}`	GET	Current top-of-book for symbol (all 5 levels)	`fh_ob/rest_server.py:50-68`
`/books`	GET	All books	`fh_ob/rest_server.py:70-80`
`/symbols`	GET	List subscribed symbols	`fh_ob/rest_server.py:82-92`
`/symbols`	POST	Add symbol (subscribe to WS, add to triangle eval)	`fh_ob/rest_server.py:94-120`
`/symbols/{symbol}`	DELETE	Remove symbol (unsubscribe from WS)	`fh_ob/rest_server.py:122-145`

HTTP/1.1 server is raw socket + simple parser. No libhttp dependency. Handles GET/POST/DELETE with Content-Type: application/json. Binds to 0.0.0.0:8000 (configurable via fh_ob.rest_port).

9. Triangle Enumeration

From oe_em/opportunity.py:111-149 and oe_em/triangle_enum.py:1-203.

Algorithm:

Fetch KuCoin symbols via GET /api/v1/symbols (oe_em/kucoin_api.py:39-56)
Filter: base != quote, symbol is active
For each pair of symbols (A, B) where A.quote == B.base or A.base == B.quote:
- Find third pair (C) that completes the triangle (C.base == A.base, C.quote == B.quote, etc.)
- Filter: triangle starts and ends in a hold currency (oe_em/config.py:55-57, default ["USDT"])
- Exclude triangles containing excluded currencies (oe_em/config.py:50-53)
Build triangle_t array with precomputed fee factors
Sort triangles, build tri_index for O(1) lookup by symbol

Fee table from GET /api/v1/base-fee (oe_em/kucoin_api.py:60-78). KCS discount: if kcs_discount_active, multiply taker fee by 0.8 (oe_em/opportunity.py:50-64).

10. Startup Sequence

1. Parse config.yaml (libyaml) — read fh_ob + oe_em sections
2. Fetch KuCoin symbols: HTTP GET /api/v1/symbols
3. Fetch KuCoin fee table: HTTP GET /api/v1/base-fee
4. Enumerate triangles → build sorted triangle[] + tri_index[]
5. Fetch public WS token: HTTP POST /api/v1/bullet-public
6. Create WS connections (N connections, each ≤400 topics)
7. Subscribe to initial symbols (from config.yaml fh_ob.symbols)
8. Bind Unix socket at executor_socket_path (oe_em config)
9. Start HTTP server on rest_port (fh_ob config)
10. Spawn Thread 1 (hot epoll loop: WS fds + timerfd)
11. Spawn Thread 2 (cold epoll loop: Unix socket, REST, eventfd)
12. Enter event loops

11. Executor Compatibility

11.1 Unix Socket Protocol

The executor's SignalSocketServer (executor/socket_server.py:1-90) expects:

Unix domain socket at /tmp/executor.sock
JSON messages, one per connection message
Message format: {"type":"signal", ...} followed by newline

The C binary writes to the same socket path. No changes needed to the executor's socket server.

11.2 Signal Payload

The signal JSON is identical to the current format, with two additional fields:

t_arrive_ms — monotonic timestamp when WS frame decode completed
t_eval_ms — monotonic timestamp when triangle evaluation started

The executor reads these in executor/executor.py:474-483. Required change:

# In executor/executor.py, around line 474:
t_arrive_ms = signal.get("t_arrive_ms", 0)
t_eval_ms = signal.get("t_eval_ms", 0)
if t_arrive_ms > 0:
    timings.append({"step": "t-3_ws_arrive", "elapsed_ms": -(executor_receive_ts_ms - t_arrive_ms)})
if t_eval_ms > 0:
    timings.append({"step": "t-4_eval_start", "elapsed_ms": -(executor_receive_ts_ms - t_eval_ms)})

11.3 Executor Disconnect Handling

The C binary keeps the executor Unix socket open and waits for reconnection. If the executor disconnects, Thread 2 logs a warning and keeps the server socket listening. New executor connections are accepted normally. (Signal queue entries for disconnected executor are dropped.)

12. Implementation Order

Phase 1: Foundation (Week 1)

CMakeLists.txt — build config, OpenSSL + libyaml dependencies
config.c/h — YAML parser, reads fh_ob + oe_em sections
hash.c/h — FNV-1a hash, symbol table
http_client.c/h — raw HTTP GET/POST (token, pairs, fees)
triangle.c/h — triangle enumeration, index builder
book.c/h — order book array, update function
Unit tests: config parse, symbol table, triangle enum

Phase 2: WS + Hot Path (Week 2)

jsmn.h — drop in library
ws_client.c/h — WS frame parser, OpenSSL BIO TLS, subscribe/unsubscribe
evaluate.c/h — triangle evaluation loop (inline, zero alloc)
signal.c/h — JSON signal formatter
Integration test: connect to KuCoin WS, verify book updates

Phase 3: IPC + REST (Week 3)

queue.c/h — SPSC ring buffer, eventfd
http_server.c/h — REST API server
events.c/h — epoll loops, timerfd, signal handling
main.c — startup, thread spawn, signal handling
Integration test: full pipeline, signal to executor

Phase 4: Polish (Week 4)

Dynamic subscribe/unsubscribe via REST
Reconnection logic (exponential backoff)
Timing field integration in executor
Performance benchmarking
Edge case handling, error paths
Deployment script update

13. Performance Targets

Metric	Current (Python)	Target (C)
WS arrive → eval start	~500µs (asyncio dispatch + IPC)	<1µs (inline)
Eval 50 triangles	~5ms (Python loop)	<50µs (C loop)
Signal dispatch	~1ms (JSON + socket write)	<10µs (SPSC push)
Total end-to-end (exchange → executor)	~15-30ms	~5-10ms (network-bound)

The dominant latency component will be network RTT to KuCoin (~5-20ms). The C binary aims to add <100µs of processing latency on top.

14. Files Affected

New Files

src/ directory with all C source files (listed in §3.4)

Modified Files

executor/executor.py — add t_arrive_ms and t_eval_ms to timing derivation (~5 lines)

Unchanged Files

config.yaml — read by both C binary and executor
executor/ (except executor.py timing change) — fully compatible
common/log.py — still used by executor
deploy.sh — updated to deploy compiled binary instead of Python files

Deleted Files

fh_ob/ — entire directory (replaced by C binary)
oe_em/ — entire directory (replaced by C binary)
common/config.py — fh_ob settings moved to C config parser; oe_em settings merged into C config

32 KiB Raw Blame History Unescape Escape