767 lines
32 KiB
Markdown
767 lines
32 KiB
Markdown
# Fused Engine — Development Plan
|
||
|
||
## 1. Objective
|
||
|
||
Fuse the `fh_ob` (Feed Handler + Order Book) and `oe_em` (Opportunity Engine + Emission)
|
||
Python components into a single C binary (`fused_engine`) that eliminates:
|
||
|
||
- Two-process IPC overhead (Unix socket serialization/deserialization between fh_ob → oe_em)
|
||
- Three-way JSON serialization (WS parse → socket serialize → socket deserialize → signal serialize)
|
||
- Python interpreter overhead on the hot path (object allocation, GIL, asyncio dispatch)
|
||
- GC pressure from transient `OrderBookTop5`, `BookLevel`, and `dict` allocations
|
||
|
||
The **design maxima**: from the moment an order book update arrives, triangle
|
||
re-evaluation and signal dispatch must be **immediate**. No blocking I/O on the hot path.
|
||
|
||
## 2. Current Architecture (for reference)
|
||
|
||
### 2.1 Process Layout
|
||
|
||
```
|
||
[KuCoin WS] → [fh_ob] --Unix socket (JSON book snapshots)--> [oe_em] --Unix socket (JSON signals)--> [executor]
|
||
```
|
||
|
||
Three processes, two IPC hops. The critical path is:
|
||
|
||
```
|
||
WS frame arrives at fh_ob
|
||
→ json.loads() (fh_ob/ws_client.py:239-240)
|
||
→ BookStore.update() (fh_ob/book_store.py:42-82) [allocates OrderBookTop5, BookLevel objects]
|
||
→ json.dumps(book) (fh_ob/book_store.py:86-91) [serializes to JSON]
|
||
→ write to Unix socket (fh_ob/socket_server.py:107-114)
|
||
→ oe_em reads from socket (oe_em/book_consumer.py:77-93)
|
||
→ json.loads() (oe_em/book_consumer.py:84-91) [second parse]
|
||
→ evaluate_triangles_for_pair() (oe_em/opportunity.py:152-230) [Python loop, dict lookups]
|
||
→ json.dumps(signal) (oe_em/opportunity.py:290-310) [third serialization]
|
||
→ write to executor socket (oe_em/socket_client.py:33-53)
|
||
```
|
||
|
||
### 2.2 Key Python Files and Line References
|
||
|
||
| File | Lines | Purpose |
|
||
|---|---|---|
|
||
| `fh_ob/ws_client.py` | 1-148 | KuCoin WS connection, multi-worker model, token fetch, subscribe/unsubscribe |
|
||
| `fh_ob/ws_client.py` | 149-168 | Dynamic subscription loop (REST-triggered add/remove symbols) |
|
||
| `fh_ob/ws_client.py` | 199-206 | Public token fetch via `/api/v1/bullet-public` |
|
||
| `fh_ob/ws_client.py` | 208-227 | Batch subscribe (100 symbols per message, topic `/spotMarket/level2Depth5:...`) |
|
||
| `fh_ob/ws_client.py` | 238-270 | Message handler: welcome/pong/ack/disconnect/message routing |
|
||
| `fh_ob/book_store.py` | 18-25 | `BookLevel` dataclass (price, size as `Decimal`) |
|
||
| `fh_ob/book_store.py` | 27-40 | `OrderBookTop5` dataclass (symbol, ts, sequence, bids/asks as lists) |
|
||
| `fh_ob/book_store.py` | 42-82 | `BookStore.update()` — parse WS message, allocate objects, store in dict |
|
||
| `fh_ob/book_store.py` | 84-91 | `to_json()` — serialize book snapshot for IPC |
|
||
| `fh_ob/socket_server.py` | 1-114 | Unix socket server, broadcast to all connected oe_em clients |
|
||
| `fh_ob/rest_server.py` | 1-189 | FastAPI REST: `/health`, `/book/{symbol}`, `/symbols` CRUD |
|
||
| `fh_ob/__main__.py` | 1-196 | Startup: config load, WS client init, dynamic subscription setup, REST server |
|
||
| `oe_em/book_consumer.py` | 1-93 | Unix socket client, connect to fh_ob, JSON parse, callback dispatch |
|
||
| `oe_em/opportunity.py` | 47-109 | `FeeTable` — fetch from KuCoin `/api/v1/base-fee`, apply KCS discount |
|
||
| `oe_em/opportunity.py` | 111-149 | `TriangleEnumerator` — enumerate triangles from pairs, filter by hold currencies |
|
||
| `oe_em/opportunity.py` | 152-230 | `evaluate_triangles_for_pair()` — hot path: cumulative rate, fee factor, max_volume |
|
||
| `oe_em/opportunity.py` | 232-288 | `create_signal()` — build signal dict with legs, books, metadata |
|
||
| `oe_em/opportunity.py` | 290-310 | `format_signal_json()` — serialize signal for executor |
|
||
| `oe_em/opportunity.py` | 312-330 | `check_cooldown()` — per-triangle cooldown enforcement |
|
||
| `oe_em/triangle_enum.py` | 1-203 | Standalone triangle enumeration utility (not used at runtime) |
|
||
| `oe_em/kucoin_api.py` | 1-107 | REST client for fee table, trading pairs fetch |
|
||
| `oe_em/socket_client.py` | 1-53 | Unix socket client to executor, reconnect loop |
|
||
| `oe_em/__main__.py` | 1-281 | Startup: config, fee table, triangle enum, book consumer, signal sender |
|
||
| `executor/executor.py` | 204-227 | `handle_signal()` — entry point, exception safety |
|
||
| `executor/executor.py` | 229-305 | `_handle_signal_impl()` — pause check, validation, stale rejection, slot check |
|
||
| `executor/executor.py` | 307-398 | `_precheck_volume()` — backward-propagation minimum calculation |
|
||
| `executor/executor.py` | 464-852 | `_execute_triangle()` — sequential 3-leg execution, paper/live mode |
|
||
| `executor/executor.py` | 474-483 | Timing markers: `t-2_book_snapshot`, `t-1_signal_created`, `signal_received` |
|
||
| `executor/executor.py` | 854-890 | `_emit_report()` — execution report with timing log |
|
||
| `executor/config.py` | 1-81 | Executor settings: paper mode, concurrency, isolation, initial capital |
|
||
| `executor/socket_server.py` | 1-90 | Unix socket server receiving signals from oe_em |
|
||
| `common/config.py` | 1-59 | fh_ob settings (YAML schema): symbols, WS URL, heartbeat, reconnect |
|
||
| `oe_em/config.py` | 1-84 | oe_em settings (YAML schema): threshold, cooldown, hold currencies, KCS discount |
|
||
|
||
### 2.3 KuCoin API References
|
||
|
||
| Endpoint / Feature | Doc URL | Notes |
|
||
|---|---|---|
|
||
| Public Token (Classic Spot) | https://www.kucoin.com/docs-new/websocket-api/base-info/get-public-token-spot-margin.md | POST `/api/v1/bullet-public`, returns token + `instanceServers` with `pingInterval`/`pingTimeout` |
|
||
| WS Connection | https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md | `wss://ws-api-spot.kucoin.com?token=...&connectId=...`, welcome message on connect |
|
||
| WS Heartbeat | https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md | Ping every `pingInterval` (18s), timeout after `pingTimeout` (10s). Any outgoing message resets timeout. |
|
||
| WS Subscribe | https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md | `{"type":"subscribe","topic":"/spotMarket/level2Depth5:BTC-USDT,...","response":true}` |
|
||
| WS Unsubscribe | https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md | `{"type":"unsubscribe","topic":"/spotMarket/level2Depth5:BTC-USDT","response":true}` |
|
||
| WS Message Format | https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md | `{"type":"message","subject":"matchLevel2","topic":"/spotMarket/level2Depth5:SYMBOL","data":{"time":...,"sequence":...,"bids":[...],"asks":[...]}}` |
|
||
| WS Topic Limit | https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md | **400 topics per connection**. Multiple connections needed for >400 symbols. |
|
||
| WS Token TTL | https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md | Token valid 24 hours, connection disconnected after 24h. |
|
||
| Get Symbols | https://www.kucoin.com/docs-new/rest/spot-trading/market-data/get-all-symbols.md | GET `/api/v1/symbols`, returns pair metadata (base, quote, min sizes, increments) |
|
||
| Get Fee Table | https://www.kucoin.com/docs-new/rest/account-info/trade-fee/get-basic-fee-spot-margin.md | GET `/api/v1/base-fee`, returns taker/maker fees per currency |
|
||
| Order Test | https://www.kucoin.com/docs-new/rest/spot-trading/orders/add-order-test.md | POST `/api/v1/orders/test`, validates order without placing |
|
||
| Place Order | https://www.kucoin.com/docs-new/rest/spot-trading/orders/add-order.md | POST `/api/v1/orders`, places market/limit order |
|
||
|
||
### 2.4 WS Message Format (Classic API)
|
||
|
||
```json
|
||
{
|
||
"type": "message",
|
||
"subject": "matchLevel2",
|
||
"topic": "/spotMarket/level2Depth5:BTC-USDT",
|
||
"data": {
|
||
"time": 1746789012345,
|
||
"sequence": 123456789,
|
||
"sequenceNum": 123456789,
|
||
"bids": [
|
||
["90701.1", "0.13918404"],
|
||
["90700.0", "1.00000000"],
|
||
["90699.5", "2.50000000"],
|
||
["90698.0", "0.75000000"],
|
||
["90697.5", "3.20000000"]
|
||
],
|
||
"asks": [
|
||
["90701.2", "0.57715830"],
|
||
["90702.0", "0.25000000"],
|
||
["90703.5", "1.10000000"],
|
||
["90704.0", "0.80000000"],
|
||
["90705.0", "2.00000000"]
|
||
]
|
||
}
|
||
}
|
||
```
|
||
|
||
### 2.5 Signal Format (oe_em → executor)
|
||
|
||
From `oe_em/opportunity.py:232-310`:
|
||
|
||
```json
|
||
{
|
||
"type": "signal",
|
||
"correlation_id": "abc123",
|
||
"triangle_key": ["USDT", "BTC", "ETH"],
|
||
"primary_quote": "USDT",
|
||
"legs": [
|
||
{
|
||
"pair": "BTC-USDT",
|
||
"input_currency": "USDT",
|
||
"output_currency": "BTC",
|
||
"fee_currency": "USDT",
|
||
"fee_rate": 0.001,
|
||
"exchange_rate": 0.00001102,
|
||
"side": "buy"
|
||
},
|
||
{
|
||
"pair": "ETH-BTC",
|
||
"input_currency": "BTC",
|
||
"output_currency": "ETH",
|
||
"fee_currency": "BTC",
|
||
"fee_rate": 0.001,
|
||
"exchange_rate": 16.5,
|
||
"side": "buy"
|
||
},
|
||
{
|
||
"pair": "ETH-USDT",
|
||
"input_currency": "ETH",
|
||
"output_currency": "USDT",
|
||
"fee_currency": "USDT",
|
||
"fee_rate": 0.001,
|
||
"exchange_rate": 90701.2,
|
||
"side": "sell"
|
||
}
|
||
],
|
||
"predicted_bps": 1.50,
|
||
"max_volume": "100.00",
|
||
"ts_ms": 1746789012349,
|
||
"book_ts_ms": 1746789012345,
|
||
"books": [
|
||
{
|
||
"symbol": "BTC-USDT",
|
||
"bids": [{"price": "90701.1", "size": "0.139"}, ...],
|
||
"asks": [{"price": "90701.2", "size": "0.577"}, ...],
|
||
"ts_ms": 1746789012345
|
||
},
|
||
...
|
||
]
|
||
}
|
||
```
|
||
|
||
The executor uses `books[i].asks[0].price` or `books[i].bids[0].price` for price lookup
|
||
in `_precheck_volume()` (executor/executor.py:344-348) and `_execute_triangle()` (executor/executor.py:539-545).
|
||
|
||
## 3. Target Architecture
|
||
|
||
### 3.1 High-Level Diagram
|
||
|
||
```
|
||
[KuCoin WS] ──▶ [ fused_engine (C binary) ] ──Unix socket──▶ [executor (Python)]
|
||
│ │
|
||
│ N WS connections │ triangle_index (precomputed)
|
||
│ each ≤400 topics │ book[] (fixed array)
|
||
│ single epoll loop │
|
||
└── REST API (port 8000) ──┘
|
||
```
|
||
|
||
Single binary replaces fh_ob + oe_em. Executor is **unchanged** except for reading
|
||
two new timing fields from the signal.
|
||
|
||
### 3.2 Threading Model
|
||
|
||
Two threads, separated by hot/cold concern:
|
||
|
||
```
|
||
Thread 1 (HOT — evaluation only) Thread 2 (COLD — I/O)
|
||
───────────────────────────────── ─────────────────────
|
||
epoll: epoll:
|
||
[WS fd #1, WS fd #2, ..., WS fd #N] [unix_socket_server_fd,
|
||
[timerfd for WS pings] rest_server_fd,
|
||
eventfd_wakeup]
|
||
|
||
Incoming WS frame: Drains SPSC ring buffer:
|
||
1. Decode WS frame (stack-only) → JSON write signal to executor socket
|
||
2. jsmn parse (0 alloc) → Handle REST API requests
|
||
3. Update book[sym_idx] in-place → HTTP token refresh, pair fetch
|
||
4. clock_gettime(CLOCK_MONOTONIC) → t_arrive → WS reconnect I/O
|
||
5. clock_gettime(CLOCK_MONOTONIC) → t_eval → Dynamic subscribe/unsubscribe
|
||
6. for tri in tri_index[sym_idx]:
|
||
compute net return (6 muls, 2 subs)
|
||
if profitable && cooldown_ok:
|
||
clock_gettime() → t_signal
|
||
format signal JSON into fixed buffer
|
||
push to SPSC ring buffer
|
||
log latency line to stderr
|
||
7. Return immediately (never blocks)
|
||
```
|
||
|
||
**Guarantees:**
|
||
- WS read + triangle evaluation is **never blocked** by signal dispatch or REST traffic
|
||
- Signal enters SPSC queue within microseconds of detection
|
||
- Thread 2 stalls → Thread 1 keeps processing WS messages
|
||
- SPSC push is lock-free: single atomic increment, < 100ns
|
||
|
||
### 3.3 SPSC Ring Buffer
|
||
|
||
```
|
||
Thread 1 pushes: atomic increment head → copy signal blob → atomic publish
|
||
Thread 2 drains: atomic read tail → copy signal blob → atomic increment tail
|
||
```
|
||
|
||
Bounded ring buffer of fixed-size entries (~4KB each, max 1024 entries).
|
||
`eventfd` used to wake Thread 2 from epoll when buffer transitions from empty to non-empty.
|
||
|
||
### 3.4 File Structure
|
||
|
||
```
|
||
src/
|
||
CMakeLists.txt — build configuration
|
||
main.c — startup, config parse, thread spawn, signal handling
|
||
config.c/h — YAML config parser (libyaml), reads fh_ob + oe_em sections
|
||
hash.c/h — FNV-1a string hash, symbol table (sorted array + bsearch)
|
||
http_client.c/h — raw HTTP GET/POST over TCP (token, pairs, symbols, fees)
|
||
http_server.c/h — minimal HTTP/1.1 server (REST API endpoints)
|
||
hmac.c/h — HMAC-SHA256 via OpenSSL EVP (for private endpoints if needed)
|
||
queue.c/h — lock-free SPSC ring buffer + eventfd wakeup
|
||
triangle.c/h — triangle enumeration, index builder, fee table
|
||
ws_client.c/h — WebSocket frame parser, OpenSSL BIO TLS, subscribe/unsubscribe
|
||
book.c/h — order book array (fixed-size, 0 alloc updates)
|
||
evaluate.c/h — triangle evaluation loop (HOT PATH, all inline)
|
||
signal.c/h — JSON signal formatter, pushes to SPSC queue
|
||
events.c/h — epoll loop, timerfd, eventfd, signal handling
|
||
jsmn.h — jsmn library (header-only, dropped in)
|
||
config.json.example — example config (same schema as config.yaml)
|
||
```
|
||
|
||
### 3.5 Dependencies
|
||
|
||
| Library | Purpose | Notes |
|
||
|---|---|---|
|
||
| **OpenSSL** | TLS (WS connections), HMAC-SHA256 | System-installed, `find_package(OpenSSL REQUIRED)` |
|
||
| **libyaml** | Parse `config.yaml` at startup | Startup-only, not on hot path |
|
||
| **jsmn** | JSON parsing (WS messages, token response, REST requests) | Header-only, 0 allocation, drop `jsmn.h` into `src/` |
|
||
| **libc** | epoll, timerfd, eventfd, Unix sockets, pthreads, clock_gettime | Linux 6.12+ only |
|
||
|
||
### 3.6 Config
|
||
|
||
`config.yaml` is **unchanged** — read by both the C binary and the executor.
|
||
The C binary uses libyaml to parse the `fh_ob` and `oe_em` sections at startup.
|
||
|
||
Relevant settings from `common/config.py:9-45` (fh_ob section):
|
||
- `symbols` — initial symbol list (can be empty, dynamic add via REST)
|
||
- `log_level` — logging verbosity
|
||
- `socket_path` — Unix socket path for executor (still `/tmp/fh_ob.sock` for fh_ob→oe_em internal, but the C binary writes directly to executor socket at `/tmp/executor.sock`)
|
||
- `rest_host`, `rest_port` — REST API bind address/port
|
||
- `ws_url` — KuCoin WS endpoint (from token response, not hardcoded)
|
||
- `token_url` — KuCoin public token endpoint
|
||
- `reconnect_base_delay`, `reconnect_max_delay` — exponential backoff
|
||
- `heartbeat_interval` — WS ping interval (from token response, default 18s)
|
||
|
||
Relevant settings from `oe_em/config.py:16-68` (oe_em section):
|
||
- `signal_threshold_bps` — minimum net return in bps to fire signal
|
||
- `hold_currencies` — base currencies for triangle enumeration (default `["USDT"]`)
|
||
- `excluded_currencies` — currencies to exclude from enumeration
|
||
- `kcs_discount_active` — multiply taker fees by 0.8
|
||
- `executor_socket_path` — Unix socket path for executor signals (`/tmp/executor.sock`)
|
||
- `send_signals` — whether to emit signals to executor
|
||
|
||
## 4. Data Structures
|
||
|
||
### 4.1 Symbol Table
|
||
|
||
```c
|
||
// Sorted array of symbol entries, indexed by FNV-1a hash → bsearch
|
||
typedef struct {
|
||
char name[16]; // e.g. "BTC-USDT\0"
|
||
uint16_t index; // index into book[] array
|
||
} symbol_entry_t;
|
||
|
||
typedef struct {
|
||
symbol_entry_t *entries;
|
||
uint32_t count;
|
||
} symbol_table_t;
|
||
|
||
// O(log N) lookup: hash string → bsearch in sorted array → return book index
|
||
uint16_t symbol_table_lookup(symbol_table_t *table, const char *name);
|
||
```
|
||
|
||
### 4.2 Order Book
|
||
|
||
```c
|
||
typedef struct {
|
||
uint16_t symbol_idx;
|
||
int64_t ts_ms; // KuCoin's "time" field from WS message
|
||
int64_t sequence; // sequence number for ordering
|
||
double bids[5][2]; // [level][0] = price, [level][1] = size
|
||
double asks[5][2];
|
||
uint8_t bid_count; // actual number of bid levels (0-5)
|
||
uint8_t ask_count; // actual number of ask levels (0-5)
|
||
} order_book_t;
|
||
|
||
// Fixed-size array, indexed by symbol index
|
||
// book[symbol_idx] gives O(1) access, cache-friendly
|
||
#define MAX_SYMBOLS 2048
|
||
order_book_t books[MAX_SYMBOLS];
|
||
```
|
||
|
||
### 4.3 Triangle
|
||
|
||
```c
|
||
typedef struct {
|
||
uint16_t symbol_idx[3]; // book[] indices for each leg
|
||
uint8_t use_bid[3]; // 1 = read bid (sell), 0 = read ask (buy)
|
||
double fee_factor[3]; // precomputed: (1.0 - taker_fee * kcs_discount)
|
||
uint16_t id; // unique triangle ID (for cooldown tracking)
|
||
uint8_t currency_ids[3]; // compact currency identifiers
|
||
uint8_t primary_quote_id; // index into hold_currencies[]
|
||
char symbol_names[3][16];// symbol strings for signal JSON
|
||
char base[16]; // base currency (e.g., "USDT")
|
||
char mid[16]; // mid currency (e.g., "BTC")
|
||
char quote[16]; // quote currency (e.g., "ETH")
|
||
} triangle_t;
|
||
```
|
||
|
||
### 4.4 Triangle Index
|
||
|
||
```c
|
||
// For each symbol, offset/count into the flat triangle[] array
|
||
// Triangles are sorted by their first symbol index, then second, then third
|
||
// so all triangles containing a given symbol are contiguous
|
||
typedef struct {
|
||
uint32_t offset;
|
||
uint32_t count;
|
||
} tri_index_entry_t;
|
||
|
||
tri_index_entry_t tri_index[MAX_SYMBOLS];
|
||
```
|
||
|
||
### 4.5 Signal (SPSC Queue Entry)
|
||
|
||
```c
|
||
#define SIGNAL_MAX_SIZE 4096
|
||
|
||
typedef struct {
|
||
char data[SIGNAL_MAX_SIZE]; // pre-formatted JSON
|
||
uint32_t len;
|
||
int64_t ts_ms; // signal creation timestamp
|
||
} signal_entry_t;
|
||
|
||
#define SPSC_CAPACITY 1024
|
||
signal_entry_t ring[SPSC_CAPACITY];
|
||
atomic_uint head; // writer (Thread 1)
|
||
atomic_uint tail; // reader (Thread 2)
|
||
```
|
||
|
||
### 4.6 Cooldown State
|
||
|
||
```c
|
||
typedef struct {
|
||
int64_t last_signal_ms; // timestamp of last fired signal (ms)
|
||
} cooldown_t;
|
||
|
||
cooldown_t cooldowns[MAX_TRIANGLES];
|
||
```
|
||
|
||
## 5. Hot Path (evaluate.c)
|
||
|
||
The hot path runs inline in the WS message handler on Thread 1. Zero heap
|
||
allocation. All data accessed by index into fixed arrays.
|
||
|
||
```c
|
||
// Called from ws_client.c message handler, immediately after book update
|
||
// sym_idx is the index of the symbol that just updated
|
||
void evaluate_and_signal(uint16_t sym_idx) {
|
||
// Timestamp: arrival (after WS decode + book update)
|
||
struct timespec ts_arrive;
|
||
clock_gettime(CLOCK_MONOTONIC, &ts_arrive);
|
||
int64_t t_arrive_ms = ts_arrive.tv_sec * 1000 + ts_arrive.tv_nsec / 1e6;
|
||
|
||
tri_index_entry_t *idx = &tri_index[sym_idx];
|
||
int64_t book_ts_ms = books[sym_idx].ts_ms;
|
||
|
||
for (uint32_t i = 0; i < idx->count; i++) {
|
||
triangle_t *tri = &triangles[idx->offset + i];
|
||
|
||
// Timestamp: evaluation start
|
||
struct timespec ts_eval;
|
||
clock_gettime(CLOCK_MONOTONIC, &ts_eval);
|
||
int64_t t_eval_ms = ts_eval.tv_sec * 1000 + ts_eval.tv_nsec / 1e6;
|
||
|
||
// Compute cumulative rate through all 3 legs
|
||
double cum = 1.0;
|
||
for (int leg = 0; leg < 3; leg++) {
|
||
order_book_t *b = &books[tri->symbol_idx[leg]];
|
||
double rate = tri->use_bid[leg]
|
||
? b->bids[0][0] // sell leg: use best bid
|
||
: 1.0 / b->asks[0][0]; // buy leg: use best ask (invert)
|
||
cum *= rate * tri->fee_factor[leg];
|
||
}
|
||
|
||
double net_bps = (cum - 1.0) * 10000.0;
|
||
|
||
if (net_bps > signal_threshold_bps
|
||
&& cooldown_ok(tri->id)
|
||
&& max_volume_ok(tri)) {
|
||
|
||
// Compute max_volume (chained minimum across 3 legs)
|
||
double max_vol = compute_max_volume(tri);
|
||
|
||
// Timestamp: signal creation
|
||
struct timespec ts_signal;
|
||
clock_gettime(CLOCK_MONOTONIC, &ts_signal);
|
||
int64_t t_signal_ms = ts_signal.tv_sec * 1000 + ts_signal.tv_nsec / 1e6;
|
||
|
||
// Format signal JSON into fixed buffer
|
||
char buf[SIGNAL_MAX_SIZE];
|
||
int len = format_signal(buf, sizeof(buf), tri, net_bps, max_vol,
|
||
t_arrive_ms, t_eval_ms, t_signal_ms, book_ts_ms);
|
||
|
||
// Push to SPSC queue (lock-free, never blocks)
|
||
spsc_push(buf, len, t_signal_ms);
|
||
|
||
// Log latency line (to stderr, not disk)
|
||
fprintf(stderr,
|
||
"SIGNAL corr=%s sym=%s tri=%s/%s/%s bps=%.2f "
|
||
"t_exchange=%ld t_arrive=%ld t_eval=%ld t_signal=%ld\n",
|
||
correlation_id, tri->symbol_names[0],
|
||
tri->base, tri->mid, tri->quote,
|
||
net_bps,
|
||
(long)book_ts_ms, (long)t_arrive_ms,
|
||
(long)t_eval_ms, (long)t_signal_ms);
|
||
|
||
set_cooldown(tri->id, t_signal_ms);
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**Performance characteristics:**
|
||
- Per triangle: 6 double multiplications, 2 subtractions, 3 array lookups
|
||
- No heap allocation, no function call overhead (all `static inline`)
|
||
- `clock_gettime(CLOCK_MONOTONIC)` is ~10ns on modern Linux
|
||
- Worst case for a symbol in 50 triangles: ~50 × 1µs = 50µs total eval time
|
||
|
||
## 6. Signal JSON Format
|
||
|
||
Same as current oe_em → executor format, with two new timing fields.
|
||
The executor reads `t_arrive_ms` and `t_eval_ms` (new) alongside existing
|
||
`ts_ms` and `book_ts_ms`.
|
||
|
||
```json
|
||
{
|
||
"type": "signal",
|
||
"correlation_id": "abc123",
|
||
"triangle_key": ["USDT", "BTC", "ETH"],
|
||
"primary_quote": "USDT",
|
||
"legs": [
|
||
{
|
||
"pair": "BTC-USDT",
|
||
"input_currency": "USDT",
|
||
"output_currency": "BTC",
|
||
"fee_currency": "USDT",
|
||
"fee_rate": 0.001,
|
||
"exchange_rate": 0.00001102,
|
||
"side": "buy"
|
||
},
|
||
{
|
||
"pair": "ETH-BTC",
|
||
"input_currency": "BTC",
|
||
"output_currency": "ETH",
|
||
"fee_currency": "BTC",
|
||
"fee_rate": 0.001,
|
||
"exchange_rate": 16.5,
|
||
"side": "buy"
|
||
},
|
||
{
|
||
"pair": "ETH-USDT",
|
||
"input_currency": "ETH",
|
||
"output_currency": "USDT",
|
||
"fee_currency": "USDT",
|
||
"fee_rate": 0.001,
|
||
"exchange_rate": 90701.2,
|
||
"side": "sell"
|
||
}
|
||
],
|
||
"predicted_bps": 1.50,
|
||
"max_volume": "100.00",
|
||
"book_ts_ms": 1746789012345,
|
||
"t_arrive_ms": 1746789012347,
|
||
"t_eval_ms": 1746789012347,
|
||
"ts_ms": 1746789012349,
|
||
"books": [
|
||
{
|
||
"symbol": "BTC-USDT",
|
||
"bids": [{"price": "90701.1", "size": "0.13918404"}, ...],
|
||
"asks": [{"price": "90701.2", "size": "0.57715830"}, ...],
|
||
"ts_ms": 1746789012345
|
||
},
|
||
...
|
||
]
|
||
}
|
||
```
|
||
|
||
### 6.1 Executor Timing Derivation
|
||
|
||
The executor (executor/executor.py:474-483) currently derives:
|
||
```python
|
||
timings.append({"step": "t-2_book_snapshot", "elapsed_ms": -(executor_receive_ts_ms - book_ts_ms)})
|
||
timings.append({"step": "t-1_signal_created", "elapsed_ms": -(executor_receive_ts_ms - signal_ts_ms)})
|
||
```
|
||
|
||
With the two new fields, the executor derives:
|
||
|
||
| Metric | Formula | Meaning |
|
||
|---|---|---|
|
||
| Network latency | `t_arrive_ms - book_ts_ms` | KuCoin server → our doorstep |
|
||
| Dispatch overhead | `t_eval_ms - t_arrive_ms` | Book update → eval start |
|
||
| Eval time | `ts_ms - t_eval_ms` | Triangle evaluation loop |
|
||
| Socket + queue latency | `executor_receive_ts_ms - ts_ms` | SPSC push → executor receive |
|
||
| Total end-to-end | `executor_receive_ts_ms - book_ts_ms` | Full pipeline |
|
||
|
||
**Executor change required:** In `executor/executor.py:474-483`, read the two new
|
||
fields and add derived timing entries. Approximately 5 lines of code.
|
||
|
||
## 7. WS Connection Model
|
||
|
||
### 7.1 Multi-Connection Architecture
|
||
|
||
KuCoin limits each WS connection to **400 topics** (KuCoin API docs:
|
||
https://www.kucoin.com/docs-new/websocket-api/base-info/introduction.md).
|
||
With ~100 symbols, a single connection suffices. But the design supports
|
||
dynamic symbol addition that could exceed 400.
|
||
|
||
```c
|
||
typedef struct {
|
||
int socket_fd; // epoll'd file descriptor
|
||
BIO *bio; // OpenSSL BIO for TLS
|
||
SSL *ssl; // OpenSSL SSL context
|
||
char buffer[65536]; // WS frame receive buffer
|
||
int buffer_len;
|
||
uint32_t topic_count; // current subscription count
|
||
uint32_t max_topics; // typically 400
|
||
int64_t last_ping_ms; // last ping timestamp
|
||
int64_t ping_interval_ms; // from token response
|
||
int64_t ping_timeout_ms; // from token response
|
||
char token[512]; // public WS token
|
||
char connect_id[64]; // unique connection ID
|
||
uint16_t symbol_indices[400]; // subscribed symbol indices
|
||
uint32_t symbol_count;
|
||
} ws_connection_t;
|
||
|
||
#define MAX_WS_CONNECTIONS 8
|
||
ws_connection_t ws_connections[MAX_WS_CONNECTIONS];
|
||
uint32_t ws_connection_count;
|
||
```
|
||
|
||
### 7.2 WS Frame Protocol
|
||
|
||
Classic KuCoin WS uses standard WebSocket framing over TLS. The C implementation
|
||
handles framing manually (no libwebsockets dependency):
|
||
|
||
**Outgoing:**
|
||
1. Construct WS frame: FIN=1, opcode=1 (text), masked, payload = JSON string
|
||
2. Send via `BIO_write(bio, frame, frame_len)`
|
||
|
||
**Incoming:**
|
||
1. Read from `BIO_read(bio, buf, sizeof(buf))`
|
||
2. Parse frame header (2-14 bytes): FIN, RSV, opcode, mask, payload length
|
||
3. Unmask payload (XOR with 4-byte masking key)
|
||
4. Pass to jsmn parser
|
||
|
||
Frame parsing is ~50 lines of C. See `rfc6455` Section 5.2 for frame format.
|
||
|
||
### 7.3 Subscribe/Unsubscribe
|
||
|
||
From `fh_ob/ws_client.py:208-227`: subscribe in batches of 100 symbols.
|
||
Topic format: `/spotMarket/level2Depth5:SYMBOL1,SYMBOL2,...`
|
||
|
||
```c
|
||
// Subscribe symbols to a WS connection
|
||
int ws_subscribe(ws_connection_t *conn, uint16_t *symbol_indices, uint32_t count);
|
||
|
||
// Unsubscribe symbols from a WS connection
|
||
int ws_unsubscribe(ws_connection_t *conn, uint16_t *symbol_indices, uint32_t count);
|
||
```
|
||
|
||
Dynamic subscribe/unsubscribe is triggered by REST API calls
|
||
(`POST /symbols` and `DELETE /symbols/{symbol}`).
|
||
|
||
## 8. REST API
|
||
|
||
Kept from fh_ob for operational use. Same endpoints as `fh_ob/rest_server.py:1-189`.
|
||
|
||
| Endpoint | Method | Purpose | Ref |
|
||
|---|---|---|---|
|
||
| `/health` | GET | Status: WS connected, book count, symbol count, uptime | `fh_ob/rest_server.py:35-48` |
|
||
| `/book/{symbol}` | GET | Current top-of-book for symbol (all 5 levels) | `fh_ob/rest_server.py:50-68` |
|
||
| `/books` | GET | All books | `fh_ob/rest_server.py:70-80` |
|
||
| `/symbols` | GET | List subscribed symbols | `fh_ob/rest_server.py:82-92` |
|
||
| `/symbols` | POST | Add symbol (subscribe to WS, add to triangle eval) | `fh_ob/rest_server.py:94-120` |
|
||
| `/symbols/{symbol}` | DELETE | Remove symbol (unsubscribe from WS) | `fh_ob/rest_server.py:122-145` |
|
||
|
||
HTTP/1.1 server is raw socket + simple parser. No libhttp dependency.
|
||
Handles GET/POST/DELETE with Content-Type: application/json.
|
||
Binds to `0.0.0.0:8000` (configurable via `fh_ob.rest_port`).
|
||
|
||
## 9. Triangle Enumeration
|
||
|
||
From `oe_em/opportunity.py:111-149` and `oe_em/triangle_enum.py:1-203`.
|
||
|
||
Algorithm:
|
||
1. Fetch KuCoin symbols via `GET /api/v1/symbols` (oe_em/kucoin_api.py:39-56)
|
||
2. Filter: `base != quote`, symbol is active
|
||
3. For each pair of symbols (A, B) where A.quote == B.base or A.base == B.quote:
|
||
- Find third pair (C) that completes the triangle (C.base == A.base, C.quote == B.quote, etc.)
|
||
- Filter: triangle starts and ends in a hold currency (`oe_em/config.py:55-57`, default `["USDT"]`)
|
||
- Exclude triangles containing excluded currencies (`oe_em/config.py:50-53`)
|
||
4. Build `triangle_t` array with precomputed fee factors
|
||
5. Sort triangles, build `tri_index` for O(1) lookup by symbol
|
||
|
||
Fee table from `GET /api/v1/base-fee` (oe_em/kucoin_api.py:60-78).
|
||
KCS discount: if `kcs_discount_active`, multiply taker fee by 0.8 (oe_em/opportunity.py:50-64).
|
||
|
||
## 10. Startup Sequence
|
||
|
||
```
|
||
1. Parse config.yaml (libyaml) — read fh_ob + oe_em sections
|
||
2. Fetch KuCoin symbols: HTTP GET /api/v1/symbols
|
||
3. Fetch KuCoin fee table: HTTP GET /api/v1/base-fee
|
||
4. Enumerate triangles → build sorted triangle[] + tri_index[]
|
||
5. Fetch public WS token: HTTP POST /api/v1/bullet-public
|
||
6. Create WS connections (N connections, each ≤400 topics)
|
||
7. Subscribe to initial symbols (from config.yaml fh_ob.symbols)
|
||
8. Bind Unix socket at executor_socket_path (oe_em config)
|
||
9. Start HTTP server on rest_port (fh_ob config)
|
||
10. Spawn Thread 1 (hot epoll loop: WS fds + timerfd)
|
||
11. Spawn Thread 2 (cold epoll loop: Unix socket, REST, eventfd)
|
||
12. Enter event loops
|
||
```
|
||
|
||
## 11. Executor Compatibility
|
||
|
||
### 11.1 Unix Socket Protocol
|
||
|
||
The executor's `SignalSocketServer` (executor/socket_server.py:1-90) expects:
|
||
- Unix domain socket at `/tmp/executor.sock`
|
||
- JSON messages, one per connection message
|
||
- Message format: `{"type":"signal", ...}` followed by newline
|
||
|
||
The C binary writes to the same socket path. No changes needed to the
|
||
executor's socket server.
|
||
|
||
### 11.2 Signal Payload
|
||
|
||
The signal JSON is identical to the current format, with two additional fields:
|
||
- `t_arrive_ms` — monotonic timestamp when WS frame decode completed
|
||
- `t_eval_ms` — monotonic timestamp when triangle evaluation started
|
||
|
||
The executor reads these in `executor/executor.py:474-483`. Required change:
|
||
|
||
```python
|
||
# In executor/executor.py, around line 474:
|
||
t_arrive_ms = signal.get("t_arrive_ms", 0)
|
||
t_eval_ms = signal.get("t_eval_ms", 0)
|
||
if t_arrive_ms > 0:
|
||
timings.append({"step": "t-3_ws_arrive", "elapsed_ms": -(executor_receive_ts_ms - t_arrive_ms)})
|
||
if t_eval_ms > 0:
|
||
timings.append({"step": "t-4_eval_start", "elapsed_ms": -(executor_receive_ts_ms - t_eval_ms)})
|
||
```
|
||
|
||
### 11.3 Executor Disconnect Handling
|
||
|
||
The C binary keeps the executor Unix socket open and waits for reconnection.
|
||
If the executor disconnects, Thread 2 logs a warning and keeps the server
|
||
socket listening. New executor connections are accepted normally.
|
||
(Signal queue entries for disconnected executor are dropped.)
|
||
|
||
## 12. Implementation Order
|
||
|
||
### Phase 1: Foundation (Week 1)
|
||
1. `CMakeLists.txt` — build config, OpenSSL + libyaml dependencies
|
||
2. `config.c/h` — YAML parser, reads fh_ob + oe_em sections
|
||
3. `hash.c/h` — FNV-1a hash, symbol table
|
||
4. `http_client.c/h` — raw HTTP GET/POST (token, pairs, fees)
|
||
5. `triangle.c/h` — triangle enumeration, index builder
|
||
6. `book.c/h` — order book array, update function
|
||
7. Unit tests: config parse, symbol table, triangle enum
|
||
|
||
### Phase 2: WS + Hot Path (Week 2)
|
||
8. `jsmn.h` — drop in library
|
||
9. `ws_client.c/h` — WS frame parser, OpenSSL BIO TLS, subscribe/unsubscribe
|
||
10. `evaluate.c/h` — triangle evaluation loop (inline, zero alloc)
|
||
11. `signal.c/h` — JSON signal formatter
|
||
12. Integration test: connect to KuCoin WS, verify book updates
|
||
|
||
### Phase 3: IPC + REST (Week 3)
|
||
13. `queue.c/h` — SPSC ring buffer, eventfd
|
||
14. `http_server.c/h` — REST API server
|
||
15. `events.c/h` — epoll loops, timerfd, signal handling
|
||
16. `main.c` — startup, thread spawn, signal handling
|
||
17. Integration test: full pipeline, signal to executor
|
||
|
||
### Phase 4: Polish (Week 4)
|
||
18. Dynamic subscribe/unsubscribe via REST
|
||
19. Reconnection logic (exponential backoff)
|
||
20. Timing field integration in executor
|
||
21. Performance benchmarking
|
||
22. Edge case handling, error paths
|
||
23. Deployment script update
|
||
|
||
## 13. Performance Targets
|
||
|
||
| Metric | Current (Python) | Target (C) |
|
||
|---|---|---|
|
||
| WS arrive → eval start | ~500µs (asyncio dispatch + IPC) | <1µs (inline) |
|
||
| Eval 50 triangles | ~5ms (Python loop) | <50µs (C loop) |
|
||
| Signal dispatch | ~1ms (JSON + socket write) | <10µs (SPSC push) |
|
||
| Total end-to-end (exchange → executor) | ~15-30ms | ~5-10ms (network-bound) |
|
||
|
||
The dominant latency component will be network RTT to KuCoin (~5-20ms).
|
||
The C binary aims to add <100µs of processing latency on top.
|
||
|
||
## 14. Files Affected
|
||
|
||
### New Files
|
||
- `src/` directory with all C source files (listed in §3.4)
|
||
|
||
### Modified Files
|
||
- `executor/executor.py` — add `t_arrive_ms` and `t_eval_ms` to timing derivation (~5 lines)
|
||
|
||
### Unchanged Files
|
||
- `config.yaml` — read by both C binary and executor
|
||
- `executor/` (except `executor.py` timing change) — fully compatible
|
||
- `common/log.py` — still used by executor
|
||
- `deploy.sh` — updated to deploy compiled binary instead of Python files
|
||
|
||
### Deleted Files
|
||
- `fh_ob/` — entire directory (replaced by C binary)
|
||
- `oe_em/` — entire directory (replaced by C binary)
|
||
- `common/config.py` — fh_ob settings moved to C config parser; oe_em settings merged into C config
|