nicolas
/
ik_llama_opt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(rpc): update ggml_backend_cuda_init to 3-arg signature (#2084) 

Commit 75a5f6d0 (Per model CUDA contexts) added a third parameter
"model" to ggml_backend_cuda_init, but rpc-server was missed.

The RPC server creates backends at startup before any model is
loaded, so passing nullptr for the model parameter is correct: the
RPC server acts as a pure compute proxy and does not host models
locally. The nullptr serves as the context grouping key in
all_ctx, which is sufficient since all RPC backends in the same
process share the same group for multi-GPU reduce operations.

Fixes: cmake build failure with GGML_CUDA=ON + GGML_RPC=ON
This commit is contained in:
rankaiyx 2026-07-06 15:26:45 +08:00 committed by GitHub
parent bbc7de4751
commit 92e60231da
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
examples/rpc/rpc-server.cpp
View File

@ -263,7 +263,7 @@ static ggml_backend_t create_gpu_backend(const rpc_server_params& params, uint32
ggml_backend_t backend = NULL;
#ifdef GGML_USE_CUDA
fprintf(stderr, "%s: using CUDA backend: CUDA%d\n", __func__, device);
backend = ggml_backend_cuda_init(device, nullptr); // init device
backend = ggml_backend_cuda_init(device, nullptr, nullptr); // init device
if (!backend) {
fprintf(stderr, "%s: ggml_backend_cuda_init() failed\n", __func__);
}