I'm on 7.9 stable.
I merely dabble in local LLM.
`pkg_add llama.cpp` is available,
but my laptop has only Intel iGPU,
and I get hard-to-comprehend errors,
online LLM suggested I disable GPU in `libggml`:
$ cd ports/devel/libggml/
$ diff Makefile Makefile.orig
39a40,46
> .if ${MACHINE_ARCH:Mamd64} || ${MACHINE_ARCH:Maarch64}
> CONFIGURE_ARGS+= -DGGML_VULKAN=on
> BUILD_DEPENDS= graphics/shaderc
> LIB_DEPENDS= graphics/vulkan-loader
> WANTLIB+= vulkan
> PKG_ARGS+= -Dvulkan=1
> .else
41a49
> .endif
You need ports fetched, compile `libggml` locally, install that.
Then, `llama.cpp` can be installed via `pkg_add`.
To run bigger models, need to bump limits up in `login.conf`.
For both `llama-cli` and `llama-server` my arguments look like this,
I'm not sure they are totally optimal,
yet to start with it's better than nothing:
--reasoning off \
--ctx-size 32768 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--flash-attn on \
--batch-size 2048 \
--ubatch-size 512 \
--threads 4 \
--threads-batch 4 \
--port 8080 \
--model $model-gguf-path
`--no-warmup` may also be useful.
KoboldCpp (a llama.cpp fork) has an OpenBSD section:
https://github.com/LostRuins/koboldcpp#compiling-on-openbsd
compiles and runs OK.
Harness. Since OpenBSD has `perl` in base already,
one may wish to look at
Langertha - The clan of fierce vikings to AId your rAId
https://metacpan.org/pod/Langertha
https://metacpan.org/dist/Langertha-Raider/view/bin/raider
Install with `cpan Langertha Langertha::Raider`.
Some dependency modules may be installed with `pkg_add`.
A brief hello world example. It loops, I didn't investigate why,
need to Ctrl-C it, but `hello.c` and `hello` files are OK.
`test_hello.sh` file not created.
There are line break formatting issues but you get the idea:
$ # first run llama-server with qwen3-8b and `--host 127.0.0.1`
$ raider -e openai -k dummy -m qwen3-8b \
> -o url=http://127.0.0.1:8080/v1 \
> -r . \
> "Write hello.c (C89, prints 'Hello, World!'), compile with cc
> -std=c89 -pedantic -Wall, write test_hello.sh that checks exit code
> and stdout, run the tests, report results." write_file
> content=#include <stdio.h> int main() { printf("Hello, World!");
> return 0; } path=hello.c
. 37b Wrote 77 bytes to /tmp/raider/hello.c
> bash command=cc -std=c89 -pedantic -Wall hello.c -o hello
> working_directory=/tmp/raider
. 319b Exit code: 0
> edit_file new_string=#include <stdio.h> int main(void) {
> printf("Hello, World!\n"); return 0; } old_string=#include <stdio.h>
> int main() { printf("Hello, World!"); return 0; } path=hello.c
. 26b Edited /tmp/raider/hello.c
> bash command=cc -std=c89 -pedantic -Wall hello.c -o hello
> working_directory=/tmp/raider
. 135b Exit code: 0
> edit_file new_string=#include <stdio.h> int main(void) {
> printf("Hello, World!\n"); return 0; } old_string=#include <stdio.h>
> int main(void) { printf("Hello, World!\n"); return 0; } path=hello.c
. 26b Edited /tmp/raider/hello.c
> bash command=cc -std=c89 -pedantic -Wall hello.c -o hello
> working_directory=/tmp/raider
. 135b Exit code: 0
^C
$ ls
hello hello.c
$ cat hello.c
#include <stdio.h>
int main(void) {
printf("Hello, World!\n");
return 0;
}
$ ./hello
Hello, World!
