fantastic: latest llama.cpp server webui can now run commands for tools into rootless sandboxed containers

Reddit r/LocalLLaMA Tools

Summary

llama.cpp server build 10423 adds an experimental --tools-runtime option that lets the webui run tool shell commands inside rootless sandboxed containers (e.g., podman:alpine or docker:alpine), downloading and instantiating the image automatically.

while having much fun testing LLMs Houdini-like attitudes and abilities to evade, excalate and escape from carefully reciprocally arranged security enhancing sandboxing VMs, containers, namespaces and jails, I've discovered a new revolutionary option to let llama-server (build 10423) to launch guest containers (running for ex. bare bones linux distros like Alpine) as sandboxes for securely (rootless) running shell commands for its tools (enabled with the "--tools" option) directly from the llama-server webui. The new (experimental) option is called "--tools-runtime" and the initial argument to it that I suggest you to try is "podman:alpine" (or "docker:alpine"). The server will also download the indicated container image and then instantiate it. Of course podman (preferred by myself) or docker have to be installed beforehand on the host. Enjoy!
Original Article

Similar Articles

Fully in-browser container builds

Lobsters Hottest

A web application that builds containers entirely in the browser using client-side code, demonstrating the power of custom container tooling. Users can pick a base image, run a shell script, and export the resulting image as a tar file.