Show HN: Filament – Fast data movement engine in Go
Summary
Filament is a fast data movement engine in Go that provides pluggable data replication with checkpointing, batching, and integrity events for reliable and durable data pipelines.
View Cached Full Text
Cached at: 09/10/26, 05:18 PM
galaxy-io/filament
Source: https://github.com/galaxy-io/filament
Filament
Filament is pluggable data replication with checkpointing, batching, and integrity events.
It moves data from sources to sinks using full, incremental, or change data capture replication. Filament keeps progress durable as data moves, verifies batches on both sides of a write, and safely pauses or resumes runs from their checkpoints.
Sources, sinks, state storage, and event transport are replaceable interfaces. Run Filament as a service with its API and web UI, deploy it to your own infrastructure, or embed the engine in a Go application.
Install
curl -fsSL https://getgalaxy.io/filament/install | sh
On macOS via Homebrew
brew install galaxy-io/tap/filament
Prebuilt binaries for Linux and macOS are on the releases page.
Getting started
To run Filament locally, install Go, Docker, just, Node.js, and pnpm. On macOS, the repository’s Brewfile installs the toolchain:
brew bundle
For Podman users, set CONTAINER_ENGINE=podman to use Podman with the commands below. See engine configuration.
Start PostgreSQL and NATS, then run Filament:
just infra
just dev
Open http://localhost:5173 to use the web UI.
When you are finished, stop the local infrastructure with:
just infra down
Deploying Filament
Kubernetes with Helm is the recommended way to run Filament in production. See the Kubernetes deployment guide for installation instructions and the Helm chart README for all supported values.
For a simpler hosted deployment without Kubernetes, deploy with 1 click:
Read the Railway, Render, and DigitalOcean guides for the deployed resources, runtime behavior, and production checklist.
The DigitalOcean deployment uses filament/standalone, an all-in-one image that includes the control-plane, server, and SQLite datastore.
How it works
A Filament pipeline connects a source to a sink and defines how selected resources should be replicated. During a run, Filament extracts records into bounded batches, writes and verifies each batch, and advances durable checkpoints only after successful work.
The runtime is event-driven: the server accepts pipeline and run requests, the control plane schedules and dispatches work, and workers execute individual runs. PostgreSQL stores durable state and NATS JetStream carries lifecycle events between components.
Developing Filament
Filament is written in Go, with a React and TypeScript web UI. Contributions to the engine, connectors, UI, deployment tooling, tests, and documentation are welcome.
See CONTRIBUTING.md for repository setup, code generation, tests, and pull request guidance.
Community
- Read the Filament documentation.
- Ask questions and meet other users in the Filament Slack community.
- Report bugs or propose features through GitHub Issues.
Similar Articles
Rive, Fast and reliable background jobs in Go
River is a robust, high-performance job processing system for Go and Postgres, emphasizing transactional enqueuing to avoid distributed system pitfalls.
Show HN: Tiny – An interpeted dynamic langauge with inline Go native functions
Tiny is a new interpreted dynamic programming language written in Go, featuring a bytecode virtual machine, JIT compilation, and inline Go native functions for high performance.
Fileregister: Tagging and reference layer for your files, in plain text
Fileregister is a portable Go CLI that creates a plain-text tagging and reference layer for files using permanent IDs, JSONL indexes, and Markdown sidecars, allowing links and metadata to survive renames and moves across disks.
HFlow
HFlow is a scalable multimodal data pipeline tool for robotics applications, recently featured on Product Hunt.
I want extern "fil-c"
The author argues for a Rust FFI that speaks the Fil-C ABI, enabling safe interop with recompiled C/C++ libraries, and calls for cross-ecosystem collaboration among Rust, Fil-C, Zig, and Nix/filnix communities.