An inter-mesh deployment pipeline

# An inter-mesh deployment pipeline Source: [https://blog.zm.is/an-inter-mesh-deployment-pipeline/](https://blog.zm.is/an-inter-mesh-deployment-pipeline/) My experience working with large\-scale environment drift and what I invented to try to remedy the problems\. I have around 72 different apps, and two environments, staging and production\. Generally, the idea is to deploy to staging, test it out, and then promote to production\. In practice, there was no tooling that made this mandatory, so it was often skipped\. Over time, environments drifted, and staging was soon forgotten as a testing ground\. I set about to change this by creating a connector that would act as a bridge between the two meshes, allowing production to ask staging to build, test, and then ship any component\. One of the main challenges with this was that checking if everything is identically configured on both environments meant needing to know exactly what should have been, which led me to first develop a way to deploy*new*applications using this method\. After all, if it knows how to deploy a new app successfully, why wouldn't it have that capability? Once deployed, updates can be instigated from production by clicking "Pull from staging" which results in a fully auditable log as the following events take place: Deploys can be triggered by API/MCP, adding another step \(pictured\)\. Black arrows are outgoing, red arrows are the return path\.`proc\-mesh`is a special component which runs unprivileged, with no network, and one unix socket\. I decided not to write any programs which continuously run as`root`, instead using`doas`for carefully\-audited privilege escalation when needed, invoked by`proc\-mesh`\. The other components have`NoNewPrivileges=yes`, so even if they wanted to use`doas`, they would be unable\. It was necessary to use a layered approach to security where gaining privileges loses the ability to access the network\. ``` permit nopass xfproc as root cmd /usr/local/sbin/xf-restart-plug permit nopass xfproc as root cmd /usr/local/sbin/xf-trustbase-verify # Install a staged artifact: <component> <sha256>. The bridge owns the # allowlist and derives both the artifact path and the destination itself. permit nopass xfproc as root cmd /usr/local/sbin/xf-install-binary ``` Shown in the log: Artifacts are pulled across the NATS bridge in binary form\. No Git releases\.And fully auditable at every stage: Now that it knows how to check the setup of existing apps and create new apps, it becomes easy to perform bulk tasks like bringing the environments back in sync\. Each app is staged first, tested, then shipped into production\. #### What's next? You may have spotted one issue:`plug\-mesh`cannot deploy itself, and still uses my old Gitea runner deployment path\. My original fix for its own deployment was to connect a new`plug\-mesh`to the isolated mesh\-to\-mesh NATS bridge \(which is different from the main bridge, which connects all components within one mesh\) and perform the tests on it there using the existing`plug\-mesh`component, before having the new one kill the old one and seamlessly transition to the main NATS network\. In theory, this would work, but in practice I found that I was not often amending this component, so did not feel the need to enable it to deploy itself\. If I do implement this, it will likely be a side effect of a smarter deployment path with more rigorous testing which stages a new component without stopping the old one, requiring an isolated NATS\. Currently, we accept that staging can break, and this is the whole point\. If staging an application fails, well, at least it wasn't on production\!