Theirs is certainly an impressive environment and I don’t mean to do Cloudflare’s achievements a disservice, but I strongly encourage engineers building these kinds of systems to treat their infrastructure as actual code, and avoid the temptation to dip in and out of wire text formats like JSON or YAML as much as possible.

The worst case scenario, in terms of engineering, is one piece of Python using Jinja templated YAML only for another piece of Python also written by you! to parse that output. Every time this happens it proves to be — as the article points out — a seized opportunity to get caught out by syntax errors, and a missed opportunity to have static analysis find errors (mypy et al., basically) before they happen at runtime, should all the logic had been done in pure Python without dipping in and out of structured text.

In the Cloudflare system the fundamental unit of action is configuration driving Python functions through gitops. My preferred version of these systems is pure python at the top emitting execve() calls, sh-scripts, and file writing over ssh or local transports, or in Dockerfiles, possibly with very small sh functions on the far side, but kept minimal in size and scope and with everything being purely declarative.

(It’s certainly an anti-pattern to return data back from the host to decide what to do next. The Python end is only allowed to declare that a package be installed, and the rest of the system ensures that is the case. People think this is limiting but the majority of these configuration systems, in my experience, hinge on 90% data structures to declare how the system out to be — IPAM arithmetic, building config files from lists of domains and accounts, processing key material etc. — and only 10% is the logic to install things much of which is very simple given a good base OS like Debian where many packages split their config into .d directories with helper scripts to enable things.)

PS: I wonder if the authors have had experience with Ansible? It was my own experience with that tool’s slowness and inflexibility that prompted a lot of my opinion forming in this area. Lots of good ideas have been borne of having first been exposed to Ansible and, alas, coming up against its limits.

Finding the grain of sand in a heap of Salt