About grlx

What is grlx?

grlx (pronounced like "garlic") is an open source fleet configuration management system that is low overhead, dependency free, and easy to install. It is written entirely in Go, and is built on top of NATS to provide secure communication and scalability. Once deployed, it can be used to automate system management tasks across targeted nodes (known as sprouts), and to ensure targeted nodes are in a known state (as defined by recipes). It is designed to follow DevOps and GitOps best practices.

Why grlx?

At ADAtomic, we manage a diverse fleet of devices – from cloud instances, to several thousands of Linux-based IoT devices that can be on or offline due to unstable networks. Having tried and deployed existing configuration management products at scale, we found that these products didn’t meet our real-world expectations when it came to resource usage, resiliency, and simplifying overall complexity. Therefore, we set out to build our own – grlx.

grlx is designed to be:

Low overhead. Applications have the focus due to grlx’s small memory footprint and low CPU usage on managed nodes (sprouts).
Simple. It’s dependency free to reduce breakage during system updates, and easy to install since it’s one binary when deployed.
Scalable and fault-tolerant. Leveraging NATS server clustering, a single management server (farmer) can be scaled to share the load of incoming messages.
Secure by default. With grlx, there’s no need for a third-party encryption suite to stay secure. All communication is encrypted using self-signed TLS certificates and NATS.io NKey encryption. Certificates are pinned to the clients on first connection as an extra security precaution.
Flexible. A management server can be spun up as a systemd service or in a container. To keep the grlx core lightweight, a plugin system is used to add features, such as supporting obscure package managers or downloading files from uncommon endpoints – all backed by Go interfaces.
Easy to understand. Binaries are written entirely in Go and most of the source code can be read through in a day or so. Our non-stdlib build dependencies are selective – kept to as short a list as reasonably possible, so there’s little happening “elsewhere.”
Supports best practices. Deploying configurations to nodes (we call it cooking recipes), will soon support direct Git integration for your GitOps workflows. Authentication and authorization is handled on a per-user basis (to support RBAC), so there‘s no need to share credentials, SSH keys, or configure sudo access to the main CNC server.
More features to come. Message delivery rules for scheduled deployments, DropShell for shell access on a managed node without opening ports, and a local web UI for management are examples of what’s next!

Anatomy of a grlx farm

A typical grlx deployment is made up of three core components:

The farmer is the management server running the grlx-farmer binary. Commands sent to managed nodes (the sprouts) are published securely over a NATS message bus.
The sprout(s) are managed nodes running the grlx-sprout binary. Sprouts receive and execute management commands issued by the farmer.
The grlx CLI communicates with the farmer in order to manage the farm of sprouts. It can be used locally on the farmer (management server), or remotely over a secure-by-default, TLS-encrypted API hosted on the farmer.

Deploying a grlx farm

Getting started is really simple with our Quick Start guide. We even have simple uninstall instructions!

You’ll need a dev machine or server where you’d like to use the grlx CLI. Here you will install the CLI and provision your admin public key.
You’ll need a place to run the farmer binary in order to create your management server. This can be a dedicated server instance or in a container. Our Quick Start has examples for each. This is also where any recipes you develop will be stored.
On all of the managed nodes (sprouts) in your farm, you‘ll need to install the grlx sprout binary. Our Quick Start installer detects your system architecture and selects the correct version to install and spin up as a systemd service.
Lastly, you will use the grlx CLI to join the sprouts to the farmer and you’re ready to cook!

How sprout and CLI auth works

grlx is secure by default. All communication to and from the farmer is encrypted – third-party encryption services are not needed. This is facilitated by NATS.io NKey encryption and self-signed TLS certificates. Here‘s how it works:

When you first install the grlx CLI using grlx init, the CLI generates an elliptic-curve public/private keypair. When configuring the farmer, you provide the public key in the config. The farmer then uses the NKey library to authenticate you on the NATS bus. Additionally, the NKey is used to generate auth tokens which can be used to authenticate requests against the farmer‘s REST API.
Every sprout automatically generates an NKey (public/private keypair) at first install, and the NKey public key is sent to the farmer where it is saved as an “unaccepted” sprout. It is on the user to use the CLI to move that key over to the accepted list by entering grlx keys accept SPROUT_ID. Once accepted, the sprout uses the same mechanisms as the CLI to communicate with the NATS bus and REST API.

How configuration management works (recipes)

Configuration Management aims to maintain the consistency of deployed IT systems, improving the stability of those systems by reducing errors, repetition and variance. This is often done by declaring the desired state of your systems, and invoking an automated process to ensure that your fleet of servers and devices reach that state. It can be used to manage both system and application level configurations, apply system security policies, and is key to fleet identification and auditing.

grlx follows the practices of other products where Infrastructure as Code (IaC) “recipes” are used to define the steps needed to put a server or application into a valid and functioning state. These recipes are YAML files – often managed in source control – that describe a set of actions that a sprout should perform when deploying (or “cooking”) the recipes. The steps in a recipe are carried out by Ingredients, which can be customized and even extended through grlx’s plugin system.
Recipe steps can be defined to set up services, deploy your application’s configuration, or validate that systems have the required files they need. grlx employs a Template Engine for recipes, where Properties (both farmer and sprout defined) can be evaluated to tailor the recipes for different situations.
grlx also provides tools to check on the state of servers, as well as automate the execution of shell commands across many targeted nodes at once.

System architecture

System components defined

To learn more about grlx, check out our docs!

Farmer (grlx-farmer): The farmer is your management server running the grlx-farmer binary. The binary can run as a systemd service or be hosted in a container. This has analogs in other similar configuration platforms. In SaltStack, this is your Salt Master, or in Chef, your Chef Server.
Sprout (grlx-sprout): The sprout is a managed node running the grlx-sprout binary. Sprouts receive commands from the farmer over the NATS message bus. These can be immediate shell commands, or actions that are performed by Ingredients defined in a recipe (e.g. ensure that a file exists). Sprouts are similar to a Salt Minion in SaltStack or a Chef Client in Chef.
CLI (grlx CLI): The grlx CLI is a tool used to interact with the farmer and manage sprouts. The CLI is used to accept keys from sprouts and “cook” (run) recipes that are hosted on the farmer. The CLI also contains tools to tail the traffic over the NATS bus, test sprout connections, and run arbitrary commands on sprouts.
Auth: All communication is encrypted using self-signed TLS certificates and NATS.io NKey encryption. Certificates are pinned to the clients on first connection as an extra security precaution. Furthermore, every CLI must accept a working certificate for any interactions with the farmer. NKeys provides a NATS native method for handling asymmetric encryption of messages between all parts of the system.
Cmd Execution: The CLI has the ability to dispatch arbitrary shell commands to sprouts using grlx cmd run. Jobs are evaluated and dispatched to sprouts and their outputs are aggregated to the output of the CLI.
Farmer Hooks (hooks): Hooks provide a way to fire off webhooks when recipe deployment (cook) starts, fails or finishes. Additionally, you can configure a hook to fire when a new sprout is added to the farm, or when a sprout’s status has changed. A “hook” ingredient is also provided (not shown) that can be fired from the sprout during steps of a recipe.
Ingredients: Ingredients are how we build configurations with grlx. They can be thought of as the building blocks for completing various file, service, or management operations. Ingredients use a provider interface to allow for simple extensibility via the Go plugin system.
Template Engine: Templating is achieved with Go’s builtin templating engine. This provides a Jinja-like templating system that hooks into the Farmer and Sprout Props to customize recipes.
Job Enqueuing and Processing: You initiate jobs from the CLI by cooking recipes. Processing is continued on the farmer, and sprouts stream results to the CLI in real-time over the message bus.
Properties (Farmer, Sprout and Fetched Sprout Props): Both farmers and sprouts have properties for use in recipes. These properties are used in recipe templates to specify conditional execution of parts of the recipe. One example of this might be the specific OS or architecture of a sprout to ensure that the correct package gets installed on the correct machine.
API: Each farmer has a REST API used for querying information about jobs or other information the farmer has access to.
Local API: The Local API is hosted on the grlx CLI to run a web front end application for grlx. The local API provides a simple yet extensible interface for interacting with the CLI without the need for a terminal.
Message Bus: An embedded NATS.io server is built into the farmer to handle connection between the sprouts, farmer, and the CLI. The bus uses NKeys to handle secure message passing between sprouts, the farmer, and CLIs. The message bus is responsible for handling communication between the CLI and the farmer as well as ensuring that jobs get dispatched to sprouts.
Managed Node: The managed node is where the grlx-sprout daemon runs. It is the physical device where a sprout will deploy files, manage services, or run commands.