BKL

bkl (short for Baklava because it has layers) is a templating configuration language without the templates. It's designed to be simple to read and write with obvious behavior.

Write your configuration in your favorite format: JSON, YAML, or TOML. Layer configurations on top of each other, even from different file formats. Use filenames to define the inheritance. Have as many layers as you like. bkl merges your layers together with sane default behavior that you can override. Export your results in any supported format for human or machine consumption. Use the CLI directly or in scripts or automate with the library.

Example

service.yaml service.test.toml

addr: 127.0.0.1
name: myService
port: 8080

port = 8081 Run it!

$ bkl service.test.toml
addr = '127.0.0.1'
name = 'myService'
port = 8081

bkl knows that service.test.toml inherits from service.yaml by the filename pattern (override with $parent) and uses filename extensions to determine formats.

Install

Using go

# Install go from go.dev
$ go install github.com/gopatchy/bkl/...@latest
# Verify that ~/go/bin is in your $PATH

macOS using brew

# Install brew from brew.sh
$ brew install gopatchy/bkl/bkl

You can also download binaries directly here.

Formats

Output defaults to the same format as the specified input file. Use -f (or --format) to override.

YAML

$ bkl -f yaml service.test.toml
addr: 127.0.0.1
name: myService
port: 8081

TOML

$ bkl -f toml service.test.toml
addr = '127.0.0.1'
name = 'myService'
port = 8081

JSON

$ bkl -f json service.test.toml
{"addr":"127.0.0.1","name":"myService","port":8081}

JSON (pretty)

$ bkl -f json-pretty service.test.toml
{
  "addr": "127.0.0.1",
  "name": "myService",
  "port": 8081
}

jsonl is an alias for json to fully support JSON Lines.

You can also set the output format implicitly by using a different file extension.

$ bkl service.test.yaml   # real file is service.test.toml
addr: 127.0.0.1
name: myService
port: 8081

Write

Output goes to stdout by default. Errors always go to stderr.

File Output $ bkl -o out.yaml service.test.toml

Output format is autodetected from output filename (unless specified with -f).

Inputs

Commandline Layering $ bkl a.b.yaml c.d.yaml # a.yaml + a.b.yaml + c.yaml + c.d.yaml

Specifying multiple input files merges all layers in order.

stdin

$ bkl -- -.yaml <<'EOF'
a: 1
EOF

Specifying an input file with the base name - and a valid format extension causes bkl to read that format from standard input. Use -- before the filename to avoid it being treated as a flag.

Symlinks

$ ln -s a.b.yaml c.yaml
$ bkl c.d.yaml   # a.yaml + a.b.yaml (c.yaml) + c.d.yaml

bkl follows symbolic links and evaluates the inherited layers on the other side of the symlink.

Inheritance

Inheritance is determined using filenames by default. After stripping the extension, the remaining filename is split on . and treated as an inheritance hierarchy (e.g. a.b.c.yaml inherits from a.b.<ext> inherits from a.<ext>). Parent layers may have any supported file extension.

Set Parent In File $parent: a.b # inherits from a.b.<ext>, from a.<ext>

bkl will check for all supported endings of a manually-specified parent file and will still evaluate layers under the parent in the normal order.

Parent List

$parent:
- a
- b

Parent Wildcard $parent: a.*

Setting $parent to a list or wildcard allows inheriting from multiple files. All parent files are loaded before the child.

* does not match .. a.* matches a.b.yaml but not a.b.c.yaml.

Set Parent In CLI

$ bkl a.b.yaml c.d.yaml      # a.yaml + a.b.yaml + c.yaml + c.d.yaml
$ bkl -P a.b.yaml c.d.yaml   # a.b.yaml + c.d.yaml

Layer order can be specified with commandline argument order. The -P flag (or --skip-parent) tells bkl to not load parent layers using filenames or $parent.

No Parent $parent: false # no further inheritance

Setting $parent to false or null stops any further inheritance regardless of filename structure.

Streams

Streams package multiple documents into a single file. YAML/TOML streams are delimited with --- or sometimes +++. JSON streams are concatenated or delimited with newlines (see JSON Lines and ndjson).

To layer streams, bkl has to match documents between layers. By default, child documents are applied to all parent documents. $match allows applying to specific parent documents.

a: 1
---
b: 2

+ c: 3 =

a: 1
c: 3
---
b: 2
c: 3

a: 1
---
b: 2

$match:
  b: 2
c: 3

a: 1
---
b: 2
c: 3

The supplied pattern can match multiple documents and the updates will be applied to all of them. $match: {} matches any documents with a map as their root element, regardless of parent.

a: 1
---
b: 2
---
a: 1

$match:
  a: 1
c: 3

a: 1
c: 3
---
b: 2
---
a: 1
c: 3

$match: null forces the updates to apply to a new document.

a: 1 +

$match: null
b: 2

a: 1
---
b: 2

Maps

Maps are merged by default. To change that, use $replace: true or remove individual entries with $delete.

a: 1 + b: 2 =

a: 1
b: 2

a: 1 +

b: 2
$replace: true

= b: 2

a: 1
b: 2

c: 3
b: $delete

a: 1
c: 3

bkl returns an error if you use $delete, $replace: true, or a key: value pair when they don't override a value from a lower layer. This helps keep upper layers minimal.

Lists

Lists are merged by default. To change that, use $replace: true or remove individual entries with $delete.

- 1 + - 2 =

- 1
- 2

- 1 +

- 2
- $replace: true

= - 2

- x: 1
- x: 2

- x: 3
- $delete:
    x: 2

- x: 1
- x: 3

bkl returns an error if you use $replace: true or $delete when they don't override a value from a lower layer. This helps keep upper layers minimal.

To update a specific list item from a parent layer, use $match. Combine it with $value for scalars.

- a: 1
- b: 2

- $match:
    b: 2
  b: 10

- a: 1
- b: 10

- 1
- 2

- $match: 2
  $value: 10

- 1
- 10

$""

Use $"" to interpolate format strings with $merge-style references.

a: 1
b:
  c: foo
d: $"{b.c} bar {a} 2"

a: 1
b:
  c: foo
d: foo bar 1 2

$env

Use $env: to substitute environment variables. This is supported in keys, values, and directives.

# export FOO=test
"$env:FOO": 1

= test: 1

# export FOO=test
a: $env:FOO

= a: test

# export FOO=a
a:
  b: 1
c:
  $output: true
  $merge: $env:FOO

= b: 1

Note that all $env: substitutions result in string values even if the substituted value is true, false, null, or all digits.

$encode

$encode transforms the subtree into the specified format.

base64

$value: a
$encode: base64

= YQ== flags

a: 1
b: 2
$encode: flags
# [tolist:=, prefix:--]

- --a=1
- --b=2

flatten

- - a
  - b
- - c
  - 4
- e
- $encode: flatten

- a
- b
- c
- 4
- e

join

$value: [a, b]
$encode: join:/

= a/b json

a: 1
b: 2
$encode: json

|
  {"a":1,"b":2}

json (list)

- a: 1
- b: 2
- $encode: json

|
  [{"a":1},{"b":2}]

prefix

- a
- 2
- $encode: prefix:X

- Xa
- X2

tolist

a: 1
b: 2
$encode: tolist:=

- a=1
- b=2

Stacking

a: 1
b: 2
$encode: [tolist:=, "join:,"]

= a=1,b=2

$required

Use $required in lower layers to force upper layers to replace the value.

a: 1
b: $required

+ c: 3 = Error

a: 1
b: $required

b: 2
c: 3

a: 1
b: 2
c: 3

a: 1
b:
  - $required

+ c: 3 = Error

a: 1
b:
  - $required

b:
  - 2
c: 3

a: 1
b:
  - 2
c: 3

$merge

Use $merge to merge the contents one subtree or scalar value into another.

foo:
  bar:
    a: 1
zig:
  b: 2
  $merge: foo.bar

foo:
  bar:
    a: 1
zig:
  a: 1
  b: 2

foo:
  bar:
    - a: 1
zig:
  - b: 2
  - $merge: foo.bar

foo:
  bar:
    - a: 1
zig:
  - b: 2
  - a: 1

foo:
  bar:
    a: 1
zig:
  b: 2
  c: $merge:foo.bar.a

foo:
  bar:
    a: 1
zig:
  b: 2
  c: 1

You can also merge across documents using $match and optionally $path:

a: 1
b: 2
---
c: 3
$merge:
  $match:
    a: 1

a: 1
b: 2
---
a: 1
b: 2
c: 3

a: 1
b:
  c: 3
---
d: 4
$merge:
  $match:
    a: 1
  $path: b

a: 1
b:
  c: 3
---
c: 3
d: 4

$merge also supports a shorthand syntax equivalent to $match and $path:

a: 1
b: 2
---
c: 3
$merge: [{a: 1}]

a: 1
b: 2
---
a: 1
b: 2
c: 3

a: 1
b:
  c: 3
---
d: 4
$merge: [{a: 1}, b]

a: 1
b:
  c: 3
---
c: 3
d: 4

$replace

Use $replace to merge the contents of one subtree or scalar value with another.

foo:
  bar:
    a: 1
zig:
  b: 2
  $replace: foo.bar

foo:
  bar:
    a: 1
zig:
  a: 1

foo:
  bar:
    - a: 1
zig:
  - b: 2
  - $replace: foo.bar

foo:
  bar:
    - a: 1
zig:
  - a: 1

foo:
  bar:
    a: 1
zig:
  b: 2
  c: $replace:foo.bar.a

foo:
  bar:
    a: 1
zig:
  b: 2
  c: 1

Note that $merge and $replace are equivalent for scalar values.

You can also replace across documents using $match and optionally $path:

a: 1
b: 2
---
c: 3
$replace:
  $match:
    a: 1

a: 1
b: 2
---
a: 1
b: 2

a: 1
b:
  c: 3
---
d: 4
$replace:
  $match:
    a: 1
  $path: b

a: 1
b:
  c: 3
---
c: 3

$replace also supports a shorthand syntax equivalent to $match and $path:

a: 1
b: 2
---
c: 3
$replace: [{a: 1}]

a: 1
b: 2
---
a: 1
b: 2

a: 1
b:
  c: 3
---
d: 4
$replace: [{a: 1}, b]

a: 1
b:
  c: 3
---
c: 3

$output

Use $output: true to select a subtree for output instead of the document root.

foo:
  bar:
  	$output: true
    a: 1
	b: 2

a: 1
b: 1

foo:
  bar:
    - $output: true
    - a: 1
    - b: 2

- a: 1
- b: 1

Multiple instances of $output: true in a document will generate multiple output documents (delimited with ---).

Use $output: false to omit a subtree or entire document from the output.

a:
  b: 1
  $output: false
c:
  d: 2

c:
  d: 2

a:
  - b: 1
  - $output: false
c:
  - d: 2

c:
  - d: 2

a: 1
$output: false
---
b: 2

= b: 2

Combine $output with $replace or $merge to have hidden "template" subtrees that don't appear in the output but can be copied in as needed.

$repeat

$repeat: at the top level of a document causes the document to be duplicated the given number of times. Within the document, $repeat can be used to reference the zero-based repeat index.

a: foo
b: $repeat
$repeat: 2

a: foo
b: 0
---
a: foo
b: 1

TOML

TOML doesn't allow unquoted $ in keys, which impacts bkl directives. Work around this by quoting keys:

"$parent" = false

bklb

bklb is a wrapper for CLI programs that take configuration files as commandline arguments but do not support bkl format. It transparently merges layers, translates formats, writes to temporary files, alters the commandline arguments, then execs the wrapped program.

$ ln -s ~/go/bin/bklb ~/go/bin/catb   # catb could be, e.g. kubectlb

$ cat service.yaml
addr: 127.0.0.1
name: myService
port: 8080

$ cat service.test.toml
port = 8081

$ catb service.test.toml
addr = "127.0.0.1"
name = "myService"
port = 8081

$ cat service.test.yaml service.test.json
cat: service.test.yaml: No such file or directory
cat: service.test.json: No such file or directory

$ catb service.test.yaml
addr: 127.0.0.1
name: myService
port: 8081

$ catb service.test.json
{"addr":"127.0.0.1","name":"myService","port":8081}

Note that the files mentioned don't have to exist; bklb will search for files with the same root name but different extensions, then merge layers and translate into the specified format.

bklb takes the name of the program it wraps from its own filename, hence the ln -s symlink creation in the example above. It trims at most one b from the end of its name before searching for the wrapped program so they can coexist in your PATH.

bkld

$ bkld <base_layer_path> <target_output_path>

bkld (d for "diff") generates the minimal intermediate layer needed to create the target output from the base layer. Along with bkli, it automates splitting existing configurations into layers.

a: 1
b: $required
c: 3

? →

a: 1
b: 2
d: 4

b: 2
c: null
d: 4

- a: 1
- b: 2

? →

- a: 1
- c: 3

- c: 3
- $delete:
    b: 2

- 1
- 2

? →

- 1
- 3

- 1
- 3
- $replace: true

bkli

$ bkli <target_output_path> <target_output_path> ...

bkli (i for "intersect") generates the maximal base layer that the specified targets have in common. Along with bkld, it automates splitting existing configurations into layers.

Any fields that exist in all upper layers but have different values will be marked $required.

a: 1
b: 2
c: 3

∩

a: 1
b: 10
d: 4

a: 1
b: $required

- a: 1
- b: 2
- c: 3

∩

- a: 1
- b: 10
- d: 4

= - a: 1

bklr

$ bklr <lower_layer_path>

bklr (r for "required") generates a document containing just the required fields and their ancestors from the lower layer. The output can be edited into a minimal upper layer.

a:
  b: $required
  c: 3

a:
  b: $required

a:
  - $required
b:
  - 2

a:
  - $required

kubectl bkl

$ kubectl bkl <kubectl_commands>

kubectl bkl is a kubectl plugin that wraps all the normal kubectl commands by evaluating any input files as bkl layers and passing the output to kubectl.

# deploy.dev.yaml and deploy.yaml are bkl layer files
$ kubectl bkl apply -f deploy.dev.yaml
deployment.apps/deploy-dev unchanged

Docker

bkl docker images are available at ghcr.io/gopatchy/bkl:latest for amd64 and arm64. Compose these images for your development and CI/CD workflows.

Dockerfile

FROM ghcr.io/gopatchy/bkl:latest AS bkl

FROM debian:latest AS build
COPY --from=bkl /bin /bin

diff

It's easy to see differences between processed bkl files using shell syntax.

$ diff -ud --color <(bkl a.yaml) <(bkl b.yaml)
--- /dev/fd/63  2023-07-12 21:00:55.412475845 -0700
+++ /dev/fd/62  2023-07-12 21:00:55.412475845 -0700
@@ -1,3 +1,2 @@
 a: 1
-b: 2
 c: 3

$ diff -y --color <(bkl a.yaml) <(bkl b.yaml)
a: 1                              a: 1
b: 2                            <
c: 3                              c: 3

Migrate

Below is an example process for migrating an existing set of configurations to bkl. It contains some Kubernetes-specific items but the use of the bkl* tools applies to any configuration source.

We assume you start with two K8s deployments called deploy-dev and deploy-prod which are similar but not identical.

First, install tools:

It's possible to export configuration directly from existing templating systems (e.g. kustomize build, helm install --dry-run --debug). Instead, we do this in a generic way and make sure we get the latest configuration by fetching it directly from the K8s API server.

Export Existing Config

# "kubectl neat" removes status and default value fields
$ kubectl neat get -- deploy deploy-dev > deploy-dev-orig.yaml
$ kubectl neat get -- deploy deploy-prod > deploy-prod-orig.yaml

Generate Base Layer

# See bkli details here
$ bkli -o deploy.yaml deploy-dev-orig.yaml deploy-prod-orig.yaml

Generate Service Layers

# See bkld details here
$ bkld -o deploy.dev.yaml deploy.yaml deploy-dev-orig.yaml
$ bkld -o deploy.prod.yaml deploy.yaml deploy-prod-orig.yaml

Check Results

# Should show no diff
$ kubectl bkl diff -f deploy.dev.yaml
$ kubectl bkl diff -f deploy.prod.yaml

Clean Up $ rm deploy-dev-orig.yaml deploy-prod-orig.yaml

You now have 3 bkl files:

deploy.yaml is the base layer containing values common to both deployments
deploy.dev.yaml is the upper layer containing values specific to deploy-dev
deploy.prod.yaml is the upper layer containing values specific to deploy-prod

The migration process isn't deterministic; there are design and aesthetic considerations. Here are some general tips:

Iterate by diffing your evaluated layers against production (e.g. kubectl bkl diff -f), making small changes, then diffing again.
Migrate as much as possible to the lowest layer to reduce duplication and complexity.
Consider making your base layer match your production configuration then overriding values for dev/test configurations. This makes it very clear where you're drifting away from production.
Remove duplication with $merge: but avoid chained merge paths.
Avoid using hidden $output: false trees as template variables; prefer overriding values in-place.
bkld doesn't know how to match entries within lists, so it may remove and replace large entries (e.g. K8s containers) that could be trivially patched. Use $match: to select the container and override values within it.

Comparison

bkl has some overlap with other configuration templating tools (e.g. Helm, Kustomize, Hiera). We believe that bkl has a combination of project goals that aren't fully served by any of the alternatives:

Configuration and templating/layering should be generic, not service-specific.
Configuration and templating/layering should be separate from deployment tooling.
Configuration files should be written in standard formats (JSON, YAML, TOML) and parseable by their standard parsers.
Basic functionality should be available without learning custom syntax.
Non-templatized configuration (e.g. StackOverflow answers) should be usable without modification.
File composition should be accomplished without meta configuration (e.g. manifest files).
Templating/layering behavior should be intuitive and produce expected results.
Documentation and examples should be excellent.