Building Admission Controllers for Kubernetes with Lightscreen

Lightscreen is one component out of the Spectral platform for Kubernetes. It’s a modular admission controller toolkit for Kubernetes built in Go for Go developers.

A Word About Admission Controllers

The concept of admission is just one way to model how we allow or forbid objects into our cluster.

Kubernetes defines a pipeline of controllers that are active on kube-apiserver and that can participate in admission, a few of them are enabled by default such as:

• MutatingAdmissionWebhook
• ValidatingAdmissionWebhook

Which means you can create your own service that offers a webhook to participate in the admission decision flow. You can add any policy that you think is valuable for your cluster, that is not pre-baked into Kubernetes already, or is too complicated or too domain-specific to be included.

In the context of Spectral — a cyber security solution for data leak prevention, we want to stop harmful images from entering into our production clusteres.

Spectral uses Lightscreen as a modular platform for creating admission webhooks to allow or forbid an image that has not been scanned and approved by Spectral.

We must note that the design of admission within Kubernetes has changed quite a few times which was a bit alarming to get around (breaking APIs, flawed implementation in the case of bypassing parts of hooks with a mutating hook, and other interesting issues). However at this stage we feel that the current Kubernetes APIs are stable enough to build on, and so we did.

Building Your Own Image Notary For Kubernetes

Unlike opa, Lightscreen encourages to build your rules, logic and workflow in Go, and produce a self-contained static binary to deploy as an admission webhook service onto your cluster.

Let’s look at how to build your own image notary with Lightscreen. The full example exists on our Github repo if you prefer to follow by reading code.

In our context for this example story, the notary will look at an incoming request, get each image name, figure out its relevant SHA signature, compare it to an existing known one from an approved signatures list, and either admit or reject the request based on that.

First, creating a new server has just a single starting point that’s provided by Lightscreen:

func main() {
    ...
    server := admission.NewServer(admission.ServerOptions{
        Config:      *config,
        Development: development,
        Address:     *address,
    }, logger)
}

We packages all of the machinery of being a first-class Kubernetes admission controller into a nice Server abstraction, that expects either a Mutation action or a Validation action to be added later.

Let’s look at the currently configured actions, both mutation and validation actions:

func main() {
    ...
    server := admission.NewServer(admission.ServerOptions{
        Config:      *config,
        Development: development,
        Address:     *address,
    }, logger)
    logger.Infof("mutations=%v validations=%v", server.Actions.MutationMap, server.Actions.ValidationMap)
}

By default, Lightscreen comes with preconfigured SHA resolver and validator actions.

Each action is stored into an action map, and keyed by a unique identifier, so that even if actions are included by default, as long as you don’t use them, they’re not coming into effect.

These actions are configured with a by-convention workflow configuration file — the main and only configuration for Lightscreen. Here’s an example:

mutations:
  - type: resolve_sha
    finder: crane
validations:
  - type: admit_sha
    admit:
      "library/nginx@sha256:a8517b1d89209c88eeb48709bc06d706c261062813720a352a8e4f8d96635d9d": true

And finally, serve:

func main() {
    ...
    server := admission.NewServer(admission.ServerOptions{
        Config:      *config,
        Development: development,
        Address:     *address,
    }, logger)
    //server.Actions.
    logger.Infof("mutations=%v validations=%v", server.Actions.MutationMap, server.Actions.ValidationMap)
    logger.Infow("running", "address", *address, "config", *config, "development", development)
    server.Serve()
}

Lightscreen will run as a Kubernetes compliant admission webhook controller.

If you want to add or remove validation or mutation actions, you can explore the .Actions API:

yourAction := NewAction()
server.Actions.Add(yourAction)

Creating Your Own Actions

Creating your own actions is done by building them in Go. There are a couple abstractions that you use in order to build either a mutation or validation actions.

Mainly, each action type should conform to the Action interface:

type Action interface {
    Name() string
    Run(context.Context, *unstructured.Unstructured) error
}

Where unstructured is a way Kubernetes internally represents unstructured requests. With unstructured you'll be able to probe a request, modify and reason about it in your Mutation or Validation actions.

Here’s a mutating action that resolves an image name to the latest SHA (error handling removed for brevity):

func (r *ResolveSHAAction) Run(ctx context.Context, p *unstructured.Unstructured) error {
    o, _ := gabs.Consume(p.Object)
    containers := o.Path("spec.containers").Data().([]interface{})

    for _, c := range containers {
        container, _ := gabs.Consume(c)
        img := container.Path("image").Data()
        resolved, _ := r.resolver.Resolve(img.(string))
        container.Set(resolved, "image")
    }
    return nil
}

This will mutate the request and place an appropriate SHA in the containers section.

The way Lightscreen works is that you build your own collection of Go-based mutation or validation actions and you mix and match these, producing as many self-contained binaries for your various controllers that you like.

Why Build Rules In Go?

The fact that you build Lightscreen actions in Go is one of the main differences between Lightscreen and OPA, and it is the only way you can build new rules.

The reasons we chose Go here, which is not something that’s natural for a “rule based” engine, are:

• Familiarity: Familiar language and no learning curve
• Tooling: you can test, observe, trace, profile rules in isolation, the same way you do Go code
• Reuse: you can use your own codebase, utilities, infra, and domain
• Safety and performance: you get the same guaranties as with Go, which you’re already familiar with. For example, say you want to perform many parallel requests within a validation action.
• Self contained: there’s nothing like shipping a single, self-contained binary
• Many ways to package: you can use parts of Lightscreen, you can use it as a starting-point for something greater, and many other ways to slice things
• YAGNI: you ain’t gonna need it. If you know exactly what you want to build in terms of your admission workflow, and it’s not going to change much, you want something that gets out of your way and lets you code and ship a working service

What’s Next?

We use a version of Lightscreen internally in the Spectral platform. Every now and then we push a latest variant into the open source repo, which is being published on Github (being a security product, we use a private-public model for security reasons).

In our Lightscreen repo there are a few more examples waiting for you, as well as documentation and code to try out.