Kubernetes Versioning for Operator Developers

As we discussed in the previous blog, API versioning isn’t as simple as having running different services to handle the different versions, whereby each service processes each version independently of the other. With Kubernetes, your controller watches the state of an object in etcd, that stored object is versioned to a single value (i.e. only 1 version is ever stored at a time), and when you set-up a watch on the resource whichever version you ask for is the version you will get.

Therefore, when writing a k8s operator, there are a set of requirements and conventions we need to be aware of to ensure that we’re able to progress the software development lifecycle of the API specification and the controller versions in a robust manner.

TL;DR

For the purpose of this document there may be references to “client” or “client’s”. A client could mean a number of things. It could mean, as the term often means, some piece of software which is constructing an HTTP request and calling a specific version of our API. However, it could also mean the combination of kubectl, kustomize, helm or some technology like Config Sync, which effectively is doing this API call on behalf of the user, and the configuration manifest provided by a user who is specifying which API version they want to use in that YAML file. E.g.

Each point will be covered in more detail within the rest of the article

Core Principles

Backwards Compatibility: Changes to existing API versions MUST NOT break existing clients. This includes:
- No removing or renaming existing fields, types, or endpoints.
- No changing the semantics of existing fields or operations.
- No tightening validation rules that would invalidate previously accepted requests.
- Any request that worked before your change MUST work after your change.
- Requests that don't use your change MUST behave exactly as before.
Lossless Conversion: Conversion between different API versions MUST be lossless. No information should be lost when converting between versions and back.
Graceful Degradation: If a user request uses a feature not supported by the API version of the resource, it MUST NOT cause errors or degrade behaviour.
User Transparency: Users with existing resources on older API versions should NOT be impacted by changes in newer versions. Their resources should continue to function as expected.

Versioning Workflow

Incremental Evolution: Follow a clear versioning progression (e.g., v1alpha1 -> v1alphaN -> v1beta1 -> v1betaN -> v1 -> v2alpha1 -> ...).
Minimise Versions: Keep the number of defined API versions to a minimum.
Storage Versioning:
- Introduce new versions gradually. Initially, serve the new version but don't make it the storage version.
- Make a version the storage version when it's stable or when new properties are added. This ensures rollback compatibility.
- Do not add a new version and make it the preferred or storage version in the same release. This prevents issues if the API server needs to be rolled back.
Additive Changes: New versions should primarily add properties. Avoid removing or modifying existing properties until they are truly obsolete.
Consistent Properties: When adding a new property to one API version, add it to all versions to maintain consistency.
Default Values:
- Any field with a non-null default value in one version MUST have a non-null default in all versions.
- Explicitly define default values in your API definitions rather than setting them in the code (late initialisation).
Hub Model: Structure your API definitions to align with the "hub model" used by the main Kubernetes API for better code organisation and future development. This likely means we should be defining an internal structure in the code which all versions should map to. More details in the conversions section.

Implementation Details

Schema Migration:
- Use nullable: true and empty defaults to introduce new properties without immediate code changes.
- Use webhooks to prune or modify old properties when necessary.
- Take special care when considering to use the x-kubernetes-preserve-unknown-fields annotation to preserve fields between API versions because it can easily have unexpected results and cause confusion for the user.
Version-Specific Logic: If supporting multiple API versions with differing logic is essential, use a webhook to inject a version identifier (e.g., spec.requestedApiVersion) into the request before storage, allowing controllers to execute the appropriate code branch. This is a controversial point and goes against the general Kubernetes conventions, so should be considered seriously before implementation
Helm Considerations: Be aware that Helm-based operators (I.e. Operator SDK’s Helm Operator) have very little functionality to be able to perform version management. Therefore, and this is generally the case with Helm Operators anyway, they should only be used for very simple operators.
Conversion Webhooks: Keep conversion webhooks simple and optimised for performance, as they impact the overall Kubernetes API server performance, so can impact the whole cluster.
Layered Commits: When introducing a new API version, structure your changes into layered commits or PRs to improve reviewability:
1. A commit/PR that just copies the <existing-version> packages to the <new-version>.
2. A commit/PR that renames <existing-version> to <new-version> in the new files.
3. A commit/PR that makes any new changes for <new-version>.
4. Any commits/PRs to <existing-version> should update <new-version> and vice versa.
Field Type Changes: Be cautious when changing field types. Ensure that conversions between old and new types are handled correctly and don't lead to data loss or corruption.
Enum Values: When adding new enum values, consider their impact on existing clients. Avoid reordering or removing existing values.
Validation Rules: New validation rules should only be added to new API versions. Avoid adding validation to existing versions that could invalidate previously accepted objects.
Deprecation Policy: Establish a clear deprecation policy for API versions and communicate it effectively to users. Provide ample time for migration before removing support for older versions.
Testing: Thoroughly test API changes with a variety of clients and scenarios, including upgrades and rollbacks, to ensure compatibility and prevent regressions.

The Details

Defaults

In general we want default values to be explicitly represented in our APIs, rather than asserting that "unspecified fields get the default behaviour". This means, set a default in the CRD and don’t just handle it in the code.

This is important so that:

default values can evolve and change in newer API versions
the stored configuration depicts the full desired state, making it easier for the system to determine how to achieve the state, and for the user to know what to anticipate

There are 3 distinct ways that default values can be applied when creating or updating (including patch and apply) a resource:

static: based on the requested API version definition, fields can be assigned values during the API call. On exception only, defaults can be assigned from values in other fields in the resource but this case is not defined in this document because of complexities it can cause.
admission control: based on the configured admission controllers and possibly other state in or out of the cluster (e.g. Gatekeep data inventory on referential constraints), fields can be assigned values during the API call
controllers: arbitrary changes (within the bounds of what is allowed) can be made to a resource after the API call has completed and the object has been stored.

Some care is required when deciding which mechanism to use and managing the semantics. Refer to defaulting docs for more information.

Static Defaults

Static default values are specific to each API version. The default field values applied when creating an object with the "v1" API may be different than the values applied when using the "v2" API. In most cases, these values are defined as literal values by the API version (e.g. "if this field is not specified it defaults to 0").

Static defaults are the best choice for values which are logically required, but which have a value that works well for most users. Static defaulting must not consider any state except the object being operated upon.

Defaulting happens on the object when

There's a request to the API server using the request version defaults,
Reading from etcd using the storage version defaults,
Mutating admission webhooks have processed the request and update the object with any non-empty defaults.

With the exception of defaults in metadata, any fields which have defaults should be pruned for any versions which don't also specify the field.

Defaulting and Nullable

Null values for fields that either don't specify the nullable flag, or give it a false value, will be pruned before defaulting happens. If a default is present, it will be applied. When nullable is true, null values will be conserved and won't be defaulted.

For example, given the OpenAPI schema below:

creating an object with null values for foo and bar and baz

leads to

with foo pruned and defaulted because the field is non-nullable, bar maintaining the null value due to nullable: true, and baz pruned because the field is non-nullable and has no default.

Admission Controlled Defaults

In some cases, it is useful to set a default value which is not derived from the object in question. For example, when creating a PersistentVolumeClaim, the storage class must be specified. For many users, the best answer is "whatever the cluster admin has decided for the default". StorageClass is a different API than PersistentVolumeClaim, and which one is denoted as the default may change at any time. Thus this is not eligible for static defaulting.

Instead, we can provide a built-in admission controller or a MutatingWebhookConfiguration. Unlike static defaults, these may consider external state (such as annotations on StorageClass objects) when deciding default values, and must handle things like race conditions (e.g. a StorageClass is designated the default, but the admission controller is written in a way that it doesn’t see the update as soon as it’s live). These admission controllers are strictly optional and can be disabled. As such, fields which are initialised this way must be strictly optional.

Like static defaults, these are run synchronously to the API operation in question, and when the API call completes, all static defaults will have been set. Subsequent GETs of the resource will include the default values explicitly.

Controller-Assigned Defaults (aka Late Initialisation)

Late initialisation is when resource fields are set by a system controller after an object is created/updated (asynchronously). For example, the scheduler sets the pod.spec.nodeName field after the pod is created.

In Kubernetes, sometimes you create a "pod" (like a container for your app) without specifying all the details. A special controller sees this incomplete pod and fills in the blanks, like which server it should run on. This happens after you create the pod.

Controller-assigned benefits:

Flexibility: You don't need to know everything upfront.
Efficiency: The controller can make smart decisions based on the current state of the system.

Challenges:

Optional fields: The controller can only fill in optional fields. It can't change things you've already specified.
Race conditions: Sometimes multiple controllers might try to update the pod at the same time. Kubernetes has ways to handle this, but it's something developers need to be aware of.
Careful updates: Controllers should only change the fields they're responsible for. They shouldn't mess with other parts of the pod.

Rules when setting defaults

All defaulting should follow these rules to only apply changes that:

Set previously unset fields
Add keys to maps
Add values to arrays which have mergeable semantics
Never override a value that was provided by the user.

These rules ensure that:

Users can override any system-defaults by explicitly setting any fields with default values
User requested values can be merged with default values

kubectl replace (PUT Operations)

Kubernetes offers several ways of updating an object which preserve existing values in fields other than those being updated. However, the kubectl replace (aka HTTP PUT) way of updating objects can have bad interactions with default values. Imagine you're updating a document. You want to change some parts, but keep the rest the same.

Here's the problem: replace assumes you're providing the entire updated document. But sometimes, Kubernetes automatically fills in some blanks with default values. These defaults can change depending on how you update the document, leading to unexpected results.

Think of it like this:

You send a document with some blank spaces.
Kubernetes fills those spaces with the current defaults (like myProp: 1 of the defaults).
Later, you update the document using replace, but still with some blanks.
Kubernetes now fills those blanks with new defaults (like myProp: 2), potentially overwriting what was there before.

This can cause trouble, especially with fields that can't be changed after they're set. You might get errors or end up with values you didn't intend.

To avoid this, Kubernetes developers need to be especially careful when adding new fields with default values. They might need to write extra code to preserve the original values during updates, even if the user technically provided an incomplete document.

For example, when adding a field with a static or admission controlled default, if the field is immutable after creation, consider adding logic to manually patch the value from the oldObject into the newObject when it has not been set by the user, rather than returning an error or allocating a different value. This will very often be what the user meant, even if it is not what they said. This may dictate that the change is performed by an admission controller. Also be particularly careful to detect and report legitimate errors where the new value is specified but is different from the old value.

For controller-defaulted fields, the situation is even more unpleasant. Controllers do not have an opportunity to patch the value before the API operation is committed. If the unset value is allowed then it will be saved, and any watch clients will be notified. If the unset value is not allowed or mutations are otherwise disallowed, the user will get an error, and there's simply nothing we can do about it.

The bottom line: replace can be tricky with default values. Developers need to be mindful of how it works to avoid unexpected behavior and provide a smoother experience for users.

Conversions

If there are any notable differences - field names, types, structural change in particular - you must add some logic to convert versioned APIs to and from the internal representation.

For example, when there is a need to update a property from a singular value (e.g. string) to an array, there needs to be something in place so that older clients, that only know the singular field, continue to succeed and produce the same results as before the change. Whereas newer clients can use your change without impacting older clients. This way the API server can be rolled back and only objects that use your new change will be impacted.

Kubernetes Hub and Spoke Model for API Version Management

In Kubernetes, the hub-and-spoke model is a strategy for managing multiple API versions of custom resources. It provides a structured approach to handling versioning, conversion, and backward compatibility.

Core Concept of the model defines a single hub version and multi spokes:

Hub Version: This is the central, most stable version of the API. It's the target for conversions from other versions and is the internally defined structure which all versions should be able to convert to.
Spoke Versions: These are newer or older versions of the API that are connected to the hub. They define conversion paths to and from the hub.

What's involved:

API Server: When a client requests a resource in a specific version, the API server checks if the requested version matches the stored version.
Version Mismatch: If there's a mismatch, the API server triggers a conversion process.
Conversion Process: If the API’s conversion strategy is none, the API server tries to perform simple map to and from the versions. If it is webhook the webhook converts the resource to the hub version, performs necessary modifications, and then converts it back to the requested version.
Conversion Webhook: A conversion webhook, often implemented using controller-runtime, handles the conversion between the requested and stored versions.

Benefits:

Simplified Conversion: By defining conversions to and from the hub, the complexity of managing multiple bidirectional conversions is reduced.
Backward Compatibility: Older versions can still be used and converted to newer versions, ensuring smooth upgrades.
Flexible Versioning: New versions can more easily be introduced without breaking existing clients.
Improved Maintainability: Centralised management of conversions in the hub version simplifies maintenance.

By effectively utilising the hub-and-spoke model, it makes it possible to manage multiple API versions of custom resources in a more flexible and maintainable way, with the intention of ensuring a smoother evolution of Kubernetes resources.

Internal structure definition

As per the Kubernetes documentation, version management should follow a hub and spoke model. This model describes that all defined API versions in the CRD must be able to covert to each other but they should do so by first converting to an internally defined structure, before being converted to the other version.

In a hypothetical API (assume we're at version v6), the Frobber struct looks something like this:

You want to add a new Width field. It is generally allowed to add new fields without changing the API version, so you can simply change it to:

With regards to pre-made operator controllers, like those from the Operator SDK which allow the developer to create operators with Helm (not relevant for the purpose of complex API versioning), Ansible and Go, there should be a similar concept of managing an internal structure even though these technologies don’t directly support this sort of concept.

Using Ansible Operator SDK

kb6

The assert module could also be used to aid validation or conversion.

Conversion Webhooks

Bringing together internal structure management and defaulting, developers will still need to be able to make some changes in-flight (i.e. mutating the requested object before being stored and read by the controller). In such cases, the way to perform this action is by writing a Conversion Webhook.

A conversation webhook is a basically a mutating or validating admission webhook which is an HTTP callback that is invoked by the Kubernetes API server when a custom resource is created or updated.

The webhook can modify the CR object (mutating webhook) or reject the request (validating webhook) in-flight during the user's request.

Conventions:

The webhook server should be a separate process from the API server.
The webhook server should be implemented in a language that is supported by Kubernetes (e.g., Go, Java).
The webhook server should be secure and should authenticate requests from the API server.

Best practices:

Use a well-defined API for communication between the webhook server and the API server.
Implement robust error handling.
Log all requests and responses.
Be efficient and avoid making unnecessary API calls.

Constraints:

The webhook server must be reachable from the API server.
The webhook server must respond to requests within a timeout period.
The webhook server must return a valid JSON response.

Requirements:

The webhook server must be implemented in a language that is supported by Kubernetes.
The webhook server must be able to authenticate requests from the API server.
The webhook server must be able to validate or mutate custom resource objects.

For a recommendation for an implementation of a custom resource conversion webhook server See the example webhook implementation (used in the K8S API E2E tests). It shows how the webhook handles the ConversionReview requests sent by the API servers, and sends back conversion results wrapped in ConversionResponse.

Note that the request contains a list of custom resources that need to be converted independently without changing the order of objects.

The example server is organised in a way to be reused for other conversions. Most of the common code are located in the framework file that leaves only one function to be implemented for different conversions.

Object references

There are more details on https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#object-references in general but it is worth highlighting some considerations for version management:

Maintain consistency: When updating your API version, carefully consider the impact on existing object references. Avoid changes that could break those references or introduce ambiguity.
Document thoroughly: Clearly document any changes to how object references are handled in new API versions, including potential impacts on existing operators and how to migrate.
Backward compatibility: Strive for backward compatibility whenever possible to ensure a smooth transition for users.

Labels and annotations

Below is some advice regarding use of labels, annotations when managing version changes. There is more details on https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#label-selector-and-annotation-conventions:

API Version Changes and Metadata
- Label selectors: If your operator relies on label selectors, ensure that API changes don't inadvertently break those selections.
- Annotation handling: When updating your API version, consider how changes might affect annotations used by your operator or by other tools.
- Documentation: Clearly document any changes to labels or annotations used by your operator, including their purpose and any compatibility implications.
- Deprecation: If you need to remove or rename labels or annotations, follow a clear deprecation policy to give users time to adapt.

References

Partner with us to master Kubernetes API versioning. From implementing best practices to crafting seamless controller workflows, we’re here to innovate together.

If you would like to discuss any of these topics in more detail, please feel free to get in touch

Blog

Case studies, strategies, and ideas shaping modern technology.