Securing Helm

Nov 20 2017

There are four steps you should take if you are running Tiller (Helm's server-side component) in a cluster that has untrusted users or pods. These steps are done at installation time, and will substantially improve Helm's security.

The easiest way to install Tiller is with the helm init command. Run just like that, it will install a version of Tiller into the cluster. But the version it installs has permissions equivalent to root (if the cluster does not have RBAC enabled). To configure Tiller with higher security, you will need to add some additional command line flags to the helm init call, and you will need to create some roles and role bindings.

Update: Since the original version, the official documentation on RBAC was revised, and the link changed. This post has been updated accordingly.

1. Enable RBAC on Your Cluster, and Create Roles, SericeAccounts, and Role Bindings

Many Kubernetes clusters still do not have Role Based Access Control (RBAC) enabled. For security reasons, you should enable this. Each Kubernetes distribution has its own mechanism for enabling RBAC. Consult your distribution's documentation.

Once RBAC is on, you can create a service account for Tiller. And once that is done, you can create roles and role bindings to dictate what Tiller is allowed to do.

Relevant flags to helm init:

  • --service-account will allow you to specify the name of the service account that Tiller should use.

2. Enable TLS Between Helm and Tiller

By default, Tiller will accept any inbound gRPC requests, and will not seek to authenticate them. But it is strongly preferable to enable full TLS support.

When TLS is enabled, the Helm client will be expected to produce a certificate that can be verified by Tiller. Likewise, Tiller will send its certificate back the the client, which can optionally verify it. Using this mechanism, both the client and server can authenticate each other.

Relevant flags to helm init:

  • --tiller-tls: Enable TLS on Tiller
  • --tiller-tls-verify: Turn on Tiller's TLS with verification mode enabled. (You can disable verification mode, but we suggest not doing so).
  • --tiller-tls-cert: Tiller's certificate
  • --tiller-tls-key: Tiller's cryptographic key, which it will not share
  • --tiller-ca-cert: The CA for Tiller that can be used to verify client certificates

3. Configure Tiller to Save Releases as Secrets

For historical reasons, Tiller stores its release information in ConfigMaps. We suggest changing the default to Secrets.

Secrets are the Kubernetes accepted mechanism for saving configuration data that is considered sensitive. While secrets don't themselves offer many protections, Kubernetes cluster management software often treats them differently than other objects. Thus, we suggest using secrets to store releases.

Enabling this feature currently requires setting the --storage=secret flag in the tiller-deploy deployment. This entails directly modifying the deployment or using helm init --override=... (docs), as no helm init flag is currently available to do this for you.

4. Run One Tiller Per Team

By default helm init installs Tiller into kube-system. This namespace is considered to be the place for core and extension services to run. And the theory is that this one Tiller instance can serve the entire cluster.

For security, though, we recommend installing multiple Tiller instances, each in a different namespace. In this way, groups of Tiller users can be segregated, and RBAC permissions can be customized per team.

Note that simply installing Tiller into a namespace does not ensure that Tiller can only install into that namespace. Tiller will still need RBACs to control its permissions. A common pattern, in fact, is to install Tiller into its own namespace (staging-tiller) and then grant it permission to write into a target namespace (staging). That way, Tiller itself is not running in the namespace in which it is deploying releases.

Relevant flags for helm int:

  • --tiller-namespace: This flag allows you to specify which namespace to install Tiller into. All of Tiller's release records will also be stored in this namespace.

Other Considerations

Tiller in its current form (2.7.x) does not provide a way to map user credentials to specific permissions within Kubernetes. That is, Tiller will not let you say that for one particular Tiller server, user A has one set of rights, and user B has another. All Tiller operations on that server are executed using the Tiller pod's credentials and permissions. This situation may change in the future.

When Helm clients are connecting from outside of the cluster, the security between the Helm client and the API server is managed by Kubernetes itself. You may want to ensure that this link is secure. Note that if you are using the TLS configuration recommended above, not even the Kubernetes API server has access to the unencrypted messages between the client and Tiller.



comments powered by Disqus