It can be tricky to diagnose problems with running Kubernetes workloads. You might be lucky and find the cause in your application’s logs, via the kubectl logs command. In some cases there’s no avoiding a live debugging session though, where you interactively engage with your Pods to uncover issues.
The kubectl debug command simplifies these debugging tasks by providing a new ephemeral container inside your Pod. This can be used to inspect the Pod’s environment so you can start troubleshooting problems that are surfacing in your existing containers.
Preparing to Use Kubectl Debug
kubectl debug was launched with v1.18 of Kubernetes and Kubectl. It relies on ephemeral containers being available in your cluster. Ephemeral containers became a beta feature in Kubernetes v1.23 and are now enabled by default. You’ll need to manually enable the feature gate if your cluster runs an older Kubernetes version.
Ephemeral containers are designed for transitory tasks where you need to temporarily connect an extra container to an existing Pod. This is ideal for debugging operations where you want to accurately inspect a Pod without affecting live container instances.
Most container images lack debugging tools; installing them within a running container would mutate its environment and potentially cause side effects. Attaching an ephemeral container to your Pod is a safer way to debug that gives you a clean working environment. You can use a heavier image that includes all the tools you need.
Although ephemeral containers become part of their host Pod, there are still some differences to be aware of. Ephemeral containers don’t support port binds, probes, or resource reservations as they’re only temporary in nature. They’ll never be automatically restarted and can’t be changed once they’ve been created. A complete list of supported capabilities is available in the documentation.
Using Kubectl Debug
Before continuing, create a basic deployment to use for testing purposes:
Next use the get pods command to find the name of your deployment’s Pod:
Our deployment’s Pod is called nginx-55649fd747-qsnr2.
Now you can use the kubectl debug command to start a debugging session inside your Pod:
The command’s syntax is similar to a hybrid of kubectl create and kubectl debug. The unnamed argument supplied to the command identifies an existing Pod to attach to. The –image argument specifies the image to use for the new container. We’re using ubuntu:20.04 here to obtain access to the familiar commands included in the Ubuntu Linux distribution.
The -it flag is equivalent to –stdin –tty. Including these arguments will allocate a TTY to the container, attach to it, and connect your terminal’s stdin stream. This gives you an interactive shell inside your new container.
Now you can carry out your debugging tasks from within your ephemeral container.
Copying Pods
Another way to use kubectl debug is with a –copy-to argument. This creates a copy of the target Pod and adds the ephemeral container to the copy. The original Pod is left intact.
This feature gives you even greater assurance that changes made during debugging won’t directly impact your production application.
Copying the Pod also lets you activate process namespace sharing. This makes the existing processes in your Pod visible to your ephemeral container. It can’t be used with existing containers as their spec.shareProcessNamespace field will usually be set to false. Running kubectl debug with the –copy-to and –share-processes flag will enable process sharing on the copied Pod, making this procedure much more intuitive:
The process list visible to your ephemeral Ubuntu container will now include an NGINX process:
This process is still running in the separate NGINX container within your Pod. Namespace sharing also provides access to the target container’s filesystem via /proc:
Copying the Pod in this way is therefore a powerful debugging tool. You can readily inspect the Pod’s files and processes using a separate container that’s prepared with familiar tools.
Optional Arguments
The –copy-to flag always leaves the original Pod intact by default. You can make the operation act as a replacement instead using –replace. This will stop the first Pod.
Kubernetes will schedule the copied Pod to any available Node. This can be problematic if you want to ensure a consistent test environment. Adding –same-node will schedule the copy to the existing Pod’s Node, eliminating any differences that may exist between machines in your cluster.
Another useful option is –env to set extra environment variables inside your ephemeral container. You may need to use this to configure debugging tools or override values inherited from your target Pod.
Finally, remember that containers created by kubectl debug don’t have to be interactive. You can easily run one-off commands against your Pods using kubectl exec-like syntax. The –attach argument is supported to control whether your shell’s connected to the container when you’re not running with -i (–stdin).
Conclusion
Ephemeral containers and the kubectl debug command provide a simplified debugging experience for Kubernetes workloads. You can run commands inside a Pod using a different image to your regular containers. This lets you access debugging tools that aren’t included in your application’s image.
kubectl debug can also create copies of Pods and share their processes with the original. This mechanism lets you inspect the processes in the target Pod’s containers, from a separate ephemeral container that you’ve got full control over. It provides more advanced debugging options when you need to interrogate running processes.
