Kubernetes Namespace Stuck in Terminating State

Photo by Daniel Tausis on Unsplash

Thursday morning IST and Slack started buzzing. I suspected something wrong and I was right, Nginx Ingress on K8s was throwing 503 Service Unavailable.

Started debugging and suddenly I pushed myself into more trouble as by mistake I deleted the namespace (Please be very careful don't ever do this in Prod K8s cluster, I was lucky I made mistake but it was my QA cluster but again we always think about the end user in case of QA environment its QA Engineering team.)

After waiting for few minutes I found out that namespace is in Terminating state from so long and which means I cannot recreate the same namespace and cannot deploy the Nginx Ingress.

Cause of this might be some Extensions which are not managed by namespace not getting deleted and hence the name space deletion got stuck.

Solution:

Delete the namespace manually

1. Get the namespace that is stuck in terminating state

kubectl get namespaces

2. Create a temporary json

kubectl get namespace aaa -o json >tmp.json

3. Edit the tmp.json. Remove the kubernetes from the finalizers and save the file.

"spec": {
         "finalizers": [    "kubernetes"                                                                      ]
      },

Read more about the Finalizers .

4. Connect the Proxy

kubectl proxy

5. Make an API call with your proxy IP and port

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/aaa/finalize

6. Check the namespace back. You will see not more namespace stuck in terminating state.

kubectl get namespaces