Kubernetes namespace Terminating: prove cleanup

A namespace stuck in Terminating is evidence that deletion has not finished. It is not, by itself, evidence that Kubernetes needs a stronger delete command. Start with the namespace conditions and the first API request that fails. Then separate discovery, remaining objects and finalizer ownership before changing any of them. ^[1][2]

This guide follows a fictional preview environment called preview-saffron. Its application Deployment has gone, yet the namespace remains. The team removed a database operator during teardown, and a custom resource still represents a database outside the cluster. The names are illustrative. No customer incident or live cluster experiment is implied.

The immediate goal is to learn which cleanup obligation remains and who can complete it. Removing the namespace name from the API is a different outcome. That distinction matters when a database, volume, cloud resource or background process can survive the object that described it. A recovery record should account for both outcomes.

What is the namespace still waiting for?#

The namespace controller's conditions identify which part of deletion it could not complete during its last reconciliation. ^[2]

Read the namespace document before repeating deletion. Save the output in a restricted incident directory if it contains names or annotations you should not share. Capture its UID, deletion timestamp, lifecycle finalizers and conditions. The UID matters because a later namespace with the same name is a new object, not proof about the old object's cleanup.

# Read-only evidence collection; replace the illustrative name.
namespace='preview-saffron'
kubectl get namespace "$namespace" -o yaml
kubectl describe namespace "$namespace"

The condition messages tell you which branch to investigate. Discovery failure means the namespace controller cannot obtain a trustworthy view of the API surface. Content failure can identify a denied operation, conversion problem or webhook. Content remaining identifies resource kinds still present. Finalizers remaining points toward cleanup obligations on those resources. Read the message as well as its boolean value. ^[2]

The last row is deliberately cautious. Conditions report observations from a controller reconciliation. They are not a permanent certificate that every dependency is healthy now. In the fictional case, content and finalizer conditions identify a database custom resource. That narrows the investigation without proving that its external database has already been deleted. ^[2]

Can you prove the inventory is complete?#

kubectl get all does not enumerate every resource kind. An operator's custom resources can remain after the familiar workloads disappear. The inventory must discover listable namespaced resource types, then successfully list each type in the affected namespace. A permission denial is a gap in the inventory, not an empty result. ^[2]

Use an identity authorized to inspect the relevant resources. Do not broaden permissions just to make a script green; ask the cluster owner to supply missing observations when necessary. The following Bash function is an illustrative read-only inventory. It stops at the first failed operation and leaves the failure visible. It is not a deletion tool.

inventory_namespace() {
  local namespace="$1" resources resource
  resources=$(kubectl api-resources --verbs=list --namespaced \
    -o name --request-timeout=30s) || return 1
  while IFS= read -r resource; do
    [ -n "$resource" ] || continue
    printf '\nResource kind: %s\n' "$resource"
    kubectl get "$resource" -n "$namespace" \
      -o name --request-timeout=30s || return 1
  done <<< "$resources"
}
inventory_namespace 'preview-saffron'

Run it alongside the namespace discovery conditions. Successful client discovery alone does not erase an error recorded by the namespace controller. Investigate the named API and repeat both observations after repair. If a command times out, preserve the resource kind and error. Do not add an ignore-error flag and describe the resulting shorter output as a complete inventory. ^[2][2]

There is a practical reason to print the kind before listing it: the failure stays attached to the question the command was answering. A log containing only a timeout leaves the next responder guessing which API failed. Conversely, a clean empty list for a specific kind is useful evidence about that kind at that time.

An unavailable aggregated API is a separate branch from an unhealthy custom-resource operator. Inspect the named APIService and its backing service when the condition points there. Restoring a required API lets the controller discover and inspect its resources again. Removing a registration may be correct for a permanently retired API, but it needs an ownership decision about the data that registration represented. ^[2]

Restore the cleanup owner#

In the worked case, the inventory finds databaseclaims.example.com/app. That group and kind are fictional. Read its full document, including managed fields if useful, and its events. Record the resource UID, deletion timestamp, finalizer keys and owner references. Do not infer the external cleanup procedure from the spelling of the finalizer alone. ^[1][2]

# Fictional custom-resource kind; use the discovered kind in your cluster.
kubectl get databaseclaims.example.com app \
  -n preview-saffron -o yaml --show-managed-fields
kubectl describe databaseclaims.example.com app -n preview-saffron

Match each remaining object to its controller and record what external resource its finalizer protects.

Suppose the finalizer promises to remove a preview database and revoke its role. An empty Kubernetes Deployment list tells you neither happened. Find the operator's supported deletion behavior, its configured external target and the logs associated with this object's identity. If the operator is missing, determine whether its supported recovery installation can run in an active management namespace and still watch the deleting resource. ^[2]

That recovery location matters. A terminating namespace is a poor place to try to create replacement controller Pods. Restoring a controller is also not just starting any newer image with similar permissions. Confirm the resource version it understands, its watch scope, required secrets and external credentials, and whether multiple controller instances would compete. Follow that operator's recovery guidance. ^[2]

Owner references provide another part of the explanation. Kubernetes garbage collection uses ownership relationships, not merely a matching label, to identify dependents. A foreground-deleting owner may remain while blocking dependents remain. Inspect the dependents' finalizers and API accessibility as well as the owner's. A clean-looking owner document cannot substitute for evidence about the object blocking it. ^[3]

For the fictional database claim, the preferred path is to restore the responsible operator, observe its cleanup attempt, repair the external permission or connectivity failure, and let it remove its own finalizer. Preserve the external database identifier before the claim disappears. That identifier is how the team later distinguishes a successful cleanup from an unrelated database with a similar name.

When is manual finalization justified?#

Manual intervention starts with a written cleanup decision. Identify each remaining finalizer, the work it represents, the evidence that work completed and the owner accepting any residual obligation. A retired controller can make manual cleanup necessary. An unreadable API does not make it complete. Kubernetes explicitly warns against clearing finalizers without understanding and satisfying their purpose another way. ^[1]

Keep object cleanup and namespace finalization separate. Removing a contained object's metadata finalizer addresses that object's deletion condition. Clearing namespace spec.finalizers through the finalize subresource addresses namespace lifecycle finalization. Neither action performs the missing external cleanup, and one is not a shorthand for the other. ^[1][2]

This guide intentionally stops short of a copy-and-run force-finalization command. The correct mutation depends on the remaining field, object identity and completed obligations. Use the documented procedure for your platform after those facts are established. Preserve the current resource version, handle conflicts by rereading, and change only the finalizer whose purpose you have satisfied. ^[2]

A broad patch that clears every key can remove protections unrelated to the original blocker. If storage protection remains, first determine whether a Pod still uses the claim and whether its process can still write. If an admission or conversion webhook fails, repair that service or follow a supported retirement procedure. Deleting a CRD to clean one namespace can affect custom resources elsewhere in the cluster. ^[2]

Force-finalizing a namespace can leave resources behind that are difficult to access. That is why the acceptance decision must cover discovery and each list request, the remaining inventory and the work represented by remaining finalizers. The namespace controller's inability to see an object is not evidence that the object ceased to exist. ^[2][2]

What evidence closes the incident?#

Return to the fictional case. The operator now reports that the matching external database is absent and the preview role is revoked. The claim disappears after its controller removes the finalizer. A fresh inventory completes successfully, and the namespace subsequently returns NotFound. Record each observation with its time rather than collapsing them into a single green check.

If the namespace disappears first, keep the external cleanup task open. If the external database is gone but inventory still fails on a different API, the namespace investigation remains open. These are useful partial results. They become misleading only when the incident record treats one as proof of the other.

Close the incident only when the namespace is gone and the cleanup obligations have evidence beside them.

Rehearse the observation procedure in a disposable environment before putting it in an on-call runbook. Include an API discovery failure, a list denial, a custom object with a pending finalizer and a controller removed too early. The expected response is a specific blocked evidence branch, not a script that reports an empty namespace in every case. No live cluster execution is claimed here.

The preventive change follows from the mechanism: stop creating custom resources, delete them while their operator and external access still exist, observe cleanup, and retire the operator afterward. Keep external identifiers and teardown ownership in the deployment record. A namespace is a container for resource names; it is not a substitute for the lifecycle of everything those resources control.

Preserve the next responder's evidence#

The incident handoff should identify the namespace UID, the last successful discovery, each failed resource kind, the remaining object UIDs and the controller expected to remove each finalizer. Beside an external cleanup obligation, record its exact provider identifier and the observation that establishes completion. Keep credentials and resource bodies out of broadly shared tickets.

Make the handoff explain what would change the next action. A recovered API means repeat inventory; a restored operator means observe its reconciliation; proof of completed external cleanup permits reviewing a narrow finalizer mutation. This is more useful than a transcript of commands with no interpretation. It also prevents a later responder from treating an earlier timeout as evidence of absence.

Frequently asked questions#

Does kubectl get all prove the namespace is empty?

No. That command covers a useful subset of resource types, not every namespaced API resource. Discover listable namespaced types and inspect each one. A failed discovery or list request makes the inventory incomplete, even when other commands print no objects. ^[2]

Are namespace finalizers stored in the usual metadata field?

A namespace can have metadata finalizers, and namespace lifecycle finalization also uses spec.finalizers through the finalize subresource. Finalizers on objects inside the namespace are another separate set. Read the actual field and condition before choosing a recovery operation. ^[2]

Should I delete an unavailable APIService?

Restore its backing API when it is still required. Removing the registration is appropriate only after confirming that the API is retired and understanding the remaining data. A False availability condition identifies a dependency to investigate, rather than permission to remove it. ^[2]

Does removing a finalizer finish external cleanup?

No. A finalizer is a condition on deletion; removing its key does not perform the work that its controller normally does. Before manual removal, identify that work and complete it another way. Keep the evidence even after the Kubernetes object disappears. ^[1]