A good Kubernetes management workflow is a repeatable loop: check health, troubleshoot with the right tool, change things through a reviewed path, and run the maintenance that keeps clusters upgradable and recoverable. Day-two operations are everything that happens after the first deploy. This guide lays out that loop task by task, with a calendar you can copy and the points where most teams lose time.
TL;DR
- Split the work into four rhythms: daily checks, per-incident debugging, change management through Git, and scheduled maintenance (upgrades, certificates, backups).
- Observe first: metrics, logs and traces together give you the cluster picture before you touch anything.
- Keep access tight with RBAC and least privilege, and keep kubeconfigs off shared drives and out of cloud sync.
- Back up etcd and rehearse the restore, because an untested backup is only a hope.
- Cut the tool-switching: one native client for logs, shells, scaling, Helm and Argo CD removes much of the daily friction.
Table of Contents
- What a Kubernetes management workflow actually covers
- Build the daily loop: observe before you act
- Troubleshoot with a fixed escalation path
- Change management: Git first, kubectl for emergencies
- Access, secrets and security
- Scheduled maintenance: the day-two calendar
- Managing more than one cluster
- Perspective
- Put the workflow in one window
- FAQ
- Sources
What a Kubernetes management workflow actually covers
Day-one work gets a cluster running: provisioning, networking, the first deployments. Day-two work keeps it healthy for years. A workflow for it has five jobs:
- Observe: know the state of nodes, pods, the control plane and your applications.
- Troubleshoot: move from a symptom to a cause quickly.
- Change: ship configuration, scale workloads and roll back safely.
- Protect: access control, secrets, backups and audits.
- Maintain: upgrades, certificate rotation, capacity and cost review.
Most pain comes not from any single job but from the seams between them: a terminal for kubectl, a browser tab for the dashboard, another for Argo CD, a VPN client, an SSH session to a bastion. A workflow is partly a decision about which of those seams you can remove.
Build the daily loop: observe before you act
The Kubernetes project describes observability as collecting and analyzing metrics, logs and traces so you can understand cluster state, performance and health. Correlating those signals gives one picture across the control plane, add-ons and applications. That is the right order of operations: read the picture, then decide.
A practical daily pass takes ten to fifteen minutes per cluster:
- Nodes: any NotReady, memory or disk pressure, or unexpected cordons?
- Pods: crash loops, pending pods, rising restart counts, failed jobs.
- Control plane: API server latency and errors, scheduler and controller health.
- Alerts: review what fired overnight, and silence or fix noisy rules instead of ignoring them.
- Capacity: requests versus actual usage, so you spot a full node pool before a deploy does.
Probes matter here too. Readiness, liveness and startup probes tell Kubernetes whether a container should receive traffic or be restarted. Bad probes create most false alarms, so tune them as part of the daily loop, not once at launch.
The daily pass is where a unified client pays off most. When metrics, live logs and resource lists sit in one window, and switching from the production cluster to staging is a click, you will actually do the check every day.
Troubleshoot with a fixed escalation path
When something breaks, a fixed order beats improvisation. This one works for most workload incidents:
- Describe and events: read the pod's events for scheduling, image pull and probe failures.
- Logs: current logs first, then the previous container's logs if it restarted. Our guide to viewing Kubernetes logs covers kubectl, live tailing and what comes after.
- Shell into the pod: check environment, mounted files and DNS from the inside. See these pod shell commands for the fast routes.
- Ephemeral containers: when the image has no shell or debug tools, attach a debug container to the running pod. We explain how in debugging running pods with ephemeral containers.
- Connectivity: if the pod is fine but the service is unreachable, walk the network layer by layer with this connectivity troubleshooting guide.
- Port-forward: reach a service or database from your laptop to test it directly. Forwarding several ports at once has its own pitfalls, listed in kubectl port-forward with multiple ports.
Write the escalation path in your runbook and keep it short. An on-call engineer at 3 a.m. should not be inventing process.
Change management: Git first, kubectl for emergencies
The most durable pattern for configuration is GitOps: the desired state lives in Git, and a controller such as Argo CD or Flux reconciles the cluster toward it. You get an audit trail, review before merge, drift detection and rollback by revert.
A few rules keep this honest:
- Treat manual edits to a live cluster as an exception that you either commit back or revert within the day.
- Package third-party software as Helm charts, and let the GitOps controller own the releases. Here is how to deploy Helm charts through Argo CD.
- Scale through the declared state when it is a lasting change, and through a direct command when it is an incident response. The five scaling commands operators use are worth keeping handy.
- Keep staging and production on the same pipeline so a change is proven once before it ships twice.
Even in a strict GitOps shop, engineers need to see what the controller did: sync status, diffs, live manifests, pod state. Having Argo CD and Helm operations a click away in the same client as your logs shortens that loop.
Access, secrets and security
The Kubernetes documentation calls RBAC a key security control and advises assigning minimal rights to users and service accounts, only what each role needs. Turn that into habits:
- Bind roles to groups, not individuals, and review bindings on a schedule.
- Avoid wildcard permissions and avoid cluster-admin for daily work.
- Use short-lived credentials where your platform supports them, rather than static tokens.
- Keep policy as code, so admission rules are reviewed like any other change.
- Treat kubeconfig files as credentials, because they are. Store them locally, protected, and never in a chat thread or a synced folder.
The kubeconfig point is where desktop tooling matters. A client that keeps kubeconfigs in secure local storage, with no cloud sync and no telemetry, keeps your cluster access on your machine. For clusters behind a bastion, an SSH tunnel built into the client avoids the pile of terminal windows and half-remembered port numbers; see how admins copy tunnel commands and fix certificate errors.
Scheduled maintenance: the day-two calendar
The tasks below recur in nearly every operations checklist: calendar upgrades, add-on updates, rightsizing, certificate rotation, incident review and restore drills. Give each an owner and a cadence. A starting calendar:
| Cadence | Task | Why it matters |
|---|---|---|
| Daily | Health pass, alert review | Catches drift before users do |
| Weekly | Review restarts, pending pods, failed jobs; check pending add-on updates | Finds slow-burn problems |
| Monthly | Rightsizing of requests and limits; cost review; RBAC binding review | Controls waste and privilege creep |
| Quarterly | Security audit, certificate expiry check, restore drill | Proves you can recover |
| Per release cycle | Cluster upgrade, staging first | Keeps you inside supported versions |
Treat this as a template, not a standard. Adjust cadences to your risk and team size, and put the tasks in a shared calendar or ticketing queue so they do not depend on memory.
Upgrades without surprises
Upgrade staging first, read the release notes for removed APIs, and move the control plane before the worker nodes. Upgrade nodes one at a time or in small batches, draining each so workloads reschedule cleanly, and do not skip minor versions. Rolling, blue-green and canary strategies all work; pick the one that matches how much spare capacity you can afford.
Backups you have actually restored
The Kubernetes documentation notes that backing up etcd is what lets you recover a cluster from disaster, for example after losing every control plane node. Managed services may handle this for you, so confirm what your provider covers. Wherever you run the control plane yourself, schedule snapshots, store them off the cluster, and run the restore drill every quarter into a scratch environment. Back up application data too, including persistent volumes, since etcd holds cluster state, not your databases.
Managing more than one cluster
Most teams end up with development, staging and production clusters, then regional or per-customer ones. The workflow problems multiply: which context am I in, which credentials apply, how do I reach a private API server?
Three habits help:
- Name contexts consistently and show the active one prominently. Wrong-cluster commands are a classic outage source.
- Standardize the stack across clusters, with the same ingress, monitoring and policy baseline, so runbooks transfer.
- Centralize reach, not credentials: one client that holds all your connections locally beats sharing admin files around.
For fleet-level provisioning, see our look at Cluster API for platform teams, and for GitOps across several clusters, Argo CD across clusters, including how to avoid static tokens.
Perspective
Most Kubernetes tooling advice assumes the hard part is a missing feature. In our experience the hard part is friction: the fifth terminal tab, the VPN that drops mid-tail, the kubeconfig someone pasted into a wiki.
That is why we built Kubezilla Desktop as a native app in Rust, designed for low memory use and fast startup, with kubeconfigs stored securely on your machine and no cloud sync or telemetry. We want a cluster to be one click away, including the ones behind SSH tunnels, so the daily loop of logs, metrics, shells, scaling, Helm and Argo CD happens in one place.
A workflow is only as good as the number of times you can run it without dreading it. We design for that.
Margus
Put the workflow in one window
Kubezilla Desktop is a native app for Linux, macOS and Windows that brings multi-cluster access, SSH tunneling, live log streaming, metrics, pod terminals, YAML editing, workload scaling, PVC file browsing, and Helm and Argo CD operations into one interface. It costs €5 per month.
If the daily loop above sounds like what your team already does across five tools, try Kubezilla Desktop and see how it feels with everything in one place. You can also read about it as a Kubernetes client for your own setup.
FAQ
What are Kubernetes day-two operations?
Day-two operations are the recurring tasks after a cluster is live: monitoring, troubleshooting, scaling, access control, upgrades, certificate rotation, backups and cost review. Day one is building and deploying. Day two is the long period of keeping it reliable and secure.
How often should we upgrade a cluster?
Plan upgrades on a regular calendar rather than waiting for an end-of-support deadline. Check your Kubernetes distribution or cloud provider's official page for the currently supported versions and their schedule. Always test in staging first and avoid skipping minor versions.
Do we need GitOps to manage Kubernetes well?
No, but it helps a lot. Declaring state in Git gives you review, an audit trail, drift detection and easy rollback. Teams without it can start with one application and a tool such as Argo CD, and still keep kubectl available for incident response.
What is the first thing to check when a pod fails?
Start with the pod's events and current logs, then the previous container's logs if it restarted. Those two views explain most scheduling, image and probe failures. If they do not, shell into the pod or attach an ephemeral debug container.
Sources
- Observability | Kubernetes
- Role Based Access Control Good Practices | Kubernetes
- Operating etcd clusters for Kubernetes | Kubernetes