Key Rotation and Secret Management in Practice

A Story: The Master Key

Imagine you are the building manager for a large office tower. On your first day, you are given a master key that opens every door — the front entrance, every office, the server room, the safe, the maintenance closet. You use it every day.

Five years pass. Dozens of employees have borrowed your key. A contractor made a copy. You lost it briefly and found it under a pile of papers. An ex-employee might still have a duplicate. The key is worn from constant use.

Would you still trust that key? Would you use it to secure the building?

Of course not. Too many people have touched it. Too much time has passed. The risk is enormous.

This is the problem with long-lived secrets in software.

An API key that never expires. A database password set when the system was deployed five years ago. A service account key that has been copied to a dozen configuration files. These secrets accumulate, get shared, and get forgotten. They become liabilities.

Key rotation is the process of replacing secrets with new ones on a regular schedule. Secret management is the practice of storing, accessing, and auditing secrets securely. Together, they are the difference between a system that stays secure and one that quietly decays into vulnerability.

This article explains why secrets must be rotated, how to manage them securely, and how to automate rotation so it happens reliably — without breaking your systems.

Part One: What Is a Secret?

A secret is any piece of sensitive information that grants access to a system, service, or data.

Secrets are everywhere in modern systems: passwords (both user and service account), API keys, access tokens (JWTs, OAuth tokens, session tokens), TLS certificates and their private keys, database credentials, SSH keys, encryption keys, and cloud credentials such as AWS access keys, Azure service principals, and GCP service account keys.

Every secret is a potential entry point for an attacker. Every secret must be protected, tracked, and eventually replaced.

Part Two: The Secret Lifecycle

Secrets are not static. They are born, they live, and they die. Understanding that lifecycle is essential to managing secrets effectively.

Generation. The secret is created using a cryptographically secure random number generator. Never use predictable values like a timestamp or a dictionary word. For API keys, use at least 128 bits of randomness. For encryption keys, follow the algorithm's key size recommendations — 256 bits for AES.

Distribution. The secret is delivered to the systems that need it. This is the most vulnerable stage. Secrets sent over email, Slack, or stored in plaintext configuration files are immediately at risk. Use secure distribution methods instead: a secrets management tool, encrypted configuration injected at deployment time, or environment variables set by a trusted orchestration system.

Storage. The secret must be stored securely at rest. Never store secrets in code repositories, configuration files tracked by version control, or unencrypted environment variables. Use dedicated secret management systems that provide encryption at rest, access controls, and audit logging.

Access. The secret is used by applications, services, or users. Every access should be logged and auditable. Secrets should be transmitted only over encrypted channels. Applications should fetch secrets at runtime, not embed them at build time.

Rotation. The secret is replaced with a new one. Rotation should happen on a regular schedule and immediately after any suspected compromise. Critically, the old secret and the new secret must coexist during a transition period to avoid outages — more on this in the challenges section.

Revocation. The old secret is deactivated. Revocation must be tested. A revoked secret that still works is a security hole, not a safety net. After revocation, the secret should be permanently deleted or archived under strong access controls.

Destruction. The secret is permanently deleted from all systems — including backups, logs, and archives. Secure deletion ensures that even forensic recovery is not possible.

Part Three: Why Long-Lived Secrets Are Dangerous

The longer a secret exists, the more opportunities it has to be compromised.

Exposure over time. A five-year-old API key has likely been copied to configuration files, backed up to multiple locations, injected into CI/CD pipelines, and possibly shared with contractors or partners. Every copy is an additional risk vector. Every year of existence increases the chance of exposure.

Former employees. People who leave the company may retain copies of secrets. They might have downloaded configuration files, saved API keys in personal password managers, or written credentials down. Long-lived secrets give former employees a permanent backdoor — one that does not disappear when their accounts are deprovisioned.

Logs and backups. Secrets leak into logs through debug output, error messages, or command-line arguments. They are captured in backups stored for years. A secret that never expires remains vulnerable in every log file and every backup snapshot where it ever appeared.

Orphaned credentials. Teams accumulate secrets over time. Old secrets for systems no longer in use are rarely cleaned up. They remain in configuration files, commented out in code, or stored in forgotten services. These orphaned secrets are rarely monitored but often still grant access.

Undetected compromise. If a long-lived secret is stolen, an attacker may have access for months or years before the breach is discovered. The longer the secret lives, the larger the window of damage.

The principle is simple: secrets should have the shortest possible lifetime that is operationally practical. A secret that rotates daily is less risky than one that rotates monthly. A secret that rotates monthly is less risky than one that rotates yearly. A secret that never rotates is the most dangerous of all.

Part Four: What Is Key Rotation?

Key rotation is the process of replacing an existing secret with a new one and deactivating the old one.

What rotation achieves:

It limits the exposure window. If a secret is compromised, rotation ensures an attacker only retains access until the next scheduled rotation. Daily rotation limits the window to 24 hours; monthly rotation to 30 days.

It supports compliance. Many regulations — PCI-DSS, HIPAA, SOC 2 — require regular key rotation. Even outside a regulated context, rotation is an accepted security best practice.

It forces dependency hygiene. Rotation requires knowing which systems use each secret. This naturally drives teams to maintain an accurate inventory of secret dependencies, which is valuable on its own.

It enables emergency response. If you already have a rotation process, responding to a suspected compromise is straightforward: rotate the affected secret immediately using the same process you have already tested.

What rotation does not fix:

Poor storage. Rotating a secret that is stored in plaintext in a public repository does not help. Fix the storage problem first.

Weak generation. Rotating a weak secret to another weak secret does nothing for security. Ensure secrets are strongly and randomly generated.

Broken revocation. If old secrets are not properly deactivated, rotation is theater. Always test that revoked secrets actually stop working.

Part Five: Why Rotation Is Hard in Practice

Rotation sounds simple. In practice, it is one of the hardest operational tasks in security engineering.

Dependency discovery. Before rotating a secret, you need to know everywhere it is used. Which applications depend on this database password? Which CI/CD pipelines use this API key? Which developers have this credential in their local environments? Without an accurate inventory, rotation will break something in production.

Zero-downtime rotation. You cannot change a secret everywhere simultaneously. The old secret must remain valid until all dependent systems have updated to the new one. This requires a grace period — a window where both the old and the new secrets are accepted.

Distributed systems. In a microservices architecture, a single secret may be used by dozens of services across multiple regions and deployment environments. Coordinating updates across all of them, accounting for services that may be offline or mid-deployment, adds significant operational complexity.

Legacy systems. Some systems cannot read secrets from environment variables or secret management APIs. They require credentials in configuration files read at startup. Rotating a secret for a legacy system may require a restart — which may mean a maintenance window.

Human factors. Developers often store secrets in local .env files, hardcode them in test scripts, or save them to personal password managers. Rotating a secret that has propagated into local development environments requires coordinating with every affected developer.

Emergency rotation. Planned rotation is hard enough. Emergency rotation — rotating immediately in response to a suspected breach, under pressure, without a maintenance window — is much harder. Teams that have never practiced rotation under calm conditions will struggle when it counts.

Part Six: Automation Is the Only Viable Solution

Manual secret rotation does not scale. For any system of meaningful size, rotation must be automated end-to-end.

Automation requires four foundational capabilities.

Centralized secret storage. All secrets must live in a single system that supports versioning, access logging, and programmatic retrieval. This is the source of truth for the entire rotation workflow.

Programmatic access. Applications must fetch secrets at runtime by calling the secrets API, rather than embedding credentials in code or build artifacts. This is what makes rotation transparent to the application — the application always gets the current secret without any deployment.

Scheduled rotation. The secret management system generates new credentials on a defined schedule, updates the target system (the database, the API provider), and makes the new secret available before revoking the old one.

Notification and coordination. Dependent systems must know when new secrets are available. This can be pull-based (applications poll for the latest version on a schedule) or push-based (the secrets system sends webhooks or triggers restarts). The approach depends on your architecture.

Here is how automated rotation works for a database password in practice:

The secret management system generates a new database password and updates the database user so that both the old and new passwords are accepted. It notifies dependent applications that a new version is available. Those applications fetch and cache the new credential. After a grace period — typically 24 hours — the old password is removed from the database. Any application that has not yet updated loses access, which is the correct and expected behavior.

For an API key, the flow is similar: the new key is created and marked active, the old key is put into a deprecation state where it still works, clients are notified to update, and after the transition window the old key is revoked.

Part Seven: Secret Management Tools

Do not build your own secret management system. Use an established tool.

HashiCorp Vault is the most widely used open-source solution. It stores secrets, enforces access policies, and supports dynamic secrets — credentials generated on demand that automatically expire after a short window. Vault can rotate database passwords and cloud credentials automatically. It is extremely flexible but requires significant operational expertise to run well. It is best suited for organizations with a dedicated platform or infrastructure security team.

AWS Secrets Manager is a fully managed service that integrates deeply with RDS, Redshift, and DocumentDB for native database credential rotation. It is straightforward to operate but is AWS-only and more expensive than self-hosted alternatives. It is the right choice for AWS-centric shops that want rotation without the operational overhead.

Azure Key Vault and Google Cloud Secret Manager are the equivalent managed offerings for their respective clouds. Both provide storage, access controls, and audit logging, with deep integrations into the surrounding platform. Like AWS Secrets Manager, they are the natural choice if you are operating primarily within their ecosystem.

Kubernetes Secrets are worth understanding even if you use one of the above. They are the native mechanism for delivering secrets to pods, and most production Kubernetes environments use them in combination with an external secrets provider. Kubernetes Secrets are not encrypted by default and have no built-in rotation — additional tooling such as the External Secrets Operator is required to bridge a managed secrets store into the cluster.

One capability worth highlighting specifically is dynamic secrets, which Vault pioneered. Rather than storing a long-lived database password and rotating it periodically, dynamic secrets work differently: each application requests a credential when it needs one, receives a unique set of credentials scoped to that request, and those credentials expire automatically after a short TTL — often minutes or hours. The credential never needs to be rotated because it never lives long enough to become a liability. This is the direction the industry is heading for high-sensitivity access.

Part Eight: Rotation Strategies

Different secrets warrant different approaches to rotation.

Scheduled (time-based) rotation is the simplest and most common strategy. Secrets are replaced on a fixed cadence — every 30 days, every 90 days, annually. The schedule should balance security (shorter is better) against operational cost (more frequent rotation requires more robust automation).

Event-driven rotation ties rotation to specific triggers: an employee departure, the end of a contractor engagement, a detected security incident, or a third-party breach notification. This does not replace a scheduled rotation cadence — it supplements it.

Usage-based rotation replaces a secret after a certain number of uses or after a certain volume of data has been processed. This is most common for encryption keys in high-throughput systems, where the cryptographic risk model is tied to data volume rather than time.

Just-in-time (JIT) secrets are generated on demand and expire after a short period — minutes or hours. The application requests a credential when it needs one and discards it when done. JIT secrets eliminate the concept of a "long-lived" credential entirely. They require solid infrastructure support but represent the most secure posture available.

Zero-touch rotation is the operational goal for mature systems. The secrets management platform handles generation, distribution, rotation, and revocation automatically. Applications fetch secrets at runtime. No human intervention is required in steady state. Humans are involved only in defining policies and responding to exceptions.

Part Nine: Recommended Rotation Frequencies

There is no single correct answer. The right frequency depends on the secret type, the sensitivity of the protected resource, and your automation maturity. The table below offers starting points based on common industry practice.

User passwords should rotate every 90 to 180 days when multi-factor authentication is in use. Forcing very frequent rotation without MFA tends to encourage password reuse and weak choices, which trades one risk for another. With MFA, a longer rotation window is acceptable.

Service account passwords and API keys warrant a 30 to 90-day cycle. Service accounts are often highly privileged, and API keys are frequently exposed in logs, client-side code, and configuration — making shorter rotation windows valuable.

TLS certificates should be rotated at 90 days or less. Let's Encrypt normalized this cadence, and it is now the industry standard. The short lifetime drives automation, which is the point.

Database and cloud access credentials also fit a 30 to 90-day window. Database access is high-value, and cloud access keys provide broad blast radius.

Encryption keys used for data at rest can rotate on a longer cycle — one to two years — because key rotation requires re-encrypting the data they protect, which carries real operational cost and risk. The rotation interval should reflect that cost while still maintaining a meaningful security posture.

The broader industry trend is toward shorter lifetimes across the board. The limiting factor is automation: fully automated rotation makes shorter cycles trivially achievable, while any manual step in the process creates pressure to lengthen the interval. Invest in automation first.

Part Ten: Secrets Across the Stack

Secrets appear at every layer of a modern system, and a compromise at any layer can cascade upward or downward.

At the infrastructure layer, database credentials, cloud API keys, and SSH keys underpin everything else. Compromise here often means unrestricted access to the data plane.

At the application layer, session keys, JWT signing keys, and encryption keys for data at rest protect the confidentiality and integrity of user data. Rotation limits the blast radius of a key compromise.

At the identity layer, service account credentials, OAuth client secrets, and directory bind credentials authenticate services to each other. A compromised service identity can lead to privilege escalation across multiple systems.

At the operational layer, CI/CD pipeline secrets, deployment keys, and monitoring system credentials grant access to the systems that build and observe everything else. Compromise here can enable supply chain attacks — malicious code inserted into the build pipeline rather than the application itself.

Every secret in this stack is a potential break point. Each one must be managed, rotated, and eventually destroyed. A chain is only as strong as its weakest credential.

Part Eleven: What to Take Away

After reading this article, you should have a clear mental model of the secret lifecycle — generation, distribution, storage, access, rotation, revocation, destruction — and understand why each stage matters.

You should be able to explain why long-lived secrets are dangerous: exposure accumulates over time, former employees retain access, secrets leak into logs and backups, orphaned credentials are never cleaned up, and compromises go undetected far longer than they should.

You should understand why rotation solves these problems — and what it does not solve. Rotation limits the exposure window. It does not fix poor storage, weak generation, or broken revocation. Fix those problems first, then rotate.

You should recognize that automation is not optional. Manual rotation does not scale and will not be done consistently. Invest in the tooling — Vault, a cloud-managed secrets service, or an equivalent — that makes rotation a background process rather than a fire drill.

And most importantly: secret management is not a one-time setup. It is an ongoing operational discipline. Secrets must be rotated. Access must be audited. Orphaned credentials must be cleaned up. A system that was secure last year is not secure this year if nothing has changed.

Closing: The Master Key Revisited

On your first day as building manager, you were given a key that opened every door. Five years later, it had passed through dozens of hands. Copies had been made. It had been lost and found. It was worn and tired.

Would you still trust that key? No.

So why do we trust API keys that are five years old? Why do we trust database passwords set at initial deployment and never changed? Why do we trust signing keys whose original owners left the company years ago?

Every secret is a key. Every day it exists, the risk it carries grows. Rotate your secrets. Automate the process. Log access. Destroy old credentials. And when a secret is compromised — because eventually one will be — know that the window of exposure is short because your rotation cadence made it so.

The master key is a liability. Every secret in your system is the same. Treat them accordingly.