Project Name
Secured 200+ Microservices: A Retail Platform’s HashiCorp Vault–Kubernetes Integration Story
Our client is a fast-growing U.S.-based eCommerce company running over 200 microservices across multiple Kubernetes clusters supporting production, staging, and regional operations. The platform processes millions of daily transactions and relies on a large volume of database credentials, API keys, and service authentication tokens.
As the environment expanded, native Kubernetes Secrets became difficult to manage, offering limited visibility, manual rotation processes, and increasing security concerns. To establish a scalable and enterprise-grade secrets management framework, the company engaged Ksolves to implement HashiCorp Vault with the Agent Injector pattern, enabling centralized, secure, and automated secret delivery across its Kubernetes infrastructure.
Two hundred microservices. No encryption at rest. No audit trail. And a rotation process so disruptive that credentials had not been changed in over a year.
- No Encryption at Rest and Limited Secret Security Across 200+ Services: Kubernetes Secrets were only base64-encoded and stored in etcd without robust encryption at rest. Anyone with namespace-level access could retrieve secrets in plain text, with no visibility into who accessed them.
- Secret Rotation Was Complex and Highly Disruptive: Rotating a database password, API key, or certificate required updating the secret and redeploying every dependent service. As a result, credential rotation was frequently delayed, leaving some secrets unchanged for over a year.
- Secrets Were Manually Managed Across Multiple Clusters: Production, staging, and regional clusters maintained separate copies of the same secrets. Without centralized synchronization, configuration drift became common, creating operational and security risks.
- No Audit Trail for Secret Access: The platform lacked a mechanism to track which service, pod, or user accessed a specific secret. This made security investigations and incident response difficult and time-consuming.
- Excessive Operational Overhead for the Platform Team: Creating, updating, and maintaining secrets across hundreds of services consumed a significant portion of the platform team's capacity, generating frequent support requests and deployment issues.
- Lack of Granular Access Control: Secret access was governed by broad namespace-level permissions rather than service-specific policies. A compromised service could potentially access credentials belonging to other applications within the same namespace.
Ksolves, an AI-first DevOps consulting services company, implemented HashiCorp Vault with the Vault Agent Injector to establish centralized, secure, and scalable secrets management across the client's Kubernetes ecosystem. Every microservice was mapped to a dedicated least-privilege Vault policy, while secrets were delivered directly to workloads through Vault Agents, enabling automated rotation, granular access control, and complete auditability. The entire Vault configuration was managed as code using Terraform and Git-based workflows.
- HashiCorp Vault Deployed with Agent Injector Across All Clusters: Vault was deployed using the official Helm chart with the Agent Injector enabled. Pods requiring secrets were automatically injected with Vault Agent containers, allowing secure secret retrieval without modifying application code or container images.
- Service-Level Authentication and Least-Privilege Access Controls: The Kubernetes Auth Method was configured to authenticate workloads using ServiceAccount identities. Each microservice was assigned a dedicated Vault role and policy, ensuring access only to its own secrets and eliminating unnecessary privilege exposure.
- Secrets Delivered Directly to Pods Through Vault Agent Templates: Vault Agent templates rendered database credentials, API keys, and certificates directly into application pods as files. This removed the dependency on Kubernetes Secrets and centralized secret storage within Vault.
- Automated Secret Rotation Without Service Redeployments: Vault Agent sidecars continuously monitored secrets for updates and refreshed them automatically. Credentials could be rotated without restarting pods, coordinating deployments, or impacting application availability.
- Vault Configuration Managed as Code with Terraform: Policies, authentication roles, secret engines, and cluster integrations were defined in Terraform and stored in Git. This provided version control, peer-reviewed changes, and consistent configuration across environments.
- Dynamic Database Credentials for Enhanced Security: Vault's Database Secret Engine was implemented for PostgreSQL and Redis workloads, generating short-lived credentials on demand. Static database passwords were eliminated, reducing credential exposure and improving auditability.
Technology Stack
| Category | Technology |
|---|---|
| Secrets Management | HashiCorp Vault |
| K8s Integration | Vault Agent Injector |
| Authentication | Kubernetes Auth Method |
| Infrastructure | Helm + Terraform |
| Dynamic Secrets | Vault Database Secret Engine |
| PKI / TLS | Vault PKI Secret Engine |
From manually managed, rarely rotated Kubernetes Secrets to a fully automated Vault-powered secrets platform with continuous rotation, complete auditability, and significantly lower operational overhead.
- 70% Reduction in Secret-Management Effort: Automated secret delivery and rotation eliminated most manual secret-management tasks, allowing the platform team to reclaim approximately 70% of the time previously spent on updates, rotations, and support requests.
- Zero-Redeploy Secret Rotation Across 200+ Services: Credentials can now be rotated directly in Vault and propagated automatically to running workloads without redeployments, downtime, or operational coordination.
- Complete Audit Visibility for Secret Access: Vault audit logs provide a detailed record of secret reads, token issuance, and credential renewals, strengthening security investigations, compliance reporting, and incident response.
- Kubernetes Secrets Fully Eliminated: All secrets were migrated from native Kubernetes Secrets to Vault, replacing an unaudited and difficult-to-manage system with centralized, encrypted, and policy-controlled secret management.
- Service-Level Secret Isolation Enforced: Each microservice now operates under its own Vault policy with access limited to its designated secrets, significantly reducing the risk of lateral credential exposure.
- Continuous Credential Rotation Enabled: Dynamic database credentials rotate automatically through short-lived leases, while application secrets can be updated and distributed within seconds, replacing annual or infrequent rotation cycles.
By replacing native Kubernetes Secrets with HashiCorp Vault and the Vault Agent Injector, Ksolves transformed secrets management across a 200+ microservice environment. The new architecture introduced centralized control, automated rotation, service-level access isolation, and complete auditability without requiring application changes or operational disruption. The result is a more secure and scalable platform where secrets are delivered dynamically, credentials rotate continuously, and access is governed by least-privilege policies.
Is Your Kubernetes Platform Still Relying on Base64-Encoded Secrets that Cannot be Rotated without Redeploying Half Your Fleet?
