Wake-Up Call
Real incident: A misconfigured Kubernetes cluster at a Fortune 500 company led to exposed customer data for 72 hours. The attack vector? Default RBAC permissions and no network policies.
Cost: $4.2M in fines + immeasurable reputation damage.
Kubernetes security isn't optional—it's critical infrastructure. After securing dozens of production clusters and passing SOC 2 and ISO 27001 audits, I've distilled the essential security practices into 10 actionable steps that every team should implement.
This isn't a theoretical guide. These are production-tested configurations currently protecting clusters handling millions of requests per day. Let's get started.
10 Security Layers
RBAC & Service Accounts
Least privilege access control
Pod Security Standards
Restrict dangerous pod configurations
Network Policies
Zero-trust pod communication
Secrets Management
External secrets & encryption
Image Security
Scanning & signing
Runtime Security
Falco & behavioral monitoring
Audit Logging
Comprehensive activity tracking
API Server Security
Hardening control plane
etcd Encryption
Secrets at rest encryption
Compliance & Scanning
CIS benchmarks & vulnerability scans
Step 1: Implement Strict RBAC
Problem: Default Kubernetes gives service accounts overly broad permissions. Attackers exploiting a pod can escalate privileges cluster-wide.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-reader
namespace: production
automountServiceAccountToken: false # Disable auto-mounting
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: app-reader-role
namespace: production
rules:
# Only allow reading ConfigMaps
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list"]
# Only allow reading Secrets (specific names)
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["app-config", "db-credentials"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: app-reader-binding
namespace: production
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: app-reader-role
subjects:
- kind: ServiceAccount
name: app-reader
namespace: production
RBAC Best Practices
- Use
automountServiceAccountToken: falseby default - Create role-specific service accounts (no shared accounts)
- Scope roles to specific namespaces (avoid ClusterRoles unless necessary)
- Use
resourceNamesto restrict access to specific resources - Never grant
*permissions in production
# List all service accounts with cluster-admin rights (dangerous!)
kubectl get clusterrolebindings -o json | jq -r '
.items[] |
select(.roleRef.name=="cluster-admin") |
.metadata.name
'
# Find service accounts with wildcard permissions
kubectl get roles,clusterroles --all-namespaces -o json | jq -r '
.items[] |
select(.rules[]?.verbs[]? == "*") |
"\(.metadata.namespace // "cluster")/\(.metadata.name)"
'
# List all service accounts and their bound roles
kubectl get rolebindings,clusterrolebindings --all-namespaces \
-o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,\
ROLE:.roleRef.name,SUBJECT:.subjects[0].name
Step 2: Enforce Pod Security Standards
Pod Security Standards (PSS) replaced Pod Security Policies in Kubernetes 1.25. Three enforcement levels: privileged, baseline, and restricted.
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
# Enforce restricted standard (most secure)
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
# Audit mode for monitoring violations
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
# Warn mode for user feedback
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
Restricted PSS Requirements
The restricted standard enforces:
- No privileged containers
- No host namespaces (network, PID, IPC)
- No host ports
- Must drop ALL capabilities
- Must run as non-root user
- Read-only root filesystem recommended
- No privilege escalation
apiVersion: v1
kind: Pod
metadata:
name: secure-app
namespace: production
spec:
serviceAccountName: app-reader # Dedicated SA
automountServiceAccountToken: false # Explicit disable
securityContext:
runAsNonRoot: true
runAsUser: 10000
runAsGroup: 10000
fsGroup: 10000
seccompProfile:
type: RuntimeDefault # Seccomp for syscall filtering
containers:
- name: app
image: myapp:1.2.3
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 10000
capabilities:
drop:
- ALL # Drop all Linux capabilities
readOnlyRootFilesystem: true # Immutable filesystem
resources:
limits:
cpu: "1000m"
memory: "512Mi"
requests:
cpu: "100m"
memory: "128Mi"
volumeMounts:
- name: tmp
mountPath: /tmp
- name: cache
mountPath: /app/cache
volumes:
- name: tmp
emptyDir: {} # Writable /tmp
- name: cache
emptyDir: {} # Writable cache dir
Step 3: Implement Network Policies
By default, all pods in a Kubernetes cluster can talk to each other. Network Policies implement zero-trust networking at the pod level.
---
# Deny all ingress traffic by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: production
spec:
podSelector: {} # Apply to all pods
policyTypes:
- Ingress
---
# Deny all egress traffic by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-server-network-policy
namespace: production
spec:
podSelector:
matchLabels:
app: api-server
policyTypes:
- Ingress
- Egress
# Allowed incoming traffic
ingress:
# Allow traffic from ingress controller only
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
- podSelector:
matchLabels:
app: nginx-ingress
ports:
- protocol: TCP
port: 8080
# Allowed outgoing traffic
egress:
# Allow DNS lookups
- to:
- namespaceSelector:
matchLabels:
name: kube-system
- podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
# Allow database access
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Allow external API calls (specific CIDR)
- to:
- ipBlock:
cidr: 52.1.2.0/24 # External API IP range
ports:
- protocol: TCP
port: 443
Testing Network Policies
Always test before enforcing:
# Test connectivity from pod to pod
kubectl exec -it api-server-pod -- curl http://postgres-service:5432
# Expected: Connection allowed (policy permits it)
kubectl exec -it api-server-pod -- curl http://redis-service:6379
# Expected: Connection refused (no policy allows it)
Step 4: Proper Secrets Management
Never store secrets in Kubernetes Secrets without encryption. Use External Secrets Operator + AWS Secrets Manager/HashiCorp Vault.
# Install External Secrets Operator first:
# helm repo add external-secrets https://charts.external-secrets.io
# helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secretsmanager
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa # SA with IAM role
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-database-credentials
namespace: production
spec:
refreshInterval: 1h # Sync every hour
secretStoreRef:
name: aws-secretsmanager
kind: SecretStore
target:
name: db-credentials # Creates this K8s Secret
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: prod/database/credentials
property: username
- secretKey: password
remoteRef:
key: prod/database/credentials
property: password
- secretKey: connection-string
remoteRef:
key: prod/database/credentials
property: connection_string
Secrets Best Practices
- Store secrets in external vaults (AWS Secrets Manager, Vault)
- Enable etcd encryption at rest (see Step 9)
- Rotate secrets automatically (use ESO refresh)
- Audit secret access with RBAC logs
- Never commit secrets to Git
- Never use Secrets as environment variables (prefer mounted files)
Step 5: Container Image Security
Image Scanning with Trivy
# Scan image for vulnerabilities
trivy image --severity HIGH,CRITICAL myapp:1.2.3
# Scan and fail CI if HIGH+ found
trivy image --exit-code 1 \
--severity HIGH,CRITICAL \
myapp:1.2.3
Image Signing with Cosign
# Sign image
cosign sign --key cosign.key \
myregistry.io/myapp:1.2.3
# Verify signature before deployment
cosign verify --key cosign.pub \
myregistry.io/myapp:1.2.3
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8sallowedrepos
spec:
crd:
spec:
names:
kind: K8sAllowedRepos
validation:
openAPIV3Schema:
properties:
repos:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8sallowedrepos
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not startswith(container.image, input.parameters.repos[_])
msg := sprintf("Image %v not from approved registry", [container.image])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
name: allowed-docker-registries
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces:
- production
parameters:
repos:
- "myregistry.io/"
- "public.ecr.aws/mycompany/"
Quick Wins: Do These Now
5 Minutes
- Enable PSS restricted mode on namespace
- Add default deny network policies
- Set
automountServiceAccountToken: false
30 Minutes
- Audit existing RBAC permissions
- Integrate Trivy image scanning in CI/CD
- Deploy External Secrets Operator
Conclusion
Security is a journey, not a destination. These 10 steps provide a solid foundation, but remember:
- Defense in depth: Multiple layers catch what one misses
- Continuous improvement: Audit quarterly, patch promptly
- Team awareness: Security is everyone's responsibility
- Stay updated: K8s security evolves constantly
Master Kubernetes Security
Get weekly DevOps & security insights in your inbox