Posts on Nimbus

Building an Observability Stack with Prometheus and Grafana

Wed, 15 Mar 2023 11:00:00 -0700

Most teams don’t have a metrics problem — they have a too many dashboards nobody looks at problem. This is the minimal setup we use to go from “no visibility” to “useful alerts and three dashboards people actually open during an incident.”

Prometheus v2.51 Grafana v10.4

Terraform State Management: Lessons from Production

Wed, 15 Feb 2023 10:00:00 -0700

Local Terraform state works fine for a weekend project. It falls apart the moment a second engineer runs terraform apply against the same infrastructure. Here’s the setup we settled on after a state file conflict took down a staging environment for an afternoon.

State files contain secrets

Terraform state can include plaintext values for anything marked sensitive in your provider — database passwords, API keys, TLS private keys. Treat your state backend’s access controls with the same care as a secrets manager, not like a build artifact.

Move to a remote backend

A local terraform.tfstate file has no locking and no history. The fix is a remote backend with state locking built in:

Deploying with Kubernetes: A Practical Guide

Sun, 15 Jan 2023 09:00:00 -0700

Getting a service running reliably on Kubernetes comes down to a handful of building blocks: a container image, a Deployment, and a Service to expose it. This walkthrough covers the essentials.

Before you start

You’ll need kubectl configured against a cluster (Minikube, kind, or a managed cluster all work fine for this walkthrough).

1. Build and tag the image

1
2


docker build -t registry.example.com/api:1.4.0 .
docker push registry.example.com/api:1.4.0

2. Define the Deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


apiVersion: apps/v1
kind: Deployment
metadata:
 name: api
spec:
 replicas: 3
 selector:
 matchLabels:
 app: api
 template:
 metadata:
 labels:
 app: api
 spec:
 containers:
 - name: api
 image: registry.example.com/api:1.4.0
 ports:
 - containerPort: 8080
 readinessProbe:
 httpGet:
 path: /healthz
 port: 8080

3. Roll it out

bash

dev@k8s-master:~/api$

kubectl apply -f deployment.yaml deployment.apps/api created kubectl rollout status deployment/api deployment “api” successfully rolled out

Watch your readiness probes

A missing or too-strict readiness probe is the most common cause of a rollout that “succeeds” but serves traffic to pods that aren’t actually ready.

Architecture

Request flow

graph LR A[Client] --> B[Ingress] B --> C[Service] C --> D[Pod 1] C --> E[Pod 2] C --> F[Pod 3]

That’s the core loop: build, define, apply, verify. From here you’d typically add a HorizontalPodAutoscaler and wire this into CI so every merge to main triggers a new rollout automatically.