Infrastructure on Nimbus

Use PostgreSQL as the primary datastore

Sat, 10 Feb 2024 00:00:00 +0000

Context

We need a primary datastore for the new billing service. The team has production experience with both PostgreSQL and MongoDB, and the data model is relational (accounts, invoices, line items with foreign keys).

Decision

We will use PostgreSQL 16, managed via RDS, as the primary datastore for all new services unless a specific workload has a strong document-store or time-series justification.

Consequences

Schema migrations require a formal review (see docs/migrations).
We standardize on one connection-pooling strategy (PgBouncer) across services, simplifying on-call runbooks.
Document-shaped data (e.g. audit logs) will need a deliberate exception request rather than defaulting to Postgres JSONB columns.

Terraform State Management: Lessons from Production

Wed, 15 Feb 2023 10:00:00 -0700

Local Terraform state works fine for a weekend project. It falls apart the moment a second engineer runs terraform apply against the same infrastructure. Here’s the setup we settled on after a state file conflict took down a staging environment for an afternoon.

State files contain secrets

Terraform state can include plaintext values for anything marked sensitive in your provider — database passwords, API keys, TLS private keys. Treat your state backend’s access controls with the same care as a secrets manager, not like a build artifact.

Move to a remote backend

A local terraform.tfstate file has no locking and no history. The fix is a remote backend with state locking built in: