Food Delivery Service
Event raceMay 2026
The payment event arrived before the order existed
The Order Service publishes order.created and order.paid to NATS JetStream. The Delivery Service creates a delivery on the first and assigns a driver on the second. Once it ran in two pods, order.paid was sometimes handled before order.created. The handler found no delivery, treated the message as invalid and dropped it for good. The order stayed paid, and no driver ever got it.
Go microservices and infrastructure as code
Services talk over gRPC, async events go through NATS JetStream. Each service follows Clean Architecture and owns its PostgreSQL database. A two-node DOKS cluster is provisioned with Terraform, and a Locust scenario covers load testing.
- Go
- gRPC
- NATS JetStream
- PostgreSQL
- Redis
- Nuxt
- Kubernetes
- Terraform
// Term = don't redeliver; Nak = retry.
// ErrNotFound is retried: order.paid can arrive
// before order.created is processed.
if errors.Is(err, usecase.ErrInvalidDelivery) {
_ = msg.Term()
} else {
_ = msg.Nak()
}- 01 / 03
A queue group instead of an exclusive subscription
Switched js.Subscribe to QueueSubscribe with the delivery-workers group, so several pods share one event stream.
- 02 / 03
Permanent and transient errors handled apart
An invalid message is dropped with Term(). "Delivery not found" is now transient: Nak() puts the message back, and it returns once order.created has been processed.
- 03 / 03
Retry cap and async email
MaxDeliver(10) stops a message from looping forever. Customer emails moved to a goroutine, so SMTP no longer blocks the event handler.
5 commits
What I would do differently
Event order is not guaranteed once there is more than one consumer. I only found the race once the service ran in two pods. Next time I would make handlers idempotent and model order states explicitly from the start.