Docker Service Discovery and Load Balancing with Haystack

Haystack provides service discovery and automatic HTTP load balancing in a Docker environment. It uses its own DNS to provide service discovery and dynamically updates its embedded load balancer as services are started or stopped.

Haystack solves two problems:

  • Service Discovery – Haystack automatically manages service registration within DNS by monitoring the Docker container lifecycle. Haystack introspects container metadata exposing available services within its own DNS. Haystack automatically manages the registration of services into a common DNS for discovery by other services.
  • Load balancing of a service as it is scaled up or down to meet demand – Haystack registers DNS service names that are load balanced over the available instances as the service is scaled up or down to meet demand. Each service name is registered in Haystack’s own DNS and Haystack manages the load balancing of the containers providing that service, from one to thousands and back again – dynamically as demand rises or falls.

This post walks through a demo of some of capabilities that Haystack provides using a simple HTTP based micro service. The service accepts HTTP GET requests and responds with the hostname of the container and the HTTP path used in the request.

To start a number of services to demonstrate load balancing in Haystack:

for i in {1..5}; do
     docker run \
     --name demo-$(printf %03d $i) \
     -d shortishly/haystack_demo;
done

Start Haystack – replace 172.16.1.218 with the location of your Docker Engine that is already listening on a tcp port.

docker run \
    -p 8080:80 \
    -e DOCKER_HOST=tcp://172.16.1.218:2375 \
    -d \
    --name haystack \
    shortishly/haystack

You should now have Haystack and 5 demo micro services running within Docker:

# docker ps --format="{{.Names}}"

haystack
demo-001
demo-002
demo-003
demo-004
demo-005

Now start a busybox that uses Haystack for DNS resolution as follows:

  docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --tty \
  --interactive \
  --rm busybox /bin/sh

In the busybox shell:

wget \
   -q \
   -O /dev/stdout \
    http://demo.haystack_demo.services.haystack/this/is/a/demo

Haystack has registered demo.haystack_demo.services.haystack in
its own DNS service, and is load balancing any
request randomly to one of the demo-001, demo-002,
demo-003, demo-003, demo-004 or demo-005
containers.

If you make a number of wget requests to the same URL you will
get responses from the different containers at random:

# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
d617596e70da: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
96c6c6f27f03: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
296b5208edf9: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
1166e110e70d: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
9909343b937a: /load/balancing

You can verify which services are available in Haystack by curling /api/info:

curl -s http://$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack)/api/info|python -m json.tool

Use you bowser to visit the Haystack UI running on your Docker host:

http://YOUR_DOCKER_HOST:8080/ui