Automatic Load Balancing Of Spring Boot Micro Services In Docker With Haystack

In Spring Boot Micro Services In Docker we saw how to create a Docker container from a Spring Boot Micro Service. In this article we shall look at load balancing the service using Haystack. Haystack is a DNS based load balancer that is integrated with the Docker API automatically creating service groups as containers stop and start.

Haystack monitors the Docker API using a tcp socket. In this case the Docker API is on port 2375 on the Docker host.

docker run \
  -e DOCKER_HOST=tcp://172.16.1.218:2375 \
  --name haystack \
   --detach \
   shortishly/haystack

Build a gsa image following the instructions in this article, starting 3 instances of our Spring Boot Micro Service:

docker run --name srv-001 -d gsa
docker run --name srv-002 -d gsa
docker run --name srv-003 -d gsa

We can confirm that we have 3 gsa services by running docker ps as follows:

$ docker ps -a --format="{{.Names}}"

srv-003
srv-002
srv-001

Haystack automatically creates a service group srv.gsa.services.haystack in DNS. Starting a new gsa container will automatically add it to the service group. Stopping a gsa container automatically will remove it from the service group.

Startup a busy box instance using Haystack’s embedded DNS:

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --tty \
  --interactive \
  --rm busybox /bin/sh

Lookup srv.gsa.services.haystack in DNS:

nslookup srv.gsa.services.haystack

Server:    172.17.0.7
Address 1: 172.17.0.7 cb1i9a6.containers.haystack

Name:      srv.gsa.services.haystack
Address 1: 172.17.0.7 cb1i9a6.containers.haystack

Note that srv.gsa.services.haystack is actually pointing to the Haystack container. This is because Haystack acts as a proxy to HTTP requests, automatically load balancing requests randomly over members of the service group. Issue a wget to the service group and the requests will be load balanced randomly over the members:

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":1,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":1,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":2,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":1,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":3,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":2,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":4,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":3,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":4,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":5,"content":"Hello, Stranger!"}

Congratulations! You now have a Spring Boot Micro Service Docker container that is automatically being load balanced by Haystack. You can add or remove further gsa services and Haystack will automatically update the service pool.

Spring Boot Micro Services In Docker

Building a RESTful Web Service with Spring Boot Actuator is a great example of building a very simple micro service using Spring Boot Actuator. In this article we will go on to package that service into a Docker container.

Firstly we need to clone the source code from the original article:

$ git clone https://github.com/spring-guides/gs-actuator-service.git
$ cd gs-actuator-service/complete/

Build it using gradle:

$ ./gradlew clean assemble

Or with maven:

$ mvn package -DskipTests=true

We need to create a Dockerfile that we will use to build a docker image of the service as follows:

FROM java:8
EXPOSE 80
COPY target/gs-actuator-service-0.1.0.jar /
ENTRYPOINT ["java", \
           "-Djava.security.egd=file:/dev/urandom", \
           "-jar", \
           "gs-actuator-service-0.1.0.jar", \
           "--server.port=80"]

A couple of notes on the Dockerfile:

  • We are using the official Java 8 docker image as our base and copying our jar into the root directory of the image. If you used gradle instead of maven replace target/gs-actuator-service-0.1.0.jar with build/libs/gs-actuator-service-0.1.0.jar
  • We are using /dev/urandom as explained by this article to avoid issues in slow Tomcat startup when there is insufficient entropy (also the reason why we skipped the tests when building above)
  • We expose the service on port 80. When managing more than a couple of containers it is much simpler to just use standard ports.

Using your favourite editor put the Dockerfile in the complete directory, and build a docker image as follows:

docker build -t gsa .

Our image has been created an has been tagged as ‘gsa’:

$ docker images
REPOSITORY TAG    IMAGE ID     CREATED       SIZE
gsa        latest 6acbba251766 4 seconds ago 683.1 MB

We can run the image using docker run as follows:

docker run --name gsa -d gsa

We can find the IP address being used by the gsa container by running docker inspect:

$ docker inspect \
    --format={{.NetworkSettings.IPAddress}} gsa

172.17.0.7

In this case our ‘gsa’ container has been given 172.17.0.7 as its IP address. We can make a HTTP request for /hello-world from our new service with the following:

curl 172.17.0.7/hello-world

Be sure to replace IP address 172.17.0.7, with the output that your docker host gives you on docker inspect.

In response you will get something like:

{"id":1,"content":"Hello, Stranger!"}

Congratulations! You now have a Spring Boot Actuator Micro Service running inside a Docker container.

Tansu is a distributed key/value and lock store

Tansu is a distributed key value store designed to maintain configuration and other data that must be highly available. It uses the Raft Consensus algorithm for leadership election and distribution of state amongst its members. By default node discovery is via mDNS and will automatically form a mesh of nodes sharing the same environment.

Features

Tansu has a REST interface to set, get or delete the value represented by a key. It also provides a HTTP Server Sent Event Stream
of changes to the store.

Tansu provides REST interface for simple Check And Set (CAS) operations.

Tansu provides test and set operations that can be used to operate locks through a simple REST based HTTP Server Sent Event Stream interface.

Quick Start

Tansu is packaged as a Docker container. To start a local 5 node cluster is as simple as:

for i in {1..5}; do
    docker run \
        --name tansu-$(printf %03d $i) \
        -d shortishly/tansu;
done

Tansu uses mDNS by default to discover other nodes and automatically forms a cluster. API requests can be made to any discovered node, which are internally routed to the appropriate node depending on the request type.

To demonstrate this, lets use the shell to randomly pick a node from our new cluster:

RANDOM_IP=$(docker inspect \
            --format={{.NetworkSettings.IPAddress}} \
            tansu-$(printf %03d $[1 + $[RANDOM % 5]]))

All key/value API operations are under the ‘/api/keys/…’ URL. We can create a stream of changes to a key (or a hierarchy of keys) before that key exists as follows:

curl \
    -i \
    -s \
    "http://${RANDOM_IP}/api/keys/hello?stream=true&children=true"

The key space in Tansu is a directory structure separated with ‘/’ characters. Any change to the key ‘hello’ will be reported in the above stream, and also any change in subdirectory below ‘hello’ will also be reported.

Leaving the stream curl running, in another shell lets assign the value “world” to the key “hello”:

curl \
    -X PUT \
    -i \
    -s \
    http://${RANDOM_IP}/api/keys/hello \
    -d value=world

Back in our stream, it will contain a ‘create’ notification:

id: 1
event: create
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
  "tansu":{
    "content_type":"text/plain",
    "created":1,
    "parent":"/",
    "updated":1}},
  "value":"world"}

Or a key that is below ‘hello’:

curl \
  -X PUT \
  -i \
  -s \
   http://${RANDOM_IP}/api/keys/hello/joe \
  -d value=mike

The stream will now contain a `create` notification:

id: 2
event: create
data: {
  "category":"user",
  "key":"/hello/joe",
  "metadata":{
    "tansu":{
      "content_type":"text/plain",
      "created":2,
      "parent":"/hello",
      "updated":2}},
  "value":"mike"}

In the above case Tansu will assume that the value has the ‘text/plain’ content type (as the value from a form url encoded body). Other content types (in particular JSON) are also supported:

curl \
  -X PUT \
  -H "Content-Type: application/json" \
  -i http://${RANDOM_IP}/api/keys/hello \
  --data-binary '{"stuff": true}'

With an update in the stream:

id: 3
event: set
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":1,
      "parent":"/",
      "updated":3}},
  "previous":"world",
  "value":{"stuff":true}}

GET

The current value of a key can be obtained simply by issuing a GET on that key:

curl \
  -i \
  -s \
  http://${RANDOM_IP}/api/keys/hello

{"stuff": true}

DELETE

Similarly a key is removed by issuing a DELETE request:

curl \
  -i \
  -X DELETE \
  http://${RANDOM_IP}/api/keys/hello

The stream will now contain a delete notification:

id: 5
event: delete
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":1,
      "parent":"/",
      "updated":5}},
  "value":{"stuff":true}}

TTL

A value can also be given a time to live by supplying a TTL header:

curl \
  -X PUT \
  -H "Content-Type: application/json" \
  -H "ttl: 10" \
  -i \
  http://${RANDOM_IP}/api/keys/hello \
  --data-binary '{"ephemeral": true}'

The event stream will contain details of the `create` together with a TTL
attribute:

id: 6
event: create
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":6,
      "parent":"/",
      "ttl":10,
      "updated":6}},
  "value":{"ephemeral":true}}

Ten seconds later when the time to live has expired:

id: 7
event: delete
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":6,
      "parent":"/",
      "ttl":0,
      "updated":7}},
  "value":{"ephemeral":true}}

Docker Service Discovery and Load Balancing with Haystack

Haystack provides service discovery and automatic HTTP load balancing in a Docker environment. It uses its own DNS to provide service discovery and dynamically updates its embedded load balancer as services are started or stopped.

Haystack solves two problems:

  • Service Discovery – Haystack automatically manages service registration within DNS by monitoring the Docker container lifecycle. Haystack introspects container metadata exposing available services within its own DNS. Haystack automatically manages the registration of services into a common DNS for discovery by other services.
  • Load balancing of a service as it is scaled up or down to meet demand – Haystack registers DNS service names that are load balanced over the available instances as the service is scaled up or down to meet demand. Each service name is registered in Haystack’s own DNS and Haystack manages the load balancing of the containers providing that service, from one to thousands and back again – dynamically as demand rises or falls.

This post walks through a demo of some of capabilities that Haystack provides using a simple HTTP based micro service. The service accepts HTTP GET requests and responds with the hostname of the container and the HTTP path used in the request.

To start a number of services to demonstrate load balancing in Haystack:

for i in {1..5}; do
     docker run \
     --name demo-$(printf %03d $i) \
     -d shortishly/haystack_demo;
done

Start Haystack – replace 172.16.1.218 with the location of your Docker Engine that is already listening on a tcp port.

docker run \
    -p 8080:80 \
    -e DOCKER_HOST=tcp://172.16.1.218:2375 \
    -d \
    --name haystack \
    shortishly/haystack

You should now have Haystack and 5 demo micro services running within Docker:

# docker ps --format="{{.Names}}"

haystack
demo-001
demo-002
demo-003
demo-004
demo-005

Now start a busybox that uses Haystack for DNS resolution as follows:

  docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --tty \
  --interactive \
  --rm busybox /bin/sh

In the busybox shell:

wget \
   -q \
   -O /dev/stdout \
    http://demo.haystack_demo.services.haystack/this/is/a/demo

Haystack has registered demo.haystack_demo.services.haystack in
its own DNS service, and is load balancing any
request randomly to one of the demo-001, demo-002,
demo-003, demo-003, demo-004 or demo-005
containers.

If you make a number of wget requests to the same URL you will
get responses from the different containers at random:

# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
d617596e70da: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
96c6c6f27f03: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
296b5208edf9: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
1166e110e70d: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
9909343b937a: /load/balancing

You can verify which services are available in Haystack by curling /api/info:

curl -s http://$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack)/api/info|python -m json.tool

Use you bowser to visit the Haystack UI running on your Docker host:

http://YOUR_DOCKER_HOST:8080/ui