My Randezvous with Tech

Loadbalancing & Service Discovery

February 20, 2017 by Ameya Advankar

Problem Statement:

We need a way a provide High Availability of the UI Portal through Load balancing and Service Discovery mechanism.

Possible Solutions:

Feature	Options
Load Balancing	HAProxy, NGINX, LVS
Service Discovery	Hashicorp Consul, Apache ZooKeeper

Solution Evaluations:

Loadbalancing

For Load Balancing, the choice of standard tool like HAProxy is taken into consideration. HAProxy or High Availability Proxy is an open source software TCP/HTTP Load Balancer and proxying solution. It is used to improve the performance and reliability of a server environment by distributing the workload across multiple servers. It provides extensive configuration capabilities since it acts on level 7 of OSI layer and thus can perform routing based on packet headers.

Registry

For building Service Discovery capabilities, we considered using either Hashicorp Consul.io or Apache Zookeeper as a Registry. We have had previous experience with using Zookeeper and it provides a plethora of capabilities. Both Consul and Zookeeper can be used for service discovery but Consul is much easier to integrate since it has the basic Service discovery fragment included in it, while with Zookeeper one would have to rather write their own Service Discovery implementation. This can be considered if there is a need for a customized service discovery mechanism in the application. Further differentiation is briefly explained over here.

Service Discovery

We have used Registrator for providing Service discovery. It monitors the system, on which it is running, for Docker containers and registers/unregisters all their exposed ports as a unique service instances to the Registry. It provides configuration capabilities to selectively ignore ports / containers based on Environment variables or Labels attached to the Docker container. This provides a simple mechanism to register any service without the need to write our own implementation of Service Discovery and attaching it in our application.

Dynamic Loadbalancing

HAProxy supports loadbalancing as per the configuration in its HAProxy.cfg file. For a distributed system, the server addresses are dynamic and hence a static configuration of HAProxy.cfg was not an option. To perform dynamic loadbalancing, Hashicorp consul-template provides a feasible solution which integrates with Consul.io and can be used to update files based on the services registered in Consul at any given time. The consul-template daemon, once started, monitors the services registered in the Consul server and performs the configured tasks whenever a service is registered/ usregistered.

True zero downtime

While consul-template is an excellent tool for reloading HAProxy on the fly, it does not truly provide zero downtime. While testing the UI portal, it was observed that for a short interval of time during HAProxy reload (using -sf flag), the requests made to the portal failed.

Conclusion:

The combination of HAProxy + Consul + consul-template + Registrator provides a feasible and simple solution to Dynamic loadbalancing with service discovery. We could evaluate further options by trying out other recipes and comparing their results with the above solution !

Details of the implementation can be found on the Loadbalancer & Service Discovery Setup details page