How to run consul in docker
By Vikrant
September 8, 2018
While working with kubernetes, I was intrigued by the service discovery topic. I read about consul in past but never got the chance to make my hands dirty with it. Today, I spend sometime to run the consul in docker environment. I used docker-machine on MAC with virtual box to quicly start a machine with docker.
Running with default options
$ docker-machine create consulserver
$ docker-machine ssh consulserver
docker@consulserver:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether b6:7e:a0:4e:14:fd brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:a4:04:e6 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fea4:4e6/64 scope link
valid_lft forever preferred_lft forever
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:f7:3b:2f brd ff:ff:ff:ff:ff:ff
inet 192.168.99.100/24 brd 192.168.99.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fef7:3b2f/64 scope link
valid_lft forever preferred_lft forever
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:1d:0f:41:23 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:1dff:fe0f:4123/64 scope link
valid_lft forever preferred_lft forever
- Start a consul container without providing any options. consul started in server mode. On eth0 port of docker container “172.17.0.2” ip address is assigned which is normal behavior.
docker@consulserver:~$ docker run -d consul agent -server -bootstrap-expect=1
43f8be212338baafd0511deb0a4629061881a91343cc70fedb0cb72dd41aec61
$ docker exec -it 43f8be212338baafd0511deb0a4629061881a91343cc70fedb0cb72dd41aec61 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
19: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
- Check the logs of container. http, dns are listening on loopback address and LAN, WAN on docker eth0 ip. That means if we start another docker-machine and try to reach conul server we would not be able to reach it. In-fact conul GUI is also no accessible.
docker@consulserver:~$ docker logs 43f8be212338
BootstrapExpect is set to 1; this is the same as Bootstrap mode.
bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.2.2'
Node ID: '65efceba-4c42-c4a9-6b8d-a9b1c005ee2e'
Node name: '43f8be212338'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: true)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 172.17.0.2 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2018/09/08 09:39:42 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:65efceba-4c42-c4a9-6b8d-a9b1c005ee2e Address:172.17.0.2:8300}]
2018/09/08 09:39:42 [INFO] serf: EventMemberJoin: 43f8be212338.dc1 172.17.0.2
2018/09/08 09:39:42 [INFO] serf: EventMemberJoin: 43f8be212338 172.17.0.2
2018/09/08 09:39:42 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2018/09/08 09:39:42 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state (Leader: "")
2018/09/08 09:39:42 [INFO] consul: Adding LAN server 43f8be212338 (Addr: tcp/172.17.0.2:8300) (DC: dc1)
2018/09/08 09:39:42 [INFO] consul: Handled member-join event for server "43f8be212338.dc1" in area "wan"
2018/09/08 09:39:42 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2018/09/08 09:39:42 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2018/09/08 09:39:42 [INFO] agent: started state syncer
2018/09/08 09:39:48 [WARN] raft: Heartbeat timeout from "" reached, starting election
2018/09/08 09:39:48 [INFO] raft: Node at 172.17.0.2:8300 [Candidate] entering Candidate state in term 2
2018/09/08 09:39:48 [INFO] raft: Election won. Tally: 1
2018/09/08 09:39:48 [INFO] raft: Node at 172.17.0.2:8300 [Leader] entering Leader state
2018/09/08 09:39:48 [INFO] consul: cluster leadership acquired
2018/09/08 09:39:48 [INFO] consul: New leader elected: 43f8be212338
2018/09/08 09:39:48 [INFO] consul: member '43f8be212338' joined, marking health alive
2018/09/08 09:39:48 [INFO] agent: Synced node info
- Start a consul agent in client mode. Now we have two containers (consul in server and client mode) running on consulserver machine.
docker@consulserver:~$ docker run -d consul agent
0ef3aac96100478fe13b4086037b4fc2ee461045e5606baae25078247e08c50b
docker@consulserver:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0ef3aac96100 consul "docker-entrypoint.s…" 3 seconds ago Up 3 seconds 8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp agitated_ardinghelli
43f8be212338 consul "docker-entrypoint.s…" About a minute ago Up About a minute 8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp upbeat_hawking
- Since no option was provided to let consul client know where conul server is present hence in the logs these messages are expected.
docker@consulserver:~$ docker logs 0ef3aac96100
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.2.2'
Node ID: '742da527-eed0-b029-0b62-95ded26ce08c'
Node name: '0ef3aac96100'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 172.17.0.3 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2018/09/08 09:40:54 [INFO] serf: EventMemberJoin: 0ef3aac96100 172.17.0.3
2018/09/08 09:40:54 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2018/09/08 09:40:54 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2018/09/08 09:40:54 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2018/09/08 09:40:54 [INFO] agent: started state syncer
2018/09/08 09:40:54 [WARN] manager: No servers available
2018/09/08 09:40:54 [ERR] agent: failed to sync remote state: No known Consul servers
2018/09/08 09:41:17 [WARN] manager: No servers available
2018/09/08 09:41:17 [ERR] agent: failed to sync remote state: No known Consul servers
2018/09/08 09:41:42 [WARN] manager: No servers available
2018/09/08 09:41:42 [ERR] agent: failed to sync remote state: No known Consul servers
2018/09/08 09:42:02 [WARN] manager: No servers available
2018/09/08 09:42:02 [ERR] agent: failed to sync remote state: No known Consul servers
2018/09/08 09:42:30 [WARN] manager: No servers available
2018/09/08 09:42:30 [ERR] agent: failed to sync remote state: No known Consul servers
Make consul client detect consul server
- Delete the previous consul client container and start new container stating the ip address of consul server so that client can join the server. Hurray!! logs confirmed that consul client is connected with server.
docker@consulserver:~$ docker run -d consul agent --retry-join=172.17.0.2
8120ddc29cb0e2634250d72a598916d383a44ed20c01f82f8ce10d0cd65f7ff8
docker@consulserver:~$ docker logs 8120ddc29cb0
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.2.2'
Node ID: '04662234-108c-7876-a0da-a16214fe3d5b'
Node name: '8120ddc29cb0'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 172.17.0.3 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2018/09/08 09:45:06 [INFO] serf: EventMemberJoin: 8120ddc29cb0 172.17.0.3
2018/09/08 09:45:06 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2018/09/08 09:45:06 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2018/09/08 09:45:06 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2018/09/08 09:45:06 [INFO] agent: started state syncer
2018/09/08 09:45:06 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os packet scaleway softlayer triton vsphere
2018/09/08 09:45:06 [INFO] agent: Joining LAN cluster...
2018/09/08 09:45:06 [INFO] agent: (LAN) joining: [172.17.0.2]
2018/09/08 09:45:06 [WARN] manager: No servers available
2018/09/08 09:45:06 [ERR] agent: failed to sync remote state: No known Consul servers
2018/09/08 09:45:06 [INFO] serf: EventMemberJoin: 43f8be212338 172.17.0.2
2018/09/08 09:45:06 [INFO] agent: (LAN) joined: 1 Err: <nil>
2018/09/08 09:45:06 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2018/09/08 09:45:06 [INFO] consul: adding server 43f8be212338 (Addr: tcp/172.17.0.2:8300) (DC: dc1)
2018/09/08 09:45:07 [INFO] agent: Synced node info
Same can be confirmed from consul server logs.
docker@consulserver:~$ docker logs 43f8be212338
BootstrapExpect is set to 1; this is the same as Bootstrap mode.
bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.2.2'
Node ID: '65efceba-4c42-c4a9-6b8d-a9b1c005ee2e'
Node name: '43f8be212338'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: true)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 172.17.0.2 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2018/09/08 09:39:42 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:65efceba-4c42-c4a9-6b8d-a9b1c005ee2e Address:172.17.0.2:8300}]
2018/09/08 09:39:42 [INFO] serf: EventMemberJoin: 43f8be212338.dc1 172.17.0.2
2018/09/08 09:39:42 [INFO] serf: EventMemberJoin: 43f8be212338 172.17.0.2
2018/09/08 09:39:42 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2018/09/08 09:39:42 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state (Leader: "")
2018/09/08 09:39:42 [INFO] consul: Adding LAN server 43f8be212338 (Addr: tcp/172.17.0.2:8300) (DC: dc1)
2018/09/08 09:39:42 [INFO] consul: Handled member-join event for server "43f8be212338.dc1" in area "wan"
2018/09/08 09:39:42 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2018/09/08 09:39:42 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2018/09/08 09:39:42 [INFO] agent: started state syncer
2018/09/08 09:39:48 [WARN] raft: Heartbeat timeout from "" reached, starting election
2018/09/08 09:39:48 [INFO] raft: Node at 172.17.0.2:8300 [Candidate] entering Candidate state in term 2
2018/09/08 09:39:48 [INFO] raft: Election won. Tally: 1
2018/09/08 09:39:48 [INFO] raft: Node at 172.17.0.2:8300 [Leader] entering Leader state
2018/09/08 09:39:48 [INFO] consul: cluster leadership acquired
2018/09/08 09:39:48 [INFO] consul: New leader elected: 43f8be212338
2018/09/08 09:39:48 [INFO] consul: member '43f8be212338' joined, marking health alive
2018/09/08 09:39:48 [INFO] agent: Synced node info
2018/09/08 09:45:06 [INFO] serf: EventMemberJoin: 8120ddc29cb0 172.17.0.3
2018/09/08 09:45:06 [INFO] consul: member '8120ddc29cb0' joined, marking health alive
- Both client and server container showing the list of consul members. Status column indicates the type of member.
docker@consulserver:~$ docker exec -it 43f8be212338 consul members
Node Address Status Type Build Protocol DC Segment
43f8be212338 172.17.0.2:8301 alive server 1.2.2 2 dc1 <all>
8120ddc29cb0 172.17.0.3:8301 alive client 1.2.2 2 dc1 <default>
docker@consulserver:~$ docker exec -it 8120ddc29cb0 consul members
Node Address Status Type Build Protocol DC Segment
43f8be212338 172.17.0.2:8301 alive server 1.2.2 2 dc1 <all>
8120ddc29cb0 172.17.0.3:8301 alive client 1.2.2 2 dc1 <default>
Change bind interface of consul server
Tear down the previous containers.
- It’s stated in official consul image document that we should start the consul in host mode hence instead of default bridge networking this time host mode is used to start the container. As now host mode is used hence all interfaces available on docker-machine will be present inside the container. Since the multiple interfaces are present on host machine hence we need to mention bind interface.
docker@consulserver:~$ docker run --net=host -e CONSUL_BIND_INTERFACE=eth1 -d consul agent -server -bootstrap-expect=1
3e9f69fc7e1fa3d62b2dbbd13fde420c5c20575f41c38519fa7b0aed958c514a
docker@consulserver:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3e9f69fc7e1f consul "docker-entrypoint.s…" 2 seconds ago Up 1 second lucid_brattain
- Logs showing that this time LAN and WAN gossip protocols listening on ipaddress assigned to eth1 interface. Still we will not be able to access the UI of consul because 8500 is listening on loopback address.
docker@consulserver:~$ docker logs 3e9f69fc7e1f
==> Found address '192.168.99.100' for interface 'eth1', setting bind option...
BootstrapExpect is set to 1; this is the same as Bootstrap mode.
bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.2.2'
Node ID: 'e6fb1865-ab8a-aafc-199d-3610c0e4324d'
Node name: 'consulserver'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: true)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 192.168.99.100 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2018/09/08 09:59:18 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:e6fb1865-ab8a-aafc-199d-3610c0e4324d Address:192.168.99.100:8300}]
2018/09/08 09:59:18 [INFO] raft: Node at 192.168.99.100:8300 [Follower] entering Follower state (Leader: "")
2018/09/08 09:59:18 [INFO] serf: EventMemberJoin: consulserver.dc1 192.168.99.100
2018/09/08 09:59:18 [INFO] serf: EventMemberJoin: consulserver 192.168.99.100
2018/09/08 09:59:18 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2018/09/08 09:59:18 [INFO] consul: Adding LAN server consulserver (Addr: tcp/192.168.99.100:8300) (DC: dc1)
2018/09/08 09:59:18 [INFO] consul: Handled member-join event for server "consulserver.dc1" in area "wan"
2018/09/08 09:59:18 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2018/09/08 09:59:18 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2018/09/08 09:59:18 [INFO] agent: started state syncer
2018/09/08 09:59:25 [ERR] agent: failed to sync remote state: No cluster leader
2018/09/08 09:59:25 [WARN] raft: Heartbeat timeout from "" reached, starting election
2018/09/08 09:59:25 [INFO] raft: Node at 192.168.99.100:8300 [Candidate] entering Candidate state in term 2
2018/09/08 09:59:25 [INFO] raft: Election won. Tally: 1
2018/09/08 09:59:25 [INFO] raft: Node at 192.168.99.100:8300 [Leader] entering Leader state
2018/09/08 09:59:25 [INFO] consul: cluster leadership acquired
2018/09/08 09:59:25 [INFO] consul: New leader elected: consulserver
2018/09/08 09:59:25 [INFO] consul: member 'consulserver' joined, marking health alive
- This time instead of starting the consul in client mode on same machine I started another docker-machine and spawned container on that node in consul client mode.
$ docker-machine create consulclient
$ docker-machie ssh consulclient
docker@consulclient:~$ docker run -d consul agent --retry-join=192.168.99.100
docker@consulserver:~$ docker exec -it 3e9f69fc7e1f consul members
Node Address Status Type Build Protocol DC Segment
consulserver 192.168.99.100:8301 alive server 1.2.2 2 dc1 <all>
f25cefea35b6 172.17.0.2:8301 alive client 1.2.2 2 dc1 <default>
- I wanted to start consul client in host mode and made it listen on interface which is externally reachable hence I teared down the setup but I forgot to remove the client from consul membership. Used force-leave command remove it from the membership.
docker@consulclient:~$ docker rm -f f25cefea35b6
f25cefea35b6
docker@consulserver:~$ docker exec -it 3e9f69fc7e1f consul members
Node Address Status Type Build Protocol DC Segment
consulserver 192.168.99.100:8301 alive server 1.2.2 2 dc1 <all>
f25cefea35b6 172.17.0.2:8301 failed client 1.2.2 2 dc1 <default>
docker@consulserver:~$ docker exec -it 3e9f69fc7e1f consul force-leave f25cefea35b6
docker@consulserver:~$ docker exec -it 3e9f69fc7e1f consul members
Node Address Status Type Build Protocol DC Segment
consulserver 192.168.99.100:8301 alive server 1.2.2 2 dc1 <all>
f25cefea35b6 172.17.0.2:8301 left client 1.2.2 2 dc1 <default>
- start the consul agent in host mode with and mention the ip address on which we want to LAN/WAN port to listen.
docker@consulclient:~$ docker run -d --name=consulagent1 --net=host consul agent --retry-join=192.168.99.100 -bind=192.168.99.101
746bedb89d60caccb17b0fbccb93500ddf0c42ac8a47558edaf36289f0e9c270
docker@consulclient:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
746bedb89d60 consul "docker-entrypoint.s…" 3 seconds ago Up 2 seconds consulagent1
docker@consulclient:~$ docker logs 746bedb89d60
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.2.2'
Node ID: '1cd1a150-52b9-a1bc-9dd6-87ed527b782c'
Node name: 'consulclient'
Datacenter: 'dc1' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 192.168.99.101 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2018/09/08 10:11:28 [INFO] serf: EventMemberJoin: consulclient 192.168.99.101
2018/09/08 10:11:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2018/09/08 10:11:28 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2018/09/08 10:11:28 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
2018/09/08 10:11:28 [INFO] agent: started state syncer
2018/09/08 10:11:28 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os packet scaleway softlayer triton vsphere
2018/09/08 10:11:28 [INFO] agent: Joining LAN cluster...
2018/09/08 10:11:28 [INFO] agent: (LAN) joining: [192.168.99.100]
2018/09/08 10:11:28 [WARN] manager: No servers available
2018/09/08 10:11:28 [ERR] agent: failed to sync remote state: No known Consul servers
2018/09/08 10:11:28 [INFO] serf: EventMemberJoin: consulserver 192.168.99.100
2018/09/08 10:11:28 [INFO] agent: (LAN) joined: 1 Err: <nil>
2018/09/08 10:11:28 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2018/09/08 10:11:28 [INFO] consul: adding server consulserver (Addr: tcp/192.168.99.100:8300) (DC: dc1)
2018/09/08 10:11:29 [INFO] agent: Synced node info
- membership confirmed that consul client is detected by consul server.
docker@consulserver:~$ docker exec -it 3e9f69fc7e1f consul members
Node Address Status Type Build Protocol DC Segment
consulserver 192.168.99.100:8301 alive server 1.2.2 2 dc1 <all>
consulclient 192.168.99.101:8301 alive client 1.2.2 2 dc1 <default>
f25cefea35b6 172.17.0.2:8301 left client 1.2.2 2 dc1 <default>
- One silly experiment - Start another consul client agent on the node on which our consul server is running, it will not run, any guesses?? because the ports are already used by consul server hence I started consul client in bridge networking mode.
docker@consulserver:~$ docker run -d --name=consulagent2 --net=host consul agent --retry-join=192.168.99.101 -bind=192.168.99.100
919fb2f51f9f7f6d703679ca01c730d61f187c0d71b45f947e39e2023fd34e42
docker@consulserver:~$ docker logs 919fb2f51f9f
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul client: Failed to start lan serf: Failed to create memberlist: Could not set up network transport: failed to obtain an address: Failed to start TCP listener on "192.168.99.100" port 8301: listen tcp 192.168.99.100:8301: bind: address already in use
docker@consulserver:~$ docker run -d --name=consulagent2 consul agent --retry-join=192.168.99.101
553bfee60b9befb4b245fb691d54962208de844815852690db7a6f7a2bd36a47
docker@consulserver:~$ docker exec -it 3e9f69fc7e1f consul members
Node Address Status Type Build Protocol DC Segment
consulserver 192.168.99.100:8301 alive server 1.2.2 2 dc1 <all>
553bfee60b9b 172.17.0.2:8301 alive client 1.2.2 2 dc1 <default>
consulclient 192.168.99.101:8301 alive client 1.2.2 2 dc1 <default>
f25cefea35b6 172.17.0.2:8301 left client 1.2.2 2 dc1 <default>
Why the UI still not accessible
- Make 8500 listen on eth1 interface by adding another env variable
CONSUL_CLIENT_INTERFACEand also add-uiflag.
docker@consulserver:~$ docker run --net=host -e CONSUL_BIND_INTERFACE=eth1 -e CONSUL_CLIENT_INTERFACE=eth1 -d consul agent -ui -server -bootstrap-expect=1
f2fdb2148166bb810e33d47d3dc21bdf4f6e088eea21c2de47a0c78a0487e1d3
docker@consulserver:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2fdb2148166 consul "docker-entrypoint.s…" 7 seconds ago Up 7 seconds sad_kowalevski
- Logs indicate that HTTP is also listening on ip address assigned on eth1 interface.
docker@consulserver:~$ docker logs f2fdb2148166
==> Found address '192.168.99.100' for interface 'eth1', setting bind option...
==> Found address '192.168.99.100' for interface 'eth1', setting client option...
BootstrapExpect is set to 1; this is the same as Bootstrap mode.
bootstrap = true: do not enable unless necessary
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.2.2'
Node ID: 'd987371a-4a8a-5edd-73a4-b5f69d32559b'
Node name: 'consulserver'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: true)
Client Addr: [192.168.99.100] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 192.168.99.100 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2018/09/08 10:35:38 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:d987371a-4a8a-5edd-73a4-b5f69d32559b Address:192.168.99.100:8300}]
2018/09/08 10:35:38 [INFO] serf: EventMemberJoin: consulserver.dc1 192.168.99.100
2018/09/08 10:35:38 [INFO] serf: EventMemberJoin: consulserver 192.168.99.100
2018/09/08 10:35:38 [INFO] agent: Started DNS server 192.168.99.100:8600 (udp)
2018/09/08 10:35:38 [INFO] raft: Node at 192.168.99.100:8300 [Follower] entering Follower state (Leader: "")
2018/09/08 10:35:38 [INFO] consul: Adding LAN server consulserver (Addr: tcp/192.168.99.100:8300) (DC: dc1)
2018/09/08 10:35:38 [INFO] consul: Handled member-join event for server "consulserver.dc1" in area "wan"
2018/09/08 10:35:38 [INFO] agent: Started DNS server 192.168.99.100:8600 (tcp)
2018/09/08 10:35:38 [INFO] agent: Started HTTP server on 192.168.99.100:8500 (tcp)
2018/09/08 10:35:38 [INFO] agent: started state syncer
2018/09/08 10:35:44 [WARN] raft: Heartbeat timeout from "" reached, starting election
2018/09/08 10:35:44 [INFO] raft: Node at 192.168.99.100:8300 [Candidate] entering Candidate state in term 2
2018/09/08 10:35:44 [INFO] raft: Election won. Tally: 1
2018/09/08 10:35:44 [INFO] raft: Node at 192.168.99.100:8300 [Leader] entering Leader state
2018/09/08 10:35:44 [INFO] consul: cluster leadership acquired
2018/09/08 10:35:44 [INFO] consul: New leader elected: consulserver
2018/09/08 10:35:44 [INFO] consul: member 'consulserver' joined, marking health alive
2018/09/08 10:35:44 [INFO] agent: Synced node info
- Now if you gonna check the membership of consul it will fail because the endpoint has changed from loopback to eth1 ip address. Specify the endpoing with eth1 ip address in command to list consul members.
docker@consulserver:~$ docker exec -it f2fdb2148166 consul members
Error retrieving members: Get http://127.0.0.1:8500/v1/agent/members?segment=_all: dial tcp 127.0.0.1:8500: connect: connection refused
docker@consulserver:~$ docker exec -it f2fdb2148166 consul members --http-addr=192.168.99.100:8500
Node Address Status Type Build Protocol DC Segment
consulserver 192.168.99.100:8301 alive server 1.2.2 2 dc1 <all>
Final commands to start consul
- Create json file with the configuration and empty data directory which will help to maintain the state of consul cluster after restarts.
docker@consulserver:~$ cat conf.json
{
"datacenter":"labsetup",
"enable_debug":true
}
- Start the consul agent in server and client mode in different nodes.
docker@consulserver:~/data$ docker run -d --net=host -v /home/docker/conf.json:/consul/config/config.json -v /home/docker/data/:/consul/data/ -e CONSUL_BIND_INTERFACE=eth1 -e CONSUL_CLIENT_INTERFACE=eth1 -d consul agent -ui -server -bootstrap-expect=1
5cecf4554a0d51f2f969c3bdfd8638f21e49789bac90528766f26364dad58b23
docker@consulclient:~$ docker run -d --net=host -e 'CONSUL_LOCAL_CONFIG={"leave_on_terminate": true}' -v /home/docker/conf.json:/consul/config/config.json --name=consulagent1 consul agent --retry-join=192.168.99.100 -bind=192.168.99.101
6b3eac6dbf7452e4012e1a6979f5b34e8ed37f5954bf3c601e0af80885198c62
- Verify that consul member is showing both.
docker@consulserver:~$ docker exec -it 5cecf4554a0d consul members --http-addr=192.168.99.100:8500
Node Address Status Type Build Protocol DC Segment
consulserver 192.168.99.100:8301 alive server 1.2.2 2 labsetup <all>
consulclient 192.168.99.101:8301 alive client 1.2.2 2 labsetup <default>
- Following logs are reported continuously in docker logs, ignore them it seems like a known issue [1].
2018/09/08 11:14:05 [INFO] consul: member 'consulclient' joined, marking health alive
2018/09/08 11:14:15 [WARN] memberlist: Was able to connect to consulclient but other probes failed, network may be misconfigured
2018/09/08 11:14:22 [WARN] memberlist: Was able to connect to consulclient but other probes failed, network may be misconfigured
2018/09/08 11:14:29 [WARN] memberlist: Was able to connect to consulclient but other probes failed, network may be misconfigured
2018/09/08 11:14:35 [WARN] memberlist: Was able to connect to consulclient but other probes failed, network may be misconfigured
[1] https://github.com/hashicorp/consul/issues/3411
References:
[2] https://store.docker.com/images/consul [3] https://www.consul.io/docs/internals/architecture.html [4] https://github.com/hashicorp/docker-consul/blob/75d3ac383a8401b41a5f80bb1713ff6924584c5f/0.X/docker-entrypoint.sh