Basic support for swarm mode services #588
Conversation
|
Hi @marcuslinke I'm not familiar with |
|
Thanks for your feedback @tuan3w. The issue seems to be related to docker remote API incompatibilities. Personally tested against docker 17.06.2-ce (remote API version v1.30) and it worked. So what version of the docker daemon do you use? As a workaround could you try to configure |
|
@tuan3w Please copy/paste output of |
|
Hi @marcuslinke, |
|
Registrator is deployed as container thats OK but it detects docker swarm vip services via docker remote API. According to your logs it tries to inspect a vip service with ID 0aowu9zo26qk468q5vqs56l7b (ip2loc:8089): It would be great if you could copy/paste the output of |
|
The output is as follow: |
|
Thanks @tuan3w. Output looks okay to me. So currently I don't know whats the problem then. Do you use the latest image pushed ( |
|
Yes, I'm using the latest image. If I use your configuration, the exception doesn't throws. However, it doesn't recognize service, both in |
|
So no services are registered in the backend? Hmmm, could you post your registrator logs please? Btw. what registry backend do you use? I've tested with consul here. And how do you deploy these services? Do you use a .yml file together with |
|
something like this: THe first two lines correspond to when I deploy service using |
|
Thats strange. In your first log there were the folllowing lines: You have to see this also when you configure |
|
I updated log output a little more. @marcuslinke , you can check it now. |
|
The first 22 lines of the log saying there where a bunch of vip services registered. And none of them appearing in the consul catalog after that? Thats very strange! What version of consul do you use? |
service naming (append port).
|
Okay. Probably found the reason for the exception you mentioned above. Will push a new docker image soon. |
|
Hi @marcuslinke, If I test with |
|
@tuan3w Registration on manager nodes only is a current architectural limitation. May be we can fix it sometimes but for now registrator (and especially its registry backend implementations for etcd, zookeeper, consul) doesn't allow cross node registrations. Each registrator instance can only register services that are running on the own node. Combined with the fact that on docker swarm worker nodes the appropriate endpoints of the docker remote API (like /services) are not available its not possible atm to register vip services on worker nodes. See initial post of this conversation. However, the current implementation should register |
|
I am very pleased to test this version. This option lacks the registrator. |
|
This version of the registrator works good for me. Thank you! |
|
I implemented my own swarm service registrator . Maybe it can help others. |
|
With : I' m using the service deploy mode like that:
1- 2- 3- I'll continue next week... Thank you. |
|
@rosebude To be clear here: I don't deploy registrator itself as a global swarm service. Registrator is deployed together with the consul agent as containers on each node because consul agent needs to be configured with the nodes ip address (via |
|
I deploy my consul as a service, using som Go magic on figuring out the IP-address needed for Explanatory details :
Basically using |
|
@marcuslinke In regards to proxy, we use Docker Flow Proxy |
|
BTW, for using registrator + consul + docker-flow-proxy. In this scheme (for me) consul is dns resolver only, port is default (80/443) and settings by docker-flow-proxy. So, i used fake port without any real port exposed: After that |
This is useful only when using registrator together with plain docker hosts (no swarm mode).
Service information seems replicated asynchronously to each swarm mode. So there are situations where service events reference services that are not replicated already to the node and therefore can't be inspected instantly.
|
Tried again with a few changes:
Create Overlay Network: Create Consul Seed server Create Consul server agents on manager nodes Create Consul agents on worker nodes Create registrator service on all nodes Once the Consul Server agents are running with Quorum, remove Consul Seed. |
|
For the files: I use the following compose file to deploy these services via |
|
Taken from your registrator service definition: |
|
When this option is set registrator registers the swarm manager service (Port 2367) within consul under the given service name. This is useful when using consul DNS capabilities to resolve a swarm manager IP from a common service name. To administrate a swarm you need to connect to one of the manager nodes. You can't "hardcode" this node name in your (deployment) scripts because a node maybe down for maintainance for example. So this way our scripts can dynamically resolve a running manager node to connect to. |
|
Related: #596 Allows you to use the IP assigned by the namespaced ingress network for the app being deployed to Swarm. Tested that PR w/ Consul - it has been working pretty well for me. Set up is a multinode Swarm cluster with Replicated/Consul on each host, deployed to workers/managers as part of the same Docker stack. |
|
I have 3 swarm nodes. And service started with docker service create \
--detach \
--name registrator \
--network host \
--mount type=bind,source=/var/run/docker.sock,destination=/tmp/docker.sock \
--mode global \
marcuslinke/registrator:2017-10-25 \
-cleanup=true \
-ttl=120 \
-ttl-refresh=55 \
-resync=30 \
-swarm-manager-servicename=swarm \
consul://localhost:8500 |
|
@a0s Could you provide an example how you started the service with |
docker service create \
--detach \
--name proxy \
--network proxy \
--publish published=80,target=80 \
--publish published=443,target=443 \
--env LISTENER_ADDRESS=swarm-listener \
--env STATS_USER=stats \
--env STATS_PASS=stats \
--env SERVICE_443_IGNORE=true \
--replicas 1 \
vfarcic/docker-flow-proxy |
|
This is working great for me! That said, I do wish the vip registration would honor the Currently it seems to honor the |
|
@marcuslinke mind updating the Dockerfile to use |
|
Ping pong. Any updates, @marcuslinke ? |
|
Sorry. I totally missed this. Currently I'm a bit busy and don't have time to maintain this PR. @josegonzalez Is there anything I can do to get this merged? |
… published port. Support multiple comma separated service names.
|
Just pushed a new image version 'marcuslinke/registrator:2019-03-28' that solves some issues. |
|
Hey folks! @marcuslinke are you still using it? I'm trying to integrate it with traefik: anotherone:
image: nginx:alpine
ports:
- "80"
environment:
- SERVICE_TAGS=traefik.enable=true,traefik.http.routers.app_anotherone.rule=PathPrefix(`/anotherone`),traefik.http.routers.app_anotherone.service=app_anotherone
- SERVICE_NAME=app_anotherone
deploy:
replicas: 6
placement:
preferences:
- spread: node.id
update_config:
parallelism: 0It's registered to consul, but for some reason SERVICE_TAGS are not added to consul, so at the end not configured in traefik (using consul catalog provider) |
I fixed it in this way: anotherone:
image: nginx:alpine
ports:
- "80"
deploy:
replicas: 6
placement:
preferences:
- spread: node.id
update_config:
parallelism: 0
labels:
- SERVICE_80_TAGS=traefik.enable=true,traefik.http.routers.app_anotherone.rule=PathPrefix(`/anotherone`),traefik.http.routers.app_anotherone.service=app_anotherone
- SERVICE_NAME=app_anotherone |
10 participants
WIP: Combines the efforts of #476 and #520 for supporting swarm mode services.
This PR implements registration of swarm mode vip services with one limitation: Ingress routed services (routed via routing mesh) will be registered on swarm manager nodes only. On worker nodes this is not possible at the moment due to the fact that swarm services can't be inspected via worker nodes remote API. Probably this isn't a big limitation as long as one have enough manager nodes within the swarm.
A short summarize of features added with this PR:
dnsrr/ingress) will be reflected in the registry backend.-swarm-manager-servicenameoption). This includes dynamic registration of all manager dependent services including ingress services. Promoting/demoting swarm nodes will be reflected dynamically.docker service update --label-rm "SERVICE_80_TAGS=foo" --label-add "SERVICE_80_TAGS=bar")-ttl-refresh)-ipoption in this case anymore.-ipoption (for example-ip eth0). This is useful on plain docker hosts to avoid hard-wiring a specific ip address with the registrator container.Two new options were added:
Image for testing is here https://hub.docker.com/r/marcuslinke/registrator/
@progrium Please review. Thanks!