Discovery across nodes behind docker

[oxlade39]

Hi there. I have a bit of a custom setup that I'm trying to get working. I have several hosts across a network that have trino running.

1 coordinator, 6 workers.
Some of the workers will be on a separate physical host.

The workers that don't reside on the same physical host as the coordinator register with the coordinator via the discovery endpoint (they are visible in the nodes list), however, when work is sent to them they aren't able to respond and queries hang.

I think this is almost certainly because the workers are attempting to communicate over their internal docker ip.

Is there a way to configure which interface and port the workers publish?

Here's an example of what I see in the logs of one of the workers that doesn't appear to be able to accept work:

2023-08-08T12:23:34.888Z WARN http-client-node-manager-scheduler-1 io.trino.metadata.RemoteNodeState Error fetching node state from http://172.17.0.7:43000/v1/info/state: Failed communicating with server: http://172.17.0.7:43000/v1/info/state

172.17.0.7 is the ip of the docker container but not the host, so that address will not be publicly available.

[nineinchnick]

There are a few settings related to this, like node.bind-ip, check out the sources here: https://github.com/airlift/airlift/blob/master/node/src/main/java/io/airlift/node/NodeConfig.java
Looks like they're not documented in the Trino docs.

[oxlade39]

thanks I think that helps for the IP address.

Do you know if there is a way to configure the advertised port separately from the port the service starts on? Because it's starting within docker the port the service starts on is always 43000 in the above example, however the host port will be different.

[nineinchnick]

Have you tried setting different node.bind-ip and node.internal-address (or node.external-address)? Or maybe I don't understand the question, are you also talking about how to setup Docker networking?

[oxlade39]

From what I can tell the node.external-address will mean the ip resolves correctly but I also need to be able to change the port to differentiate between the internal port and the host port.

To add some context:

Each of the workers is running in docker behind a load balancer, the workers may or may not be on the same host and so won't necessarily be running in the same docker stack. Each worker will have its external port dynamically allocated by the load balancer but the internal port will be fixed (43000 as set by http-server.http.port).
The load balancer can route traffic to each of the workers (I can use host header routing and give each of the workers a unique host address worker1.trino.mycorp.com, worker2.trino.mycorp.com, etc) but I would need to then change the port to 443 for the https load balancer.

[cahya-wirawan]

Hi, we are having the same issue. We use 6 workers in 3 different docker hosts and 1 coordinator in another docker host. Since they are in different network, they can't communicate using the internal IP addresses.
I am new with Trino, so I am wondering where I can try to configure these node.bind-ip , node.internal-address and node.external-address. Thanks