Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

no_node_available_exception - Version 2.3.1 (push indices from one cluster to another) #101

Open
MobileAppVault opened this issue Apr 26, 2016 · 3 comments
Open

no_node_available_exception - Version 2.3.1 (push indices from one cluster to another) #101

MobileAppVault opened this issue Apr 26, 2016 · 3 comments

Comments

@MobileAppVault
Copy link

Hi,

I've tried to push indices from one cluster to another and got the following error.

I already checked my firewall settings and all firewalls deactivated and I checkt the transport ports and they are also available. The file export is working fine.

Could you please have a look into that issue?

Request
http://10.00.00.00:9200/products/_push?&host=10.00.00.13&port=9300&cluster=TEST

Response

{
  "error": {
    "root_cause": [
      {
        "type": "no_node_available_exception",
        "reason": "no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}"
      }
    ],
    "type": "no_node_available_exception",
    "reason": "no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}"
  },
  "status": 503
}

Configuration

CLUSTER 1

  • Master Node ( no data )
  • Data Node

CLUSTER 2

  • Master Node ( with data )

Elastic Version 2.3.1

Here is the log file:

[2016-04-26 17:42:10,594][INFO ][org.xbib.elasticsearch.helper.client.BaseTransportClient] creating transport client on Windows 7 Java HotSpot(TM) 64-Bit Server VM Oracle Corporation 1.8.0_91-b14 25.91-b14 with effective settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}
[2016-04-26 17:42:10,599][INFO ][plugins                  ] [Sphinx] modules [], plugins [helper], sites []
[2016-04-26 17:42:10,640][INFO ][org.xbib.elasticsearch.helper.client.BaseTransportClient] trying to connect to [10.00.00.13:9300]
[2016-04-26 17:42:10,656][ERROR][KnapsackPushAction       ] no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}
NoNodeAvailableException[no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}]
    at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:163)
    at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66)
    at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
    at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
    at org.xbib.elasticsearch.rest.action.knapsack.push.RestKnapsackPushAction.handleRequest(RestKnapsackPushAction.java:76)
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
    at org.elasticsearch.rest.RestController.executeHandler(RestController.java:205)
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166)
    at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)
    at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)
    at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:449)
    at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-04-26 17:42:10,688][WARN ][rest.suppressed          ] /products/_push Params: {cluster=TEST, size=1000, port=9300, host=10.00.00.13, index=products, sort=_doc}
NoNodeAvailableException[no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=TEST, flush_interval=5s, host=10.00.00.13, max_actions_per_request=1000, max_concurrent_requests=8, path.plugins=.dontexist, port=9300, timeout=30s}]
    at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:163)
    at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101)
    at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66)
    at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
    at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
    at org.xbib.elasticsearch.rest.action.knapsack.push.RestKnapsackPushAction.handleRequest(RestKnapsackPushAction.java:76)
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
    at org.elasticsearch.rest.RestController.executeHandler(RestController.java:205)
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166)
    at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)
    at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)
    at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:449)
    at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
@avgl-sstraub
Copy link

same issue still existent in 2.3.3

@mahuxiaozi
Copy link

mahuxiaozi commented Aug 4, 2016 •
edited
Loading

same issue in 2.3.0, in debug model , i found the error below:
[2016-08-04 15:45:01,476][DEBUG][client.transport ] [Dionysus] node_sampler_interval[5s]
[2016-08-04 15:45:01,493][DEBUG][org.xbib.elasticsearch.common.GcMonitor] enabled [true], interval [1s], gc_threshold [{default=GcThreshold{name='default', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, young=GcThreshold{name='young', warnThreshold=1000, infoThreshold=700, debugThreshold=400}, old=GcThreshold{name='old', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}}]
[2016-08-04 15:45:01,503][INFO ][org.xbib.elasticsearch.helper.client.BaseTransportClient] trying to connect to [192.168.10.4:9300]
[2016-08-04 15:45:01,504][DEBUG][org.xbib.elasticsearch.helper.client.TransportClient] [Dionysus] adding address [{#transport#-1}{192.168.10.4}{192.168.10.4:9300}]
[2016-08-04 15:45:01,504][DEBUG][org.xbib.elasticsearch.helper.client.TransportClient] [Dionysus] failed to connect to node [{#transport#-1}{192.168.10.4}{192.168.10.4:9300}], removed from nodes list
ConnectTransportException[[][192.168.10.4:9300] general node connection failure]; nested: IllegalStateException[await_() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await_() from a different thread.];
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:937)
at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888)
at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
at org.xbib.elasticsearch.helper.client.TransportClient.connect(TransportClient.java:348)
at org.xbib.elasticsearch.helper.client.TransportClient.addTransportAddresses(TransportClient.java:281)
at org.xbib.elasticsearch.helper.client.BaseTransportClient.connect(BaseTransportClient.java:98)
at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:162)
at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112)
at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101)
at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66)
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
.......
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: await_() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await_() from a different thread.
at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:320)
at org.jboss.netty.channel.DefaultChannelFuture.await0(DefaultChannelFuture.java:284)
at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:263)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:950)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916)
... 63 more
[2016-08-04 15:45:01,505][ERROR][KnapsackPushAction ] no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=liveyc, flush_interval=5s, host=192.168.10.4, max_actions_per_request=1000, max_concurrent_requests=4, path.plugins=.dontexist, port=9300, timeout=30s}
NoNodeAvailableException[no cluster nodes available, check settings {client.transport.ignore_cluster_name=true, client.transport.ping_timeout=30s, client.transport.sniff=true, cluster.name=liveyc, flush_interval=5s, host=192.168.10.4, max_actions_per_request=1000, max_concurrent_requests=4, path.plugins=.dontexist, port=9300, timeout=30s}]
at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:164)
at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:112)
at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:101)
at org.xbib.elasticsearch.action.knapsack.push.TransportKnapsackPushAction.doExecute(TransportKnapsackPushAction.java:66)
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)

@jprante
Copy link
Owner

jprante commented Aug 4, 2016

Thanks. There are limitations in Elasticsearch, I can only think of fixing this by disabling Netty dead lock checking. Will be in next release #110

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jprante @MobileAppVault @avgl-sstraub @mahuxiaozi