Trino 424 and 425 fails to start in Docker with exception when loading tpcds plugin

[eltimen]

I tried to start latest version of Trino (424) in a Docker container following this instruction. The container crashes with the exit code 100 during loading tpcds plugin with the following errors in the log

2023-08-25T07:46:06.822Z        INFO    main    io.trino.server.PluginManager   -- Finished loading plugin /data/trino/plugin/thrift --
2023-08-25T07:46:06.822Z        INFO    main    io.trino.server.PluginManager   -- Loading plugin /data/trino/plugin/tpcds --
2023-08-25T07:46:06.834Z        INFO    main    io.trino.server.PluginManager   -- Loading plugin /data/trino/plugin/tpch --
2023-08-25T07:46:06.840Z        ERROR   main    io.trino.server.Server  No service providers of type io.trino.spi.Plugin in the classpath: [file:/data/trino/plugin/tpcds/HdrHistogram-2.1.9.jar, file:/data/trino/plugin/tpcds/aopalliance-1.0.jar, file:/data/trino/plugin/tpcds/bootstrap-235.jar, file:/data/trino/plugin/tpcds/byte-buddy-1.14.5.jar, file:/data/trino/plugin/tpcds/checker-qual-3.33.0.jar, file:/data/trino/plugin/tpcds/classmate-1.5.1.jar, file:/data/trino/plugin/tpcds/concurrent-235.jar, file:/data/trino/plugin/tpcds/configuration-235.jar, file:/data/trino/plugin/tpcds/error_prone_annotations-2.21.1.jar, file:/data/trino/plugin/tpcds/failureaccess-1.0.1.jar, file:/data/trino/plugin/tpcds/guava-32.1.1-jre.jar, file:/data/trino/plugin/tpcds/guice-7.0.0.jar, file:/data/trino/plugin/tpcds/hibernate-validator-8.0.0.Final.jar, file:/data/trino/plugin/tpcds/http-client-235.jar, file:/data/trino/plugin/tpcds/http2-client-11.0.15.jar, file:/data/trino/plugin/tpcds/http2-common-11.0.15.jar, file:/data/trino/plugin/tpcds/http2-hpack-11.0.15.jar, file:/data/trino/plugin/tpcds/http2-http-client-transport-11.0.15.jar, file:/data/trino/plugin/tpcds/j2objc-annotations-2.8.jar, file:/data/trino/plugin/tpcds/jackson-core-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-databind-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-guava-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-jdk8-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-joda-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-jsr310-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-module-parameter-names-2.15.2.jar, file:/data/trino/plugin/tpcds/jakarta.annotation-api-2.1.1.jar, file:/data/trino/plugin/tpcds/jakarta.inject-api-2.0.1.jar, file:/data/trino/plugin/tpcds/jakarta.validation-api-3.0.2.jar, file:/data/trino/plugin/tpcds/jboss-logging-3.4.1.Final.jar, file:/data/trino/plugin/tpcds/jcl-over-slf4j-2.0.7.jar, file:/data/trino/plugin/tpcds/jetty-alpn-client-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-alpn-java-client-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-client-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-http-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-io-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-util-11.0.15.jar, file:/data/trino/plugin/tpcds/jmxutils-1.23.jar, file:/data/trino/plugin/tpcds/joda-time-2.12.5.jar, file:/data/trino/plugin/tpcds/json-235.jar, file:/data/trino/plugin/tpcds/jsr305-3.0.2.jar, file:/data/trino/plugin/tpcds/log-235.jar, file:/data/trino/plugin/tpcds/log-manager-235.jar, file:/data/trino/plugin/tpcds/log4j-api-2.17.1.jar, file:/data/trino/plugin/tpcds/log4j-over-slf4j-2.0.7.jar, file:/data/trino/plugin/tpcds/log4j-to-slf4j-2.17.1.jar, file:/data/trino/plugin/tpcds/logback-core-1.4.8.jar, file:/data/trino/plugin/tpcds/modernizer-maven-annotations-2.6.0.jar, file:/data/trino/plugin/tpcds/node-235.jar, file:/data/trino/plugin/tpcds/opentelemetry-semconv-1.26.0-alpha.jar, file:/data/trino/plugin/tpcds/security-235.jar, file:/data/trino/plugin/tpcds/slf4j-api-2.0.7.jar, file:/data/trino/plugin/tpcds/slf4j-jdk14-2.0.7.jar, file:/data/trino/plugin/tpcds/stats-235.jar, file:/data/trino/plugin/tpcds/tpcds-1.4.jar, file:/data/trino/plugin/tpcds/trace-token-235.jar, file:/data/trino/plugin/tpcds/trino-cache-424.jar, file:/data/trino/plugin/tpcds/trino-matching-424.jar, file:/data/trino/plugin/tpcds/trino-plugin-toolkit-424.jar, file:/data/trino/plugin/tpcds/trino-tpcds-424-services.jar, file:/data/trino/plugin/tpcds/trino-tpcds-424.jar, file:/data/trino/plugin/tpcds/units-1.9.jar]
java.lang.IllegalStateException: No service providers of type io.trino.spi.Plugin in the classpath: [file:/data/trino/plugin/tpcds/HdrHistogram-2.1.9.jar, file:/data/trino/plugin/tpcds/aopalliance-1.0.jar, file:/data/trino/plugin/tpcds/bootstrap-235.jar, file:/data/trino/plugin/tpcds/byte-buddy-1.14.5.jar, file:/data/trino/plugin/tpcds/checker-qual-3.33.0.jar, file:/data/trino/plugin/tpcds/classmate-1.5.1.jar, file:/data/trino/plugin/tpcds/concurrent-235.jar, file:/data/trino/plugin/tpcds/configuration-235.jar, file:/data/trino/plugin/tpcds/error_prone_annotations-2.21.1.jar, file:/data/trino/plugin/tpcds/failureaccess-1.0.1.jar, file:/data/trino/plugin/tpcds/guava-32.1.1-jre.jar, file:/data/trino/plugin/tpcds/guice-7.0.0.jar, file:/data/trino/plugin/tpcds/hibernate-validator-8.0.0.Final.jar, file:/data/trino/plugin/tpcds/http-client-235.jar, file:/data/trino/plugin/tpcds/http2-client-11.0.15.jar, file:/data/trino/plugin/tpcds/http2-common-11.0.15.jar, file:/data/trino/plugin/tpcds/http2-hpack-11.0.15.jar, file:/data/trino/plugin/tpcds/http2-http-client-transport-11.0.15.jar, file:/data/trino/plugin/tpcds/j2objc-annotations-2.8.jar, file:/data/trino/plugin/tpcds/jackson-core-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-databind-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-guava-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-jdk8-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-joda-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-datatype-jsr310-2.15.2.jar, file:/data/trino/plugin/tpcds/jackson-module-parameter-names-2.15.2.jar, file:/data/trino/plugin/tpcds/jakarta.annotation-api-2.1.1.jar, file:/data/trino/plugin/tpcds/jakarta.inject-api-2.0.1.jar, file:/data/trino/plugin/tpcds/jakarta.validation-api-3.0.2.jar, file:/data/trino/plugin/tpcds/jboss-logging-3.4.1.Final.jar, file:/data/trino/plugin/tpcds/jcl-over-slf4j-2.0.7.jar, file:/data/trino/plugin/tpcds/jetty-alpn-client-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-alpn-java-client-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-client-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-http-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-io-11.0.15.jar, file:/data/trino/plugin/tpcds/jetty-util-11.0.15.jar, file:/data/trino/plugin/tpcds/jmxutils-1.23.jar, file:/data/trino/plugin/tpcds/joda-time-2.12.5.jar, file:/data/trino/plugin/tpcds/json-235.jar, file:/data/trino/plugin/tpcds/jsr305-3.0.2.jar, file:/data/trino/plugin/tpcds/log-235.jar, file:/data/trino/plugin/tpcds/log-manager-235.jar, file:/data/trino/plugin/tpcds/log4j-api-2.17.1.jar, file:/data/trino/plugin/tpcds/log4j-over-slf4j-2.0.7.jar, file:/data/trino/plugin/tpcds/log4j-to-slf4j-2.17.1.jar, file:/data/trino/plugin/tpcds/logback-core-1.4.8.jar, file:/data/trino/plugin/tpcds/modernizer-maven-annotations-2.6.0.jar, file:/data/trino/plugin/tpcds/node-235.jar, file:/data/trino/plugin/tpcds/opentelemetry-semconv-1.26.0-alpha.jar, file:/data/trino/plugin/tpcds/security-235.jar, file:/data/trino/plugin/tpcds/slf4j-api-2.0.7.jar, file:/data/trino/plugin/tpcds/slf4j-jdk14-2.0.7.jar, file:/data/trino/plugin/tpcds/stats-235.jar, file:/data/trino/plugin/tpcds/tpcds-1.4.jar, file:/data/trino/plugin/tpcds/trace-token-235.jar, file:/data/trino/plugin/tpcds/trino-cache-424.jar, file:/data/trino/plugin/tpcds/trino-matching-424.jar, file:/data/trino/plugin/tpcds/trino-plugin-toolkit-424.jar, file:/data/trino/plugin/tpcds/trino-tpcds-424-services.jar, file:/data/trino/plugin/tpcds/trino-tpcds-424.jar, file:/data/trino/plugin/tpcds/units-1.9.jar]
        at com.google.common.base.Preconditions.checkState(Preconditions.java:835)
        at io.trino.server.PluginManager.loadPlugin(PluginManager.java:167)
        at io.trino.server.PluginManager.loadPlugin(PluginManager.java:157)
        at io.trino.server.ServerPluginsProvider.lambda$loadPlugins$1(ServerPluginsProvider.java:57)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
        at io.trino.util.Executors.executeUntilFailure(Executors.java:41)
        at io.trino.server.ServerPluginsProvider.loadPlugins(ServerPluginsProvider.java:52)
        at io.trino.server.PluginManager.loadPlugins(PluginManager.java:139)
        at io.trino.server.Server.doStart(Server.java:141)
        at io.trino.server.Server.lambda$start$0(Server.java:91)
        at io.trino.$gen.Trino_424____20230825_074553_1.run(Unknown Source)
        at io.trino.server.Server.start(Server.java:91)
        at io.trino.server.TrinoServer.main(TrinoServer.java:38)

Full logs is available in the attachment: trino_error.txt
I also tried the old 416 image and it starts successfully.

Steps to reproduce:

1. docker run --name trino -d -p 8080:8080 trinodb/trino:latest
2. watch docker ps -a and wait the container to stop with error
3. docker logs trino

Environment:

• Docker 20.10.21
• Ubuntu 20.04 in WSL

[eltimen]

I tried the new 425 image and it also crashes at the same stage:

2023-08-28T07:40:31.730Z        ERROR   main    io.trino.server.Server  java.nio.file.FileSystemException: plugin/tpcds: Too many open files
java.io.UncheckedIOException: java.nio.file.FileSystemException: plugin/tpcds: Too many open files
        at io.trino.server.ServerPluginsProvider.listFiles(ServerPluginsProvider.java:82)
        at io.trino.server.ServerPluginsProvider.buildClassPath(ServerPluginsProvider.java:66)
        at io.trino.server.ServerPluginsProvider.lambda$loadPlugins$0(ServerPluginsProvider.java:58)
        at io.trino.server.PluginManager.loadPlugin(PluginManager.java:148)
        at io.trino.server.ServerPluginsProvider.lambda$loadPlugins$1(ServerPluginsProvider.java:57)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
        at io.trino.util.Executors.executeUntilFailure(Executors.java:41)
        at io.trino.server.ServerPluginsProvider.loadPlugins(ServerPluginsProvider.java:52)
        at io.trino.server.PluginManager.loadPlugins(PluginManager.java:139)
        at io.trino.server.Server.doStart(Server.java:141)
        at io.trino.server.Server.lambda$start$0(Server.java:91)
        at io.trino.$gen.Trino_425____20230828_074017_1.run(Unknown Source)
        at io.trino.server.Server.start(Server.java:91)
        at io.trino.server.TrinoServer.main(TrinoServer.java:38)
Caused by: java.nio.file.FileSystemException: plugin/tpcds: Too many open files
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:440)
        at java.base/java.nio.file.Files.newDirectoryStream(Files.java:482)
        at io.trino.server.ServerPluginsProvider.listFiles(ServerPluginsProvider.java:74)

trino_425_error.txt

[hashhar]

Check the ulimits on the host which get passed down to the container.

[eltimen]

Thank you, increasing the ulimit -n helped.