Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with AWS GLUE 4 and AWSGlueDynamicSchema-0.9.0 #210

Open
anthonybastidas49 opened this issue May 13, 2024 · 0 comments
Open

Issue with AWS GLUE 4 and AWSGlueDynamicSchema-0.9.0 #210

anthonybastidas49 opened this issue May 13, 2024 · 0 comments

Comments

@anthonybastidas49
Copy link

I have a project with glue 4 and the appropriate scala and spark versions according to the AWS documentation.

"com.amazonaws" % "AWSGlueETL" % "4.0.0" % Provided, "com.vitorsvieira" %% "scala-iso" % "0.1.2", "org.apache.logging.log4j" % "log4j-core" % "2.13.1" % Provided, "org.apache.spark" %% "spark-core" % "3.3.2" % Provided, "org.apache.spark" %% "spark-mllib" % "3.3.2" % Provided, "org.apache.spark" %% "spark-sql" % "3.3.2" % Provided,
but I am getting the following error.

image Since the error occurred, the method does not receive 2 parameters in the jar (the dependency that is downloaded with the issue's upload date), but when commenting on this error I obtained a jar that I downloaded last April in which that method contains the syntax of the 2 parameters. And in addition to that there is no problem when running it in AWS As part of my tests I replaced the automatically downloaded (current) jar with the one I had previously downloaded and it worked without problems.

image

My question is why in AWS it does not download the same one as on my (current) computer, there is a risk that AWS will download the jar that is not compatible and my processes will give errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant