Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue #1809] Add load-transform SFN #2004

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

coilysiren
Copy link
Collaborator

Summary

Fixes #1809

Time to review: 2 mins

Changes proposed

Activates the load-transform job in every environment. In dev it runs with --no-* on everything, so it runs but doesn't do anything.

Copy link
Collaborator

@jamesbursa jamesbursa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for implementing!

Comment on lines +16 to +17
"poetry",
"run",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think running poetry in production isn't needed and adds additional complexity.

It's a development & build tool and the docker image has clean dependencies and is an isolated environment by running in a container.

When testing, I was using this command line:
["flask", "--app=src.app:create_app", "data-migration", "load-transform"]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task]: Setup Step Function / Orchestration for Transformations
3 participants