Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry mechanism for the given error #322

Open
1 task
hovaesco opened this issue Jul 12, 2023 · 1 comment
Open
1 task

Add retry mechanism for the given error #322

hovaesco opened this issue Jul 12, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@hovaesco
Copy link
Contributor

Describe the feature

Add retry mechanism for the given error, like for example DELTA_LAKE_BAD_WRITE.
That feature should be configurable, meaning that the user can specify one or more errors on which the query from the model will be retried. It should also include max number of retries and wait/sleep before the next attempt.

Describe alternatives you've considered

No response

Who will benefit?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@hovaesco hovaesco added the enhancement New feature or request label Jul 12, 2023
@Thelin90
Copy link

Thelin90 commented Jul 12, 2023

In dbt we utilise something called artifacts:

We use this to store test results into a table called dbt_results , on our project level on-run-end.

Which outputs test results to a single table with all test results.

We don’t run all jobs in one single DBT job but they are spread out.

This leads to multiple DBT jobs finishing, trying to write on

on-run-end:
    - "{{ store_dbt_results(results) }}"

The delta write lock issue then kicks in here since multiple DBT jobs, within the same single cluster, try to write their output to the results table from the actual dbt run and dbt test.

So even if we spread out the run of the jobs, collision can happen. So this retry would be good when you have a few collisions, or rare ones, we can try to avoid it as much as possible due to how we run and schedule the DBT jobs, but we still need to be able to avoid collision, when having multiple run, test output tables in this case for each run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants