Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[metadata] Remove metadata from empty enriched items #848

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[metadata] Remove metadata from empty enriched items #848

wants to merge 3 commits into from

Conversation

animeshk08
Copy link
Contributor

Fixes: #846
This commit updates the enricher for
github and gitlab to prevent addition of
metadata fields to empty enriched items

Signed-off-by: Animesh Kumar [email protected]

@animeshk08
Copy link
Contributor Author

animeshk08 commented Apr 15, 2020 •
edited
Loading

Hi @valeriocos. I have checked the enrichers for all the backends. Even though most of the backend override the enrich_items method, they do not require this fix as there no category based checks.
In other words, in all the other backends except GitHub and GitLab the enrich items are never empty.

Please have a look at this PR :)

Also, I will update the tests and separate the commits tomorrow :)

@coveralls
Copy link

coveralls commented Apr 15, 2020 •
edited
Loading

Pull Request Test Coverage Report for Build 2135

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 219 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.06%) to 79.573%
Files with Coverage Reduction New Missed Lines %
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/github2.py 16 94.19%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/gitlab.py 20 87.8%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/github.py 38 75.93%
/home/travis/build/chaoss/grimoirelab-elk/grimoire_elk/enriched/enrich.py 145 73.6%
Totals Coverage Status
Change from base Build 2133: 0.06%
Covered Lines: 7904
Relevant Lines: 9933
💛 - Coveralls

@animeshk08
[metadata] Remove metadata from empty enriched items
719604f 
This commit updates the enricher to prevent
addition of metadata fields to empty enriched items.

Signed-off-by: Animesh Kumar <[email protected]>
@animeshk08
[github] Remove metadata from non-enriched items
0574e55 
This commit prevents addition of repository label
and metadata filter raw to items which are not
to be enrihced. This results in creation of empty
items which are not uploaded to ElasticSearch.
Tests have been added accrodingly.

Signed-off-by: Animesh Kumar <[email protected]>
@animeshk08
[gitlab] Remove metadata from non-enriched items
0036e06 
This commit prevents addition of repository label
and metadata filter raw to items which are not
to be enrihced. This results in creation of empty
items which are not uploaded to ElasticSearch.
Tests have been added accrodingly.

Signed-off-by: Animesh Kumar <[email protected]>
@animeshk08
Copy link
Contributor Author

Hi, @valeriocos. Please have a look at this PR :)
SInce only GitHub and GitLab had such a category check of items the PR covers only these backends.
Do you think we should extend this check to other backends as well? This will serve two main purposes:

  • In case a raw item is not to be enriched at all.
  • In case fetching of a new category of item is integrated and work is being done on creating its enricher.
    WDYT?

@valeriocos
Copy link
Member

Hi @animeshk08 , sorry for the late reply! ^^

We are testing it in the pre-production environment, and the initial tests are good.
It will be probably go into the release 0.2.41 (not the upcoming one, but the next one).

@animeshk08
Copy link
Contributor Author

Thank you for the quick reply @valeriocos. Let me know when there are any updates.
We can extend the discussion on #848 (comment) then :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[enrich] Update decorator to add metadeta only for non-empty items
3 participants
@animeshk08 @coveralls @valeriocos