[Gitee] Add Gitee backend support to GrimoireLab ELK

[heming6666]

This PR chaoss/grimoirelab-perceval/pull/645 is working on adding Gitee backend support to Perceval. So maybe I can try to implement the Gitee backend to store the indexes in GrimoireLab ELK.

Since Gitee provides similar service as Github does, I will look at the raw and enrich connectors of Github for reference.

WDYT @valeriocos ?

[valeriocos]

Hi @heming6666 thank you for your proposal, it would be great! Let's wait some time to see when/where the Gitee backend will be added.

Since Gitee provides similar service as Github does, I will look at the raw and enrich connectors of Github for reference.

The GitLab connectors can be also useful to check, since they are simplified version of the ones for GitHub.

Thanks!

[heming6666]

Thank you @valeriocos for your reply!

I will follow the discussion raised in grimoirelab-perceval/pull/645 and grimoirelab-perceval/pull/640 around the integration of Gitee backend.

The GitLab connectors can be also useful to check, since they are simplified version of the ones for GitHub.

Thanks for your suggestion. I am trying to dive into the code of GitHub/GitLab connectors and will submit the pull request ASAP once Gitee backend is added into perceval.

[heming6666]

Hi @valeriocos , I saw that there are some proposals and discussion about integrating Airtable, Gitee and Trello backend to Perceval. And the current scenario we have is that people have their own repositories for these backends. In the case of Gitee, here's the grimoirelab-perceval-gitee repository.

So I wonder if we can have the similar way when adding ELK support for new backends. In this case, grimoirelab-elk-gitee would act as a plugin and can be loaded or used by the core of grimoirelab-elk.

• For the plugins, they need to export the connectors (e.g. Gitee, GiteeOcean, GiteeEnrich and GiteeCommand) and register themselves for discovery.
• For grimoirelab-elk, it only needs to discover the registered plugins automatically and then load the specific modules to import the connectors.

Inspired by @WillemJiang, I went through some examples about creating and dynamically loading plugins in python. One way is to use entry_points[1] for plugins to register themselves and use __import__()[2] in grimoirelab-elk to dynamically load the plugins. In this way, grimoirelab-elk can load all of the connectors without knowing the plugins in advance.

I have built a repository for testing this idea and here's some disscusion we have made about this idea.

I would really glad to hear some opinios from you @valeriocos and move this forward. If possible, I can open another issue to disscuse this idea and submit a PR to work on it.

Reference:
[1] https://packaging.python.org/guides/creating-and-discovering-plugins/#using-package-metadata
[2] https://www.geeksforgeeks.org/how-to-dynamically-load-modules-or-classes-in-python/

[valeriocos]

Hi @heming6666 , sorry for the late reply !

I think it's a great idea and overcome the current blockers to include new backends to Perceval.
Thank you for opening the PR!

[heming6666]

Here's the initial implementation of grimoirelab-elk-gitee. It registered the named entry points grimoire_elk in setup.py:

setup(name="grimoire-elk-gitee",
      packages=['grimoire_elk_gitee', 'grimoire_elk_gitee.enriched', 'grimoire_elk_gitee.raw', 'grimoire_elk_gitee.identities'],
      entry_points={"grimoire_elk": "gitee = grimoire_elk_gitee.utils:get_connectors"},
      )

And in grimoire_elk_gitee/utils.py, it exported the necessary connectors :

# Connectors for Perceval
from perceval.backends.gitee.gitee import Gitee, GiteeCommand
# Connectors for EnrichOcean
from .enriched.gitee import GiteeEnrich
# Connectors for Ocean
from .raw.gitee import GiteeOcean

def get_connectors():
    return {"gitee": [Gitee, GiteeOcean, GiteeEnrich, GiteeCommand]}

Then grimoirelab-elk can easily discover the gitee plugin and load all of the connectors correctly.

I will continue to implement grimoirelab-elk-gitee plugin (improve the code, add tests and documentation etc.)

[vchrombie]

The work is currently maintained at grimoirelab-gitee/grimoirelab-elk-gitee.

Closing the issue wrt chaoss/grimoirelab#439 (comment)