[Proposal] Integrate Gateway as a Data Provider

[ZKentin]

Integrate Gateway as a Data Provider will enable to create new Groups from holders of a specific Gateway Credential NFT.

Specifications

{
  "name": "Gateway",
  "iconUrl": "",
  "providerClassName": "GatewayProvider",
  "functions": [
    {
      "name": "Get holders of credential NFT",
      "functionName": "TODO",
      "countFunctionName": "TODO",
      "description": "Returns all holders of a specific Gateway credential",
      "args": [
        {
          "name": "credential",
          "argName": "TODO",
          "type": "string",
          "required": "true",
          "example": "TODO",
          "description": "A specific credential identifier"
        }
      ]
    },
    {
      "name": "Get holders of organization",
      "functionName": "TODO",
      "countFunctionName": "TODO",
      "description": "Returns all holders of any Gateway credential of a specific organization",
      "args": [
        {
          "name": "organization",
          "argName": "TODO",
          "type": "string",
          "required": "true",
          "example": "TODO",
          "description": "A specific organization identifier"
        }
      ]
    }
  ]
}

Tips on how to integrate the Data Provider

Gateway NFTs are non-transferrable ERC-721 NFTs minted on Polygon with this contract address.

They don’t have an API, so the easiest way to extract the list of addresses is by using BigQuery & Arweave.

Have a look at Sismo documentation to understand what is a Data Provider, and checkout the tutorial to know how to integrate a Data Provider.

Have fun!

[sampolgar]

Notes

1. The Gateway metadata (Credential Name, Credential Type, Organization Id) are metadata of the Gateway NFT as available here.
2. There are currently 762,000 NFTs issued
3. We will probably need to use something that indexes data like Dune or The Graph

[MartinGbz]

Agree with this, again The Graph well be more welcomed than Dune 😅
But I think as it is specified in the issue: using BigQuery + Arweave seems ok to me isn't it?

[sampolgar]

I had a quick look on BigQuery, we can query the token_transfers which returns an array of objects like this:

{
  "token_address": "0x506402dbeeec75b2425d80b6f51e806684fd9f40",
  "from_address": "0x0000000000000000000000000000000000000000",
  "to_address": "0x5187c076ed0f393ead95d8f74c6ee98b12c08282",
  "value": "764065",
  "transaction_hash": "0xb7650d28e137bc39812e9953b6c15554bcb2d88e67fc55ee966641dad090d301",
  "log_index": "89",
  "block_timestamp": "2023-05-20 04:24:58.000000 UTC",
  "block_number": "42921051",
  "block_hash": "0x631a3bc77d94e428257dfcaebe74a48a2a6733868c88b0cb44c5bb9bd4bf8114"
}

Here we have the owner and tokenId but not the Credential Name, Credential Type, or Organization Id. Not sure how to use Arweave as it looks like these NFT's don't have metadata stored on Arweave.

Do you know how I can use BigQuery + Arweave?

[MartinGbz]

By using BigQuery (especially the getAllTransactionsForSpecificMethod of the BigQuery Provider) you can get fetch all NFT mints, and get so these 2 arguments:

So with the tokenURI_ (https://tws5ihsn6k6tlq5zd3re62mbwuzh6tp4pfie6rfxeajb34jipnlq.arweave.net/naXUHk3yvTXDuR7iT2mBtTJ_Tfx5UE9EtyASHfEoe1c) you can have all these info stored (including the attributes) on Arweave.
So it seems possible to do it, but I don't know if it's so scalable (but it worth to test it).

[sampolgar]

Awesome, thanks @MartinGbz, I'll have a play with the BigQuery, it looks like we just need to add a node env GOOGLE_APPLICATION_CREDENTIALS to run it locally?

RE Arweave - it seems some NFT metadata is stored on Arweave and some isn't
(results from the Alchemy getNFTMetadata query)
e.g. the Piggylet Podcast is on Arweave

{ "contract": { "address": "0x506402dbeeec75b2425d80b6f51e806684fd9f40" }, "id": { "tokenId": "762000", "tokenMetadata": { "tokenType": "ERC721" } }, "title": "Piggylet Podcast: Episode 12", "description": "Hey, lads'n'lasses. Welcome to Piggylet Podcast! Every week we share stats about NFT-Backed Lending, NFT sales, news about both cryptocurrencies and NFTs, and of course development notes of our NFT-Backed Lending Platform, Piggylet. You can claim your Piggylet Podcast Credential NFT for each episode!", "tokenUri": { "gateway": "https://arweave.net/i52Llq7NGbUlwN66CM_h4Ovct1-blli6sm8qw0Gws34", "raw": "https://arweave.net/i52Llq7NGbUlwN66CM_h4Ovct1-blli6sm8qw0Gws34" }, "media": [ { "gateway": "https://nft-cdn.alchemy.com/matic-mainnet/b2affa6e572103c6dd0d6010e8208d1a", "thumbnail": "https://res.cloudinary.com/alchemyapi/image/upload/thumbnailv2/matic-mainnet/b2affa6e572103c6dd0d6010e8208d1a", "raw": "https://node.mygateway.xyz/storage/file?id=d60740ad-8bc1-49cf-954a-87ebc700cf85", "format": "jpeg", "bytes": 462428 } ], "metadata": { "image": "https://node.mygateway.xyz/storage/file?id=d60740ad-8bc1-49cf-954a-87ebc700cf85", "issuanceDate": "2023-05-11T05:11:39.423Z", "name": "Piggylet Podcast: Episode 12", "description": "Hey, lads'n'lasses. Welcome to Piggylet Podcast! Every week we share stats about NFT-Backed Lending, NFT sales, news about both cryptocurrencies and NFTs, and of course development notes of our NFT-Backed Lending Platform, Piggylet. You can claim your Piggylet Podcast Credential NFT for each episode!", "attributes": [ { "value": 7, "trait_type": "Number of Tasks" }, { "value": "Piggylet Podcast: Episode 12", "trait_type": "Credential Name" }, { "value": "de458a10-025a-4f5a-8e8f-3d848b5de0e5", "trait_type": "Organization ID" } ], "proof": { "proofPurpose": "assertionMethod", "type": "HS256", "proofValue": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXNjcmlwdGlvbiI6IkhleSwgbGFkcyduJ2xhc3Nlcy4gV2VsY29tZSB0byBQaWdneWxldCBQb2RjYXN0ISBFdmVyeSB3ZWVrIHdlIHNoYXJlIHN0YXRzIGFib3V0IE5GVC1CYWNrZWQgTGVuZGluZywgTkZUIHNhbGVzLCBuZXdzIGFib3V0IGJvdGggY3J5cHRvY3VycmVuY2llcyBhbmQgTkZUcywgYW5kIG9mIGNvdXJzZSBkZXZlbG9wbWVudCBub3RlcyBvZiBvdXIgTkZULUJhY2tlZCBMZW5kaW5nIFBsYXRmb3JtLCBQaWdneWxldC4gWW91IGNhbiBjbGFpbSB5b3VyIFBpZ2d5bGV0IFBvZGNhc3QgQ3JlZGVudGlhbCBORlQgZm9yIGVhY2ggZXBpc29kZSEiLCJpbWFnZSI6Imh0dHBzOi8vbm9kZS5teWdhdGV3YXkueHl6L3N0b3JhZ2UvZmlsZT9pZD1kNjA3NDBhZC04YmMxLTQ5Y2YtOTU0YS04N2ViYzcwMGNmODUiLCJuYW1lIjoiUGlnZ3lsZXQgUG9kY2FzdDogRXBpc29kZSAxMiIsImlzc3VhbmNlRGF0ZSI6IjIwMjMtMDUtMTFUMDU6MTE6MzkuNDIzWiIsInByb29mT2ZXb3JrIjpbeyJ0aXRsZSI6IlN1YnNjcmliZSBQaWdneWxldCdzIENoYW5uZWxzIGFuZCBXYXRjaC8gTGlzdGVuIFBpZ2d5bGV0IFBvZGNhc3Q6IEVwaXNvZGUgMTIiLCJkZXNjcmlwdGlvbiI6IllvdVR1YmUgaXMgb25lIG9mIHRoZSBiZXN0IHBsYWNlcyB0byBsaXN0ZW4gdG8gdGhlIFBpZ2d5bGV0IFBvZGNhc3QhIERvbid0IG1pc3MgYW55IGVwaXNvZGVzIGJ5IHN1YnNjcmliaW5nIHRvIFBpZ2d5bGV0J3MgWW91VHViZSBjaGFubmVsIGFuZCBoaXR0aW5nIHRoZSBiZWxsIGJ1dHRvbiEiLCJ0YXNrX3R5cGUiOiJzZWxmX3ZlcmlmeSIsImNvbXBsZXRlZCI6dHJ1ZX0seyJ0aXRsZSI6IlBpZ2d5bGV0IERpc2NvcmQgSW52aXRhdGlvbiBMaW5rIiwiZGVzY3JpcHRpb24iOiJUbyByZWFjaCBQaWdneWxldCBEaXNjb3JkLCB5b3UgbmVlZCBhIGxpbmssIHNvIGhlcmUgaXQgaXMhIiwidGFza190eXBlIjoic2VsZl92ZXJpZnkiLCJjb21wbGV0ZWQiOnRydWV9LHsidGl0bGUiOiJGb2xsb3cgQHBpZ2d5bGV0ZGVmaSIsImRlc2NyaXB0aW9uIjoiRm9sbG93IHVzIGFuZCB0b2dldGhlciBsZXQncyBkZW1vY3JhdGl6ZSBORlQtQmFja2VkIExlbmRpbmcgc29sdXRpb25zISIsInRhc2tfdHlwZSI6InR3aXR0ZXJfZm9sbG93IiwiY29tcGxldGVkIjp0cnVlfSx7InRpdGxlIjoiUmV0d2VldCBQaWdneWxldCBQb2RjYXN0OiBFcGlzb2RlIDEyIiwiZGVzY3JpcHRpb24iOiJTaGFyZSBQaWdneWxldCBQb2RjYXN0IHdpdGggeW91ciBmcmllbmRzISIsInRhc2tfdHlwZSI6InR3aXR0ZXJfcmV0d2VldCIsImNvbXBsZXRlZCI6dHJ1ZX0seyJ0aXRsZSI6Ikxpa2UgUGlnZ3lsZXQgUG9kY2FzdDogRXBpc29kZSAxMiIsImRlc2NyaXB0aW9uIjoiU2hvdyB5b3VyIGxvdmUgdG8gUGlnZ3lsZXQgUG9kY2FzdCEiLCJ0YXNrX3R5cGUiOiJ0d2l0dGVyX2xpa2UiLCJjb21wbGV0ZWQiOnRydWV9LHsidGl0bGUiOiJQaWdneWxldCBXZWJzaXRlIGFuZCBtb3JlISIsImRlc2NyaXB0aW9uIjoiVGhlcmUgYXJlIG1vcmUgY2hhbm5lbHMgdG8gY29ubmVjdCB3aXRoIFBpZ2d5bGV0ISIsInRhc2tfdHlwZSI6InNlbGZfdmVyaWZ5IiwiY29tcGxldGVkIjp0cnVlfSx7InRpdGxlIjoiRW50ZXIgdGhlIEhpZGRlbiBQaHJhc2UiLCJkZXNjcmlwdGlvbiI6IldhdGNoIG9yIGxpc3RlbiB0byBFcGlzb2RlIDEyIG9mIHRoZSBQaWdneWxldCBQb2RjYXN0IHRvIGRpc2NvdmVyIHRoZSBoaWRkZW4gcGhyYXNlISBXZSdsbCBwcm92aWRlIGEgd2FybmluZyBkdXJpbmcgdGhlIHBvZGNhc3QganVzdCBiZWZvcmUgd2UgcmV2ZWFsIHRoZSBzZWNyZXQgcGhyYXNlIG9uIHRoZSBzY3JlZW4sIHNvIHN0YXkgYWxlcnQhIFRoZXJlIGFyZSBubyBjYXBpdGFsIGxldHRlcnMuIiwidGFza190eXBlIjoibWVldGluZ19jb2RlIiwiY29tcGxldGVkIjp0cnVlfV0sImF0dHJpYnV0ZXMiOlt7InRyYWl0X3R5cGUiOiJOdW1iZXIgb2YgVGFza3MiLCJ2YWx1ZSI6N30seyJ0cmFpdF90eXBlIjoiQ3JlZGVudGlhbCBOYW1lIiwidmFsdWUiOiJQaWdneWxldCBQb2RjYXN0OiBFcGlzb2RlIDEyIn0seyJ0cmFpdF90eXBlIjoiT3JnYW5pemF0aW9uIElEIiwidmFsdWUiOiJkZTQ1OGExMC0wMjVhLTRmNWEtOGU4Zi0zZDg0OGI1ZGUwZTUifV0sImlhdCI6MTY4Mzc4MTg5OX0.a6bIV1RkirakeaOcIcHuLkimJbJQ_um3g6_2kgiV4P0", "created": "2023-05-11T05:11:39.423Z" }, "proofOfWork": [ { "description": "YouTube is one of the best places to listen to the Piggylet Podcast! Don't miss any episodes by subscribing to Piggylet's YouTube channel and hitting the bell button!", "completed": true, "title": "Subscribe Piggylet's Channels and Watch/ Listen Piggylet Podcast: Episode 12", "task_type": "self_verify" }, { "description": "To reach Piggylet Discord, you need a link, so here it is!", "completed": true, "title": "Piggylet Discord Invitation Link", "task_type": "self_verify" }, { "description": "Follow us and together let's democratize NFT-Backed Lending solutions!", "completed": true, "title": "Follow @piggyletdefi", "task_type": "twitter_follow" }, { "description": "Share Piggylet Podcast with your friends!", "completed": true, "title": "Retweet Piggylet Podcast: Episode 12", "task_type": "twitter_retweet" }, { "description": "Show your love to Piggylet Podcast!", "completed": true, "title": "Like Piggylet Podcast: Episode 12", "task_type": "twitter_like" }, { "description": "There are more channels to connect with Piggylet!", "completed": true, "title": "Piggylet Website and more!", "task_type": "self_verify" }, { "description": "Watch or listen to Episode 12 of the Piggylet Podcast to discover the hidden phrase! We'll provide a warning during the podcast just before we reveal the secret phrase on the screen, so stay alert! There are no capital letters.", "completed": true, "title": "Enter the Hidden Phrase", "task_type": "meeting_code" } ] }, "timeLastUpdated": "2023-05-11T22:50:52.087Z", "contractMetadata": { "name": "Credential NFT", "symbol": "GNFT", "tokenType": "ERC721", "contractDeployer": "0x47640c1cca02c8209e989e02625e303e0a52406a", "deployedBlockNumber": 31814028, "openSea": { "lastIngestedAt": "2023-05-16T13:54:08.000Z" } }, "spamInfo": { "isSpam": "true", "classifications": [ "NoSalesActivity", "HighAirdropPercent" ] } }

or this is on Pinata

{ "contract": { "address": "0x506402dbeeec75b2425d80b6f51e806684fd9f40" }, "id": { "tokenId": "76400", "tokenMetadata": { "tokenType": "ERC721" } }, "title": "Goldfinch FAQ", "description": "This credential shows that you have read our Goldfinch FAQ in Discord and understand the basics about Goldfinch.", "tokenUri": { "gateway": "https://alchemy.mypinata.cloud/ipfs/Qme2k4PccAxsJZ2mBy1hDNfwMsNR7gzqRTvjGfQMBzDix9", "raw": "ipfs://Qme2k4PccAxsJZ2mBy1hDNfwMsNR7gzqRTvjGfQMBzDix9" }, "media": [ { "gateway": "https://nft-cdn.alchemy.com/matic-mainnet/f15537550d34da8bf82ee35ec4b2f84b", "thumbnail": "https://res.cloudinary.com/alchemyapi/image/upload/thumbnailv2/matic-mainnet/f15537550d34da8bf82ee35ec4b2f84b", "raw": "https://node.mygateway.xyz/storage/file?id=63d67262-79e3-4ccf-9061-e1d233057e3a", "format": "jpeg", "bytes": 1041301 } ], "metadata": { "image": "https://node.mygateway.xyz/storage/file?id=63d67262-79e3-4ccf-9061-e1d233057e3a", "issuanceDate": "2022-10-22T02:11:23.019Z", "name": "Goldfinch FAQ", "description": "This credential shows that you have read our Goldfinch FAQ in Discord and understand the basics about Goldfinch.", "attributes": [ { "value": 2, "trait_type": "Number of Tasks" } ], "proof": { "proofPurpose": "assertionMethod", "type": "HS256", "proofValue": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXNjcmlwdGlvbiI6IlRoaXMgY3JlZGVudGlhbCBzaG93cyB0aGF0IHlvdSBoYXZlIHJlYWQgb3VyIEdvbGRmaW5jaCBGQVEgaW4gRGlzY29yZCBhbmQgdW5kZXJzdGFuZCB0aGUgYmFzaWNzIGFib3V0IEdvbGRmaW5jaC4iLCJpbWFnZSI6Imh0dHBzOi8vbm9kZS5teWdhdGV3YXkueHl6L3N0b3JhZ2UvZmlsZT9pZD02M2Q2NzI2Mi03OWUzLTRjY2YtOTA2MS1lMWQyMzMwNTdlM2EiLCJuYW1lIjoiR29sZGZpbmNoIEZBUSIsImlzc3VhbmNlRGF0ZSI6IjIwMjItMTAtMjJUMDI6MTE6MjMuMDE5WiIsInByb29mT2ZXb3JrIjpbeyJ0aXRsZSI6IlJlYWQgb3VyIEZBUSBpbiB0aGUgR29sZGZpbmNoIERpc2NvcmQgQ2hhbm5lbCIsImRlc2NyaXB0aW9uIjoiUmVhZCB0aGUgRkFRIENoYW5uZWwiLCJ0YXNrX3R5cGUiOiJzZWxmX3ZlcmlmeSIsImNvbXBsZXRlZCI6dHJ1ZX0seyJ0aXRsZSI6IkhhdmUgeW91IHJlYWQgb3VyIEZBUT8iLCJkZXNjcmlwdGlvbiI6IkdvbGRmaW5jaCB3b3VsZCBsaWtlIHRvIHRlc3QgeW91ciBrbm93bGVkZ2Ugb24gb3VyIEZBUS5cblxuWW91IG11c3QgYW5zd2VyIDEwIG91dCBvZiAxMCBxdWVzdGlvbnMgcmlnaHQhIiwidGFza190eXBlIjoicXVpeiIsImNvbXBsZXRlZCI6dHJ1ZX1dLCJhdHRyaWJ1dGVzIjpbeyJ0cmFpdF90eXBlIjoiTnVtYmVyIG9mIFRhc2tzIiwidmFsdWUiOjJ9XSwiaWF0IjoxNjY2NDA0NjgzfQ.5_OHQy8_crdsaFggoCyw9zQkt29cu8XvmvkApbwqLGY", "created": "2022-10-22T02:11:23.019Z" }, "proofOfWork": [ { "description": "Read the FAQ Channel", "completed": true, "title": "Read our FAQ in the Goldfinch Discord Channel", "task_type": "self_verify" }, { "description": "Goldfinch would like to test your knowledge on our FAQ.\n\nYou must answer 10 out of 10 questions right!", "completed": true, "title": "Have you read our FAQ?", "task_type": "quiz" } ] }, "timeLastUpdated": "2023-04-16T04:03:44.882Z", "contractMetadata": { "name": "Credential NFT", "symbol": "GNFT", "tokenType": "ERC721", "contractDeployer": "0x47640c1cca02c8209e989e02625e303e0a52406a", "deployedBlockNumber": 31814028, "openSea": { "lastIngestedAt": "2023-05-16T13:54:08.000Z" } }, "spamInfo": { "isSpam": "true", "classifications": [ "NoSalesActivity", "HighAirdropPercent" ] } }

[MartinGbz]

Hey @sampolgar, in order to use BigQuery, you need to have a Google Cloud account and run this command gcloud auth login (no need to use the GOOGLE_APPLICATION_CREDENTIALS)

Regarding your remark: but here you are comparing 2 different NFTs, so that's normal, maybe the second one don't have attributes at all (it seems possible isn't it?)

[sampolgar]

Ok, I think we're stuck, can we meet to discuss how we will do this?

Here is what I see:

On Chain Data
We can use TheGraph or BigQuery to query the onchain data

• TokenId
• NFT URI (Arweave or IPFS link)

Off Chain Data
Seems we have 2 options to fetch metadata

1. query Arweave & IPFS
2. query Alchemy

There are over 750 000 NFTs minted, so querying Arweave/IPFS would be a 1 to 1 query (query 750000 times) or querying Alchemy is 65 NFTs at a time

We should definitely cache the results somewhere and query on a schedule, etc.

[sampolgar]

An update - the Alchemy SDK supports async queries & rate-limit handling. This would significantly reduce the time to bulk query the endpoint.
I'll test it soon
Rate limit
SDK

edit
ANKR supports 90 000 RPC call per minute (this is still about 10 minutes for Gateway) ANKR rate limit

[MartinGbz]

📢 Recap and Current State of the Issue - Integrate Gateway as a Data Provider

• There are over 750 000 NFTs minted
• NFTs are onchain
• NFT metadata ara off chain on Arweave
• The link between NFT and their metadata is a Arweave link write on chain

🔍 Solution found

• Solution1: Query with BigQuery each token and then query one by one the NFT metadata on Arweave => It seems not to be so scallable, but nobody try this method so maybe it can works well.
  The perfect solution would be to cache NFT metadata in order to not retrieve it each time.
• Solution2: Use Ankr, or Alchemy that they have surely a cache system on their side, so we can get easily the NFT holders and their metadata. In the Sismo Hub
  • We don't have an Alchemy premium account so we are rate limited
  • We have an Ankr premium account so we can do 90000 call per minute

👉 Next Steps

Implement one of these 2 solutions
Looking forward to getting this integration finalized.

If there are any further suggestions or ideas, please share them here.