Skip to content

Latest commit

 

History

History

dataplex

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 

Dataplex instance with lake, zone & assets

This module manages the creation of Dataplex instance along with lake, zone & assets in single regions.

Simple example

This example shows how to setup a Dataplex instance, lake, zone & asset creation in GCP project.

module "dataplex" {
  source     = "./fabric/modules/dataplex"
  name       = "terraform-lake"
  prefix     = "test"
  project_id = "myproject"
  region     = "europe-west2"
  zones = {
    landing = {
      type      = "RAW"
      discovery = true
      assets = {
        gcs_1 = {
          resource_name          = "gcs_bucket"
          cron_schedule          = "15 15 * * *"
          discovery_spec_enabled = true
          resource_spec_type     = "STORAGE_BUCKET"
        }
      }
    },
    curated = {
      type      = "CURATED"
      discovery = false
      assets = {
        bq_1 = {
          resource_name          = "bq_dataset"
          cron_schedule          = null
          discovery_spec_enabled = false
          resource_spec_type     = "BIGQUERY_DATASET"
        }
      }
    }
  }
}

# tftest modules=1 resources=5

IAM

This example shows how to setup a Dataplex instance, lake, zone & asset creation in GCP project assigning IAM roles at lake and zone level.

module "dataplex" {
  source     = "./fabric/modules/dataplex"
  name       = "lake"
  prefix     = "test"
  project_id = "myproject"
  region     = "europe-west2"
  iam = {
    "roles/dataplex.viewer" = [
      "group:[email protected]",
      "group:[email protected]"
    ]
  }
  zones = {
    landing = {
      type      = "RAW"
      discovery = true
      assets = {
        gcs_1 = {
          resource_name          = "gcs_bucket"
          cron_schedule          = "15 15 * * *"
          discovery_spec_enabled = true
          resource_spec_type     = "STORAGE_BUCKET"
        }
      }
    },
    curated = {
      type      = "CURATED"
      discovery = false
      iam = {
        "roles/viewer" = [
          "group:[email protected]",
          "group:[email protected]"
        ]
        "roles/dataplex.dataReader" = [
          "group:[email protected]",
          "group:[email protected]"
        ]
      }
      assets = {
        bq_1 = {
          resource_name          = "bq_dataset"
          cron_schedule          = null
          discovery_spec_enabled = false
          resource_spec_type     = "BIGQUERY_DATASET"
        }
      }
    }
  }
}

# tftest modules=1 resources=8

TODO

  • support multi-regions

Variables

name description type required default
name Name of Dataplex Lake. string
project_id The ID of the project where this Dataplex Lake will be created. string
region Region of the Dataplax Lake. string
zones Dataplex lake zones, such as RAW and CURATED. map(object({…}))
iam Dataplex lake IAM bindings in {ROLE => [MEMBERS]} format. map(list(string)) {}
location_type The location type of the Dataplax Lake. string "SINGLE_REGION"
prefix Optional prefix used to generate Dataplex Lake. string null

Outputs

name description sensitive
assets Assets attached to the lake of Dataplex Lake.
id Fully qualified Dataplex Lake id.
lake The lake name of Dataplex Lake.
zones The zone name of Dataplex Lake.