Example of IaC project with Terraform and Gitlab

Overview

Code presented below is the result of a project whose goal was to deploy several similar platforms (qualif, preprod, prod) each comprising several hundred VMs, distributed in different VAPP and connected to several networks.

Networks, VAPPs and VMs are defined in CSV input files, with one file for each type of object. They will be managed in an Infra as code (IaC) mode through the Terraform and Gitlab tools.

PFor example, each VM is defined by the vAPP that contains it, the amount of CPU and RAM allocated to it, the template used, the size of the system disk, the number and size of additional disks, the list of networks connected to the VM.

The code also contains a sample of Gitlab CI configuration, which allows the infrastructure to be automatically deployed whenever the code or input files are changed.

The Gitlab CI configuration defines a Pipeline that will be executed in a docker container. This pipeline will launch the execution of all the necessary Terraform commands (init, plan, apply) using some variables managed by the Gitlab (for example the connection identifiers to Cloud Avenue, the deployment environment depending on the modified git branch, etc…).

Project Organization

The entire Terraform configuration consists of one or more Terraform files (extension: .tf). Configuration is usually splited into several files for better readability.

If the configuration uses variables, a variables file (extension: .tfvars) will also be needed. You can have one variable file per environment.

terraform_ngp/
  .gitlab-ci.yml
  backend.tf
  inputs.tf
  network.tf
  output.tf
  output.tmpl
  provider.tf
  exemple.tfvars
  terraform.tf
  vapps.tf
  variables.tf
  vms.tf
  inputs/
    networks.csv
    vapps.csv
    vms.csv
  vm_modules/
    main.tf
    outputs.tf
    terraform.tf
    variables.tf

Terraform mandatory files

The 4 files below can be found in most Terraform configurations.

File terraform.tf

Contains the definition of the Terraform providers used: the Vmware VCD provider to use the VCD API and the local provider to read and write the csv files.

terraform {
  required_providers {
    # list provider used by the configuration.
    vcd = {
      source = "vmware/vcd"
      version = ">=3.5.1"
    }
    local = {
      source = "hashicorp/local"
      version = ">=2.1.0"
    }
  }
}

File backend.tf

Contains the definition of the backend where the status file will be stored, in this example we use the Gitlab functionality to store the status file.

# using gitlab registry to store tfstate
terraform {
  # all parameters are defined in environment by the gitlab CI/CD 
  backend "http" {
  }
}

File provider.tf

Contains the definition of the parameters for the provider(s) used: connection information to the vCD API, the parameters are defined as a variable.

provider "vcd" {
  # parameteres of provider vmware/vcd
  user                 = var.vcd_user
  password             = var.vcd_password
  org                  = var.vcd_org
  url                  = var.vcd_url
  vdc                  = var.edgegw_vdc
  allow_unverified_ssl = var.vcd_allow_unverified_ssl
}

# the provider local has no parameter

File variables.tf

Defines the variables used by the Terraform configuration, all variables must be declared to be used in the configuration and defined in the tfvars file, or in the environment a when running Terraform.

variable "vcd_user" {
    type = string
    description = "VCD User"
    sensitive = true
}
variable "vcd_password" {
    type = string
    description = "VCD Password"
    # sensitive variable are not displayed in standard output
    sensitive = true
}
variable "vcd_url" {
    type = string
    description = "VCD URL"
}
variable "vcd_allow_unverified_ssl" {
    type = bool
    description = "skip ssl certificate verification"
    default = false
}
variable "vcd_org" {
    type = string
    description = "VCD Organization name"
}
variable "catalog_name" {
    type = string
    description = "name of the catalog that contains the templates"
}
variable "edgegw_vdc" {
    type = string
    description = "Name of vdc that contains the edge gateway"
}
variable "edgegw_name" {
    type = string
    description = "Name of the edge gateway"
}
variable "net_csv" {
    type = string
    description = "filename of networks definition"
}
variable "vapp_csv" {
    type = string
    description = "filename of vapps definition"
}
variable "vm_csv" {
    type = string
    description = "filename of vms definition"
}
variable "output_csv" {
    type = string
    description = "finename of output data"
}

Project Specific Files

The following files are specific to this Terraform project.

File inputs.tf

Reading of input files. The code contained in this file reads the input csv files present in the “inputs” directory and transforms them into variables used in the rest of the configuration.

# read input data from csv file
data "local_file" "input_network" {
  # read the list of network in csv format
    filename = "${path.root}/inputs/${var.net_csv}"
}
data "local_file" "input_vapp" {
  # read the list of vapp in csv format
    filename = "${path.root}/inputs/${var.vapp_csv}"
}
data "local_file" "input_vm" {
  # read the list of vm in csv fornat
    filename = "${path.root}/inputs/${var.vm_csv}"
}
  
# transform the files content to a list of map for each input file
locals {
    net_data = csvdecode(data.local_file.input_network.content)
    vapp_data = csvdecode(data.local_file.input_vapp.content)
    vm_data = csvdecode(data.local_file.input_vm.content)
}

File networks.tf

Construction of objects of type network. This file contains 2 types of resources “vcd_network_routed_v2” and “vcd_network_isolated_v2”, it creates the routed and isolated networks, the routed networks are connected to an edge gateway.

data "vcd_nsxt_edgegateway" "edgegateway1" {
  # get the data for the edge gateway,
  # edge gateway id is needed for router network
     name = var.edgegw_name
     org  = var.vcd_org
     vdc  = var.edgegw_vdc
}

resource "vcd_network_routed_v2" "rnetwork" {
  # create a network for each network in the list with a network type of 'routed'
  for_each = {for x, net in local.net_data : net.NETWORK_NAME => net
                if lower(net.NETWORK_TYPE) == "routed" &&
                    net.ORG_NAME == var.vcd_org }
    org             = each.value.ORG_NAME
    vdc             = each.value.VDC_NAME
    name            = each.key

    # use the vcd_nsxt_edgegateway data to find the gateway id
    edge_gateway_id = data.vcd_nsxt_edgegateway.edgegateway1.id

    # define the ip address used by the gateway in the network
    # and define the network prefix. input is in format : x.x.x.x/y
    gateway         = split("/", each.value.GATEWAY)[0]
    prefix_length   = split("/", each.value.GATEWAY)[1]

    # define the ip pool address for the network,
    # input format is x.x.x.x|y.y.y.y
    static_ip_pool {
        start_address = split("|", each.value.STATIC_ADDRESS_POOL)[0]
        end_address   = split("|", each.value.STATIC_ADDRESS_POOL)[1]
    }
}

resource "vcd_network_isolated_v2" "inetwork" {
  # create a network for each network in the list with a network type of 'isolated'
  for_each = {for x, net in local.net_data : net.NETWORK_NAME => net
                if lower(net.NETWORK_TYPE) == "isolated" &&
                    net.ORG_NAME == var.vcd_org }
    org             = each.value.ORG_NAME
    vdc             = each.value.VDC_NAME
    name            = each.key

    # even if the network is not routed it need a ip address
    # for the gateway in the network
    # and define the network prefix. input is in format : x.x.x.x/y
    gateway         = split("/", each.value.GATEWAY)[0]
    prefix_length   = split("/", each.value.GATEWAY)[1]

    static_ip_pool {
        start_address = split("|", each.value.STATIC_ADDRESS_POOL)[0]
        end_address   = split("|", each.value.STATIC_ADDRESS_POOL)[1]
    }
}

File vapps.tf

Construction of vAPP objects and association of networks to each vAPP.

resource "vcd_vapp" "vapp1" {
  # define a vapp for each line in input file
    for_each    = {for x, vapp in local.vapp_data : vapp.VAPP_NAME => vapp if vapp.ORG_NAME == var.vcd_org }
    name        = each.key
    org         = each.value.ORG_NAME
    vdc         = each.value.VDC_NAME
}

locals {
  # transform data format since each vapp has a variable number of networks
    vapp_net = flatten([
        for vapp in local.vapp_data : [
            for net in split("|",vapp.ORG_NETWORK) : {
                org     = vapp.ORG_NAME
                vdc     = vapp.VDC_NAME
                vapp    = vapp.VAPP_NAME
                netname = net
                }
            ] if vapp.ORG_NAME == var.vcd_org
        ])
}

resource "vcd_vapp_org_network" "network1" {
  # attach org network into each vapp
    depends_on          = [vcd_vapp.vapp1,vcd_network_routed_v2.rnetwork, vcd_network_isolated_v2.inetwork]
    for_each            = {for x, net in local.vapp_net : "${net.vapp}-${net.netname}" => net }
    vapp_name           = each.value.vapp
    org                 = each.value.org
    vdc                 = each.value.vdc
    org_network_name    = each.value.netname
}

File vms.tf

Construction of VM objects. This file uses a Terraform “module” because each VM consists of several resources: a VM and one or more add-ons. See below the section “Using Terraform Module”.

module "vm_instance" {
    depends_on          = [vcd_vapp.vapp1, vcd_vapp_org_network.network1]
    source              = "./vm_modules"
    for_each            = {for x,vm in local.vm_data : vm.HOSTNAME => vm if
                              vm.ORGANIZATION == var.vcd_org }
    vm_name             = each.key
    organization        = each.value.ORGANIZATION
    vdc                 = each.value.VDC_NAME
    vapp_name           = each.value.VAPP
    catalog_name        = var.catalog_name
    template_name       = each.value.TEMPLATE
    memory              = tonumber(each.value.RAM_QUANTITY)
    cpus                = tonumber(each.value.CPU_QUANTITY)
    vm_storage_profile  = each.value.STORAGE_POLICY
    networks            = [
                            {name = each.value.NET1, ip = each.value.IP1},
                            {name = each.value.NET2, ip = each.value.IP2},
                            {name = each.value.NET3, ip = each.value.IP3},
                          ]
    osdisk              = each.value.DISK0
    disks               = [each.value.DISK1, each.value.DISK2, each.value.DISK3]
}
# the module returns the ip assigned to each interface and the admin password
output "out_vm" {
  value = module.vm_instance
  sensitive = true
}

File output.tf and output.tmpl

The output.tf file creates a result file in CSV format that contains the same information as the VM input file with the addition of the IP address assigned to the VM and the admin password defined when the VM was created. The output.tmpl file is a template for the creation of this result file.

locals {
  # transform the output of module vm_instance in a variable usable by the template
  out_data = {for k,v in module.vm_instance : k => {ip = v.vm_ip, admin = v.admin_password}}
}

resource "local_file" "output" {
  # create the output file form input vm data et output variable
    filename = var.output_csv
    file_permission = 0644
    content = templatefile("output.tmpl", {input = local.vm_data, out = local.out_data})
}

Terraform can use templates to generate files, the directives that can be used in a template are documented here

HOSTNAME,ORGANIZATION,VAPP,TEMPLATE,VDC_NAME,CPU_QUANTITY,RAM_QUANTITY,STORAGE_POLICY,DISK0,DISK1,DISK2,DISK3,NET1,IP1,NET2,IP2,NET3,IP3,ADMIN
%{ for host in input ~}
${host.HOSTNAME},${host.ORGANIZATION},${host.VAPP},${host.TEMPLATE},${host.VDC_NAME},${host.CPU_QUANTITY},${host.RAM_QUANTITY},${host.STORAGE_POLICY},${host.DISK0},${host.DISK1},${host.DISK2},${host.DISK3},${host.NET1},${out[host.HOSTNAME]["ip"][0]},${host.NET2},%{ if length(out[host.HOSTNAME]["ip"]) > 1 }${out[host.HOSTNAME]["ip"][1]}%{ endif },${host.NET3},%{ if length(out[host.HOSTNAME]["ip"]) > 2 }${out[host.HOSTNAME]["ip"][2]}%{ endif },${out[host.HOSTNAME]["admin"]}
%{ endfor ~}

File exemple.tfvars

Example file for defining variables. All the variables used by the configuration must be defined during Terraform execution. They can be defined in this file or as environment variable, otherwise Terraform will interactively ask for the value of each undefined variable. For sensitive variables (login, password) it is preferable not to put them in a file, but rather to define them in the environment which the CI/CD pipeline can do automatically.

vcd_org = "cav01xxxxxxxxxxxx"
vcd_url = "https://console2.cloudavenue.orange-business.com/api"
vcd_allow_unverified_ssl = true

edgegw_name = "tn01xxxxxxxxxxx"
edgegw_vdc = "your_vdc"
catalog_name = "your_catalog"

net_csv = "networks.csv"
vapp_csv = "vapps.csv"
vm_csv = "vms.csv"
output_csv = "output.csv"

Input Files

Inputs files are stored in the “inputs” directory and their names are defined in the variables file.

Define Networks

Each network is defined by the vdc that contains it, its name, the network type (isolated or routed), the IP address of the gateway, the mask in CIDR notation, the definition of the static address pool with its first and last IPs.

ORG_NAME,VDC_NAME,NETWORK_NAME,NETWORK_TYPE,GATEWAY,STATIC_ADDRESS_POOL
cav01xxxxxxxxxxxx,your_vdc,rtr_01,routed,192.168.0.1/24,192.168.0.10|192.168.0.100
cav01xxxxxxxxxxxx,your_vdc,iso_02,isolated,192.168.2.1/24,192.168.2.10|192.168.2.100
cav01xxxxxxxxxxxx,your_vdc,iso_03,isolated,192.168.3.1/24,192.168.3.10|192.168.3.100

Define vAPPs

Each vAPP is defined by the vdc that contains it, its name, and the list of networks that are attached to it separated by the “|” character

ORG_NAME,VDC_NAME,VAPP_NAME,ORG_NETWORK
cav01xxxxxxxxxxxx,your_vdc,vapp_01,rtr_01|iso_02|iso_03
cav01xxxxxxxxxxxx,your_vdc,vapp_02,rtr_01|iso_03
cav01xxxxxxxxxxxx,your_vdc,vapp_03,rtr_01|iso_03

Define VMs

Each VM is defined by its name, the vAPP and the vdc that contains it, the number of cpu, the amount of memory, the storage policy, the size of the system disks (disk0) and additional disks (disk1 to disk3), the name of the connected networks and possibly an ip if we want to fix it, otherwise it is assigned to the pool.

HOSTNAME,ORGANIZATION,VAPP,TEMPLATE,VDC_NAME,CPU_QUANTITY,RAM_QUANTITY,STORAGE_POLICY,DISK0,DISK1,DISK2,DISK3,NET1,IP1,NET2,IP2,NET3,IP3
vm-01,cav01xxxxxxxxxxxx,vapp_01,ubuntu-20,your_vdc,2,4,gold,4,0,0,0,rtr_01,,iso_02,,iso_03,
vm-02,cav01xxxxxxxxxxxx,vapp_01,ubuntu-20,your_vdc,2,2,gold,10,10,20,0,rtr_01,,iso_03,,,
vm-03,cav01xxxxxxxxxxxx,vapp_03,ubuntu-20,your_vdc,2,2,gold,15,10,0,0,rtr_01,,iso_02,,,

Using Terraform Module

In order to factorize the code, Terraform modules can be defined, which take the form of a directory containing Terraform files.

Here we will use a module to define VMs because they are made of several resources (a VM and additional disks).

Directory vm_module

Terraform module to create a vm with additional disks, this module itself consists of 4 files :

File vm_module/main.tf

Contains the definition of a VM and its additional disks.

# create a vm, most argument are variables
resource "vcd_vapp_vm" "vapp_vm" {
  name                   = var.vm_name
  computer_name          = var.vm_name
  org                    = var.organization
  vdc                    = var.vdc
  vapp_name              = var.vapp_name
  catalog_name           = var.catalog_name
  template_name          = var.template_name
  memory                 = var.memory * 1024
  cpus                   = var.cpus
  storage_profile        = var.vm_storage_profile
  cpu_hot_add_enabled    = true
  memory_hot_add_enabled = true

  dynamic "network" {
    for_each = [for net in var.networks : net if net.name != ""]
    content {
      adapter_type       = "VMXNET3"
      type               = "org"
      name               = network.value.name
      ip_allocation_mode = network.value.ip == "" ? "POOL" : "MANUAL"
      ip                 = network.value.ip
    }
  }

  override_template_disk {
    bus_type            = "paravirtual"
    size_in_mb          = var.osdisk * 1024
    bus_number          = 0
    unit_number         = 0
  }

  customization {
    allow_local_admin_password = true
    auto_generate_password     = true
    enabled                    = true
    force                      = false
  }
}

resource "vcd_vm_internal_disk" "diskadd" {
  depends_on  = [vcd_vapp_vm.vapp_vm]
  for_each    = {for idx, disk in var.disks : idx => disk if disk != "0"}
  org         = var.organization
  vdc         = var.vdc
  vapp_name   = var.vapp_name
  vm_name     = var.vm_name
  size_in_mb  = each.value * 1024
  bus_type    = var.disk_bus
  bus_number  = 0
  unit_number = each.key < 6 ? each.key + 1 : each.key + 2
}

File vm_module/terraform.tf

Defines the provider(s) used by the module, here as in the main configuration the Vmware VCD provider.

terraform {
  required_providers {
    vcd = {
      source  = "vmware/vcd"
      version = "3.5.1"
    }
  }
}

File vm_module/output.tf

Defines values returned by the module, here the IP address assigned to the VM and the admin password defined when building the VM.

output "vm_ip" {
  value = flatten([
    for network in resource.vcd_vapp_vm.vapp_vm.network : [
      network.ip
    ]
  ])
}

output "admin_password" {
  value = resource.vcd_vapp_vm.vapp_vm.customization[0].admin_password
  sensitive = true
}

File vm_module/variables.tf

Declare the variables used by the module. These variables are the call parameters of the module, and must be defined when calling the module.

variable "vm_name" {
  type        = string
  description = "name of vm (hostname)"
}
variable "organization" {
  type        = string
  description = "name of organization"
}
variable "vdc" {
  type        = string
  description = "name of vdc"
}
variable "vapp_name" {
  type        = string
  description = "name of the vapp containing the vm"
}
variable "catalog_name" {
  type        = string
  description = "name of catalog"
}
variable "template_name" {
  type        = string
  description = "name of vm template"
}
variable "memory" {
  type        = number
  description = "vm memory in GB"
}
variable "cpus" {
  type        = number
  description = "number of cpus"
}
variable "vm_storage_profile" {
  type        = string
  description = "storage profile for vm disks"
}
variable "networks" {
  type        = list(any)
  description = "network description"
}
variable "osdisk" {
  type        = number
  description = "size of system disk"
}
variable "disks" {
  type        = list(any)
  description = "list of additional disks size"
}
variable "disk_bus" {
  type        = string
  description = "type of disk bus"
  default     = "paravirtual"
}

Deploy with Gitlab CI

File .gitlab-ci.yml

Definition of CI/CD pipeline.

In this example, the pipepline automatically executes a plan Terraform task, then an apply Terraform task, it uses a docker image provided by gitlab.com that contains terraform plus a gitlab-terraform script that simplifies the pipeline configuration.

The sensitive variables (API user and password: QUALIF_VCD_USER and QUALIF_VCD_PASSWORD) are stored in Gitlab (procedure to define variables in Gitlab). And the pipeline defines them in the environment, as well as the parameters to store the state file in the Gitlab using the variables predefined by the CI (List of predefined variables). The output file is saved in the Gitlab artifacts.

image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest

variables:
  TF_ROOT: ${CI_PROJECT_DIR}

cache:
  paths:
    - ${TF_ROOT}/.terraform

before_script:
  - cd ${TF_ROOT}
  - terraform --version

stages:
  - qualif-plan
  - qualif-apply

.qualif:
  variables:
    TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/qualif
    TF_VAR_vcd_user: ${QUALIF_VCD_USER}
    TF_VAR_vcd_password: ${QUALIF_VCD_PASSWORD}
  tags:
    - private
    - prod
    - docker

qualif-plan:
  extends: .qualif
  stage: qualif-plan
  script:
    - gitlab-terraform init -backend-config=skip_cert_verification=true
    - gitlab-terraform validate
    - gitlab-terraform plan -var-file qualif.tfvars
    - gitlab-terraform plan-json
  artifacts:
    name: qualif-plan
    paths:
      - ${TF_ROOT}/plan.cache
    reports:
      terraform: ${TF_ROOT}/plan.json

qualif-apply:
  extends: .qualif
  stage: qualif-apply
  script:
    - gitlab-terraform apply
  dependencies:
    - qualif-plan
  artifacts:
    name: qualif-output
    paths:
      - ${TF_ROOT}/output.csv

Note regarding credentials

This project uses Gitlab variables to store the credentials used, for greater security it is recommended to use an external secret manager, the Gitlab documentation describes how to use for example Hashicorp Vault.

Overview

Practical sheets

API

Backup

Cloud Customer Space

Contractual documents

Help and technical assistance

Online help

Roles and users

Technical Dashboards

The Support

Virtual Data Center ressources

Kubernetes

Additional components

KaaS

Tooling & Troubleshooting

Network

Networks and External Interconnections (links)

VDC Networks

Secnumcloud

Security

Services Area

Log in to the service area

Services

Storage

Objet Storage (S3)

Advanced features implementation guide

Getting Started

Service user manual

Shared storage (Network storage)

VDC & Virtual Machines

vCOD

Virtual Machines

Advanced Operations

Known Issues

Main Operations

VMware Cloud Director (VCD)

Catalogs

VMWare Cloud Director Availability (vCDA)

Q & A

Services

Virtual datacenter

XaaS services

DBaaS

Kubernetes

Example of IaC project with Terraform and Gitlab

Overview

Project Organization

Terraform mandatory files

File terraform.tf

File backend.tf

File provider.tf

File variables.tf

Project Specific Files

File inputs.tf

File networks.tf

File vapps.tf

File vms.tf

File output.tf and output.tmpl

File exemple.tfvars

Input Files

Define Networks

Define vAPPs

Define VMs

Using Terraform Module

Directory vm_module

File vm_module/main.tf

File vm_module/terraform.tf

File vm_module/output.tf

File vm_module/variables.tf

Deploy with Gitlab CI

File .gitlab-ci.yml

Note regarding credentials