Deploy NGINX Ingress Controller on Cloud Container Engine

Technical additional documentation

Version Supported

This document is based on Cloud Container Engine version 1.21.

Introduction

In this document we will explain how you can replace the default Cloud Container Engine (CCE) Ingress with a different Ingress Controller. In this example we will use NGINX Ingress Controller but you can use any other you prefer.

You can find more informations about default CCE Ingress in the Help Center.

Prerequisite :

  • Virtual Private Cloud (VPC)
  • Cloud Container Engine cluster (CCE)
  • Elastic Loadbalancer (ELB)

Creating an Elastic Load Balancer

First you need to create a Elastic LoadBalancer (ELB) and bind it to a ElasticIP (EIP) if you don’t already have one. You can either do it with terraform or over the UI. I would recommend you go with terraform and use Flexible Engine Terraform Provider.

If you install it over the UI you need to do the following: Go to the ELB Dashboard and click on “Create Elastic Load Balancer”. Then create a new LoadBalancer inside the VPC where the CCE is hosted. After the creation if you click on the newly created ELB you will now see the following page :

Note the ELB ID (in green) : 055fdf89-9f35-46f9-a880-xxxxxxxxxxx

Default Ingress in CCE

If you create a ingress or a loadbalancer service inside your CCE, you can specify which controller should be used. This is defined by the annotation kubernetes.io/ingress.class, by default it looks like this:

apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
annotations: .... 
kubernetes.io/ingress.class: cce 
... 
spec: ...

In the example above the default CCE Ingress will be used. That means your traffic and ssl domain certificates will be handled by Flexible Engine Ingress (ELB). The good thing is, you don’t need to care about additional services and configuring a ingress controller. But on the other hand you have no control over the ingress controller.

Deploy an Ingress Controller

Using an existing Shared Elastic LoadBalancer

Before deploying NGINX Ingress Controller we need to customize the Helm chart and edit a values.yaml file as following :

controller:
  replicaCount: 1
  service:
    externalTrafficPolicy: Cluster
    annotations:
      kubernetes.io/elb.id: "ELB_ID"

Inside this file you need to set the ELB ID noted previously (ex : 055fdf89-9f35-46f9-a880-xxxxxxxxxxx).

Replace the part “ELB_ID” with your ELB ID, the values.yaml file shoud look like that :

controller:
  replicaCount: 1
  service:
    externalTrafficPolicy: Cluster
    annotations:
      kubernetes.io/elb.id: "055fdf89-9f35-46f9-a880-xxxxxxxxxxx"

Note : This method doesn’t support Dedicated ELB

Auto Create a Shared Elastic LoadBalancer

To automatically create a Shared ELB on the Ingress controller you use the following example :

controller:
  replicaCount: 1
  service:
    externalTrafficPolicy: Cluster
    annotations:   
      kubernetes.io/elb.class: union
      kubernetes.io/elb.autocreate: 
          '{
              "type": "public",
              "bandwidth_name": "cce-bandwidth-1551163379627",
              "bandwidth_chargemode": "traffic",
              "bandwidth_size": 5,
              "bandwidth_sharetype": "PER",
              "eip_type": "5_bgp",
              "name": "test"
            }'

You can find more detail in the Help Center : https://docs.prod-cloud-ocb.orange-business.com/usermanual2/cce/cce_01_0014.html

Auto Create a Dedicated Elastic LoadBalancer (Beta)

To automatically create a Dedicated ELB on the Ingress controller you use the following example :

controller:
  replicaCount: 1
  service:
    externalTrafficPolicy: Cluster
    annotations:
      kubernetes.io/elb.class: performance
      kubernetes.io/elb.autocreate: 
          '{
              "type":"public",
              "bandwidth_name":"cce-bandwidth-1655142139284",
              "bandwidth_chargemode":"traffic",
              "bandwidth_size":20,
              "bandwidth_sharetype":"PER",
              "eip_type":"5_bgp",
              "available_zone": ["eu-west-0a", "eu-west-0b"],
              "l4_flavor_name": "L4_flavor.elb.s1.small"
           }'
​

Dedicated ELB will be fully supported with CCE 1.23 release

Deploy with Helm

Once it is done, you can now deploy your Helm Chart with your customized values.yaml file :

helm upgrade -f values.yaml --install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespace

Wait up to a minute for the EIP and the Loadbalancer to connect.

You can see that IP address or FQDN with the following command :

kubectl get svc -n ingress-nginx

You can see that the EIP of the ELB is associated to the NGINX Controller with the following result :

NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.247.234.164   10.0.21.169,90.84.177.26   80:31360/TCP,443:32340/TCP   16h
ingress-nginx-controller-admission   ClusterIP      10.247.198.139   <none>                     443/TCP                      16h

Testing Ingress Controller

Let’s create a simple web server and the associated service :

kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo

Once you have the external IP address (or FQDN), set up a DNS record pointing to it. Then you can create an ingress resource. The following example assumes that you have set up a DNS record for www.demo.io:

kubectl create ingress demo --class=nginx \
  --rule="www.demo.io/*=demo:80"

Now that we switched the ingress class from “cce” to “nginx” our new Ingress Controller will now be responsible for all the traffic which is configured inside the Ingress configuration.

To Access your domain or the EIP, the traffic will now be handled by our newly created NGINX Ingress controller pod. The traffic flow looks like this:

Example of rewrite rule

Create an Ingress rule with a rewrite annotation :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2/
  name: rewrite
spec:
  ingressClassName: nginx
  rules:
  - host: demo.io
    http:
      paths:
      - path: /something(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: demo
            port: 
              number: 80

In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2, which is then used as a parameter in the rewrite-target annotation.

For example, the ingress definition above will result in the following rewrites:

  • demo.io/something rewrites to demo.io.com/
  • demo.io.com/something/ rewrites to demo.io.com/
  • demo.io/something/new rewrites to demo.io/new

Important

Other annotations like kubernetes.io/elb.subnet-id or kubernetes.io/elb.port are almost every time unnecessary and should not be added !

If you have trouble connecting the ELB it is best, to take a look at the events of the nginx service like this :

kubectl -n ingress-nginx describe svc ingress-nginx-controller