Configuring Advanced Forwarding Actions for a LoadBalancer Ingress

Prerequisites

• A CCE standard or Turbo cluster is available, and the cluster version meets the requirements.
• The cluster can be accessed using kubectl. For details, see Accessing a Cluster Using kubectl.

Notes and Constraints

• This feature is only available when dedicated load balancers are used.
• This feature relies on the ELB advanced forwarding policies. Once enabled, the priority is not determined by the domain name or path matching. You can customize the forwarding policy priority as needed. For details about the forwarding policy priority, see Forwarding Policy Priorities of LoadBalancer Ingresses.

Configuring Header Writing or Deletion for a LoadBalancer Ingress

1. Use kubectl to access the cluster. For details, see Accessing a Cluster Using kubectl.
2. Create a YAML file named ingress-test.yaml. The file name can be customized.

   vi ingress-test.yaml

   An example YAML file of an ingress created using an existing load balancer is as follows:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/elb.class: performance
       kubernetes.io/elb.id: 034baaf0-40e8-4e39-b0d9-bf6e5b883cf9
       kubernetes.io/elb.port: "80"
       # Configure header writing or deletion for the Service named test-service.
       kubernetes.io/elb.actions.test-service: |
         [{
             "type": "InsertHeader",
             "InsertHeaderConfig": {
                 "key": "aa",
                 "value_type": "USER_DEFINED",
                 "value": "aa"
             }
         },
         {
             "type": "InsertHeader",
             "InsertHeaderConfig": {
                 "key": "bb",
                 "value_type": "SYSTEM_DEFINED",
                 "value": "ELB-ID"
             }
         },
         {
             "type": "InsertHeader",
             "InsertHeaderConfig": {
                 "key": "cc",
                 "value_type": "REFERENCE_HEADER",
                 "value": "cc"
             }
         },
         {
             "type": "RemoveHeader",
             "RemoveHeaderConfig": {
                 "key": "dd"
             }
         },
         {
             "type": "RemoveHeader",
             "RemoveHeaderConfig": {
                 "key": "ee"
             }
         }]      
     name: ingress-test
     namespace: default
   spec:
     ingressClassName: cce
     rules:
       - http:
           paths:
             - backend:
                 service:
                   name: test-service
                   port:
                     number: 8888
               path: /
               pathType: ImplementationSpecific
               property:
                 ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH

   Table 2 Annotations for advanced forwarding actions

   Parameter	Type	Description
   kubernetes.io/elb.actions.${svc_name}	String	Configure an advanced forwarding action for an ingress. ${svc_name} indicates the Service name, which can contain a maximum of 51 characters. If the annotation value is set to [], all advanced forwarding actions will be deleted. The value of the annotation for configuring header writing or deletion is a JSON string array. For details, see Table 3. For example: [{"type":"InsertHeader","InsertHeaderConfig":{"key":"aa","value_type":"USER_DEFINED","value":"aa"}}]

   Table 3 Parameters for rewriting a header

   Parameter	Description	Example
   type	Type of header rewriting. Options: InsertHeader: indicates that the header will be written. This field must be used with InsertHeaderConfig. RemoveHeader: indicates that the header will be deleted. This field must be used with RemoveHeaderConfig. NOTE: For advanced forwarding actions, you can add a maximum of five header writing or deletion configurations to an annotation.	None
   InsertHeaderConfig	Indicates that the header will be written. This parameter is used only when type is set to InsertHeader. key: key of the rewritten header. A key consists of 1 to 40 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed. A key cannot contain any of the following characters (case-insensitive): connection, upgrade, content-length, transfer-encoding, keep-alive, te, host, cookie, remoteip, authority, x-forwarded-host, x-forwarded-for, x-forwarded-for-port, x-forwarded-tls-certificate-id, x-forwarded-tls-protocol, x-forwarded-tls-cipher, x-forwarded-elb-ip, x-forwarded-port, x-forwarded-elb-id, x-forwarded-elb-vip, x-real-ip, x-forwarded-proto, x-nuwa-trace-ne-in, or x-nuwa-trace-ne-out value_type: type of the header to be written. If the header is deleted, this parameter becomes invalid. Options: USER_DEFINED: custom header REFERENCE_HEADER: header referenced by a user SYSTEM_DEFINED: header defined by the system value: value of the header to be written. If the header is deleted, this parameter becomes invalid. If value_type is set to SYSTEM_DEFINED, the value can only be CLIENT-PORT, CLIENT-IP, ELB-PROTOCOL, ELB-ID, ELB-PORT, ELB-EIP, or ELB-VIP.	{ "type": "InsertHeader", "InsertHeaderConfig": { "key": "aa", "value_type": "USER_DEFINED", "value": "aa" } }
   RemoveHeaderConfig	Indicates that the header will be deleted. This parameter is used only when type is set to RemoveHeader. key: key of the deleted header. A key consists of 1 to 40 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed. A key cannot contain any of the following characters (case-insensitive): connection, upgrade, content-length, transfer-encoding, keep-alive, te, host, cookie, remoteip, authority, x-forwarded-host, x-forwarded-for, x-forwarded-for-port, x-forwarded-tls-certificate-id, x-forwarded-tls-protocol, x-forwarded-tls-cipher, x-forwarded-elb-ip, x-forwarded-port, x-forwarded-elb-id, x-forwarded-elb-vip, x-real-ip, x-forwarded-proto, x-nuwa-trace-ne-in, or x-nuwa-trace-ne-out	{ "type": "RemoveHeader", "RemoveHeaderConfig": { "key": "ee" } }

3. Create an ingress.

   kubectl create -f ingress-test.yaml

   If information similar to the following is displayed, the ingress has been created:

   ingress/ingress-test created

4. Check the created ingress.

   kubectl get ingress

   If information similar to the following is displayed, the ingress has been created:

   NAME          CLASS    HOSTS     ADDRESS          PORTS   AGE
   ingress-test  cce      *         121.**.**.**     80      10s

Returning a Specific Response Body for a LoadBalancer Ingress

1. Use kubectl to access the cluster. For details, see Accessing a Cluster Using kubectl.
2. Create a YAML file named ingress-test.yaml. The file name can be customized.

   vi ingress-test.yaml

   An example YAML file of an ingress created using an existing load balancer is as follows:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/elb.class: performance
       kubernetes.io/elb.id: 034baaf0-40e8-4e39-b0d9-bf6e5b883cf9
       kubernetes.io/elb.port: "80"
       # Configure the capability of returning a fixed response body for the Service named test-service.
       kubernetes.io/elb.actions.test-service: |
        [{ 
          "type": "FixedResponse", 
          "fixedResponseConfig": { 
              "contentType": "text/plain", 
              "statusCode": "503",
              "messageBody": "503 error text"  
          } 
        }]
     name: ingress-test
     namespace: default
   spec:
     ingressClassName: cce
     rules:
       - http:
           paths:
             - backend:
                 service:
                   name: test-service
                   port:
                     number: 8888
               path: /
               pathType: ImplementationSpecific
               property:
                 ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH

   Table 4 Annotations for advanced forwarding actions

   Parameter	Type	Description
   kubernetes.io/elb.actions.${svc_name}	String	Configure an advanced forwarding action for an ingress. ${svc_name} indicates the Service name, which can contain a maximum of 51 characters. If the annotation value is set to [], all advanced forwarding actions will be deleted. The annotation value of a fixed response is a JSON string array. For details, see Table 5.

   Table 5 Parameters for a fixed response

   Parameter	Description	Example
   type	The value is fixed at FixedResponse, indicating that a fixed response will be returned. NOTE: For advanced forwarding actions, you can add a maximum of one fixed response configuration to an annotation.	None
   fixedResponseConfig	contentType: format of the returned content. The options are text/plain, text/css, text/html, application/javascript, and application/json. statusCode: By default, 2xx, 4xx, and 5xx status codes are supported. (Mandatory) messageBody: Enter 0 to 1024 characters. NOTICE: This parameter cannot be left empty when rate limiting is configured.	{ "type": "FixedResponse", "fixedResponseConfig": { "contentType": "text/plain", "statusCode": "503", "messageBody": "503 error text" } }

3. Create an ingress.

   kubectl create -f ingress-test.yaml

   If information similar to the following is displayed, the ingress has been created:

   ingress/ingress-test created

4. Check the created ingress.

   kubectl get ingress

   If information similar to the following is displayed, the ingress has been created:

   NAME          CLASS    HOSTS     ADDRESS          PORTS   AGE
   ingress-test  cce      *         121.**.**.**     80      10s

Configuring Traffic Limit for LoadBalancer Ingresses

1. Use kubectl to access the cluster. For details, see Accessing a Cluster Using kubectl.
2. Create a YAML file named ingress-test.yaml. The file name can be customized.

   vi ingress-test.yaml

   An example YAML file of an ingress created using an existing load balancer is as follows:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/elb.class: performance
       kubernetes.io/elb.id: 034baaf0-40e8-4e39-b0d9-bf6e5b883cf9
       kubernetes.io/elb.port: "80"
       # Configure ELB traffic limit for the test-service Service.
       kubernetes.io/elb.actions.test-service: |
        [{ 
          "type": "TrafficLimit", 
          "trafficLimitConfig": { 
              "QPS": 4,
              "perSourceIpQps": 2,
              "burst": 2
          } 
        }]
     name: ingress-test
     namespace: default
   spec:
     ingressClassName: cce
     rules:
       - http:
           paths:
             - backend:
                 service:
                   name: test-service
                   port:
                     number: 8888
               path: /
               pathType: ImplementationSpecific
               property:
                 ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH

   Table 6 Annotations for advanced forwarding actions

   Parameter	Type	Description
   kubernetes.io/elb.actions.${svc_name}	String	Configure an advanced forwarding action for an ingress. ${svc_name} indicates the Service name, which can contain a maximum of 51 characters. If the annotation value is set to [], all advanced forwarding actions will be deleted. The value of an annotation for configuring traffic limit is a JSON array. For details, see Table 7.

   Table 7 Parameters for configuring traffic limit

   Parameter	Description	Example
   type	The value is fixed at TrafficLimit, indicating that traffic limit is enabled for a load balancer. NOTE: For advanced forwarding actions, you can add a maximum of one traffic limit configuration to an annotation.	None
   trafficLimitConfig	QPS: the total traffic limit, which specifies the maximum QPS value. The value ranges from 0 to 100000. Value 0 means no limit. If the number of requests reaches the specified value, new requests will be discarded and 503 Service Unavailable will be returned to the client. (Optional) perSourceIpQps: traffic limit per client source IP address. The value ranges from 0 to 100000. Value 0 means no limit. If both QPS and perSourceIpQps are configured, the latter value must be smaller than the former. If the number of requests reaches the specified value, new requests will be discarded and 503 Service Unavailable will be returned to the client. (Optional) burst: the size of a burst buffer. The value ranges from 0 to 100000. The burst rate enables temporary surges above the average rate to manage sudden spikes in requests. For instance, if the limit is set to 5 but the burst rate is 10, five more requests can be processed within 1 second. However, if the number of requests exceeds 10, only five requests are allowed within 1 second.	{ "type": "TrafficLimit", "trafficLimitConfig": { "QPS": "4", "perSourceIpQps": "2", "burst": "2" } }

3. Create an ingress.

   kubectl create -f ingress-test.yaml

   If information similar to the following is displayed, the ingress has been created:

   ingress/ingress-test created

4. Check the created ingress.

   kubectl get ingress

   If information similar to the following is displayed, the ingress has been created:

   NAME          CLASS    HOSTS     ADDRESS          PORTS   AGE
   ingress-test  cce      *         121.**.**.**     80      10s

Configuring a Custom Backend Server Group for a LoadBalancer Ingress

• Configuring a custom backend server group for a LoadBalancer ingress is subject to the capabilities of your ELB service. Before using this feature, submit a service ticket to enable the required ELB capabilities.
• Custom backend server groups cannot be used for grayscale release or fixed responses.
• If you configure a custom backend server group for a LoadBalancer ingress, the number field for the backend Services of the ingress will be invalid and the port name must be set to use-annotation. For details, see the example provided in this section.
• When configuring a custom backend server group for a LoadBalancer ingress, ensure that each Service name and backend server group ID are unique.

1. Use kubectl to access the cluster. For details, see Accessing a Cluster Using kubectl.
2. Create a YAML file named ingress-test.yaml. The file name can be customized.

   vi ingress-test.yaml

   An example YAML file of an ingress created using an existing load balancer is as follows:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       kubernetes.io/elb.class: performance
       kubernetes.io/elb.id: 034baaf0-40e8-4e39-b0d9-bf6e5b883cf9
       kubernetes.io/elb.port: "80"
       # Configure a custom backend server group for Service forward-demo, where the Service name in paths must match forward-demo and the port name must be use-annotation.
       kubernetes.io/elb.actions.forward-demo: |
        [{ 
          "type": "ForwardPool", 
          "forwardConfig": [{ 
             "serviceName": "test",
             "servicePort": 80,
             "weight": 90,
             "type": "service"
             },{ 
               "poolID": "53d8516e-xxxx-xxxx-xxxx-b15ffd6b3cca", 
               "weight": 70,
               "type": "pool"
            }]
        }] 
     name: ingress-test
     namespace: default
   spec:
     ingressClassName: cce
     rules:
       - host: example.com
         http:
           paths:
             - backend:
                 service:
                   name: forward-demo
                   port:
                     name: use-annotation   # When forwardPool is enabled, this parameter becomes invalid, and the port name must be set to use-annotation.
               path: /
               pathType: ImplementationSpecific
               property:
                 ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH

   Table 8 Annotations for advanced forwarding actions

   Parameter	Type	Description
   kubernetes.io/elb.actions.${svc_name}	String	Configure an advanced forwarding action for an ingress. ${svc_name} indicates the Service name, which can contain a maximum of 51 characters. If the annotation value is set to [], all advanced forwarding actions will be deleted. The value of an annotation for configuring a custom backend server group is a JSON string array. For details, see Table 9.

   Table 9 Parameters for configuring a custom backend server group

   Parameter	Description	Example
   type	The value is fixed at ForwardPool, indicating that a custom backend server group is configured for a load balancer. NOTE: A maximum of one custom backend server group can be added to an advanced forwarding action annotation.	None
   forwardConfig	NOTE: A maximum of five custom backend server groups can be added to this parameter. type: type of a custom backend server group, which can be service or pool. service: CCE automatically creates a backend server group for each Service and manages its lifecycle. pool: You need to create and maintain the backend server group on the ELB console. poolID: mandatory when type is set to pool. This parameter specifies the ID of the backend server group under the target ingress's load balancer. The ID must be unique for each group. You can go to the backend server group list, search for the backend server group of a load balancer using the name or ID of the load balancer associated with the ingress, and obtain the group ID. serviceName: mandatory when type is set to service. Before configuring a Service name, ensure that it is unique and the Service exists in the cluster. servicePort: mandatory when type is set to service. This parameter specifies a Service port. weight: weight for distributing traffic to each backend server group, ranging from 0 to 100.	{ "type": "ForwardPool", "forwardConfig": [{ "serviceName": "test", "servicePort": 80, "weight": 90, "type": "service" },{ "poolID": "53d8516e-xxxx-xxxx-xxxx-b15ffd6b3cca", "weight": 70, "type": "pool" }] }

3. Create an ingress.

   kubectl create -f ingress-test.yaml

   If information similar to the following is displayed, the ingress has been created:

   ingress/ingress-test created

4. Check the created ingress.

   kubectl get ingress

   If information similar to the following is displayed, the ingress has been created:

   NAME          CLASS    HOSTS         ADDRESS          PORTS   AGE
   ingress-test  cce      example.com   121.**.**.**     80      10s