Question

Proxy Protocol not working with L4 -> Ambassador TLS termination

Posted April 8, 2020 255 views
Kubernetes

Describe the bug
When proxy protocol is enabled on my L4 load balancer, ambassador does not terminate TLS/map to the service.

To Reproduce
Use the config below:

apiVersion: getambassador.io/v2
kind:  Module
metadata:
  name:  ambassador-users
  namespace: xxxx-user
spec:
  ambassador_id: ambassador-user
  config:
    use_remote_address: false
    use_proxy_proto: true
---
apiVersion: getambassador.io/v2
kind: TLSContext
metadata:
  name: global-tls
  namespace: xxxx-user
spec:
  ambassador_id: ambassador-user
  hosts:
    - xxxx.com
    - "*.xxxx.com"
  redirect_cleartext_from: 8080
  secret: xxxx-tls
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: frontend
  namespace: xxxx-user
spec:
  ambassador_id: ambassador-user
  host: "(\\w+).xxxx.com"
  host_regex: true
  prefix: /
  service: frontend.production:80

Ambassador is installed from the helm chart with the following config

---
# Default values for ambassador.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Manually set metadata for the Release.
#
# Defaults to .Chart.Name
nameOverride: ""
# Defaults to .Release.Name-.Chart.Name unless .Release.Name contains "ambassador"
fullnameOverride: ""

replicaCount: 3
daemonSet: false

# Enable autoscaling using HorizontalPodAutoscaler
# daemonSet: true, autoscaling will be disabled
autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
    - type: Resource
      resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 60

podDisruptionBudget: {}

# namespace:
  # name: default

# Additional container environment variable
# Uncomment or add additional environment variables for the container here.
env:
  AMBASSADOR_ID: ambassador-user
  # Exposing statistics via StatsD
  # STATSD_ENABLED: true
  # STATSD_HOST: statsd-sink
  # sets the minimum number of seconds between Envoy restarts
  # AMBASSADOR_RESTART_TIME: 15
  # sets the number of seconds that the Envoy will wait for open connections to drain on a restart
  # AMBASSADOR_DRAIN_TIME: 5
  # sets the number of seconds that Ambassador will wait for the old Envoy to clean up and exit on a restart
  # AMBASSADOR_SHUTDOWN_TIME: 10
  # labels Ambassador with an ID to allow for configuring multiple Ambassadors in a cluster

imagePullSecrets: []

securityContext:
  runAsUser: 8888

image:
  repository: quay.io/datawire/aes
  tag: 1.3.2
  pullPolicy: IfNotPresent

dnsPolicy: "ClusterFirst"
hostNetwork: false

service:
  type: LoadBalancer

  # Note that target http ports need to match your ambassador configurations service_port
  # https://www.getambassador.io/reference/modules/#the-ambassador-module
  ports:
    - name: http
      port: 80
      targetPort: 8080
      # protocol: TCP
      # nodePort: 30080
    - name: https
      port: 443
      targetPort: 8443
      # protocol: TCP
      # nodePort: 30443
    # TCPMapping_Port
      # port: 2222
      # targetPort: 2222
      # protocol: TCP
      # nodePort: 30222

  externalTrafficPolicy:

  annotations:
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "false"
    service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"

  #############################################################################
  ## Ambassador should be configured using CRD definition. If you want
  ## to use annotations, the following is an example of annotating the
  ## Ambassador service with global configuration manifest.
  ##
  ## See https://www.getambassador.io/reference/core/ambassador and
  ## https://www.getambassador.io/reference/core/tls for more info
  #############################################################################
  #
  #  getambassador.io/config: |
  #    ---
  #    apiVersion: ambassador/v1
  #    kind: TLSContext
  #    name: ambassador
  #    secret: ambassador-certs
  #    hosts: ["*"]
  #    ---
  #    apiVersion: ambassador/v1
  #    kind: Module
  #    name: ambassador
  #    config:
  #      admin_port: 8001
  #      diag_port: 8877
  #      diagnostics:
  #        enabled: true
  #      enable_grpc_http11_bridge: false
  #      enable_grpc_web: false
  #      enable_http10: false
  #      enable_ipv4: true
  #      enable_ipv6: false
  #      liveness_probe:
  #        enabled: true
  #      lua_scripts:
  #      readiness_probe:
  #        enabled: true
  #      server_name: envoy
  #      service_port: 8080
  #      use_proxy_proto: false
  #      use_remote_address: true
  #      xff_num_trusted_hops: 0
  #      x_forwarded_proto_redirect: false
  #      load_balancer:
  #        policy: round_robin
  #      circuit_breakers:
  #        max_connections: 2048
  #      retry_policy:
  #        retry_on: "5xx"
  #      cors:

adminService:
  create: true
  type: ClusterIP
  port: 8877
  # NodePort used if type is NodePort
  # nodePort: 38877
  annotations:
    {}

rbac:
  # Specifies whether RBAC resources should be created
  create: true
  podSecurityPolicies: []
  # Name of the RBAC resources defaults to the name of the release.
  # Set nameOverride when installing Ambassador with cluster-wide scope in
  # different namespaces with the same release name to avoid conflicts.
  nameOverride:

scope:
  # tells Ambassador to only use resources in the namespace or namespace set by namespace.name
  singleNamespace: false

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

deploymentStrategy:
  type: RollingUpdate

restartPolicy:

initContainers: []

sidecarContainers: []

livenessProbe:
  initialDelaySeconds: 30
  periodSeconds: 3
  failureThreshold: 3

readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 3
  failureThreshold: 3


volumes: []

volumeMounts: []

podLabels:
  {}

podAnnotations:
  {}
  # prometheus.io/scrape: "true"
  # prometheus.io/port: "9102"

deploymentAnnotations:
  {}
  # configmap.reloader.stakater.com/auto: "true"

resources:
  {}
  # If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

priorityClassName: ""

nodeSelector: {}

tolerations: []

affinity: {}

ambassadorConfig: ""

crds:
  enabled: true
  create: false
  keep: true

# The Ambassador Edge Stack is free for limited use without a license key.
# Go to https://{ambassador-host}/edge_stack/admin/#dashboard to register
# for a community license key.

enableAES: true

# Set createSecret: false is installing multiple releases of The Ambassador
# Edge Stack in the same namespace.
licenseKey:
  value:
  createSecret: true
  secretName:

# The DevPortal is exposed at /docs/ endpoint in the AES container. 
# Setting this to true will automatically create routes for the DevPortal.
createDevPortalMappings: true

# The Ambassador Edge Stack uses a redis instance for managing authentication,
# rate limiting, and sharing minor configuration details between pods for
# centralized management. These values configure the redis instance that ships
# by default with The Ambassador Edge Stack.
#
# URL of your redis instance. Defaults to redis instance created below.
redisURL:
# Ambassador ships with a basic redis instance. Configure the deployment with the options below.
redis:
  create: true
  # Annotations for Ambassador Pro's redis instance.
  annotations:
    deployment:
      {}
    service:
      {}
  resources: {}
  # If you want to specify resources, uncomment the following
  # lines and remove the curly braces after 'resources:'.
  # These are placeholder values and must be tuned.
  #   limits:
  #     cpu: 100m
  #     memory: 256Mi
  #   requests:
  #     cpu: 50m
  #     memory: 128Mi
  nodeSelector: {}


# Configures the AuthService that ships with the Ambassador Edge Stack.
# Setting authService.create: false will not install the AES AuthService and
# allow you to define your own.
#
# Typically when using the AES, you will want to keep this set to true and use 
# the External Filter to communicate with a custom authentication service.
# https://www.getambassador.io/reference/filter-reference/#filter-type-external
authService:
  create: false
  # Set additional configuration options. See https://www.getambassador.io/reference/services/auth-service for more information
  optional_configurations:
    # include_body:
    #   max_bytes: 4096
    #   allow_partial: true
    # status_on_error:
    #   code: 403
    # failure_mode_allow: false
    # retry_policy:
    #   retry_on: "5xx"
    #   num_retries: 2
    # add_linkerd_headers: true
    # timeout_ms: 30000


# Configures the RateLimitService in the Ambassador Edge Stack. 
# Keep this enabled to configure RateLimits in AES.
rateLimit:
  create: false

# DEPRECATED: Ambassador now exposes the /metrics endpoint in Envoy.
# DEPRECATED: See https://www.getambassador.io/user-guide/monitoring#deployment for more information on how to use the /metrics endpoint
#
# DEPRECATED: Enabling the prometheus exporter creates a sidecar and configures ambassador to use it
prometheusExporter:
  enabled: false
  repository: prom/statsd-exporter
  tag: v0.8.1
  pullPolicy: IfNotPresent
  resources: {}
  # If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  #   limits:
  #     cpu: 100m
  #     memory: 256Mi
  #   requests:
  #     cpu: 50m
  #     memory: 128Mi
  # You can configure the statsd exporter to modify the behavior of mappings and other features.
  # See documentation: https://github.com/prometheus/statsd_exporter/tree/v0.8.1#metric-mapping-and-configuration
  # Uncomment the following line if you wish to specify a custom configuration:
  # configuration: |
  #   ---
  #   mappings:
  #   - match: 'envoy.cluster.*.upstream_cx_connect_ms'
  #     name: "envoy_cluster_upstream_cx_connect_time"
  #     timer_type: 'histogram'
  #     labels:
  #       cluster_name: "$1"

Expected behavior
Ambassador terminates TLS and routes to the mapping

Versions (please complete the following information):

  • Ambassador: 1.32
  • Kubernetes environment: Digitalocean
  • Version: 1.16

Additional context
Routing works when proxy-protocol is disabled on the Load balancer.

Submit an answer

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!