Question

Proxy Protocol not working with L4 -> Ambassador TLS termination

Posted April 8, 2020 597 views
Kubernetes

Describe the bug
When proxy protocol is enabled on my L4 load balancer, ambassador does not terminate TLS/map to the service.

To Reproduce
Use the config below:

apiVersion: getambassador.io/v2
kind:  Module
metadata:
  name:  ambassador-users
  namespace: xxxx-user
spec:
  ambassador_id: ambassador-user
  config:
    use_remote_address: false
    use_proxy_proto: true
---
apiVersion: getambassador.io/v2
kind: TLSContext
metadata:
  name: global-tls
  namespace: xxxx-user
spec:
  ambassador_id: ambassador-user
  hosts:
    - xxxx.com
    - "*.xxxx.com"
  redirect_cleartext_from: 8080
  secret: xxxx-tls
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: frontend
  namespace: xxxx-user
spec:
  ambassador_id: ambassador-user
  host: "(\\w+).xxxx.com"
  host_regex: true
  prefix: /
  service: frontend.production:80

Ambassador is installed from the helm chart with the following config

---
# Default values for ambassador.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Manually set metadata for the Release.
#
# Defaults to .Chart.Name
nameOverride: ""
# Defaults to .Release.Name-.Chart.Name unless .Release.Name contains "ambassador"
fullnameOverride: ""

replicaCount: 3
daemonSet: false

# Enable autoscaling using HorizontalPodAutoscaler
# daemonSet: true, autoscaling will be disabled
autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
    - type: Resource
      resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 60

podDisruptionBudget: {}

# namespace:
  # name: default

# Additional container environment variable
# Uncomment or add additional environment variables for the container here.
env:
  AMBASSADOR_ID: ambassador-user
  # Exposing statistics via StatsD
  # STATSD_ENABLED: true
  # STATSD_HOST: statsd-sink
  # sets the minimum number of seconds between Envoy restarts
  # AMBASSADOR_RESTART_TIME: 15
  # sets the number of seconds that the Envoy will wait for open connections to drain on a restart
  # AMBASSADOR_DRAIN_TIME: 5
  # sets the number of seconds that Ambassador will wait for the old Envoy to clean up and exit on a restart
  # AMBASSADOR_SHUTDOWN_TIME: 10
  # labels Ambassador with an ID to allow for configuring multiple Ambassadors in a cluster

imagePullSecrets: []

securityContext:
  runAsUser: 8888

image:
  repository: quay.io/datawire/aes
  tag: 1.3.2
  pullPolicy: IfNotPresent

dnsPolicy: "ClusterFirst"
hostNetwork: false

service:
  type: LoadBalancer

  # Note that target http ports need to match your ambassador configurations service_port
  # https://www.getambassador.io/reference/modules/#the-ambassador-module
  ports:
    - name: http
      port: 80
      targetPort: 8080
      # protocol: TCP
      # nodePort: 30080
    - name: https
      port: 443
      targetPort: 8443
      # protocol: TCP
      # nodePort: 30443
    # TCPMapping_Port
      # port: 2222
      # targetPort: 2222
      # protocol: TCP
      # nodePort: 30222

  externalTrafficPolicy:

  annotations:
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "false"
    service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"

  #############################################################################
  ## Ambassador should be configured using CRD definition. If you want
  ## to use annotations, the following is an example of annotating the
  ## Ambassador service with global configuration manifest.
  ##
  ## See https://www.getambassador.io/reference/core/ambassador and
  ## https://www.getambassador.io/reference/core/tls for more info
  #############################################################################
  #
  #  getambassador.io/config: |
  #    ---
  #    apiVersion: ambassador/v1
  #    kind: TLSContext
  #    name: ambassador
  #    secret: ambassador-certs
  #    hosts: ["*"]
  #    ---
  #    apiVersion: ambassador/v1
  #    kind: Module
  #    name: ambassador
  #    config:
  #      admin_port: 8001
  #      diag_port: 8877
  #      diagnostics:
  #        enabled: true
  #      enable_grpc_http11_bridge: false
  #      enable_grpc_web: false
  #      enable_http10: false
  #      enable_ipv4: true
  #      enable_ipv6: false
  #      liveness_probe:
  #        enabled: true
  #      lua_scripts:
  #      readiness_probe:
  #        enabled: true
  #      server_name: envoy
  #      service_port: 8080
  #      use_proxy_proto: false
  #      use_remote_address: true
  #      xff_num_trusted_hops: 0
  #      x_forwarded_proto_redirect: false
  #      load_balancer:
  #        policy: round_robin
  #      circuit_breakers:
  #        max_connections: 2048
  #      retry_policy:
  #        retry_on: "5xx"
  #      cors:

adminService:
  create: true
  type: ClusterIP
  port: 8877
  # NodePort used if type is NodePort
  # nodePort: 38877
  annotations:
    {}

rbac:
  # Specifies whether RBAC resources should be created
  create: true
  podSecurityPolicies: []
  # Name of the RBAC resources defaults to the name of the release.
  # Set nameOverride when installing Ambassador with cluster-wide scope in
  # different namespaces with the same release name to avoid conflicts.
  nameOverride:

scope:
  # tells Ambassador to only use resources in the namespace or namespace set by namespace.name
  singleNamespace: false

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

deploymentStrategy:
  type: RollingUpdate

restartPolicy:

initContainers: []

sidecarContainers: []

livenessProbe:
  initialDelaySeconds: 30
  periodSeconds: 3
  failureThreshold: 3

readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 3
  failureThreshold: 3


volumes: []

volumeMounts: []

podLabels:
  {}

podAnnotations:
  {}
  # prometheus.io/scrape: "true"
  # prometheus.io/port: "9102"

deploymentAnnotations:
  {}
  # configmap.reloader.stakater.com/auto: "true"

resources:
  {}
  # If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

priorityClassName: ""

nodeSelector: {}

tolerations: []

affinity: {}

ambassadorConfig: ""

crds:
  enabled: true
  create: false
  keep: true

# The Ambassador Edge Stack is free for limited use without a license key.
# Go to https://{ambassador-host}/edge_stack/admin/#dashboard to register
# for a community license key.

enableAES: true

# Set createSecret: false is installing multiple releases of The Ambassador
# Edge Stack in the same namespace.
licenseKey:
  value:
  createSecret: true
  secretName:

# The DevPortal is exposed at /docs/ endpoint in the AES container. 
# Setting this to true will automatically create routes for the DevPortal.
createDevPortalMappings: true

# The Ambassador Edge Stack uses a redis instance for managing authentication,
# rate limiting, and sharing minor configuration details between pods for
# centralized management. These values configure the redis instance that ships
# by default with The Ambassador Edge Stack.
#
# URL of your redis instance. Defaults to redis instance created below.
redisURL:
# Ambassador ships with a basic redis instance. Configure the deployment with the options below.
redis:
  create: true
  # Annotations for Ambassador Pro's redis instance.
  annotations:
    deployment:
      {}
    service:
      {}
  resources: {}
  # If you want to specify resources, uncomment the following
  # lines and remove the curly braces after 'resources:'.
  # These are placeholder values and must be tuned.
  #   limits:
  #     cpu: 100m
  #     memory: 256Mi
  #   requests:
  #     cpu: 50m
  #     memory: 128Mi
  nodeSelector: {}


# Configures the AuthService that ships with the Ambassador Edge Stack.
# Setting authService.create: false will not install the AES AuthService and
# allow you to define your own.
#
# Typically when using the AES, you will want to keep this set to true and use 
# the External Filter to communicate with a custom authentication service.
# https://www.getambassador.io/reference/filter-reference/#filter-type-external
authService:
  create: false
  # Set additional configuration options. See https://www.getambassador.io/reference/services/auth-service for more information
  optional_configurations:
    # include_body:
    #   max_bytes: 4096
    #   allow_partial: true
    # status_on_error:
    #   code: 403
    # failure_mode_allow: false
    # retry_policy:
    #   retry_on: "5xx"
    #   num_retries: 2
    # add_linkerd_headers: true
    # timeout_ms: 30000


# Configures the RateLimitService in the Ambassador Edge Stack. 
# Keep this enabled to configure RateLimits in AES.
rateLimit:
  create: false

# DEPRECATED: Ambassador now exposes the /metrics endpoint in Envoy.
# DEPRECATED: See https://www.getambassador.io/user-guide/monitoring#deployment for more information on how to use the /metrics endpoint
#
# DEPRECATED: Enabling the prometheus exporter creates a sidecar and configures ambassador to use it
prometheusExporter:
  enabled: false
  repository: prom/statsd-exporter
  tag: v0.8.1
  pullPolicy: IfNotPresent
  resources: {}
  # If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  #   limits:
  #     cpu: 100m
  #     memory: 256Mi
  #   requests:
  #     cpu: 50m
  #     memory: 128Mi
  # You can configure the statsd exporter to modify the behavior of mappings and other features.
  # See documentation: https://github.com/prometheus/statsd_exporter/tree/v0.8.1#metric-mapping-and-configuration
  # Uncomment the following line if you wish to specify a custom configuration:
  # configuration: |
  #   ---
  #   mappings:
  #   - match: 'envoy.cluster.*.upstream_cx_connect_ms'
  #     name: "envoy_cluster_upstream_cx_connect_time"
  #     timer_type: 'histogram'
  #     labels:
  #       cluster_name: "$1"

Expected behavior
Ambassador terminates TLS and routes to the mapping

Versions (please complete the following information):

  • Ambassador: 1.32
  • Kubernetes environment: Digitalocean
  • Version: 1.16

Additional context
Routing works when proxy-protocol is disabled on the Load balancer.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

I have the same issue, i couldn’t make ambassador work with proxy protocol enabled.

Submit an Answer