By juandow
Describe the bug When proxy protocol is enabled on my L4 load balancer, ambassador does not terminate TLS/map to the service.
To Reproduce Use the config below:
apiVersion: getambassador.io/v2
kind: Module
metadata:
name: ambassador-users
namespace: xxxx-user
spec:
ambassador_id: ambassador-user
config:
use_remote_address: false
use_proxy_proto: true
---
apiVersion: getambassador.io/v2
kind: TLSContext
metadata:
name: global-tls
namespace: xxxx-user
spec:
ambassador_id: ambassador-user
hosts:
- xxxx.com
- "*.xxxx.com"
redirect_cleartext_from: 8080
secret: xxxx-tls
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
name: frontend
namespace: xxxx-user
spec:
ambassador_id: ambassador-user
host: "(\\w+).xxxx.com"
host_regex: true
prefix: /
service: frontend.production:80
Ambassador is installed from the helm chart with the following config
---
# Default values for ambassador.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Manually set metadata for the Release.
#
# Defaults to .Chart.Name
nameOverride: ""
# Defaults to .Release.Name-.Chart.Name unless .Release.Name contains "ambassador"
fullnameOverride: ""
replicaCount: 3
daemonSet: false
# Enable autoscaling using HorizontalPodAutoscaler
# daemonSet: true, autoscaling will be disabled
autoscaling:
enabled: false
minReplicas: 2
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 60
podDisruptionBudget: {}
# namespace:
# name: default
# Additional container environment variable
# Uncomment or add additional environment variables for the container here.
env:
AMBASSADOR_ID: ambassador-user
# Exposing statistics via StatsD
# STATSD_ENABLED: true
# STATSD_HOST: statsd-sink
# sets the minimum number of seconds between Envoy restarts
# AMBASSADOR_RESTART_TIME: 15
# sets the number of seconds that the Envoy will wait for open connections to drain on a restart
# AMBASSADOR_DRAIN_TIME: 5
# sets the number of seconds that Ambassador will wait for the old Envoy to clean up and exit on a restart
# AMBASSADOR_SHUTDOWN_TIME: 10
# labels Ambassador with an ID to allow for configuring multiple Ambassadors in a cluster
imagePullSecrets: []
securityContext:
runAsUser: 8888
image:
repository: quay.io/datawire/aes
tag: 1.3.2
pullPolicy: IfNotPresent
dnsPolicy: "ClusterFirst"
hostNetwork: false
service:
type: LoadBalancer
# Note that target http ports need to match your ambassador configurations service_port
# https://www.getambassador.io/reference/modules/#the-ambassador-module
ports:
- name: http
port: 80
targetPort: 8080
# protocol: TCP
# nodePort: 30080
- name: https
port: 443
targetPort: 8443
# protocol: TCP
# nodePort: 30443
# TCPMapping_Port
# port: 2222
# targetPort: 2222
# protocol: TCP
# nodePort: 30222
externalTrafficPolicy:
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "false"
service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
#############################################################################
## Ambassador should be configured using CRD definition. If you want
## to use annotations, the following is an example of annotating the
## Ambassador service with global configuration manifest.
##
## See https://www.getambassador.io/reference/core/ambassador and
## https://www.getambassador.io/reference/core/tls for more info
#############################################################################
#
# getambassador.io/config: |
# ---
# apiVersion: ambassador/v1
# kind: TLSContext
# name: ambassador
# secret: ambassador-certs
# hosts: ["*"]
# ---
# apiVersion: ambassador/v1
# kind: Module
# name: ambassador
# config:
# admin_port: 8001
# diag_port: 8877
# diagnostics:
# enabled: true
# enable_grpc_http11_bridge: false
# enable_grpc_web: false
# enable_http10: false
# enable_ipv4: true
# enable_ipv6: false
# liveness_probe:
# enabled: true
# lua_scripts:
# readiness_probe:
# enabled: true
# server_name: envoy
# service_port: 8080
# use_proxy_proto: false
# use_remote_address: true
# xff_num_trusted_hops: 0
# x_forwarded_proto_redirect: false
# load_balancer:
# policy: round_robin
# circuit_breakers:
# max_connections: 2048
# retry_policy:
# retry_on: "5xx"
# cors:
adminService:
create: true
type: ClusterIP
port: 8877
# NodePort used if type is NodePort
# nodePort: 38877
annotations:
{}
rbac:
# Specifies whether RBAC resources should be created
create: true
podSecurityPolicies: []
# Name of the RBAC resources defaults to the name of the release.
# Set nameOverride when installing Ambassador with cluster-wide scope in
# different namespaces with the same release name to avoid conflicts.
nameOverride:
scope:
# tells Ambassador to only use resources in the namespace or namespace set by namespace.name
singleNamespace: false
serviceAccount:
# Specifies whether a service account should be created
create: true
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name:
deploymentStrategy:
type: RollingUpdate
restartPolicy:
initContainers: []
sidecarContainers: []
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 3
failureThreshold: 3
readinessProbe:
initialDelaySeconds: 30
periodSeconds: 3
failureThreshold: 3
volumes: []
volumeMounts: []
podLabels:
{}
podAnnotations:
{}
# prometheus.io/scrape: "true"
# prometheus.io/port: "9102"
deploymentAnnotations:
{}
# configmap.reloader.stakater.com/auto: "true"
resources:
{}
# If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
priorityClassName: ""
nodeSelector: {}
tolerations: []
affinity: {}
ambassadorConfig: ""
crds:
enabled: true
create: false
keep: true
# The Ambassador Edge Stack is free for limited use without a license key.
# Go to https://{ambassador-host}/edge_stack/admin/#dashboard to register
# for a community license key.
enableAES: true
# Set createSecret: false is installing multiple releases of The Ambassador
# Edge Stack in the same namespace.
licenseKey:
value:
createSecret: true
secretName:
# The DevPortal is exposed at /docs/ endpoint in the AES container.
# Setting this to true will automatically create routes for the DevPortal.
createDevPortalMappings: true
# The Ambassador Edge Stack uses a redis instance for managing authentication,
# rate limiting, and sharing minor configuration details between pods for
# centralized management. These values configure the redis instance that ships
# by default with The Ambassador Edge Stack.
#
# URL of your redis instance. Defaults to redis instance created below.
redisURL:
# Ambassador ships with a basic redis instance. Configure the deployment with the options below.
redis:
create: true
# Annotations for Ambassador Pro's redis instance.
annotations:
deployment:
{}
service:
{}
resources: {}
# If you want to specify resources, uncomment the following
# lines and remove the curly braces after 'resources:'.
# These are placeholder values and must be tuned.
# limits:
# cpu: 100m
# memory: 256Mi
# requests:
# cpu: 50m
# memory: 128Mi
nodeSelector: {}
# Configures the AuthService that ships with the Ambassador Edge Stack.
# Setting authService.create: false will not install the AES AuthService and
# allow you to define your own.
#
# Typically when using the AES, you will want to keep this set to true and use
# the External Filter to communicate with a custom authentication service.
# https://www.getambassador.io/reference/filter-reference/#filter-type-external
authService:
create: false
# Set additional configuration options. See https://www.getambassador.io/reference/services/auth-service for more information
optional_configurations:
# include_body:
# max_bytes: 4096
# allow_partial: true
# status_on_error:
# code: 403
# failure_mode_allow: false
# retry_policy:
# retry_on: "5xx"
# num_retries: 2
# add_linkerd_headers: true
# timeout_ms: 30000
# Configures the RateLimitService in the Ambassador Edge Stack.
# Keep this enabled to configure RateLimits in AES.
rateLimit:
create: false
# DEPRECATED: Ambassador now exposes the /metrics endpoint in Envoy.
# DEPRECATED: See https://www.getambassador.io/user-guide/monitoring#deployment for more information on how to use the /metrics endpoint
#
# DEPRECATED: Enabling the prometheus exporter creates a sidecar and configures ambassador to use it
prometheusExporter:
enabled: false
repository: prom/statsd-exporter
tag: v0.8.1
pullPolicy: IfNotPresent
resources: {}
# If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 256Mi
# requests:
# cpu: 50m
# memory: 128Mi
# You can configure the statsd exporter to modify the behavior of mappings and other features.
# See documentation: https://github.com/prometheus/statsd_exporter/tree/v0.8.1#metric-mapping-and-configuration
# Uncomment the following line if you wish to specify a custom configuration:
# configuration: |
# ---
# mappings:
# - match: 'envoy.cluster.*.upstream_cx_connect_ms'
# name: "envoy_cluster_upstream_cx_connect_time"
# timer_type: 'histogram'
# labels:
# cluster_name: "$1"
Expected behavior Ambassador terminates TLS and routes to the mapping
Versions (please complete the following information):
Additional context Routing works when proxy-protocol is disabled on the Load balancer.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
I’m having the same issue. When Proxy Protocol is disabled, everything is fine. I tried many different configs with ambassador, i couldn’t make it work. Example log from ambassador pod when enabled: ACCESS [2020-06-04T23:53:24.145Z] “- - HTTP/1.1” 0 DPE 0 0 0 - “-” “-” “-” “-” “-”
I have the same issue, i couldn’t make ambassador work with proxy protocol enabled.
I have the same problem…
This is still not working for me in 1.9.1. All http:// requests get a 404, while all https:// ones work.
Mapping:
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
name: wetsnow.com
spec:
prefix: /
host: www.wetsnow.com
resolver: endpoint
service: nginx.web:80
Host:
apiVersion: getambassador.io/v2
kind: TLSContext
metadata:
name: www.wetsnow.com-context # for the host above
spec:
hosts:
- "www.wetsnow.com"
secret: www.wetsnow.com
alpn_protocols: h2,http/1.1 # required for C# grpc client
curl request:
% curl -v -I http://www.wetsnow.com
* Trying 167.99.20.238...
* TCP_NODELAY set
* Connected to www.wetsnow.com (167.99.20.238) port 80 (#0)
> HEAD / HTTP/1.1
> Host: www.wetsnow.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< content-security-policy: frame-ancestors 'none'
content-security-policy: frame-ancestors 'none'
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< x-frame-options: deny
x-frame-options: deny
< date: Tue, 29 Dec 2020 04:06:40 GMT
date: Tue, 29 Dec 2020 04:06:40 GMT
< x-envoy-upstream-service-time: 0
x-envoy-upstream-service-time: 0
< x-content-type-options: nosniff
x-content-type-options: nosniff
< server: envoy
server: envoy
< transfer-encoding: chunked
transfer-encoding: chunked
Ambassador log line:
ACCESS [2020-12-29T04:06:40.893Z] "HEAD / HTTP/1.1" 404 - 0 0 1 0 "96.239.115.74" "curl/7.64.1" "f11b5605-de60-4cbf-b2da-1fe425c04b81" "www.wetsnow.com" "127.0.0.1:8500"
Match from diag screen:
{
"match": {
"case_sensitive": true,
"headers": [
{
"exact_match": "www.wetsnow.com",
"name": ":authority"
}
],
"prefix": "/",
"runtime_fraction": {
"default_value": {
"denominator": "HUNDRED",
"numerator": 100
},
"runtime_key": "routing.traffic_shift.cluster_nginx_web_80_default"
}
},
"route": {
"cluster": "cluster_nginx_web_80_default",
"prefix_rewrite": "/",
"priority": null,
"timeout": "3.000s"
}
}
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.