Preserve the source IP in Istio with HAProxy as the reverse proxy

This is useful for situations where you want to whitelist/blacklist certain IP addresses with the Istio authorization policy.

By default, when using a reverse proxy, the X-Forwarded-For header is lost when the request passes through the proxy. This way, Istio will recognize the source IP as the IP of the pod where the request was meant to end. This is problematic if you want to set authorization policies and you are not controlling this in the application itself.

This will also configure HAProxy in a TCP mode which will allow SSL Passthrough, which means that the SSL termination will happen at the Istio Ingress Gateway level inside the cluster.

The solution

The solution for this is to enable the proxy protocol on both HAProxy and Istio. As HAProxy explains it, the proxy protocol is designed to chain proxies or reverse proxies without losing the client information. This is what we need to solve it.

The HAProxy configuration

The HAProxy configuration is done for both :80 and :443 ports. You can replace this if your configuration is done in another way. Also, replace the IP addresses with your own. Here is the configuration, and you will place it in /etc/haproxy/haproxy.cfg on your host machine:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

global
    log /dev/log  local0 warning
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

   stats socket /var/lib/haproxy/stats

defaults
  log global
  option  httplog
  option  dontlognull
        timeout connect 5000
        timeout client 50000
        timeout server 50000

frontend istio_http
    bind *:80
    option tcplog
    mode tcp
    default_backend istio_http

backend istio_http
    mode tcp
    balance roundrobin
    server app_http 192.168.0.50:80 check send-proxy-v2

frontend istio_https
    bind *:443
    option tcplog
    mode tcp
    default_backend istio_https

backend istio_https
    mode tcp
    balance roundrobin
    server app_https 192.168.0.50:443 check send-proxy-v2

Now restart the HAProxy service:

1

sudo systemctl restart haproxy

As you can see from the configuration above, we are setting the frontend and backend configurations for both http and https, and we are using the send-proxy-v2 HAProxy module which allows for the client information, including the X-Forwarded-For header to pass through the reverse proxy. Now you have to enable it on Istio level as well so it can receive this information.

The Istio configuration

Setting the number of trusted proxies

First and foremost, you need to add a mesh configuration which lets Istio and Envoy itself know the number of proxies it can trust. We will set the number to 2. Create a new file named topology.yaml and paste this inside:

1
2
3
4
5
6
7

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        numTrustedProxies: 2

Save the file, and if you haven’t installed Istio yet, run the following:

1

istioctl install -f topology.yaml -y

If you already have Istio installed in your cluster, run the following:

1

istioctl upgrade -f topology.yaml -y

You can check if the setting is applied if the meshConfig is present in the spec field of the installed-state istio operator. You can get it by doing:

1

kubectl get istiooperators.install.istio.io -n istio-system installed-state -o yaml

Setting up the proxy protocol

Istio itself doesn’t support the proxy protocol, but we can enable an Envoy Proxy filter which is configured to inspect TLS and allow the client headers to pass through the proxy with the proxy_protocol and tls_inspector filters.

After the number of trusted proxies is set up, create a new file named envoy-proxy-filters.yaml and paste the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.filters.listener.proxy_protocol
        - name: envoy.filters.listener.tls_inspector
  workloadSelector:
    labels:
      istio: ingressgateway

This will enable the proxy protocol to listen to incoming information, which will come from HAProxy. Apply this manifest:

1

kubectl apply -f envoy-proxy-filters.yaml

After you have applied it, restart the istio-ingressgateway pod in the istio-system namespace:

1

kubectl delete pod -n istio-system -l app=ingressgateway

Testing the setup

Now it’s time to test if the proxy protocol is working. To do so, you can enable the debug mode of the Envoy proxy logs inside the istio-ingressgateway pod:

1

kubectl get pods -n istio-system -o name -l istio=ingressgateway | sed 's|pod/||' | while read -r pod; do istioctl proxy-config log "$pod" -n istio-system --level rbac:debug; done

Now follow the logs:

1

kubectl logs -f -n istio-system -l app=ingressgateway

Send a request to any application which is routed through the Istio Ingress Gateway:

1

curl -v https://example.com

And finally, check the logs of the istio-ingressgateway pod to see that the X-Forwarded-For header contains your public IP. That means that the setup is complete.