Preserve the source IP in Istio with NGINX as the reverse proxy

This is useful for situations where you want to whitelist/blacklist certain IP addresses with the Istio authorization policy.

By default, when using a reverse proxy, the X-Forwarded-For header is lost when the request passes through the proxy. This way, Istio will recognize the source IP as the IP of the pod where the request was meant to end. This is problematic if you want to set authorization policies and you are not controlling this in the application itself.

This will also configure NGINX in a TCP mode which will allow SSL Passthrough, which means that the SSL termination will happen at the Istio Ingress Gateway level inside the cluster.

The solution

The solution for this is to enable the proxy protocol on both NGINX and Istio. As NGINX explains it, the proxy protocol is designed to chain proxies or reverse proxies without losing the client information. This is what we need to solve it.

The NGINX configuration

The NGINX configuration is done for both :80 and :443 ports. You can replace this if your configuration is done in another way. Also, replace the IP addresses with your own.

First, add this snippet to /etc/nginx/nginx.conf:

1
2
3

stream {
  include /etc/nginx/streams/*.conf;
}

For the :80 port, we will not create a stream, but a simple http server configuration. You can change it to a stream if you want, I did this because I didn’t want the :80 port to be configured as a stream. Create a new file istio.conf inside /etc/nginx/sites-enabled and add the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80 default_server proxy_protocol;
    listen [::]:80 default_server;

    server_name _;

    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://192.168.0.50;
    }
}

Then, create the streams directory:

1

sudo mkdir /etc/nginx/streams

Create the file istio.conf inside /etc/nginx/streams to configure the :443 port and add the following to it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

upstream https {
    server 192.168.0.50:443;
}

map $ssl_preread_protocol $upstream {
    "TLSv1.2" https;
    "TLSv1.3" https;
}

server {
    listen 443;

    proxy_pass $upstream;
    ssl_preread on;
    proxy_protocol on;
}

As you can see from the configuration above, we are setting the proxy_protocol NGINX module to on for both :80 and :443, which allows for the client information, including the X-Forwarded-For header to pass through the reverse proxy. Now you have to enable it on Istio level as well so it can receive this information.

The Istio configuration

Setting the number of trusted proxies

First and foremost, you need to add a mesh configuration which lets Istio and Envoy itself know the number of proxies it can trust. We will set the number to 2. Create a new file named topology.yaml and paste this inside:

1
2
3
4
5
6
7

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        numTrustedProxies: 2

Save the file, and if you haven’t installed Istio yet, run the following:

1

istioctl install -f topology.yaml -y

If you already have Istio installed in your cluster, run the following:

1

istioctl upgrade -f topology.yaml -y

You can check if the setting is applied if the meshConfig is present in the spec field of the installed-state istio operator. You can get it by doing:

1

kubectl get istiooperators.install.istio.io -n istio-system installed-state -o yaml

Setting up the proxy protocol

Istio itself doesn’t support the proxy protocol, but we can enable an Envoy Proxy filter which is configured to inspect TLS and allow the client headers to pass through the proxy with the proxy_protocol and tls_inspector filters.

After the number of trusted proxies is set up, create a new file named envoy-proxy-filters.yaml and paste the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.filters.listener.proxy_protocol
        - name: envoy.filters.listener.tls_inspector
  workloadSelector:
    labels:
      istio: ingressgateway

This will enable the proxy protocol to listen to incoming information, which will come from NGINX. Apply this manifest:

1

kubectl apply -f envoy-proxy-filters.yaml

After you have applied it, restart the istio-ingressgateway pod in the istio-system namespace:

1

kubectl delete pod -n istio-system -l app=ingressgateway

Testing the setup

Now it’s time to test if the proxy protocol is working. To do so, you can enable the debug mode of the Envoy proxy logs inside the istio-ingressgateway pod:

1

kubectl get pods -n istio-system -o name -l istio=ingressgateway | sed 's|pod/||' | while read -r pod; do istioctl proxy-config log "$pod" -n istio-system --level rbac:debug; done

Now follow the logs:

1

kubectl logs -f -n istio-system -l app=ingressgateway

Send a request to any application which is routed through the Istio Ingress Gateway:

1

curl -v https://example.com

And finally, check the logs of the istio-ingressgateway pod to see that the X-Forwarded-For header contains your public IP. That means that the setup is complete.