Service access from outside minikube

JONATHAN.T.PARKER · November 9, 2021, 2:12pm

In the process of running a few integration style tests, we have attempted to enable access to kafka which is deployed locally using skaffold (minikube running within a local docker container) from an application running on my host machine (a kafka producer or another application acting as a stimulator of the test). We have tried various combinations including port forwarding and changing from ClusterPort to NodePort. So far, no success. How should this be done?

Stephen.A.Gorman · November 9, 2021, 5:27pm

Sometimes I need to see what’s on a kafka topic and I hate using a console consumer. I found that if I edited my hosts file and port-forwarded kafka/zookeeper I was able to connect to kafka.

Host file:

127.0.0.1 kafka-0.kafka-headless.kafka.svc.cluster.local
127.0.0.1 kafka-zookeeper-0.kafka-zookeeper-headless.kafka.svc.cluster.local

Port Forwards:

kubectl.exe port-forward -n kafka kafka-zookeeper-0 2181:2181

kubectl.exe port-forward -n kafka kafka-0 9092:9092

This lets me configure a consumer (on my local OS, not in the cluster) to connect using localhost:9092.

joshua.l.ward · November 9, 2021, 5:48pm

We recently added support for mongodb and postgresql to be accessible from outside of a cluster (running in a container) and the solution involved creating virtual services for each, updating the Istio Gateway resource and updating the Istioctl command that is performed by leafctl. I imagine the same could be done for kafka if desired. Feel free to make a feature request to get this added to leafctl:

https://tasks.leidos-leaf.com/servicedesk/customer/portal/9

If you want to take a crack at it yourself, you can deploy istio similar to this (assuming you’re using istioctl… which if you’re using leafctl you are)

istioctl install -y \
    --namespace=istio-system \
    --set values.global.istioNamespace=istio-system \
    --set values.gateways.istio-ingressgateway.runAsRoot=true \
    --set profile=default \
    --set "components.ingressGateways[0].name=istio-ingressgateway" \
    --set "components.ingressGateways[0].k8s.service.externalIPs[0]=<MINIKUBE_IP>" \
    --set "components.ingressGateways[0].k8s.service.ports[0].name=status-port" \
    --set "components.ingressGateways[0].k8s.service.ports[0].port=15021" \
    --set "components.ingressGateways[0].k8s.service.ports[0].targetPort=15021" \
    --set "components.ingressGateways[0].k8s.service.ports[1].name=http2" \
    --set "components.ingressGateways[0].k8s.service.ports[1].port=80" \
    --set "components.ingressGateways[0].k8s.service.ports[1].targetPort=80" \
    --set "components.ingressGateways[0].k8s.service.ports[2].name=https" \
    --set "components.ingressGateways[0].k8s.service.ports[2].port=443" \
    --set "components.ingressGateways[0].k8s.service.ports[2].targetPort=443" \
    --set "components.ingressGateways[0].k8s.service.ports[3].name=mongodb" \
    --set "components.ingressGateways[0].k8s.service.ports[3].port=27017" \
    --set "components.ingressGateways[0].k8s.service.ports[3].targetPort=27017" \
    --set "components.ingressGateways[0].k8s.service.ports[4].name=postgresql" \
    --set "components.ingressGateways[0].k8s.service.ports[4].port=5432" \
    --set "components.ingressGateways[0].k8s.service.ports[4].targetPort=5432" \
    --set "components.ingressGateways[0].k8s.service.ports[5].name=kafka" \
    --set "components.ingressGateways[0].k8s.service.ports[5].port=9092" \
    --set "components.ingressGateways[0].k8s.service.ports[5].targetPort=9092"

Then make a corresponding Istio Gateway resource and kubectl apply it:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: istio-tls
      hosts:
        - "*"
    - port:
        number: 27017
        name: mongodb
        protocol: TCP
      hosts:
        - "*"
    - port:
        number: 5432
        name: postgresql
        protocol: TCP
      hosts:
        - "*"
    - port:
        number: 9092
        name: kafka
        protocol: TCP
      hosts:
        - "*"

Then make a corresponding Istio VirtualService resource for kafka and kubectl apply it:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kafka
  namespace: kafka
spec:
  gateways:
  - istio-system/istio-ingressgateway
  hosts:
  - '*'
  tcp:
  - match:
    - port: 9092
    route:
    - destination:
        host: kafka.kafka.svc.cluster.local
        port:
            number: 9092

Full disclosure, I haven’t had time to test the above, but it’s the approach I would probably take if looking to add this as a feature to leafctl. As with any leafctl command… this is only recommeneded for local development purposes. If everything works as intended… you should be able to access kafka at localhost:9092 without needing to modify your /etc/hosts

JONATHAN.T.PARKER · November 12, 2021, 1:57am

Well, I am definitely at the beginning of the learning curve with istio.

The istioctl install command responds with the following error:

zsh: no matches found: components.ingressGateways[0].name=istio-ingressgateway

joshua.l.ward · November 12, 2021, 1:49pm

~~ahh try applying the Gateway resource before running the istioctl command (the second code block in the above) my apologies for transposing the order~~
Please see latest reply, this was incorrect

JONATHAN.T.PARKER · November 12, 2021, 2:06pm

Yeah, I tried that last night all ready. The second two commands seemed to succeed (the gateway and virtual service commands). Note I was using dry-run, because I was not sure what was going wrong. Anyway, still the same error.

~/code/deployment-scripts  dev !1  minikube                                                                                                                                        ≡ 
 ❯  istioctl install --dry-run -y \
    --namespace=istio-system \
    --set values.global.istioNamespace=istio-system \
    --set values.gateways.istio-ingressgateway.runAsRoot=true \
    --set profile=default \
    --set components.ingressGateways[0].name=istio-ingressgateway \
    --set components.ingressGateways[0].k8s.service.externalIPs[0]=192.168.49.2 \
    --set components.ingressGateways[0].k8s.service.ports[0].name=status-port \
    --set components.ingressGateways[0].k8s.service.ports[0].port=15021 \
    --set components.ingressGateways[0].k8s.service.ports[0].targetPort=15021 \
    --set components.ingressGateways[0].k8s.service.ports[1].name=http2 \
    --set components.ingressGateways[0].k8s.service.ports[1].port=80 \
    --set components.ingressGateways[0].k8s.service.ports[1].targetPort=80 \
    --set components.ingressGateways[0].k8s.service.ports[2].name=https \
    --set components.ingressGateways[0].k8s.service.ports[2].port=443 \
    --set components.ingressGateways[0].k8s.service.ports[2].targetPort=443 \
    --set components.ingressGateways[0].k8s.service.ports[3].name=mongodb \
    --set components.ingressGateways[0].k8s.service.ports[3].port=27017 \
    --set components.ingressGateways[0].k8s.service.ports[3].targetPort=27017 \
    --set components.ingressGateways[0].k8s.service.ports[4].name=postgresql \
    --set components.ingressGateways[0].k8s.service.ports[4].port=5432 \
    --set components.ingressGateways[0].k8s.service.ports[4].targetPort=5432 \
    --set components.ingressGateways[0].k8s.service.ports[5].name=kafka \
    --set components.ingressGateways[0].k8s.service.ports[5].port=9090 \
    --set components.ingressGateways[0].k8s.service.ports[5].targetPort=9090
zsh: no matches found: components.ingressGateways[0].name=istio-ingressgateway
 ~/code/deployment-scripts  dev !1                                                                                                                                                  ≡ 
 ❯  ls
README.md  instioGatewayCreation.txt  istioSetupVirtualServices.txt  istioctl_InstallWithServiceAccess.txt
 ~/code/deployment-scripts  dev !1  minikube                                                                                                                                        ≡ 
 ❯  kubectl apply -f instioGatewayCreation.txt
gateway.networking.istio.io/istio-ingressgateway configured

 ~/code/deployment-scripts  dev !1  minikube                                                                                                                                        ≡ 
 ❯  istioctl install --dry-run -y \
    --namespace=istio-system \
    --set values.global.istioNamespace=istio-system \
    --set values.gateways.istio-ingressgateway.runAsRoot=true \
    --set profile=default \
    --set components.ingressGateways[0].name=istio-ingressgateway \
    --set components.ingressGateways[0].k8s.service.externalIPs[0]=192.168.49.2 \
    --set components.ingressGateways[0].k8s.service.ports[0].name=status-port \
    --set components.ingressGateways[0].k8s.service.ports[0].port=15021 \
    --set components.ingressGateways[0].k8s.service.ports[0].targetPort=15021 \
    --set components.ingressGateways[0].k8s.service.ports[1].name=http2 \
    --set components.ingressGateways[0].k8s.service.ports[1].port=80 \
    --set components.ingressGateways[0].k8s.service.ports[1].targetPort=80 \
    --set components.ingressGateways[0].k8s.service.ports[2].name=https \
    --set components.ingressGateways[0].k8s.service.ports[2].port=443 \
    --set components.ingressGateways[0].k8s.service.ports[2].targetPort=443 \
    --set components.ingressGateways[0].k8s.service.ports[3].name=mongodb \
    --set components.ingressGateways[0].k8s.service.ports[3].port=27017 \
    --set components.ingressGateways[0].k8s.service.ports[3].targetPort=27017 \
    --set components.ingressGateways[0].k8s.service.ports[4].name=postgresql \
    --set components.ingressGateways[0].k8s.service.ports[4].port=5432 \
    --set components.ingressGateways[0].k8s.service.ports[4].targetPort=5432 \
    --set components.ingressGateways[0].k8s.service.ports[5].name=kafka \
    --set components.ingressGateways[0].k8s.service.ports[5].port=9090 \
    --set components.ingressGateways[0].k8s.service.ports[5].targetPort=9090
zsh: no matches found: components.ingressGateways[0].name=istio-ingressgateway
 ~/code/deployment-scripts  dev !1                                                                                                                                                  ≡ 
 ❯  ls
README.md  instioGatewayCreation.txt  istioSetupVirtualServices.txt  istioctl_InstallWithServiceAccess.txt
 ~/code/deployment-scripts  dev !1  minikube                                                                                                                                        ≡ 
 ❯  kubectl apply -f istioSetupVirtualServices.txt 
virtualservice.networking.istio.io/kafka created

 ❯  istioctl install --dry-run -y \
    --namespace=istio-system \
    --set values.global.istioNamespace=istio-system \
    --set values.gateways.istio-ingressgateway.runAsRoot=true \
    --set profile=default \
    --set components.ingressGateways[0].name=istio-ingressgateway \
    --set components.ingressGateways[0].k8s.service.externalIPs[0]=192.168.49.2 \
    --set components.ingressGateways[0].k8s.service.ports[0].name=status-port \
    --set components.ingressGateways[0].k8s.service.ports[0].port=15021 \
    --set components.ingressGateways[0].k8s.service.ports[0].targetPort=15021 \
    --set components.ingressGateways[0].k8s.service.ports[1].name=http2 \
    --set components.ingressGateways[0].k8s.service.ports[1].port=80 \
    --set components.ingressGateways[0].k8s.service.ports[1].targetPort=80 \
    --set components.ingressGateways[0].k8s.service.ports[2].name=https \
    --set components.ingressGateways[0].k8s.service.ports[2].port=443 \
    --set components.ingressGateways[0].k8s.service.ports[2].targetPort=443 \
    --set components.ingressGateways[0].k8s.service.ports[3].name=mongodb \
    --set components.ingressGateways[0].k8s.service.ports[3].port=27017 \
    --set components.ingressGateways[0].k8s.service.ports[3].targetPort=27017 \
    --set components.ingressGateways[0].k8s.service.ports[4].name=postgresql \
    --set components.ingressGateways[0].k8s.service.ports[4].port=5432 \
    --set components.ingressGateways[0].k8s.service.ports[4].targetPort=5432 \
    --set components.ingressGateways[0].k8s.service.ports[5].name=kafka \
    --set components.ingressGateways[0].k8s.service.ports[5].port=9090 \
    --set components.ingressGateways[0].k8s.service.ports[5].targetPort=9090
zsh: no matches found: components.ingressGateways[0].name=istio-ingressgateway

joshua.l.ward · November 12, 2021, 3:42pm

I gave this a quick test and my apologies… the order is actually correct. The issue is that we’re trying to run that istioctl command outside of the context of a script directly in your shell and I believe it doesn’t know how to handle the array index references. I’ve wrapped the istioctl flags that reference indices in quotes and this appears to work (just replace minikube ip):

istioctl install -y \
    --namespace=istio-system \
    --set values.global.istioNamespace=istio-system \
    --set values.gateways.istio-ingressgateway.runAsRoot=true \
    --set profile=default \
    --set "components.ingressGateways[0].name=istio-ingressgateway" \
    --set "components.ingressGateways[0].k8s.service.externalIPs[0]=<MINIKUBE_IP>" \
    --set "components.ingressGateways[0].k8s.service.ports[0].name=status-port" \
    --set "components.ingressGateways[0].k8s.service.ports[0].port=15021" \
    --set "components.ingressGateways[0].k8s.service.ports[0].targetPort=15021" \
    --set "components.ingressGateways[0].k8s.service.ports[1].name=http2" \
    --set "components.ingressGateways[0].k8s.service.ports[1].port=80" \
    --set "components.ingressGateways[0].k8s.service.ports[1].targetPort=80" \
    --set "components.ingressGateways[0].k8s.service.ports[2].name=https" \
    --set "components.ingressGateways[0].k8s.service.ports[2].port=443" \
    --set "components.ingressGateways[0].k8s.service.ports[2].targetPort=443" \
    --set "components.ingressGateways[0].k8s.service.ports[3].name=mongodb" \
    --set "components.ingressGateways[0].k8s.service.ports[3].port=27017" \
    --set "components.ingressGateways[0].k8s.service.ports[3].targetPort=27017" \
    --set "components.ingressGateways[0].k8s.service.ports[4].name=postgresql" \
    --set "components.ingressGateways[0].k8s.service.ports[4].port=5432" \
    --set "components.ingressGateways[0].k8s.service.ports[4].targetPort=5432" \
    --set "components.ingressGateways[0].k8s.service.ports[5].name=kafka" \
    --set "components.ingressGateways[0].k8s.service.ports[5].port=9092" \
    --set "components.ingressGateways[0].k8s.service.ports[5].targetPort=9092"

joshua.l.ward · November 12, 2021, 4:04pm

I have made a few edits as I was referencing the wrong port (was using 9090 in my examples instead of the correct 9092). I still don’t have a convenient means of testing this on hand, but you’re still welcome to see if it works for you.

EDIT: So I finally had a little time to give this approach a proper test… and unfortunately it doesn’t appear to be working as I would have believed it would (this is the approach we’ve used to make mongo and postgres available from outside of the cluster… I assumed it would be pretty cut and dry to do the same for kafka, but looks like I was mistaken) I did give @Stephen.A.Gorman’s solution a test drive and it worked well, kudos to him Until we are able to investigate making this a possibility without modifying the etc/hosts file, I would suggest just using Stephen’s suggestion. Apologies for any and all misinformation here.