Istio 101 (0.8.0) on GKE

In one of my previous posts, I showed how to install Istio on minikube and deploy the sample BookInfo app. A new Istio version is out (0.8.0) with a lot of changes, especially changes on traffic management, which made my steps in the previous post a little obsolete.

In this post, I want to show how to install Istio 0.8.0 on Google Kubernetes Engine (GKE), deploy the sample BookInfo app and show some of the add-ons and traffic routing.

Create Kubernetes cluster

First, we need a Kubernetes cluster to install Istio. On GKE, this is a single command:

gcloud container clusters create hello-istio \
 --cluster-version=1.9.7-gke.1 \
 --zone europe-west1-b \
 --num-nodes 4

A couple of things to note. First, I’m not using the latest cluster-version on GKE (1.10) because Istio 0.8.0 does not work against it (yet). Second, I’m using 4 worker nodes. That’s the recommended number of nodes for BookInfo sample.

Once the cluster is created, we also need to create a clusterrolebinding for Istio to be able to manage the cluster:

kubectl create clusterrolebinding cluster-admin-binding \
 --clusterrole=cluster-admin \
 --user=$(gcloud config get-value core/account)

Download & Setup Istio

Now that we have a cluster, let’s download the latest Istio (0.8.0 as of today):

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=0.8.0 sh -

Add Istio’s command line tool istioctl to your PATH. We’ll need it later:

export PATH="$PATH:./istio-0.8.0/bin"

Install Istio

It’s time to install Istio with mutual authentication between sidecars:

kubectl apply -f install/kubernetes/istio-demo-auth.yaml

Once it’s done, you can check that pods are running under istio-system namespace:

kubectl get pods -n istio-system

You’ll realize that in addition to Istio base components (eg. pilot, mixer, ingress, egress), a number of add-ons are also installed (eg. prometheus, servicegraph, grafana). This is different from the previous versions of Istio.

Deploy BookInfo app

Let’s deploy the BookInfo sample app now:

kubectl apply -f <(istioctl kube-inject --debug -f samples/bookinfo/kube/bookinfo.yaml)

And make sure all the pods are running:

kubectl get pods

Deploy BookInfo Gateway

In Istio 0.8.0, traffic management completely changed and one of those changes is that you need to create a gateway for ingress traffic. Let’s go ahead and create a gateway for BookInfo app:

istioctl create -f samples/bookinfo/routing/bookinfo-gateway.yaml

Use BookInfo app

We can finally take a look at the app. We need to find ingress gateway IP and port:

kubectl get svc istio-ingressgateway -n istio-system

To make it easier for us, let’s define a GATEWAY_URL variable:

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

Let’s see if the app is working. You should get 200 with curl:

curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage

You can also open a browser and see the web frontend for product page. At this point, we got the app deployed and managed by a basic installation of Istio.

Next, we’ll take a look at some of the add-ons. Unlike previous versions, add-ons are automatically installed already. Let’s start sending some traffic first:

for i in {1..100}; do curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage; done

Grafana dashboard

There’s Grafana for dashboarding. Let’s setup port forwarding first:

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 8080:3000

Navigate to http://localhost:8080 to see the dashboard:

Istio Dashboard in Grafana

Prometheus metrics

Next, let’s take a look at Prometheus for metrics. Set port forwarding:

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 8083:9090

Navigate to http://localhost:8083/graph to see Prometheus:

Prometheus in Istio

ServiceGraph

For dependency visualization, we can take a look at ServiceGraph:

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=servicegraph -o jsonpath='{.items[0].metadata.name}') 8082:8088

Navigate to http://localhost:8082/dotviz:

Screen Shot 2018-06-07 at 10.02.38 AM.png

Tracing

For HTTP tracing, there is Jaegar and Zipkin. Let’s take a look at Jaeger. Setup port forwarding as usual:

kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 8084:16686

Navigate to http://localhost:8084

Screen Shot 2018-06-07 at 10.05.11 AM

Traffic Management

Traffic Management changed dramatically in 0.8.0. You can read more about it here but basically instead of routing rules, we now have VirtualServices and DestinationRules.

You can see the existing VirtualServices and DestinationRules like this:

istioctl get virtualservices -o yaml
istioctl get destinationrules -o yaml

When you go to the product page of BookInfo application and do a browser refresh a few times, you will see that the reviews section on the right keeps changing (the stars change color). This is because there are 3 different reviews microservices and everytime, a different microservice is invoked. Let’s pin all microservices to version1:

istioctl create -f samples/bookinfo/routing/route-rule-all-v1-mtls.yaml

This creates VirtualServices and DestinationRules needed to pin all microservices to version1. Now, if you back to the product page and do a browser refresh, nothing changes because reviews microservice is pinned to version1 now.

To pin a specific user (eg. Jason) to a specific version (v2), we can do the following:

istioctl replace -f samples/bookinfo/routing/route-rule-reviews-test-v2.yaml

With this rule, if you login to the product page with username “Jason”, you should see the v2 version of reviews microservice.

To clean up all destination rules, run the following and now we’re back to the beginning with 3 different versions of the microservices:

istioctl delete -f samples/bookinfo/routing/route-rule-all-v1.yaml

Cleanup

This wraps up all the basic functionality of Istio 0.8.0 that I wanted to show on GKE. To cleanup, let’s first delete the BookInfo app:

samples/bookinfo/kube/cleanup.sh

Confirm that BookInfo app is gone:

istioctl get gateway
istioctl get virtualservices
kubectl get pods

Finally, cleanup Istio:

kubectl delete -f install/kubernetes/istio-demo.yaml

Confirm that Istio is gone:

kubectl get pods -n istio-system
Advertisements

Codemotion in Amsterdam, Devoxx in London

After my trip in Istanbul, I visited my parents in Nicosia, Cyprus for a long weekend. Then, I stopped by in Amsterdam for Codemotion before coming back to London for Devoxx. 4 cities in 4 countries in 1 week was exhausting but also a lot of fun in many ways.

Codemotion Amsterdam

Amsterdam is almost a second home to me nowadays. There’s a great tech scene and a lot of tech events throughout the year, as a result, I end up visiting Amsterdam at least 2-3 times a year.

Codemotion is a European tech conference that happens in many locations. As you might remember, I spoke at Codemotion Rome earlier this year (trip report). This was my second time speaking at Codemotion Amsterdam. Last year, I spoke about gRPC and this year about Istio, both open source projects .

Codemotion Amsterdam is a mid-size conference, my guess is about 1000/1500 developers. I love the venue of Codemotion Amsterdam. It’s in an old factory kind of place, right next to the river. They did a great job with the venue decoration, lighting both last year and this year as well.

Talk & Questions

I did my usual Istio 101 talk to a group of about 100 developers. After my talk, I got the following questions:

  • How does Istio compare to Conduit? (apparently, Conduit is an Istio like project but I didn’t know much about it).
  • How can we have sticky sessions with Istio? (i.e. make sure certain users always go to the same pod).
  • Is it possible to have a message queue between services? This is a common pattern in microservices and a couple of people were wondering if this is possible in Istio.

Devoxx London

After Amsterdam, I arrived back to London for Devoxx UK. Even though I’m based in London, I don’t get to speak as much as I’d like in London, mainly due to my travels, so I was happy to be part of Devoxx UK.

Devoxx is another European conference that happens in places like Brussels, Krakow, Casablanca and London. It started as a Java conference but nowadays, it’s much more than just Java. I got to speak at Devoxx Brussels, Krakow and Casablanca in previous years but this was my first time speaking in Devoxx London.

As a side note, Devoxx Brussels is one of the best tech conferences I ever attended with great technical content, huge cinema like screens for previous and awesome attendees and speakers. In comparison, London is smaller but still a nice conference.

Talk & Questions

In Devoxx London, I did my Istio 101 talk again. It’s great to see so much interest in Istio from the community. The talk of the video is already online, so you can watch it here if you like:

After the talk, I got the following questions:

  • Kafka or some message queue between services: Again, people are curious about how to have an async architecture with message queues between services.
  • Zipkin add-on: Where does it save its data? If Istio is restarted, does the data persist?
  • Zipkin: Can we have it to look at our custom headers for tracing?
  • Pilot stability: What happens if Pilot does? Does the service mesh still work? Does Pilot’s state persist somewhere?

Istanbul: The city where the East and the West meet

Istanbul is one of those crazy dynamic cities with friendly people, amazing history, great shopping and above all, a food heaven. Naturally, I was excited to be back for Java Day Istanbul conference.

I came a day early to meet with a partner and visit a customer. They had lots of questions on Kubernetes and hybrid-cloud. It was quite useful for me to hear about their challenges about moving to the cloud and propose some solutions.

The biggest benefit of conferences for me is meeting or reconnecting with people. This time, I got to meet my friends Edson and Rustam. We went to the famous steak house called Nusret where the owner is a social media sensation.

Java Day

Java Day is a community driven conference that happens all over Europe. I spoke in Java Day Istanbul for the first time last year about Kubernetes. This year, it was at a different venue and seemed a little bigger than last year with about 800 developers (my guess). I got to speak about Istio in the main room to a group of about 100 developers.

They also have my video talk if you want to watch.

After my talk, I had to leave immediately to catch my flight, as I went to visit my parents for a long weekend.

Questions

The questions I received about Istio were mostly about the performance overhead of Envoy and also the production readiness of Istio. I’m sure I received some other questions that I cannot think of anymore but overall, no unique question stood out.

 

 

 

First time in China

Visiting a new country is always extra special for me and a couple of weeks ago, I got to do just that and visited Beijing, China for the first time.

After TDC in Floripa, I flew to Beijing for QCon conference and for a Google Developer Group (GDG) meetup. It was the first time for me in China (my 48th country!) and first time speaking at QCon, so I was naturally excited. I didn’t know what to expect in Beijing but I was pleasantly surprised. Beijing is a modern city with good public transportation and interesting sites to visit. I got the feeling that it’s also very international with many expats, definitely more than I expected. I visited the iconic sites like Tiananmen Square and the Great Wall of China. I also visited Google’s Beijing office (my 35th Google office visit!)

QCon Beijing

QCon is a global conference that happens in multiple locations like New York, London, San Francisco, Sao Paulo, Beijing, Shanghai. QCon Beijing was a multi-day multi-track conference with more than 1000+ attendees on a diverse set of topics. I could only attend the last day of the conference on Sunday.

I did my Istio 101 talk to a full room of probably 200 people. I was surprised by the number of attendees and the level of questions I received after my talk. It seems like many people in Beijing are active users of the cloud related technologies like Kubernetes and they were very curious about Istio.

Questions

After my talk, I received the usual Istio questions such as:

  • When will Istio be production ready?
  • What’s Istio roadmap, what features are planned in the upcoming releases?
  • What’s the performance overhead of Envoy proxy?
  • What are the performance characteristics of Istio?

I also received a couple unique questions such as:

  • How do you talk from Istio to an external database?
  • What’s the state of CloudFoundry with Istio?
  • How to upgrade Istio? What’s the recommended procedure?

GDG Beijing

A couple of days after QCon Beijing, I did the same Istio talk at Google Developer Group in Beijing. It was at a co-working space and organizers did a great job in putting the event together with banners, videos before the event and a great Q&A session after my talk.

Questions

After the talk, we had an extensive Q&A session. There were the usual Istio questions on roadmap, production readiness and envoy overhead. More interesting questions I remember were:

  • How long lived connections are handled in Istio?
  • How to delete route rules without yaml?
  • How does Istio see user agent headers when encrypted?
  • Will Istio replace or integrate with Spring Cloud?
  • How is Istio different from Rancher?

Overall, I was very happy to get the chance to visit this great city and country and hope to come again in the near future.

Istio 101 with Minikube

 

As part of my Istio 101 talk, I like to show demos locally (because conference Wifi can be unreliable) and Minikube is perfect for this. Minikube gives you a local Kubernetes cluster on top of which you can install Istio.

In this post, I want to show how to do Istio 101 on Minikube. More specifically, I will show how to install Istio, deploy a sample application, install add-ons like Prometheus, Grafana, Zipkin, ServiceGraph and change traffic routes dynamically.

Minikube and kubectl

First, you need to install Minikube. There are instructions are on Minikube page. Once installed, you can start it with minikube start command and make sure it is running with minikube status:

> minikube status
minikube: Running
cluster: Running
kubectl: Correctly configured: pointing to minikube-vm at ...

Also make sure kubectl points to your minikube cluster with this:

> kubectl config use-context minikube
Switched to context "minikube"

Download & Setup Istio

Let’s download the latest Istio (0.7.1 as of today):

> curl -L https://git.io/getLatestIstio | sh -

Add Istio’s command line tool istioctl to your PATH. We’ll need it later:

> export PATH="$PATH:./istio-0.7.1/bin"

Install Istio

Now it’s time to install Istio. You can install Istio with or without mutual TLS enabled between services. In this case, let’s go with mutual TLS. Inside istio folder:

> kubectl apply -f install/kubernetes/istio-auth.yaml
...
service "istio-pilot" created
serviceaccount "istio-pilot-service-account" created
deployment.extensions "istio-pilot" created
service "istio-ingress" created
serviceaccount "istio-ingress-service-account" created
deployment.extensions "istio-ingress" created
serviceaccount "istio-ca-service-account" created
deployment.extensions "istio-ca" created
...

This installs Istio and its core components like ingress, mixer, pilot into a separate istio-system namespace. Let’s make sure Istio related pods and services are running:

> kubectl get svc -n istio-system
NAME          TYPE          CLUSTER-IP     EXTERNAL-IP 
istio-ingress LoadBalancer  10.99.175.236  <pending> 
istio-mixer   ClusterIP     10.101.16.135  <none>
istio-pilot   ClusterIP     10.97.241.47   <none> 

> kubectl get pods -n istio-system
NAME                           READY STATUS
istio-ca-86f55cc46f-p8npd      1/1   Running
istio-ingress-868d5f978b-nsjt6 1/1   Running
istio-mixer-65dc5549d6-cqg4j   3/3   Running
istio-pilot-657cb5ddf7-8dsr7   2/2   Running

Deploy BookInfo app

Now, we can deploy a sample app and get it managed by Istio. You have two choices here. First, you can install Istio initializer. This way, when a pod is created, a sidecar Envoy proxy will be created automatically but this requires a more complex setup with WebHooks and certificates. Your second option is to use istioctl command and inject the sidecar proxy yourself when you create the application pod. I’ll go for the second option as it’s easier to setup.

Istio comes with a number of sample apps under samples folder. Let’s go with the BookInfo app:

> kubectl create -f <(istioctl kube-inject -f samples/bookinfo/kube/bookinfo.yaml
service "details" created
deployment.extensions "details-v1" created
service "ratings" created
deployment.extensions "ratings-v1" created
service "reviews" created
deployment.extensions "reviews-v1" created
deployment.extensions "reviews-v2" created
deployment.extensions "reviews-v3" created
service "productpage" created
deployment.extensions "productpage-v1" created
ingress.extensions "gateway" created

As you can see, this creates bunch of microservices and some microservices have different versions as well.

Show BookInfo app

Now, let’s take a look at the app. External load balancers are not supported in Minikube. However, you can use the host IP of the ingress service, along with the NodePort, to access the ingress. To do that, we’ll set a GATEWAY_URL variable:

> export GATEWAY_URL=$(kubectl get po -l istio=ingress -n istio-system -o 'jsonpath={.items[0].status.hostIP}'):$(kubectl get svc istio-ingress -n istio-system -o 'jsonpath={.spec.ports[0].nodePort}')

And test with curl:

> curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage
200

You can also open a browser and see the web frontend for product page:

BookInfo app managed by Istio

At this point, we got the app deployed and managed by a basic installation of Istio. Next, we’ll install some add-ons.

Prometheus for metrics

The first add-on we’ll install is Prometheus for metrics.

> kubectl apply -f install/kubernetes/addons/prometheus.yaml
configmap "prometheus" created
service "prometheus" created
deployment.extensions "prometheus" created
serviceaccount "prometheus" created
clusterrole.rbac.authorization.k8s.io "prometheus" configured
clusterrolebinding.rbac.authorization.k8s.io "prometheus" configured

In a separate command tab, start sending some traffic to our app, so we can have some metrics flowing in our system:

> for i in {1..10}; do curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage; done
200
200
...

And setup port forwarding, so we can take a look at Prometheus:

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090

You can navigate to http://localhost:9090 and query some metrics, such as istio_request_count:

Prometheus in Istio

Grafana for dashboard

To visualize metrics in a nice dashboard, we can install Grafana. First install Grafana add-on:

> kubectl apply -f install/kubernetes/addons/grafana.yaml

And then setup port forwarding:

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000

Navigate to http://localhost:3000 and go to Istio Dashboard to see the dashboard:

Istio Dashboard in Grafana

Zipkin for tracing

Another useful add-on is Zipkin for distributed HTTP tracing. Just like other add-ons, first install it:

> kubectl apply -f install/kubernetes/addons/zipkin.yaml
deployment.extensions "zipkin" created
service "zipkin" created

Then, setup port forwarding:

> kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=zipkin -o jsonpath='{.items[0].metadata.name}') 9411:9411

Navigate to http://localhost:9411 to and take a look at traces for productpage:

Zipkin in Istio

ServiceGraph for dependency visualization

Finally, the last add-on we’ll take a look is my-favorite ServiceGraph. It gives us a nice graph of all microservice dependencides in our service mesh. First, install it:

> kubectl apply -f install/kubernetes/addons/servicegraph.yaml
deployment.extensions "servicegraph" created
service "servicegraph" created

Then, setup port forwarding:

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=servicegraph -o jsonpath='{.items[0].metadata.name}') 8088:8088

There are a few different graphs you can take a look:

ServiceGraph in Istio

Traffic Management

One of the nice things about Istio is that you can change routing rules between microservices dynamically. By default, there are no routing rules and you can see that by:

> istioctl get routerules 
No resources found.

When you go to the product page of BookInfo application and do a browser refresh a few times, you will see that the reviews section on the right keeps changing (the stars change color). This is because there are 3 different reviews microservices and everytime, a different microservice is invoked .

It is very easy though to pin microservices to a certain version. For example, let’s pin all microservices to version1:

> istioctl create -f samples/bookinfo/kube/route-rule-all-v1.yaml
Created config route-rule/default/productpage-default at revision 185533
Created config route-rule/default/reviews-default at revision 185534
Created config route-rule/default/ratings-default at revision 185535
Created config route-rule/default/details-default at revision 185536

Now, if you back to the product page and do a browser refresh, nothing changes because reviews microservice is pinned to version 1 now.

You can do even more complicated routing rules. For example, you can pin certain users to certain versions of a microservice. In this case, let’s pin the user “Jason”:

> istioctl create -f samples/bookinfo/kube/route-rule-reviews-test-v2.yaml
Created config route-rule/default/reviews-test-v2 at revision 185630

With this rule, if you login to the product page with username “Jason”, you should see the v2 version of reviews microservice.

Cleanup

This wraps up all the basic functionality of Istio that I wanted to show in Minikube.

To cleanup, let’s first delete the route rules:

> istioctl delete -f samples/bookinfo/kube/route-rule-all-v1.yaml
Deleted config: route-rule/default/productpage-default
Deleted config: route-rule/default/reviews-default
Deleted config: route-rule/default/ratings-default
Deleted config: route-rule/default/details-default

> istio-0.7.1 istioctl delete -f samples/bookinfo/kube/route-rule-reviews-test-v2.yaml
Deleted config: route-rule/default/reviews-test-v2

To delete Istio:

> kubectl delete -f install/kubernetes/istio-auth.yaml

At this point, everything related about Istio is deleted but the sample app is running. It’s just not managed by Istio anymore. If you want to delete the sample app as well:

> kubectl delete -f samples/bookinfo/kube/bookinfo.yaml
service "details" deleted
deployment.extensions "details-v1" deleted
service "ratings" deleted
deployment.extensions "ratings-v1" deleted
service "reviews" deleted
deployment.extensions "reviews-v1" deleted
deployment.extensions "reviews-v2" deleted
deployment.extensions "reviews-v3" deleted
service "productpage" deleted
deployment.extensions "productpage-v1" deleted
ingress.extensions "gateway" deleted