First time in China

Visiting a new country is always extra special for me and a couple of weeks ago, I got to do just that and visited Beijing, China for the first time.

After TDC in Floripa, I flew to Beijing for QCon conference and for a Google Developer Group (GDG) meetup. It was the first time for me in China (my 48th country!) and first time speaking at QCon, so I was naturally excited. I didn’t know what to expect in Beijing but I was pleasantly surprised. Beijing is a modern city with good public transportation and interesting sites to visit. I got the feeling that it’s also very international with many expats, definitely more than I expected. I visited the iconic sites like Tiananmen Square and the Great Wall of China. I also visited Google’s Beijing office (my 35th Google office visit!)

QCon Beijing

QCon is a global conference that happens in multiple locations like New York, London, San Francisco, Sao Paulo, Beijing, Shanghai. QCon Beijing was a multi-day multi-track conference with more than 1000+ attendees on a diverse set of topics. I could only attend the last day of the conference on Sunday.

I did my Istio 101 talk to a full room of probably 200 people. I was surprised by the number of attendees and the level of questions I received after my talk. It seems like many people in Beijing are active users of the cloud related technologies like Kubernetes and they were very curious about Istio.


After my talk, I received the usual Istio questions such as:

  • When will Istio be production ready?
  • What’s Istio roadmap, what features are planned in the upcoming releases?
  • What’s the performance overhead of Envoy proxy?
  • What are the performance characteristics of Istio?

I also received a couple unique questions such as:

  • How do you talk from Istio to an external database?
  • What’s the state of CloudFoundry with Istio?
  • How to upgrade Istio? What’s the recommended procedure?

GDG Beijing

A couple of days after QCon Beijing, I did the same Istio talk at Google Developer Group in Beijing. It was at a co-working space and organizers did a great job in putting the event together with banners, videos before the event and a great Q&A session after my talk.


After the talk, we had an extensive Q&A session. There were the usual Istio questions on roadmap, production readiness and envoy overhead. More interesting questions I remember were:

  • How long lived connections are handled in Istio?
  • How to delete route rules without yaml?
  • How does Istio see user agent headers when encrypted?
  • Will Istio replace or integrate with Spring Cloud?
  • How is Istio different from Rancher?

Overall, I was very happy to get the chance to visit this great city and country and hope to come again in the near future.


Trip Report: TDC in Florianópolis

Back in my favorite country: Brazil

Last week, I was in my favorite country, Brazil, for The Developer Conference (TDC) in Florianópolis (aka Floripa). I went to Brazil for the first time last July. Since then, I’ve been there 3 more times and I gradually fell in love with it. People are friendly, their BBQ is amazing, scenery is beautiful. I always have a good time in Brazil and this time wasn’t an exception.

In all previous times, I was mainly in Sao Paulo for a conference followed by a short trip to Rio (my favorite city!) for myself. This time, I went to Florianópolis which is a city to the south of Sao Paulo. It has a similar feeling as Rio (i.e. beach town) but smaller and felt safer than Rio. It had amazing views of mountains and sea, great food and friendly people, as always the case with Brazil.


TDC is probably the biggest tech conference in Brazil, along with QCon. It’s a community driven conference that happens in multiple locations (Sao Paulo, Porto Alegre, Floripa) throughout the year. It is usually multi-day, multi-track conference on a number of diverse topics (cloud, microservices, big data, .NET, Java) and a big technical audience of usually 1000+ people. I was told that TDC Floripa was close to 4000 people which is amazing. Most of the talks are in Portuguese with a few exceptions in English, like my talk.

At TDC, I got to meet my friend, Rafael Benevides from RedHat and made some new friends such as Elder Moraes and Rodrigo Cândido Silva who were the track leads for microservices track that I was speaking at. Although the conference was 4 days, I could only attend the first day. I had to fly out the next day for another conference.

My talk on Istio

I first spoke at TDC last July in Sao Paulo about running .NET Containers with Kubernetes on Google Cloud. This year, I was happy to be selected again for Floripa for my Istio talk. It was a big room full of people, in the end probably about 120+ people.

It was the same talk I did in Codemotion Rome a couple weeks ago on introduction to containers, Kubernetes and Istio with demos sprinkled along the talk.


After the talk, the audience was engaged and asked a number of questions. These are some of them that I remember.

Production readiness and roadmap. People asked when Istio will be considered production ready and what’s the roadmap of features coming up.

What’s the overhead of Envoy? People want to know what’s the performance overhead of Envoy and Istio in general. While the overhead of Envoy is supposed to be minimal, I don’t have exact figures. I intend to look into this further to give people more guidance.

Existing ingress rules. Someone mentioned that they have an existing Kubernetes cluster with a number of existing ingress rules and they were curious how Istio ingress work or doesn’t work with existing ingress rules.

DOS attacks with ingress. Someone asked if Istio ingress handles Denial of Service attacks. I don’t think ingress itself has DOS protection but a loadbalancer infront of DOS would handle that in Google Cloud.

Rancher and Istio. How’s Istio different than Rancher? Rancher is about deploying Kubernetes to multiple clouds and making that process easy. Istio is built on top of Kubernetes and provides microservice level features. So it’s much more than just Kubernetes and Rancher.

WebSockets with Envoy. Someone mentioned that they had issues using WebSockets with Envoy. I told him that latest versions of Istio have WebSocket support, you just need to enable it but I couldn’t really comment on the issue without seeing the details.

After talk interview

After my talk, an Oracle LATAM DevRel person, briefly interviewed me about my talk and here’s the video:

Istio 101 with Minikube


As part of my Istio 101 talk, I like to show demos locally (because conference Wifi can be unreliable) and Minikube is perfect for this. Minikube gives you a local Kubernetes cluster on top of which you can install Istio.

In this post, I want to show how to do Istio 101 on Minikube. More specifically, I will show how to install Istio, deploy a sample application, install add-ons like Prometheus, Grafana, Zipkin, ServiceGraph and change traffic routes dynamically.

Minikube and kubectl

First, you need to install Minikube. There are instructions are on Minikube page. Once installed, you can start it with minikube start command and make sure it is running with minikube status:

> minikube status
minikube: Running
cluster: Running
kubectl: Correctly configured: pointing to minikube-vm at ...

Also make sure kubectl points to your minikube cluster with this:

> kubectl config use-context minikube
Switched to context "minikube"

Download & Setup Istio

Let’s download the latest Istio (0.7.1 as of today):

> curl -L | sh -

Add Istio’s command line tool istioctl to your PATH. We’ll need it later:

> export PATH="$PATH:./istio-0.7.1/bin"

Install Istio

Now it’s time to install Istio. You can install Istio with or without mutual TLS enabled between services. In this case, let’s go with mutual TLS. Inside istio folder:

> kubectl apply -f install/kubernetes/istio-auth.yaml
service "istio-pilot" created
serviceaccount "istio-pilot-service-account" created
deployment.extensions "istio-pilot" created
service "istio-ingress" created
serviceaccount "istio-ingress-service-account" created
deployment.extensions "istio-ingress" created
serviceaccount "istio-ca-service-account" created
deployment.extensions "istio-ca" created

This installs Istio and its core components like ingress, mixer, pilot into a separate istio-system namespace. Let’s make sure Istio related pods and services are running:

> kubectl get svc -n istio-system
NAME          TYPE          CLUSTER-IP     EXTERNAL-IP 
istio-ingress LoadBalancer  <pending> 
istio-mixer   ClusterIP  <none>
istio-pilot   ClusterIP   <none> 

> kubectl get pods -n istio-system
NAME                           READY STATUS
istio-ca-86f55cc46f-p8npd      1/1   Running
istio-ingress-868d5f978b-nsjt6 1/1   Running
istio-mixer-65dc5549d6-cqg4j   3/3   Running
istio-pilot-657cb5ddf7-8dsr7   2/2   Running

Deploy BookInfo app

Now, we can deploy a sample app and get it managed by Istio. You have two choices here. First, you can install Istio initializer. This way, when a pod is created, a sidecar Envoy proxy will be created automatically but this requires a more complex setup with WebHooks and certificates. Your second option is to use istioctl command and inject the sidecar proxy yourself when you create the application pod. I’ll go for the second option as it’s easier to setup.

Istio comes with a number of sample apps under samples folder. Let’s go with the BookInfo app:

> kubectl create -f <(istioctl kube-inject -f samples/bookinfo/kube/bookinfo.yaml
service "details" created
deployment.extensions "details-v1" created
service "ratings" created
deployment.extensions "ratings-v1" created
service "reviews" created
deployment.extensions "reviews-v1" created
deployment.extensions "reviews-v2" created
deployment.extensions "reviews-v3" created
service "productpage" created
deployment.extensions "productpage-v1" created
ingress.extensions "gateway" created

As you can see, this creates bunch of microservices and some microservices have different versions as well.

Show BookInfo app

Now, let’s take a look at the app. External load balancers are not supported in Minikube. However, you can use the host IP of the ingress service, along with the NodePort, to access the ingress. To do that, we’ll set a GATEWAY_URL variable:

> export GATEWAY_URL=$(kubectl get po -l istio=ingress -n istio-system -o 'jsonpath={.items[0].status.hostIP}'):$(kubectl get svc istio-ingress -n istio-system -o 'jsonpath={.spec.ports[0].nodePort}')

And test with curl:

> curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage

You can also open a browser and see the web frontend for product page:

BookInfo app managed by Istio

At this point, we got the app deployed and managed by a basic installation of Istio. Next, we’ll install some add-ons.

Prometheus for metrics

The first add-on we’ll install is Prometheus for metrics.

> kubectl apply -f install/kubernetes/addons/prometheus.yaml
configmap "prometheus" created
service "prometheus" created
deployment.extensions "prometheus" created
serviceaccount "prometheus" created "prometheus" configured "prometheus" configured

In a separate command tab, start sending some traffic to our app, so we can have some metrics flowing in our system:

> for i in {1..10}; do curl -o /dev/null -s -w "%{http_code}\n" http://${GATEWAY_URL}/productpage; done

And setup port forwarding, so we can take a look at Prometheus:

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0]}') 9090:9090

You can navigate to http://localhost:9090 and query some metrics, such as istio_request_count:

Prometheus in Istio

Grafana for dashboard

To visualize metrics in a nice dashboard, we can install Grafana. First install Grafana add-on:

> kubectl apply -f install/kubernetes/addons/grafana.yaml

And then setup port forwarding:

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0]}') 3000:3000

Navigate to http://localhost:3000 and go to Istio Dashboard to see the dashboard:

Istio Dashboard in Grafana

Zipkin for tracing

Another useful add-on is Zipkin for distributed HTTP tracing. Just like other add-ons, first install it:

> kubectl apply -f install/kubernetes/addons/zipkin.yaml
deployment.extensions "zipkin" created
service "zipkin" created

Then, setup port forwarding:

> kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=zipkin -o jsonpath='{.items[0]}') 9411:9411

Navigate to http://localhost:9411 to and take a look at traces for productpage:

Zipkin in Istio

ServiceGraph for dependency visualization

Finally, the last add-on we’ll take a look is my-favorite ServiceGraph. It gives us a nice graph of all microservice dependencides in our service mesh. First, install it:

> kubectl apply -f install/kubernetes/addons/servicegraph.yaml
deployment.extensions "servicegraph" created
service "servicegraph" created

Then, setup port forwarding:

> kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=servicegraph -o jsonpath='{.items[0]}') 8088:8088

There are a few different graphs you can take a look:

ServiceGraph in Istio

Traffic Management

One of the nice things about Istio is that you can change routing rules between microservices dynamically. By default, there are no routing rules and you can see that by:

> istioctl get routerules 
No resources found.

When you go to the product page of BookInfo application and do a browser refresh a few times, you will see that the reviews section on the right keeps changing (the stars change color). This is because there are 3 different reviews microservices and everytime, a different microservice is invoked .

It is very easy though to pin microservices to a certain version. For example, let’s pin all microservices to version1:

> istioctl create -f samples/bookinfo/kube/route-rule-all-v1.yaml
Created config route-rule/default/productpage-default at revision 185533
Created config route-rule/default/reviews-default at revision 185534
Created config route-rule/default/ratings-default at revision 185535
Created config route-rule/default/details-default at revision 185536

Now, if you back to the product page and do a browser refresh, nothing changes because reviews microservice is pinned to version 1 now.

You can do even more complicated routing rules. For example, you can pin certain users to certain versions of a microservice. In this case, let’s pin the user “Jason”:

> istioctl create -f samples/bookinfo/kube/route-rule-reviews-test-v2.yaml
Created config route-rule/default/reviews-test-v2 at revision 185630

With this rule, if you login to the product page with username “Jason”, you should see the v2 version of reviews microservice.


This wraps up all the basic functionality of Istio that I wanted to show in Minikube.

To cleanup, let’s first delete the route rules:

> istioctl delete -f samples/bookinfo/kube/route-rule-all-v1.yaml
Deleted config: route-rule/default/productpage-default
Deleted config: route-rule/default/reviews-default
Deleted config: route-rule/default/ratings-default
Deleted config: route-rule/default/details-default

> istio-0.7.1 istioctl delete -f samples/bookinfo/kube/route-rule-reviews-test-v2.yaml
Deleted config: route-rule/default/reviews-test-v2

To delete Istio:

> kubectl delete -f install/kubernetes/istio-auth.yaml

At this point, everything related about Istio is deleted but the sample app is running. It’s just not managed by Istio anymore. If you want to delete the sample app as well:

> kubectl delete -f samples/bookinfo/kube/bookinfo.yaml
service "details" deleted
deployment.extensions "details-v1" deleted
service "ratings" deleted
deployment.extensions "ratings-v1" deleted
service "reviews" deleted
deployment.extensions "reviews-v1" deleted
deployment.extensions "reviews-v2" deleted
deployment.extensions "reviews-v3" deleted
service "productpage" deleted
deployment.extensions "productpage-v1" deleted
ingress.extensions "gateway" deleted


Trip Report: Codemotion in Rome

The Eternal City

Last week, I was in one of my favorite cities, Rome, for Codemotion conference. There are many cities to see in the world and normally, I do not like to revisit cities that I’ve been before. However, Rome is a great exception. It was my fifth visit there over the years and it was still as exciting as the first time. I also found out that Google has a small office in Rome, so I paid a quick visit next morning before attending the conference.


Codemotion is a tech conference that happens in multiple locations around Europe like Amsterdam, Tel Aviv and of course, its original home Rome. Codemotion Rome lasts 2 days and there are all sorts of topics on all sorts of languages, frameworks and technologies. The audience is mainly software engineers from Italy and speakers are from all over the world. I don’t know the official numbers but my guess is that there were more than 1500 developers over 2 days. Overall, it’s a very diverse and fun conference to be part of.

Google Cloud was one of the sponsors and we had good presence in Codemotion with 4 speakers. On the first day, Giovanni Galloro, a customer engineer from Rome office, had a talk in the opening keynote. It was followed by a Big Data and Machine Learning talk by my colleague and good friend, Felipe Hoffa. I had my talk in the afternoon right after lunch and Alfredo Morresi, our community manager in Italy, did his talk on the second day. Azzura Rogone, our community specialist was there as well with a number of Google Developer Group people and their booth. I had to unfortunately fly out of Rome next day for another conference, so I missed the second day.

My talk on Istio

It was my second time speaking at Codemotion Rome. Last year, I talked about gRPC and this year I did a talk on Istio, an open source project to manage microservices. The room was full of about 80 people, with some people standing.

In my talk, I briefly talked about containers and Kubernetes and how they fall short in managing microservices. Then, I gave an overview of Istio and how it helps with managing microservices in production. Finally, I got into details of Istio building blocks like Envoy, Pilot, Mixer, Traffic Management and showed demos along the way.

My laptop refused to connect to the projector, so I had to use someone else’s laptop to present. I was planning on doing live demos of Istio running on Kubernetes in Minikube but without my laptop, that was not possible. Thankfully, I had videos of my demos, so I could show those instead. If you want to have my slides, here they are:

Stop reinventing the wheel with Istio



After the talk, I got some questions. These are some of them that I remember.

First question was whether Kubernetes and Istio will merge into a single project at some point as they seem to complement each other well. My answer was probably not because someone people just need Kubernetes, some people need more. These are two open source projects that will coexist and complement each other but it will be up to people to decide if they want to use it together.

There was a question on whether you can have custom metrics in Istio and the answer was yes. Istio is quite pluggable. By default, you get some default metrics but it’s quite easy to add your custom metrics.

Another question was whether there’s a way to plug in their default web gateway to Istio. I wasn’t too sure of this question but in Istio, there’s Ingress that controls all the incoming traffic, so whatever the gateway is doing, it should be possible to limit things with Ingress.

Finally, the last question was about custom dashboards. Someone was wondering whether it’s possible to create a custom dashboard and plug that into Istio, instead of Grafana. Again, since Istio is pluggable, it should be possible to create your own add-ons and plug that in but I told him that it’s not something I have tried myself.

Istio + Kubernetes on Windows


I’ve been recently looking into Istio, an open platform to connect and manage microservices. After Containers and Kubernetes, I believe that Istio is the next step in our microservices journey where we standardize on tools and methods on how to manage and secure microservices. Naturally, I was very excited to get my hands on Istio.

While setting up Istio on Google Kubernetes Engine (GKE) is pretty straightforward, it’s always useful to have a local setup for debugging and testing. I specifically wanted to setup Istio on my local Minikube Kubernetes cluster on my Windows machine. I ran into a few minor issues that I want to outline here in case it is useful to someone out there.

I assume you have a Minikube cluster setup already and running. If not, you can check out my previous post on how to setup and run a Minikube cluster on your Windows machine. Istio has a Quickstart tutorial for Kubernetes. I’ll follow that but it’s Linux-centric and some of the commands have to be adopted for Windows.

Download Istio

Here is the command to download Istio from Quickstart:

curl -L | sh -

This is a Linux shell command and it won’t work on Windows cmd or PowerShell. Thankfully, someone already wrote an equivalent PowerShell script here. I used the script as is, only changed the IstioVersion to 0.5.1, the latest Istio version as of today:

 [string] $IstioVersion = "0.5.1"

The script downloads Istio and sets an ISTIO_HOME as environment variable.

PS C:\dev\local\istio> .\getLatestIstio.ps1
Downloading Istio from
0.5.1/ to path C:\dev\local\istio

Then, I added %ISTIO_HOME%\bin to PATH  to make sure I can run istoctl commands.

Install and Verify Istio

To install Istio and enable mutual TLS authentication between sidecars, I ran the same command in the quickstart:

PS C:\istio-0.5.1> kubectl apply -f install/kubernetes/istio-auth.yaml
namespace "istio-system" created
clusterrole "istio-pilot-istio-system" created
clusterrole "istio-sidecar-injector-istio-system" created
clusterrole "istio-mixer-istio-system" created
clusterrole "istio-ca-istio-system" created
clusterrole "istio-sidecar-istio-system" created

And verify that all the Istio pods are running:

PS C:\istio-0.5.1> kubectl get pods -n istio-system
NAME                           READY STATUS  RESTARTS AGE
istio-ca-797dfb66c5-x4bzs      1/1   Running  0       2m
istio-ingress-84f75844c4-dc4f9 1/1   Running  0       2m
istio-mixer-9bf85fc68-z57nq    3/3   Running  0       2m
istio-pilot-575679c565-wpcrf   /2    Running  0       2m

Deploy the sample app

Deploying an app is a little different on Windows as well. To deploy the BookSample app with Envoy container injection, this is the command you would normally run on Linux:

kubectl create -f <(istioctl kube-inject -f samples/bookinfo/kube/bookinfo.yaml)

The redirection causes problems on PowerShell. Instead, you can first run the istioctl command and save it to an intermediate yaml:

istioctl kube-inject -f .\samples\bookinfo\kube\bookinfo.yaml > bookinfo_inject.yaml

Then, you can apply the intermediate yaml:

PS C:\istio-0.5.1> kubectl create -f .\bookinfo_inject.yaml
service "details" created
deployment "details-v1" created
service "ratings" created
deployment "ratings-v1" created
service "reviews" created
deployment "reviews-v1" created
deployment "reviews-v2" created
deployment "reviews-v3" created
service "productpage" created
deployment "productpage-v1" created
ingress "gateway" created


With that, you will have BookInfo app deployed and managed by Istio. Hope this was useful to get Istio + Kubernetes running in Minikube on Windows.