Application metrics in Istio


The default metrics sent by Istio are useful to get an idea on how the traffic flows in your cluster. However, to understand how your application behaves, you also need application metrics.

Prometheus has client libraries that you can use to instrument your application and send those metrics. This is good but it raises some questions:

  • Where do you collect those metrics?
  • Do you use Istio’s Prometheus or set up your own Prometheus?
  • If you use Istio’s Prometheus, what configuration do you need to get those metrics scraped?

Let’s try to answer these questions.

Istio vs. standalone Prometheus

In Prometheus, there’s a federation feature that allows a Prometheus server to scrape metrics from another Prometheus server. If you want to keep Istio metrics and application metrics separate, you can set up a separate Prometheus server for application metrics. Then, you can use federation to scrape those application metrics scraped with Istio’s Prometheus server.

A simpler approach is to scrape the application’s metrics using Istio’s Prometheus directly and that’s what I want to focus on here.

Sending application metrics

To send custom metrics from your application, you need to instrument your application using Prometheus’ client libraries. Which library to use depends on the language you’re using. As a C#/.NET developer, I used the .NET client for Prometheus and this blog post from Daniel Oliver has step-by-step instructions on how to send custom metrics from an ASP.NET Core application and see them in a local Prometheus server.

One thing you need to pay attention to is the port where you’re exposing your Prometheus metrics. In ASP.NETCore, the default port is 5000. When running locally, application metrics are exposed on localhost:5000/metrics. However, when you containerize your app, it’s common to expose your application over a different port, like 8080, and this becomes relevant later when we talk about configuration.

Assuming that you containerized and deployed your application on an Istio-enabled Kubernetes cluster, let’s now take a look at what we need to do to get these application metrics scraped by Istio’s Prometheus.


In Istio 1.0.5, the default installation files for Kubernetes, istio-demo.yaml or istio-demo-auth.yaml, already have scraping configurations for Prometheus under a ConfigMap. You can just search for prometheus.yml. There are 2 scraping jobs that are relevant for application metrics:

- job_name: 'kubernetes-pods'
- role: pod
- job_name: 'kubernetes-pods-istio-secure' 
  scheme: https

These are the jobs that scrape metrics from regular pods and pods where mTLS is enabled. It looks like Istio’s Prometheus should automatically scrape application metrics. However, in my first try, it didn’t work. I wasn’t sure what was wrong but Prometheus has some default endpoints:

  • /config: to see the current configuration of Prometheus.
  • /metrics: to see the scraped metrics.
  • /targets: to see the targets that’s being scraped and their status.

All of these endpoints are useful for debugging Prometheus:


Turns out, I needed to add some annotations in my pod YAML files in order to get Prometheus scrape the pod. I had to tell Prometheus to scrape the pod and on which port with these annotations:

kind: Deployment
  name: aspnetcore-v4
  replicas: 1
        app: aspnetcore
        version: v4
      annotations: "true" "8080"

After adding the annotations, I was able to see my application’s metrics in Prometheus:


However, it was only working for regular pods and I was not able to see metrics with mTLS enabled between pods.

Issue with Istio certs and Prometheus

After some investigation, I contacted the Istio team and turns out, there’s a bug. When Prometheus starts, it will attempt to mount the Istio-supplied certificates. However, they may not have been issued by Istio Citadel yet. Unfortunately, Prometheus does not retry to load the certificates, which leads to an issue scraping mTLS-protected endpoints.

It’s not ideal but there’s an easy workaround: restart the Prometheus pod. Restart forces Prometheus to pick up certificates and the application’s metrics start flowing for mTLS enabled pods as well.


Getting application metrics scraped by Istio’s Prometheus is pretty straightforward once you understand the basics. Hopefully, this post provided you the background info and configuration you need to achieve that.

It’s worth noting that Mixer is being redesigned and, in future versions of Istio, it will be directly embedded in Envoy. In that design, you’ll be able to send application metrics through Mixer and it’ll flow through the same overall metrics pipeline of the sidecar. This should make it simpler to get application metrics working end to end.

Thanks to the Istio team and my coworker Sandeep Dinesh for helping me to debug through issues, as I got things working.


Dialogflow fulfillment with C# and App Engine


Dialogflow is a developer platform for building voice or text-based conversational apps on a number of platforms such as Google Assistant, Facebook Messenger, Twilio, Skype and more. Earlier this year, we used Dialogflow to build a Google Assistant app and extended it to use the power of Google Cloud. You can read more about it on Google Cloud blog here, see the app code on GitHub here and one of my talk videos about the app is here.

Out of the box, Dialogflow provides features such as intents for matching user input to responses, entities to extract relevant information from conversations, contexts to maintain the state of the conversation. It also provides a way to extend its functionality via fulfillments.


A fulfillment is a HTTPS webhook that Dialogflow can forward requests to. From then on, your code is in charge of handling the request and generating a response. This is very useful for having custom logic or introducing further intelligence/data to your app. In our app, we used fulfillment to search for images using Google Custom Search, used Vision API to analyze images with Machine Learning and performed rich data analysis with BigQuery.

An easy way of implementing a fulfillment webhook is via the Inline Editor in Dialogflow console. Inline Editor enables you to write a Node.js function to handle Dialogflow requests and deploy it to Cloud Functions with Firebase (see this codelab for details). While Inline Editor is easy, it only allows a single webhook and it’s not developer friendly. Ideally, you’d develop code locally and use some kind of version control.

You can setup a local environment and use Firebase to deploy the function to Cloud Function (this codelab shows how). While this is a more realistic setup, it’s still constrained by Cloud Functions limitations and Node.js is the only language you can use.

AppEngine for fulfillment

In our app, we took a different approach to implement the webhook. Instead of Node.js, we decided to use C# and instead of Cloud Functions, we containerized our app and deployed to App Engine for more flexibility.

Since it’s a container, we could have deployed to Kubernetes Engine as well but we chose App Engine because:

  1. App Engine gives us a HTTPS endpoint out of the box with minimal hassle whereas Kubernetes Engine HTTPS setup is not trivial.
  2. Versioning in App Engine allowed us to update and deploy our app easily, test it out and forward all the traffic to the new version when we’re ready.

HelloWorld fulfillment with C#

To give you an idea on how to write a Dialogflow fulfillment on AppEngine using C#, let’s write a HelloWorld fulfillment webhook.

First, you need to first create an ASP.NET Core web app:

dotnet new web -n fulfillment

And add Dialogflow NuGet package to the project. This package will allow us to parse requests from Dialogflow and generate responses:

dotnet add package Google.Cloud.Dialogflow.V2 --version 1.0.0-beta02

Now, we can edit Startup.cs file and add logic to handle Dialogflow requests. We need to first refer to Dialogflow NuGet library and some of its dependencies:

using Google.Cloud.Dialogflow.V2;
using Google.Protobuf;
using System.IO;

Then, we need to create a Protobuf jsonParser that we’ll use to parse the body of HTTP requests:

private static readonly JsonParser jsonParser = 
   new JsonParser(JsonParser.Settings.Default.WithIgnoreUnknownFields(true));

Finally, let’s change the Configure method to parse the request from Dialogflow and simply echo back with a Hello message with the intent name:

app.Run(async (context) =>
    WebhookRequest request;

    using (var reader = new StreamReader(context.Request.Body))
         request = jsonParser.Parse<WebhookRequest>(reader);

    var response = new WebhookResponse
        FulfillmentText = "Hello from " + request.QueryResult.Intent.DisplayName

    await context.Response.WriteAsync(response.ToString());

Notice how the HTTP request body is parsed as a WebhookRequest. This is the request Dialogflow sends to our code. In response, we create a WebhookResponse with FulfillmentText and send back. Dialogflow will use the FulfillmentText to say what we asked.

Deploy to App Engine

Now, we’re ready to deploy our code to App Engine. We need to publish the app first:

dotnet publish -c Release

This will create a DLL inside the bin/Release/netcoreapp2.0/publish/ folder. Inside this folder, create an app.yaml file for App Engine with the following contents:

env: flex
runtime: aspnetcore

This will tell App Engine that this is an ASP.NET Core app and it should deploy to App Engine flex.

Finally, deploy to App Engine and let’s call this initial version v0:

gcloud app deploy --version v0

Once the deployment is done, you will have an HTTPS endpoint in the form of https://<yourprojectid&gt; that you can use a fulfillment webhook in Dialogflow.

Test with Dialogflow

We’re finally ready to test our fulfillment webhook. In Dialogflow console, specify fulfillment url:


Then, create an intent with some training phrases:


Finally, make sure the fulfillment webhook is enabled for the intent:


We can use the simulator in Dialogflow console to test out the intent:

hello .net

When we say “Hello .NET”, this triggers the webhook and our webhook code simply echoes back the intent name which is “hello c# intent” in this case.

Obviously, a real world webhook would do much more than a simple reply. For example, you would need to keep track of user sessions and get or create conversations based on those sessions (see DialogflowApp of our app on GitHub). You also need to find a way to match intents to handlers on the server (see Conversation of our app on GitHub on this).

Hopefully, this blog post provided the basics to get started with fulfillment webhook using C# and App Engine.


.NET Days in Zurich, Shift Conf in Split

Last week was a quite interesting week in terms of travel. First I got to visit Zurich again after a while for .NET Day and then I got to visit the Croatian coastal town Split for the first time for Shift Conference.

.NET Days in Zurich

When I used to work at Adobe, part of my team was based in Basel, Switzerland. As a result, I used to visit Basel, Zurich and other Swiss cities quite often. Since I left Adobe, I visited Switzerland only once 2 years ago. I was naturally excited to visit Zurich again for .NET Day.

I arrived a day early for the conference and explored Zurich a little bit. Google has a big office in Zurich with strong engineering. I got to visit that office as well for the first time and spent half day there working from the office.

Talk & Questions

.NET Day is a small .NET focused conference with 2 tracks and about 200 attendees. It was the first time presenting there. I did my “Google Home meets .NET Containers” talk where I show how to connect Google Home mini to a .NET container running in Google Cloud. It’s a fun talk and always get a good reaction from the crowd.

Conference organizers did a couple of special things for speakers. First, we got speaker t-shirts with our names on it. I think this was the first time I got a t-shirt with my name which was nice. Second, they organized a photo shoot with the conference photographer, Irene Bizic. She did an amazing job and as a result, I got a few very nice pictures of myself.

After my talk, I got some questions on the pricing model of Vision API. Someone also asked me about how to test Dialogflow end to end.

Shift Conference in Split

After Zurich, I flew to Split, Croatia. As you might remember, I was in Zagreb, the Croatian capital last October but this was the first time I got to visit the coastal part of Croatia.

I have to say I was impressed with Split. It’s a small town with rich history, great food and good beaches. The weather was very good with 30 degrees and sunny almost every day. I tried food in 3-4 different places and every place was very good. I had opened the beach season back in January in Rio but it had been a while since then and it was nice to swim again one afternoon in Split.

Talk & Questions

This was the first time I spoke at Shift Conference. I was expecting a small conference in a small town but I was totally wrong. Shift is a big well-organized conference (1000+ attendees) with a single track (and a workshop) over 2 days. There were lots of speakers from all over the place, a ton of technical content. The conference happens in an old theatre kind of place and I was super impressed with the stage. It was probably the most impressive stage I ever spoke at.

I did my “Google Home meets .NET containers” talk again. It was super fun again and I got reaction from the crowd both during and after my talk. After the conference, I got some general questions about Google Cloud and Dialogflow.

I have to say the organizers did an amazing job with the conference. There were speaker dinners and parties every night and they really tried to make it a fun event not just for attendees but for speakers as well.

I hope to visit Split again next year and explore more of Croatia and surroundings.

Deploying ASP.NET Core apps on Kubernetes/Container Engine

In my previous post, I talked about how to deploy a containerised ASP.NET Core app to App Engine (flex) on Google Cloud. App Engine (flex) is an easy way to run containers in production: Just send your container and let Google Cloud figure out how to run it at scale. It comes with some nice default features such as versioning, traffic splitting, dashboards and autoscaling. However, it doesn’t give you much control.

Sometimes, you need to create a cluster of containers and control how each container is deployed and scaled. That’s when Kubernetes come into play. Kubernetes is an open source container management platform that helps you to manage a cluster of containers and Container Engine is Kubernetes managed by Google Cloud.

In this cloud minute, I show how to deploy an ASP.NET Core app to Kubernetes running on Container Engine.

If you want to go through these steps yourself, we also have a codelab for you that you can access here.

Deploying ASP.NET Core apps on App Engine

I love how easy it is to deploy and run containerized ASP.NET Core apps on App Engine (flex). So much so that, I created a Cloud Minute recently to show you how, here it is.

It basically involves 3 steps:

  1. Create your ASP.NET Core app using dotnet command line tool inside Cloud Shell and publish your app to get a self-contained DLL.
  2. Containerize your app by creating a Dockerfile, relying on the official App Engine image and pointing to the self-contained DLL of your app.
  3. Create an app.yaml file for App Engine and use gcloud to deploy to App Engine.

That’s it! If you want to go through these steps yourself, we also have a codelab for you that you can access here.