Christian Weyer's Blog

Colorful Windows Azure Compute monitoring & diagnostics with Azure Diagnostics Manager

One of the open secrets of Windows Azure is that diagnostics is powerful but also very tedious and partly error-prone. In addition, there is no really good and useful tooling provided by Microsoft. We want to have a light-weight means to monitor application, look at performance counters, see event logs etc.

There is the SCOM MP for Windows Azure, but this is more an enterprise-targeted solution which cannot really be called light-weight. This leaves us with 3rd party tools.

My 3rd party tool of choice for doing diagnostics and monitoring of Windows Azure applications is Cerebrata’s Azure Diagnostics Manager (ADM).

But before we use any tools to visualize data we need to set up and instruct our code to collect diagnostics data. In the following example we set up three performance counters and two event logs to be collected and be transferred over to the configured storage account:

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        var diagConfig = DiagnosticMonitor.GetDefaultInitialConfiguration();

        var procTimeConfig = new PerformanceCounterConfiguration();
        procTimeConfig.CounterSpecifier = @"\Processor(*)\% Processor Time";
        procTimeConfig.SampleRate = System.TimeSpan.FromSeconds(5.0);
        diagConfig.PerformanceCounters.DataSources.Add(procTimeConfig);

        var diskBytesConfig = new PerformanceCounterConfiguration();
        diskBytesConfig.CounterSpecifier = @"\LogicalDisk(*)\Disk Bytes/sec";
        diskBytesConfig.SampleRate = System.TimeSpan.FromSeconds(5.0);
        diagConfig.PerformanceCounters.DataSources.Add(diskBytesConfig);

        var workingSetConfig = new PerformanceCounterConfiguration();
        workingSetConfig.CounterSpecifier =
            @"\Process(" +
            System.Diagnostics.Process.GetCurrentProcess().ProcessName +
            @")\Working Set";
        workingSetConfig.SampleRate = System.TimeSpan.FromSeconds(5.0);
        diagConfig.PerformanceCounters.DataSources.Add(workingSetConfig);
        diagConfig.PerformanceCounters.ScheduledTransferPeriod = 
          TimeSpan.FromSeconds(30);

        diagConfig.WindowsEventLog.DataSources.Add("System!*");
        diagConfig.WindowsEventLog.DataSources.Add("Application!*");
        diagConfig.WindowsEventLog.ScheduledTransferPeriod = 
          TimeSpan.FromSeconds(30);

        DiagnosticMonitor.Start(
          "Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", 
          diagConfig);

        return base.OnStart();
    }
}

When you look at your storage account after a while you see that for the above example two tables have been created (if they haven’t been already in place), namely WADPerformanceCountersTable and WADWindowsEventLogsTable:

Of course you can use your storage explorer of choice and look at the raw data collected Or we choose the above mentioned ADM to use its Dashboard feature to visualize our performance counter and event logs (note that ADM polls the data from the storage account and you can specify the poll interval):

If we like we can even drill into each of the diagnostics categories. The following picture shows a detailed view of three performance counters:

Again: the Azure Diagnostics Manager is a handy tool and is what I personally have been using in projects and recommending our clients.

Hope this helps.

Troubleshooting a typical Windows Azure Storage Emulator database issue

By default the Windows Azure storage emulator on your local machine works above a SQL Server Express instance which needs to be named ‘SQLExpress’. If you open up SQL Server Management Studio you can see the DevelopmentStorageDb20090919 database:

But not everybody has SQL Express installed or enabled. In this case you may see the following error in Visual Studio when you try to debug your Windows Azure project:

The detailed error message from the Output window look like this:

Windows Azure Tools: Failed to initialize Windows Azure storage emulator.
Unable to start Development Storage. Failed to start Development Storage:
the SQL Server instance ‘localhost\SQLExpress’ could not be found.
Please configure the SQL Server instance for Development Storage using
the ‘DSInit’ utility in the Windows Azure SDK.

OK. So let’s start the dsinit.exe tool with the /? parameter to get some basic help information:

With this information we know that we can initialize our local SQL Server default instance with the following command:

dsinit /sqlInstance:.

Now you can run and debug your Windows Azure projects locally with a non-SQL Express instance-based development storage.

Hope this helps.

Poor man’s approach to application pool warm-up for IIS in a Windows Azure web role

This is another small but quite useful idea and snippet from a real Windows Azure project.

We all know that after deploying our web role to the Cloud that the first request would take a looong time. A web site’s application pool needs to be spawned, ASP.net compilation kicks in etc.
In order to already have a web site up and running and pre-warmed there are several approaches, some of them being intrinsically supported by ASP.NET – but a bit complex, IMO - and some still in beta, AFAIK. But we do not need no magic feature for this. We already have the WebRole.cs with the RoleEntryPoint-derived class. Here we do our initialization and potentially our background work.

And what could be a better candidate than periodically calling a certain (maybe the base) URL of a web site in order to keep the app pool up (if you want so!).

The following code exactly does this – based on the name of a Windows Azure internal or input endpoint (which belongs to a web site) we simply call the base address of that endpoint for the current role instance:

using System;
using System.Net;
using Microsoft.WindowsAzure.ServiceRuntime;

namespace Thinktecture.WindowsAzure
{
    public static class WebSiteUtilities
    {
        public static void WarmUp(string endpointName)
        {
            try
            {
                var endpoint = RoleEnvironment.CurrentRoleInstance
                  .InstanceEndpoints[endpointName];
                var address = String.Format("{0}://{1}:{2}",
                    endpoint.Protocol,
                    endpoint.IPEndpoint.Address,
                    endpoint.IPEndpoint.Port);
                var webClient = new WebClient();
                webClient.DownloadString(address);
            }
            catch (Exception)
            {
                // intentionally swallow all exceptions here.
            }
        }
    }
}

The actual initial and recurring warm-up call can then be placed into the Run method our our web role:

public class WebRole : RoleEntryPoint
{
    public override void Run()
    {
        try
        {
            while (true)
            {
                Trace.WriteLine("Warming up web site...");
                WebSiteUtilities.WarmUp("HttpIn");

                Thread.Sleep(TimeSpan.FromMinutes(19));
            }
        }
        catch (Exception ex)
        {
            DiagnosticsHelper.WriteExceptionToBlobStorage(ex);
        }
    }
}

Done.
Note: I chose 19 minutes here between the calls as the default idle timeout of an IIS application pool is 20 minutes. You may want to change this to your own defaults or logic.

Hope this helps.

Creating custom Windows event logs and sources in a Windows Azure Compute startup task

Yet another chapter in the Windows Azure Compute startup tasks kung-fu saga.

There still needs to be a single real-world application I have seen that does not write one or the other information to the Windows event log. As you already know, we need to have elevated privileges in order to create either a new event log source or a new event log.

When you hear ‘elevated privileges’ in the context of Windows Azure you should immediately think about startup tasks. Therefore, here is our today’s startup task to create a new event log and event log source.
Let’s start with the .cmd file to spawn the task:

powershell -Command "Set-ExecutionPolicy Unrestricted"
powershell .\CreateEventLog.ps1

The according .ps1 file looks like this (for PowerShell 2.0, i.e. Windows Server 2008R2, i.e. osFamily=’2’ in a Windows Azure project):

new-eventlog -logname 'My Azure Logs' -Source 'AzureEventLogs'

With this in place, we can then use the usual suspects in our .NET code to write to the event log, e.g.:

var azureEventLog = new EventLog();
azureEventLog.Source = "AzureEventLogs";
azureEventLog.WriteEntry("Event log entry from da cloud.", 
    EventLogEntryType.Warning);

Voila.
Just to prove it:

Hope this helps.

Hosting a public web site and an internal services site in one Windows Azure web role

Imagine you want to host some (Web) services in an IIS web site inside your Windows Azure compute web role. In addition, you also want to host your web application/portal in another web site in Full IIS in the *same* web role.

So, basically, the scenario looks like the following – where the endpoint of the web portal is an input endpoint which is available through the firewalls and the load balancer, but the endpoint of the services web site is an internal endpoint which is (and which should) only be reachable within our Azure application system.

Let’s take a look at a sample solution.
We have an Azure project, two web applications (for the portal and for the services) and a Contracts project which contains the WCF service and data contracts:

What we need to achieve is having two web sites configured in one web role our Azure project – one with the portal, one with the services. And we need two endpoints: one input (external) and one internal endpoint:

<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="WCFinWebRoleInternalEndpoints" 
 xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
  <WebRole name="WebPortal">
    <Sites>
      <Site name="Web">
        <Bindings>
          <Binding name="WebPortal" endpointName="WebPortal" />          
        </Bindings>
      </Site>
      <Site name="Services" physicalDirectory="..\BackendServices">
        <Bindings>
          <Binding name="Services" endpointName="Services" />
        </Bindings>
      </Site>
    </Sites>
    
    <Endpoints>
      <InputEndpoint name="WebPortal" protocol="http" port="80" />
      <InternalEndpoint name="Services" protocol="http" />
    </Endpoints>
    
    <Imports>
      <Import moduleName="Diagnostics" />
    </Imports>
  </WebRole>
</ServiceDefinition>

OK. Now our services web site runs inside our web role VM on a port that is not publicly available.
If you look at the internal endpoint definition you can see there is no port definition. This is because Windows Azure will assign a port dynamically.

Well, turns out this is a bit of a problem if our web portal wants to talk to the WCF services in that other (internal) web site. Which address, which port to use to talk to on the client/consumer side?

Of course there is help. The RoleEnvironment API from Windows Azure is here to heal our wounds.
What we need is the address and the port that Azure has assigned to our abstract internal endpoint (which we named ‘Services’ in the above .csdef). With this information we simply create a new endpoint URL and assign it to a WCF ChannelFactory: Here we have some generic helper method which can look at the endpoint of a ChannelFactory and adjust the address for an internal Windows Azure endpoint:

using System;
using System.ServiceModel;
using Microsoft.WindowsAzure.ServiceRuntime;

namespace Thinktecture.WindowsAzure
{
    public static class ChannelFactoryInternalEndpointExtensions
    {
        public static void AdjustEndpointAddress(this ChannelFactory channelFactory, 
          string internalEndpointName)
        {
            var serviceEndpoint = RoleEnvironment.CurrentRoleInstance
              .InstanceEndpoints
                [internalEndpointName].IPEndpoint;

            if (serviceEndpoint != null)
            {
                var serviceAddress = channelFactory.Endpoint.Address.Uri;
                var serviceUrl = new Uri(String.Format("{0}://{1}:{2}{3}",
                    serviceAddress.Scheme,
                    serviceEndpoint.Address,
                    serviceEndpoint.Port,
                    serviceAddress.AbsolutePath));
                channelFactory.Endpoint.Address = 
                  new EndpointAddress(serviceUrl);
            }
        }
    }
}

With this tiny extension method in place the actual code for calling the WCF services hosted in the internal web site looks like the following
(illustrated for both, ‘Add Service Reference…’ and a pure shared contracts with ChannelFactory scenario::

public partial class _Default : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {  
        // ASR... approach:
        var client = new EchoServiceClient();
        client.ChannelFactory.AdjustEndpointAddress("Services");

        var result = client.Echo("Hello Cloud");
        labelResult.Text = result;


        // ChannelFactory and shared contracts approach:
        var channelFactory = new ChannelFactory<IEchoService>("*");
        channelFactory.AdjustEndpointAddress("Services");
        var clientChannel = channelFactory.CreateChannel();

        result = clientChannel.Echo("Hello Cloud");
        labelResult.Text = result;
    }        
}

Voila. Nice architectural cloud approach made real.

Hope this helps.

Automating backup of a SQL Azure database to Azure Blob Storage with the help of PowerShell and Task Scheduler

OK, I am being a bit lazy on this one . I am not writing any custom PowerShell CmdLets here but use the powerful Azure Management CmdLets.

1. After you have downloaded the trial version of the CmdLets, make sure you have followed the setup instructions outlined in the “Azure Management Cmdlets - Getting Started Guide” document (which can be found in the installation folder of the CmdLets).
   
2. Start PowerShell (this has been tested with PS 2.0 which is installed by default on Windows 7 and Windows Server 2008 R2).
   
3. After having registered the CmdLets (again please see “Azure Management Cmdlets - Getting Started Guide”) you can run a backup script using the SQL Azure backup CmdLet (there are tons more in the package!).

Here is a sample script:

#########################################################################################
## Powershell script to backup a SQL Azure database by downloading its tables data to  ##
## local disk using "bcp" utility and uploading these data files to Azure Blob Storage.##
#########################################################################################
# Name of your SQL Azure database server. Please specify just the name of the database server.
# For example if your database server name is "abcdefgh", just specify that 
# and NOT "tcp:abcdefgh.database.windows.net" or "abcdefgh.database.windows.net"
$databaseServerName = "<your_sql_azure_server_name>";
# User name to connect to database server. Please specify just the user name.
# For example if your user name is "myusername", just specify that
# and NOT "myusername@myservername".
$userName = "<your_sql_azure_user_name>";
# Password to connect to database server.
$password = "<your_sql_azure_user_password>";
# Name of the database e.g. mydatabase.
$databaseName = "<your_sql_azure_database_name";
# Download location where the contents will be saved.
$downloadLocation = "c:\Temp\SQLAzure";
# Windows Azure Storage account name to use to store the backup files.
$storageAccount = "<your_windows_azure_storage_account_name>";
# Windows Azure Storage account primary key.
$storagekey = "<your_windows_azure_storage_account_key>";
# Blob container to use for storing the SQL Azure backup files.
$storageContainer = "sqlazurebackup";
Backup-Database -Name $databaseName -DownloadLocation $downloadLocation 
  -Server $databaseServerName -Username $userName -Password $password 
  -SaveToBlobStorage -AccountName $storageAccount -AccountKey $storageKey 
  -BlobContainerName $storageContainer -CompressBlobs

The above script first backups data in all tables to a local temporary folder and then uploads the data in compressed form into blob storage. When everything worked successfully you will see something like this:

Make sure to verify with a storage explorer tool that your backup files actually ended up in blob storage – e.g.:

Last but not least, you could create a recurring task via Windows task scheduler to periodically run your backup PS script (e.g. http://blog.pointbeyond.com/2010/04/23/run-powershell-script-using-windows-server-2008-task-scheduler/).
I am not showing the details here – should be pretty straight-forward.

Hope this helps.

Trying to troubleshoot Windows Azure Compute role startup issues

This one is actually borrowed from a Steve Marx blog post – I just wanted to explicitly set it in context with role startup issues and provide a small helper class. I guess we will have been struggling with finding out what exactly is going wrong during startup of our Azure role instances.

There are several possible reasons – and some of these issues you can actually see when trying to log exceptions happening the startup phase.

Try to wrap the code in your OnStart methods and catch any exception and immediately write it directly to blob storage.

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        try
        {
            // do your startup stuff here...
            return base.OnStart();
        }
        catch (Exception ex)
        {
            DiagnosticsHelper.WriteExceptionToBlobStorage(ex);
        }

        return false;
    }
}

And this is the little helper:

using System;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;

namespace Thinktecture.WindowsAzure
{
    public static class DiagnosticsHelper
    {
        public static void WriteExceptionToBlobStorage(Exception ex)
        {
            var storageAccount = CloudStorageAccount.Parse(
                RoleEnvironment.GetConfigurationSettingValue(
                  "Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
            
            var container = storageAccount.CreateCloudBlobClient()
              .GetContainerReference("exceptions");
            container.CreateIfNotExist();
            
            var blob = container.GetBlobReference(string.Format(
              "exception-{0}-{1}.log",
               RoleEnvironment.CurrentRoleInstance.Id,
               DateTime.UtcNow.Ticks));
            blob.UploadText(ex.ToString());
        }
    }
}

… which would then lead to your exceptions showing up in blob storage:

Hope this helps.

Querying Windows Azure Table Storage with LINQPad

Last week I wanted to quickly query and test data from one of my Windows Azure Table Storage tables. I did not want to use a storage tool as I wanted to have more fine-grained query control – and I sure did not want to write a custom program in Visual Studio.

What I ended up with is using the wonderful LINQPad to query Azure Table Storage. How does this work?

1. Add the necessary DLLs to LINQPad by opening Query Properties from the Query menu. Then add references to these assemblies:
   -System.Data.Services.Client.dll
   -Microsoft.WindowsAzure.StorageClient.dll
   -<all assemblies you need for querying table storage via your custom DataServiceContext />
   
   
   
2. Next we need to import the .NET namespaces we need in our code. In my case it looks like this:
   
   

That’s it.
Now we can use a C# program snippet like the following to successfully query our table(s):

void Main()
{
    var accountName = "…";    
    var sharedKey = "…";    
    var storageCredentials = new StorageCredentialsAccountAndKey(accountName, sharedKey);
    var storageAccount = new CloudStorageAccount(storageCredentials, true);
           
    var storageService = new MeldungenDataServiceContext(
        storageAccount.TableEndpoint.ToString(), 
        storageCredentials);
    
    var competitions =
        from competition in storageService.Wettkaempfe    
        select competition;            
    competitions.Dump();    
}

Just to prove it :

Nice.

Note: make sure to spend a few bucks on LINQPad to get full IntelliSense – this just rocks

Hope this helps.

Redirecting standard-out trace messages from custom processes to Windows Azure diagnostics manager

Recently I met with our friends from runtime and pusld to do a workshop on service-orientation and WCF – and we also chatted about some Windows Azure topics.

One of the discussion points was running a custom .exe server application in a Windows Azure worker role. Nothing new or spectacular so far.
The interesting tidbit was that this .exe already traced a number of useful data via standard-out. How could we direct this output to Windows Azure so that we are able to see it with any Azure storage or even diagnostics tool?

This is what the pusld guys came up with - here we go:

1. As usual, define a configuration setting for the storage connection string in the configuration file:
   

   <Setting name="DiagnosticsConnectionString" 
    value="DefaultEndpointsProtocol=https;AccountName=foobar;AccountKey=…" />

2. In the OnStart() method,  configure the default Azure logs to be copied to your storage account every 1 minute (in this case with Verbose log level):
   

   DiagnosticMonitorConfiguration diagnosticConfig = 
      DiagnosticMonitor.GetDefaultInitialConfiguration();
   diagnosticConfig.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
   diagnosticConfig.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
   DiagnosticMonitor.Start("DiagnosticsConnectionString", diagnosticConfig);

3. In the method where we start the executable tell the process info to redirect standard output and standard error:
   

   ProcessStartInfo info = new ProcessStartInfo()
   {
      FileName = targetexecutable,
      WorkingDirectory = targetPath,
      UseShellExecute = false,
      ErrorDialog = false,
      CreateNoWindow = true,
      RedirectStandardOutput = true,
      RedirectStandardError = true
   };

4. Add two event handlers which would log all the standard output and standard error to the Azure logs and begin to asynchronously read the streams after starting the process:
   

   Process runningProgram = new Process();
   runningProgram.StartInfo = info;
   runningProgram.OutputDataReceived += new DataReceivedEventHandler(
      RunningProgramOutputDataReceived);
   runningProgram.ErrorDataReceived += new DataReceivedEventHandler(
      RunningProgramErrorDataReceived);
   runningProgram.Start();
   runningProgram.BeginOutputReadLine();
   runningProgram.BeginErrorReadLine();

5. In the asynchronous handler methods just simply trace the output to the Azure logs, e.g.:
   

   private static void RunningProgramOutputDataReceived(object sender, 
      DataReceivedEventArgs e)
   {
       if (e.Data != null)
       {
           string output = e.Data;
           Trace.WriteLine("[…] Standard output –> " 
             + output, "Information");
       }
   }

6. Voila!
   Finally, use a diagnostics tool like Cerebrata’s Azure Diagnostics Manager just add the Azure storage account specified in your application above and choose to display the default Trace Logs:

 

Hope this helps.

thinktecture WSCF.blue has hit the 10,000 downloads landmark on CodePlex

After thinktecture StarterSTS has hit the 5,000 downloads mark a few weeks ago (congrats again, Dom!), we now have hit 10,000 for our WCF-based Web Services contract-first tool WSCF.blue.

[Want to learn more about it and the idea behind it…? Read this MSDN Magazine article]

Thanks to a great team!