Fixing Windows Azure SDK 1.3 Full IIS diagnostics and tracing bug with a startup task & a grain of salt

We have been doing quite a few Azure consulting gigs lately. All in all Windows Azure looks quite OK with the SDK 1.3 and the features available.

But there is one big issue with web roles and Full IIS since SDK 1.3. I have hinted at it and my friend and colleague Dominick also blogged about the problem. The diagnostics topic seems like a never-ending story in Azure, but it got even worse with the bug introduced in SDK 1.3 – quoting the SDK release notes:

Some IIS 7.0 logs not collected due to permissions issues - The IIS 7.0 logs that are collected by default for web roles are not collected under the following conditions:

  • When using Hostable Web Core (HWC) on instances running guest operating system family 2 (which includes operating systems that are compatible with Windows Server 2008 R2).
  • When using full IIS on instances running any guest operating system (which is the default for applications developed with SDK 1.3).
The root cause to both of these issues is the permissions on the log files. if you need to access these logs, you can work around this issue by doing one of the following:
  • To read the files yourself, log on to the instance with a remote desktop connection. For more information, see Setting Up a Remote Desktop Connection for a Role.
  • To access the files programmatically, create a startup task with elevated privileges that manually copies the logs to a location that the diagnostic monitor can read.

Wonderful.

But I really needed this to work, to get my log files written in my web roles automatically collected by the diagnostics monitor.
Well, startup tasks to the rescue. I decided to change the ACLs on the said folder from the Azure local resources via a PowerShell script.
The script will be spawned through a .cmd file like this:

powershell -ExecutionPolicy Unrestricted .\FixDiagFolderAccess.ps1

 

And here is the beginning of PowerShell code. It tries to register the Windows Azure-specific CommandLets:

Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime
$localresource = Get-LocalResource "TraceFiles"
… some more stuff omitted …

I tried it locally and it seemed to work, great! Deployed it to the Cloud… didn’t work… Sad smile Then I got in touch with Steve Marx. He played around a bit and found out that there is some kind of timing issue with registering the CommandLets… argh…

So, with that knowledge the working PS script now looks like this:

Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime

while (!$?)
{
    echo "Failed, retrying after five seconds..."
    sleep 5

    Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime
}

echo "Added WA snapin."

$localresource = Get-LocalResource "TraceFiles"
$folder = $localresource.RootPath

$acl = Get-Acl $folder

$rule1 = New-Object System.Security.AccessControl.FileSystemAccessRule(
"Administrators", "FullControl", "ContainerInherit, ObjectInherit",
"None", "Allow") $rule2 = New-Object System.Security.AccessControl.FileSystemAccessRule(
"Everyone", "FullControl", "ContainerInherit, ObjectInherit",
"None", "Allow") $acl.AddAccessRule($rule1) $acl.AddAccessRule($rule2) Set-Acl $folder $acl echo "Done changing ACLs."

 

Yes, I do realize that I am adding Everyone with full access – mea culpa. I need to further investigate to actually cut down the ACLs. And let’s hope that Microsoft fixes this annoying bug quickly with a new SDK release.

So, how do we get things working properly now?
Another new feature since SDK 1.3 and with Full IIIS is that we now have two processes in the game: the w3wp.exe for your web application and the WaIISHost.exe for the RoleEntryPoint-based code. For this reason if you want to trace also in your WebRole.cs you need to register stuff in OnStart:

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        base.OnStart();

        Trace.WriteLine("Entering OnStart in WebRole...");

        var traceResource = RoleEnvironment.GetLocalResource("TraceFiles");
        var config = DiagnosticMonitor.GetDefaultInitialConfiguration();
        config.Directories.DataSources.Add(
            new DirectoryConfiguration
            {
                Path = traceResource.RootPath,
                Container = "traces",
                DirectoryQuotaInMB = 100
            });
        config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);

        DiagnosticMonitor.Start(
"Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString",
config); return true; } }

 

In addition, let’s use my custom trace listener I demonstrated earlier. This goes into the WaIISHost.exe.config file for being picked up for tracing in your WebRole.cs (which will be compiled into WebRole.dll):

<configuration>
  <system.diagnostics>
    <trace autoflush="true">
      <listeners>
        <add type="Thinktecture.Diagnostics.Azure.LocalStorageXmlWriterTraceListener,
AzureXmlWriterTraceListener, Version=1.0.0.0, Culture=neutral,
PublicKeyToken=null"
         name="AzureDiagnostics" 
         initializeData="TraceFiles\web_role_trace.svclog" />
      </listeners>
    </trace>
  </system.diagnostics>
</configuration>

 

Last, but not least also configure tracing for your web application, most likely in global.asax:

public class Global : System.Web.HttpApplication
{
    void Application_Start(object sender, EventArgs e)
    {
        Trace.WriteLine("Application_Start...");

        var traceResource = RoleEnvironment.GetLocalResource("TraceFiles");
        var config = DiagnosticMonitor.GetDefaultInitialConfiguration();
        config.Directories.DataSources.Add(
            new DirectoryConfiguration
            {
                Path = traceResource.RootPath,
                Container = "traces",
                DirectoryQuotaInMB = 100
            });
        config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
    }
} 

 

And the appropriate entries in web.config:

<configuration>
  <system.diagnostics>
    <trace autoflush="true">
      <listeners>
        <add type="Thinktecture.Diagnostics.Azure.LocalStorageXmlWriterTraceListener,
AzureXmlWriterTraceListener, Version=1.0.0.0, Culture=neutral,
PublicKeyToken=null"

name="AzureDiagnostics"
initializeData="TraceFiles\web_trace.svclog" />
      </listeners>
    </trace>
  </system.diagnostics>

 

With all that set up I can see my trace (.svclog) files – after the configured transfer period - now in my blob storage. Hooray!

image

 

FYI, this is how my sample solution looks like in VS 2010. With the .cmd and .ps1 files in place (and make sure they are set to Copy always !):

image

Oh yeah: it finally also works as expected in the real Cloud Smile

Note: make sure to read through Steve’s tips and tricks for using startup tasks.

 

Hope this helps.