Christian Weyer's Blog

Writing trace data to your beloved .svclog files in Windows Azure (aka ‘XmlWriterTraceListener in the cloud’)

Tracing is probably one of the most discussed topics in the Windows Azure world. Not because it is freaking cool – but because it can be very tedious and partly massively counter-intuitive.

One way of doing tracing is to use System.Diagnostics features like traces sources and trace listeners. This has been in place since .NET 2.0. Since .NET 3.0 and the rise of WCF (Windows Communication Foundation) there was also extensive usage of the XmlWriterTraceListener. We can see numberless occurrences of the typical .svclog file extension in many .NET projects around the world. And we can view these files with the SvcTraceViewer.exe tool from the Windows SDK.

All nice and well. But what about Windows Azure?
In Windows Azure there is a default trace listener called Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener from the Microsoft.WindowsAzure.Diagnostics assembly.

If you use this guy and want to trace data via trace sources, your data will be stored in Windows Azure Storage tables. Take some time to play around with it and find out that the data in there is close to useless and surely not very consumer-friendly (i.e. try to search for some particular text or error message. Horror).

So, taking these two facts I thought it would be helpful to have a custom trace listener which I can configure just through my config file which uses Azure local storage to store .svclog files. From there on I am using scheduled transfers (which I demonstrated here) to move the .svclog files (which are now custom error logs for Windows Azure) to Azure blob storage. From there I can just open them up with the tool of my choice.

Here is the simplified code:

using System.Configuration;
using System.Diagnostics;
using System.IO;
using Microsoft.WindowsAzure.ServiceRuntime;

namespace Thinktecture.Diagnostics.Azure
{
    public class LocalStorageXmlWriterTraceListener : XmlWriterTraceListener
    {        
        public LocalStorageXmlWriterTraceListener(string initializeData)
            : base(GetFileName(initializeData))
        {
        }

        public LocalStorageXmlWriterTraceListener(string initializeData, string name)
            : base(GetFileName(initializeData), name)
        {
        }

        private static string GetFileName(string initializationData)
        {
            try
            {
                var localResourceItems = initializationData.Split('\\');
                var localResourceFolder = localResourceItems[0];
                var localResourceFile = localResourceItems[1];

                var localResource = RoleEnvironment.GetLocalResource(localResourceFolder);

                var fileName = Path.Combine(localResource.RootPath, localResourceFile);

                return fileName;
            }
            catch
            {
                throw new ConfigurationErrorsException("No valid Windows Azure local 
                  resource name found in configuration.");
            }
        }
    }
}

In my Azure role (a worker in this particular case, but it also works with a web role) I configure the trace listener like this:

<configuration>
  <system.diagnostics>
    <trace autoflush="true">
      <listeners>
        <add type="Thinktecture.Diagnostics.Azure.LocalStorageXmlWriterTraceListener, 
               AzureXmlWriterTraceListener, Version=1.0.0.0,
               Culture=neutral, PublicKeyToken=null"
             name="AzureDiagnostics"
             initializeData="TraceFiles\worker_trace.svclog" />
      </listeners>
    </trace>
    </system.diagnostics>
</configuration>

After scheduling the transfer of my log files folder I can use a tool like Cerebrata’s Cloud Storage Studio to look at my configured blob container (named traces)– and I can see my .svclog file.

Double-clicking on the file in blob storage opens it up in Service Trace Viewer. From here on it is all the good ole’ tracing file inspection experience

Note: as you can see the Service Trace Viewer tool is not just for WCF – but you knew that before!

 

UPDATE: this does not properly work with Azure SDK 1.3 and Full IIS due to permission issues – there is more information in the SDK release notes. Very unfortunate

Hope this helps.

Transferring your custom trace log files in Windows Azure for remote inspection

You can write your trace data explicitly to files or use tracing facilities like .NET’s trace source and listener infrastructure (or third party frameworks like log4net or nlog or…). So, this is not really news

In your Windows Azure applications you can configure the diagnostics monitor to include special folders – which you obtain a reference to through a local resource in VM’s local storage - to its configuration whose files will then be transferred to the configured Azure Storage blob container.

Without any further ado:

public override bool OnStart()
{
    Trace.WriteLine("Entering OnStart...");

    var traceResource = RoleEnvironment.GetLocalResource("TraceFiles");
    var config = DiagnosticMonitor.GetDefaultInitialConfiguration();
    config.Directories.DataSources.Add(
        new DirectoryConfiguration
        {
            Path = traceResource.RootPath,
            Container = "traces",
            DirectoryQuotaInMB = 100
        });
    config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(10);

    DiagnosticMonitor.Start("DiagnosticsConnectionString", config);
         
    return base.OnStart();
}        

Note: remember that there are special naming conventions in Azure Storage, e.g. for naming blob storage containers. So, do not try to to use ‘Traces’ as the container in the above code!

And a side note: of course this whole process incurs costs. Costs for data storage in Azure Storage. Costs for transactions (i.e. calls) against Azure Storage and costs for transferring the data from Azure Storage out of the data center for remote inspection.

Alright – this is the base for the next blog post which shows how to use a well-known trace log citizen from System.Diagnostics land in the cloud.

Hope this helps (so far).

Monitoring Windows Azure applications with System Center Operations Manager (SCOM)

Windows Azure contains a few options to collect monitoring data at runtime, including event log data, performance counters, Azure logs, your custom logs, IIS logs etc. There are not really good ‘official’ monitoring tools by Microsoft – besides a management pack for System Center Operation Manager (SCOM).

To get started monitoring your Azure applications with an enterprise-style systems management solution, you need to do the following:

• Install SCOM. SCOM is a beast, but the good news is you can install all the components (including Active Directory and SQL Server) on a single server. Here is a very nice walkthrough – long and tedious, but very good.
• Download and import the Azure management pack (MP) for SCOM. Note that the MP is still RC at the time of this writing but Microsoft support treats it like an RTM version already.
• Follow the instructions in the guide from the download page on how to discover and start monitoring your Azure applications.
  

Voila. If everything worked then you will see something like this:

 

Note: This is a very ‘enterprise-y’ solution – I surely hope to see a more light-weight solution by Microsoft soon targeted at ISVs and the like.

Hope this helps.

Running a 32-bit IIS application in Windows Azure

Up there, everything is 64 bits. By design.

What to do if you have 32-bit machines locally – erm, sorry: on-premise – and want to move your existing applications, let’s say web applications, to Windows Azure?

For this scenario you need to enable 32-bit support in (full) IIS in your web role. The following is a startup task script that enables 32-bit applications in IIS settings by using appcmd.exe.

enable32bit.cmd:

%windir%\system32\inetsrv\appcmd set config -section:applicationPools
-applicationPoolDefaults.enable32BitAppOnWin64:true

And this is the necessary startup task defined in the service definition file:

ServiceDefinition.csdef:

<Startup>
   <Task commandLine="enable32bit.cmd" executionContext="elevated" taskType="simple" />
</Startup>

Hope this helps.

Windows Azure VM Role is still PaaS - if you want IaaS choose Amazon EC2, seriously

‘Nuff said?

Maybe we could blame Microsoft to name the new VM role feature (Windows Server 2008 R2 is supported as guest OS, 2011 should show broader OS support) introduced at PDC10 in a confusing way – but the fact remains:
it is based on the Azure service model & the Windows Azure Fabric Controller (FC) is still in charge of everything. Although you uploaded your own prepared VM the FC may and will decide to take your VM instances offline, start new instances, reprogram the load balancers etc.

Yes, Windows Azure Compute is about PaaS (Platform-as-a-Service), also with the VM role now in place. Don’t get confused by the new role feature name. Your applications (and thus roles!) need to be state-agnostic.

BTW: What is not possible with the VM role today is automatic OS updates/patching.
This means that you have no feasible option to have an up-to-date OS. When you try to run Windows Update this might work (actually it should). But then two things can happen:

• your VM needs to reboot due to the Windows Update patches
• FC decides to reboot your VM, or take it offline and hook up an new instance

In either case you end up with your original VM – bingo (and in the first case you will feel like in Groundhog Day… “Hey babe –dududu – I got you babe…”). Therefore, the official hands-on lab illustrates to disable WU entirely.

Think about the Windows Azure VM role, twice.

Sending JSON push notification messages to Urban Airship service with C#

A quick follow-up from a previous post on how to receive push notifications in iOS, e.g.through the Urban Airship cloud service: this time I will show you some snippets you can use to send out notifications from your code to Urban Airship which in turn then notifies subscribed devices/apps. This code can be running in a service (like in my app it is a WCF service) or you can use it anywhere else you see fit.

They have a really straight-forward JSON API for sending push messages.
This time I tried to model the required JSON payload with WCF’s data contract feature. Following is part of the supported JSON payload modeled in C# (mind the necessary body wrapper):

[DataContract(Name = "apsNotification")]
public class APSNotification
{
    [DataMember(Name = "aps")]
    public APSBody APS { get; set; }

    [DataContract(Name = "apsBody")]
    public class APSBody
    {
        [DataMember(Name = "badge")]
        public int Badge { get; set; }

        [DataMember(Name = "alert")]
        public string Alert { get; set; }

        [DataMember(Name = "sound")]
        public string Sound { get; set; }
    }

    [DataMember(Name = "aliases")]
    public List<string> Aliases { get; set; }

    public string ToJsonString()
    {
        var ms = new MemoryStream();
        var ser = new DataContractJsonSerializer(typeof(APSNotification));
        ser.WriteObject(ms, this);
        ms.Seek(0, SeekOrigin.Begin);
        var sr = new StreamReader(ms);
        var result = sr.ReadToEnd();
        ms.Close();
        ms.Dispose();

        return result;
    }
}

With the help of a little helper class called PushNotifications we can now send out push notifications (given that we have an account with Urban Airship and set up all necessary application-specific thing) – exception handling omitted:

public static class PushNotifications
{
    public static void SendNotificationMessage(
        string alertText, int badge, string sound,
        string alias, string username, string password)
    {
        var aps = new APSNotification
        {
            APS = new APSNotification.APSBody
            {
                Badge = badge,
                Alert = alertText,
                Sound = sound
            },
            Aliases = new List<string>
            {
                alias
            }
        };
        var json = aps.ToJsonString();

        var uri = new Uri("https://go.urbanairship.com/api/push/");
        var encoding = new UTF8Encoding();
        var request = (HttpWebRequest)WebRequest.Create(uri);
        request.Method = "POST";
        request.Credentials = new NetworkCredential(username, password);
        request.ContentType = "application/json";
        request.ContentLength = encoding.GetByteCount(json);

        using (var stream = request.GetRequestStream())
        {
            stream.Write(encoding.GetBytes(json), 0, encoding.GetByteCount(json));
            stream.Close();
            var response = request.GetResponse();
            response.Close();
        }
    }
}

Now, when anything ‘of interest’ happens in my WCF services I can notify my iOS users by using the above code.

Hope this helps.

No, I am not kidding: COA (Cost-Oriented Architecture)

You care about success in the Cloud? The embrace COA.

While architects (and partly developers) up-to-date did not (and did not have to) care about costs – when thinking about, using and moving to the Cloud you need to be aware of costs.

Yes, the architect becomes a controller (kind of).
People need to understand that and not blindly believe in ‘cool’ conference demos This is going to be interesting times…

Update: Or maybe we should coin it COCA (Cost-Oriented Cloud Architecture)

Flame away.

Sending emails from Windows Azure using Exchange Online web services (BPOS, for the search engines)

There already have been several blog posts (e.g. 1, 2, 3) about why and how to send emails from an Azure-hosted application.
I just wanted to summarize the essence and show some code on how to send email from Azure code via Exchange Online web services if you have an Exchange Online email subscription.

Turns out I was able to register a nice domain for my Exchange Online trial: windowsazure.emea.microsoftonline.com 

So, here is the essential code snippet to send an email via EWS (Exchange Web Services) by leveraging the EWS Managed API 1.1 (get the download here):

var service = new ExchangeService(ExchangeVersion.Exchange2007_SP1);
service.Url = new Uri(
  "https://red002.mail.emea.microsoftonline.com/ews/exchange.asmx");
service.Credentials = new WebCredentials(userName, password);

var message = new EmailMessage(service);
message.ToRecipients.Add("joe@doe.com");
message.From = new EmailAddress(
  "foobarbaz@windowsazure.emea.microsoftonline.com");
message.Subject = "Hello EMail - from Windows Azure";
message.Body = new MessageBody(BodyType.HTML, "Email from da cloud :)");
            
message.SendAndSaveCopy();

In the code above I am sending the email via the EWS host for Europe – you may need different URLs for your location:

Asia Pacific (APAC): https://red003.mail.apac.microsoftonline.com
Europe, the Middle East, and Africa (EMEA): https://red002.mail.emea.microsoftonline.com
North America: https://red001.mail.microsoftonline.com

Hope this helps.

Implementing Push Notifications for iOS with C# & MonoTouch using the Cloud & Urban Airship

A bit of mobile goodness today.

Communication-wise push notifications are one of my personal favorite features on any mobile device platform So, it seems natural to write a bit about it.
This time we will see how to use a third-party (but still free) service in the cloud to handle the push notifications for us (independent of Apple, Google, Blackberry or Microsoft). The sample shown here in this blog entry is for iOS and is written and built in C# with MonoTouch.

Although, the ideas of push notifications are similar between iOS; Windows Phone 7 and other device platforms, there are differences in implementation (and partly in architecture). What I wanted to achieve for my push notification-enabled apps was two-folded:

1. Being able to easily scale out and handle potentially huge numbers of user/subscribers and push messages without having to write, host and maintain my own code and services and servers
2. Not having to implement push notifications logic (like register, unregistering, receiving, error handling etc) for each and every platform over and over

This lead me to two solutions:

1. Use C# everywhere (oops, is this trademarked anywhere? ) with MonoTouch, MonoDroid and WP7 developer tools
2. Use a cloud service offered by an independent third party. In my case this is Urban Airship (free for 1,000,000 messages per month)

Rock da cloud.

Turns out that Urban Airship has a web-friendly REST API (oh wonder!) which accepts JSON messages. Honestly, it was not hard to decide to write some C# to utilize this service instead of having to deal with Apple’s APNs directly.

Note: the code illustrated here is a bit outdated. It relies on some iOS intrinsics like UIApplication and NSUserDefaults and therefore is not portable across devices. I have some platform-independent code lurking here but need to fully implement, test and polish it first.

The most important piece of code is the PushNotifications class which handles all the necessary communication, especially with the Urban Airship service:

public static class PushNotifications
{
     public static void Subscribe ()
     {
          UIApplication.SharedApplication.RegisterForRemoteNotificationTypes (
            UIRemoteNotificationType.Alert | UIRemoteNotificationType.Badge);
     }
     
     public static void Unsubscribe ()
     {
          UIApplication.SharedApplication.UnregisterForRemoteNotifications ();
     }
     
     public static void EnsureDeviceRegistration (NSData deviceToken)
     {
          var str = (NSString)Runtime.GetNSObject (
                Messaging.intptr_objc_msgSend (deviceToken.Handle, new Selector(
                  "description").Handle));
          var deviceTokenString = str.ToString ().Replace ("<", "").Replace (">", "")
            .Replace (" ", "");
     
          var token = LoadFromSettings ();
          
          if (String.IsNullOrEmpty (token) || token != deviceTokenString)
          {
                Console.WriteLine (String.Format ("EnsureDeviceRegistration: {0}, {1}", 
                  deviceTokenString, deviceTokenString.Length));
                
                var server = @"https://go.urbanairship.com";
                var url = String.Format("{0}{1}{2}", server, "/api/device_tokens/", 
                  deviceTokenString);
                
                Console.WriteLine ("EnsureDeviceRegistration: URL: " + url);
                     
                var putUrl = new Uri(url);
                
                var request = (HttpWebRequest)HttpWebRequest.Create(putUrl);
                request.Method = "PUT";
                request.ContentType= "application/json";
                request.Credentials = new NetworkCredential(Settings.APNsUser, 
                  Settings.APNsPassword);
                
                var payload = @"{""alias"": """ + Settings.DeviceAlias + @"""}";
                
                Console.WriteLine ("EnsureDeviceRegistration: Payload: " + payload);
                
                var byteArray = Encoding.UTF8.GetBytes(payload);
                request.ContentLength = byteArray.Length;
                var dataStream = request.GetRequestStream();
                dataStream.Write(byteArray, 0, byteArray.Length);
                dataStream.Close();
                
                var response = (HttpWebResponse)request.GetResponse();
                
                Console.WriteLine("EnsureDeviceRegistration: Status: " + 
                  response.StatusCode);
                     
                SaveToSettings (deviceTokenString);
          }
          else
          {
                Console.WriteLine("EnsureDeviceRegistration: device already registered.");
          }
     }
     
     private static string LoadFromSettings ()
     {
          var prefs = NSUserDefaults.StandardUserDefaults;
          
          return prefs.StringForKey("DeviceToken");
     }
     
     private static void SaveToSettings (string deviceTokenString)
     {
          var prefs = NSUserDefaults.StandardUserDefaults;
          prefs["DeviceToken"] = new NSString (deviceTokenString);
     }
}

Pretty simple – I did deliberately decide not to use any JSON API or serializer here.

Then, for my iOS apps I usually implement a couple of overrides in my AppDelegate class like this:

public override void RegisteredForRemoteNotifications (UIApplication application, 
  NSData deviceToken)
{
     Console.WriteLine ("RegisteredForRemoteNotifications");

     PushNotifications.EnsureDeviceRegistration (deviceToken);
}

public override void ReceivedRemoteNotification (UIApplication application, 
  NSDictionary userInfo)
{
     Console.WriteLine ("ReceivedRemoteNotification");
     
     var aps = userInfo.ObjectForKey (new NSString (
       CommunicationConstants.ApsRootElement)) as NSDictionary;
     var alert = aps.ObjectForKey (new NSString (
       CommunicationConstants.ApsAlertElement)).ToString ();
     
     Console.WriteLine ("Alert: " + alert);
     
     var sound = SystemSound.FromFile (new NSUrl (ResourcesConstants.ExplodeSound));
     sound.PlayAlertSound ();
     
     var av = new UIAlertView("Notification:", alert, null, "OK", null);
     av.Show ();
     
     (navigationController.VisibleViewController as EpisodesTableViewController)
       .LoadEpisodesData ();
}

public override void FailedToRegisterForRemoteNotifications (UIApplication application,
  NSError error)
{
    Console.WriteLine (String.Format ("FailedToRegisterForRemoteNotifications: 
      {0}, {1}, {2}", error.Domain, error.Code, error.LocalizedDescription));    
}

Then in my FinishedLaunching method (or where you decide to do it) I subscribe to push notifications:

PushNotifications.Subscribe();

That’s it – hope this helps someone. If you have more questions, please let me know.

P.S.: a big Kudos to the MonoTouch and MonoDroid teams and their immediate help with technical questions & issues!

Whitepaper: The ins and outs of data type interop between .NET/WCF and Java – add it to your [web] services tool belt

Recently, Buddhike and I wrote a whitepaper for MSDN which details the mapping of .NET/WCF and Java data types in a XML/XSD-ruled world.

As a developer it’s almost inevitable running into XML-based Web Services these days. Literally, they are everywhere. Web Services are often available as either REST style services or SOAP messaging style services built using various tools and running in different operating environments. Therefore, a number of standards have been defined over the years in order to ensure seamless interoperability between Web Services running in different operating environments. However, it is still somewhat challenging to develop interoperable Web Services because of how different platform vendors support industry standards. This whitepaper is written based on an interoperability analysis carried out between .NET 4 and three major Java based platforms - IBM Web Sphere, Oracle WebLogic and Oracle Metro (referred as Java client/server systems hereinafter).

Interoperable Web Services use universal type system - XML Schema Definition (XSD) constructs to define the data and message structures used for communication. However, the advanced tools and runtimes used for Web Service development usually abstract the details of XSD (and other protocols) from the developer. As a result developers can elaborate their Web Services using the programming language artifacts they are familiar with and leave the heavy lifting to the tooling/runtimes to translate those into the standard format. Even though some developers favor developing their type system using XSD (also known as schema first development), interoperability issues still arise due to limitations in various platforms for some XSD features. In this whitepaper, our analysis primarily focuses on the interoperability problems that occur while translating XSD artifacts to various platform level artifacts and vice versa.

[…]

Some illustrations used in paper are considered as bad practices for designing Web Service contracts. This paper acknowledges those shortcomings but provides those samples nevertheless to demonstrate the consequences.

Topics being covered:

• Primitive Types
  • .NET Server to Java Clients Mapping
  • Java Server to .NET Client Mapping
• Arrays and Abstract Data Types
• Date & Time
• Guid
• List
• Dictionary
• Queue
• Stack
• Tuple
• Binary Data
• Programming Language Constructs
  • Inheritance
  • Generics
  • Interfaces
    

Hope this article helps someone out there.