Using WAWS for Tenant Isolation

Introduction

In my second talk at Microsoft MVP Summit 2013 in Redmond/Bellevue I was speaking about a characteristic of Windows Azure Mobile Services (WAMS): The way that WAMS uses the infrastructure of Windows Azure Websites (WAWS) for tenant isolation. As people asked me to share the content of my talk, I summarize the most important take-aways in this article. Please note that all the code in this blog post is POC code. It should demonstrate the ideas and is by far not production ready (e.g. no error handling, no async programming, no good names, etc.).

Slides

Here are the slides that I have used during the talk. Only a few slides - I spent most of the time in Visual Studio showing/writing code:

The Problem

If you are a developer writing a mobile application (e.g. a Windows Store app) you might need to store some data in the cloud (e.g. settings, high scores, devices for push notifications, etc.). Windows Azure gives you multiple types of data stores you could use. The problem is that you might run into problems trying to access these data stores directly from your mobile app. Here are just some examples:

1. You cannot directly access a SQL database in Azure from your Windows Store app simply because ADO.NET is not available.
2. Table Storage, the NoSQL option for Azure, offers you a REST-based API. However, where should you store the credentials?

In each of these cases you will need to create, deploy, and maintain a service in the cloud. Your mobile app will use the service and the service accesses the underlying data store. The problem is that you might not want to be responsible for such a service. The solution that Microsoft offers for this problem is a ready-made service provided by Microsoft that all mobile app vendors can use. Such services with a shared infrastructure are called multi-tenant services.

Multi-tenant services like WAMS have a special problem when it comes to user-provided code (e.g. scripts): Running the code in the shared infrastructure is not possible because the risk of a script from tenant A doing bad things to tenant B is too high. There has to be some kind of sandbox in which non- or semi-trusted code can run. It turns out that the WAMS developers did not invent their own sandbox. They decided to use an existing one: The sandbox that comes with Windows Azure Websites (WAWS).

Windows Azure Websites as an Isolation Mechanism

Although the name Windows Azure Websites indicates that this service has been built only for websites. In fact WAWS is much more versatile. You can run nearly everything that you can run in IIS including Web APIs. This leads to the core question of my session: Can't we use WAWS in customizable, multi-tenant line-of-business (LOB) SaaS solutions for tenant isolation similar to WAMS does? We can.

For my session at MS MVP Summit I have created a small POC scenario. Here is the setup that I have shown:

1. Demo web application representing the website on which new customers can subscribe to the fictitious SaaS solution (of course this website also ran in WAWS).
2. A worker responsible for provisioning new tenants. In my demo it was a persisted VM role in Azure; in practice I would recommend an Azure Cloud Service.
3. A simple .NET-based API that can be used by power users for scripting. In my example I used the existing scripting technology in .NET (Dynamic Language Runtime, DLR) with a Python implementation built  on top of it (IronPython). We have been using IronPython in our own SaaS offering time cockpit for quite a long time and are very happy with it.
4. A sample full-client (WPF) representing e.g. a powerful ERP client for the SaaS offering.
5. A template implementation for a Web API that is capable of running user-provided scripts. My example uses ASP.NET's Web API.

The following diagram shows an overview how the different components play together. I will go into more details about what goes on in the sample.

Architecture Diagram (click to enlarge)

Code Walkthrough

When a user subscribes to our service, the website queues a tenant creation request using Azure's Service Bus. This service makes communicating between servers in the cloud a piece of cake. Just as an example, here is the code I used in the sample to enqueue a tenant creation request.

protected void LinkButton1_Click(object sender, EventArgs e)
{
    var tenant = string.Format("tenant{0}", DateTime.Now.Ticks);

    var queueClient = QueueClient.Create(
        "createtenant",
        ReceiveMode.ReceiveAndDelete);
    var bm = new BrokeredMessage();
    bm.Properties["Tenant"] = tenant;
    queueClient.Send(bm);
}

Here is the code for the worker waiting for tenant creation request. I share this code because it demonstrates how you can use e.g. WAWS's command line tools to automate the creation of websites in Azure.

class Program
{
    static void Main(string[] args)
    {
        // Create queue if it does not exist already
        var nsm = NamespaceManager.Create();
        if (!nsm.QueueExists("createtenant"))
        {
            var queue = nsm.CreateQueue(new QueueDescription("createtenant")
            {
                DefaultMessageTimeToLive = TimeSpan.FromHours(1.0),
                EnableBatchedOperations = false,
                EnableDeadLetteringOnMessageExpiration = false,
                IsAnonymousAccessible = false,
                MaxSizeInMegabytes = 1,
                MaxDeliveryCount = 2,
                RequiresDuplicateDetection = false,
                RequiresSession = false
            });
        }

        var queueClient = QueueClient.Create("createtenant", ReceiveMode.ReceiveAndDelete);
        while (true)
        {
            // Wait for a tenant creation message
            var msg = queueClient.Receive(TimeSpan.FromSeconds(60));
            if (msg != null)
                if (true)
                {
                    // Get tenant ID from message
                    var tenant = msg.Properties["Tenant"].ToString();

                    // Create the tenant's database by copying a template
                    CreateTenantDatabase(tenant);

                    // Create and configure a website using the command line tool for WAWS
                    RunProcessAsync("azure", string.Format("site create {0} --git --location \"West US\" < c:\\temp\\CanBeDeleted\\Empty.txt", tenant)).Wait();
                    RunProcessAsync("azure", string.Format("site config add tenant={0} {0}", tenant)).Wait();

                    // Copy the website template using git
                    RunProcessAsync("git", "init", @"C:\temp\CanBeDeleted\template").Wait();
                    RunProcessAsync("git", "pull https://myuser@erptenantisolationtemplate.scm.azurewebsites.net/ErpTenantIsolationTemplate.git", @"C:\temp\CanBeDeleted\template").Wait();
                    RunProcessAsync("git", string.Format("remote add azure https://myuser@{0}.scm.azurewebsites.net/{0}.git", tenant), @"C:\temp\CanBeDeleted\template").Wait();
                    RunProcessAsync("git", "push azure master", @"C:\temp\CanBeDeleted\template").Wait();

                    // Cleanup
                    RunProcessAsync("cmd", "/C rd . /s /q", @"C:\temp\CanBeDeleted\template").Wait();
                }
                else
                {
                    Trace.WriteLine("No tenant creation request, waiting...");
                }
        }
    }

    private static void CreateTenantDatabase(string tenant)
    {
        using (var conn = new SqlConnection(ConfigurationManager.ConnectionStrings["ErpDatabase"].ConnectionString))
        {
            conn.Open();
            using (var cmd = conn.CreateCommand())
            {
                // Create a transactionally consistant copy of the template database
                cmd.CommandText = string.Format("CREATE DATABASE {0} AS COPY OF northwind", tenant);
                cmd.ExecuteNonQuery();
            }
        }
    }

    /// <summary>
    /// Simple helper that executes a process async
    /// </summary>
    private static Task RunProcessAsync(string command, string commandline, string workingDirectory = null)
    {
        var result = new TaskCompletionSource<object>();
        var psi = new ProcessStartInfo(command, commandline);
        if (workingDirectory != null)
        {
            psi.WorkingDirectory = workingDirectory;
        }
        var p = Process.Start(psi);
        p.EnableRaisingEvents = true;
        p.Exited += (_, __) =>
        {
            result.SetResult(null);
        };
        return result.Task;
    }
}

Scripting

Adding script capabilities to a .NET program is quite simple. The .NET Framework contains the Dynamic Language Runtime (DLR). Multiple dynamic languages have been implemented on top of the DLR. Here are some examples:

• IronPython
• IronRuby
• IronJS

In my opinion IronPython is the most mature implementation of a dynamic language on the DLR. If you want to play with it, I encourage you to install the following components:

• IronPython
• Python Tools for Visual Studio (you will even get debugging capabilities for your scripts; Simon Opelt, a colleague of mine has written a blog article about it)
• Add the IronPython Nuget package to the .NET projects in which you want to use it

The following code snippet demonstrates how the isolated WAWS website loads a Python script from the tenant's database and executes it. Be aware that in practice you would have to invest some time in making this code more robust (e.g. collect console output from script, special handling of syntax errors, etc.). However, you do not have to care about sandboxing. WAWS will take care of that.

public class ScriptController : ApiController
{
    // GET api/<controller>/5
    public HttpResponseMessage Get(HttpRequestMessage request, string id)
    {
        // Use the ERP's API to connect to the tenant's datbase
        var context = new DataContext();
        context.Open(ConfigurationManager.ConnectionStrings["ErpDatabase"].ConnectionString);
            
        // Load the script from the tenant's database
        var scriptSource = context.GetScriptSource(id);
        if (scriptSource == null)
        {
            return new HttpResponseMessage(HttpStatusCode.NotFound);
        }

        try
        {
            // Execute the script and give it access the the ERP's API
            var engine = Python.CreateEngine();
            var scope = engine.CreateScope();
            scope.SetVariable("Context", context);
            var script = engine.CreateScriptSourceFromString(scriptSource);
            script.Execute(scope);

            return request.CreateResponse(HttpStatusCode.OK, "Success");
        }
        catch (Exception ex)
        {
            return request.CreateResponse(HttpStatusCode.InternalServerError, ex.Message);
        }
    }
}

Summary and Critical Evaluation

Technically it is possible to use WAWS as a tenant isolation mechanism for multi-tenant SaaS solutions. However, the solution does not scale for solutions with thousands and thousands of tenants because the number of websites per Azure subscription is limited today. In my experience this approach is feasible for SaaS offerings with a small to medium number of tenants.