Yesterday we represented Microsoft and their Team Foundation Server in the Tools Challenge at the Software Quality Days 2014 conference in Vienna. The topic was Model Driven Development and Testing. The  great news: We won :-) In this article you can read more about what we have demonstrated during the challenge.

You can find more images from the Tools Challenge in my Flickr album.

The Challenge

The idea of the tools challenge is to bring multiple vendors of leading software quality tools on stage and let them present a solution for a specific software quality-related problem in front of the audience of the Software Quality Days conference. The vendors get the requirements for the presentation in the morning. They have the entire day to prepare a solution and a presentation. In the evening, every vendor has exactly seven minutes for the presentation - and this is a hard time limit with a loud bell ringing when the time is over. Afterwards a panel of domain specialists ask three questions and the presenter has exactly one minute for each answer.

After all presentations everyone in the audience has one vote for her favorite vendor. The vendor who gets the most votes is the winner of the Tools Challenge. Additionally, the jury is also electing a winner who gets the Jury Price of the conference.

In 2014 the topic was Model-driven Development and Testing. The participating vendors were:

1. IBM
2. imbus AG
3. Microsoft
4. Polarion
5. Ranorex
6. Tricentis

It was explicitly allowed to use 3rd party extensions or tools from partners.

We are a close partner of Microsoft and heavy user of their Team Foundation Server product when developing our SaaS time tracking solution time cockpit. Therefore Microsoft asked us to do the Tools Challenge on their behalf. Last year we also represented Microsoft at the 2013 Tools Challenge and we managed to reach the second place. Polarion has been the winner in 2013. This year we succeeded and reached the first place of the Tools Challenge. We want to congratulate all vendors for their great presentations - especially imbus AG who won the Jury Price.

Our Presentation

Team Foundation Server (TFS) is a great platform for application lifecycle management. It covers many different aspects like requirements management, test management, issue tracking, source code management, automated build, automated testing, etc. For the Tools Challenge we had to concentrate on model-based testing and development. The presentations of the different vendors have shown that this means a lot of different things to a lot of different people. We decided to show to scenarios at the show:

1. A top-down scenario that starts with a visual model from which we generate tests. The tests should be exported to TFS for further processing (test execution, test result management, reporting, etc.).
2. A bottom-up scenario where we start with C# source code and build a model from the source code from which we can automatically create test cases that point us to scenarios which would lead to errors.

Top-Down Scenario: Visual Model

TFS does not contain a module for visually building a model from which you can generate tests. However, TFS is extremely flexible and a large ecosystem of partners create TFS extensions. On of these partners is the German company sepp.med gmbh.

For the Tools Challenge we integrated the MBTsuite from sepp.med with Team Foundation Server. It added visual modelling and automatic generation of tests to the feature set of TFS.

For the demo we used a model that everyone is familiar with: The turn indicator system of a car. The MBTSuite can use Microsoft Office Visio for visual modelling. The following gallery contains some screenshot of the visual model we showed in the contest presentation.

Top level diagram
This top level diagram contains links to subdiagrams which contain the logic for directional blinking and hazard flashing. Directional Blinking Diagram
This diagram shows the functionality of directional blinking. It contains links to subdiagrams like left and right blinking. Left Blinking Diagram
This diagram contains the logic for left blinking. Note that the diagram also contains the requirements (e.g. Indicator on) linked to the corresponding actions in the model.

We imported the model from Visio into the MBTSuite. There we could choose from different strategies for test generation. During the tools challenge we only had seven minutes for the entire presentation. Therefore we prepared two guided paths through the model and generated two test scenarios for them. The following gallery contains some screenshots of the MBTSuite showing the imported model and the generation of tests.

Model in MBT
Here you see the imported model in the MBTSuite. Test Generation
MBT knows different strategies for test generation. Full Path Coverage would have been resulted in way to many test cases for the demo at Software Quality Days. Therefore we have chosen Guided Path. Guided Path Test Generation
For the demo we created guided paths through our model (workflow_1 to 4). We decided to generate tests for two of them.

After we generated the tests based on our model, we had to export them to TFS. To show the power of the TFS integration, we started with an entirely empty project in TFS. The MBTSuite recognized that it had never exchanged data with the TFS project before and created User Stories (based on the requirements in the model) and Test Plans. The following gallery contains some screenshots showing the export process.

Triggering TFS Export in MBT
Here you see how easy it is to trigger the TFS export in MBT. Export to TFS
MBT recognizes that the target project is empty and generates user stories and test cases. Result in TFS
Here you see the resulting user stories with the linked test cases in the TFS web interface.

Last but not least we showed how to run the generated tests in TFS's Testing Center. As we did not have a real car on stage during our presentation, we wrote a "Blinking Simulator" containing an intentional bug. The following gallery contains some screenshots showing the test in TFS Testing Center.

Generated Test Cases in TFS Testing Center
Here you see the generated test case with the associated test steps in TFS Testing Center. Run the test
During the presentation we decided to run the first test. Found a bug!
Here you see our "Blinking Simulator" which we had to write because we did not have a real car during the demo. But wait - it contains a bug! The test run failed and now we could use TFS functionality to create a bug entry and manage its lifecycle.

Bottom-Up Scenario: Code Contracts, PEX, and CodeDigger

We are passionate programmers and therefore we could not resist showing a more code-driven aspect of model-driven development in the Tools Challenge, too.

In our presentation we used Microsoft's PEX (automated white-box unit testing tool) and Microsoft's Code Digger (Visual Studio add-in using PEX in the background) to let Visual Studio generate a model from our source code and suggest test cases.

First we introduced PEX and CodeDigger with a very simple procedure (click to enlarge):

In this simple scenario Code Digger comes up with three different tests. Two of them result in an error (click to enlarge):

Next we wanted to demo the power of PEX and Code Digger in a slightly more complex scenario. Here is the code we used (don't analyze the logic, it does not have any deeper meening):

public class Bill
{
	public Bill(Region outsideEU, decimal amount, string supplier)
	{
		this.Region = outsideEU;
		this.Amount = amount;
		this.Supplier = supplier;
	}

	public Region Region { get; private set; }

	public decimal Amount { get; private set; }

	public string Supplier { get; private set; }

	public bool ValidateSupplier()
	{
		Contract.Ensures(Contract.OldValue(this.Region) == this.Region);
		Contract.Ensures(Contract.OldValue(this.Amount) == this.Amount);
		Contract.Ensures(Contract.OldValue(this.Supplier) == this.Supplier);

		if (this.Region == Region.ASIA)
		{
			// internal bills are always ok.
			if (this.Supplier == "INTERNAL")
			{
				return true;
			}

			if (this.Amount > 150.00m)
			{
				// supplier required if amount > 150 outside of EU
				if (this.Supplier.Length == 0)
				{
					// if no supplier is available ->
					return false;
				}
			}

			return true;
		}
		else if (this.Region == Region.US)
		{
			this.Amount = -this.Amount;
			return true;
		}
		else if (this.Region == Region.EU && this.Supplier.Length > 0)
		{
			return true;
		}

		return false;
	}
}

Trying to come up with interesting test cases for the ValidateSupplier method would not be super easy. We let PEX do the math (click on image to enlarge):

You can not only run PEX through CodeDigger in Visual Studio. You can also use it as a stand-alone tool. You will get an HTML result showing things like code coverage, test cases (including code to copy into your automated unit test projects), etc. Here are some screenshots of the PEX report for our demo scenario:

Pex Report with Test Cases
This PEX report shows the suggested test cases with their results. Note that the last one fails because of a code contract. PEX Report with Test Cases
Here you see the details for some of the test cases. You get the code that you can copy into your unit test project to further investigate potential bugs.
comments powered by Disqus