RESTful Reporting with Visual Studio Online

image

My team uses Visual Studio Online for work item tracking and generally speaking it has pretty good baked-in reporting.  I can see an overview of the current sprint, I can see capacity and I can see the burndown.  One area that I’ve always felt it was missing, however, is a way to analyse the accuracy of our estimations.

We actually make pretty good estimations, in general terms: we rarely over-commit and it’s unusual for us to add anything significant to a sprint because we’ve flown through our original stories.  This is based on a completely subjective guess at each person’s capacity and productivity which – over time – has given us a good overall figure that we know works for us.

But is that because our estimates are good, or because our bad estimates are fortuitously averaging out?  Does our subjective capacity figure still work when we take some people out of the team and replace them with others?

This is an area where the reporting within VSO falls down and the limitation boils down to one issue: there is no way to (easily) get the original estimate for a task once you start changing the remaining work.  So how can we get at this information?

Enter the API

I had seen a few articles on the integration options available for VSO but hadn’t really had a chance to look into it in detail until recently.  The API is pretty extensive and you can run pretty much any query through the API that you can access through the UI, along with a bunch of useful team-related info.  Unfortunately the API suffers the same limitation as the VSO portal, but we can work around it using a combination of a little effort and the Work Item History API.

Getting the Data

There is nothing particularly complicated about pulling the relevant data from VSO:

  1. Get a list of sprints using the ClassificationNode API to access iterations
  2. Use Work Item Query Language to build a dynamic query and get the results through the Query API.  This gives us the IDs of each Task in the sprint
  3. For each Task, use the Work Item History API to get a list of all updates
  4. Use the update history to build up a picture of the initial state of each task

Point 4 has a few caveats, however.  The history API only records the fields that have changed in each revision so we don’t always get a complete picture of the Task from a single update.  There are a few scenarios that need to be handled:

  1. Task is created in the target sprint and has a time estimate assigned at the same time.  This is then reduced during the sprint as the Task moves towards completion
  2. Task is created in the target sprint but a time estimate is assigned at a later date before having time reduced as the sprint progresses
  3. Task is created in another sprint or iteration with a time assigned, then moved to the target sprint at a later date
  4. Task is created and worked on in another sprint, then is moved to the target sprint having been partially completed

The simplest scenario (#1 above) would theoretically mean that we could take the earliest update record with the correct sprint.  However, scenario 2 means that the first record in the correct sprint would have a time estimate of zero.  Worse, because we only get changes from the API we wouldn’t have the correct sprint ID on the same revision as the new estimate: it wouldn’t have changed!

The issue with scenario 3 is similar to #2: when the Task is moved to the target sprint the time estimate isn’t changed so isn’t included in the revision.

A simplistic solution that I initially tried was to simply take the maximum historical time estimate for the task (with the assumption that time goes down as the sprint progresses, not up).  Scenario 4 puts an end to this plan as the maximum time estimate could potentially be outside of the current sprint.  If I move a task into a sprint with only half it’s work remaining, I don’t really want to see the other half as being completed in this sprint.

Calculating the Original Estimate: Solution

The solution that I eventually went with here was to iterate through every historical change to the work item and store the “current” sprint and remaining work as each change was made.  That allows us to get the amount of remaining work at each update alongside the sprint in which it occurred; from this point, taking a maximum of the remaining work values gives us a good number for the original amount of work that we estimated.

It does rely on the assumption that Tasks estimations aren’t increased after they have started work (e.g. start at 2 hours, get 1 hour done then realise there’s more work so increase back to 2) but in this scenario we tend to create new tasks instead of adjusting existing ones (we did find more work, after all) which works for us.

Tying it all Together

Once I was able to get at the data it was relatively simple to wrap a reporting service around the implementation.  I went with node & express for the server-side implementation with a sprinkling of angular on top for the client, but visualising the data wasn’t the challenge here!

With this data available I can see a clear breakdown of how different developers affect the overall productivity of the team and can make decisions off the back of this.  I have also seen that having a live dashboard displaying some of the key metrics acts as a bit of a motivator for the people who aren’t getting through the work they expect to, which can’t be a bad thing.

I currently have the following information displayed:

  • Total remaining, completed and in-progress work based on our initial estimates
  • %age completion of the work
  • Absolute leaderboard; i.e. who is getting through the most work based on the estimates
  • Adjusted leaderboard; i.e. who is getting through the most work compared to our existing subjective estimates
  • Current tasks

I hope that the VSO API eventually reaches a point that I can pull this information out without needing to write code, but it’s good to know I can get the data if I need it!

Advertisements

Keep IIS Express Running in Visual Studio 2013

Since upgrading to Visual Studio 2013 I’ve noticed a change in behaviour in IIS.  It still starts up when you start debugging a web project – same as it always has – but since the upgrade it automatically shuts down when you stop debugging.

As with most IDE behaviour, I was pretty familiar with the old way and so I found it incredibly frustrating whenever this happened.  The good news is that there’s a very simple solution: disable Edit and Continue for the project in the Properties dialog.

edit continue

Hopefully this will save someone else some pain!

Explicitly Controlling Cache Dependencies in MVC

In the past, I have always used the OutputCacheAttribute  when I wanted to cache the result of an Action in my MVC application; it’s simple, and it gets basic caching up and running with very little effort.

Unfortunately “simple and quick” are not quite as useful when you need a little more control…

Scenario

Let’s say you have a resource that you automatically generate in your controller, but that creating that resource takes a long time…

public async Task<ActionResult> Slow()
{
	var resource = await _service.GenerateResource();
	// ...5 minutes later...
	return View(resource);
}

Understandably, you want to cache the result of this action so that subsequent visitors don’t all have to wait around for the resource to be created.  [OutputCache] to the rescue, right?

Not So Fast!

Unfortuantely, the resource is not entirely constant – it can be altered by certain user actions in other areas of the application, and when that happens it will need to be regenerated.

[OutputCache] will allow us to vary the cached result by a number of variables (host, action arguments, custom strings…) and supports timeouts etc, but it does not allow another piece of code somewhere in the application to say “that resource is not longer valid”.

An Alternative to [OutputCache]

One alternative means of caching the results of an action is to call the AddCacheDependency method on the Response object:

public async Task<ActionResult> Index()
{
	Response.AddCacheDependency(new CacheDependency(...));
			
	//...
}

The CacheDependency instance in this example is an ASP.NET class that is used by the caching framework to determine when the resource we created has been changed.

The base CacheDependency implementation allows you to specify one or more file system paths that will be monitored, and will invalidate itself when any of those files is updated.  There is also a SqlCacheDependency class that observes the results of a SQL query and invalidates when they change.

Neither of these implementations will give us quite what we are looking for –  the ability to invoke a method from anywhere within the codebase that explicitly marks the cached content as changed – but they are leading us in the right direction.

If we can create a CacheDependency instance that is uniquely identifiable for the slow resource then we can add it to the response and force it to invalidate itself at a later date.

Extending CacheDependency

Before we can get into how we add our CacheDependency instance, we need to create an implementation that will allow us to explicitly invalidate it through code.  Thankfully, CacheDependency exposes a number of methods to it’s inheriting classes that mean we can achieve our explicit invalidate very easily.

The minimum that we need to do to make the CacheDependency work is to pass a list of paths into the base constructor and to provide a valid value from the GetUniqueID method.  We know that we do not want to depend on any file system resources so we can just pass an empty list into the constructor, and as we need a unique ID anyway (to identify the cached resource later) we can just pass this into the constructor.

class ExplicitCacheDependency : CacheDependency
{
	private string _uniqueId;

	public ExplicitCacheDependency(string uniqueId)
		: base(new string[0]) //no file system dependencies
	{
		_uniqueId = uniqueId;
	}
		
	public override string GetUniqueID()
	{
		return _uniqueId;
	}
}

CacheDependency has a protected NotifyDependencyChanged method that will notify the caching framework that the cached item is no longer valid. In most implementations this would be invoked in some callback, but for our purposes we can just add a new Invalidate method and invoke it directly:

public void Invalidate()
{
	base.NotifyDependencyChanged(this, EventArgs.Empty);
}

Voila – a cache dependency that we can explicitly invalidate.  But how can we get a reference to this in the various places that we need it?

CacheDependencyManager

Creating a new cache dependency doesn’t help us much if we can’t get a reference to it later – otherwise, how can we call Invalidate?  Let’s create a new class that handles the creation and invalidation of the new cache dependencies: CacheDependencyManager.

class CacheDependencyManager
{
	private Dictionary<string, ExplicitCacheDependency> _dependencies
		= new Dictionary<string, ExplicitCacheDependency>();

	public CacheDependency GetCacheDependency(string key)
	{
		if (!_dependencies.ContainsKey(key))
			_dependencies.Add(key, new ExplicitCacheDependency(key));

		return _dependencies[key];
	}

	public void InvalidateDependency(string key)
	{
		if (_dependencies.ContainsKey(key))
		{
			var dependency = _dependencies[key];
			dependency.Invalidate();
			dependency.Dispose();
			_dependencies.Remove(key);
		}
	}
}

Note: in the interests of brevity I have not included the thread-safe version here; this is a terrible idea in the real world, so make sure you include some appropriate locking!

This CacheDependencyManager is intended to hide the detail of how the dependency instances are created and invalidated from calling classes, which it achieves through 2 public methods:

  • GetCacheDependency that creates a new ExplicitCacheDependency if none exists for the specified key, or returns the cached one if it has been previously created
  • InvalidateDependency that attempts to locate an existing cache dependency for the specified key, and invalidates and removes it if one is found.  If one doesn’t exist then it does nothing, so callers don’t need to know whether or not the resource has already been cached

One quick singleton pattern later and we can call these methods from throughout our codebase. When we invoke our slow action for the first time we need to add the dependency to the response using a unique key to identify the slow resource:

Response.AddCacheDependency(
		CacheDependencyManager.Instance.GetCacheDependency("ResourceKey"));

And how do we invalidate the resource after some arbitrary user action? We can just pass that same key into the Invalidate:

public class SomeOtherController : Controller
{
	public ActionResult SomeRandomUserAction()
	{
		CacheDependencyManager.Instance.Invalidate("ResourceKey")
		//...
	}
}

Obviously you could (should!) use dependency injection to pass the CacheDependencyManager to your controllers instead of accessing it directly, but the singleton will suffice for this example.

That’s All Folks

That’s the lot – now we can manually invalidate our slow resource whenever and from wherever we want, and our users can enjoy speedy cached resources the rest of the time!