Cross-platform unit testing in .NET, Mono and mobile platforms

Automated unit testing of .NET UI-less libraries in .NET is simple if all you need is to run a test suite on your build server. But when testing on multiple platforms such Xamarin.iOS, Xamarin.Android, .NET Compact Framework or Mono (on Linux or OS X), things get complicated. Fortunately, Rebex components can help with that.

Running tests locally

At Rebex, we use the popular NUnit testing framework. NUnit provides a console test runner (nunit-console.exe), so running tests locally (at your workstation or your build server) under normal .NET is just a matter of executing a single command:

nunit-console.exe Ftp.Test.dll /xml=Ftp.Test-results.xml /out=Ftp.Test-output.txt

Running tests remotely

But how to automatically trigger test runs in different environments? We quickly figured out we simply need a utility that would allow us to execute nunit-console.exe (or a compatible utility) on a remote machine, regardless of its operating system - it might be Windows, Linux or OS X server, or even a mobile device running Android, iOS or Windows CE with .NET Compact Framework.

Testing on remote Linux or OS X machines

Surprisingly, it turned out that the most remote-testing-friendly platforms are Linux and OS X with Mono - all we needed to do was to install Mono, NUnit and OpenSSH server, which made it possible to upload the test suites (and test data) using Rebex SFTP component and run them by executing nunit-console.exe over SSH with the help of Rebex Terminal Emulation component.

To make this solution nicely fit into our existing build system, we wrote a remote-runner.exe utility that accepts the same arguments as nunit-console.exe, but runs tests on a specified remote machine. It's highly configurable and supports a wide range of complex scenarios that cover all our testing needs on different platforms and environments.

Testing on remote Windows machines

But what about testing on platforms that don't come with an SSH/SFTP server, such as Microsoft Windows? Obviously, running tests at the build server is not sufficient - we need to test on different versions of Windows or .NET Framework, and also on machines where only FIPS-compliant cryptographic algorithms are enabled.

Installing a third-party SSH/SFTP server does the trick, but our C# SSH/SFTP server component comes in handy as well - just add a custom command called nunit-console that executes the actual NUnit test runner.

Testing on .NET Compact Framework

Yes, we still support platforms such as Windows CE, Windows Mobile or Pocket PC that run a lightweight variant of .NET called .NET Compact Framework. In the past, we used to run tests on these platforms manually and this process not only consumed lot of our time and effort, but it was prone to mistakes. It was clear that there must be a better way.

Fortunately, Rebex File Server supports .NET CF as well, which made it possible to apply the same solution that we were already using to run tests on remote Windows, Linux and OS X platforms. However, we had to make several changes - NUnit didn't support .NET CF and there was no nunit-console.exe utility. But since we were already using NUnitLite on this platform, adopting its test runner and adding a nunit-console.exe-like command line interface on top of it was rather simple. (This part of the solution would be even easier now - NUnit and NUnitLite have been merged.

And this is what our 'cluster' of Pocket PC / Windows Mobile emulators ready to run automated test suites look like in action:

Executing a set of unit tests remotely on any of these is just a matter of running remote-runner.exe with a suitable configuration file.

Testing on Xamarin.Android

From TCP component developer's point-of-view, Xamarin.Android is almost identical to Linux with Mono. With a bit of hackery, one could even install an SSH server. But we skipped that step because reusing the solution we developed for .NET CF turned out to be even easier - we just had to add a new GUI façade and modify the internals slightly to fit the new platform.

Only one issue remained unsolved - it turned out that Android emulator's networking support was very unstable. Fortunately, actual devices don't suffer from this issue, so the solution was obvious:

Testing on Xamarin.iOS

Unlike Xamarin.Android, it's iOS counterpart is a very different platform.

There is no Mono/.NET at runtime and .NET bytecode has to be statically compiled to native code before deployment. Reflection capabilities are severely limited and it's not possible to dynamically load a .NET assembly at runtime (Assembly.Load is not supported), which means the simple remote-runner approach we used on Windows, Windows CE, Linux, OS X and Android can't work. Although we could still run our SFTP/SSH server on an iPhone or iPad and upload all our tests suites and test data, there is no way to actually run them. Not without statically compiling them first.

But instead of trying to come up with a way to load statically compiled "assemblies", we chose a much simpler approach. Instead of connecting directly to the iOS device, we use remote-runner to connect to a Mac with iPad or iPhone attached and deploy the test suites and test data. Then, instead of running the tests, we generate a Xamarin Studio solution that references the test suites and test data, deploy it to the actual device and launch it. The test suite "app" immediately establishes a connection to the host device and reports its progress. Despite all this additional work, the whole process appears to be the same from the outside - the test runner still just uses remote-runner to execute a set of test suites.

What about Windows Phone?

We don't yet run automated tests on Windows Phone 8.1 or Windows Store 8/8.1/10 devices. This is going to be slightly more complicated as well - the limitations of these platforms resemble those of Xamarin.iOS, which means we would have to use a similar solution. Once we get to automating our tests on Microsoft's new platforms, we will let you know.