Browse Tag: unit testing

Unit Testing Large And Complex Data

This week I’ve been helping some developers with unit testing adoption which raise an interesting topic that I’ve not seen explicitly addressed.

Unit testing is great when you are working with simple data but what if you have larger or more complex data such as waveforms or images?

I’ve used a couple of techniques over the years:

Simplify

Maybe this is an obvious one – but the first option is to identify whether the method can still be applicable on a subset of the real data.

For example, on an application where I’m doing some image processing, the real data will be 256×256 pixels

However, my tests run over 3×3.

This is still applicable to the larger image as many algorithms involve handling edge conditions and the normal implementation in the middle. The larger arrays will have more of the normal implementation but the smaller tests will still cover the edge conditions as well (which is often where things can go wrong!).

Generate

In some cases, you need a full input set but you are just measuring some properties.

An example might be a frequency processing function where we want to extract the size of the peak at 1kHz.

The FFT parameters change a lot based on the size of the data you put in so really we want to use the actual size we expect in the application. So instead what I have done in the past is to write a basic routine to generate an appropriate signal.

In the example above I use generators to produce a multitone signal, perform the frequency analysis and manipulation which I am testing and then compare just the components of interest.

(Note: this is before I got a bit more structured with my testing!)

Credit to Piotr Demski at Sparkflow for pointing out an important point I missed when I first published this. If you are generating data – it should be the same every time i.e. beware random data sources. Your tests may fail without anything changing.

Golden Data

The approaches above may not work if it isn’t obvious how to generate the data or you can’t generate the important elements easy. They also only work where the problem is importing data – but what if you need to compare a large result.

Here I will revert to storing a constant of some reference data. Normally by running the system, using a probe on the data and copying it to my test.

Quick Demo of Golden Data Process

On the input, this can work without any compromise.

For expected cases there is an obvious catch – if you generate data from the VI under test it will of course pass. However, if you have no way of generating an expected output then we have to compromise.

Instead, we can write the algorithm until it works (validated manually) and capture that data for the test. It means the test doesn’t guarantee it is right, however, it will fail if you do anything that alters the output, which gives you a chance to catch potential regressions.

Another path is to use a known good algorithm (perhaps from another language) to capture the expected output in a similar way. This is fantastic when you are looking to replace or replicate legacy systems since you can take data directly from them.

Catch It At Different Levels

In some cases, it may simply not be viable to create a unit test for the algorithm. That’s OK – 100% coverage isn’t a useful goal in most cases anyway. Instead, consider:

  • Try and unit test the components inside the algorithm (depending on the structure of your code). This will provide some confidence.
  • Make sure this algorithm is verified in system or integration level tests. Ideally, find ways to make it fail loudly so it isn’t likely to be missed.

I hope that gives you some ideas for your use case. Feel free to post any questions you have.

Testing Events In VI Tester

The APIs that you have to test are not always simple. As well as passing data they may involve events (with the front panel or with user events).

The other day I needed to test that an event fired as part of a test case. I could see a generic solution, so I created a template for it. I had two requirements:

  1. If the event doesn’t fire – test fails.
  2. If the event fires with the wrong data – test fails.

In my given when then sequence then we end up with a test that follows the structure:

  • Given: Who knows, in this case, a UI library has been tied to a control.
  • When: We take some action that should cause an event on that control.
  • Then: Check the event.

To check the event we create an event structure outside of a loop as we don’t want to handle multiple events. We need two cases:

  1. A timeout case with a suitable timeout – In this case, we call the Test Case.lvclass:fail.vi to fail the test. This should never run if the when code fired the event.

    Failing Path
  2. A case that handles the event – If you don’t care about the data then you can do nothing here, otherwise, include tests on the data included in the event.

    Passing Path

 

Additional Complexity

  1. Dynamic Event Registration: If this is a user event then you will need to register for the event. I’ve included this in my template, but you must move the event registration to the given case. If you haven’t registered the event before the action in the when case, it won’t ever fire.
  2. Parallel/Dynamic Event Generation: If your event is in some dynamic code you may need to have this running. My advice: DONT. Try and pull out the internal API and test synchronously. Asynchronous testing in LabVIEW introduces timing concerns which make your tests much more complicated.

Where To Get It

If you want to use this template, or even if you are just using VI tester you can download the new version of the VI Tester Advanced Comparisons (VITAC) tool from https://github.com/WiresmithTech/VITAC/releases/tag/v1.1.0.

 

Given, When, Then In LabVIEW Tests

A few months ago in the Austrian alps I was skiing and attempting my second ever slalom run on my trip. Those of you that I have seen recently will now it didn’t end well!

I gathered too much speed, caught some ice and tore my ACL.

Since then I have to do some exercises each morning to prepare it for surgery. I tend to try and watch some interesting talks while I do this and make use of my “bonus” time. They are often TED talks but I had watched many of the latest ones so, super geeky, I switched to Go To Conference talks which are software talks – mainly based around web technologies.

I find it interesting to watch some of the talks that skirt the edge of the technical and understand how they can be applied to LabVIEW. This was certainly true of Level Up Your Unit Tests.

Descriptive Tests

The talk is somewhat the story of a developers transition to a new testing tool and there is one piece that really appealed to me.

There is a concept I have come across before for a structure for acceptance tests called Given, When, Then. The idea is it clearly describes every aspect of a test situation:

  • Given: The pre-conditions.
  • When: The trigger or action.
  • Then: What the software/system should do in response.

For example:

Given we have a high temperature alarm, when the user clears the alarm then an alarm should no longer show as active.

In the video Trisha Gee describes a test framework that was new to her that actually describes the tests in this structure which greatly helps with clarity and highlights problems with the code if any section gets too large. Ideally:

  • Given is small. If it starts to get quite big this starts to sound more like an integration test and less of a unit test. It should also contain no tests – this is not the subject of the unit test.
  • When is tiny. This should ONLY be the code you are actually testing.
  • Then is tiny. This contains your actual tests and assertions. Given a unit test should test one thing there should only be one assertion here or multiple tightly related assertions.

Why Looks Matter

What struck me was that my unit tests – while quite effective – are a mess compared to my normal code. I tend to rattle through then and not give them the full attention that they need. This can hurt me when I have to return to them to understand why they fail.

So I have experimented by taking the descriptive structure of the framework that Trisha describes in the video and implementing it in LabVIEW. The idea is we want clearly separate sections for these with defined boundaries so I found flat sequence structures work well.

Let me give you a (kind of) before and after.

Before:

old-test-style

This is the old style. It works, well. However just looking at the code there are test cases spread throughout (5 in total) and it isn’t clear from the code alone what is being tested.

After:

given-when-then-test

(Yes this is a different test, I haven’t rewritten all of my tests to this format)

Here it is much clearer what is just setup code, what is the code under test and then what the conditions are that we really care about.

It also makes it really obvious if I had tests that were really just checking that the setup has worked which is what some of the tests in the before case are doing (Sometimes this can be really useful though, I think the answer is though that this code should have been tested somewhere else – but I need to think this through more)

Now I really am running out of things to say on unit testing! I have a few more OO posts in the pipeline as well as a couple of tips & tricks posts that I hope to do this year. It has been very busy the past couple of months but I will be having some time off over the summer while they reconstruct my knee! So expect a few more posts then.

 

UPDATE:

Thanks to the commenters below I wanted to include a couple of links.

  1. This template is now built into the VITAC toolkit at https://github.com/WiresmithTech/VITAC/releases (thanks for the prompt Fab!)
  2. Ivan’s tip below for adding this to the default class is great and I have this on all my systems now – http://kosist.org/2018/08/modify-default-vi-tester-testcase-template/

Spies In LabVIEW

In my last post I covered how I am using unit testing in LabVIEW. In this post I want to share a useful tool on that journey.

Test Doubles

One of the first rules I operate under while testing is using as much of the real application as possible in testing a behaviour. Then each test becomes more valuable as it covers the integration of all the units as well as their contents.

Sometimes however this simply isn’t possible. You may need to talk to hardware that isn’t available, or a database or file that is hard to verify as part of the test (It’s normally possible somehow, but perhaps not worth it). This is where test doubles come in.

A test double is simply a substitute for a software component that you want to exclude from the test. In my last post I mentioned that OO make testing easier, this is why. If these components are written to an abstracted interface we can replace them with a double using dynamic dispatch.

Sometimes JohnnyDepp.lvclass can't be used (source
Sometimes JohnnyDepp.lvclass can’t be used (source)

There are different types of test doubles that are well covered in an article by Martin Fowler. Without getting bogged down into terminology these can return canned answers, simulate some behaviour or simply act as black holes for data!

Spies

One type is called a spy. A spy object is a way for us to peek inside an object. Essentially it is a double that stores information about how it was called so that you can query it later and make sure it was called in the way that you expected.

For example if we are sending a logging library a value of 1 to log, we want to see that the file writing function was called with a value of 1 for the channel name for itself.

Brittle Tests Health Warning – Overuse of this can create very brittle tests. The advantage of taking a high level approach to testing is that your tests aren’t coupled to implementation details which will change more than behaviours. If you use spies too much and you see a lot of implementation details you risk making tests that frequently break and/or require more maintenance.

Spies In LabVIEW

So how do we do this in LabVIEW? Because LabVIEW is a compiled language we must specifically implement new components that contain the spy abilities.

Essentially this component must be able to store the details of each call in a way that it can be read back later. You can do this however you like! But I have created a library to try and take some of the hard work out of it.

LabSpy API

By creating these spies inside of your class you can create a spy class that you can use in your testing. 

 

Typically when we do these there are a few steps:

  1. Create a setup and delete spies method which can be called in the setup and teardown of your test. Make sure the references they create are accessible (either output as an indicator or make accessible through another accessor).
  2. Create the override methods and add the register calls. If you want to track input parameters create a type def for them and wire into the parameters input of the register call function.
  3. Write the test and check the calls using the LabSpy API. The image below shows a simplified diagram showing what this could look like.

Simplified Test View

Now you can check that your software is making the calls with the correct parameters.

Where To Get It

This project is released under an open source license and you can find it on my github account at https://github.com/JamesMc86/LabSpy. There is a VI Package so that you can install it into the palettes which you can download from https://github.com/JamesMc86/LabSpy/releases.

Feel free to download it, play with it, suggest fixes or add feature that you would find useful.

How We Unit Test LabVIEW Code

So if you have been following me for a while you will know that one of my new years resolutions was to really get to grips with unit testing LabVIEW code.
It has been a successful year for this and I thought I would share my results so far.

Result #1 – Always Unit Testing

The immediate question is whether it is worth it and my experience says it is. There is a time investment, that much is obvious but I have seen many benefits:
  • Better development process – by taking a component in isolation and being able to focus and test that thoroughly on its own means less time running full applications, clicking buttons in the right order to create the scenario you need to make sure the latest change works. This means more time writing code, less time debugging and testing. (as well as less code coupling).
  • Higher Confidence – When you have a good suite of tests around your code that you can run in an instant you are more confident in changing it and not causing bugs which looks good to customers and feels great. It also encourages refactoring as it becomes a low risk activity.
  • Smoother Deployments: Compared to last year my commissioning visits have been far smoother with less issues and those that do come up are easier to narrow down.
This is and will continue to be core to how we develop software at Wiresmith Technology.

Result #2 – JKI VI Tester and LabVIEW OO are the Tools of Choice

As late as May I was trying to make NI UTF work for me. I like the concepts and having the option of coverage testing is great but the usability sucks so I transitioned to VI Tester as the framework of choice.
VI tester generates a lot of code as you go! But it lets you write very specific tests in whatever style you like and follows standard testing conventions. My only concern is that it seems unclear what the state of development is and if it will continue. For example I would love to see a “Before” and “BeforeEach” function as opposed to a single “SetUp” VI. It is also very clunky with multiple project targets which I would love to understand what can be done.
Slightly more controversially, I feel that to be really effective you need to be using OO. This is simply because using OO and good design practices/patterns allows you to substitute test doubles where items are difficult to test (i.e. IO) or not relevant to the tests (i.e. another QMH that you don’t want to start, just see what interface calls are made). I just don’t see an effective way to do this with more traditional methods.

Result #3 – Test First not Test Driven

What I refer to here is the purist Test Driven Development. This says that the tests drive the design of your code. You write just enough code to pass each test, refactor the code and by the end you end up with an optimal design.
I have not found much success with this. I have tried a couple of projects where I did try to do this rather than having a proper up front design and the code did not feel very clear. Perhaps it is my style or not enough refactoring but it did not feel good to me.
What I will say is I do follow a test first process.
Unit Testing LabVIEW Process
The thing to remember with the test code is we are trying to call it from as high a level as possible, substituting any test doubles as low as possible to test as many of the real parts as possible, all the time trading off having to spend hours creating test doubles or the test code itself.
Why test first, it has a couple of key benefits:
  • You make sure your test fails first, if the test passes when you haven’t written the code your testing is wrong!
  • It helps you consider the problem domain, what parts are interacting with the behaviour you are working on.
  • It feels like (not saying it is, just feels like) taking two steps back when you are writing tests for code you have already written and know works.
So I think that is one of the first new years resolutions I have ever kept! There is not huge amounts of information out there on unit testing with LabVIEW so I hope this helps. I have a few specific posts in mind to cover some techniques and tools that I have developed along the way that I hope to put up over the next few months.
In the meantime I will once again plug the unit testing group on the community which remains a great resource on this topic.
EDIT: I wrote this post before the announcement of JKI’s new testing framework. I will be looking for a project to evaluate this new approach on and will report back!

External Video – TDD, Where Did It All Go Wrong?

Things have been a bit quiet as we have been going through a number of changes at Wiresmith Technology that has been taking up my time. In the last month we have moved into our first offices, so much time has been spent on making the dream development cave!

We have also taken on a Javascript contractor to help with some work which has taken up time, but he is also keeping me busy with plenty of great resources that he has been using to push his own development skills on things like test driven development, so much of my spare time has been taken with my head in books and youtube videos.

So I don’t have anything new to say today as I’m still absorbing all of this information and I hope to spit it out over the next few months in the form of various experiments, thoughts and translation to the LabVIEW world.

In the meantime, one of the great talks I have watched recently explains why Unit Testing != Testing Units and I’m trying to understand how best to apply this. It’s worth a watch and don’t worry, I have noted that this someone contradicts my last post! This is why I’m not adding anything until I have had a chance to process it properly.

Ian Cooper: TDD, where did it all go wrong from NDC Conferences on Vimeo.

Fixing a Simple Bug with Test Driven Development

So it was the CLA Summit last week that gave me more opportunity to bang on about software testing either further and was great to discuss it with various people and see the themes coming out of it.

My common theme was that I really like the interactive nature of the Unit Test Framework, I think it plays to LabVIEW’s strengths and allows for a nice workflow (for basic tests, using test vectors is far more long winded than it needs to be!).

Another positive I took was from Steve Watts’ talk on debugging and immediacy. He talked about the advantages of ‘runnable code’, that is having logic contained in subVIs that can run independently which aids the debugging process.

So as I worked this week I came across a bug which, the process of fixing, highlighted this well. I took a screencast of the process to highlight some of the benefits that I have found and I think highlights one of the most commonly cited benefits of testing, better code structure. (Go easy, I’m not as natural on camera!)

4 Lessons Learnt Unit Testing LabVIEW FPGA Code

After declaring my intent earlier in the year on moving towards an increasingly test driven methodology, one of the first projects I aimed to use it on has been based on FPGA, which makes this less than trivial.

I was determined to find a way to make the Unit Test Framework work for this, I think it has a number of usability flaws and bugs but if they are fixed it could be a great product and I want to make the most of it.

So can it be used for unit testing LabVIEW FPGA code? Yes but there are a few things to be aware of.

1. You Can’t Directly Create A Unit Test

Falling at the first hurdle, creating unit tests on VIs under an FPGA target isn’t an available option on the right-click menu, normally the easiest way to create a test.

Instead you must manually create a unit test under a supported target type but as LabVIEW FPGA VIs are simply VIs, you can then point this at the FPGA code as long as it doesn’t contain any FPGA only structures.

This has it’s frustrations though. You must manually link the VI (obviously) and rename the test. This bit is a pain as I cannot seem to rename new tests without closing and re-opening the project. Re-arranging tests on disk can also be a frustrating task!

untitled-tests
Argh! Project Hell!

2. Test Vectors Are Your Friend

Unlike processor based code, FPGA logic generally works on data point by point and builds an internal history to perform processing. This means you may have to run the VI multiple times to test it well.

The test vectors function of the unit test framework allows you to do this, specifying inputs for several iterations of the VI and being able to test the VI over time.

Mini-tip: Using Excel or a text editor to do large edits can save you losing your hair! (Right-Click and Open In External Editor)

3. Code Resets

Related to 2, because FPGA code commonly uses feedback nodes/feedforward nodes/registers to store data at some point that has to be reset.

The lazy route to this is to simply wire inputs to the feedback nodes however this defaults to a reset on first call mode. This won’t work with test vectors as each iteration is counted as a new call.

At a minimum you should change the feedback node behaviour to initialize on compile or load, but better practice on FPGA is to have a reset input to the VI to allow explicit control of this. Then you must simply set this on the first element of the vector to get the expected behaviour.

4. FPGA Specific Code or Integration Testing

There will be code that the UTF can’t test directly if it can’t run under an Windows target. In this case the best way to include it in automated testing is with a user defined test. This allows you to create a custom VI for the test, which can include a FPGA desktop execution node to allow for testing the VI. Potentially this will have to be accompanied with a custom FPGA VI as well to be able to pipe the correct data into FIFOs or similar.

 

Hopefully this will help you get over the first few hurdles, I’m sure more will follow and I will try to update them here as I find them.

My 2015 LabVIEW Resolution

It’s a new year which means we must assume everything we did before last week was rubbish and change everything.

OK, that’s a slightly cynical view but it does always amaze me how quickly advertising, blogs and news changes at this time of year. That’s also not to say that I haven’t joined in, as I ease back into work I have enjoyed looking back at 2014 as a year of great change (starting Wiresmith Technology) and now look forward to improving in 2015.

 Quality means doing it right when no one is looking – Henry Ford

One big focus point for me is going to be software quality. As old projects finish and new ones start, I want to make sure I have done everything in my power to prevent old projects coming back with issues. Some problems are inevitable but it costs time to fix and lost time on other projects to have to refocus.

I find it interesting that when it comes to software bugs are considered inevitable, even as I started by business and got contracts drawn up, the templates all include clauses stating this. Whilst there is an element of truth (software is a complex system) I also think it can lead to a relaxed attitude towards defects that you wouldn’t find in other fields.

Software never was perfect and won’t get perfect. But is that a license to create garbage? The missing ingredient is our reluctance to quantify quality – Boris Beizer

When it comes to quality I think the first step has to be testing. I have been using the unit test framework from NI on a couple of projects now as well as unit test frameworks in javascript and I am convinced it is the way forward. By the end of the year I want to be doing something akin to TDD.

The key reason is simple, when I have been writing tests as I have written code (not always strictly first, but as I am initially developing) I have found bugs. Therefore, there is only two possible outcomes to me not testing:

  1. I discover that bug as I test the integration of that code into the main product. This could take a lot of time if it is not clear what subVI is the source of the bug and I have to go back and fix it. Even knowing the subVI it means I have to get back into the same mentality as when I wrote it, which also takes time.
  2. I still miss it and the customer finds it instead. This is more costly to debug as you are likely going to find it harder to debug from the customers descriptions down to a subVI, never mind knocking the customers confidence in you.

To do this requires two things, the right mentality and the right tools.

For the mentality, discipline is the biggest requirement to begin with. I know the process and it will feel unnatural at first but I hope to push through the initial pain to get to the rolling green pastures on the other side.

For the tools, there are really two in existence for LabVIEW. The Unit Test Framework (UTF) from NI and JKI’s VI Tester. I have tried UTF quite a bit and want to return and evaluate VI Tester over the next couple of months to understand its advantages.

For both of these, keep an eye out over the next few months when I hope to report back on my progress and findings with them. No doubt I will also be discussing this on the NI community as well. Check out the Unit Testing group over there if you want to learn more (and from more experienced people).


By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close