The APIs that you have to test are not always simple. As well as passing data they may involve events (with the front panel or with user events).
The other day I needed to test that an event fired as part of a test case. I could see a generic solution, so I created a template for it. I had two requirements:
If the event doesn’t fire – test fails.
If the event fires with the wrong data – test fails.
In my given when then sequence then we end up with a test that follows the structure:
Given: Who knows, in this case, a UI library has been tied to a control.
When: We take some action that should cause an event on that control.
Then: Check the event.
To check the event we create an event structure outside of a loop as we don’t want to handle multiple events. We need two cases:
A timeout case with a suitable timeout – In this case, we call the Test Case.lvclass:fail.vi to fail the test. This should never run if the when code fired the event.
Failing Path
A case that handles the event – If you don’t care about the data then you can do nothing here, otherwise, include tests on the data included in the event.
Passing Path
Additional Complexity
Dynamic Event Registration: If this is a user event then you will need to register for the event. I’ve included this in my template, but you must move the event registration to the given case. If you haven’t registered the event before the action in the when case, it won’t ever fire.
Parallel/Dynamic Event Generation: If your event is in some dynamic code you may need to have this running. My advice: DONT. Try and pull out the internal API and test synchronously. Asynchronous testing in LabVIEW introduces timing concerns which make your tests much more complicated.
In my last post I covered how I am using unit testing in LabVIEW. In this post I want to share a useful tool on that journey.
Test Doubles
One of the first rules I operate under while testing is using as much of the real application as possible in testing a behaviour. Then each test becomes more valuable as it covers the integration of all the units as well as their contents.
Sometimes however this simply isn’t possible. You may need to talk to hardware that isn’t available, or a database or file that is hard to verify as part of the test (It’s normally possible somehow, but perhaps not worth it). This is where test doubles come in.
A test double is simply a substitute for a software component that you want to exclude from the test. In my last post I mentioned that OO make testing easier, this is why. If these components are written to an abstracted interface we can replace them with a double using dynamic dispatch.
Sometimes JohnnyDepp.lvclass can’t be used (source)
There are different types of test doubles that are well covered in an article by Martin Fowler. Without getting bogged down into terminology these can return canned answers, simulate some behaviour or simply act as black holes for data!
Spies
One type is called a spy. A spy object is a way for us to peek inside an object. Essentially it is a double that stores information about how it was called so that you can query it later and make sure it was called in the way that you expected.
For example if we are sending a logging library a value of 1 to log, we want to see that the file writing function was called with a value of 1 for the channel name for itself.
Health Warning – Overuse of this can create very brittle tests. The advantage of taking a high level approach to testing is that your tests aren’t coupled to implementation details which will change more than behaviours. If you use spies too much and you see a lot of implementation details you risk making tests that frequently break and/or require more maintenance.
Spies In LabVIEW
So how do we do this in LabVIEW? Because LabVIEW is a compiled language we must specifically implement new components that contain the spy abilities.
Essentially this component must be able to store the details of each call in a way that it can be read back later. You can do this however you like! But I have created a library to try and take some of the hard work out of it.
By creating these spies inside of your class you can create a spy class that you can use in your testing.
Typically when we do these there are a few steps:
Create a setup and delete spies method which can be called in the setup and teardown of your test. Make sure the references they create are accessible (either output as an indicator or make accessible through another accessor).
Create the override methods and add the register calls. If you want to track input parameters create a type def for them and wire into the parameters input of the register call function.
Write the test and check the calls using the LabSpy API. The image below shows a simplified diagram showing what this could look like.
Now you can check that your software is making the calls with the correct parameters.
So if you have been following me for a while you will know that one of my new years resolutions was to really get to grips with unit testing LabVIEW code.
It has been a successful year for this and I thought I would share my results so far.
Result #1 – Always Unit Testing
The immediate question is whether it is worth it and my experience says it is. There is a time investment, that much is obvious but I have seen many benefits:
Better development process – by taking a component in isolation and being able to focus and test that thoroughly on its own means less time running full applications, clicking buttons in the right order to create the scenario you need to make sure the latest change works. This means more time writing code, less time debugging and testing. (as well as less code coupling).
Higher Confidence – When you have a good suite of tests around your code that you can run in an instant you are more confident in changing it and not causing bugs which looks good to customers and feels great. It also encourages refactoring as it becomes a low risk activity.
Smoother Deployments: Compared to last year my commissioning visits have been far smoother with less issues and those that do come up are easier to narrow down.
This is and will continue to be core to how we develop software at Wiresmith Technology.
Result #2 – JKI VI Tester and LabVIEW OO are the Tools of Choice
As late as May I was trying to make NI UTF work for me. I like the concepts and having the option of coverage testing is great but the usability sucks so I transitioned to VI Tester as the framework of choice.
VI tester generates a lot of code as you go! But it lets you write very specific tests in whatever style you like and follows standard testing conventions. My only concern is that it seems unclear what the state of development is and if it will continue. For example I would love to see a “Before” and “BeforeEach” function as opposed to a single “SetUp” VI. It is also very clunky with multiple project targets which I would love to understand what can be done.
Slightly more controversially, I feel that to be really effective you need to be using OO. This is simply because using OO and good design practices/patterns allows you to substitute test doubles where items are difficult to test (i.e. IO) or not relevant to the tests (i.e. another QMH that you don’t want to start, just see what interface calls are made). I just don’t see an effective way to do this with more traditional methods.
Result #3 – Test First not Test Driven
What I refer to here is the purist Test Driven Development. This says that the tests drive the design of your code. You write just enough code to pass each test, refactor the code and by the end you end up with an optimal design.
I have not found much success with this. I have tried a couple of projects where I did try to do this rather than having a proper up front design and the code did not feel very clear. Perhaps it is my style or not enough refactoring but it did not feel good to me.
What I will say is I do follow a test first process.
The thing to remember with the test code is we are trying to call it from as high a level as possible, substituting any test doubles as low as possible to test as many of the real parts as possible, all the time trading off having to spend hours creating test doubles or the test code itself.
Why test first, it has a couple of key benefits:
You make sure your test fails first, if the test passes when you haven’t written the code your testing is wrong!
It helps you consider the problem domain, what parts are interacting with the behaviour you are working on.
It feels like (not saying it is, just feels like) taking two steps back when you are writing tests for code you have already written and know works.
So I think that is one of the first new years resolutions I have ever kept! There is not huge amounts of information out there on unit testing with LabVIEW so I hope this helps. I have a few specific posts in mind to cover some techniques and tools that I have developed along the way that I hope to put up over the next few months.
EDIT: I wrote this post before the announcement of JKI’s new testing framework. I will be looking for a project to evaluate this new approach on and will report back!
Things have been a bit quiet as we have been going through a number of changes at Wiresmith Technology that has been taking up my time. In the last month we have moved into our first offices, so much time has been spent on making the dream development cave!
We have also taken on a Javascript contractor to help with some work which has taken up time, but he is also keeping me busy with plenty of great resources that he has been using to push his own development skills on things like test driven development, so much of my spare time has been taken with my head in books and youtube videos.
So I don’t have anything new to say today as I’m still absorbing all of this information and I hope to spit it out over the next few months in the form of various experiments, thoughts and translation to the LabVIEW world.
In the meantime, one of the great talks I have watched recently explains why Unit Testing != Testing Units and I’m trying to understand how best to apply this. It’s worth a watch and don’t worry, I have noted that this someone contradicts my last post! This is why I’m not adding anything until I have had a chance to process it properly.
So it was the CLA Summit last week that gave me more opportunity to bang on about software testing either further and was great to discuss it with various people and see the themes coming out of it.
My common theme was that I really like the interactive nature of the Unit Test Framework, I think it plays to LabVIEW’s strengths and allows for a nice workflow (for basic tests, using test vectors is far more long winded than it needs to be!).
Another positive I took was from Steve Watts’ talk on debugging and immediacy. He talked about the advantages of ‘runnable code’, that is having logic contained in subVIs that can run independently which aids the debugging process.
So as I worked this week I came across a bug which, the process of fixing, highlighted this well. I took a screencast of the process to highlight some of the benefits that I have found and I think highlights one of the most commonly cited benefits of testing, better code structure. (Go easy, I’m not as natural on camera!)
Continuing on my theme from the last post I wanted to talk about what goes around writing great software, other than the software itself. How can you improve your coding process?
This consists of 12 questions and you should be aiming to answer yes to 10 or more of these.
Do you use source control?
Can you make a build in one step?
Do you make daily builds?
Do you have a bug database?
Do you fix bugs before writing new code?
Do you have an up-to-date schedule?
Do you have a spec?
Do programmers have quiet working conditions?
Do you use the best tools money can buy?
Do you have testers?
Do new candidates write code during their interview?
Do you do hallway usability testing?
Stop and have a think about these with your team, what do you score?
Now what if I told you this article was 15 years old! I find this interesting for two reasons:
Some of these points seem quite out of date with new developments around agile methods of software development building on many of these points. There is a good article which reviews these points here.
I still know lots of developers that haven’t adopted many of these practices.
I must confess there are differences for most developers in the LabVIEW way of working. While many techniques are aimed towards 4-8 developers working on software for hundreds/thousands of users, many LabVIEW applications are developed by individuals or small teams for internal projects where you are probably already speaking to all of the end users.
Can we still infer great tips for those teams? Let’s look at what we can do with LabVIEW to add some more yes’s.
1. Do you use source control?
If you aren’t using this now, stop what your doing and go learn about it!
There are so many resources and technologies out there. SVN is extremely popular in the LabVIEW community although I would probably recommend trying Git or Mercurial as there are more services out there available to use now.
If your interested in these try the Atlassian SourceTree client which solved my previous concerns about a good Windows client for Git.
2. Can you make a build in one step?
3.Do you make daily builds?
Builds is a funny word in the LabVIEW world as this varies from many languages as LabVIEW is constantly compiling.
The key point behind this is that you should be continuously making sure the application can run. In LabVIEW this can boil down to being able to run the application consistently, especially as if you can’t run it, you can’t test it.
I make a point of running the full application after every change worth a commit into source code control or a bug in the bug database. This saves context switching of having to go back to old features because you didn’t test them at the time.
4. Do you have a bug database?
This should be step 2 after getting your source code control set up.
If you haven’t used these before, I find them invaluable as a means of tracking the state of the application.
Not just bugs but feature requests can go in the database and get assigned things like priorities, developers, due dates etc. and it becomes where you go when you need to know what to do next.
They also typically integrate with source code control. When I make a commit I mention a bug ID in the commit message and the system links the two.
So I now have a system that tracks what needs doing, who needs to do it, when they have done it and you can even find the exact commit so you can identify what code was changed to complete it.
Again, in this world of cloud computing this doesn’t even require any outlay or time to set up. Most source code hosting services have this built in.
Bitbucket is an easy one to start with as you can have free, private projects. I use a hosted Redmine server from Planio (which costs 9 Euro/month), others use Github (free for public projects).
Go set up a trial account and take a look around. As with many things it will take a bit of getting used to but I find it a far better way to work.
9. Do you use the best tools money can buy?
I’m skipping a few that aren’t specific to LabVIEW here so we fall on the best tools.
Firstly you chose LabVIEW! I think most people reading this will hopefully agree this is a good choice, but it doesn’t stop there.
There are probably two key things with LabVIEW to consider from a hardware point of view. a) It isn’t exactly lightweight and b) It uses a lot of Windows!
You don’t want to be trying to code LabVIEW on a 13″ laptop with a track pad. Get a mouse, get at least one large monitor, preferably two and a decent, modern machine.
My setup- Filling the desk with monitors!
The extra costs will soon be recovered in productivity gains of not having to wait for the software to load a new library or trying to switch between a couple of VIs and the probe window when debugging.
Also consider this from a software perspective as well. It is difficult but try and get familiar with some of the toolkits on the tools network or LAVA forums as they could save you a lot of time over writing features yourself.
There are many free tools but even if there is a cost associated you have to factor in your time to design, develop and maintain that extra code (which is probably not the area you are expert in).
10. Do you have testers?
This is one that is falling out of vogue at the minute as Test Driven Development (TDD) is becoming popular (see my previous post) which means the developers tend to write the tests themselves. That being said it is still useful to have people available for higher level testing as well but as mentioned previously we tend to have smaller teams in LabVIEW.
I think the key point right now is to have a plan for testing. Likely this should be a mix of automated testing and having some test procedure/schedule for higher level testing.
One of my first projects at Wiresmith Technology was where this failed me. I didn’t re-test the software thoroughly enough after changes (normally only the specific section which I had changed) which meant more problems went to the customer than I would like. They all got fixed but each problem means time wasted on communicating the problem as well as affecting the confidence of the customer.
Since then I now keep a procedure for testing the various areas of the software so I can do this before I send a release to the customer which has improved my hit rate as well as saving time in the long run.
With bringing in things like this there is always some pain adjusting but pick one, push through and you will come out writing better software.
It’s a new year which means we must assume everything we did before last week was rubbish and change everything.
OK, that’s a slightly cynical view but it does always amaze me how quickly advertising, blogs and news changes at this time of year. That’s also not to say that I haven’t joined in, as I ease back into work I have enjoyed looking back at 2014 as a year of great change (starting Wiresmith Technology) and now look forward to improving in 2015.
Quality means doing it right when no one is looking – Henry Ford
One big focus point for me is going to be software quality. As old projects finish and new ones start, I want to make sure I have done everything in my power to prevent old projects coming back with issues. Some problems are inevitable but it costs time to fix and lost time on other projects to have to refocus.
I find it interesting that when it comes to software bugs are considered inevitable, even as I started by business and got contracts drawn up, the templates all include clauses stating this. Whilst there is an element of truth (software is a complex system) I also think it can lead to a relaxed attitude towards defects that you wouldn’t find in other fields.
Software never was perfect and won’t get perfect. But is that a license to create garbage? The missing ingredient is our reluctance to quantify quality – Boris Beizer
When it comes to quality I think the first step has to be testing. I have been using the unit test framework from NI on a couple of projects now as well as unit test frameworks in javascript and I am convinced it is the way forward. By the end of the year I want to be doing something akin to TDD.
The key reason is simple, when I have been writing tests as I have written code (not always strictly first, but as I am initially developing) I have found bugs. Therefore, there is only two possible outcomes to me not testing:
I discover that bug as I test the integration of that code into the main product. This could take a lot of time if it is not clear what subVI is the source of the bug and I have to go back and fix it. Even knowing the subVI it means I have to get back into the same mentality as when I wrote it, which also takes time.
I still miss it and the customer finds it instead. This is more costly to debug as you are likely going to find it harder to debug from the customers descriptions down to a subVI, never mind knocking the customers confidence in you.
To do this requires two things, the right mentality and the right tools.
For the mentality, discipline is the biggest requirement to begin with. I know the process and it will feel unnatural at first but I hope to push through the initial pain to get to the rolling green pastures on the other side.
For the tools, there are really two in existence for LabVIEW. The Unit Test Framework (UTF) from NI and JKI’s VI Tester. I have tried UTF quite a bit and want to return and evaluate VI Tester over the next couple of months to understand its advantages.
For both of these, keep an eye out over the next few months when I hope to report back on my progress and findings with them. No doubt I will also be discussing this on the NI community as well. Check out the Unit Testing group over there if you want to learn more (and from more experienced people).
About
This LabVIEW blog provides advice on development best practices and tips from the experts at Wiresmith Technology.
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
Recent Comments