Getting VI Analyzer Into Our Workflow
We have long known that VI Analyzer is a good idea – much like unit testing – people on the other side of adoption swear by it.
We’ve found a few hurdles for mainstream adoption into our process, and I suspect yours too.
1. Understanding Why
The first step in adopting VI Analyzer (and keeping that adoption going) is understanding why you are doing it!
What I mean by this is you need to make the tool fit your process – not the other way round. If you use VI Analyzer because NI says so then you’re going to see less benefit and it will feel a lot more effort.
For me – I believe that consistent style and code inspection reduces bugs and improves readability of the system. I have a style guideline, but I don’t always follow it. As I typically work on my own, then code reviews aren’t an option.
Your “why” will probably be very similar but the subtle differences will make some of the following items slightly different from mine.
2. Complicated Setup
As I said above, VI Analyzer is all about consistency for us. We want every project to follow the same style guidelines. Unfortunately, VI Analyzer does not make it very natural to create a standard test configuration and share it between projects since there is a single configuration file for test setup and which VIs to test.
These are the steps that I went through to build a standard configuration:
- Start with your style guidelines. I made mine into a table and identified what already had a VI Analyzer test, what had a custom test in the community, what could have a custom test and what was not testable.
Style Guide With Tests - I downloaded the custom tests that are available online and created a couple of key tests myself. I didn’t do them all, and I will expand my coverage over time.
- I used a project to create a VI Analyzer configuration file. I worked through the tests and configured them to my style guideline. Then I removed all of the VIs to be tested. I saved this configuration file as my standard.
- I created a VI package which would install the custom tests and the configuration file to a shared location in my documents. Full credit to Fabiola De La Cueva and Delacor for this idea. They have been helping their customers with this for some time. (You can see their video of this on Youtube)
By completing these steps up front, I can introduce a consistent VI Analyzer configuration to a new project quickly and easily.
3. Defining A Trigger
As with any good habit, you need a trigger to tell you when to do it else you often won’t follow through. There are a few common triggers that I have seen people using:
- Post-Commit in a CI Server – I’m not a big fan of this one because you now need another trigger to review the results and implement the changes.
- Pre-Code Review – This is a great one if you are on a team. You should test with VI Analyzer before a code review. You don’t want to waste time picking up things a machine can identify faster. As I am a solo developer, this one is limited for me.
- Feature Branch Merge – If you are using a branching workflow in Git or Mercurial, then you can use a feature branch merge as a trigger. This is a good trigger as it should limit the scope of what needs testing. However, if the change list gets long, then there could be a lot to review.
- Pre-Commit – I use a Javascript IDE that has a tick box in the commit tool to check style before the commit. This workflow is what I want in LabVIEW. Testing at each commit makes the tests quick (since only a few files have changed) and it means that you fix issues while you are in the mindset of that code. It also means that once you commit the code, it is “done”. The main problem with this in VI Analyzer is the execution time of the tests.
4. Speed
I wanted my trigger to be a commit. I feel like the code should go through VI Analyzer to be “finished” and only “finished” code should be committed.
This is a problem though. Analyzing a whole project can take many minutes, and I might commit 10+ times a day.
One way to solve this would be to test only changed code, but VI Analyzer lacks any native support for this workflow.
I have developed a tool to tie it to git changes. It isn’t great, so we haven’t shared it yet (it still lacks many essential features), but it has started us using VI Analyzer regularly.
This tool will take a configuration file and then run it against only the code that Git shows as changed. Testing only the changes cuts the test time and now means that it is possible to check at each commit and fix the changes before that commit.
By overcoming these hurdles, VI Analyzer has become a standard part of our workflow. I hope you can use this to incorporate it as well.
Joerg
July 24, 2018Great article, as always!
I do trigger VI Analyzer post-commit on our CI server, with all the drawbacks you describe. As I can run our “app builder” (the work-in-progress name of our CI toolchain) manually via a LabVIEW UI, I also implemented the feature you describe, to only check on those files that were changed for a given commit or since the last tag.
And much like you, I would love to see this integrated into the workflow somehow. But the git client I use doesn’t offer any hooks that I know of…
Fabiola De la Cueva
July 26, 2018Excellent post! Thanks for the shout out and for sharing with us your insight into the why and triggers.
At Delacor we have discussed some ideas on how to by only run VI Analyzer on changed VIs. If we come up with a workable solution, we will make sure to share it with you.
Thanks,
Fab
DNatt
July 26, 2018> Unfortunately, VI Analyzer does not make it very natural to create a standard test configuration and share it between projects since there is a single configuration file for test setup and which VIs to test.
As you mention in the document, starting with a template .cfg that contains your desired test configurations is the recommended approach. If you have ideas for a more efficient way to accomplish this, let me know.
> I downloaded the custom tests that are available online and created a couple of key tests myself. I didn’t do them all, and I will expand my coverage over time.
If you feel so inclined, please share any generally useful tests you’ve developed with the VI Analyzer Enthusiasts community.
> Defining A Trigger
I agree with all the trigger points you have identified, and I’ve been involved in projects that use all 4 of them. And in most cases, having a different .cfg file for each one makes a lot of sense. There are certain things I expect a developer to have corrected before I review the code (Pre-Code Review), and there are certain things I expect a manager to receive an email about when they somehow make it past code review (Post-Commit in a CI Server).
> The main problem with this in VI Analyzer is the execution time of the tests.
For a Pre-Commit action, I would expect a small number of tests running on a small number of VIs.
> I have developed a tool to tie it to git changes.
That would be a great tool to share with the VI Analyzer Enthusiasts community once you feel ready to share it. There is already a similar tool shared with the community that works with Perforce (see here: https://forums.ni.com/t5/VI-Analyzer-Enthusiasts/Quick-Launch-options-for-VI-Analyzer/ta-p/3625965) to only analyze VIs that are set to writeable (i.e. files that are checked out from Perforce).
Also, I noticed in the Twitter discussion of this thread that there are concerns about performance of the VI Analyzer API. We’ve discussed performance-related issues a bit here (https://forums.ni.com/t5/VI-Analyzer-Enthusiasts/Configuration-File-Load-Times/m-p/3783277 ), but in general, make sure the VIs you’re analyzing are mass-compiled, the tests you’re running are mass-compiled, and you’re only running the tests you need, and not the whole test suite. There were lots of changes made to VI Analyzer in 2018, but none that I think would directly affect performance.
I will say that while reading through this thread and looking at the source code for VI Analyzer, I did get an idea for a possible performance improvement when starting a new programmatic analysis task. I will put this on my list of potential improvements to investigate for VI Analyzer 2019.
James McNally
July 26, 2018Awesome thanks for the comments!
So, several of the custom tests I created were added in 2018 – this is where I’m hoping for the best increase as I believe from our discussions there is some overhead in loading the tests (and I dare say you’ve implemented better than I will have!). I had a go at running the mass compile again but I don’t recall that I saw much improvement (and then moved on to other projects)
To jump back a bit to this:
> Unfortunately, VI Analyzer does not make it very natural to create a standard test configuration and share it between projects since there is a single configuration file for test setup and which VIs to test.
As you have stated there is a good standard approach around this. My comments are based more on a UX feel. The fact that you select tests and then VIs and save them together in a config ‘feels’ like you have tied it to that project. If you could save out a test configuration without VIs and load that when configuring VI analyzer in a new project I think it would feel more natural in that use case.
DNatt
July 26, 2018One of the changes we made in VI Analyzer 2018 is that the configuration file is a new XML-based format (and a new file extension — .viancfg), and this format makes it easier to have a human-readable (and editable) template configuration file that just defines tests and nothing else. So you could make a .viancfg that only contains the following:
1
[all your tests]
That’s pretty darn close to having a file that only defines the tests that you can use for a template.
DNatt
July 26, 2018Hah, all my XML disappeared in that previous post. It was supposed to say:
<Config>
<FileFormatVersion>1</FileFormatVersion>
<TestConfigData>
[all your tests]
</TestConfigData>
</Config>
Stas
August 5, 2018James, very inspiring post. I have a question I hope you can answer. In order to tie VI Analyzer to pre-commit hook batch file, did you have to make your tool executable and call it from within the batch file with command line arguments? Assuming that LV is already running, it would be impossible to invoke a tool it is were a simple VI.
James McNally
August 6, 2018Hi Stas,
I had a similar discussion with a friend of mine who has been working on this already! You can see Joerg’s progress at https://twitter.com/joerghampel/status/1023369726926446594. Think he has something working well.
Cheers,
James
Pingback: Getting VI Analyzer Into Our Workflow | Delacor
Russell Blake
September 14, 2018Hello James,
Thanks for another great post! If you do end up sharing the tool you created to tie VI Analyzer into git changes I would definitely be interested in trying it out.
Pingback: Getting VI Analyzer Into Our Workflow – Alliance of LabVIEW™ Architects.
Pingback: VI Analyzer and Git – Felipe's Thoughts