I am sure this has been asked…
What do people use for Quality Assurance regression testing of their software?
We just have a bunch of spreadsheets with every main function by 3rd party template to test like; Capesoft’s, Noyantis’, Boxsoft etc…
And when we update any of those or even CW we test each section of our code.
Does anyone use anything more automated? It would also be nice to be able to modify existing testing scripts if the data flow (or screen) changes.
Kevin Erskine
In the Clarion Mag link in this post, Andrew Barnham talks about some of the automated testing he does.
I think there are a few different articles on automated testing in Clarion mag.
ClarionTest is one such article ClarionTest - Clarion Content - Confluence (jira.com)
There is thread here on automated testing tools with some suggestions Clarion, Automated test tools with Clarion ? (computer-programming-forum.com)
Another term for Automated Testing is Fuzzing and Fuzzer’s. This is a link to one of Google’s Fuzzer’s which can be used for programming languages. I’m sure there will be other fuzzer’s on github.
GitHub - googleprojectzero/domato: DOM fuzzer
Havent tested or tried any of these yet personally, so it will be interesting to see if anyone else has used a fuzzer of sorts and what their experiences were.
Some of our products - especially the ones which are mostly interface free - ship with an example called UnitTest. Always built in the latest clarion (in this case 11.1) The most up-to-date is StringTheory. xFiles and jFiles also have them though - although they cover less ground.
I wish I had coverage for everything like I do for StringTheory, but it’s a lot of work, so sadly I don’t. but it means StringTheory builds at least come with an exceptionally high level of confidence.
cheers
Bruce
Thanks - we need to test very specific entry screens and data flow to other APIs, so a more general\standard approach instead of a lot of handcoding is what we are looking for. Thanks I will still loook st your suggestion for future uses.
Back in the 16-bit days used Microsoft Visual Test. It was sold and became Rational Visual Test. It worked great until compiling 32-bit when it would often get lost, possibly due to the old threading. So that company gave up on testing that way. We should have found another way as the Forms got more and more complicated where numbers entered resulted in other numbers plus feilds enabled/disabled/hide/unhide.
It had all the features needed to find Controls various ways. It had a control picker like Spy++ that helped find some unique attributes of the control to ID it. Part of the test scripting work is to ID Controls in some way that can still work if the window changes.
VT could write values or select radios or drop lists. It also allowed reading values so you could confirm it was as expected. So a form could be filled in, then read various fields to see if they were as expected. Read the Browse values to see it was as expected. Often a Browse had totals that made it easy to confirm. Could also read the state of a field to confirm that if a Box was checked, or certain amount entered, other fields would unhide or enable.
It was a bunch of work to setup the scripts but then it validated that inserting records of all types worked as expected, there were no regressions. Also all changes worked as expected with no regressions.
AFAIK Visual Test no longer exists. It was exactly what was needed. Last I recall looking at it I thought that .Net had the features needed … in some API. There are also programs like AutoHotkey to automate forms as macro players that may work. You need that ability to read control values and state, then log problems and continue.
Anything that can enumerate controls on a window like Spy++ will be useful, but what I’ve started to notice with software and this might be because of what language its written in, is some controls can no longer be enumerated.
For example, its not longer possible to enumerate the different controls on Firefox so its no longer possible to pull the web address out of the web address field. Other browsers and programs have also gone the same way, and its also now no longer possible to enumerate controls in parts of windows like the new Settings (control panel) window. I dont know if these are called App X packages or not.
So whilst this wont affect Clarion, I do wonder how automated testing is carried out on these programs or parts of window, if they can no longer enumerate the controls and stuff them with data.
Anyone know?
I think there some interesting lessons to be learned by the recent NOTAM shutdown across North America. Canada suffers same mysterious air system outage as US, delaying more flights | The Independent
It seems the US FAA are blaming two contractors and are investigating them for malicious intent.
Now as someone, like many on here, have been asked to write code on contract and knowing how vague words are which lawyers exploit in the courts all the time, why is the Press not questioning the FAA about their own failings instead of buying the line that it was the fault of two contractors?
FAA says computer failure that grounded thousands of flights caused by 2 contractors | Fortune
I’ve worked at plenty of places which dont have procedures in place to test software properly, this is one of the reasons why I use Clarion, the templates and generated code is tested by thousands if not millions of people on a daily basis, yet it seems the FAA didnt have the necessary procedures implemented to test software updates before they get pushed into live production environments, other than restoring software backups for a 30year old system, only to find it had buggered up the database as well!
I bet the FAA werent using Clarion and the templates to generate their code! 
Back at CIDC2013 I did a presentation on refactoring and optimizing code and on the general theme of “quality”. One of the points was that the amount of testing you should do is proportional to the consequences of getting it wrong.
Consider these contrasting scenarios:
You are doing some batch update where you take a backup, then do the update. If it crashes mid-stream you simply restore to the backup point, fix the problem and re-run. Rinse and repeat.
You write a system where life depends on the software. For example, software for a driverless vehicle or a rocket launch with people onboard.
Obviously case 2 has a much higher cost (lives) of failure. Therefore the testing needs to be appropriately much, much higher.
In the mid 1980’s I was at a job where there was basically no formal QA (Quality Assurance) apart from one’s own professional pride. We were trusted to test our work well before putting it into production. This setup works as well as the quality of the programmers.
My next job was the complete opposite - a large team with multiple layers of bureaucracy. A job or fix that might have taken an hour or two at the previous job now took days or even weeks with numerous forms to fill in with documentation and tests that were passed to a separate QA team who vetted your changes and tested everything. And had lots of meetings to discuss everything.
It was a bit of a culture shock moving from one environment to the other and I felt somewhat stifled by the lack of productivity.
There had to be a better way!
I started my own company so I only had to answer to myself (and my clients if I got something wrong).
Mind you I never wrote software that sent anyone to the moon.
I hadn’t really thought of that before. This is true of the “boiler plate” code but not your custom code that you put in embed points. Dave Harms wrote a series of articles in his Clarion Magazine basically suggesting you take your code out of embeds and place it in classes that could be tested automatically using a unit test framework.
Clarion can be divided into two parts - “open” and “closed”. The templates and ABC library are “open” - you can view the source and fix bugs or do your own thing.
The compiler, run-time and drivers are a different matter. My concern in recent years has been that some new versions of Clarion seem to be “two steps forward, one step back”. They fix up some things. And break others.
You hear of people using a later version of Clarion with some particular driver from an earlier version “because it works”.
if true then that’s a worry. The costs of a mistake here are high. Grounding all the planes for a few hours must have cost millions in productivity (not to mention aggravation).
Reports on this were unclear and contradictory. I think it comes down to media reporters not having an IT background. When I first read about this I thought they meant they had restored to an earlier version of the software but there was some corruption in a data file so of course that fixed nothing. Other reports suggested the data corruption was caused by some “bug” in the software. Any of us who in the past used tps files in a networked environment probably know about data corruption. Not necessarily caused by your software.
You are probably right but that didn’t stop some wags (jokers) on the Skype groups suggesting it must have been tps file corruption!
Automation testing systems and Fuzzers do this.
Nasa’s Space Shuttle runs 4 systems + 1 backup with a majority rule because the radiation in space can affect cpu’s. Ch4-3 (nasa.gov) The same radiation also affects high altitude aircraft to a lessor extent, but most people will know this with the science that frequent flyers have an increased risk of cancer.
Automated fuzzers, automated form filling and decision tree’s to guide the resulting behaviours would have speeded those things up. This is what makes Fuzzers, their key feature being they create and do random stuff which is favourable to the standard unit tests.
I couldnt help but notice that lots of 9/11 twin tower stuff was circulating quite high up on the front page of reddit, like a video of the collapse of WTC7. Other websites discussing the NOTAM, ie forums & newspapers with comments to discuss the NOTAM, I didnt see any mention of 9/11 so was it a psychological exercise to monitor across various sections of society who had forgotten about 9/11? 
Edit. I did find it suspicious well before the 9/11 posts on Reddit and my initial thoughts were was this an exercise, was this a threat from a satellite, Russia’s Zircon missile Putin deploys Russian warship with Zircon hypersonic missile, TASS says | CNN or something else. Thats what I love about the internet, its a Natural Sciences scientists wet dream.
Where would you put code templates then?
An app which can take free form embed code and make it a code template applies consistency which can remove the code placed in the wrong embed and the copy and paste errors that occur. Currently the Wizatrons are the closest to an app which meets this criteria.
An insight into the processes being used in the development of Clarion?
In fairness this is a hardware failure because the networking checksums and/or OS have failed to deliver the data properly.