Medium sized organizations are hardly capable of controlling their IT systems, the testing of their software is too complex. Even the relative easy software that is needed to anonymize production data, is too much work. This is not because of unwillingness, say René Ceelen, but inability: It’s too expensive.
Software packages are hugely popular. At a lot of medium sized organizations like healthcare, housing associations they form the core of the IT system, and all the ins and outs of the organization completely depend on the ups and downs of the software package. However medium sized organizations underestimate the complexity of the information system that has been built with the software package. They inadequately realize that they hardly are able to control the quality of that information system, because testing these have become too specialized and time consuming.
The complexity of such information systems usually starts when multiple software packages are mutually linked. Subsequently this whole will be linked to the outside world – hospitals with general practitioners and pharmacies, and soon also with a national switchboard (our situation in the Netherlands). In this kind of configuration, the information streams wiggle themselves through a spider web of systems and links, as confusing as an Eastern Kasbah, where tourist easily get lost, filled with houses, slums and alleys.
Because of the dependence to their IT-solutions, it isn’t an unnecessary luxury for organizations to demonstrate (again) that everything is still functional with every new release. Under the specific situation of where the organization utilizes the information system, including the links with other software, not just for one type of software packet that the software developer tests in his private laboratory situation. Recognized standards for information security like the Code of Practice for information security management or the derivative thereof for information security management in the healthcare industry (Dutch NEN7510) constitute clear requirements to this type of work.
To start off with an easy demand, that is mentioned in the standard for information security management in the healthcare industry (Dutch NEN7511-1): if test data is a derivative from production data, then this data should be made anonymous, a demand coming straight from the Data Protection Act (DPA) amongst other things. Completely correct, especially for an institution for mental health care. The same goes for simple looking financial administration of for example housing associations. Again with an organization like the aforementioned, there is confidential data traffic like information about social issues, late payments and income data. Too often, however the relatively simple software which is necessary for anonymizing of production data, is already too much work. The core requirements of good testing are availability of acceptance criteria that are developed in test scenarios with specific test kits, approval procedures for changes and a test environment that is separate from the production environment. Numerous medium-sized organizations have only partially completed the requirements. This is not a matter of unwillingness but mostly inability because all of the work is hardly affordable.
This way risk and accidents arise, for example: sending correspondence to deceased patients or renters, losing medical data, inability to send out invoices for months et cetera. The risks are further amplified when complicated arrangements get introduced under high time pressure, whereby further testing is put under even more pressure.. As were when the complex Diagnosis Treatment Combinations (DTC’s) were introduced into the healthcare industry with a high tempo, and it’s conceivable that at numerous organizations calculation errors are made that stay(ed) unnoticed. This has happened before, even with banks that have much more money to spend on IT compared to medium-sized organizations. According to a report in the Dutch newspaper the Volkskrant, the ING bank discovered that in 2005 they miscalculated many commissions, for years, and they had to repay 70 million euros to their customers.
One can, with a simple checklist verify if their organization has supplied the basic requirements for good testing. If that is not the case, apparently they trust the software supplier to test her own software adequately before it’s released. With this expectation they will not only be disappointed regularly, but it’s also not realistic. Every organization should carry its own responsibility to control critical business software before taking it into production this is a fortiori the case for the many mutual links between different software packages. Many software suppliers also indicate this by mentioning it in their terms and conditions, that the customer automatically accepts a release when they put the software into production.
For the medium-sized organizations, however, it’s difficult to bear that responsibility, not only because of the cost but sometimes also because of externally imposed deadlines, think of adjustments within legislation, new policy by the government or European guidelines, et cetera.
If one realizes that the customers of a similar software developer do the same test effort on the same moment, then there is also a solution to the problem above. By merging these efforts and placing it under professional direction, the test approach and software quality are improved for an acceptable price. This approach is in fact a professionalization that requires close cooperation between software users and suppliers.
Affordable and good quality testing is available for the medium-sized organizations.
The legal division of responsibilities. The test organization can in this approach not give a 100 percent coverage for the quality of the tested software. Because the test organization is working for multiple organizations, each with its own production environment, multiple test environments have to be maintained. The management of licenses can be costly if the software developer doesn’t want to support this approach and create obstacles in the form of high licensing fees.
The following basic requirements are needed for decent testing. Medium sized organizations want to suffice to these terms, but usually they can’t.
Without a measurable acceptance an information system can’t be accepted with a foundation. But in reality during the selection and purchase of software the extensive programs of demands is hardly refashioned to the measurable acceptance criteria of the ISO-9126 quality attributes: functionality, reliability, usability, efficiency, maintainability and portability. For example: errors in the blood group of a patient are unacceptable, errors in a postal code are mostly just annoying.
In reality when transferring software to the production environment settings often have to be modified. There is a real probability of creating unwanted errors, while the chance of quickly discovering them is minute, especially when an erroneous test database gets connected with in there a recent copy of the production database.
Only with this approach a systematic and planned consideration of satisfaction of the information system can be made. Testing by simply finding some usable situations from the production database, is clearly inferior and often insufficient of depth. However, the preparation of test scenarios is labor-intensive and costly, and therefore done too little.
Testing is too much manual labor while systems are getting more complex and need more test actions. Automated testing of new releases saves a lot of work, performance testing can hardly be done without automation and even the possibility to change system dates can save a lot of work. However, most medium sized organization only have a limited toolkit of test aids.
Testing usually is carrying water to the sea because a new release contains errors and bugs that were solved at a prior release, after much testing. Then serious conversations should be made with the software supplier about the development procedures. And if you tested you’ll have all facts on the table.
Drs. René Ceelen (rceelen@testmonitor.com) is the founder of TESTMONITOR, the specialized tool for software testing and implementations.
Drs. Jaap van der Wel (jvdwel@comfort-ia.nl) is director of Comfort-IA and a member of the standards commission for Information security in the Healthcare industry.