Regression Testing an ITSM tool Avoid the pitfalls and heart-ache of a botched ITSM software upgrade by utilizing some basic principles of regression testing the latest release of your helpdesk software. What is regression testing? In a software context, regression testing refers to testing the latest release of a software product to ensure that it is stable, functional and does not “break” any existing integrations with other systems such as reporting, custom code or other business processes. Why regression testing is needed Any progressive ITSM / help desk software will be updated over time. Depending on the vendor and their particular product release and lifecycle management cycle and processes, releases will usually fall into the following categories. Maintenance (bugfix / performance) Point release (maintenance + minor enhancements) Major release (all of the above + major new features) Due to the fact that ITSM / helpdesk software is usually business-critical software, it is important to approach any upgrade, install and configuration with due diligence and consideration.A bad install, a missing configuration item, a change in technical specification, or a change in an existing, or new feature behavior can quickly cause a disruption to service, and/or a reduction in the quality of an IT Service – an ITIL incident if you will… The irony of course, is if this happens to your actual ITSM software, who you gonna call? Where do you log this incident, when your incident tracker is dead? Don’t be the company that kills their helpdesk, with the very software that provides the helpdesk. It’s not good for business…or careers. Here’s some things to consider when upgrading your ITSM / helpdesk software Consider your testing environment A full range of tests against your ITSM system will no-doubt include creating new incidents, problems, email automation, web, automation triggers, API integration etc etc etc. Consider using a dedicated test platform and database to perform at least some portion of your regression testing against. This platform should include specific test-only email addresses, test-only Active Directory structures, test-only clients, site, assets etc. Don't fall into the testing trap of running tests again a copy of your live database/configuration, only to find that you just send email out to real people, and real organizations and/or modified your Active Directory contents, or worse. Build this test platform up over time so that it can successfully emulate and run a wide range of tests. An alternative to this, may be to run scripts against your live configuration/database to blank-out email addresses etc, and/or update them to specific test-only accounts. Are you entitled to use the latest version? Check to see whether you are actually entitled to use the latest version. When new versions of your software is released, use the opportunity to review your licensing entitlements and product availability. Are you currently within a maintenance contract, or otherwise entitled to actually implement and use the software? Verify with your vendor if unsure. Always read the release notes When new software is released, there should always be some documentation associated with it. Release notes vary from vendor to vendor. Some release notes are comprehensive, explanatory and helpful. Others are just terse descriptions of fixed errors and issues, no-doubt written by the developer that performed the code update. Whatever style of documentation you have, read it thoroughly and understand the implications. Discuss any concerns with your vendor. It may be the case that you don’t need to upgrade to a particular release. If there is nothing in the change-log that directly affects you, consider not upgrading to it – why go through the hassle of upgrading for no discernible benefit!? Scan the documentation for specific issues that affect you. Verify the technical specifications – have they changed? Updated software, can often mean updated specifications, particularly so with major feature releases, or new products. Have the technical specification changed? Things to check include: Machine specification (processor, memory, drive configuration etc) Windows version, service pack. Microsoft .Net Framework version Microsoft Internet Information Server version IIS. Java requirements SQL Server (including Reporting Services) Crystal Reports (doesn’t play well with other versions of itself) Miscellaneous 3rd party libraries Email systems – IMAP, SMTP etc Encryption libraries and technology If unsure about the requirements and specifications, check with your vendor, and discuss with your technical team to ensure that all requirements are met. Custom development API / SDK – Integrations with other systems Do you use any custom-developed code libraries for your solution that were developed against the product API/SDK or web-services? Custom code is often developed and used to integrate other systems, and/or provide functionality to other operational areas and personnel. Oftentimes, custom solutions are vitally important to the business areas utilizing it. Things to check include: Read the API documentation. Give a copy to your developers to review. Has the API been updated? Have methods and functions been deprecated, or changed in some way? Discuss any changes with the developers of the code to understand the implications. You may need to re-write code before any update takes place. Your developers wrote test scripts right? (insert chirping cricket sounds here…) Run these against the upgraded API. Do all functions and methods work correctly? Verify this with key people from the business areas that are affected. Reports and reporting Updates to software often mean updates to the underlying database structure. This may affect any custom reporting or data access that may have been developed against the database schema. Things to check include: Do you utilize any custom reporting solutions? Has the database schema changed? How? Is there an updated data dictionary? Are you using a database reporting abstraction layer to produce reports? BI solution/framework? Will this reporting layer require updating? Test all reports, ETL, and other database scripting against an updated database Database upgrade / migration Updates in software often requires an update to the underlying database schema, or (worse), a data migration to a new database. This is an essential area to test and check. Create a backup of your existing live database. Use this database to test the upgrade process. Check with your vendor first to see if there is anything you need to know about this before performing it. If the database upgrade fails, or provides warning/information, research these and find solutions before progressing further with the upgrade and testing. Even though a database upgrade/migration may progress without issue, verify the results by looking at your upgraded data through an updated version of the software. Are all the jobs/tickets/incident intact? Is everything still there, and OK? Verify this will members of the team that know the data. A DBA, or a technical person doing the upgrade won’t necessarily be the best person to verify data integrity. Test the known issues Test the issues you have experienced and are of interest to you. Only you know the bug/issue, especially so if you reported it in the first place. Test the bug and see if it’s really fixed. Re-run your test cases. Read through your documentation on the issue. Talk to other people about the nature of the bug. Often times, bugs and anomalies will be found by the front-line staff who have a tendency to use the software more rigorously than 2nd level staff, or technical personnel. Get them involved – they will tell you whether the issue is truly resolved or not. Understand and test new features Get to know the new features of the software. Seek to understand the following questions about each feature. Why was it developed? What does it do? Does it make other features obsolete – is there now a better way of doing something? How can you utilize this new feature? Will this new feature require group discussion, planning and configuration? Will this new feature require training? Do we need this new feature? Is this feature ready for use? Has the vendor implemented this correctly? Discuss new features and opportunities with your team. Test for accessibility Do you have special needs for your software? If you do, test for these things, and involve the people that know what to look for. Test for high visibility, screen reader integrations, access keys, appropriate imagery etc. Document everything Throughout the installing, testing, and configuration of any software, be sure to document your experience. Every environment is unique and there is a lot of environmental and configuration data that needs to be recorded. By continually documenting and improving the documentation of your ITSM / helpdesk software installation, configuration and upgrade processes, you’ll be building a valuable reference guide not only for your organization, but for anyone who will require this information in the future. Get help If at any time you require assistance, have questions or need to clarify any matter, contact the vendor of the software. The people that build the software (should) know it inside and out, and they (should) be able to provide guidance and support throughout your testing and upgrading experience. Consider even getting your vendor involved on-site to assist with any upgrade, testing and training needs. You may find that you can save a lot of time and trouble by leveraging the know-how of your vendor. Conclusion Upgrading software to the latest and greatest release is an exciting time. Bugfixes, new features and increased performance can really grease the wheels of your IT service delivery and provide benefits to your team and the clients you support. View upgrading software as a natural, and on-going part of the software development cycle – it’s a good thing. Be pro-active about it, and plan for it. Embrace the upgrade! By adhering to a few basic principles and approaching the transition with due diligence and care, you’ll find that not only will your upgrade progress smoothly, but you’ll also continue to build a valuable foundation of product lifecycle management, documentation and process maturity.