Computer System Validation (CSV) is the validation process used in regulated industries throughout the world to verify that a computerized system performs as it is designed to and that it maintains the integrity of its data to ensure the safety and effectiveness of the product. In the United States, the Food and Drug Administration requires regulated companies to perform validation of computerized systems that operate or support the production of the regulated products, including Pharmaceuticals, Biologicals, Medical Devices, Infant Formulas, Blood and Blood Components, Human cell and tissue products, etc.
Perform CSV when implementing a new computerized system or when you make a change to a validated computerized system. CSV processes are validation processes, based on applicable regulations and guidance documents, industry best practices, and the intended use of the computerized system being validated. In this post we are going to discuss a few best practice recommendations for effective, risk-based CSV.
Computer System Validation 101
FDA defines software validation as “Confirmation by examination and provision of impartial evidence that software specifications follow user needs as well as proposed uses, and that the precise requirements implemented through software can be consistently achieved.” In the FDA regulated world, a “computer system” is not just the computer hardware and software, but the entire system, including any equipment or instruments connected to the system, as well as the actions of the users that operate the system, and the Standard Operating Procedures, manuals, and documentation.
CSV ensures that both new and existing computer systems are suitable for their intended use(s), produce correct and trustworthy results that enable regulatory compliance, meet the User Requirements Specifications, maintain required audit trails, and provide the ability to detect invalid or altered records.
The CSV process uses a variety of static and dynamic testing activities that you should conduct throughout your entire software development lifecycle, from system conception to retirement. You should plan to validate that your system will work in all the situations that you want the system to work in. Write a CSV Master Plan. Document your CSV processes. If you didn’t document your CSV, your CSV didn’t happen.
Plan your CSV activities using some or all of these elements based on your particular computerized system and its intended use(s).
System inventory and assessments determine which systems need to be validated.
System boundary documents determine where your computerized system starts and ends.
Validation plans define the purpose, scope, and plan for the validation study.
User requirement specifications explain what the system should do. For example, “I want to drive 6 people from Reno to Las Vegas (438 miles) in not more than 7 hours”.
Functional need specifications explain how the system will look and function for the user to be able to achieve the user requirements. For example, “The vehicle speed has to be at least 80 miles per hour.”
Validation Risk assessments analyze the failure scenarios in order to determine the scope of validation efforts. For example, “Speeding ticket, flat tire, etc.”
Validation Traceability Matrix references each specification to one or more specific validation tests in the Installation, Operational, and Performance Qualifications.
Network and Infrastructure qualification verifies that the infrastructure, hardware, and software supporting the application system being validated have been installed properly and are working correctly.
Installation Qualification Protocols, Test Scripts, and Final Reports demonstrate that the system has been installed correctly in the user environment. For example, “What kind of car did I buy?”
Operational Qualification Protocols, Tests Scripts, and Final Reports demonstrate that the various functions of the system do what they are intended to do in the user environment. For example, “Do the tail lights work? Do the brakes work?, Can I do at least 80 miles per hour?, etc.”
Performance Qualification Protocols, Test Scripts, and Final Reports determine that system does what it is intended to do along with qualified people following SOPs in the production environment even under the worst case. For example, “I drive me and 5 ‘people’ from Reno to Las Vegas. Did I make the trip in less than 7 hours?”
Validation Reports include reported results, deviation resolutions, and conclusions of all activities in the validation study.
System Release Documentation documents that validation activities are completed, that the system was released by the authorized quality function, and the system is available to use.
What Is Computerized System Validation (CSV)?
Computerized System Validation (CSV) is defined by the FDA as “confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled” – General Principles of Software Validation: Final Guidance for Industry and FDA Staff.”
In layman’s terms, CSV is the line of work where regulated companies validate their software applications by executing different validation projects in order to prove their software is working properly.
Why Is Computerized System Validation Important to My Business?
There are many reasons as to why Computerized System Validation is important, especially if you work in a highly regulated industry. If your business does fall into that category, it is likely you’re familiar with the validation of methods, processes, equipment or instruments to ensure your science is of high quality. CSV is no different. CSV is integral to ensuring the quality and integrity of the data that supports the science. If the FDA or any other regulatory body inspects your company, you can guarantee they will check on this.
Common Computerized System Validation Mistakes
The whole goal of CSV is to prove that computers and software will work accurately on a consistent basis in any situation as it complies with relevant regulatory bodies.
The timeline of CSV testing activities is never ending. CSV happens throughout the whole software development lifecycle (SDLC) – from system implementation to retirement.
With that being said, this leaves ample opportunity for error. Some of the more common ones in the industry include:
- Poor Planning – We run into poor planning issues when there are insufficient resources and inaccurate timelines.
- Inadequate Requirements – Typically, we see too few, too many, too detailed, or too vague.
- Test Script Issues – This is commonly seen with execution errors, inadequate testing, poor test incident resolution, over reliance on vendor testing.
- Project Team Issues – Associated issues include poor buy-in from all stakeholders, unavailability of key personnel at key times.
- Inadequate Focus on the Project – Resources often are pulled off to their day jobs, insufficient managerial support.
- Wasting Time on Low Value Testing Activities – There are typically inadequate risks and critical assessments.
If you have been following this blog series so far, I have discussed my perspective on why I feel so many project managers are unsuccessful at their attempts to earn their CAPM® or PMP® certifications. In the previous edition of this blog, I expounded upon the importance of developing a plan at the outset of your CAPM® or PMP® journey. While excitement and drive may be enough to carry someone through the first phase of their journey, without a solid plan most people will fall short of their certification goals. Although I don’t have statistics to support my assertion, I would bet that are probably more CAPM® and PMP® candidates who fail to pass their certification exam as a result of failure to perform in the middle phase of their study plan – that phase in which they cover the “meat” of the PMBOK® material – than there are those that fail because of their performance on during the certification exam itself. They will make a great showing at the start where there is a lot of help available and many people are interested in helping to set them up for success. Similarly, if they manage to complete their training requirements and announce that they are ready to sign-up for the exam, they will likely experience a wave of support and encouragement from their managers, trainers and peers. However, if they enter into the examination phase without a solid foundation provided by the middle phase of their study plan, their success on the certification exam will hang in the balance.
The key to success on the exam lies not in developing a bag of tricks to release on exam day, but in finding a way to bridge the gap between those the first and last phases of your training, the phase which I refer to as “The Belly of the Beast” in my PMP® training plan. I believe this middle phase to be both the most difficult phase of certification process as well as the phase where you can shine as a soon-to-be project manager. In this phase, your mental toughness and dedication will be challenged as you attempt to balance your study regimen with your other commitments at work, at school and at home. Succeeding in this phase will require re-prioritization and re-alignment of your commitments in order to arrive at a regimen that is both robust enough that it can be sustained despite the inevitable distractions that will arise over the course of your studies but that is also malleable enough that it can accommodate new categories of information from the later part of the PMBOK® Guide – both of which are skills that are expected from a certified PMP® / CAPM®.
So, let’s not delay it any longer – it’s time to get to work.
The Spiral Method: Studying with a Purpose
My high school psychology teacher, Mrs. McBride, first introduced me to what she termed “The Spiral Method” of teaching and studying during her Brain and Behavior / Advanced Placement Psychology course. With this teaching method, the teacher and students purposefully revisit the course material multiple times during a unit, engaging the material is a different manner or with a different learning modality in each pass. For example, for the introductory unit which focused on brain anatomy, we started off by reading about the anatomy of the brain in the textbook. In the second pass, we created a table of the parts of the brain in our notebooks and developed pneumonics to help remember each of their functions. In the final pass, we watched a video-series that detailed the equipment and the methods scientists use to study the brain as well as some experimental footage of subjects with traumatic brain injuries performing common, everyday activities while simultaneously being monitored by a functional magnetic resonance imaging (fMRI) machine.
The goal of this first unit was two-fold: to show the students that brain physiology was a very real topic – something that could be both observed and measured – as well as to prime us for the teaching methods that we would encounter for the remainder of the course. As the semester progressed, I came to appreciate Mrs. McBride’s teaching style more and more, and none more so than when the College Board’s Advanced Placement (AP) tests came around later that year. After having spiraled through the course content multiple times, the AP Psychology exam felt just like another leg of the learning spiral and expressing my understanding during the exam felt more like a formality than a chore.
That memory has stuck with me for all these years, so when I was staring down at my newly-purchased PMBOK® Guide back in 2016 and about to start my own PMP® journey, although there were many things of which I was unsure of in that first week or two, there was one thing that was certain: I wanted to experience that feeling again – that feeling of knowing that when I sit down in the exam room several months from now and first open up that test booklet, earning my project management certification at that point will just be a formality. I wanted to feel that I had given my best during my studying of the PMBOK®, had spiraled through the material until I knew it inside and out and had erased all feelings of doubt from my mind.
So, what does it take to implement a “Spiral Method” approach in your studies for the PMP® or CAPM® exam and the confidence boost that accompanies it? Let’s dive right in and examine the fours legs of this exam preparation strategy in more detail.
The First Leg: An Introduction to the Course Material
Consider your first pass to be an introduction to the course material. You could start off by reading the chapter for one of the knowledge areas in the PMBOK® Guide and then summarizing the main points in your study journal. Make notes on the key concepts, the skills and techniques, the vocabulary and key terms as well as the important formulae and figures as you read. Don’t worry if you don’t understand everything being presented in this this first pass, but do be sure to flag items that were not clear and that require further explanation or research.
The Second Leg: Collaboration and Engagement
For your second pass at the course material, attend or watch the lecture on the PMBOK® knowledge area or topic as part of your classroom training. Collaborate and engage with the presenter in person during their “office hours” or through e-mail or other communication method to discuss and debate the material. In particular, be sure to ask about the items that you flagged in your first pass in order to clarify your doubts and to bolster your understanding. Update your study journal and expound upon your new-found knowledge and understanding.
The Third Leg: Application of Knowledge
Although the first and second passes in the spiral method are fairly basic and familiar to most people from their school days, the third pass is where things begin to get uncomfortable. Fear not, because the same rule applies in academia as it does in personal fitness: the improvement is in the struggle. In this third pass at the course material, the focus should move towards the application of the knowledge you have learned, rather than simply re-reading content over and over again. This leg of the spiral demands a shift of focus: a shift away from the trainer and towards the student as the driver of their own learning – a sensation that is uncomfortable for most and a sensation that many professionals have forgotten from their school days.
When I take my third pass at the course material, it often begins something along these lines: imagine for a moment that you weren’t studying for an exam. (What a relief!) Instead, put yourself in the shoes of a project manager about to use this new information on a project or in the shoes of a trainer about to teach the next group of project management recruits about the significance of this information. What tool or template would you distribute to your project team to be sure that they put the best practices from this knowledge area into use in their work? What reference document or study guide would you give to your trainees to help them grapple with this information? Try your hand at creating that tool, template or summary document – perhaps something along the lines of a CliffsNotes® or SparkNotes summary. As you work your way through creating the reference guide, based on your ability to create a succinct and accurate summary of each topic, you will be able to gauge your level of understanding more accurately and you will quickly discover which topics you are already confident in as well as discover those areas in which you still need clarification.
If you are stuck and you are not sure what type of tool or reference to create, you can always try the school teacher’s method for helping students to learn to summarize large volumes of information in very a limited space – the “cheat sheet”. You could attempt to summarize everything a PMP®/CAPM® student could want to know about a particular topic or knowledge area on a single 3” x 5” index card, or the entire PMBOK® on one 8.5” x 11” piece of paper in order to make a “cheat sheet” such as the one a high school teacher or college professor might let a student bring into a final exam.
Although most students think their teacher is letting them off easy, the teacher knows the student won’t be able to fit everything they needed to know to take a final exam in such a limited space. The challenge (and benefit) of this assignment lies in forcing the student to distill the target material into its most basic form and in creating memory cues that the student can use to recall or to re-create the information that they will need on the spot in order to work through a problem on that topic during the exam. When done correctly and with purpose, the most mindful students will barely need to refer to their card during the exam save for a quick check of a formula, as it is the process of creating the reference card that helps to engrain the information in the student’s memory. (And here you thought your teachers were throwing you a bone for all these years!)
The Fourth Leg: The Broader Implications
In your final pass at the material, you should begin to widen your perspective and to consider the broader implications of the material you are studying. One way to do this would be to test out your study tool or reference by using your reference to teach another person about the PMBOK® topic. It could be a fellow project management student, your project management mentor, an existing PMI® credential holder, or someone else well-versed in the project management topic. Share your study tool with them and ask them for their feedback on it: Is the tool easy to use? Are my explanations clear and accurate? Are there any important topics I have left out?
You could even carry out the exercise with a study buddy who is not so strong in project management, such as another project management student who is still studying for the certification exam themselves, or better yet, a colleague or an unsuspecting family member who is not involved in project management at all. Try explaining the PMBOK® concept to them using only the tool or reference that you created. If your cross-disciplinary study buddies are anything like the friends and family whom I called upon to assist me with my studies, you will find that they bring a perspective to the table that is quite refreshing. Rather than challenge your ability to memorize the nitty-gritty details, such as definitions, formulas, and calculations, they will likely ask you probing questions such as: “Developing a project management plan and all its smaller plans seems like a big waste of time that takes attention away from the actual project work – why should I even bother making any of those plans at all?”. Although it may not seem like it at first, it is a very valid question that challenges your understanding of the logic, the purpose and the reasoning behind the project management plan and the role that the planning processes serve on a project.
After your mini-lesson, test your study buddy by having them reflect back to you what they learned either through a skills demonstration or by having them try their hand at answering a few questions on the topic from the practice exam. If your study buddy is able to properly explain the concept or can demonstrate proficiency in using your tool to answer the practice exam questions, then you know that you are proficient on that topic and are ready to take the exam on that section. If, on the other hand, if your study buddy struggles with the concept, or if their explanation is incomplete, or if they are confused by your reference tool, then you should consider it as a reflection of your own understanding of the topic – they couldn’t demonstrate proficiency on the topic because you didn’t know understand the material well enough when you presented it to them. You must devote more time to or try a different approach with that topic in order to iron out the wrinkles from your own understanding prior to sitting for the certification exam.
Although it may seem like a lot of work, after a bit of practice the “Spiral Method” of studying can become second nature. When approaching a new topic in the PMBOK®, especially those that are very large and complicated, start by breaking the topic into more manageable chunks and then applying your study approach. Customize the legs of the spiral and alter the number of passes you take at a topic as needed based on your level of experience and familiarity with the topic. As many project management skills cut across multiple PMBOK® knowledge areas and the techniques used to analyze their performance are similar (e.g. time, cost, and scope management), consider double-dipping to incorporate several knowledge areas into a single activity in your later passes through the spiral. The payoff from following the spiral method is enormous as it can give you the sense of control over the material that is elusive for many project management students.
In the next edition of this blog, I will explore how to build upon the knowledge you have gained through your studies and how to use your past experiences to boost your level of confidence and to set yourself up success on the certification exam. I hope you’ll join me.