We’ve heard considerable discussion regarding the delay in availability of Covid-19 testing in the United States. The laboratory test method validation and verification process is part of the delay. Here are some other reasons test kits were not available immediately:

• SARS-CoV-2 was a new virus and testing kits and reagents did not exist
• it took time to identify exactly what the virus was
• developing new test methods takes time and money
• approvals from regulatory agencies takes time
• laboratories must ensure prior to patient testing that the test method is accurate
• manufacturers must scale up production and prioritize which customers receive the test kits based on need

In this blog, we’ll discuss the method validation and verification process.

It’s like making cookies

A laboratory cannot simply start testing for something if the method does not exist. It would be like trying to bake cookies and not knowing what to use or even what kind of cookie you’re trying to make. You need to identify the cookie you’re making. You need to identify what ingredients you’ll need. You need to acquire the ingredients. You need to create a recipe from scratch. You need to bake many batches of cookies with this recipe to ensure it gives the type of cookie you’re trying to make. You need to ensure the recipe does not make people sick. And finally, you need to make consistent batches of cookies based on the new recipe that you can share with others.

Lab tests work the same way. You need to know what you’re testing for (Covid-19 or some other lab test), need to acquire the ingredients to make the test (reagents, quality control, calibrators, supplies), develop the test (create your recipe), run the test many times with quality control and specimens to verify it is giving the expected result (make sure the recipe gives you the right kind of cookie), ensure the test does not give you false results (the cookie should not make anyone sick), and finally that it gives you consistent results that you and the health care providers can rely on to diagnose a patient.

Federal Regulations

As laboratory professionals, our responsibility to patient safety is evaluating, validating, and verifying each test method as we add it to the laboratory test menu. Many regulations, accreditation standards, and industry guidance documents exist to ensure the laboratory performs method validation in a scientifically sound manner.

The federal government regulates how laboratories perform the process. The Code of Federal Regulations (CFR) Title 42 Part 493.1253 explicitly lists actions to be performed and documented for methods approved by the Food and Drug Administration (FDA) and for methods modified by or created by a laboratory on its own. The regulation states these actions must be completed prior to any patient testing, hence the delay in availability of tests. In the case of Covid-19, many were laboratory-developed tests, requiring a more rigorous evaluation.

The CFR states for laboratory developed tests, the following performance characteristics must be specified and proven acceptable prior to testing patient specimens:

• Accuracy: how close your results are to the expected value. This is like having all your arrows hit really close to the bullseye on your target.
• Precision: how close your results are to each other. You can reproduce essentially the same result when you run a specimen multiple times. This is like having all your arrows hit near the same place on the target, but it is way at the edge of the target, not necessarily near the bullseye (although you want them to be near the bullseye).
• Analytical sensitivity: the lowest amount of analyte that can be detected reliably by the method. If you are testing for potassium and your method cannot detect potassium below 0.3 mmol/L (let’s pretend that’s a tablespoon) but you know your specimen has at least 0.1 mmol/L of potassium (let’s pretend that’s a teaspoon) because you put a teaspoonful of potassium in it, your method will not detect the teaspoonful, because it needs at least a tablespoonful to detect that potassium is present.
• Analytical specificity to include interfering substances (lipemia, hemolysis, icterus, drugs): the ability to truly detect what you want to detect. If you are testing for cocaine, but your test detects glucose, then the analytical specificity for cocaine isn’t so great. You want a test designed to detect cocaine to detect it when it’s there, and not detect it when it’s absent. You also don’t want it to give a false cocaine result when something else is interfering, which lipemia, hemolysis, and icterus can do.
• Reportable range of results: the full range of results that the method can reliably report for the analyte. Inside the reportable range, the results are accurate and precise. Outside the range, the results are inaccurate and cannot be trusted. Results outside the reportable range must never be used for patient care.
• Reference intervals (normal values): the range of results that is considered normal or expected for a healthy person. If results fall outside this range, the provider would try to determine why and treat you to get the result back into the normal range the next time you get your blood drawn.

Clinical Laboratory Improvement Amendments (CLIA)

Each laboratory that performs human testing must be licensed with the government according to the Clinical Laboratory Improvement Amendments (CLIA). Part of that licensure includes a Laboratory Director who is legally bound to uphold the federal regulations for every aspect of the laboratory operation. This includes method validation.

The Laboratory Director must oversee the development of and must approve the method validation processes for the laboratory she/he oversees.

Clinical & Laboratory Standards Institute (CLSI)

Guidance for meeting the federal regulations is available in the form of industry-accepted standards from a respected organization. A free document is available from the Clinical & Laboratory Standards Institute (CLSI) that provides a framework for evaluating, validating, and verifying a test method from the point of design and development by the vendor through verification in the clinical laboratory and ending at the retirement of the method (EP19: A Framework for Using CLSI Documents Evaluate Clinical Laboratory Measurement Procedures). Let’s focus on the guidance documents from CLSI for method validation and verification by the end-user in the clinical laboratory.

Keep in mind, not all of these documents will be appropriate for each method. Clinical Chemistry methods have different requirements than do Microbiology methods because they are reporting analytes in different ways. Clinical Chemistry results typically provide a number value for the level of analyte in the specimen. Microbiology provides identification of an organism and its susceptibility to antibiotics. Each type of analyte needs to have performance criteria established based on its characteristics.

The following documents are helpful in establishing the performance characteristics for a Clinical Chemistry method and are available for purchase from CLSI:

• EP05-A3 | Evaluation of Precision of Quantitative Measurement Procedures
• EP06-A | Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach
• EP07 | Interference Testing in Clinical Chemistry
• EP07 Supplement (EP37) | Supplemental Tables for Interference Testing in Clinical Chemistry
• EP10-A3-AMD | Preliminary Evaluation of Quantitative Clinical Laboratory Measurement Procedures
• EP15-A3 | User Verification of Precision and Estimation of Bias
• EP28-A3c | Defining, Establishing, and Verifying Reference Intervals in the Clinical Laboratory

Step 1: Calibrate & Run Quality Control (QC)

Typically, the first thing a laboratory does for a new method is load the reagent, calibrate the method, and run quality control (QC). If the QC is within the expected range, gathering patient samples for parallel testing with the current method can begin.

Some institutions require review of the specimen collection process by the Institutional Review Board to ensure patient rights are not being violated and/or to obtain patient permission to have their specimens used in this fashion.

Step 2: Run Patient Specimens in Parallel

The Laboratory Director will determine the minimum number of patient specimens to be analyzed to satisfy statistical limits for linearity between the old and new method.

Typically, 20-50 specimens are analyzed that cover the full reportable range for the method. The results for the old method are plotted against the results for the new method and linear regression is performed.

The slope of the regression line should meet a 95% confidence interval, usually 0.97-1.06 (Tietz, 8th edition). The results plot should show minimal outliers, and the outliers should be investigated to see if including them in the data set is appropriate.

This data may also be used to verify the reportable range. The Results should be linear within the reportable range and you can expect to see nonlinear results either above or below the range.

Final determination of acceptability must be made by the Laboratory Director and his/her approval documented.

Step 3: Accuracy & Precision

Quality control material is run according to manufacturer’s instructions daily for approximately 30 days. The data points collected are used to calculate the mean and standard deviation to compare with expected values for accuracy and precision if assayed control materials are used. These statistics are used to set the laboratory’s acceptable range for each quality control level for each analyte.

The Laboratory Director is also responsible for setting acceptable criteria for accuracy and precision and must document approval.

Step 4: Analytical Sensitivity

Analytical sensitivity can be verified by analyzing specimens at the lower end of the reportable range. If the new method provides results at or below the reportable range when expected, the sensitivity of the assay has been proven.

Step 5: Interfering Substances

Interfering substances evaluation is primarily done using manufacturer’s claims because it is difficult to obtain the interfering substances to test at the end user’s clinical laboratory.

For example, if cocaine interferes with a method and causes falsely elevated results, the laboratory has no way of legally obtaining cocaine (unless it has a controlled substance permit – which most labs don’t have) to mix with specimens to analyze the possible interference.

The majority of laboratories can evaluate icterus, hemolysis, and lipemia by performing studies mixing normal specimens with icteric, hemolyzed, or lipemic specimens, then calculating the expected result and comparing to the result obtained when the mixed specimen is analyzed.

If for example, the manufacturer stated hemolysis increases a glucose value, you would see a higher result from the analyzer than what you expected based on your calculation. This would prove the interference from hemolysis.

Laboratories must make available to health care providers information regarding interfering substances as needed to properly interpret the results.

Step 6: Reference Intervals

The final item the laboratory must verify prior to patient testing is the reference range for the patient population it serves. Most of the time, this is done by analyzing patient specimens from known healthy outpatients who consent to providing an extra specimen for the analysis.

This process may need to be reviewed by the Institutional Review Board prior to obtaining the specimens. The laboratory must also implement a process for addressing unexpectedly abnormal results so that a provider may follow up with the patient for care, if needed.

The theoretical minimum number of patients to include for a 95% reference interval is 40 (Tietz, 8th edition). This means that 2 of the “normal” patients may have results that fall outside the range set for the other 38, and the reference interval is still valid for the patient population. Of course, the more specimens analyzed, the better the statistical data set and increased validity of the reference interval. 100-120 specimens may be recommended by the Laboratory Director for each reference interval needing verification to support a statistically sound interval.

Method Validation Document

Once all this testing is complete, the data is compiled into a validation document for the Laboratory Director to review. Data interpretation and evaluation narratives are included. If the information supports acceptable method performance, the Laboratory Director will sign and date the document noting her/his approval and release the method for use in patient testing.

Final Outcome: Patient Safety

All this takes a considerable amount of time and effort to implement testing for patient care. Keep in mind, laboratory professionals care about our patients just as much as nurses and physicians do. We show our caring by providing accurate laboratory results so our patients receive the correct diagnosis and treatment. We take the time to get it right the first time — to keep you safe.