Covid-19 rapid antigen detection kits and CMPT PT samples

Early this year, CMPT in collaboration with BC Centre for Disease Control (BCCDC) created a COVID-19 proficiency testing program geared to point of care (POC) testing sites. As testing became widespread and rapid testing systems began to proliferate, the College of Physicians and Surgeons of British Columbia, Diagnostic Accreditation Program (DAP), required these sites to be accredited.

Participating in a COVID-19 proficiency testing program is one of the requirements for this accreditation.

CMPT’s COVID-19 PT program

CMPT created a simulated sample using inactivated (non-infectious) material prepared by Dr. Martin Petric (BCCDC) from a virus culture lysate that has a concentration of ~6×10⁴ (Median Tissue Culture Infectious Dose) TCID50/ml.   The inactivated virus, as measured by PCR has a Combining time (Ct) of approximately 13 which signifies a very dense concentration of viral nucleic acid.  It also contained ample concentrations of the protein antigens produced by COVID-19 virus that were determined to be useful detection targets for making a laboratory confirmation of the presence of the virus.   Because of the mixed proteinaceous content of the culture lysate, it is not possible to easily define the precise concentration of the specific proteins, and so by convention, the PCR Ct is used as a surrogate measure of the protein.

In order to develop a useful proficiency testing scheme, we adopted a plan to send samples with different concentrations of the viral material to better represent clinical samples.  It was not in the interest of CMPT or laboratories to create a sample that would be too close to the limit of detection of any testing system.  Two concentrations were chosen with the strongest being a 1:10 dilution of the material provided and the second 1:50 (referred to as medium positive).  A third sample which did not contain any viral material was developed to give a negative result but test positive for RNAse P.

Figure 1. Testing results for Covid-19 PT samples using the Abbot Panbio RAD test. A: strong positive; B: medium positive

Both positive samples (called strong and medium) have a Ct value of <19 when tested by PCR, which is well within the accepted threshold for test positivity (Ct less than 35); when tested by rapid antigen testing assay the strong positive sample would present a strong band on the testing area of the test while the medium positive sample would produce a weaker but definitely positive band (Figure 1).

The different samples were tested for stability at different temperatures during a period of at least one week and up to 14 days since some of the POC testing sites are in remote areas and would experience delays in receiving the material.   Both positive samples showed to be stable at room temperature and 4°C for at least 14 days, and at 35°C for at least 7 days.

Proficiency testing results

The first survey of the CMPT COVID-19 PT program was sent in March 2021 and a second one was sent in May 2021. Each survey consisted of a three-samples set.

The two predominant rapid testing methods included Abbott PanBio (Antigen) and Abbott ID Now (by thermostable Nucleic Acid amplification). Additional methods were also used and included in the results but were present at a sufficiently low number that they could not be fairly analyzed by brand within the larger pool. As the number of test sites increases, the additional methods will be identified if their number increases.

After two surveys, data collected showed important differences between the RNA methods and the Rapid Antigen Detection (RAD) methods (Table 1).

Table 1. Results obtained by the different testing methods after two PT surveys. Acceptable responses

The strong positive sample was easily identified by both methods; 80% of the sites using RAD methods reported the sample positive compared to 92% of sites using RNA methods.

There were no issues observed with negative samples for any of the methods. All sites reported the negative sample as such with the exception of a few laboratories that reported “inconclusive” mostly because of operator error.

100% of the sites using RNA methods reported the medium positive sample as positive however only 34% of the testing sites using RAD methods correctly reported a positive result.

Investigation process

The results obtained by the testing sites using the RAD methods were unacceptable considering these kits are being used across the province at many POC testing places. The lack of sensitivity observed with a sample that is clearly positive was of concern.

The first point to consider was the proper training of personnel performing these tests since many of them may not have laboratory experience. This was tackled by some of the Health Authorities that sent trainers to oversee the testing using replacement samples.  It was determined that some of the testers were not attuned to seeing the fainter bands, but with training, they became much more adept.

Of some concern, even with supervision, some sites testing medium positive samples still were having difficulty seeing a faint band on the testing area of the RAD testing cartridge. This was puzzling because we would run the same sample at CMPT and have a clear positive result.  This led us to considering other possible explanations for inaccurate readings, which might include other issues that could impact the quality and stability of the sample such as elevated temperatures in transit, delays in testing, and/or sample manipulation.    We were unable to identify any single factor, as the medium positive sample always tested positive in our hands in all these conditions when tested with the Abbot Panbio Rapid Antigen kit.

Lastly, after finding out from different testing site that the RAD kits were stored at room temperature –while CMPT stores the kits at 4°C– we decided to test if storage temperature for the RAD kit affected the ability of the test to detect the medium positive samples.

In order to test this, we took some cartridges, tubes, swabs, and reagents from a kit stored at 4°C and placed them in a box at room temperature (21°C).  Mid positive samples were left at room temperature and tested with the kits stored at 4°C and at RT at different time intervals.

While the capacity to detect a strong positive sample with the kit stored at RT was intact after 7 days, the RAD kit appeared to have lost its ability to detect the medium positive sample after 14 days, however, these results could not be reproduced, and a sample tested positive after 21 days of the kit being stored at RT.

These findings would have been significant as the manufacturer’s instructions state that the kit should be stored between 2 and 30°C and many POC testing sites may not have a fridge at hand to store the kits. There are also high through-put testing sites such as airport hubs that have thousands of kits in storage and are not at 4°C.  What we don’t know is if any or many of the storage rooms are a lot closer to 30°C than may be appreciated.

A Panbio RAD kit was again divided and components (buffer and cartridges) were exposed to 4C (fridge), 19C (RT), 25C (water bath), and 35C (incubator), for different periods of time.

A testing schedule was created and both, strong (+++) and medium positive (++) samples were tested (Table 2).

Table 2. RAD stored at different temperatures – Testing scheme and results

The ability of the Panbio RAD testing kit to detect strong and medium positive samples was not affected after being stored for 4 days at 35°C; or 14 days at 19, or 25°C (Figure 2).

Figure 2. Testing results for Covid-19 PT samples using the Abbot Panbio RAD test stored at different temperatures. From left to right: Day 4, 35°C, strong (S) and medium (M) positive samples; Day 14, 25°C, S and M samples; Day 14, 4°C, S and M samples; Day 14, 19°C, S and M samples.

We don’t know what caused that first sample to test negative after 14 days of storing the kit at RT and consequently, we have not found a link between the storage temperature and the poor performance of the POC sites with the medium positive sample.

By coincidence, at the same time as we were looking for explanations for some invalid sample results, western Canada developed an unusual climatic phenomenon referred to a high-pressure heat dome that resulted in temperatures greatly exceeding 30°C across British Columbia, in some regions even exceeding 40^oC.  This did lead to a more thorough investigation in coordination with the BCCDC to test kits stored at different warehouses that might have been exposed to high temperatures during the recent BC heat wave.

A random selection of testing kits stored at different locations around the province was tested to verify their performance.  BCCDC performed QC on the Abbott Panbio, Abbott ID Now, and BD Veritor and all passed. CMPT tested the Panbio kits from the different warehouses and found no issues in their performance. Fortunately, no issues were found with the different testing kits as all of them performed to the expected standards.

While these results would indicate that storing the kits at room temperature does not decrease their performance when testing samples, the exercise nonetheless was useful as it highlighted the importance and value of ensuring big warehouses storing thousands of kits monitor their temperature; this was implemented after the preliminary results were discussed.

There are three points that we would like to communicate to the testing sites:

(1) The manufacturer advises that their reagents are temperature sensitive and must be stored in a way to ensure they stay within the posted range. 

(2) during the current heatwave that range was potentially breached, but investigation found no evidence of harm,

(3) organizations are advised to monitor storage facilities, especially at times of extreme temperatures to ensure the kits remain at low risk.

Veronica Restelli, CMPT Editor

Michael Noble CMPT Chair