Download
Journal of Biological Methods | 2024 | Volume 11 | Issue 1 | e99010009
DOI: 10.14440/jbm.2024.409
Research Article

Validation of 12 Rapid Antigen Tests for the Detection of SARS-CoV-2

Minghang Yu1,2,3,4,#, Yang Xiong1,2,#, Pu Liang1,2,3,4,#, Danying Chen1,2,3,4, Yuting Zhang1,2,3,4, Huan Liu1,2,3,4, Yuanyuan Zhang1,2,3,4,*, Xuesen Zhao1,2,3,4,*, Ronghua Jin1,2,3,4,*, Xi Wang1,2,3,4,*
1Beijing Key Laboratory of Emerging Infectious Diseases, Institute of Infectious Diseases, Beijing Ditan Hospital, Capital Medical University, Beijing 100015, P.R.China. 2Beijing Institute of Infectious Diseases，Beijing, 100015, P.R.China. 3National Center for Infectious Diseases, Beijing Ditan Hospital, Capital Medical University, Beijing 100015, P.R. China. 4National Key Laboratory of Intelligent Tracking and Forecasting for Infectious Diseases, Beijing,100015, P.R.China.
#These authors contributed equally to this work.
*Corresponding author:
Yuanyuan Zhang, Email: zhangyuanyuan@ccmu.edu.cn.
Xuesen Zhao, Email: zhaoxuesen@ccmu.edu.cn.
Ronghua Jin, Email: ronghuajin@ccmu.edu.cn.
Xi Wang, Email: xiwang@ccmu.edu.cn.
Abstract
The rapid identification SARS-CoV-2 virus has become the basis for the control of the COVID-19 outbreak. The rapid antigen tests for SARS-CoV-2 are quick, widely available, and inexpensive. Rapid antigen tests have gradually replaced the time-consuming and costly RT-PCR. Currently, although several RAT kits have been extensively used for the diagnosis of COVID-19, validity data are limited due to the inconsistent sensitivity and poor reproducibility. Meanwhile, WHO does not recommend specific commercial RAT kits. Therefore, it is crucial to establish a method to evaluate the effectiveness of different rapid antigen tests kits. This study aimed to develop an evaluation system for rapid antigen tests to provide an efficient and accurate technique for screening SARS-CoV-2 antigen detection kits. Given large number of rapid antigen tests kits available, this study only focused on those that are representative and commonely used in China. By minimzing biases through randomization, concealment, and blinding, we eventually found that the Test 1 had the lowest sensitivity and the Test VI had the highest sensitivity. This study provided an evaluation platform that can potentially serve as a reference for COVID-19 diagnostic strategies.
Keywords: antigen, COVID-19, rapid antigen test, SARS-CoV-2, sensitivity
Received 2023-06-29; Revision received 2023-12-14; Accepted 2023-12-16; Published 2024-01-12

© 2024 Journal of Biological Methods published by POL Scientific

1 Introduction

The coronavirus disease 2019 (COVID-19) caused by the SARS-CoV-2 virus infection, has spread rapidly across the globe. According to the World Health Organization (WHO), nearly 800 million people worldwide are infected with SARS-CoV-2, and more than 6 million died of the condition[1]. Identifying COVID-19 by testing for SARS-CoV-2 infection (COVID-19 testing) is essential for the control of the epidemic. Reverse transcriptase polymerase chain reaction (RT-PCR) has been established as the gold standard for SARS-CoV-2 diagnosis[2]. This method, however, has significant drawbacks when used for community-based asymptomatic screening, since it requires laboratory testing and delays in result reporting may lead to failure to timely isolate the infected patients. Nevertheless, the SARS-CoV-2 Antigen Rapid Diagnostic Test (RAT) is a cheaper, faster, and accurate analysis at the protein level. Prior researches indicated that the antigen RAT, as a diagnostic tool, outperformed RT-PCR, in early detection of the COVID-19 and timely initiation of treatment [3]. In many countries, Omicron variants are currently the main epidemic strains. And the antigen RATs kit can successfully detect the recombinant SARS-CoV-2 nucleocapsid protein (NP) of wild-type, Alpha-, Beta-, Gamma-, Delta-, Epsilon-, Kappa-, and Omicron-variants[4,5]. Multiple in vitro rapid antigen diagnosis kits, based on the immunochromatographic principles, are now commercially available for the detection of SARS-CoV-2. RATs can be easily performed without additional equipment or staff and provide results within 15 minutes. In view of the current global pandemic, antigen RATs play a pivotal role as countries are incrementally lifing travel bans and easing physical distancing measures. WHO currently does not recommend specific diagnostic tests for COVID-19 screening using RAT kit[6]. Nevertheless, no standards are available for comparing the performance of various kits.
We previously introduced a high-sensitivity antigen test that was based on the fully-automated light-initiated chemiluminescent immunoassay (LiCA®). LiCA® showed a suerior capability in detecting SARS-CoV-2 antigen and could detect roughly 100~5,000-fold lower levels of the analyte as compared to other rapid tests studied[7]. At present, many RAT kits are also commercially available. The differences in test performance are reported between studies using the same RAT kits and between studies that employed different kits[7,8]. Therefore, it is urgent to develop novel effective evaluation methods for COVID-19 detections. In this study, the sensitivity of various SARS-CoV-2 RAT kits was evaluated in terms of the limit of detection (LOD) using a set of serial ten-fold diluted SARS-CoV-2 nucleocapsid protein. To date, a direct comparison of the twelve RATs has been performed. Therefore, the goal of this paper was to provide a method and applications of COVID-19 detection.

2 MATERIALS AND METHODS

2.1 Analytical validation

This research was primarily performed at the National Infectious Disease Medical Center of the Capital Medical University, affiliated with Beijing Ditan Hospital, Beijing, China. Our research was in compliance with the protocol of Clinical and Laboratory Standards Institute (CLSI) EP5-A3: User Verification of Precision and Estimation of Bias[9]; and EP12-A2 guidelines: User Protocol for Evaluation of Qualitative Test Performance[10].

2.2 2019-nCoV nucleocapsid protein

Nucleocapsid Protein Solution Reference Material of 2019-nCoV was manufactured by the Chinese National Institute of Metrology, China. The sequence of SARS-CoV-2 N gene was derived from NCBI GenBank database (28274 to 29533 nt), the plasmid containing the full length of the N protein was constructed. E. coli recombinantly expressed and further purified to obtain the recombinant 2019-nCoV NP protein. The sample was finally diluted in PBS buffer solution and then dispensed into freezing tubes.
The 2019-nCoV NP protein was diluted into 100000 pg/mL, 50000 pg/mL, 25000 pg/mL, 12500 pg/mL, and 10000 pg/mL in PBS buffer. As recommended by the manufacturers, these samples were added to the lysis solution of the RAT kits and diluted into 5000 pg/mL, 2500 pg/mL, 1250 pg/mL, 1000, pg/mL 500 pg/mL, 250 pg/mL, 125 pg/mL, 100 pg/mL, and 62.5pg/mL, with supplemental dilution concentrations of 50 and 25 pg/mL added, if necessary. One hundred microliters of NP protein dilutions were loaded into the sample well of the cassette, and the test result was read within 15 min.

2.3 Rapid Antigen Test (RAT)

The RAT kits used in the present study were all commercially available in China. These kits demonstrated good clinical performance. With all kits, an immunochromatographic (ICT) format was utilized to detect viral antigens by using the immobilized coated SARS-CoV-2 antibody on the device. These labeling materials include colloidal gold and latex.

2.4 Qualitative assessment of RAT test performance

The experiments consisted of coders, operators, and outcome raters. To avoid biases, all experimental data were collected in a blinded fashion. RATs and diluted samples were numbered randomly. All of them were blinded to the RATs and samples. All participants received standardized training before the study was initiated.
The assay was carried out under the same conditions and against judgment criteria. There were a total of 8 results (4 repetitions ×2 interpretations). The overall RAT result was judged positive if outcome raters identified at least 5 positive results. Weakly and medium positive band intensity needed to fulfill 10 positive repetitions with uniform color rendering. The lowest concentration of the recombinant NP protein with weakly positive band intensity was designated the LOD of the RAT. The protocol for the qualitative assessment of RAT test performance is shown in Figure 1. Result interpretation, concentration gradient testing and repeatability test of minimum concentration for 2019-nCoV NP protein is listed in Figure 2.

2.5 Statistical analysis

All analyses were performed using SPSS 26.0 software package. Figures were generated by means of GraphPad prism 9 and Adobe illustration. The Kolmogorov-Smirnov test was conducted to test for normality. We used a two-tailed Student’s t-test to compare normally-distributed data and a two-tailed Mann-Whitney U test for non-normal distribution data. The criterion for significance was set at α = 0.05. Difference was deemed significant when a p-value <0.05.
Figure 1. Flow of the evaluation protocol for RAT qualitative test performance. The study was conducted by using a blind method. With researchers and interpreters blinded, we finally obtained eight results interpreted as positive more than or equal to 5 times, while ten repetitions were positive and showed uniform color, which we rated as positive. All other cases were deemed negative.

3 Results

3.1 Rapid comparative evaluation of SARS-CoV-2 antigen detection kits

To analyze the sensitivity and repreducibility for assessing SARS-CoV-2 virus detection kits using diluted recombinant antigen, we evaluated 12 RATs (I-XII) that were approved for clinical diagnosis in China. All information of RATs kits is listed in Table 1. The lists include all three latex Ag-RATs test (I, X, XI) and the other tests were colloidal gold tests. All tests were based on lateral flow chromatography.
The assay was performed as described in Materials and Methods section[9, 10]. Comparison sensitivity of RATs for 2019-nCoV nucleocapsid protein. The recombinant SARS-CoV-2 nucleocapsid protein (NP) was diluted to concentrations ranging from 2500 pg/mL to 50 pg/mL (refer to Table 2. and Fig. 2 for details). The sensitivity of RATs for 2019-nCoV NP is shown in Table 2 and Figure 3A. As shown in Table 2, the 12 RATs had different detection sensitivities. We found that Flowflex COVID-19 Ag RAT (ACON) tests showed a slightly higher sensitivity than others RATs, and the minimum detectable concentration was 50 pg/mL (refer to Fig. 2 and 3). Reciprocally, LEPU Technology to test 2019-nCoV antigen showed lower sensitivity than the other RATs, with the lowest detectable concentration being 2500 pg/mL (refer to Table 2). Interestingly, Latex-labeled RATs had a significantly lower detection sensitivity than colloidal gold-labeled RATs (P=0.0273, Fig. 3B). However, analysis based on the data provided by the manufacturer, there was no statistically significant difference in the lowest detection line between the RATs of the two labeling materials (P=0.3165, Fig. 3C). In summary, we were led to conclude that the model for the NP protein was the most accurate.
Figure 2. Concentration gradient and Repeatability test for 2019-nCoV nucleocapsid protein. A. Result interpretation of COVID-19 Ag-RATs; B. The addition of sample dilutions of different concentrations to the RAT test plate allowed for the observation of lines with varying display levels. With this RAT, at 50,000 pg/ml and 25,000 pg/ml, the T line was more conspicuous than the C line, and gradually became lighter. It was judged to be weakly positive until 50 pg/ml, when the T-line was the faintest; C. Ten replicates were performed for RAT plates showing weak (50 ng/ml) and moderate (62.5 ng/ml) positivity in this RAT. Each replicate was positive at each concentration and showed consistent color shade.

DISCUSSION

The rapid spread of COVID-19 represents a major global medical challenge. The diagnostic tests for COVID-19 disease are a subject of great interest to policy makers and regulators. Currently, diagnostic tests for COVID-19 include serological, antigen and viral nucleic acid tests. RT-PCR is the gold-standard for diagnosing COVID-19. However, RT-PCR analysis has some inherent limitations: It entails specialized equipment, requires professionals for result interpretation and is time-consuming. While serological COVID-19 assay is a blood test for the detection of the specific COVID-19 antibodies produced by the immune system[11]. However, this method also has obvious restrictions, including a protracted window of detection, a lack of sensitivity and specificity, and possible false positives. Therefore, less expensive, and easy-to-use RATs are needed for the detection of SARS-CoV-2 NP, not only for diagnosing COVID-19, but also for characterizing the course of disease.
Various RATs are available on the China market and subject to China regulations with the mandatory marked/labeling for sales. Most of RATs process upper respiratory tract swabs, including nasopharyngeal (NP), oropharyngeal (O), or nasal (N) swabs. Here, we evaluated 12 different marked commercial assays for the laboratory detection of SARS-CoV-2 NP. We found the lowest detectable concentration was limited to 50 pg/ml by Flowflex (Acon Biotech), these results were similar to those reported by prior studies[5,12]. The Flowflex RAT of ACON Biotech achieved the highest sensitivity for the detection of Delta and Omicron variants[5]. However, LEPU Technology 2019-nCoV antigen test showed marginally lower detection sensitivity for NP than the other RATs, the lowest concentration being 2500 pg/mL (refer to Table 2). The sensitivity of the RAT depends on the binding kinetics and epitopes of the monoclonal antibody used as well as the composition of the lysis buffer[13]. Our analytical study suggests that strong heterogeneity exists among different RATs in the detection of SARS-CoV-2 NP antigen.
Figure 3. The minimum concentration of standard substances in different Rapid Antigen Test kits. A. Heatmap of the minimum concentration of standard substances in different Rapid Antigen Test/RAT kits; B. Minimum detection limits for colloidal gold and latex in RATs; C. Minimum TCID50 for colloidal gold and latex according to manuals provided by the manufacturers. Data were presented as means ± S.D., *P < 0.05.
Table 1. Overview of RAT kits evaluated in the study.
Number Kits Manufacturer Detection method Recommended Test Samplea Minimum detection limits (TCID50/mL)
I Flowflex 2019-nCoV antigen test ACON biotech Latex NP or N or O swab 160
II 2019-nCoV antigen test FOSUN diagnostic colloidal gold N swab 150
III 2019-nCoV antigen test EasyDiagnosis colloidal gold NP or N or O swab 500
IV 2019-nCoV antigen test KHB colloidal gold NP or N swab 500(CT=31)
V 2019-nCoV antigen test Vazyme colloidal gold NP or N or O swab 50
VI 2019-nCoV antigen test LEPU Technology colloidal gold N swab 200
VII 2019-nCoV antigen test XABT colloidal gold NP or N swab 200
VIII 2019-nCoV antigen test Biobase colloidal gold N or O swab 115
IX 2019-nCoV antigen test Zybio colloidal gold NP or N or O swab 70
X 2019-nCoV antigen test Livzon Latex NP or N or O swab 100
XI GenFocus 2019-nCoV antigen test Jinwofu Latex NP or N or O swab 100(CT=31)
XII 2019-nCoV antigen test YHLO colloidal gold NP or N swab 250
aNP, nasopharyngeal; N, nasal; O, oropharyngeal.
Several previous related studies had more or less limitations. For example, no material was available to compare the results from alternative assays with different cross-reactivities. Furthermore, there was no clinical samples and untailored sampling participants in this study. We used this method mainly because the sample source in China is limited to patients who are symptomatic at the time of sample collection. And we were unable to compare the performance of the RATs assays to other high-throughput antigen assays. Meanwhile, SARS-CoV-2 antigen assays were not standardized, and the results may thus not be directly extrapolated to other populations. In a previous study, we performed a series of serial dilutions using high-value pools diluted in low-value pools[7]. Therefore, dilution linearity and assay accuracy were not assessed in this study.
In summary, this study provided an evaluation method for the systematic detection of SARS-CoV-2 antigen by RATs. Although all of RATs were in compliance with the current laws of the China, there are still significant differences in terms of detection limits of the SARS-CoV-2 recombinant proteins. Therefore, it is necessary to establish a standard manufacturer’s instructions for their optimal operation on testing sites. We analyzed the detection performance of 12 RATs through an assay platform established in this study, which provides reference information for national governments, international donor agencies and global health policy-making bodies.
Table 2. Test results of different kits of standard substances concentration.
The concentration of 2019-nCoV（pg/mL） I II III IV V VI VII VIII IX X XI XII
5000.00 + + + + + + + + + + + +
2500.00 + + + + + + + + + + + +
1250.00 + + + + + - + + + + + +
1000.00 + + - + + - - + + + + +
500.00 + + - + + - - + + + + +
250.00 + + - + - - - - - + + -
125.00 + + - - - - - - - - + -
62.50 + - - - - - - - - - - -
50.00 + - - - - - - - - - - -
25.00 - - - - - - - - - - - -
The table summarizes the results of the standard substance concentration tests for/using different kits. Various kits are denoted by different colors. ‘+’means the test result is weakly, moderately, or strongly positive, ‘-’ indicates the test result is negative.

Acknowledgements

This work was supported by National Key R&D Program of China (2023YFC0872400), Beijing Municipal Science and Technology Project (Z221100007922020).

Competing interests

The authors have declared that no competing interests exist.

Abbreviation used

COVID-19, Corona Virus Disease 2019; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; rRT-PCR, real-time reverse transcription polymerase chain reaction; RAT, rapid antigen Test; ICT, immunochromatography; NC, Nitrocellulose membrane; CLSI, Clinical and Laboratory Standards Institute; NP, nasopharyngeal; O, oropharyngeal; N, nasal; TCID50, 50% tissue culture infective dose; Ct, Cycle threshold.

References

  1. World Health Organization [Internet]. COVID-19 Epidemiological Update - 24 November 2023. Available from: https://www.who.int/publications/m/item/covid-19-epidemiological-update---24-november-2023.
  2. Summer S, Schmidt R, Herdina AN, Krickl I, Madner J, Greiner G, et al. Detection of SARS-CoV-2 by real-time PCR under challenging pre-analytical conditions reveals independence of swab media and cooling chain. Sci Rep. 2021;11(1):13592. doi: 10.1038/s41598-021-93028-8. PMID: 34193912
  3. Lee H, Kang H, Cho Y, Oh J, Lim TH, Ko BS, et al. Diagnostic Performance of the Rapid Antigen Test as a Screening Tool for SARS-CoV-2 Infection in the Emergency Department. J Pers Med. 2022;12(7). doi: 10.3390/jpm12071172. PMID: 35887669
  4. Szekely J, Mongkolprasert J, Jeayodae N, Senorit C, Chaimuti P, Swangphon P, et al. Development, Analytical, and Clinical Evaluation of Rapid Immunochromatographic Antigen Test for SARS-CoV-2 Variants Detection. Diagnostics (Basel). 2022;12(2). doi: 10.3390/diagnostics12020381. PMID: 35204473
  5. Bekliz M, Adea K, Puhach O, Perez-Rodriguez F, Marques Melancia S, Baggio S, et al. Analytical Sensitivity of Eight Different SARS-CoV-2 Antigen-Detecting Rapid Tests for Omicron-BA.1 Variant. Microbiol Spectr. 2022;10(4):e0085322. doi: 10.1128/spectrum.00853-22. PMID: 35938792
  6. Gomes JC, Masood AI, Silva LHS, da Cruz Ferreira JRB, Freire Júnior AA, Rocha A, et al. Covid-19 diagnosis by combining RT-PCR and pseudo-convolutional machines to characterize virus sequences. Sci Rep. 2021;11(1):11545. doi: 10.1038/s41598-021-90766-7. PMID: 34078924
  7. Yu M, Chen D, Tang X, Zhang Y, Liang P, Xiong Y, et al. Evaluation of a high-sensitivity SARS-CoV-2 antigen test on the fully automated light-initiated chemiluminescent immunoassay platform. Clin Chem Lab Med. 2023. Epub 2023/01/20. doi: 10.1515/cclm-2022-1039. PMID: 36656975
  8. Dinnes J, Deeks JJ, Adriano A, Berhane S, Davenport C, Dittrich S, et al. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. 2020;8(8):Cd013705. doi: 10.1002/14651858.Cd013705. PMID: 32845525
  9. CLSI. Evaluation of precision of quantitative measurement procedures; approved guideline-3rd edition. CLSI document EP5-A3. Wayne, PA: Clinical and Laboratory Standards Institute. 2014.
  10. CLSI. User protocol for evaluation of qualitative test performance; approved guideline-second edition. CLSI document EP12-A2. Wayne, PA: Clinical and Laboratory Standards Institute. 2008.
  11. Padoan A, Cosma C, Sciacovelli L, Faggian D, Plebani M. Analytical performances of a chemiluminescence immunoassay for SARS-CoV-2 IgM/IgG and antibody kinetics. Clin Chem Lab Med. 2020;58(7):1081-8. doi: 10.1515/cclm-2020-0443. PMID: 32301749
  12. Schneider UV, Forsberg MW, Leineweber TD, Jensen CB, Ghathian K, Agergaard CN, et al. A nationwide analytical and clinical evaluation of 44 rapid antigen tests for SARS-CoV-2 compared to RT-qPCR. J Clin Virol. 2022;153:105214. doi: 10.1016/j.jcv.2022.105214. PMID: 35738151
  13. Sakai-Tagawa Y, Yamayoshi S, Halfmann PJ, Kawaoka Y. Comparative Sensitivity of Rapid Antigen Tests for the Delta Variant (B.1.617.2) of SARS-CoV-2. Viruses. 2021;13(11). doi: 10.3390/v13112183. PMID: 34834991