User menu

Accès à distance ? S'identifier sur le proxy UCLouvain

How to construct and implement script concordance tests: insights from a systematic review

Primary tabs

Dory, Valérie [UCL] Gagnon, Robert Vanpee, Dominique [UCL] Charlin, Bernard

CONTEXT: Programmes of assessment should measure the various components of clinical competence. Clinical reasoning has been traditionally assessed using written tests and performance-based tests. The script concordance test (SCT) was developed to assess clinical data interpretation skills. A recent review of the literature examined the validity argument concerning the SCT. Our aim was to provide potential users with evidence-based recommendations on how to construct and implement an SCT. METHODS: A systematic review of relevant databases (MEDLINE, ERIC [Education Resources Information Centre], PsycINFO, the Research and Development Resource Base [RDRB, University of Toronto]) and Google Scholar, medical education journals and conference proceedings was conducted for references in English or French. It was supplemented by ancestry searching and by additional references provided by experts. RESULTS: The search yielded 848 references, of which 80 were analysed. Studies suggest that tests with around 100 items (25-30 cases), of which 25% are discarded after item analysis, should provide reliable scores. Panels with 10-20 members are needed to reach adequate precision in terms of estimated reliability. Panellists' responses can be analysed by checking for moderate variability among responses. Studies of alternative scoring methods are inconclusive, but the traditional scoring method is satisfactory. There is little evidence on how best to determine a pass/fail threshold for high-stakes examinations. CONCLUSIONS: Our literature search was broad and included references from medical education journals not indexed in the usual databases, conference abstracts and dissertations. There is good evidence on how to construct and implement an SCT for formative purposes or medium-stakes course evaluations. Further avenues for research include examining the impact of various aspects of SCT construction and implementation on issues such as educational impact, correlations with other assessments, and validity of pass/fail decisions, particularly for high-stakes examinations.

Document type Article de périodique (Journal article) – Synthèse de littérature
Access type Accès restreint
Publication date 2012
Language Anglais
Journal information "Medical Education" - Vol. 46, no. 6, p. 552-563 (2012)
Peer reviewed yes
Publisher Wiley-Blackwell Publishing Ltd. ((United Kingdom) Oxford)
issn 0308-0110
e-issn 1365-2923
Publication status Publié
Affiliations UCL - SSS/IRSS - Institut de recherche santé et société
UCL - (MGD) Services des urgences
Links
  1. Norman Geoffrey, Research in clinical reasoning: past history and current trends, 10.1111/j.1365-2929.2005.02127.x
  2. Croskerry Pat, A Universal Model of Diagnostic Reasoning : , 10.1097/acm.0b013e3181ace703
  3. Pelaccia Thierry, Tardif Jacques, Triby Emmanuel, Charlin Bernard, An analysis of clinical reasoning through a recent and comprehensive approach: the dual-process theory, 10.3402/meo.v16i0.5890
  4. De Neys Wim, Glumicic Tamara, Conflict monitoring in dual process theories of thinking, 10.1016/j.cognition.2007.06.002
  5. Eva Kevin W, What every teacher needs to know about clinical reasoning, 10.1111/j.1365-2929.2004.01972.x
  6. Custers E J, Regehr G, Norman G R, Mental representations of medical diagnostic knowledge : a review, 10.1097/00001888-199610000-00044
  7. Charlin Bernard, Tardif Jacques, Boshuizen Henny P. A., Scripts and Medical Diagnostic Knowledge : Theory and Applications for Clinical Reasoning Instruction and Research, 10.1097/00001888-200002000-00020
  8. Charlin Bernard, Boshuizen Henny P A, Custers Eugene J, Feltovich Paul J, Scripts and clinical reasoning : clinical expertise, 10.1111/j.1365-2923.2007.02924.x
  9. van der Vleuten C., Newble D.I., How can we test clinical reasoning?, 10.1016/s0140-6736(95)90763-7
  10. Kreiter Clarence D, Bergus George, The validity of performance-based measures of clinical reasoning and alternative approaches, 10.1111/j.1365-2923.2008.03281.x
  11. Schuwirth Lambert, Is assessment of clinical reasoning still the Holy Grail?, 10.1111/j.1365-2923.2009.03290.x
  12. Elstein A S, Beyond multiple-choice questions and essays : the need for a new way to assess clinical competence, 10.1097/00001888-199304000-00002
  13. Case Susan M., Swanson David B., Extended‐matching items: A practical alternative to free‐response questions, 10.1080/10401339309539601
  14. Miller G E, The assessment of clinical skills/competence/performance : , 10.1097/00001888-199009000-00045
  15. Wass Val, Van der Vleuten Cees, Shatzer John, Jones Roger, Assessment of clinical competence, 10.1016/s0140-6736(00)04221-5
  16. van der Vleuten Cees P M, Schuwirth Lambert W T, Assessing professional competence: from methods to programmes, 10.1111/j.1365-2929.2005.02094.x
  17. Norcini John, Anderson Brownell, Bollela Valdes, Burch Vanessa, Costa Manuel João, Duvivier Robbert, Galbraith Robert, Hays Richard, Kent Athol, Perrott Vanessa, Roberts Trudie, Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference, 10.3109/0142159x.2011.551559
  18. Charlin B., Brailovsky C., Leduc C., Blouin D., 10.1023/a:1009741430850
  19. Lubarsky Stuart, Charlin Bernard, Cook David A, Chalk Colin, van der Vleuten Cees P M, Script concordance testing: a review of published validity evidence : Validity evidence for script concordance tests, 10.1111/j.1365-2923.2010.03863.x
  20. Haig, Med Teach, 25, 352 (2003)
  21. Gantelet M Impact du panel de référence sur les résultats d’un test de concordance de script (TCS) développé en formation initiale des sages-femmes Dissertation for the completion of a diploma of the Ecole des cadres sages-femmes Ecole de Cadres Sages-Femmes 2008
  22. Lamia B Sitbon O Evaluation du raisonnement et de la compétence clinique par le test de concordance de script Universités Paris V, VI, XI, XII Inter-university diploma dissertation Universités Paris V, VI, XI, XII 2006
  23. Deschênes Marie-France, Charlin Bernard, Gagnon Robert, Goudreau Johanne, Use of a Script Concordance Test to Assess Development of Clinical Reasoning in Nursing Students, 10.3928/01484834-20110331-03
  24. Cohen Laura J., Fitzgerald Shirley G., Lane Suzanne, Boninger Michael L., Minkel Jean, McCue Michael, Validation of the Seating and Mobility Script Concordance Test, 10.1080/10400430902945546
  25. Carrière Benoit, Gagnon Robert, Charlin Bernard, Downing Steven, Bordage Georges, Assessing Clinical Reasoning in Pediatric Emergency Medicine: Validity Evidence for a Script Concordance Test, 10.1016/j.annemergmed.2008.07.024
  26. Ramaekers Stephan, Kremer Wim, Pilot Albert, Beukelen Peter van, Keulen Hanno van, Assessment of competence in clinical reasoning and decision‐making under uncertainty: the script concordance test method, 10.1080/02602938.2010.500103
  27. Goulet François, Jacques André, Gagnon Robert, Charlin Bernard, Shabah Abdo, Poorly performing physicians: Does the script concordance test detect bad clinical reasoning?* : , 10.1002/chp.20076
  28. Meterissian Sarkis, Zabolotny Brent, Gagnon Robert, Charlin Bernard, Is the script concordance test a valid instrument for assessment of intraoperative decision-making skills?, 10.1016/j.amjsurg.2006.10.012
  29. Monnier Patricia, Bédard Marie-Josée, Gagnon Robert, Charlin Bernard, The relationship between script concordance test scores in an obstetrics-gynecology rotation and global performance assessments in the curriculum, 10.5116/ijme.4d21.bf89
  30. Boulouffe Caroline, Charlin Bernard, Vanpee Dominique, Evaluation of Clinical Reasoning in Basic Emergencies Using a Script Concordance Test, 10.5688/aj7410194
  31. Ruiz Jorge G., Tunuguntla Renuka, Charlin Bernard, Ouslander Joseph G., Symes Stephen N., Gagnon Robert, Phancao Francois, Roos Bernard A., The Script Concordance Test as a Measure of Clinical Reasoning Skills in Geriatric Urinary Incontinence : SCT IN GERIATRIC URINARY INCONTINENCE, 10.1111/j.1532-5415.2010.03136.x
  32. Lubarsky Stuart, Chalk Colin, Kazitani Driss, Gagnon Robert, Charlin Bernard, The Script Concordance Test: A New Tool Assessing Clinical Judgement in Neurology, 10.1017/s031716710000706x
  33. Lambert Carole, Gagnon Robert, Nguyen David, Charlin Bernard, The script concordance test in radiation oncology: validation study of a new tool to assess clinical reasoning, 10.1186/1748-717x-4-7
  34. Tunuguntla, J Gen Intern Med, 24, 10 (2009)
  35. Gagnon Robert, Lubarsky Stuart, Lambert Carole, Charlin Bernard, Optimization of answer keys for script concordance testing: should we exclude deviant panelists, deviant responses, or neither?, 10.1007/s10459-011-9279-2
  36. Charlin Bernard, Gagnon Robert, Pelletier Jean, Coletti Michel, Abi-Rizk Grace, Nasr Claudine, Sauve Evelyne, van der Vleuten Cees, Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel, 10.1111/j.1365-2929.2006.02541.x
  37. Marie I., Sibert L., Roussel F., Hellot M.-F., Lechevallier J., Weber J., Le test de concordance de script : un nouvel outil d’évaluation du raisonnement et de la compétence clinique en médecine interne ?, 10.1016/j.revmed.2004.10.028
  38. Sibert Louis, Charlin Bernard, Corcos Jacques, Gagnon Robert, Lechevallier Joel, Grise Philippe, Assessment of Clinical Reasoning Competence in Urology with the Script Concordance Test: an Exploratory Study Across Two Sites from Different Countries, 10.1016/s0302-2838(02)00053-2
  39. Gibot Sébastien, Bollaert Pierre-Edouard, Le test de concordance de script comme outil d’évaluation formative en réanimation médicale, 10.1051/pmed:2008037
  40. Fournier Jean-Paul, Thiercelin Didier, Pulcini Céline, Alunni-Perret Véronique, Gilbert Elise, Minguet Jean-Marc, Bertrand François, Évaluation du raisonnement clinique en médecine d’urgence : les tests de concordance des scripts décèlent mieux l’expérience clinique que les questions à choix multiples à contexte riche, 10.1051/pmed:2006020
  41. CHARLIN B., Script questionnaires: their use for assessment of diagnostic knowledge in radiology, 10.1080/01421599880300
  42. Sibert Louis, Giorgi Roch, Dahamna Badisse, Doucet Jean, Charlin Bernard, Darmoni Stefan J., Is a web-based concordance test feasible to assess therapeutic decision-making skills in a French context?, 10.1080/01421590802572783
  43. Lemay, Can Med Educ J, 1, 89 (2010)
  44. Charlin Bernard, Desaulniers Martin, Gagnon Robert, Blouin Daniel, van der Vleuten Cees, Comparison of an Aggregate Scoring Method With a Consensus Scoring Method in a Measure of Clinical Reasoning Capacity, 10.1207/s15328015tlm1403_3
  45. Fournier, Pédagogie Médicale, 10, 40 (2009)
  46. Collard Anne, Gelaes Sabine, Vanbelle Sophie, Bredart Serge, Defraigne Jean-Olivier, Boniver Jacques, Bourguignon Jean-Pierre, Reasoning versus knowledge retention and ascertainment throughout a problem-based learning curriculum : Biomedical reasoning versus knowledge retention and ascertainment, 10.1111/j.1365-2923.2009.03410.x
  47. Sibert Louis, Darmoni Stefan J, Dahamna Badisse, Hellot Marie-France, Weber Jacques, Charlin Bernard, On line clinical reasoning assessment with Script Concordance test in urology: results of a French pilot study, 10.1186/1472-6920-6-45
  48. Gagnon Robert, Charlin Bernard, Roy Louise, St-Martin Monique, Sauve Evelyne, Boshuizen Henny P. A., van der Vleuten Cees, The Cognitive Validity of the Script Concordance Test: A Processing Time Study, 10.1207/s15328015tlm1801_6
  49. Bland Andrew C., Kreiter Clarence D., Gordon Joel A., The Psychometric Properties of Five Scoring Methods Applied to the Script Concordance Test : , 10.1097/00001888-200504000-00019
  50. Sibert, Prog Urol, 11, 1213 (2001)
  51. Williams Reed G., Klamen Debra L., Hoffman Rebecca M., Medical Student Acquisition of Clinical Working Knowledge, 10.1080/10401330701542552
  52. Joly, Pédagogie Médicale, 10, S39 (2009)
  53. Joly L., Braun M., Benetos A., Apprentissage du raisonnement clinique et test de concordance de script en gériatrie : intérêt dans la formation en stage des étudiants hospitaliers, 10.1016/j.revmed.2009.03.302
  54. Khonputsa Panrasri, Besinque Kathleen, Fisher Dixie, Gong William C., Use of script concordance test to assess pharmaceutical diabetic care: a pilot study in Thailand, 10.1080/01421590600825292
  55. Demeester A Evaluation du raisonnement clinique des étudiants sages-femmes par le test de concordance de script Master's dissertation Université Paris 13 2004
  56. Gagnon Robert, Charlin Bernard, Lambert Carole, Carrière Benoit, Van der Vleuten C., Script concordance testing: more cases or more questions?, 10.1007/s10459-008-9120-8
  57. Bertrand, Pédagogie Médicale, 10, 42 (2009)
  58. Bursztejn, Pédagogie Médicale, 11, 69 (2010)
  59. Charlin Bernard, Gagnon Robert, Sauvé Evelyne, Coletti Michel, Composition of the panel of reference for concordance tests: Do teaching functions have an impact on examinees’ ranks and absolute scores?, 10.1080/01421590601032427
  60. Sibert Louis, Charlin Bernard, Corcos Jacques, Gagnon Robert, Grise Philippe, Vleuten Cees van der, Stability of clinical reasoning assessment results with the Script Concordance test across two different linguistic, cultural and learning environments, 10.1080/0142159021000012599
  61. Charlin, The Script Concordance Test as a Tool for Assessment in Context of Uncertainty: a Scoring Process Study (2006)
  62. Kreiter CD Bland AC Gordon JA Comparing Two Methods of Scoring a Script Concordance Test
  63. Vanbelle Sophie, Massart Valérie, Giet Didier, Albert Adelin, Test de concordance de script : un nouveau mode d’établissement des scores limitant l’effet du hasard, 10.1051/pmed:2007002
  64. Gagnon R, Charlin B, Coletti M, Sauve E, van der Vleuten C, Assessment in the context of uncertainty: how many members are needed on the panel of reference of a script concordance test?, 10.1111/j.1365-2929.2005.02092.x
  65. Holloway Robert, Nesbit Kathryn, Bordley Donald, Noyes Katia, Teaching and evaluating first and second year medical students' practice of evidence-based medicine, 10.1111/j.1365-2929.2004.01817.x
  66. Park Amy J., Barber Matthew D., Bent Alfred E., Dooley Yashika T., Dancz Christina, Sutkin Gary, Jelovsek J. Eric, Assessment of intraoperative judgment during gynecologic surgery using the Script Concordance Test, 10.1016/j.ajog.2010.04.010
  67. Schuwirth Lambert, Colliver Jerry, Gruppen Larry, Kreiter Clarence, Mennin Stewart, Onishi Hirotaka, Pangaro Louis, Ringsted Charlotte, Swanson David, Van Der Vleuten Cees, Wagner-Menghin Michaela, Research in assessment: Consensus statement and recommendations from the Ottawa 2010 Conference, 10.3109/0142159x.2011.551558
  68. Fournier Jean Paul, Demeester Anne, Charlin Bernard, Script Concordance Tests: Guidelines for Construction, 10.1186/1472-6947-8-18
  69. Downing Steven M, Validity: on the meaningful interpretation of assessment data, 10.1046/j.1365-2923.2003.01594.x
  70. Charlin Bernard, Gagnon Robert, Lubarsky Stuart, Lambert Carole, Meterissian Sarkis, Chalk Colin, Goudreau Johanne, van der Vleuten Cees, Assessment in the Context of Uncertainty Using the Script Concordance Test: More Meaning for Scores, 10.1080/10401334.2010.488197
  71. Duggan, BMC Med Educ
  72. Humbert Aloysius J., Johnson Mary T., Miech Edward, Friedberg Fred, Grackin Janice A., Seidman Peggy A., Assessment of clinical reasoning: A Script Concordance test designed for pre-clinical medical students, 10.3109/0142159x.2010.531157
  73. Hammick Marilyn, Dornan Timothy, Steinert Yvonne, Conducting a best evidence systematic review. Part 1: From idea to data coding. BEME Guide No. 13, 10.3109/01421590903414245
Bibliographic reference Dory, Valérie ; Gagnon, Robert ; Vanpee, Dominique ; Charlin, Bernard. How to construct and implement script concordance tests: insights from a systematic review. In: Medical Education, Vol. 46, no. 6, p. 552-563 (2012)
Permanent URL http://hdl.handle.net/2078.1/114871