Readers may use articles without permission of copyright owners, as long as the author and MLA are acknowledged and the use is educational and not for profit. This article has been cited by other articles in PMC. Two approaches were used to investigate if the search assessment tool could capture performance differences in search strategy construction. Second, a cross-section of search strategies from residents in one incoming class was compared with strategies of residents graduating a year later.
MEDLINE search strategies formulated by faculty who had been identified as having search expertise were used as a gold standard comparison. Two librarians rated the blinded search strategies. Search strategy scores were significantly higher for residents who received training than the comparison group with no training.
There was no significant difference in search strategy scores between senior residents who received training and faculty experts. This assessment tool can measure improvements in information-seeking skills and provide data to fulfill Accreditation Council for Graduate Medical Education competencies.
Implications This validated assessment tool can serve as an effective means to measure improvements in residents' information-seeking skills and provide data to fulfill Accreditation Council for Graduate Medical Education competencies. The Accreditation Council for Graduate Medical Education ACGME has listed practice-based learning and improvement PBLI as a core competency, defining PBLI as the ability to 1 investigate and evaluate the care of patients, 2 appraise and assimilate scientific evidence, and 3 continuously improve patient care based on constant self-evaluation and lifelong learning [ 1 ].
Of these, the ability to appraise and assimilate scientific evidence is infrequently taught in many postgraduate medical education programs, and currently no one standardized method is used to assess this competency [ 2 ].
Being able to search the medical literature effectively is an essential skill in the practice of evidence-based medicine EBM.
Moreover, developing and maintaining strong literature searching skills use of appropriate search keywords, use of appropriate limits, development of evidence-based search strategies, and use of correct search syntax help physicians identify appropriate literature that affects medical decision making [ 3 ]. Due to their training in the areas of MEDLINE search strategies and search retrieval, medical librarians are recognized as specialists in biomedical literature searching and search assessment [ 4 ].
Historically, a variety of methods and tools have been used to assess residents' search skills [ 5 ]. Instruments have typically been based on the development of a gold standard search strategy, as determined by an expert searcher, against which all residents' searches for a specific clinical question are compared.
The comparison includes identified elements determined by the expert searchers and agreed upon search criteria that would be included in an effective and efficient search. METHODS Tool development Interest at the University of Michigan in developing a validated tool dates back to , when a residency program director and two librarians developed a blinded randomized controlled trial to measure librarians' impact on residents' searching and retention of skills while on rotation in the neonatal intensive care unit [ 6 ].
In evaluating the resident searches, a gold standard search on a relevant topic was established by using all identified search criteria and searching elements. Search scores were determined by the librarians, and points were deducted for common search errors identified in the literature [ 6 ].
This method of search strategy assessment was found to have limitations. It was too subjective and lacked the flexibility needed to measure the variety of search skills among searchers. Subsequently, in , Nesbit and Glover developed a search assessment instrument based on feedback gathered in a national survey of medical librarians, in which the librarians were asked to weight the importance of a series of search elements for a successful search [ 8 ]. The structure of the instrument allowed assessment with greater flexibility.
With permission from the authors, the instrument template was adapted by adding additional search elements and a section to allow subtraction of points for search errors. UMMSA maintains flexibility and can be used in various biomedical and health sciences curricula.
It is not necessarily limited to assessing search skills in clinical medicine. The UMMSA tool is a matrix that measures the use of identified search elements and other important criteria. UMMSA is customized to apply to a specific question. There are eleven search criteria for which points are awarded and two search elements for which points are deducted Table 1.
Because the ACGME competencies had been recently introduced and required supporting documentation of competency achievement, the department of pediatrics had a specific interest in finding a meaningful method to assess information retrieval skills as a part of the PBLI competency. The pediatrics liaison librarian in collaboration with department of pediatrics designed the assessments.
Participants Participants in this study included all 22 incoming pediatric and medicine-pediatric first-year residents in cohort 1 ; 10 self-selected pediatric residents who graduated the year later, in cohort 2 ; and 9 faculty volunteers who had been identified by their expertise in EBM concepts and principles. These faculty members had EBM teaching responsibilities, demonstrated professional knowledge, and published on the topic of EBM. The 10 self-selected residents in cohort 2 were recruited through an email message sent out to all third-year residents by the department of pediatrics.
No incentives were offered for their participation. The incoming residents cohort 1 were tested twice, once in , just after receiving training in MEDLINE searching and once just before graduation. The 9 faculty volunteers and the self-selected pediatric residents were tested in July Search scoring procedure Two librarians from the University of Michigan's Health Sciences Libraries authors Rana and Bradley rated each search strategy performance in a blinded fashion, using the new instrument.
Study participants were presented with a case of a pediatric patient with bronchiolitis, asked to identify an appropriate search question, and conduct a MEDLINE search Appendix , online only. After reading the clinical scenario, searchers were expected to identify the clinical question: The identified gold standard search strategy included the elements that both librarians agreed were required in an effective MEDLINE search to address this specific clinical question.
The raters used UMMSA to independently assess the search strategy effectiveness for each of the study participant's searches. The Ovid MEDLINE search interface was used because, at the time of this study, the process of search strategy development was distinguishable and search strategies were easily captured using the Ovid interface, as opposed to the PubMed interface.
Thus, an aspect of the searcher's information-seeking behavior would be revealed by their selection of appropriate MeSH terms. In the basic PubMed search, terms are mapped to MeSH automatically without the searcher needing to select appropriate headings, so the searcher's judgment could not be assessed in this same fashion. Study design Two approaches were used to validate the instrument's ability to measure improvements in search strategy that would be expected from training and experience.
Their search skills were tested following instruction in and again, upon graduation, in To assess improvement in scores, fifteen of the incoming residents of cohort 1 were assessed on the same case prior to graduation of the original twenty-two residents, seven either dropped out of the program or moved on quickly near graduation time.
The scores of these same twenty-two incoming residents were compared with the scores of the ten volunteer senior residents, who had not received any training, and the nine faculty volunteers.
Due to such experimentation with, testing of, and repetition of terms or search features i. For instance, it might seem as if a very in-depth and lengthy search strategy was created when, in truth, the searcher only used two search terms in the final search strategy. The librarians only graded the final identified search strategy.
Points were deducted for lack of search efficiency in such searches. Construct validity Significant improvements were found when comparing the and scores for pediatric and medicine-pediatric first-year residents cohort 1 mean improvement: Search strategy scores were significantly higher for residents who received training upon graduation than for the 9 senior residents who had received no training cohort 2 median Item analysis Using the average ratings of both raters, item statistics were evaluated to determine whether any of the items could be consolidated with other items.
After removing the items with the lowest item-total correlations, the range of item-total score correlations improved to 0. This process resulted in five specific search elements being identified as critical in the development of an effective MEDLINE search strategy: