THE POLYGLOT SEARCH TRANSLATOR Justin Clark Centre for Research in - PowerPoint PPT Presentation

SYSTEMATIC REVIEW ACCELERATOR: THE POLYGLOT SEARCH TRANSLATOR Justin Clark Centre for Research in Evidence Based Practice (CREBP) Bond University, Gold Coast, Australia

SR tasks – full list of tasks and tools Task No. Classification Task Description Automation tool SRA – TRIP data analysis 1 Preparation Formulate review Decide on the research question of the question review. 2 Preparation Find previous or Search for a SR that answers the same Cochrane library, Epistemonikos, PROSPERO upcoming SR question. 3 Write up Write the protocol Provide an objective, reproducible, sound Reporting guidelines, existing protocol templates methodology for peer review. 4 Preparation Devise search strategy Decide on databases and keywords to find SRA - Registered Trials Finder (in development), SRA - Word Frequency all relevant trials. Analyser 5 Retrieval Search Aim to find all relevant citations even if SRA - Polyglot Search Translator many irrelevant ones included. 6 Retrieval De-duplicate Remove identical citations. SRA - De-duplicator. Rayyan 7 Appraisal Screen abstracts Based on titles and abstracts, remove SRA - Endnote Helper, Covidence, Rayyan definitely irrelevant citations. Endnote - find full text feature, SRA – Endnote Helper 8 Retrieval Obtain full text Download, request copies from authors, inter-library loan, etc. 9 Appraisal Screen full text Exclude irrelevant citations. SRA - Endnote Helper, Covidence 10 Retrieval Citation analysis Follow citations, cited and citing, from SRA - SpiderCite (in development) included trials to find additional trials. 11 Synthesis Extract data Extract outcome numbers and associate Robot Reviewer with trial arm 12 Synthesis Synthesize data Convert extracted data to common Revman representation (usually average and SD). 13 Retrieval Recheck literature Repeat the search to find new literature SRA - Polyglot Search Translator (future development) published since the initial search. 14 Synthesis Meta-analyze Statistically combine the results from all RevMan, MetaXL included trials. SRA – Revman Replicant 15 Write up Write up review Produce and publish the final report

Polyglot Search Translator (PST) • The PST translates either a PubMed or Ovid Medline search strategy to the correct syntax for different databases

Polyglot Search Translator • Copy your PubMed or Ovid Medline search and paste it in to the ‘Your Query’ box • To see how the search translates across databases, click the arrow next to the name of the database.

Polyglot Search Translator • For some databases you will need to convert the subject terms (e.g. change MeSH terms to Emtree terms)

Polyglot Search Translator • The PST is designed to make searching databases easier, BUT it does not remove the need to have knowledge of the process Medline Ovid to Cochrane Central

Evaluation of the Polyglot Search Translator a tool for improving searches in systematic reviews: a multi-centre randomized crossover trial • Clark J 1 , Carter M 1 , Honeyman D 2 , Cleo G 1 , Auld Y 3 , Booth D 4 , Condron P 5 , Dalais C 6 , Dern S 2 , Linthwaite B 3 , May N 7 , Munn J 8 , Ramsay L 9 , Rickett K 6 , Rutter C 10 , Sanders S 1 , Smith A 11 , Sondergeld P 10 , Wallin M 11 , Beller E 1 • 1 Centre for Research in Evidence Based Practice, Bond University, Australia • 2 Bond University Library, Bond University, Australia • 3 Gold Coast Health Library Service, Gold Coast University Hospital, Australia • 4 University Library, University of Newcastle, Australia • 5 University Library, University of Melbourne, Australia • 6 University Library, University of Queensland, Australia • 7 Sturt Library, Flinders University, Australia • 8 Centre for Teaching and Learning, Southern Cross University, Australia • 9 University Library, Australian Catholic University, Australia • 10 University Library, Queensland University of Technology, Australia • 11 Hunter New England Health Libraries, NSW Health • 12 University Library, Southern Cross University, Australia

Background • Searching the literature to identify relevant studies is an important part of a SR • Translating the search to run in multiple databases is needed for a good SR • To improve the speed of this the Polyglot Search Translator (PST) was developed.

Objectives • To determine whether a PST-assisted search translation improves the speed without an increase in errors when translating search strings from one database to another.

Methods • Participants (n=20) were recruited, 4 dropped out leaving 16 (10 of whom completed all 10 translations) • Participants were randomly allocated ten search strategies from a pool of 20 • They were then randomly assigned to either use the PST to help translate it, or to not use it • We compared how long it took, how many errors were made and how many search results they found

Results • A PST assisted search strategy translation was faster with, a mean time saving of 14 minutes and 18 seconds per translation. • If you approximate a translation time of 1 hour per search this meant a 24% reduction in time.

Figure 1 Mean time taken (minutes) to translate searches 5.8 PubMed to Web of Science 11.4 20.0 PubMed to Scopus 42.0 18.1 PubMed to MEDLINE 23.7 24.5 PubMed to Embase 28.3 PST-A method 15.5 PubMed to Cochrane 26.8 22.5 Manual method PubMed to CINAHL 33.6 17.7 PubMed to All databases 27.6 30.9 Ovid MEDLINE to Web of Science 41.5 56.7 Ovid MEDLINE to Scopus 44.7 52.5 Ovid MEDLINE to Pubmed 70.6 49.8 Ovid MEDLINE to Embase 72.8 23.5 Ovid MEDLINE to Cochrane 66.8 49.7 Ovid MEDLINE to CINAHL 79.1 43.8 Ovid MEDLINE to All databases 62.6 30.8 All databases 45.1 0 10 20 30 40 50 60 70 80 90

Search hits: median percentage difference from the expected number to be found PST-A method Manual method Median % Median % Range Range Difference Difference Translation -1.47 (-87.3 to 600.0) 22.8 (-77.2 to 625.7) PubMed-Web of Science 768.3 (-95.2 to 18700.0) 46.1 (0.8 to 28000) PubMed-Scopus -12.9 (-66.7 to 82.4) -17.3 (-57.8 to 160.2) PubMed-MEDLINE -2.5 (-60.7 to 86.6) -17.9 (-37.9 to 19.8) PubMed-Embase -1.1 (-100.0 to 486.9) 34.6 (-48.5 to 1467.2) PubMed-Cochrane 13.8 (-38.6 to 1262.5) 43.0 (-4.6 to 300.0) PubMed-CINAHL -1.5 (-100.0 to 18700) 2.1 (-77.2 to 28000.0) PubMed - All databases 11.7 (-75.9 to 596.7) 7.0 (-78.3 to 731.9) MEDLINE-Web of Science 22.5 (-41.2 to 2012.5) 21.9 (-91.4 to 7424.8) MEDLINE-Scopus 2.4 (-98.8 to 753.1) 20.4 (-97.5 to 4658.5) MEDLINE-PubMed -23.1 (-99.6 to 7.2) -12.4 (-89.7 to 13.7) MEDLINE-Embase 2.4 (-100.0 to 39716.7) 1.4 (-98.9 to 1562.9) MEDLINE-Cochrane 40.6 (-84.9 to 563.0) 104.0 (-36.6 to 2984.7) MEDLINE-CINAHL 3.0 (-100.0 to 39716.7) 7.0 (-98.9 to 4658.5) MEDLINE - All databases 1.7 (-100.0 to 39716.7) 6.3 (-98.9 to 4658.5) All databases

Errors: mean number of total errors PubMed-Web of Science 0 2 PubMed-Scopus 9 6 PubMed-MEDLINE 13 15 PubMed-Embase 18 7 PubMed-Cochrane 8 0 PubMed-CINAHL 6 7 PubMed-total 9 5 Ovid MEDLINE-Web of Science 2 3 Ovid MEDLINE-Scopus 17 23 Ovid MEDLINE-PubMed 15 17 Ovid MEDLINE-Embase 13 16 Ovid MEDLINE-Cochrane 27 1 Ovid MEDLINE-CINAHL 7 14 Ovid MEDLINE-total 15 12 Total 14 0 5 10 15 20 25 30 Mean total errors Polyglot (n=95) Mean total errors Manual (n=103)

Errors : Number of type of errors Type of error All databases Polyglot (n=95) All databases Manual (n=103) Wrong wildcard or truncation syntax - recall 28 21 Missing or added truncation or wildcard - recall 6 14 Missing or added truncation or wildcard - Precision 0 1 Truncated term at wrong point - recall 2 5 Truncated term at wrong point - Precision 0 3 Missing subject term - recall 3 3 Missing keyword term - recall 4 12 Restricting to major subject terms - recall 3 3 Not exploding subject terms - recall 7 8 Exploding subject terms - precision 3 17 Incorrect MeSH conversion - recall 3 5 Incorrect MeSH conversion - Precision 3 1 Incorrect adjacency syntax - recall 5 1 Used incorrect adjacency - recall 0 2 Used incorrect adjacency- Precision 3 6 Incorrect field syntax - recall 9 2 Incorrect field syntax - precision 0 1 Chose incorrect fields - recall 4 13 Chose incorrect fields - precision 7 13 Incorrect phrase translation - recall 6 3 Incorrect phrase translation - Precision 1 6 Extra out of place characters inserted - recall 8 4 Spelling mistake in search - recall 1 7 Incorrect boolean operator - recall 2 1 Incorrect search structure - recall 6 7 Extra components added - recall 0 0 Extra components added - Precision 3 0 Total unique errors 117 159

Conclusion • For people skilled in conducting searches of the literature the PST can increase the speed of translating search strategies h which this can be done, without a decrease in quality