How to report Screen4Me in Cochrane protocols and reviews

This guidance is for authors of Cochrane protocols and reviews of interventions who are planning to use - or have used - the Cochrane Crowd Screen4Me service.

Additional references
Protocols
Reviews
Updates
Multiple uses of Screen4Me for an individual review
Communicating with the Crowd

Additional references

When reporting use of Screen4Me in your Cochrane protocol or review, add these references to your additional references in RevMan first, so that you can link to them when inserting the template text below.

Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, Wisniewski S, McDonald S, Murano M, Glanville J, Foxlee R, Beecher D, Ware J, Thomas J. An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. J Clin Epidemiol. 2021 May;133:130-139. doi: 10.1016/j.jclinepi.2021.01.006. Epub 2021 Jan 18. PMID: 33476769.

Thomas J, McDonald S; Noel-Storr AH, Shemilt I, Elliott J, Mavergames C, Marshall I. Machine learning reduces workload with minimal risk of missing studies: development and evaluation of an RCT classifier for Cochrane Reviews. Journal of Clinical Epidemiology 2020; DOI:https://doi.org/10.1016/j.jclinepi.2020.11.003

Noel-Storr AH, Dooley G, Wisniewski S, Glanville J, Thomas J, Cox S, Featherstone R, Foxlee R, Cochrane Centralised Search Service showed high sensitivity identifying randomised controlled trials: A retrospective analysis. Journal of Clinical Epidemiology 2020; 127: 142-150 https://doi.org/10.1016/j.jclinepi.2020.08.008

Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide. Res Synth Methods. 2018;9(4):602-614

Protocols

RevMan section: Selection of studies

Template text to add at the start of this section:

We will be using Cochrane’s Screen4Me workflow to help assess the search results. Screen4Me comprises three components: known assessments – a service that matches records in the search results to records that have already been screened in Cochrane Crowd and been labeled as an RCT or as Not an RCT; the RCT classifier – a machine learning model that distinguishes RCTs from non-RCTs; and if appropriate, Cochrane Crowd – Cochrane’s citizen science platform where the Crowd help to identify and describe health evidence.

For more information about Screen4Me and the evaluations that have been done, please go to the Screen4Me webpage on the Cochrane Information Specialist’s portal. For more detailed information regarding evaluations of the Screen4Me components, see [add in-text citations to additional references Noel-Storr 2021, Thomas 2020, Noel-Storr 2020, Marshall 2018].

Reviews

RevMan section: Selection of studies

Template text to add at the start of this section:

We used Cochrane’s Screen4Me workflow to help assess the search results. Screen4Me comprises three components: known assessments – a service that matches records in the search results to records that have already been screened in Cochrane Crowd and been labeled as an RCT or as Not an RCT; the RCT classifier – a machine learning model that distinguishes RCTs from non-RCTs, and if appropriate, Cochrane Crowd – Cochrane’s citizen science platform where the Crowd help to identify and describe health evidence.

RevMan section: Results of the search

The search identified a total of [insert number] search results. In assessing the studies we used Cochrane’s Screen4Me workflow to help identify potential reports of randomised trials. The results of the Screen4Me assessment process can be seen in Figure x. We then assessed the remaining [insert number] records left in after Screen4Me. [Author team then continues to describe their screening process e.g. assessing remaining records based on a title and abstract assessment]

RevMan section: Figures: Screen4Me summary diagram

We recommend that you export the Screen4Me summary diagram from your Screen4Me project and import it into the Figures sections of the review. Figure type: Other figure. Caption: Screen4Me summary diagram. Here is an example. Screen4Me produces a flow diagram per search so if you use S4M multiple times for a single review, you will have generated multiple Screen4Me flow diagrams.

RevMan section: Figures: PRISMA flow diagram example

In the example below, the key information about what was excluded at the Screen4Me stage of the screening process is detailed in the Screening exclusion box.

RevMan section: Acknowledgements

If you use Cochrane Crowd component of Screen4Me, you must acknowledge by name those Crowd contributors who screen 250 or more records for your Screen4Me review/project. You can see who should get acknowledgement within the Screen4Me project summary screen. You can name Screen4Me volunteers in the Acknowledgements without seeking permission to name them individually. Volunteers give this permission when agreeing to take part in the Crowd task.

Template text for the Acknowledgements section:

We would like to acknowledge and thank the following people for their help in assessing the search results for this review via Cochrane’s Screen4Me workflow: [insert names].

Updates

Some users will make use of Screen4Me for updates of reviews where Screen4Me has not been used previously. Below we give an example of how to reflect the use of Screen4Me in these circumstances.

RevMan section: Figures: PRISMA flow diagram example

Multiple uses of Screen4Me for an individual review

Some users will use Screen4Me more than once for a specific review. Below we give an example of how to reflect the use of Screen4Me in these circumstances.

RevMan section: Figures: PRISMA flow diagram example

Communicating with the Crowd

If you have used Cochrane Crowd, we would like to alert Crowd contributors when reviews they have contributed to are published. Please email support@cochrane.org when the review is published. The Cochrane Crowd team will send an alert to the Crowd contributors and flag the review as published on the Cochrane Crowd platform.