What Can We Learn and Apply from Journal Peer Review Rick Reiss - PowerPoint PPT Presentation

1 What Can We Learn and Apply from Journal Peer Review Rick Reiss Exponent Alexandria, VA

2  What are the advantages and disadvantages of journal and Agency peer reviews?  Can journals teach us anything about Agency peer reviews?

3 Issues in the Peer Review Process  How do we select peer reviewers?  How can we ensure a balance of perspectives?  Does the conflict process exclude those with real experience with chemicals?  How should the public and stakeholders be considered in the peer review process?  How do we ensure that Agencies adequately respond to peer reviews?  How do we encourage a more robust review and evaluation process?

4 Goals for Selection of Peer Review Panel  Covers all of the technical areas of relevance  Can be broad with chemical assessments  Achieves a balanced perspective of views  Resulting review should reflect views that a majority of scientists would share  Includes reviewers that are free to conclude that the assessment is good or bad  Viewed as credible  Though, those unhappy with result may always complain

5 How Journals Review Papers Review Reject Revise Accept Editor Publish Major Minor Review

6 Issue Agency Peer Review Journal Peer Review Peer reviewers – Extensive review Disclosure but no conflicts of interest and disclosure means of confirming information Peer reviewers – Large panel size Some balance balance of allows different possible, but limited perspective disciplines and can by 2-3 reviewers allow balance of perspective Peer reviewers – Possible pressure, No, because peer pressure to reach given that it’s a review is particular conclusion public process anonymous

7 Issue Agency Peer Review Journal Peer Review Peer reviewers – Can interact with Cannot interact with interactions authors and other authors and other reviewers – good for reviewers – good for information objectivity and exchange and removing pressures avoiding errors Leader of process Head of panel is Editor is free to influential, but not make decision alone usually dominant based on peer review

8 Issue Agency Peer Review Journal Peer Review Result of process A recommendation Accept or reject to Agency that it can follow or not follow Revisions to Usually does not Formal process to assessment during occur request revisions review and review if response is adequate Further review? At the discretion of Only if author the Agency to call substantially revises panel back (or form paper and resubmits new one)

9 Advantages and Disadvantages of Peer Review Panels Advantages  Large panel with different types of experts  Interaction and exchange of information Disadvantages  Pressure of public process  No ability to force changes in assessment  No review after changes made in assessment

10 Advantages and Disadvantages of Journal Peer Review Advantages  Anonymous process  Power to reject or accept  Ability to ask for revisions and make sure they are adequately made Disadvantages  Small number of reviewers  Lack of interaction with authors or other reviewers

11 Should Panels Make Recommendations on Future Review?  Agree up front that this will be done  Possible problem: Public agencies cannot necessarily empower decision-making to a different entity  Options could include:  Acceptable, but should consider the following comments … (i.e., minor revisions)  No re-review necessary  Upgradable, but need to respond to detailed comments and re- review  Subset of panel reviews changes, possibly without meeting  Rejected, scientific flaws are substantial  Panel needs to reconvene to review changes

12 Different Possible Processes for Agencies to Address Comments  Allow agency discretion for addressing comments  Agency publishes a detailed response  Allow panel to decide whether they need to see the revised assessment.  Full reconvening of panel  Ask panelists to separately respond to revision  Have agencies establish an independent ombudsman to ascertain if agency has adequately responded to comments

13 Conclusions  Peer reviews for agencies and in journals have advantages and disadvantages  One aspect of journal peer review that would improve agency peer review are mechanisms to:  Rate an assessment as acceptable, not acceptable, or upgradeable  Allow for a revision and response to revision process to assure adequate changes are made based on peer review