Topics for Dialogues:
There are numerous practices to be discussed at the conference, some of which would be selected for production as dialogues. Here are listed just a few examples in the form of questions. A current listing of topics that have been suggested and attendees and others who have expressed interest in producing them has been sent to most attendees. If you do not have that listing or want to see a current version, can request it by emailing shiffrin@indiana.edu. If you have not already done so, send an email to Rich Shiffrin listing a few topics that are of particular interest to you, ones that you might want to help produce.
- “Is reproducibility the chief goal of science?”
- "Are scientific inference and statistical inference misaligned?”
- “Would more or fewer rules and regulations be good for science?”
- “Should HARKing (Hypothesizing After the Results are Known) be inhibited or embraced?”
- “Should pre-registration be required for publication?”
- “Should reviewing be blind?”
- “Should various forms of affirmative action be employed in practicing science?”
- “Should publications require a much stronger statistical support than present practice?"
- “Should ‘badges’ be used to induce proper practice?”
- “What should be the role of what kind of theory in experimental design and inference?”
Relevant Domains of Science:
The issues to be addressed span all areas of science, but it would be unwieldy and essentially impossible to include participants from all of science. Because the ‘reproducibility crisis’ was sparked by concerns of replicability in the social sciences, psychology, and cognitive science, because remedies for the ‘crisis’ were proposed by methodologists and statisticians, because computer scientists have produced simulations of likely outcomes of different practices, and because philosophers and historians of science have addressed these issues for many years, the organizers have focused primarily on these domains when proposing and inviting participants. The attendees are quite diverse, including including a mixture of young and older scientists, women and men, diverse fields, diverse geography and institutions, and particularly diverse views
Organizers:
The main organizer, to whom questions should be addressed: Richard Shiffrin NAS Section 52 shiffrin@indiana.edu; 812-855-492; 812-219-5892.
The associate organizers:
David Kellen, Syracuse University, davekellen@gmail.com
Joachim Vandekerckhove, UC Irvine, joachim@uci.edu
Workshop Goals:
The SISI Workshop, Scientific Inference and Statistical Inference, is designed to highlight different approaches to the practice of science. One goal is assessment of the validity of claims that there is a reproducibility crisis. Its title was chosen because most arguments that science needs repair, and most proposed remedies, are based on statistical methodology.
It is clearly important to practice science in ways that best accomplish its many goals, including progress, societal good, ethical approach, feasibility, public communication (and more). However, it is hard to imagine how to carry out a definitive experimental test of alternatives: Scientists cannot run experimental and control groups in alternative universes. Even were it possible to make a mostly universal change in one or more current practices, it would be hard to know if the results produced net benefit or harm. Some lessons can be gleaned from historical analyses, but different researchers interpret them in different ways.
These difficulties aside, scientists have strong views about the best ways to practice science. These views are typically based on scientists’ personal histories, their understanding of the history of science, on analyses of publications on the web, or on the results of simulations of different scenarios. Particularly in recent years we have seen that these views differ strongly as seen in many publications advocating for one or another position and remedy.
Product of the Conference:
Given the strong and differing views, and the lack of definitive evidence, the workshop is not aimed to reach a consensus. Rather it is aimed to foster reasonable discussions among advocates for differing positions. Thus the planned product of the conference would be a series of ‘dialogues’. During the first five days of the workshop the participants will discuss a large number of practices that have been suggested as deserving of discussion and debate. On the last conference day eight will be selected and described for a formal proposal to PNAS for submission for consideration for a special issue with eight “perspectives”. (PNAS likes this idea in principle but won’t give formal approval until seeing the actual proposal with the eight topics).
For each of the eight dialogues/perspectives that are chosen, the attendees will identify a ’leader’ who will submit the result to PNAS, and will identify subgroups of attendees (and a few others who were unable to attend) who would work together to produce a dialogue/perspective on each such practice. If the proposal is approved by PNAS, the dialogues would be produced and submitted for publication by a date decided at the conference, but likely near the end of 2023.
An example of a dialogue has been published. It is a discussion of the criterion to be used for publishing unlikely results or claims. Links to it may be found on this website on the link “Published Dialogue”. The format of this dialogue is too informal for PNAS so a format more organized and scholarly will be used for each.
If more than eight dialogues are viewed as worth producing, it is possible that additional ones will be produced for submission to another journal.
Workshop Daily Organization:
The conference room (Pitla Sala from the city of Selva) is directly across the street behind the hotel (2 minutes walk) and will be available all day for use by the participants. We will try to arrange to have microphones and a sound system there so the discussions can be recorded (recording will be useful when the organizers produce the formal proposal to PNAS).
Discussions of topics and final selection will be held each day from 4-8. The final session will be used to select eight topics to produce as perspectives, and for each, assign a ‘leader’ who will handle the submission, and assign a set of attendees to produce it. (After the conference some who cannot attend in person will be added to the production teams).
Following the conference the organizers (Rich Shiffrin, David Kellen, Joachim Vandekerckhove) will produce the formal proposal to PNAS.
If there is sufficient interest, it is possible the attendees will decide to produce additional dialogues on other practices for submission to another journal.
The time each day before 4 PM (16:00) can be used for self-organized meetings, and/or the attendees can take advantage of the many outdoor activities in the spectacular Dolomites surrounding Selva. Light finger food and refreshments will be available in the hotel starting about 3:40 (15:40) each day.
Participants:
All those wanting to participate in production of the dialogues should register on this website on the link “Participants”, and should fill out the form on that page showing the conference days on which they can participate and/or lead discussions.
A limited amount of funding will be available to provide partial reimbursement of travel and local expenses. That funding will be directed primarily to the less senior attendees. If you are a graduate student, postdoctoral researchers or assistant professor and would want to request reimbursement indicate so on the form on the participants page.
