| tags: [ modelling QRP replication reproducibility ] categories: [planning ]

Advisory Committee Meeting

Chapter 2, QRPs

Scope

Trying to cover the scope of the entire decision process might be too large for just one chapter. Just examining QRPs for system modelling is quite a task. PV suggested restricting the focus to decision tools and focussing on one application area, such as conservation planning / reserve design. This would also give more traction and uptake among ecologists. Conservation planning isn’t using the term ‘reproducibility’ to describe their problems. But they are disussing how problems in data, for example, are propagating into decisions from Species Distribution Models. Other known issues are where decision-makers keep pushing for new reserve designs, after seeing the results of the modelling, and then pushing for new ones until it fits what they desire. This QRP seems akin to HARKing. An alternative application area to focus on, could be Bayesian Networks (Ann Nicholson has written a lot about best-practice for BN dev).

So a new breakdown for Chapter 2 / Chapter 3 could be as follows:

  • Chapter 2: focussing on system models
  • Chapter 3: can turn to the broader problem context, objectives and alternatives. Review literature for transparency around problem development / specification.

Defining QRPs (as opposed to ethical misconduct)

FF: As moving onto non-frequentist QRPs, must hold onto the formal definition of a QRP, as well as the set of external conditions promoting their occurrence: i.e., that a QRP increases the type 1 error rate, and their likelihood is increased by broader biases, such as publication bias, etc. In terms of the broader conditions, thinking back to the conservation-planning harking-esque error, we could separate broader biases into general and agenda-based ones. I.e. for a decision-making problem, decision-makers bring with them their own agendas, and this is context-dependent. For example, CH discussed the term ‘facipulator’, e.g. a facilitator who brings an agenda along to the elicitation workshop. She suggested a mini-case study using the Mallee-Fowl workshop. From 3 different workshop facilitators, 3 completely different models emerged!

So what would an equivalent be for decisions? PV: Perhaps it is an error in being able to descriminate between decision alternatives… So a QRP is something that increases discriminatory error.

Chapter 4, Case Study in Replication Experiment

Study Design

HF suggested using a cross-over design. This would involve multiple teams of researchers solving a decision problem, and then attempting to replicate the work of other teams. Variation can be measured in terms of both the final decision, but also in terms of divergence in methods throughout the process.

This would serve three different purposes, allowing us to investigate:

  1. At what points in the decision-process are divergent outcomes observed?
  2. How reproducible are the DSSs?
  3. It would also allow for a test of the QRP / transparency criteria designed from chapter 2.

Format and Incentives

Workshop, is probably the easiest format - have the data, and Chris available.

Might be able to host during coding-club for a ‘challenge session / hackathon’.

The length of time / commitment needed to complete the task isn’t so important in terms of the feasibility of this work. What’s crucial is getting the incentives right.

Chapter 1 Typology:

See notes under definition of QRPs, for considering how we measure reproducibility.

LR emphasised that when attempting a replication of a DSS / DST, DM values need to be held constant. How can we control for this? This could constitute one element of the replication criteria, and would be akin to elements like ‘population’ used in other typologies.

Next Actions:

  1. Scope of the QRP chapter - just decision ‘tools’ or system models? Focus on a particular application area.
  2. Start pulling together visual schematic of vocabulary - SDM, DT’s, Decision Support System, etc.
  3. Start considering the definition of QRPs for non-frequentist work.
  4. Ask PV / CH / supervisors for good examples of a confirmation report
  5. To discuss next meeting: Conflict of interest with CH and PV participating in reproducibility retreat workshop?
  6. Look into HF’s crossover replication design.