ILARWorkshoponReproducibilityIssuesin

ResearchwithAnimalsandAnimalModels.
AnOunceofPreventionworthaBillionof
Cure:ProactivePlanninginthePreclinical
ResearchArena
PaulBraunschweigerPh.D.,
Professor,RadiationOncology
CoFounderoftheCITIProgramattheUniversityofMiami
pbraunsc@med.miami.edu
Talkingpoints
Responsibleconductofresearch
Misconductvs.QuestionableResearchPractices
Physiologicconditionsaffectingreproducibilityin
preclinicalscience.
Environmentalsituationsthataffectreproducibility
PotentialSolutions.
WhatistheResponsibleConductofResearch
(RCR)
RCRisDisplayingIntegrityandProfessionalisminthePlanningof
Research,TheAccurateCollectionofDataandTheAnalysisof
ExperimentalResults.RCRAlsoEmbodiesReportingResearch
ResultsHonestlyandWithoutDeceptionorDeceit.
Althoughwearelivinginaculturethatisincreasinglymore
dishonest,moredeceitful,lesscivilandmorecorrupt,researchers
areheldtoahigherstandard.Researchersandtheresearch
enterpriseisunderincreasingscrutinyandexpectationtoconduct
sciencecollegially,professionallyandtothehighestethical
standards.
MisconductandQuestionableResearchPractices.
PHS/NIHandbytheNSF.
ResearchMisconduct
TheUSfederalgovernment(45CFR689)has
definedresearchmisconductas
Fabrication
Falsification
Plagiarism
1. Veryrigorousdefinitions.
2. Intentional
3. Preponderanceofevidence.
HowManyScientistsFabricateandFalsifyResearch?A
SystematicReviewandMetaAnalysisofSurveyData
DanieleFanelli PLOS:May29,2009
IncidentofMisconduct
Selfreport
2.0%(95%CI:0.94.5)ofscientistsadmittedtoFFP
Colleagues
1412%(95%CI:9.919.7)FF
Misconductwasreportedmorefrequentlybybiomedical
researchers.
Thedataseemstoindicatethattruemisconductisrare,but
otherssuggestthatitislikelyunderreported.
HowManyScientistsFabricateandFalsifyResearch?A
SystematicReviewandMetaAnalysisofSurveyData
DanieleFanelli PLOS:May29,2009
IncidentofQuestionableBehavior
Colleagues
Upto72%forotherquestionableresearchpractices
Selfreport
Upto33.7%admittedotherquestionableresearchpractices.
Reportedmorefrequentlybybiomedicalresearchersthan
others.
QuestionableResearchPractices
intheplanningofresearch,collectingresearchdata
andthereportingofResearch.
Inadequateresearchrecords
Incompletedatareporting.
Inappropriatesamplesize.Underpoweredstudydesign.
Addingadditionalanimalstoastudywherethedatafailsto
showasignificanteffect,mergingthedata.
Usinginappropriatestatisticalmethodsinappropriately.
Discardingoutliers.
Incentivetopublishresultsquickly
Whyissomuchofthedata
suspect?
Poorpeerreview
CITIProgramsatisfactionsurveyRCRcourse
~70%offacultyrespondersindicatedthatpeerreviewisdoneby
studentsandpostdocs.
Studentresponsesandfacultyresponsesweresimilar.
Poororincompletetraininginresearchethics.
Poororincompletetraininginresearchstudydesign.
Internationally,researchstudydesignisusuallylearnedfrom
theresearchadvisor.
Reasonsforlackofreproducibility
Poorqualitycontrolofreagents,cellslinesoranimalstrains.
Theuseofmodelsystemsthatmaybereproducibleand
reliableinthelab,butarenotpredictableforhuman
conditions.
Biasintheexperimentaldesignimposedbylackof
randomizationstrategies.
Biasimposedbytheresearcherknowingwhichgroupswere
gettingthetestagent.Noblinding.
Noorinappropriatecontrolgroups.
Reasonsforlackofreproducibility
Labenvironment
Intensecompetitionamonglaboratoriesandintralab
personnelpromotes:
Riskybehavior.TakingShortcuts.Thequickanddirtyendpoint
increasingthechanceoferrorandinappropriateconclusions.
Questionableresearchpracticesandperhapsmisconduct
Poorcontrolofexperimentalconditionsfromexperimentto
experiment.
Busy,inattentivesupervisors.
SloppyScience
Reasonsforlackofreproducibility
QuestionablePublicationPractices
Failuretoreportbasicelementsofexperimentaldesign.For
example:
Animalcharacteristics e.g.,Strain,age,gender,bodyweight,source,
acclimationperiod.
Husbandryissues.cagechangefrequency,beddingtype,enrichment,
temp,lightdarkcycles(circadianandcircannual rhythms),activity
cycles.
Treatmentbasics. TreatmentAMorPM,Reagentqualitycontrol.
Machinecalibrations.
Studydesign.Blinding,randomization,replications,samplesize
calculationandgenderdifferences.
Reasonsforlackofreproducibility
QuestionablePublicationPractices
Failuretoreportbasicelementsofexperimental
design.Forexample:
The'secretsauce' tomaketheirexperimentswork
withholddetailsfrompublicationordescribethemonly
vaguelytoretainacompetitiveedge.
Dishonestanddeceptiveauthorshippractices.
Sloppyscience
Unprofessionalirresponsiblescience
Whatcanbedonetoenhance
reproducibility
Engagenationalanimalresearchassociationsto
promotereproducibilityofscience.
PRIMR,AALAS,AAALACInt.,SCAW,NABR,ILAR
Promoteaccreditationofthelabanimalprograms.
EncourageIACUCstobemorerigorousintheirprotocol
review.
Hypothesistesting,modelappropriateness,modelfailure.
Askwhatwillbedonetoassuredataqualityand
reproducibility.
Promotetheinclusionsofstatisticians(consultants)inthe
researchteam.
Whatcanbedonetoenhance
reproducibility
Nationalorganizationscanpromote
Open,transparentandcollaborativeresearch
Promotetheuseofwebbasedenotebooks.
Opendata,promotecriticalanalysis,collaboration.
RetaintheIP
MichaelRubacha,AnilK.RattanandStephenC.Hosselet.AReviewofElectronic
LaboratoryNotebooksAvailableintheMarketToday.JournalofLaboratory
Automation201116:90
https://www.slas.org/default/assets/File/2012_JALA_Readers_Choice_Award.pdf
Whatcanbedonetoenhance
reproducibility
NationalorganizationscanPromote
Multicenterefficacytrialsofagentsinpipelineforclinical
trial.
NationalCenterforAdvancingTranslationalSciences (NCATS)
CTSA.
Hugegrantsdesignedtotransformrecipientinstitutions(CTSI)intoa
centersthatencourage,enableandexpeditethetranslationofbasic
preclinicalsciencetoclinicalapplication.
Mostlytheclinicalsideoftranslationalresearch(GCP,biostatistics,
clinicalresearchstudydesign).
Whatcanbedonetoenhance
reproducibility
EngagetheEducationProviders
ALAS,CITIProgramattheUofMiami,SQA
Topics
Labanimalwelfare,andtheRCR
Responsiblepublicationpractices
Researchstudydesignwithanimals.
Quantitativestudydesign
Factorsknowntoaffectresponses.
Biostatistics.
GoodLaboratoryPractice;GLPlabs.
Conclusions
Thekeytopromotingreproduciblescienceisforall
membersoftheresearchteamtohaveaclear
understandingthat:
1. Promotingdataintegrityandreproducibilityofscienceisa
sharedresponsibilityinourcommunity.
2. Sloppinessinplanning,executionandreportingofanimal
researchisunprofessionalandunacceptable.
3. LargerLabs,morestudentsandpostdocs,playingagreater
role.MoreresponsibilityfallsonthePItoinsurestaffare
technicallyproficientandthattheplanning,executionand
reportingofresearchistothehigheststandards.
4. Peerreviewofmanuscriptsandgrantapplicationsshould
notberelegatedstudentsandpostdocsw/opermission.
Further:
4. Greateremphasisshouldbegiventotrainingstudentsand
younginvestigatorsinresearchstudydesign.
5. Nationalorganizationsthatpromoteanimalwelfareand
animalresearchshouldtakeanactiveroleinpromotingthe
reproducibilityofscience.
6. Ifwedonotgetthehouseinorder,theissueswillbecome
politicizedandadditionalregulatoryburdenwillbeimposed
toinsurethatthePublicsTrust inresearchandthepublically
fundedresearchenterpriseispreserved.