An experiment to assess the cost-benefits of code

IEEETRANSACTIONSONSOFTWAREENGINEERING,VOL.23,NO.6,JUNE1997329AnExperimenttoAssesstheCost-BenefitsofCodeInspectionsinLargeScaleSoftwareDevelopmentAdamA.Porter,Member,IEEE,HarveyP.Siy,Member,IEEEComputerSociety,CarolA.Toman,andLawrenceG.Votta,Member,IEEEAbstract—Weconductedalong-termexperimenttocomparethecostsandbenefitsofseveraldifferentsoftwareinspectionmethods.Thesemethodswereappliedbyprofessionaldeveloperstoacommercialsoftwareproducttheywerecreating.Becausethelaboratoryforthisexperimentwasalivedevelopmenteffort,wetookspecialcaretominimizecostandrisktotheproject,whilemaximizingourabilitytogatherusefuldata.Thisarticlehasseveralgoals:1)todescribetheexperiment’sdesignandshowhowweusedsimulationtechniquestooptimizeit,2)topresentourresultsanddiscusstheirimplicationsforbothsoftwarepractitionersandresearchers,and3)todiscussseveralnewquestionsraisedbyourfindings.Foreachinspection,werandomlyassignedthreeindependentvariables:1)thenumberofreviewersoneachinspectionteam(1,2,or4),2)thenumberofteamsinspectingthecodeunit(1or2),and3)therequirementthatdefectsberepairedbetweenthefirstandsecondteam’sinspections.Thereviewersforeachinspectionwererandomlyselectedwithoutreplacementfromapoolof11experiencedsoftwaredevelopers.Thedependentvariablesforeachinspectionincludedinspectioninterval(elapsedtime),totaleffort,andthedefectdetectionrate.Ourresultsshowedthatthesetreatmentsdidnotsignificantlyinfluencethedefectdetectioneffectiveness,butthatcertaincombinationsofchangesdramaticallyincreasedtheinspectioninterval.IndexTerms—Softwareinspection,controlledexperiments,industrialexperimentation,ANOVA,poweranalysis.——————————✦——————————1INTRODUCTIONORalmost20years,softwareinspectionshavebeenpromotedasacost-effectivewaytoimprovesoftwarequality.Althoughthebenefitsofinspectionshavebeenwellstudied,theircostsareoftenjustifiedbysimplyobservingthatthelongeradefectremainsinasystem,themoreex-pensiveitistorepair,andthereforethefuturecostoffixingdefectsisgreaterthanthepresentcostoffindingthem.However,thisargumentissimplistic—forexample,itdoesn’tconsidertheeffectinspectionshaveonschedule[23].WehaveobservedthatatypicalreleaseofLucentTech-nologies’5ESS´switch[15](0.5Mlinesofaddedandchangedcodeperreleaseonabaseof5Mlines)canrequireroughly1,500inspections,eachwithfour,five,orevenmoreparticipants.Besidestheobviouslaborcosts,holdingsuchalargenumberofmeetingscanalsocausedelays,whichmaysignificantlylengthenthedevelopmentinterval(calendartimetocompletedevelopment).1Sincelongdevelopmentintervalsrisksubstantialeconomicpenalties,thishiddencostmustbeconsidered.Wehypothesizedthatdifferentinspectionapproachescreatedifferenttradeoffsbetweenminimuminterval,minimumeffort,andmaximumeffectiveness.Butuntilnowtherehavebeennocontrolledexperimentstoiden-tifythemechanismsthatdrivethesetradeoffs.Wecon-ductedsuchastudy,andourresultsindicatethatthechoiceofapproachsignificantlyaffectsthecost-effectivenessoftheinspection.Below,wereviewtherelevantresearchlitera-ture,describethevariousinspectionapproachesweex-amined,andpresentourexperimentaldesign,analysis,andconclusions.1.1InspectionProcessSummaryandLiteratureReviewToeliminatedefects,manyorganizationsuseaniterative,three-stepinspectionprocedure:preparation,collection,andrepair[11].First,ateamofreviewerseachreadstheartifactseparately,detectingasmanydefectsaspossible.Next,thesenewlydiscovereddefectsarecollected,usuallyatateammeeting.Theyarethensenttotheartifact’sauthorforrepair.Undersomeconditionstheentireprocessmayberepeatedoneormoretimes.Theresearchliteratureshowsthatseveralvariantsofthisapproachhavebeenproposedinordertoimproveinspec-tionperformance.TheseincludeFaganInspections[8],Ac-tiveDesignReviews[16],N-FoldInspections[19],PhasedInspections[13],andTwo-PersonInspections[2].0098-5589/97/$10.00©1997IEEE————————————————•A.A.PorteriswiththeComputerScienceDepartment,UniversityofMaryland,CollegePark,Maryland20742.E-mail:[e-mail].•H.P.SiyandL.G.VottaarewiththeSoftwareProductionResearchDe-partment,BellLaboratories,InnovationsforLucentTechnologies,Room1G-347,1000E.WarrenvilleRd.,Naperville,IL60566.E-mail:{hpsiy,votta}@research.bell-labs.com.•C.A.TomaniswiththeGlobalDataDevelopmentDepartment,LucentTechnologies,2B-265,2000N.NapervilleRd.,P.O.Box3033,Naperville,IL60566.E-mail:[e-mail].3,1996;revisedFeb.24,1997.RecommendedforacceptancebyJ.C.Knight.Forinformationonobtainingreprintsofthisarticle,pleasesende-mailto:[e-mail],andreferenceIEEECSLogNumber101201.1.F1.Asdeveloper’scalendarsfillup,itbecomesincreasinglydifficulttoschedulemeetings.Thispushesmeetingdatesfartherandfartherintothefuture,increasingthedevelopmentinterval[1].330IEEETRANSACTIONSONSOFTWAREENGINEERING,VOL.23,NO.6,JUNE1997Eachoftheseiscreatedbyrestructuringthebasicprocess,e.g.,rearrangingthesteps,changingthenumberofpeopleworkingoneachstep,orthenumberoftimeseachstepisexecuted.Severalalsorequiretheuseofspecialdefectdetec-tionmethods.Althoughsomeofthesevariantshavebeenevaluatedempirically,thefocushasbeenontheiroverallperformance.Veryfe