VOT2016presentation

MatejKristan,AlešLeonardis,JiriMatas,MichaelFelsberg,RomanPflugfelder,LukaČehovin,GustavoFernandez,TomašVojir,GustavHager,AlanLukežič,etal.TheVisualObjectTrackingVOT2016:ChallengeandresultsKristanetal.,VOT2016resultsOutline1.ScopeoftheVOTchallenge2.VOT2016challengeoverview•Evaluationsystem•Dataset•Performanceevaluationmeasures3.VOT2016resultsoverview4.Summaryandoutlook2/42SCOPEOFTHEVOT2016CHALLENGEVOT20163/42Kristanetal.,VOT2016resultsSelectedclassoftrackers•Single-object,single-camera,model-free,short-term,causaltrackers•Model-free:•NothingbutasingletrainingexampleisprovidedbytheBBoxinthefirstframe•Short-term:•Trackerdoesnotperformre-detection•Onceitdriftsoffthetargetweconsiderthatafailure•Causality:•Trackerdoesnotuseanyfutureframesforposeestimation•Objectstatedefinedasarotatedboundingbox(rectangle)4/42VOT2016EVALUATIONSYSTEMVOT20165/42Kristanetal.,VOT2016resultsVOT2016Challengeevaluationkit•Matlab-basedkittoautomaticallyperformabatteryofstandardexperiments•Plugandplay!•Supportsmultipleplatformsandprogramminglanguages(C/C++/Matlab/Python,etc.)•Easytoevaluateyourtrackeronallourbenchmarks•BackwardcompatibilitywithVOT2013/VOT2014/VOT2015•DownloadfromourhomepageEval.KitTracker(C++/Matlab/Python,…)TraXprotocoli/ostreams•Currenttrend[Wuetal.CVPR2013,Smeuldersetal.PAMI2013,Wangetal.arXiv2015,Wuetal.PAMI2015]:•Largedatasetsbycollectingmanysequencesfrominternet•Largedataset≠diverseoruseful•VOT2013/2014/2015approach:•Keepitsufficientlysmall,wellannotatedanddiverse•DevelopedtheVOTdatasetconstructionmethodology8/42Kristanetal.,VOT2016resultsTheVOT2016dataset•TheperformanceonVOT2015datasetdidnotsaturatein2015challenge•Keptall60sequencesfromVOT2015challenge•NEW:Objectsre-annotated!9/42Kristanetal.,VOT2016resultsObjectannotationAutomaticboundingboxplacement1.Segmentthetarget(semi-automatic)2.Automaticallyfitaboundingboxbyoptimizingacostfunction•Visualverificationoftheresults•12%revertedtotheVOT2015annotation10/39Kristanetal.,VOT2016resultsVOT2016dataset–objectannotation•AverageoverlapbetweenVOT2015andVOT2016BB:0.7411/42Kristanetal.,VOT2016resultsAnnotationuncertainty•Segmentationuncertaintyresultsinboundingboxuncertainty•Uncertainty:Averageofoverlapsbetweenoptimalboundingboxandthosewithin7%𝐶increase.12/42𝐶7%Kristanetal.,VOT2016resultsPracticaldifferences13/42ReducedbyhalfcomparedtoVOT2015Kristanetal.,VOT2016resultsVOT2016dataset–frameannotation•ManuallyandautomaticallylabeledeachframewithVOT2013visualattributes(sameasVOT2015):(i)(ii)(iii)(iv)(v)(vi)000100100100100100000001iv.Objectsizechange(A)v.Cameramotion(M)vi.Unassigned(A)i.Occlusion(M)ii.Illuminationchange(M)iii.Objectmotion(A)M...manualannotation,A...automaticannotation14/42EVALUATIONMETHODOLOGYVOT201615/42Kristanetal.,VOT2016resultsPerformancemeasures•TargetlocalizationpropertiesmeasuredusingtheVOT2013/VOT2014/VOT2015methodology.•ApproachinVOT2013/VOT2014:•Interpretabilityofperformancemeasures•Selectasfewaspossibletoprovideclearcomparison•Basedonarecentstudy1twobasicweakly-correlatedmeasuresarechosen:•Robustness•Accuracy16/421Čehovin,Leonardis,Kristan.Visualobjecttrackingperformancemeasuresrevisited,IEEETIP2016Kristanetal.,VOT2016resultsVOTperformancemeasures•Robustness:Numberoftimesatrackerdriftsoffthetarget.•Accuracy:Averageoverlapduringsuccessfultracking.17/42Kristanetal.,VOT2016resultsVOTperformanceevaluation•Rankingmethodologyw.r.t.AccuracyandRobustness•Assignequalrankto“equally”performingtrackers:•Statisticalsignificanceofresultsandpracticaldifference•AprincipledwaytomergeAccuracyandRobustness:•Expectedaverageoverlap(EAO)poorwellpoorwell18/39Probabilityofstilltrackingafter𝑁𝑓framesKristanetal.,VOT2016resultsVOT2016Speedmeasurement•Reducethehardwarebiasinreportingtrackingspeed.•Approach:TheVOT2014speedbenchmark•Dividetrackingtimewithtimerequiredtoperformthefilteringoperation•EquivalentFilterOperations(EFO)600x600imageMaxoperationin30x30windowApplythisfiltertoallpixelsMeasurethetimeforfiltering19/42CHALLENGEPARTICIPATIONANDSUBMITTEDTRACKERSVOT201620/42Kristanetal.,VOT2016resultsVOT2016Challenge:participation•Participantswoulddownloadtheevaluationkit:•Evaluationsystem+Dataset•Integratetheirtrackerintotheevaluationsystem•Predefinedsetofexperimentsautomaticallyperformed–submittheresultsback•Requiredtosubmitbinaries/source•RequiredtooutperformaNCCtrackerParticipantVOT2016Page21/42Kristanetal.,VOT2016results70trackerstested!Diversesetofentries:70=48submissions+22existing•Deepconvolutionalneuralnetworks(MLDF,SiamFC-R,SiamFC-A,TCNN,DNT,SO-DLT,MDNet-N,SSAT)•Correlationfilters(SRDCF,SWCF,FCF,GCF,ART-DSST,DSST2014,SMACF,STC,DFCT,KCF2014,SAMF2014,OEST,sKCF,Staple,Staple+,MvCFT,NSAMF,SSKCF,ACT,ColorKCF,deepMKCF,HCF,DDC,DeepSRDCF,C-COT,RFD-CF2,NCC)•Discriminativemodels–singlepart(MIL,Struck2011,EBT,TGPR)•Globalgenerative-model-based(DAT,SRBT,ASMS,LoFT-Lite,IVT,CCCT,DFT)•Part-basedtrackers(LT-FLO,SHCT,GGTv2,MatFlow,Matrioska,CDTT,BST,TRIC-track,DPT,SM