计算机研究与发展ISSN

ISSN100021239PCN1121777PTPJournalofComputerResearchandDevelopment43(1):145152,2006:2004-09-02;:2005-05-27:(20040509190);-1,221(100083)2(100080)(wangzhiming@tsinghua1org1cn)AReviewofText2to2VisualSpeechSynthesisWangZhiming1,2andTaoJianhua21(DepartmentofComputerScienceandTechnology,BeijingUniversityofScienceandTechnology,Beijing100083)2(NationalLaboratoryofPatternRecognition,InstituteofAutomation,ChineseAcademyofSciences,Beijing100080)AbstractVisualinformationisimportanttotheunderstandingofspeech1Notonlyhearing2impairedpeo2ple,butpeoplewithnormalhearingalsomakeuseofvisualinformationthataccompaniesspeech,especiallywhentheacousticspeechisdegradedinthenoiseenvironment1Astext2to2speech(TTS)synthesismakescomputerspeaklikehuman,text2to2visualspeech(TTVS)synthesisbycomputerfaceanimationcanincor2poratebimodalityofspeechintohuman2computerinteractioninterfaceinordertomakeitfriendly1Thestate2of2the2artoftext2to2visualspeechsynthesisresearchisreviewed1Twoclassesofapproaches,parame2tercontrolapproachanddatadrivenapproach,aredevelopedinvisualspeechsynthesis1Fortheparametercontrolapproach,threekeyproblemsarediscussed:facemodelconstruction,animationcontrolparametersdefinition,andthedynamicpropertiesofcontrolparameters1Forthedatadrivenapproach,threemainmethodsareintroduced:videosliceconcatenation,keyframemorphing,andfacecomponentscombination1Finally,theadvantagesanddisadvantagesofeachapproacharediscussed1Keywordstext2to2visualspeech(TTVS);viseme;co2articulation;facemodel;facialanimation1,,1,-,1-1:1,1-(TTVS);;;;TP3911,1,1,©1994-2006ChinaAcademicJournalElectronicPublishingHouse.Allrightsreserved.[1],,11dB1McGurk[2],,,,,1(),1:,;,1,,(),,1,,,1213;;121130,1:121111Parke3D[3,4],11,,1,,,13D3D[5,6]13D,3D13D,1,3D1,,[79]121112,111Waters[10,11]11,6:1,;,1[1214],12123D,1,11,,EkmanFACS(facialactioncodingsystem)MPEG24FAP(facialanimationparameter)121211FACS1978,Ekman[15],6412006,43(1)©1994-2006ChinaAcademicJournalElectronicPublishingHouse.Allrightsreserved.(actionu2nit),,1FACS,FACS,1FCAS,[1618],1Sayette[19]FACS,AUAU,:FACS1Fig11Parametercontrolvisualspeechsynthesis1121212FAPMPEG24[20]FAP,:(visemesandexpressions)661,6:(joy)(sadness)(anger)(fear)(disgust)(surprise)11,FAP[21,22]1,MPEG24FDPFAP[2325]1213,,1(performancebased)[26],1[27],1,11,,1[28,29][12,30]1MPEG243:Hermite1,1CohenMassaro[31],,1,Cohen[23,32,33]1Cohen,[34]1,,1Cohen,1,,1Edge[35]741:-©1994-2006ChinaAcademicJournalElectronicPublishingHouse.Allrightsreserved.(spacetimeconstraintsmethod),1,1,1,1TTS,1FAP113,,1,1,,11997Bregler(videorewrite)[36],,3:1311Bregler[36]11,1,-(TTS),1,(visualtri2phones),(tri2viseme),1,HMM,1,1,,,12:Fig12Visualspeechsysthesisbasedonimagesequenceconcatenation12Cosatto,GrafHuang[3740]13D,1,8412006,43(1)©1994-2006ChinaAcademicJournalElectronicPublishingHouse.Allrightsreserved.(),(PCA)1Bregler,,1,Viterbi1[40],1[39],11Cao[41,42],[41](anime),[42]1,,,1312Bregler,Ezzat[4345]1,,,11,,1,1,1,Tony,,1313,CosattoGraf[4648]1(),,,3()1,,1,,1,,,1,1,1,,1,,1430,1,3:;,31;;,;1,,1,1:Cosatto[47]3D,,941:-©1994-2006ChinaAcademicJournalElectronicPublishingHouse.Allrightsreserved.[49],Edge[50]Sumedha[51],,1,[42,47,52],,1,,11A1Macleod,A1Q1Summerfield1Quantifyingthecontributionofvisiontospeechperceptioninnoise1BritishJournalofAudiology,1987,21(2):1311412H1McGurk,J1MacDonald1Hearinglipsandseeingvoices1Na2ture,1976,264(5588):7467483F1Parke1Computergeneratedanimationoffaces1ACMAnnualConf1,Boston,Massachusetts,UnitedStates,19724F1Parke1Amodelforhumanfacesthatallowsspeechsynchro2nizedanimation1JournalofComputersandGraphics,1975,1(1):145Y1Lee,D1Terzopoulos,K1Waters1Realisticmodelingforfacialanimation1The22ndAnnualACMConf1ComputerGraphics,LosAngeles,19956U1Neumann,J1Li,J1Y1Enciso,etal1Constructingarealisticheadanimationmeshforaspecificperson1UniversityofSouthernCalifornia,TechnicalReport:USC2CS2TR992691,19997A1C1A1Valle,J1Ostermann13Dtalkingheadcustomizationbyadaptingagenericmodeltooneuncalibratedpicture1The2001IEEEIntlSymposiumonCircuitsandSystems,Sydney,Aus2tralia,20018V1Blanz,T1Vetter1Amorphablemodelforthesynthesisof3Dfaces1SIGGRAPH99,LosAngeles,19999M1M1Cohen,J1Beskow,D1W1Massaro1Recentdevelop2mentsinfacialanimation:Aninsideview1TheWorkshoponAu2dio2VisualSpeechProcessing,Terrigal,Australia,199810K1Waters,D1Terzopoulos1Aphysicalmodeloffacialtissueandmusclearticulation1TheFirstConf1VisualizationinBiomedicalComputing,Atlanta,USA,199011D1Terzopoulos,K1Waters1Analysisandsynthesisoffacialim2agesequencesusingphysicalandanatomicalmodels,IEEETrans1PatternAnalysisandMachineIntelligence,1993,15(6):56957912B1Uz,U1Gudukbay,B1Ozguc1Realisticspeechanimationofsyntheticfaces1IEEEComputerAnimation(CA98),Philadel2phia,USA,199813Y1Zhang,E1Sung,E1C1Prakash1Aphysically2basedmodelforreal2timefacialexpressionanimation1ThirdIntlConf132DDigitalImagingandModeling,QuebecCity,Canada,200114R1M1Koch,M1H1Gross,A1A1Bosshard1Emotioneditingusingfiniteelements1ComputerGraphicsForum(Eurographics98)