基于实例和特征的迁移学习算法研究

上海交通大学硕士学位论文基于实例和特征的迁移学习算法研究姓名：戴文渊申请学位级别：硕士专业：计算机应用技术指导教师：俞勇20081201TransferLearningboosting–I–Instance-basedandFeature-basedTransferLearningABSTRACTTraditionalmachinelearningtechniquesmakeabasicassumptionthatthetrainingandtestdatashouldbeunderthesamedistributions.However,inmanycases,thisidentical-distributionassumptiondoesnothold.Theviolationoftheassumptionmighthappenwhenthetrainingdataareoutofdate,butnewdataareexpensivetolabel.Thisleavesplentyoflabeledexamplesthatareunderasimilarbutdifferentdistribution,whichisawastethrowawayentirely.Inthissituation,transferlearningbecomesimportanttotaketheroleoflever-agingtheseexistingdataknowledge.Transferlearningaimsatusinglearnedknowledgefromonecontexttobenefitfur-therlearningtasksinothercontexts.Thus,transferlearningdoesnotmaketheidentical-distributionassumptionastractionalmachinelearningalgorithms.Inthisthesis,webroadlyreviewthewholefieldoftransferlearning,andthenintroduceourrecentworkontransferlearningaccordingly.Ourworkcanbedividedintotwoparts:instance-basedtransferlearn-ing,andfeature-basedtransferlearning.Wewillshowthatinstance-basedtransferlearninghasbetterstrengthinknowledgetransferring,whilefeature-basedtransferlearningiswithmoregenerality.Wepresenttwotransferlearningalgorithmsbasedonboostingtechniqueandfeaturetranslationrespectively.Thesetwoalgorithmscorrespondstoinstance-basedandfeature-basedtransferlearning.Ourextensiveexperimentsshowthatouralgorithmscangreatlyimproveseveralstate-of-the-artalgorithmsinthesituationoftransferlearning,includingneartransferandfartransfer.KEYWORDS:TransferLearning,Instance,Feature–II–1–1........................23–1TrAdaBoost..........93–2TrAdaBoost..........................123–3..........................223–4peoplevsplaces............243–5TrAdaBoostpeoplevsplaces....244–1............264–2Flickr().....274–312.......................344–4¸TLRisk12..........34–IV–3.120Newsgroups.......................223.2SRAA............................233.31%.............234.1.......................32–V–(TransferLearning)1–1–1–1–1Figure1–1Transferlearninginourdailylifeboosting–2––3–(TransferLearning)90learninghowtolearn[1]lifelonglearning[2]multi-tasklearning[3][4,5](DARPA)EffectiveBayesianTransferLearning,EBTL˜russell/ebtl/workshopMichaelI.Jordan(ICML)(AAAI)(IJCAI)10NIPS2005transferlearningemphasizesthetransferofknowledgeacrossdomains,tasks,anddistributionsthataresimilarbutnotthesame.–4–2.1Caruana1997[3]BakkerHeskes[6]Jebara[7]Argyriou[8]Obozinski[9]ArgyriouICMLAAAIIJCAIMachineLearningJournalofMachineLearningResearch[10–12]2.2(AAAI)(KDD)AAAI2007TransferringNaiveBayesClassifiersforTextClassification[13](NaiveBayesClassifier)KDD2007Co-clusteringbasedClassificationforOut-of-domainDocuments[14][15][16][17–22]2.3–5–WebWeb[23][24]Heckman1979Econometrica[25]2000Zadrozny2004ICML[23]NIPS[26–29](ICML2007)BoostingforTransferLearning[30]boostingPAC2.4DoNg[31]Raina[32]Raina[33][33][34][35]–6–2.5AAAI[36,37]BickelScheffer[38][39]Rigutini[40]EM[41]–7–3.1Boosting[42,43]basetrainingdataauxiliarytrainingdata3.1.1boosting[3]AdaBoost[44]–8–(a)(b)“+”“¡”(c)“¡”(d)TrAdaBoost3–1TrAdaBoostFigure3–1AnintuitiveillustrationtoTrAdaBoost–9–boostingboostingAdaBoost[44]Hedge(¯)[44]boostingTransferAdaBoostTrAdaBoostTrAdaBoost3–1TrAdaBoost3.1.23.1():²XbbaseinstancespaceXaauxiliaryin-stancespacetargetinstancespace²Y=f0;1g²Tµf(X=Xb[Xa)£Yg²c:X7!Yx2Xc(x)2Y3.2():S=f(xti)g,xti2Xb,i=1;2;:::;k;–10–SkST3.3():Ta=f(xai;c(xai))g;xai2Xa;i=1;2;:::;n;Tb=f(xbj;c(xbj))g;xbj2Xb;j=1;2;:::;m.c(x)xTaTbnmT=f(xi;c(xi))gxi=8:xdi;i=1;:::;n;xsi;i=n+1;:::;n+m::(3.1)TaTbTbSTaSP(x;y)jx2Ta6=P(x;y)jx2STbTaSS²TaTbS²Learnerhf:X7!YSTbSTb–11–⑤䆁㒗᭄᥂Tb䕙ࡽ䆁㒗᭄᥂Ta䆁㒗᭄᥂THedge(ȕ)AdaBoost3–2TrAdaBoostFigure3–2ThemechanismofTrAdaBoostalgorithm3.1.3TrAdaBoostTrAdaBoostAdaBoost[44]AdaBoostAdaBoostFreundSchapireAdaBoost[44]TransferAdaBoostTrAdaBoostAdaBoostAdaBoostAdaBoostAdaBoostTbTaHedge(¯)[44]3–2TrAdaBoostAdaBoostTrAdaBoost–12–1TrAdaBoostTaTb(3.1)T=Ta[TbSLearnerN1.w1=(w11;::::::;w1n+m),aw1i=½1=ni=1;:::;n1=mi=n+1;:::;n+m2.¯=1=(1+p2lnn=N).Fort=1;:::;N1.ptpt=wtPn+mi=1wti:2.LearnerTTptSSht:X7!Y3.htTb²t=n+mXi=n+1wtijht(xi)¡c(xi)jPn+mi=n+1wti:4.¯t=²t=(1¡²t).b5.wt+1i=½wti¯jht(xi)¡c(xi)j;i=1;:::;nwti¯¡jht(xi)¡c(xi)jt;i=n+1;:::;n+mhf(x)=½1;PNt=dN=2eln(1=¯t)ht(x)¸12PNt=dN=2eln(1=¯t)0;aw1b²t1=2²t1=2²t1=2–13–TrAdaBoost1¯jht(xi)¡c(xi)j¯jht(xi)¡c(xi)j2(0;1]TrAdaBoostAdaBoostTrAdaBoostAdaBoost3.2TrAdaBoostTrAdaBoostTrAdaBoostAdaBoostHedge(¯)[44]3.2.13.4:lti=jht(xi)¡c(xi)j(i=1;:::;nt=1;:::;N)htTaloss3.5:btatTbTatthati=wtiPnj=1wtji=1;:::;nbti=wti+nPn+mj=n+1wtji=1;:::;m3.6:TrAdaBoost(Ta)NLa=NXt=1nXi=1atilti=NXi=1at¢lt:–14–3.7:TrAdaBoostxi(i=1;:::;n)NL(xi)=NXt=1lti:TrAdaBoost3.2.23.8Pni=1wN+1i3.8:l1;:::;lNlnÃnXi=1wN+1i!·¡(1¡¯)La:(3.2):t=1;:::;NnXi=1wt+1i=nXi=1wti¯lti:(3.3)¯lti·1¡(1¡¯)lti¯;lti2[0;1],nXi=1wt+1i·nXi=1wti(1¡(1¡¯)lti)=nXi=1wti¡(1¡¯)nXi=1wtilti:(3.4)ati=wtiPnj=1wtji=1;:::;n,nXi=1wt+1i·nXi=1wti¡(1¡¯)nXi=1ÃatiltinXi=1wti!=ÃnXi=1wti!¡1¡(1¡¯)at¢lt¢:(3.5)lnÃnXi=1wt+1i!·lnÃnXi=1wti!+ln¡1¡(1¡¯)at¢l