海量Web信息搜集系统优化设计

Web(CERN)[3]19992Internet16.0M[4]10()20001[5]350[2]1000(Webgather)CERNETWebGather[3]19971029CERNET19987199812CERNET5,000,00030,000“”internet2.1“”,[refer]”restart”gatherURLgatherCPU,gather,gather—[ref]stopword”5URLP0P0hyperlink,URLURLURL,JunghooCho3.1://:20001URL””3.2(P)P1://(hitnum)(clicknum)(url_node),1494,4,URL129,4,md54,4,Md516444444urlurl_nodemd5nodemd5md5_node,40,URL,url_nodechainp_md5,md5_nodep_url3.3url_nodemd5_nodeurl3.3url_nodemd5_nodeUrl_nodeurlurl,,url_node1,000,000urlurl442*1,999,993md5_nodeMd5_nodeMd516urlMd5_hashurl_hash1,000,000,1,999,993md5_hash42MBurl_hash157MB(mmap)malloc—mallocmalloc,1000000V1,V2,V3,......,V999999,V1000000H1,H2,H3,......,H999999,H1000000,C1,C2,C3,......C1000000,URLD1,D2,......,D1000000,S1,S2,......,S1000000,Si=1URLV’i=Vi/’’’’’’’’’’’–’2.604480e-04mis_D=’–’6.342357e-05mis_C=’–’6.321719e-05mis_S=’–’6.298261e-05,1://hff.21cn.com:88/book/index.htm423.423.4{Hi}mis_H’–’7.16635e-04mis_D=’–’9.55940e-03mis_C=’–’6.93268e-03mis_S=’–’7.17072e-03868357131906%15.1902962876462589868%20.506720999999160896%16.0896176479796947179%14.6174323.5-3.83.53.63.83.73.5-3.8~xzm/~xzm/index.html,index.htmlURLURL8()MicorsoftInternetInformationServer2.0Manuls4HTML,Postscript20%30%512htmltitleURLmd516100099%TCP(30)IP(29)201610MAC54.1222||||||βαβα+−=misNmd5Nmd52.Nmd5NNN+1n100n1,182,899url_nodemd5_nodeLIMIT=0.01,N=10(%)(%)1,182,899199,76316.89981,182,899212,91418.00981,182,899210,47517.79981,182,899219,84118.58971,182,899225,91919.10971,182,899265,75222.4797512LIMITLIMIT=0.1(%)(%)1,182,899211,71817.90981,182,899222,15118.7897LIMITNN4.2,,10NNNM(DocID),(ChunkID),(signature)Naraynan24,000,000%461749225422CPU(PnO(n)IEEE802.3htmlTagtitle....../titleiWi,nWeight(Tag)4.1∑=+=njjiTagWeightBaseW1)(5.1HTMLtagTagWeight(tag)TagWeight(tag)TITLE40DT4CITE8LI4EM2UL4STRONG4A4B4FONTSIZE=716I2FONTSIZE=612BIG4FONTSIZE=58H112FONTSIZE=44H28FONTSIZE=31H34FONTSIZE=21H41FONTSIZE=11H51FONTSIZE=+14W},255min{∑=i=”*.edu.cnURLURL1