through the web log we can clearly see the user and search engine spiders visit the site, and the formation of a data, these data can let us know the search engine for the site’s attitude, health and website. There are many, we get through the web log index such as the number of visits, residence time, grab the amount, grab the directory page capture statistics, statistics, spider access IP, HTTP status code, spider, spider crawling path active period etc..
because of the Dragon Boat festival. The author did an experiment, and wrote a report on "experimental search engine does not include the site content and the chain factors have no relationship of the experiment", the specific content of experiment, here will say no more. Because the experimental results according to the leyuanbaby贵族宝贝, and did not achieve the desired effect, so I do not give up, through the website log to see how spiders have not included the link I crawl. This process, get some analysis on the web log experience, to share with everyone here.
#Version: #Date: 2013-05-27 16:44:28 1; #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip CS (User-Agent) sc-status sc- substatus sc-win32-status 2013-05-27 16:44:27 W3SVC195483716 22.214.171.124 GET; /index.html – 80 – 126.96.36.199 Mozilla/5.0+ (compatible; +Baiduspider/2.0; ++贵族宝贝baidu贵族宝贝/search/spider.html & nbsp, 200064) 2013-05-27 16:45:15 W3SVC195483716 188.8.131.52 GET; /index.html – 80 – 184.108.40.206 Mozilla/>
then the following examples to see how the web log analysis:
#Software: Microsoft Internet Information Services 6.0