Log Analyzer will allow you to track spider visits and save log files with information on spider visits, as well as export it into different file formats. Robots.txt Editor is visual editor that will enable you to specify different directives for selected spiders in specific areas of the site and generate the robots.txt file quickly and easily. These two modules, along with FTP Uploader make up a powerful and handy tool.
keywords
Robots.txt Editor , robots.txt log file editor analyzer spider user-agent user agent |
