Jcrawler Component - GOLIBU | Professional Link Building Services

Written by Lee Ng.

Saturday, 16 January 2010 02:41

JCrawler does the same as the Google crawler. It crawls your Joomla site and stores all the urls in a XML file without any plugins. After crawling you can ping Google, Ask.com, Moreover, Yahoo and MSN.

Features:

- sitemap generation on the fly, no additional plugins needed
- submit sitemap to 5 searchengines (google, yahoo, msn, ask.com, moreover)
- Automatic priority calculation based on internal PageRank (experimental)
- shows bad links, or not crawlable sites
- with curl: max 250 parallel connections to spider urls (configurable)
- exclude list
- modification of robots.txt with the location of the sitemap
- crawling with curl of fopen
- saves config in a xml-file
- Performance and compatibility improvments
- Submission to search engines via curl
- fixed empty loc-tag problem
- improved link detection

Note:
- fopen or Curl is required to crawl the site

Download