HobbySEO - Marathon XML Sitemap and Link Crawler

HobbySEO - Marathon XML Sitemap and Link Crawler 1.1

A quick way to create a XML SItemaps for one ore more websites

PROS

  • Steady run - like on Marathon

CONS

  • New features that are in plan to be added.

A quick way to create a XML SItemaps for one ore more websites

The main functionality for the Marathon XML SItemap is to create reliable XML Sitemaps. It is a need to compare different results generated by different softwares or services that provide this functionality. It incorporates a crawler that has a parametrizable timing section: the delay between two consecutive sites can be set and the delay for browsing the same IP. In this way web servers are not overloaded with bulk requests. For each site crawled a list of pages is generated. In fact, for performance reasons, multiple physical files are created on disk. After the process of crawling is completed, the user can generate the actual XML SItemap.

The Marathon XML Sitemap Crawler detects also all the outbound links for the crawled sites. This is a useful feature when an analysis is performed for the monitored sites. Also, it takes into account the robots.txt files downloaded from the crawled sites. In the sitemap are included only pages for which the HTTP response codes are 200, 301 and 302.

It does not use a relational database, It's using a local database based on text files, and due to this approach the browsing speed remains steady even if the total number of pages browsed increase in time.

This software can be used by SEO companies that provide XML Sitemap services, by students and by hobbyists.

HobbySEO - Marathon XML Sitemap and Link Crawler

Download

HobbySEO - Marathon XML Sitemap and Link Crawler 1.1