A statement from google
We use a huge set of computers to fetch (or "crawl") billions of pages on the web. Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.
จะเห็นได้ว่า Algorithm ของเขาไม่ได้เปิดเผย แต่หากดู Pattern ออกก็พอทำ Algorithm ย้อนกลับได้ ส่วน Web crawler ที่ฟรีก็มีหลายตัว เอาไปศึกษาดูครับ
Abot (Developer Site Link) is a C# web crawler built for speed and flexibility, Apache License 2.0 License (free for commercial and personal use)
Aspseek is a crawler, indexer and a search engine written in C++ and licensed under the GPL
DataparkSearch is a crawler and search engine released under the GNU General Public License.
GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used to mirror Web and FTP sites.
GRUB is an open source distributed search crawler that Wikia Search used to crawl the web.
Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java.
ht://Dig includes a Web crawler in its indexing engine.
HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL.
ICDL Crawler is a cross-platform web crawler written in C++ and intended to crawl Web sites based on Website Parse Templates using computer's free CPU resources only.
mnoGoSearch is a crawler, indexer and a search engine written in C and licensed under the GPL (Linux machines only)
Nutch is a crawler written in Java and released under an Apache License. It can be used in conjunction with the Lucene text-indexing package.
Open Search Server is a search engine and web crawler software release under the GPL.
PHP-Crawler is a simple PHP and MySQL based crawler released under the BSD. Easy to install it became popular for small MySQL-driven websites on shared hosting.
the tkWWW Robot, a crawler based on the tkWWW web browser (licensed under GPL).
Scrapy, an open source webcrawler framework, written in python (licensed under BSD).
Seeks, a free distributed search engine (licensed under Affero General Public License).
YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL).
พัฒนาเว็บไซต์
