ถ้าเว็บของ จขกท เป็น wordpress ลองใช้อันนี้ก็ได้ครับ
อนุญาติเฉพาะตัวหลักๆบอทอื่นบล็อคหมด
แต่บางทีก็มีบอททำตัวมึนๆไม่อ่าน robots.txt ก็มี
User-agent: Googlebot
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-agent: Adsbot-Google
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-agent: Googlebot-Image
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-agent: Googlebot-Mobile
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-agent: MSNBot
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-agent: bingbot
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-agent: Slurp
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-Agent: Yahoo! Slurp
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-Agent: facebookexternalhit
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Allow: /
User-agent: *
Disallow: /
Sitemap: http://www.example.com/sitemap.xml
ความรู้ทั่วไป
