Many people are asking what is the best and common robots.txt file for a wordpress site/blog?
So here is the answer below. Copy this and paste in robots.txt file and save it in your wordpress root folder.
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content Disallow: /tag Disallow: /author Disallow: /wget/ Disallow: /httpd/ Disallow: /i/ Disallow: /f/ Disallow: /t/ Disallow: /c/ Disallow: /j/ User-agent: Mediapartners-Google Allow: / User-agent: AdsBot-Google Disallow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Disallow: / User-agent: ia_archiver-web.archive.org Disallow: / Sitemap: https://www.freedigitalworld.com/sitemap.xml #NOte: please replace freedigitalworld.com with your sitename..