Home Online Strategies for Beginners Optimize Your Robot.txt File for Optimal Indexation

Optimize Your Robot.txt File for Optimal Indexation

27
0
Optimize Your Robot.txt File for Optimal Indexation

Optimizing your  robots.txt file, which is locted at the end of your url as such, https://yourdomain.com/robots.txt

This file plays a vital role in how the robots scripts read and index your site. We made it easy for your by offering you a free optimized robots file. Just download and change the location of your site map, upload and then connect it to your Google™ search console.

What an optimized robot file is:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /*.css
Allow: /*.js
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /readme.html
Disallow: /license.txt
Disallow: /xmlrpc.php
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: */disclaimer/*
Disallow: *?attachment_id=
Disallow: /privacy-policy

User-agent: Googlebot
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Mediapartners-Google
Allow: /

User-agent: AdsBot-Google
Allow: /

User-agent: AdsBot-Google-Mobile
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Msnbot
Allow: /

User-agent: msnbot-media
Allow: /wp-content/uploads/

User-agent: Applebot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Qwantify
Allow: /

Host: https://your_domain.com  (change this)
Sitemap: https://your_domain/sitemap_index.xml (change this)

LEAVE A REPLY

Please enter your comment!
Please enter your name here