Learn Something About Robots.txt In An Easy Way - Testing and Verification | Part 2/2

I explained in the first part a lot about robots.txt by showing examples like http://www.nike.com/robots.txt that has a logo at the end because # means nothing will be counted after #. I also explained almost all the key elements Allow: Disallow: User-agent etc as well as the disadvantage being not used robots.txt for the secret information on your site - the example of NTS site, students data leaked.
--
How to Check A Specific File is Being Blocked By Robots.txt Or Not?
--
Matlab Agr Robots.txt Block Kar Rha Hai to Google Engine Crawl Nhi Kary Ga Aur Phr Search Result Main Show Bhi Nhi Hoga.
Ab Suppose Apki Site Par Ek JPG Hai (Ya PDF) Jo aap Chahty Han Block Ho Jye Jo Google Search Main Show Na Ho. Is Ka Direct Link ye Hoga
To Aap Is Ko esy Likhy Gy Robots.txt Main
Disallow: /wp-content/uploads/2016/12/Bloggers-Funda.jpg
Jis Ka Look Esy Hoga Search Console main, See Screenshot Here http://prntscr.com/dhoxw6
Ab Check Karny Ky Liye Yahan Paste Karain Link, Then Aap Dekhain Gy Aapko Show Hoga "Blocked" Matlab Google Is File Ko Crawl Nhi Kary Ga Aur Na Hi Search Result Main Show Kary Ga. See here http://prntscr.com/dhoyjh
Agr Allow A Jye to Matlab Aap Ky Code Main Kye Mistake Hye Hai Jesy Mery Sy ye Hye Thi Ky Main yahan \\ Lga Diya Tha To Is Ki waja Sy Block Nhi Howa Aur Allow Ho Gya. See here http://prntscr.com/dhozkt
Isi Tarha Aap Ek Directory Ko Block Kary Then Check Kary Asal Main Wo File Ya Directory Block Hye Bhi Ya Nhi Ye Mainy Wesy Hi Likh Di. Ye Ek Tester Hai Jo Bta Dy Ga. Phr Aap Apni Site Ky Robots.txt Editor Main Ja Kr Changes Kar Lain Ya Direct Hosting Ki File Manager Main Change Kar Lain Then Yahan A Kar Submit Bhi Kar Dain Ta Ky Google Ko Pta Chal Jye Aap Ny Changes Ki Hai. See here http://prntscr.com/dhp1pn
If you haven't read its first part, then you will not get anything. Read that for the best understanding.
Note:
Sometimes Google shows result even you include the file in Robots.txt; the reason is that when a website gives a link (backlink) to the blocked file, Google crawls that and index. So, don't worry, if it's getting indexed. You can make use of https://www.google.com/webmasters/tools/url-removal in order to remove the result from the Google Search.
Learn Something About Robots.txt In An Easy Way - Testing and Verification | Part 2/2 Learn Something About Robots.txt In An Easy Way - Testing and Verification | Part 2/2 Reviewed by Hamza Bashir Ahmad on 09:25:00 Rating: 5