Robots.txt tester failed

Question

Hi, I would like to understand the reason why my robots.txt tester keeps on failing.
I have emailed tech support who told me to review my sitemap xml file, however I’m running 3 blogs with identical sitemap and robots structure and this problem only occurs in 1 of them.

I have done robots testing on Bing Webmaster and the results stated that nothing crawling is allowed. Some my pages are already indexed by Bing as well.

Could someone let me know step by step on what I could do to remedy this issue?

Thanks in advance!

ctomczyk · Answer

shenyihui&nbsp;Can you paste the error message here?

shenyihui · Answer

It says “Failed to update robots.txt file. Please try again after some time.”

ctomczyk · Answer

shenyihui I'd suggest searching for "robots.txt validators" and validating your robots.txt first. If the structure of robots.txt is valid but some pages aren't yet indexed, then something else must block crawling.

Also, it takes some time for Bing to update its results after changing anything in robots.txt.

shenyihui · Answer

Thank you for your reply,&nbsp;ctomczyk&nbsp;. Will defo try that.An additional detail though, my blog is updated several times a week and the newer posts are getting indexed. However the old ones are not, I even checked on the url inspection where the status stated “Discovered but not crawled”.Even after I fetched the sitemap or submit the urls, the discovery date never changed nor the status move on to crawling stage.Do you happen to have any suggestions for this?

ctomczyk · Answer

shenyihui “Discovered but not crawled” - that might be the issue with canonical link or you may have duplicated content so the search engines sees same content under the two different URLs.&gt; Even after I fetched the sitemap or submit the urls, the discovery date never changed nor the status move on to crawling stage.If the sitemap contains correct URLs then all you can do is wait until search engine will update their data. Or eventually contact Webmaster Support - Bing Webmaster Tools: https://www.bing.com/webmasters/help/webmasters-support-24ab5ebf

Forum Discussion

Robots.txt tester failed

Share

Resources