Robots.txt tester failed
Hi, I would like to understand the reason why my robots.txt tester keeps on failing. I have emailed tech support who told me to review my sitemap xml file, however I’m running 3 blogs with identica...
Forum Discussion
It says “Failed to update robots.txt file. Please try again after some time.”
shenyihui I'd suggest searching for "robots.txt validators" and validating your robots.txt first. If the structure of robots.txt is valid but some pages aren't yet indexed, then something else must block crawling.
Also, it takes some time for Bing to update its results after changing anything in robots.txt.
Also, it takes some time for Bing to update its results after changing anything in robots.txt.
Thank you for your reply, ctomczyk . Will defo try that.
An additional detail though, my blog is updated several times a week and the newer posts are getting indexed. However the old ones are not, I even checked on the url inspection where the status stated “Discovered but not crawled”.
Even after I fetched the sitemap or submit the urls, the discovery date never changed nor the status move on to crawling stage.Do you happen to have any suggestions for this?
- shenyihui “Discovered but not crawled” - that might be the issue with canonical link or you may have duplicated content so the search engines sees same content under the two different URLs.
> Even after I fetched the sitemap or submit the urls, the discovery date never changed nor the status move on to crawling stage.
If the sitemap contains correct URLs then all you can do is wait until search engine will update their data. Or eventually contact Webmaster Support - Bing Webmaster Tools: https://www.bing.com/webmasters/help/webmasters-support-24ab5ebf