.Google's John Mueller answered an inquiry concerning why Google indexes webpages that are disallowed coming from creeping by robots.txt as well as why the it is actually risk-free to ignore the relevant Browse Console files regarding those crawls.Bot Website Traffic To Inquiry Guideline URLs.The individual asking the question recorded that bots were actually making hyperlinks to non-existent inquiry criterion Links (? q= xyz) to pages with noindex meta tags that are actually additionally blocked out in robots.txt. What motivated the inquiry is actually that Google is crawling the hyperlinks to those web pages, obtaining shut out through robots.txt (without noticing a noindex robots meta tag) after that receiving reported in Google Explore Console as "Indexed, though blocked through robots.txt.".The person inquired the adhering to inquiry:." But here is actually the big question: why will Google.com index web pages when they can't also find the web content? What is actually the perk because?".Google.com's John Mueller verified that if they can not crawl the webpage they can't view the noindex meta tag. He likewise creates an intriguing acknowledgment of the website: hunt driver, recommending to dismiss the results given that the "common" customers won't view those results.He composed:." Yes, you're right: if we can not creep the webpage, our team can not see the noindex. That said, if our experts can not crawl the pages, at that point there's certainly not a whole lot for us to index. Thus while you may see a few of those webpages along with a targeted internet site:- inquiry, the common customer won't see them, so I wouldn't bother it. Noindex is actually additionally great (without robots.txt disallow), it only implies the URLs will certainly find yourself being crept (as well as wind up in the Look Console document for crawled/not indexed-- neither of these standings lead to concerns to the rest of the website). The integral part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limits in using the Internet site: search progressed hunt operator for analysis factors. Some of those causes is actually because it is actually not attached to the regular search mark, it is actually a separate point entirely.Google's John Mueller talked about the website hunt driver in 2021:." The brief solution is actually that an internet site: concern is not indicated to be full, neither utilized for diagnostics purposes.A web site question is actually a details sort of search that limits the results to a specific site. It is actually primarily just the word site, a colon, and then the internet site's domain name.This concern limits the results to a specific web site. It is actually not indicated to be an extensive selection of all the pages from that internet site.".2. Noindex tag without using a robots.txt is great for these type of situations where a crawler is actually linking to non-existent pages that are actually obtaining found out through Googlebot.3. URLs along with the noindex tag are going to generate a "crawled/not indexed" entry in Explore Console which those will not possess an adverse result on the remainder of the website.Read the concern as well as answer on LinkedIn:.Why will Google mark webpages when they can not even view the information?Included Picture by Shutterstock/Krakenimages. com.