.Google's John Mueller answered a concern regarding why Google marks pages that are prohibited coming from creeping through robots.txt and why the it is actually risk-free to overlook the associated Search Console files regarding those crawls.Crawler Website Traffic To Inquiry Parameter URLs.The person asking the question chronicled that bots were actually developing hyperlinks to non-existent query specification Links (? q= xyz) to web pages with noindex meta tags that are likewise obstructed in robots.txt. What prompted the inquiry is actually that Google.com is actually creeping the links to those web pages, obtaining blocked through robots.txt (without noticing a noindex robots meta tag) after that obtaining turned up in Google.com Search Console as "Indexed, though shut out by robots.txt.".The individual inquired the observing concern:." But listed below is actually the major concern: why will Google.com index web pages when they can not also find the web content? What's the advantage in that?".Google.com's John Mueller verified that if they can't crawl the webpage they can't see the noindex meta tag. He likewise produces an interesting reference of the web site: hunt driver, recommending to neglect the outcomes considering that the "ordinary" customers will not find those end results.He wrote:." Yes, you are actually proper: if we can't crawl the web page, our experts can't find the noindex. That said, if our team can't crawl the web pages, after that there is actually certainly not a lot for our company to index. So while you could observe a few of those web pages with a targeted web site:- question, the typical user won't find them, so I would not bother it. Noindex is actually additionally great (without robots.txt disallow), it only means the URLs will find yourself being crawled (and also end up in the Explore Console document for crawled/not catalogued-- neither of these standings trigger problems to the rest of the web site). The essential part is actually that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the constraints being used the Website: search evolved search operator for diagnostic explanations. Some of those causes is actually considering that it is actually certainly not connected to the frequent search mark, it is actually a separate point completely.Google.com's John Mueller commented on the website hunt driver in 2021:." The brief answer is actually that a web site: question is not suggested to be comprehensive, neither made use of for diagnostics purposes.A website concern is actually a specific sort of hunt that confines the end results to a specific site. It is actually primarily just the word web site, a digestive tract, and after that the internet site's domain.This inquiry restricts the outcomes to a specific internet site. It is actually certainly not meant to become a thorough assortment of all the webpages from that web site.".2. Noindex tag without utilizing a robots.txt is actually great for these sort of scenarios where a bot is actually linking to non-existent webpages that are obtaining found through Googlebot.3. URLs along with the noindex tag will generate a "crawled/not catalogued" item in Search Console and that those won't have an unfavorable result on the remainder of the web site.Read the inquiry and also respond to on LinkedIn:.Why would Google index web pages when they can't even view the web content?Included Picture through Shutterstock/Krakenimages. com.