Noindex dilemma - How to please the search engines

Sure thing :slight_smile: The one thing I’ve always tried to keep in mind with a robots.txt as well is that not all “spiders” will actually listen to it and some will use it to view the files you specifically don’t want viewed - leaving them out of the robots.txt will result in the files not being crawled unless they’re referenced/indexed somewhere else as others have said above.

You are right. However when someone talks about SEO he or she keeps in mind certain SE and does not cares about others. And the most important to know and understand that your SE you are interested in listen your robots.txt