I need to create a robots.txt file that does the following:
Disallow all robots from accessing certain pages on the site.
But, as an exception, allow robot X to access the entire site.
My robots.txt contains something like this:
Can someone tell me if this will achieve my aim? I'm not sure whether the second record will completely override the first (in which case, the * would also apply to robot-x) or whether the * means, in effect, "all robots except the one mentioned above".
Hope this makes sense.