Robots.txt allow only /.well-known/assetlinks.json

General Web Dev
#1

Hello everyone. I have an android app that’s using my site URLs to get and post data. Those URLs are private and are hidden to search engines, that’s why my robots.txt are disabled like this:

User-agent: *
Disallow: /

However, Google asks me to verify my “domain” in order to use app links

So I need to add one file into my website root like this: https://mywebsite/.well-known/assetlinks.json

But as said before I don’t want to open robots.txt because I want to keep all my other URLs private (hidden).

How would be the best way to permit robots.txt only to /.well-known/assetlinks.json and disallow everything else?

I have tried 3 ways but not sure what’s the best one (if its).

User-agent: *
Allow: /$
Disallow: /

User-agent: *
Allow: /.well-known/assetlinks.json
Disallow: /

User-agent: *
Allow: /.well-known/
Disallow: /

Thanks!

#2

You should probably have the disallow before the allow.

1 Like
#3

Well, the files as written do slightly different things:

  1. Allow robots to look at the root page of your website only. (not what you want)
  2. Allow robots to look specifically and only at the file /.well-known/assetlinks.json
  3. Allow robots to look at any file in the /.well-known/ directory.
1 Like
#4

Does the order matter in robots.txt? As far I can understand reading the Google guidelines; is that Allow and Disallow are just executed at the same time no matters what position on the file they have

Thanks.

Thanks, that’s what I was figuring myself, pretty sure will be using the 3º option because in the near future will add another .well-known file to root directory from Apple.

Thanks.