Are we allowed to make alterations to the robots.txt file? We are digging into our Google Seach Console and Google is trying to index our API, and we don't want this to happen. I want to add it to the disallow list
Are we allowed to make alterations to the robots.txt file? We are digging into our Google Seach Console and Google is trying to index our API, and we don't want this to happen. I want to add it to the disallow list
When you say API are you meaning a REST API? If using the standard process for creating REST Apis they are already excluded in the robots.txt file
When you say API are you meaning a REST API? If using the standard process for creating REST Apis they are already excluded in the robots.txt file
Robots.txt:
User-agent: * Disallow: /*/tags/ Disallow: /tags/ Disallow: *.axd$ Disallow: /api/core/
URLs that are failing indexing:
If this is your file its already been modified. There is no such thing as /api/core/ and there should have been an entry for Disallow: /api.ashx. You are also missing several other important entries, I would go back to a default install package and get the the correct file.
We are cloud hosted, should I put in a request with Support to compare what we currently have to the v12 default?
Yes