Great Answers to
Questions About Everything


I have salesforce community (salesforce site) and have to index that site by google.

  1. How can I index my site by google? Ex:- https://Mywebsite.com/

  2. I have to allow few pages only(not all pages), how can I do this ?

  3. I have some user profiles which is a public page. it can be with query string parameter. How to index those user profiles by google ? Ex:- https://Mywebsite.com/apex/UserProfile?id=001K000000sLyjhIAC

This id will be changed based on the user profile .

{ asked by Ramesh }


Go to Setup > Develop > Sites. There should be an entry there corresponding to your community (there's a 1-1 relationship between a Site and a Community)

First create a Visualforce page containing (for example) the following:

<apex:page contentType="text/plain">
    User-agent: *
    Allow: /

Choose this page on the Site Robots.txt attribute of the Site associated with your Community.

Then confirm you can see its contents when you visit your community at https://mywebsite.com/robots.txt

User-agent: *
Allow: /

A bit more reading on how robots.txt works will help you allow/disallow the appropriate pages. Anything behind a login cannot be crawled, and those custom user profile pages must be discoverable via links.

{ answered by user320 }