Sitecore Robots.txt

Integrating the robots.txt on sitecore is not so difficult. In this post, I will explain how you can implement the robots.txt on Sitecore.

Supposed you have multiple site and each site has its own robots.txt file. Placing the robot.txt file in the root folder, i.e, the Website folder is good. But since you are using Sitecore, instead of placing the file in the Website folder, you can place it in an item on Sitecore directly.

Sample Structure

In this post, I will take the following scenario, 3 sites in which each one will have its own robots.txt

site structure

Now, under each site, you may create an item named Robot or for a proper hierarchy, you can create a Setting Item which will inherite a template that will contain the Robot.txt description and values.

I will opt to create a setting item which will stored various setting of the site and which will inherit the Robot template.

Robot Template

The template will looks as follows:

SEO

After you have created the template, you need to link the template with your desired item. I will be associating the template to the Sample Item provided by Sitecore. Below is the final look how it will be. You will just need to input your values in the Multi-Line Text field.

robot

Code Implementation

In this section, I will explain the implementation of displaying the Robots.txt.

Configuration Setup

First of all, you will need to set the CustomHandlers in a configuration file as below:

 <customHandlers>
    <handler trigger="robots.txt" handler="RobotsTxtHandler.ashx" />
 </customHandlers>

In order to make it dynamic, you will need to setup a Provider. So for its configuration file, it will look as follows:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <robotsTxtSection>
      <!-- Set the value of the Robots.txt provider component here -->
      <robotsTxtProvider type="Your Namespace Here, Your Assembly Name" />
    </robotsTxtSection>
  </sitecore>
</configuration>

Code Setup

Create an ASHX file in your Solution and leave it blank. In the CodeBehind use the following code:

 public class RobotsTxtHandler : IHttpHandler
 {
     public bool IsReusable
     {
         get
         {
             return false;
         }
     }

     public void ProcessRequest(HttpContext context)
     {
         var siteRoot = Context.Database.Items.GetItem(string.Format("{0}{1}", Context.Site.RootPath, Context.Site.StartItem));
           
         if (siteRoot != null)
         {
             context.Response.Clear();
             context.Response.ContentType = "text/plain";
             context.Response.ContentEncoding = System.Text.Encoding.UTF8;

             if (Provider.RobotsTxtProvider != null)
             {
                 context.Response.Write(Provider.RobotsTxtProvider.GetRobotsTxtFileContent(siteRoot));
             }
         }

         context.Response.End();
     }
 }

You need to implement the Provider. The code is as shown below:

 public static class Provider
 {
     public static IRobotsTxtProvider robotsTxtProvider = InitializeProvider<IRobotsTxtProvider>("robotsTxtSection/robotsTxtProvider");

     public static IRobotsTxtProvider RobotsTxtProvider
     {
         get
         {
             return robotsTxtProvider;
         }
     }

     public static T InitializeProvider<T>(string providerConfigElement) where T : class
     {
         try
         {
             var config = Factory.GetConfigNode(providerConfigElement);

             return Factory.CreateObject<T>(config);
         }
         catch (Exception)
         {
             return null;
         }
     }
 }

The Provider will fetch the namespace that is required to search for the Robot.txt.

The last part is the code that will read the sitecore item field of the robot and return its value

 public class RobotsTxtProvider : IRobotsTxtProvider
 {
     public string GetRobotsTxtFileContent(Item siteRoot)
     {
         if (siteRoot != null)
         {
             var robotSettings = Context.Database.Items.GetItem(string.Format("{0}/Robot Settings", siteRoot.Paths.FullPath));

             if (robotSettings != null)
             {
                 var robotField = robotSettings.Fields["Robots File Content"].Value;

                 return robotField.Replace("[site domain]", HttpContext.Current.Request.Url.Host);
             }
         }
         return string.Empty;
     }
 }

You can find the sample code on my GitHub Repository

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s