Best Practices for Handling Robots

How to control search engine indexing and crawling for an AEM site using page-level robots metadata, site-wide bulk metadata, the robots.txt file, and the default X-Robots-Tag header on preview and live domains.

Quick reference

Page-level directives via metadata

Metadata is managed at the page level using a metadata block placed at the bottom of a document. Each row is a key/value pair, and the robots property controls how that single page is indexed and crawled.

Add a robots row to the page's Metadata block and set it to the directive you want. Common values:

Property names are lower-cased in the generated HTML, and the value is rendered as a <meta name="robots" content="..."> tag in the page head. Page-level metadata always takes precedence over site-wide bulk metadata, so use it only for one-off exceptions (for example, excluding a single thank-you or landing page).

Remember to preview and publish the document after editing its Metadata block for the change to take effect.

See also Page Metadata.

Site-wide directives via bulk metadata

If you need to apply robots rules to many pages or a whole section at once, use a bulk metadata sheet instead of editing every single page. Create a spreadsheet named metadata in the root folder of your site content (metadata.xlsx in SharePoint).

The sheet has a URL column holding the path pattern, plus one column per metadata property you want to assign (for example robots, template, or theme). The wildcard * can be used as a prefix or suffix for flexible path matching.

Example: keep a whole draft or internal section out of search results:

Important rules:

Preview and publish the metadata sheet for changes to be reflected on the site.

See also Bulk Metadata.

robots.txt

When accessed directly, *.aem.page and *.aem.live domains always serve a robots.txt which disallows crawling. This is intentional to prevent accidental indexing of backend URLs and avoids duplicate-content issues between the platform domains and your real production domain.

Once you set up a production CDN, and traffic is coming in through your production domain, AEM will automatically serve a default or custom robots.txt.

Default robots.txt

The default robots.txt allows crawling and references the sitemap:

User-agent: *
Allow: /

Sitemap: https://<your-domain>/sitemap.xml

Custom robots.txt

To configure your own custom robots.txt, you can either use the API or our handy robots.txt Editor.

It is important to understand that a Disallow in robots.txt is potentially helping with crawler budget issues under certain conditions, but does not prevent a page from being added to the Google index, and conversely will make its removal via noindex impossible. URLs of pages that have a Disallow can still be discovered in a SERP.

An effective way to remove a URL from the index is to Allow crawlers and put a noindex robots on the page using metadata.

See also Go-live Checklist.

X-Robots-Tag header

By default, every response served from the *.aem.page and *.aem.live domains includes the following HTTP response header:

This is intentional to prevent accidental indexing of backend URLs and avoids duplicate-content issues between the platform domains and your real production domain.

Once you set up a production CDN, and traffic is coming in through your production domain, that header will be removed automatically, so indexing is then governed by your robots.txt and the per-page robots metadata described above. Verify behavior by checking the response headers of a live URL after go-live.

See also Network Profile and Go-live Checklist.