Manage sitemaps for versioned documentation¶
Use this guide to publish a sitemap index for versioned documentation.
The sphinx-sitemap extension doesn’t support multiple versions by default. Configuring your versioned documentation to use an appropriate version may be sufficient, as search engines and other web systems crawl websites for the purposes of indexing.
If you want sitemaps for all your documentation’s versions, you need to deploy your own robots.txt file and sitemap index. Supporting multiple versions is recommended for documentation with LTS releases because it makes past versions more prominent to search engines.
Note
If you are implementing a sitemap on an RTD instance that is not a subproject, and it uses {link} for the sitemap_url_scheme, RTD will replace your sitemap with their own.
This is a known bug. The only current workaround is to use a different sitemap name
and a custom robots.txt pointing to it.
Create a sitemap index¶
Before you begin, ensure each version of your documentation has a sitemap generated by the sphinx-sitemap extension with the appropriate version. For the available sitemap settings and their default values, see Sitemap configuration.
Next, create a sitemapindex.xml file in the same directory as the conf.py configuration file, and point to the sitemap files of each of your documentation sets.
Take the Sphinx Stack for an example. Assume that it has three versions, 1.0, 2.0, and 3.0, and uses the URL schema of <version>/<filename>.
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://canonical-sphinx-stack.readthedocs-hosted.com/stable/sitemap.xml</loc>
</sitemap>
<sitemap>
<loc>https://canonical-sphinx-stack.readthedocs-hosted.com/3.0/sitemap.xml</loc>
</sitemap>
<sitemap>
<loc>https://canonical-sphinx-stack.readthedocs-hosted.com/2.0/sitemap.xml</loc>
</sitemap>
<sitemap>
<loc>https://canonical-sphinx-stack.readthedocs-hosted.com/1.0/sitemap.xml</loc>
</sitemap>
</sitemapindex>
Select the sitemap that crawlers use¶
A robots.txt file specifies the sitemap that crawlers use to index a website. Create the file in the same directory as the configuration file.
If necessary, block any paths you don’t want crawled. Google describes how to do this in How to write and submit a robots.txt file.
At the end of robots.txt, point to the future path of sitemapindex.xml:
Sitemap: https://canonical-sphinx-stack.readthedocs-hosted.com/stable/sitemapindex.xml
Publish the sitemap index and robots file¶
Add both files to the configuration file:
html_extra_path = [
"sitemapindex.xml",
"robots.txt",
]
This provides a sitemapindex.xml file, which points to the sphinx-sitemap generated sitemap for each version.
Automate sitemap-index generation¶
You may want to automate the generation of the sitemapindex.xml file. To see how this is done for the Ubuntu documentation library project, which generates a sitemap
containing subproject sitemaps, see the script here.