Alter robots.txt disallow indexing

This module changes the /robots.txt file that search engines read when they visit your website. Normally Odoo's website allows crawlers to index your pages; after installing this module the file tells them not to.

• It adds a configuration setting called website.crawler.disable, which is switched on by default when the module is installed.
• When this setting is on, the /robots.txt file contains "Disallow: /", which asks search engines to stay away from the whole site.
• When the setting is off, the file no longer blocks crawling, so search engines can index the site again.
• The file still points crawlers to your sitemap (sitemap.xml) so they can find your pages when indexing is allowed.

The module works by replacing the template that Odoo uses to generate /robots.txt and by taking over the web route that serves that file. It depends on the Website app and does not add any new menus or screens of its own; the behaviour is controlled through the existing configuration parameter. No other Odoo areas, models or business documents are affected.