How to configure meta-tags and media sharing in the admin
app
, we have the SEOPLP
and SEOPDP
blocks)Disable indexing
).
robots.txt
represents a standard file used by bots on the internet to
define how they should process your site.
To do this, it is possible to edit the file in the admin itself:
User-agent
s, at least one allow
or disallow
rule, and other optional rules
(depending on the bot).
The values for these rules can include *
to represent all or any set of bots
or paths.
With the growth of generative AI bots, we do not recommend granting access to any bot and any path by default. An ideal configuration:
- for search engines (like google/bing/etc): allow access to all paths, with some exceptions
- for all bots, block everything except very specific or AI-oriented paths, such as institutional pages, or even representative product shelves or catalogs for AI.