pim/static
Archived
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
This repository has been archived on 2024-04-26. You can view files and clone it, but cannot push or open issues or pull requests.
static/robots.txt

9 lines
136 B
Text
Raw Normal View History

block semrush bot through robots.txt
2023-08-23 17:03:22 +00:00
User-agent: SemrushBot
Disallow: /
block AhrefsBot in robots.txt
2023-08-24 14:07:44 +00:00
User-agent: AhrefsBot
Disallow: /
update RP2 grading status block dotbot in robots.txt
2023-08-25 08:37:49 +00:00
User-agent: DotBot
Disallow: /
disallow facebookbot using robots.txt
2023-08-27 12:04:37 +00:00
User-Agent: FacebookBot
Disallow: /
Reference in a new issue Copy permalink