robots.txt 생성기
검색 엔진이 페이지를 크롤링하는 방식을 제어하기 위해 웹사이트용 robots.txt 파일을 만듭니다.
Presets:
Rules
User-agent: *
Disallow: /admin/
Disallow: /private/robots.txt - 기술 세부 정보
The robots.txt file tells web crawlers which URLs they can access on your site. It's placed at the root of your domain (e.g., example.com/robots.txt). While crawlers should respect these rules, they are not enforceable—use authentication for truly private content.
명령줄 대안
# Test your robots.txt\ncurl https://example.com/robots.txt\n\n# Check if a URL is blocked\n# Use Google Search Console's robots.txt Tester