> For the complete documentation index, see [llms.txt](https://www.marialc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.marialc.com/pentesterlab-labs/recon-badge/recon_00-robots.txt.md). # recon\_00 (/robots.txt) View the exercise here: [PentesterLab: Recon 00](https://pentesterlab.com/exercises/recon-00) ### **OBJECTIVE** For this challenge, your goal is to retrieve the `robots.txt` from the main website for `hackycorp.com`. ### THE `ROBOTS.TXT` FILE The `robots.txt` file is used to tell web spiders how to crawl a website. To avoid having confidential information indexed and searchable, webmasters often use this file to tell spiders to avoid specific pages. This is done using the keyword `Disallow`. You can find more about the `robots.txt` file by reading [Robots exclusion standard](https://en.wikipedia.org/wiki/Robots_exclusion_standard) ### **SOLUTION** Checking the `/robots.txt` file reveals paths that are restricted from search engine indexing. These paths may contain sensitive or hidden information that attackers can exploit, making it essential to review during reconnaissance. {% hint style="success" %} We'll find the flag for this challenge in **`hackycorp.com/robots.txt`**. {% endhint %}