How to find a list of pages I can't access on a certain domain
April 27, 2016 4:27 PM   Subscribe

An odd question I can't quite figure out how to google: I'm trying to detective my way into finding out the names of unannounced products. I've figured out that if I plug the code names of these products in as the final part of a url, I get an Access Denied message. If I type in random letters or words I do not get this message. How do I find out what other pages exist that I can't access?

Someone is lording this information over Twitter, saying he's found out the info but doesn't want to share because this loophole will be closed if discovered. I don't want to bust it open to the world either, but I figure I've gotten almost all the way there but I can't find the right search terms to get me the rest of the way.
posted by yellowbinder to Computers & Internet (4 answers total) 1 user marked this as a favorite
If you don't have a list of likely code names, and they don't follow some predictable pattern, the only way to find out what URLs you can't access is to test all possible words and see which return error code 401 instead of 404. This could be done with something like
cat /usr/share/dict/words | while read w; do curl -s -I$w | grep 401 && echo $w; done
to search all dictionary words, but the site's owners will probably notice and be mad.
posted by nicwolff at 5:11 PM on April 27, 2016 [1 favorite]

Unannounced products don't get put on web pages with easily discoverable or guessable URLs, unless somebody doesn't know what they are doing. If they are on web pages at all, those pages will have password protection, maybe multiple layers of such. If not, it's probably not a product worth hacking your way into. And the very smart people here at Metafilter how might know how to hack their way in, are not going to tell you.
posted by beagle at 5:20 PM on April 27, 2016

Yeah I get that I'm probably out of my depth, not able or looking to hack and further Googling has me pretty much sold it's beyond me. This individual has in the past said "The reveal of x confirms rumor y" and is again claiming to have info, giving hints that lead me to believe he has such a list. From his public persona I wouldn't expect him to be a hacker, just someone who found some unsecured info.

Anyone with any ideas feel free to share, but I can barely hack a tuna sandwich let alone anything to do with computers and wouldn't want to if I could.
posted by yellowbinder at 5:31 PM on April 27, 2016

There's also the very distinct possibility that this twitter user is a damn dirty liar. Unless they're willing to allow for independent verification of their claims, then I'd suggest that you have pretty good odds against them being truthful.

Also keep in mind that there are various reasons for a webserver to reply with a 401 Unauthorized or 403 Forbidden response, and it does not necessarily follow that it's protecting some secret cache of documents about unreleased products.

Finally, I'd suggest that even if you are doing this against a publicly facing webserver with public URLs, the possibility exists that a company could detect your attempts and try to sue you for trying to hack it. If they have deep enough pockets, it doesn't really matter if they have a case when it comes time to make your life a living hell, and frankly if you're not technical enough to be asking this question, you're likely not technical enough to take precautions against this eventuality.
posted by Aleyn at 1:49 PM on April 28, 2016

« Older Here and there and back again   |   Can I create an online petition and moderate who... Newer »
This thread is closed to new comments.