dvd.com complete inventory

April 22, 2023 9:34 AM

Before dvd.com closes at the end of September, I would like to gather a complete list of their available dvds. Does anyone know if such a thing already exists? If not, do you have any suggestions about how to go about compiling it? Thanks!
posted by metadave to Computers & Internet (4 answers total)

I see they have a browse all genres link, which seems to list all available titles. You can probably scrape the links (which has the titles embedded in the url) for a list?
posted by pseudoinversedly at 10:23 AM on April 22, 2023

If what they are showing as autocomplete results is an unthrottled set of what is there, I recommend this strategy to get back end text data

1. Make a series of direct requests with cURL or something similar to this list:

https://portal.dvd.netflix.com/search/searchall?term=a&dataType=Movies
https://portal.dvd.netflix.com/search/searchall?term=b&dataType=Movies
https://portal.dvd.netflix.com/search/searchall?term=c&dataType=Movies
...
https://portal.dvd.netflix.com/search/searchall?term=z&dataType=Movies

2. Collate all the results

3. Strip the strackId key from the results, because it is dynamic and would interfere with deduplication

4. Deduplicate

5. Reformat from JSON to a friendly tabular format

6. Profit!
posted by Glomar response at 3:09 PM on April 22, 2023

Response by poster: Thanks, Glomar response! That's very useful. How did you figure that out?
posted by metadave at 6:53 PM on April 22, 2023

Search bar autocomplete is usually set up this way, fetching an array of results.

I'm not a coder, just a pro web searcher. It's something I use a lot to get the data I need.
I used the developer tools pane in my browser and looked for behavior in the Fetch/XHR tab.

By the way, don't forget to make additional requests with digits instead of letters - there are a handful of movies with names like "3" and "1923"
posted by Glomar response at 6:53 AM on April 23, 2023