Quantcast
Channel: Recent Questions - Stack Overflow
Viewing all articles
Browse latest Browse all 12141

Getting the status code of 2000 URLs and storing them as a dictionary output in a fast way

$
0
0

I want to get the status code of 2000 urls. I want to store the status code of the url as a dictionary key and the value being the url itself. I also want to do this as quickly as possible. I've seen stuff about async and ThreadPoolExecutor but I don't know how to use any of them yet. How could I solve this problem efficiently?

Here's what I have tried:

import requests def check_urls(list_of_urls):    result = {"200": [], "404": [], "anything_else": []}    for url in list_of_urls:        try:            response = requests.get(url)            if response.status_code == 200:                result["200"].append(url)            elif response.status_code == 404:                result["404"].append(url)            else:                result["anything_else"].append((url, f"HTTP Error {response.status_code}"))        except requests.exceptions.RequestException as e:            result["anything_else"] = ((url, e))    return result 

Is there any way to improve this code by making it faster to process 2000 URLs? I have tried requests.head but it's not accurate.


Viewing all articles
Browse latest Browse all 12141

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>