I want to get the status code of 2000 urls. I want to store the status code of the url as a dictionary key and the value being the url itself. I also want to do this as quickly as possible. I've seen stuff about async and ThreadPoolExecutor but I don't know how to use any of them yet. How could I solve this problem efficiently?
Here's what I have tried:
import requests def check_urls(list_of_urls): result = {"200": [], "404": [], "anything_else": []} for url in list_of_urls: try: response = requests.get(url) if response.status_code == 200: result["200"].append(url) elif response.status_code == 404: result["404"].append(url) else: result["anything_else"].append((url, f"HTTP Error {response.status_code}")) except requests.exceptions.RequestException as e: result["anything_else"] = ((url, e)) return result
Is there any way to improve this code by making it faster to process 2000 URLs? I have tried requests.head
but it's not accurate.