thanks a lot for hosting this challenging and interesting competition!
I’m trying to connect to the NOAA API to pull climate data, but unfortunately I only seem to be able to get data from 2021/03/21, when inserting earlier dates I get an error message:
<?xml version="1.0" encoding="utf-8"?>BlobNotFoundThe specified blob does not exist.
We’ve added a second approved data access location for HRRR data. You should be able to pull climate data for earlier dates using the new AWS endpoint instead of the planetary computer endpoint. There are more details on the problem description page.
Good luck, please don’t hesitate to reach out if you run into any other trouble!
To get started, here’s a snippet of example code to pull data from the AWS access point instead of the Planetary Computer. It uses the same general approach that is outlined in the Planetary Computer example notebook (also linked from the problem description page).
sector = "conus"
cycle = 12 # noon
forecast_hour = 1 # offset from cycle time
product = "wrfprsf" # 3D Pressure Levels
sample_date = "20190826" # August 8 2019
file_path = f"hrrr.t{cycle:02}z.{product}{forecast_hour:02}.grib2"
url = f"https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.{sample_date}/{sector}/{file_path}"
print(f"Full URL: {url}")
r = requests.get(f"{url}.idx")
idx = r.text.splitlines()
> Full URL: https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20190826/conus/hrrr.t12z.wrfprsf01.grib2
Hi, I can’t seem to download any climate data prior to September 2014. From looking online it seems that the HRRR data was not available prior to Sep’ 14, however there are ~1000 test points that are from 2013-2014. Am I missing something? were you able to get the climate data for 2013/14?
It is possible that HRRR data is not available for every sample in the dataset. You are correct that HRRR first went into operation in 2014, although I’m not sure the exact date. I’m also finding that most points before September 2014 do not have HRRR data available.
HRRR is not a required data source, so it is up to you whether and how you incorporate it. This mirrors the real world nature of the problem, where they may be missing values that have to be handled. There are a number of strategies for handling missing values if you choose to incorporate HRRR data.
Good luck, don’t hesitate to reach out with any more questions!
