Available dates for Climate Data?

Dear organizers & fellow competitors,

thanks a lot for hosting this challenging and interesting competition!
I’m trying to connect to the NOAA API to pull climate data, but unfortunately I only seem to be able to get data from 2021/03/21, when inserting earlier dates I get an error message:

<?xml version="1.0" encoding="utf-8"?>BlobNotFoundThe specified blob does not exist.

RequestId:9e33f454-801e-00d4-4c74-1b7cb4000000
Time:2022-12-29T10:57:33.5971818Z

Is there any way to access earlier data that I’ve been missing so far?

Thank you very much and best regards!
David

1 Like

I have same problem to use climate data. What is happened?

1 Like

Hi @DavidEis and @Nyantaro! Thank you for flagging this.

We’ve added a second approved data access location for HRRR data. You should be able to pull climate data for earlier dates using the new AWS endpoint instead of the planetary computer endpoint. There are more details on the problem description page.

Good luck, please don’t hesitate to reach out if you run into any other trouble!

2 Likes

To get started, here’s a snippet of example code to pull data from the AWS access point instead of the Planetary Computer. It uses the same general approach that is outlined in the Planetary Computer example notebook (also linked from the problem description page).

sector = "conus"
cycle = 12  # noon
forecast_hour = 1  # offset from cycle time
product = "wrfprsf"  # 3D Pressure Levels
sample_date = "20190826"  # August 8 2019

file_path = f"hrrr.t{cycle:02}z.{product}{forecast_hour:02}.grib2"
url = f"https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.{sample_date}/{sector}/{file_path}"
print(f"Full URL: {url}")

r = requests.get(f"{url}.idx")
idx = r.text.splitlines()

> Full URL: https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20190826/conus/hrrr.t12z.wrfprsf01.grib2

1 Like

Oh! Really thanks for your regard!!

Hi @kwetstone,

thank you very much for adding the additional data source - the new AWS endpoint works great :slight_smile:

Best regards
David

1 Like

Hi, I can’t seem to download any climate data prior to September 2014. From looking online it seems that the HRRR data was not available prior to Sep’ 14, however there are ~1000 test points that are from 2013-2014. Am I missing something? were you able to get the climate data for 2013/14?

1 Like

Hi @asaphs. Good question!

It is possible that HRRR data is not available for every sample in the dataset. You are correct that HRRR first went into operation in 2014, although I’m not sure the exact date. I’m also finding that most points before September 2014 do not have HRRR data available.

HRRR is not a required data source, so it is up to you whether and how you incorporate it. This mirrors the real world nature of the problem, where they may be missing values that have to be handled. There are a number of strategies for handling missing values if you choose to incorporate HRRR data.

Good luck, don’t hesitate to reach out with any more questions!