Evaluation Arena Question

Water Supply Forecast Rodeo
9

Hi @mmiron,

We’ll try to debug this issue further, but as a workaround that you can use in your code and have some control over, you can try something like import time; time.sleep(5) to just delay your code for a short amount of time, like 5 seconds. There seems to be some transient effect when mounting the data drive that makes Python not able to properly iterate over the files (even though the files are actually there) until after some delay.

For example, the following preprocess function works for me:

from collections.abc import Hashable
import glob
import os
from pathlib import Path
import time
from typing import Any

from loguru import logger


def preprocess(src_dir: Path, data_dir: Path, preprocessed_dir: Path) -> dict[Hashable, Any]:
    time.sleep(5)
    glob_results = glob.glob(os.path.join(data_dir, "*"))
    logger.info("glob.glob: {}", glob_results)
    list_dir_results = os.listdir(data_dir)
    logger.info("os.listdir: {}", list_dir_results)
    raise Exception
    return {}

Resulting logs:

2023-12-05 22:04:52.740 | INFO     | __main__:main:51 - Running function 'preprocess'
2023-12-05 22:04:57.841 | INFO     | src.solution:preprocess:14 - glob.glob: ['/code_execution/data/cdec_snow_stations.csv', '/code_execution/data/cpc_climate_divisions.gpkg', '/code_execution/data/geospatial.gpkg', '/code_execution/data/metadata.csv', '/code_execution/data/smoke_submission_format.csv', '/code_execution/data/submission_format.csv', '/code_execution/data/test_monthly_naturalized_flow.csv', '/code_execution/data/cdec', '/code_execution/data/cpc_outlooks', '/code_execution/data/grace_indicators', '/code_execution/data/modis_vegetation', '/code_execution/data/pdsi', '/code_execution/data/snodas', '/code_execution/data/snotel', '/code_execution/data/teleconnections', '/code_execution/data/usgs_streamflow']
2023-12-05 22:04:57.895 | INFO     | src.solution:preprocess:16 - os.listdir: ['cdec_snow_stations.csv', 'cpc_climate_divisions.gpkg', 'geospatial.gpkg', 'metadata.csv', 'smoke_submission_format.csv', 'submission_format.csv', 'test_monthly_naturalized_flow.csv', 'cdec', 'cpc_outlooks', 'grace_indicators', 'modis_vegetation', 'pdsi', 'snodas', 'snotel', 'teleconnections', 'usgs_streamflow']