Live logs in local testing environment

MagNet: Model the Geomagnetic Field
1

Is there a way to obtain live logs in the local testing environment? Currently, the logs are dumped after the execution is completed - it would be nice to have these logs generated in livetime instead. I understand that in “/runtime/entrypoint.sh”, one may add --live stream to the line “conda run -n py python main.py” (even though this won’t be a possible step for the actual submission), but I think the docker image we pulled won’t reflect this change. Is it possible then that the repo’s dockerfile can be built with the --live stream attribute appended in entrypoint.sh?

Any suggestions would be greatly appreciated, thanks!

1 Like
2

This sounds like a Docker timing issue about when output gets output to the console running the container, but the Python process within the container pipes STDOUT and STDERR to log.txt in the submission folder.

You should be able to tail -f submission/log.txt and watch as files are written out.

3

Hi @suyashk12,

you can use logging instead of loguru by using following in main.py
that will create file ‘my_runtime_log.log’ under submission folder.

import logging

logging.basicConfig(filename=RUNTIME_DIRECTORY/'my_runtime_log.log', 
                    level=logging.INFO, 
                    format='%(asctime)s | %(name)s | %(levelname)s | %(message)s')

logging.info('This message will be logged')
4

This looks like it’ll work, just wondering if this is different than looking at log.txt in the submission folder which should already be working in the same way?

6

I think loguru is logging into stdout and later at end of the execution that is copied to log.txt. Where in above approach logging is done directly to the file. There should be way to do similar even with loguru. May be here is loguru equivelant loguru._logger.Logger.add for this.

7

@MineshJ1291 @isms Is this resolved? I do not see loguru doing a realtime logging to the file.

8

Yes, by simply adding one line to main.py:

from loguru import logger
logger.add(RUNTIME_DIRECTORY/'my_runtime_log.txt')
logger.info("This message will be logged")
9

OK @MineshJ1291 … That’s clear.
Do you have any suggestion to driven data team for live logging?

10

What worked for me was adding the attribute --no-capture-output to the conda run statement for main.py in entrypoint.sh in the runtime folder, and then building a local docker image with make test-container. Sure, we won’t be able to do this over the actual submission website, but for local testing purposes, this works well. The issue is that entrypoint.sh is invoking the python script main.py (and everything the latter calls) in the background, and it is streaming to stdout and stderr instead of the terminal. We don’t want that to happen, and therefore, we want to add the attribute --no-capture-output to the conda run statement for main.py in entrypoint.sh.

1 Like
11

Added just now. As soon as the new image is built and pushed this change will be reflected on new runs.

Thanks @suyashk12!

1 Like