Clarification on federated pandemic model state storage

PETs Prize Challenge
1

Hi @jayqi,

We’ve followed this guideline so far:

  • Your code must use the intended Flower APIs for client–server communication. Your code must only use the provided client and server directories for saving and loading state. Any use of other I/O or global variables to pass information between calls is forbidden.

but it seems a bit confusing that in the example_src, the model is stored (and updated) within the class instance, like so:

class TestClient(fl.client.NumPyClient):
    """Custom Flower NumPyClient class for test."""

    def __init__(
        self,
        cid: str,
        model: SirModel,
        disease_outcome_df: pd.DataFrame,
        preds_format_df: pd.DataFrame,
        preds_path: Path,
    ):
        super().__init__()
        self.cid = cid
        self.model = model
        self.disease_outcome_df = disease_outcome_df
        self.preds_format_df = preds_format_df
        self.preds_path = preds_path

    def evaluate(self, parameters: Parameters, config: dict) -> Tuple[float, int, dict]:
        """Make predictions on the test split. Use model parameters from server."""
        set_model_parameters(self.model, parameters)
        predictions = self.model.predict(self.disease_outcome_df)
        predictions.loc[self.preds_format_df.index].to_csv(self.preds_path)
        logger.info(f"Client test predictions saved to disk for client {self.cid}.")
        # Return empty metrics. We're not actually evaluating anything
        return 0.0, 0, {}

Could you clarify what this limitation means in terms of model storage in this example? If it applies only to communication/disk storage between client and server (or different clients), do we have to store the model to disk even within the same client (or server) class instance?

Additionally, if it’s allowed to reuse the same dynamic class instance variables between calls, we are wondering if we can reuse our Go process within the same class instance as well - this would save a lot of overhead we currently have on creating a new process on each call to the same client.

Thank you

2

Hi @denis.broadinstitute,

Client instances are ephemeral. Every time a particular client (of a given client ID cid) needs to invoke a method (like fit or evaluate), a new instance is created by calling the client factory function that is provided by the solution. There is no in-memory persistence of a client instance beyond a single method call.

So, in the example you’ve cited, it’s actually the case that every individual fit or evaluate method invocation creates a new client instance. Each of those times, the factory function loads the model from disk and passes it into the client instantiation.

The fact that the model object is passed into instantiation and bound to an instance variable is a non-meaningful implementation detail. The factory function could instead pass in the client_dir storage directory for that client, and the fit or evaluate methods could equivalently read the model from disk within the method and use it without binding it to the instance.

This means instance variables can’t be reused between method calls, because those instances are created and deleted for each method call.

3

That makes sense, thank you! I assume this also means we cannot have a per-client long-running Go process that stores some per-client state between calls?

4

Right, please stick with the provided directories for storing state as documented. This keeps things consistent with the setup in Python, and it is more straightforward for ensuring compliance with expected client data access.

1 Like