# IMPORTANT: How to speed up Inference & Queue time

**URL:** <https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986>\
**Category:** On Cloud N\
**Created:** [December 31, 2021, 8:09am UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986 "2021-12-31T08:09:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Max\_Schaefer](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@Max\_Schaefer](https://community.drivendata.org/u/Max_Schaefer)\
**Post date:** [December 31, 2021, 8:09am UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986/1 "2021-12-31T08:09:46Z")

</div>

Hey fellow participants.

I think i found the reason for long queue waiting time. I noticed that the code provided in the [benchmark notebook](https://www.drivendata.co/blog/cloud-cover-benchmark/) doesn´t leverage GPU, but instead lets your model do inference on CPU. Since probably most participants start with that notebook, i´m guessing that most didn´t adjust the code in main.py to let their model do inference on GPU. I adjusted the code and tested the difference:

**CPU: 1h 44 minutes**  
**GPU: 0h 23 minutes**

Here is a little hard coded solution to speed up Inference:

**main.py**

**line 79:**  
change to:  
x = batch[“chip”].to(“cuda”)

instead of:  
x = batch[“chip”]

**line 81:**  
change to:  
preds = (preds \> 0.5).detach().to(“cpu”).numpy().astype(“uint8”)

instead of:  
preds = (preds \> 0.5).detach().numpy().astype(“uint8”)

**line 117:**  
change to:  
model = CloudModel(bands=bands, hparams={“weights”: None, “gpu”: True})

instead of:  
model = CloudModel(bands=bands, hparams={“weights”: None})

**Please make sure to adjust your code. It would benefit everybody. It would help if the blog post with the benchmark notebook gets adjusted as well to avoid running into the same issues of long queues again.**

---

<div class="post-metadata">

**Author:** ![ajijohn](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@ajijohn](https://community.drivendata.org/u/ajijohn)\
**Post date:** [December 31, 2021, 1:15pm UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986/2 "2021-12-31T13:15:18Z")

</div>

This is great ! Thanks for taking the time to write this up.

One follow-up question, on the benchmark code side, to speed up the training, are there other places that needs to be modified to use GPU?

Thank you

---

<div class="post-metadata">

**Author:** ![Max\_Schaefer](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@Max\_Schaefer](https://community.drivendata.org/u/Max_Schaefer)\
**Post date:** [January 1, 2022, 9:37am UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986/3 "2022-01-01T09:37:37Z")

</div>

Hey @ajijohn.

I just saw that you are struggling with waiting hours per epoch. Your model very likely trains on CPU.

1. You should add the parameter “GPUs” = True, when setting up your model:

```auto
cloud_model = CloudModel(
    bands=BANDS,
    x_train=train_x,
    y_train=train_y,
    x_val=val_x,
    y_val=val_y,
    hparams={"gpu": True}
)

```

1. Set **gpus = 1** , when setting up the pytorch\_lightning.Trainer object:

```auto
trainer = pl.Trainer(
    gpus=1,
    fast_dev_run=False,
    callbacks=[checkpoint_callback, early_stopping_callback],
)

```

One epoch in google colab takes about 5-7 minutes.

---

<div class="post-metadata">

**Author:** ![ajijohn](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@ajijohn](https://community.drivendata.org/u/ajijohn)\
**Post date:** [January 4, 2022, 6:42pm UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986/4 "2022-01-04T18:42:10Z")

</div>

Thx @Max_Schaefer , I’ve the settings you indicated, but only difference is that the images are on my google drive. Will try to debug it.

---

<div class="post-metadata">

**Author:** ![harish5p](https://avatars.discourse-cdn.com/v4/letter/h/779978/32.png) [@harish5p](https://community.drivendata.org/u/harish5p)\
**Post date:** [January 4, 2022, 8:49pm UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986/5 "2022-01-04T20:49:54Z")

</div>

@Max_Schaefer

1. I got `CUDA out of memory` error, would changing the batch\_size help fix this error?
2. I trained on batch\_size of 8, would testing on a different batch\_size give an error?

Full error:  
`RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 11.17 GiB total capacity; 10.54 GiB already allocated; 99.88 MiB free; 10.67 GiB reserved in total by PyTorch)`

---

<div class="post-metadata">

**Author:** ![ajijohn](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@ajijohn](https://community.drivendata.org/u/ajijohn)\
**Post date:** [January 4, 2022, 10:59pm UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986/6 "2022-01-04T22:59:37Z")

</div>

@harish5p , I’m having the issue, and per suggestion by a community member, I modified the batch size to resolve OOM error. I think it depends on your backbone too, I modified my batch\_size to 10 from 23, and monitored the usage, and its hovering around 7GB (well below 11GB max we get). Also, I’m only using 4 bands now with backbone resnet34, so you might want to experiment with with your model framework.

I plan to resubmit soon, but looks like it might be fine to go through inference.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![Max\_Schaefer](https://avatars.discourse-cdn.com/v4/letter/m/f17d59/32.png) [@Max\_Schaefer](https://community.drivendata.org/u/Max_Schaefer)\
**Post date:** [January 5, 2022, 2:19am UTC](https://community.drivendata.org/t/important-how-to-speed-up-inference-queue-time/6986/7 "2022-01-05T02:19:04Z")

</div>

Make sure to restart your runtime before trying another batch size. @ajijohn is right. Changing your batch size will solve the problem. Deeper backbones often require smaller batch sizes. Using **batch\_size = 16 with a resnet34 backbone on google colab environment** works for me.  
Pytorch lightning provides a function to find the right batch\_size for your current environment. [Tutorial for using auto\_scale\_batch\_size](https://www.youtube.com/watch?v=HkKyLE1IBFM&ab_channel=PyTorchLightning). Solution that worked for me:

1. add **auto\_scale\_batch\_size=True** flag to trainer:

```auto
trainer = pl.Trainer(
    gpus=1,
    fast_dev_run=False,
    callbacks=[checkpoint_callback, early_stopping_callback],
    auto_scale_batch_size=True
)

```

1. Use **trainer.tune** instead of trainer.fit to find the right size:

`trainer.tune(cloud_model)`

output should look similar to this:

> LOCAL\_RANK: 0 - CUDA\_VISIBLE\_DEVICES: [0]  
> Global seed set to 42 Batch size 2 succeeded, trying batch size 4  
> LOCAL\_RANK: 0 - CUDA\_VISIBLE\_DEVICES: [0]  
> Global seed set to 42 Batch size 4 succeeded, trying batch size 8 LOCAL\_RANK: 0 - CUDA\_VISIBLE\_DEVICES: [0]  
> …
> 
> Finished batch size finder, will continue with full run using batch size 32  
> Restoring states from the checkpoint path at …  
> {‘scale\_batch\_size’: 32}

Although batch\_size 32 succeeded in my case, i still run into issues using it. Batch size 16 worked for me. Make sure to remove these flags again and restart runtime before training again.  
There is even ‘auto\_lr\_find’ flag that finds the right learning rate for you. I recommend you to checkout [Pytorch Lightning Tutorials](https://www.youtube.com/c/PyTorchLightning/videos) to find more tricks to optimize your pipeline.