Use the color remover tool to make a list of colors you wish to remove, simply by tapping on the image.
Play around with the sensitivity sliders and RGB sliders to get the exact colors of your background.
If the colors you pick aren’t enough to erase every thing you wish, you can use the brush tool to get the job done. Use your finger or Apple Pencil to manually erase. You can also zoom in on the image to increase accuracy.