Do I need to have my media in the media library?

Strictly speaking, no. Although we recommend that you add your media to the media library. If you know for certain that you'll never use an asset/file other than in the context that you are uploading it to, it's safe to not add the asset/file to the media library. When in the media library, you can then instance the same asset (image, document, video or audio) in more than one page or record without the need to re-upload it.