Sometimes, as in the case of the nopetopus and the WTF cake, a certain amount of time in Inkscape is going to be necessary, but I’d say that I whisper around ⅔ of my emoji via downloading images just two websites, and what’s more you can too! (It’s just that easy!)
openclipart contains whatever people want it to contain. Aside from providing sources for many of the puzzlemoji, I’ve recently found good witch and wizard hats there. Most of the images can’t be shrunk to 128px and still be made out, but there’s enough that can to make the search worth it.