Downsides of embedding raster images in SVGs?

I have a technical site with tutorial content, much of it being annotated raster images (think rectangles, arrows, and text overlayed on a screenshot). Historically I would draw my annotations in a raster editor, then upload it to my Wordpress site, but after needing to edit my annotations from time to time, I've beginning to think there has to be a better way.

A while back I switch to vector based equations (via LaTeX) as I was having scaling issues with raster version on mobile browser, and so far have had great success. Renders nice and quick and looks great. Which got me thinking... why not do something similar with my larger annotated diagrams?

Here's my thoughts:

Pros: Painless to modify annotations (and no need to retain source raster graphic), Wordpress doesn't need to make scaled duplicates of the graphic (less files = quicker backups), and the ability to embed author information. Maybe even including animations at some point.

Cons: Incompatibility with older browsers

Unknowns: Rendering penalty versus loading raster graphic? Possible security issues? Issues with SEO?

Any other major downsides for embedding raster graphics into an SVG for use on a modern site? Can you help me answer some of the unknowns listed above?

Thanks