What's the rule of thumb when deciding whether to use a single Apostrophe' or a Quote"?

While both will work when quoting HTML attributes, the usual convention is to use double quotes, and single quotes inside them if needed.

For example:

<img alt="The good ship 'Lollipop' in dry dock">

Using single quotes will cause problems if your text includes an apostrophe, because that will be seen as the closing quote.

e.g.

<a href='http://whatever.com' title='Visit Mabel's website'>