Is setting font size by css a good thing for the end user?

I am debating whether or not to use css to specify font size so my precious design does not go astray by how the user sets his/her browser (i.e. text size, font)

But, I am wondering if this would necessarily be a good thing, because some people who are hard of sight may want to see bigger fonts, while people with smaller monitors may want to view the text smaller.

Does anyone have an opinion on this subject?

I feel it is of utmost importance to satisfy the end user's experience.