How you remove it depends on what you intend to put there in its place.
Since those using it for tracking where they are on the page expect it to look like that you will need it to look at least somewhat similar so they can tell where they are.
If you don't intend replacing it with anything then the best alternative is to remove the button completely - at least that way you will not end up discriminating against disabled people who need that focus marker to tell where they are on the page.
Try navigating the web for a while without touching your mouse and you will soon realise why that marker is there.
For those using Opera who have trouble seeing where the focus is there is an extension available that converts it to a blue outline that matches the way it is marked in Safari.