well, i'm assuming that you're making a selection from the original and then filling it with a color? correct? make sure the selection is set to anti-alias. if that doesn't work, do a very slight blur, 2 pixel maybe, before you do the effects.
but then, exactly when is the aliased edge showing up?