For any server where you are both web serving and encoding/transcoding video (which is processor intensive) you’re best to implement an encoding queue system so that bursts of many simultaneous processes don’t hinder the rest of your site. You’d likely want a port that burst to 1gbps rather than a 100mb port if you foresee a lot of simultaneous downloads (or you could offload download activity to a CDN)
You would probably best offloading content to a CDN if you plan to stream lots of videos, equally, if you only plan to stream a few at any one time to paid subscribers for example, you might be better investing in a decent powerhouse of a server yourself with plenty of bandwidth.