One of the questions I was asked is where is the best place to store video data. This concerns raw video data that comes directly from the camera. And not yet processed.
Now I run into a few things with this question:
Video files are large (but really big), so uploading takes quite a while when you do this to Google Drive, for example.
To make a comparison, I uploaded a Debian iso and it already took about 10 minutes (I think it was 3 to 4 GB in size). Video files are many times bigger than this.
Now I can spin on a Nas to store this data, back it up to Amazon Glacier, for example, but is there a better way for this, which is also fairly cheap?
Must they be stored on an on-line server?
Presumably they do for the purpose of collaborating with others or working at different locations.
Otherwise it will always be better to store large files locally.
With extremely large files it is always going to be slow to up/download on the web, there is no getting away from that.
Multi-part upload will significantly reduce upload time for large files.
Multipart Upload allows you to upload a single object as a set of parts . After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size.
The upload process to s3 can further be optimized by uploading files directly in the browser. This eliminates any potential server limitations or load since the server is removed completely from the equation.
You can create a bucket in s3 and upload one of the files via the aws s3 gui to test. It seems like you might be familiar with this considering you are talking about glacier but that is for backup. If you need to access the files frequently than glacier isn’t a good option.
Unless you are managing your own physical server s3 is extremely low cost. You only pay for storage and aws provides a generous free tier.
Another option if you don’t need public internet access is adobe creative cloud. Adobe creative cloud would be a great option if you just need to store things in the cloud and already have a adobe subscription.
A data point: I recently bought a Synology NAS and did some testing with it installed on my local 1Gb network. From my PC to the NAS RAID I was able to get 83MB/s for 125GB of video files. From the NAS RAID to the local (on the Synology, so no network) USB3 backup disk, I was able to back up the same 125GB at 173MB/s.
At the end harddrives are the fastest and cheapest solution. If you calculate around 50 Euro for 2tb and use a mirror raid 0 (2 mirrored disks) to have no loss in case of a disk defect, it’s 100 Euro for 2tb, that’s 10 Euro for 200gb what is one video.
Is it possible to compress the video before uploading? I believe there are quite a few utilities available which may help also try zipping locally and unzipping online may be another solution.
Edit;
Zip I believe is usually installed by default but it is not the best compression tool, try the following alternatives:
What is best for someone might be lousy for another.
For some people the best solution could be to send the files physically. A Blu-ray Disc can hold up to 50 GB. If there is a service that will return the media, there are SSDs of 4 TB and greater. Thumb drives cost less. There are many cheap SSDs available, such as 16 TB for under $40 but their reliability is assumed to be low.
There might be a nearby business that can do the uploads overnight.
For Hollywood most of Los Angeles has fiber-optic services capable of 2 Gbps.
Yes HD movies in a Tivo are often greater than 10 GB.
Lets say you use Blueray disk (just got a Dutch site).
They have an price of 46,50 for 50 disks
46,50 / (So 50 disks * 25GB = 1250GB) = Has an price of 0.0372 Euro once
For the HDD, just took the WD Red 2TB 3.5 Inch 2 TB
This one is about 98,32 Euro.
98,32 / 2000 GB = 0,04916 Euro once
For a usb 3.0 flash drive (SanDisk® Ultra Shift™ USB 3.0) you pay 35 Euro (rounded) for 128GB
35 / 128 = 0,2734375 Euro once
There is fibre, and there is fibre.
The fibre optic broadband you have is just that, your Wi-Fi or Ethernet by which you connect to the internet, using the regular internet protocols, through most likely copper wires until you get to the fibre part of the pipeline, it’s only as fast as the slowest part of the pipeline.
What I believe @SamuelCalifornia is referring to is a pure fibre data connection which goes from a fibre port in your workstation/server, through a pure fibre channel into the receiving port in another workstation/server. There is no internet involved and it’s super fast compared to any broadband connection.
That’s the system used by some of these high capacity storage devices that are used in (video/film) production pipelines.
When I was working in that business we had a thing called a “Facilis Terrablock” which was a 4U, rack mounted box with 24 HDDs in the front and 4 I/O fibre ports in the back. These connected to fibre cards in the server and workstations.
This was used locally at just one small production facility. But theoretically, those fibre lines could be any length and spread all across Hollywood. Not that I was ever in Hollywood.
The closest thing I was to that kind of set-up was at Granada Studios, where at the time they shot Coronation Street (British soap). The cables you saw in the street set going from telegraph poles to the buildings, you would assume were just dead wires as part of the set dressing. They were actually functioning fibre optic lines taking footage from the cameras on set to the production facilities. They used the Avid Isis system for their mass storage.
I suppose it depends on your use case and how fast you need it to be. Those kind of fibre enabled storage devices are very fast with a high capacity, but come at a price.
They are used to access the footage in real-time during actual production. The workstation just sees it as if it were a local drive within it, not an external, remote device.
But it may be you just want it for storage or archiving, as opposed to live use, in which case fibre speed may be overkill and there may be cheaper options available for local mass-storage.
