DIRECT S3 FILE UPLOAD

Direct upload of files to Amazon S3 was designed and successfully applied in the existing service (marketplace) within a shorter period of time than the upload system that was used before: half a month compared to 2 months. Moreover, it also enhanced security for downloaded files.

• Users can upload files up to 500Mb or up to 1 Gb (depending on association with a specific user group)
• New items (Video Files) became available for upload
• S3 server usage decreased the time and resources required for server maintenance compared to the hardware servers that were used before
• Enhanced security of downloaded files

Existing service allowed users to upload files and attach them to products. However, this service puts a solid limit on the size of files that are selected for upload. As a result, it was not possible to upload a file that is larger than 200 Mb. Also, it worked in a way that when a file upload failed (due to the network error), the user was forced to start the entire process all over again. A major problem is that many users have files that are larger than 200Mb. Consequently, there was a lot of requests to upload such types of files. Besides, the NGINX module which was used for file upload was very old and unsupportable.

New upload schema

We decided to upload files directly to S3 and no longer store them on our servers. Also, we've chosen to use a 3rd party JavaScript library to upload files to S3 directly instead of creating our own library. We selected https://github.com/TTLabs/EvaporateJS. This library has already implemented and tested features for all known web browsers:

• Secured direct S3 file upload
• Resume failed file upload

Now all files are uploading to S3 into a temporary bucket for each user's item. Also, we use unique paths and file names. We decided to split files into a smaller parts before upload and run uploads to S3 in parallel threads, each part is signed by our server so it can't be catched and replaced by someone else. Now when a user selects a file to upload to S3, the following occurs:

1. Evaporate JS uploads a file into a temporary bucket directly
   During the upload process, it makes a request to our servers only for signing each part of the file. This is done in order to avoid cases when somebody can catch part of the uploaded file and replace it.

2. After uploading files into the temporary bucket we started to process the files in the following order:

   1. Check for viruses:
      If a virus was detected, we stop processing the file, move it into a quarantine bucket (it clears automatically after 4 days) and notify the user about this event.
   2. Generate preview files
   3. Collect information about the file (size, checksum, type of file, etc.)
      During this process, the user sees an upload progress bar. And does not see a frozen 100% upload as it was with old upload system.
3. User fills in required text fields and clicks on submit button.
4. Process push job of funneling item into a queue:
   When the user wants to create several items, we organize all the processes into a queue. Once the first item is submitted for creation/update, the user is able to select the next one and doesn't have to wait until the create/update process is finished for the first item. Once the item is created/updated, we display a message to inform the user.
5. In the background, worker process takes jobs from the queue and performs the next job in parallel threads:
   1. Save item details into database
   2. Move uploaded files from temporary bucket into regular.
      In a regular S3 bucket, we used MD5 from the file content as the file name. It gives us an opportunity to know when the file was changed and then store the whole file on S3 along with the file upload history. Now we give users a new feature - opportunity to download and edit older versions of downloaded file.
6. In case the background saving process has failed (network issue, Amazon issue, etc.), the queue process always restarts failed tasks a couple of times.