Building an App to Automate Video Editing

When I started on this project, I was also working as a Multimedia Editor for a newspaper/magazine where I created videos regularly for various social media platforms. Over time, some parts of the process, mostly burning subtitles into the video and re-encoding the videos to fit within the different platform requirements, started taking up a significant portion of my time. To free up my schedule, I decided to try my hand at automating this segment of the process.

I was looking for a solution that not only allowed me to process multiple files at the same time but also did this without me having to open a video editor. At my job, I wasn't working alone, so my coworker and I had different folders where we would upload videos throughout the various stages of the editing process. I decided that this tool would work with these folders by watching them and then acting upon any changes.

Ingredients

• Python
• Javascript
• Dropbox
• Transloadit
• Amazon Simple Queue Service (SQS)
  

I was more comfortable with web development at this time, so the plan was to manage all this in a web app, posing an interesting challenge in that I now needed a way for the web app to know whenever a file is uploaded to a local folder. Cue Dropbox, which has a local folder sync functionality that allows users to have files in a folder on their device sync with their cloud storage automatically. Plus, Dropbox has REST APIs that allowed me to check for changes in users' dropbox folders.

To get this setup, I had to integrate Dropbox's OAuth user authentication in my app to retrieve an access token. With that, I could make API calls on behalf of users, for instance, to fetch a list of files in a specific Dropbox directory or get the contents of a specific file. Most importantly, I could track file changes using a cursor Dropbox provides when you call some of their APIs. This cursor acts as a pointer to a specific record or state of a folder. For instance, if I make a call to the `/files/list_folder` endpoint for a folder in my Dropbox, it would return a response with all the files and subfolders present like this:

{
    "cursor": "ZtkX9_EHj3x7PMkVuFIhwKYXEpwpLwyxp9vMKomUhllil9q7eWiAu",
    "entries": [
        {
            ".tag": "file",
            "client_modified": "2015-05-12T15:50:38Z",
            "content_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
            "file_lock_info": {
                "created": "2015-05-12T15:50:38Z",
                "is_lockholder": true,
                "lockholder_name": "Imaginary User"
            },
            "has_explicit_shared_members": false,
            "id": "id:a4ayc_80_OEAAAAAAAAAXw",
            "is_downloadable": true,
            "name": "Prime_Numbers.txt",
            "path_display": "/Homework/math/Prime_Numbers.txt",
            "path_lower": "/homework/math/prime_numbers.txt",
            "property_groups": [
                {
                    "fields": [
                        {
                            "name": "Security Policy",
                            "value": "Confidential"
                        }
                    ],
                    "template_id": "ptid:1a5n2i6d3OYEAAAAAAAAAYa"
                }
            ],
            "rev": "a1c10ce0dd78",
            "server_modified": "2015-05-12T15:50:38Z",
            "sharing_info": {
                "modified_by": "dbid:AAH4f99T0taONIb-OurWxbNQ6ywGRopQngc",
                "parent_shared_folder_id": "84528192421",
                "read_only": true
            },
            "size": 7212
        },
        {
            ".tag": "folder",
            "id": "id:a4ayc_80_OEAAAAAAAAAXz",
            "name": "math",
            "path_display": "/Homework/math",
            "path_lower": "/homework/math",
            "property_groups": [
                {
                    "fields": [
                        {
                            "name": "Security Policy",
                            "value": "Confidential"
                        }
                    ],
                    "template_id": "ptid:1a5n2i6d3OYEAAAAAAAAAYa"
                }
            ],
            "sharing_info": {
                "no_access": false,
                "parent_shared_folder_id": "84528192421",
                "read_only": false,
                "traverse_only": false
            }
        }
    ],
    "has_more": false
}

For my use case, the important fields were the 'cursor' and the 'has_more' fields, the latter of which tells me whether there are more entries in the folder that were not included in this response. If true, I would need to make a call to the `/list_folder/continue` endpoint and pass the provided cursor value to get the remaining entries. In simpler terms, by passing along the cursor, I am giving the Dropbox API the marker from which it should continue to traverse the folder. Because the cursor also holds information about the state of a folder at a specific time, if I were to add or remove a file, and call the `/files/list_folder_continue` endpoint again with the same cursor, it would return information about the changes that have occurred since then and return a new cursor for the current state.

As I needed the web app to act whenever a new file is uploaded to a folder, I had to figure out a way to listen for this event at any time. I could check at a specific interval (e.g every 5 minutes), essentially polling Dropbox's API often to see if anything had changed, but that felt like an inefficient use of server resources, especially if multiple people ended up using the platform. Fortunately, Dropbox also uses Webhooks, or notifications sent to my server whenever a connected user's account has been updated. As such, my app would only need to check for new files whenever Dropbox tells us that a change has occurred. To make sure the app could handle the potential traffic, I set up a Python worker to queue the webhooks from Dropbox (using Amazon SQS) and process them at a more manageable pace.

Now, I had to decide how I wanted to process the incoming files. I looked into using FFmpeg but felt it was a bit too complicated for what I was trying to achieve, so I switched to moviepy, a Python module for video editing. I got a basic video processor worker running on AWS, however, I wasn't comfortable with the potential bills I would run into if I misconfigured something, so I decided to look into a managed encoding solution. I landed on Transloadit, which provides an API for video encoding and general file processing. The platform was well documented and easy to integrate, so I was up and running pretty quickly.

For the interface,  I had been using automation solutions like Zapier and Integromat at the time, and I wanted to implement a workflow-building UX similar to theirs instead of using a timeline like most video editing tools do. Furthermore, I wanted it to be easy to read the entire process being automated at a glance, so I broke down the various actions into step blocks that users could add to a sequence. Each block would generate a JSON file with instructions for how to process the sequence.

To illustrate this, let's use an example where we resize videos for Instagram stories. As a user, I could create a new task, and then add a Trigger step, which is an action that initiates a workflow. In this case, I'll select the ‘New File Uploaded to Watch Folder’ action for Dropbox, which means this workflow will be triggered whenever I upload a file to a folder in Dropbox. I would then be presented with a screen to configure my Dropbox account, if I hadn't already, and select a folder I want to watch. 

I still need to do something with the video, so let's add a ‘Resize Video’ step to the workflow and configure it a bit more. The aspect ratio for Instagram stories is 9:16, so I could set 1080 x 1920 pixels for the width and height parameters.

The ‘Which Input Should We Process’ field allows me to choose which prior step will provide the video that will be resized. For this use case, we would be referencing the trigger step from Dropbox, but if we had, for example, included a step before this that added subtitles to the source video, we could have used the result of that step instead. 

Finally, I might want to do something with the result of the 'Resize Video' step, so I could add an 'Upload to Dropbox' step to the workflow and select a different folder to store the resized video.

And if I wanted to know when the process is complete, I could add a 'Send an Email Notification' step that would send an email to me or whomever I feel needs to know that the file is ready. Once this is all set up and the workflow has been activated, the process would happen automatically without me having to open the web app again.

After a few months of tinkering, I had finally reached the finish line. The project was complete, but unfortunately, I no longer needed it as I had switched to a different job that didn't require a lot of video production. Instead of scrapping the project immediately, I spent some time trying to see if it might be helpful to others, and to some extent, it was. However, most people who showed interest in the platform wanted a version that ran locally instead of in the cloud, mostly due to data privacy reasons and cost. And while I thought it was great feedback, it would've required rebuilding the app for the most part. As I no longer had the bandwidth to properly dedicate more time to the project, I decided to put it on ice. While I still don't have any immediate plans to revisit it, if this is an idea you're currently working on, I would love to hear about it. 

‍