The Best AI Tools for Video Editors
What are the best AI powered tools for video and audio post production that video editors, colorists and sound editors should be using in their workflows?
That’s the question I set out to answer in this article, yet with the explosion of “AI” based tools (really its machine learning folks!) available today there are a huge range of apps and services that you could be using, but which ones are actually worth your time and money?
Latest Update: October 2023
The 8 Best AI Tools for Post Production shortlist should help you get straight to the good stuff, while the long-list that comes after that will give you a sense of everything that’s out there, although I left a few dubious options out, that were just ‘not good.’
Shortlist of the 8 Best AI Tools for Post Production
- Runway.ml
- DaVinci Resolve Studio
- Descript
- Blanc
- Colour Lab.ai
- Fylm.ai
- ElevenLabs.io
- Altered Studio
- (Bonus) Swell.ai
I’ll be updating this list regularly as I find new and decent AI-based tools to add to it, but if you see something I’ve missed, do get in touch to let me know.
At the end of this article you’ll find a breakdown of some of the AI powered tools already available on your timeline in DaVinci Resolve Studio and Adobe Premiere Pro.
Table of Contents
- AI and Post Production Frequently Asked Questions
- The Best AI Based Tools for Video Post Production
- AI Video Editing and VFX Tools
- AI Colour Grading Tools
- AI Audio Post Production Tools
- AI Transcription,Text Editing, Subtitles, Translation
- AI Media and Metadata Search
- AI CGI & 3D Animation Tools
- AI Image & Video Creation Tools
- AI Image Enhancement Tools
- AI Voice Over Generation Tools
- AI Tools Already in Your Video Editing Software
- AI Creativity, Ethics and The End of Humanity
- Actually Using AI in Post-Production
FAQ about AI in Post-Production
With AI The Best UI Wins
My guess is that many of the services listed in this article are likely accessing the same AI and ML open source models underneath their browser-based UI. So, really, the race is on for whoever can develop the best user friendly interface, affordable pricing structure and win over enough paying users to hog market share and establish a sustainable business.
That said, some brands like Runway.ml, are actively developing their own proprietary models such as Stable Diffusion and have the VC-backed funds to develop new and existing features, market themselves well and play the long game.
Will AI Take My Job?
Whether you’re fearful of AI taking your job, or just want to get your day-to-day tasks done more efficiently, the AI toothpaste is already on it’s way out of the tube so we may as well make the most of it.
If you don’t, others will.
Personally, I’m not worried about creative video editors being relegated to the annals of history just yet, as the creative decision making, client relationships and cognitive processes involved in making excellent content just aren’t going to be beat by AI, at leat not just yet!
That said, obviously AI editing tools can slap clips and audio together and create an ‘edit’ but it’s not an apples to apples comparison.
What some of these tools excel at is removing the (human!) manual labour from complex tasks such as rotoscoping, transcription or asset tagging. Yet all of this work still needs to be managed, tweaked, corrected, edited and approved by a human.
AI Vs ML – Marketing Terms vs Technical Reality
To Nerd-splain for a minute, A.I. and M.L. are quite different things.
A.I. is mostly what you think of when you think of A.I.
Intelligent machines that can think and act upon their own objectives, usually involving world domination and the subjugation of the human race. They can do multiple things with dexterity.
Machine Learning, however is a branch of A.I. and far more constrained and it’s behind what most of these tools are doing.
Provide your image recognition algorithm with a large enough dataset of images, for example of cats, the computer program would be able to iteratively learn what a cat generally looks like based on patterns of pixels and so (eventually) predictively tag photos of cats with the metadata ‘cat’.
So there’s no need to start building your survival bunker just yet.
The Best AI Based Tools for Video Post Production
As there are so many tools on this list, I’ve curated out my top 8 AI post-production tools at the start to save you from scrolling too far and to give you some more details on these specific services.
As I update this list, these might swap and change as new tools and services become available. I’ll also suggest why exactly a video editor like myself would want to leverage these in their workflow.
Update October 2023 – Adobe Max Sneak – Project FastFill
Adobe Max is Adobe’s chance to own the headlines and demonstrate some of their (potentially) up-coming tools, such as this ‘sneak’ (read early demo) of ProjectFastFill, which allows you to enhance your video clips with generative AI.
This allows you to paint out elements of the frame or add new features into the frame, all in a couple of clicks.
Whether or not you’re an Adobe user, this kind of implementation of generative AI will certainly be coming to Premiere Pro or DaVinci Resolve in the near future, as the race to stay at the cutting edge continues.
If you’re an Avid or Final Cut editor, I’m confident the generative additions feature will appear in something like Runway.ml, who, by the way, are $141 million richer as of June 2023.
Watch more Adobe Max 2023 sneaks here.
TOP 8 AI POST PRODUCTION TOOLS
Runway.ml – The Swiss Army knife of AI powered browser-based tools, Runway.ml has a huge range of useful features such as rotoscoping, object removal, image creation, motion tracking, colour grading, audio noise removal, transcription, text to image, image to image and much more.
Imagine this as your (temp) VFX tool for rapidly fixing shots, creating new assets or backgrounds and solving problems for your clients like you are able to do magic.
If all goes well you could complete the shot inside of Runway’s online editor or download elements like Alpha mattes, new green screen elements, background plates and more.
Sign up for free to try Runway.ml for yourself here and save 10% with code: ELWYNJX10
DaVinci Resolve Studio – Less features than Runway but with the huge benefit of being available right there in your timeline, no uploading or downloading required. Also a one-time price of $295 for a lifetime of updates and usage it’s a pretty economical option.
DaVinci Resolve’s ‘Neural Engine’ features include facial recognition, magic mask, super scale, 3D depth maps, scene cut detection and a whole lot more (see the end of this article for more details!)
Buy DaVinci Resolve Studio on Amazon Global Stores
Descript – Offers an ‘all-in-one’ solution for text-based audio and video editing which includes the ability to ‘overdub’ words and phrases with an AI generated version of the speaker.
Descript is already really popular with podcasters and anyone who doesn’t mind a jump cut in their video. It can automatically remove pauses and filler words (ums, ahs etc.) and allows you to edit your transcribed video like a text document.
It also features a comprehensive browser-based multi-track editing tool, very impressive ‘Studio Sound‘ noise reduction and audio rebuilding, transcription, caption creation and the ability to export to professional applications such as Premiere, Final Cut Pro and others.
The Free account includes 1 hour of transcription, one 720p watermark free export, ums & ahs removed, 1000 word overdub vocabulary and 10 minutes of Studio Sound per month.
Useblanc.com – Blanc’s super power is its ability to convert one transcription into multiple languages, there by exponentially increasing the reach of the content to new audiences.
Currently this means you can create 47 other versions of your video with either translated subtitles, AI-generated voice over, or even AI-generated replacement lipsync.
One of the cool things that Useblanc does is to translate the original text accurately, but also in a way that mitigates vast differences in word-length or sentence structure between languages, in order to deliver the most natural sounding results.
You can get $10 worth of free credit to test Useblanc.com for yourself, with the promo code JONNYELWYN.
ColourLab.ai & Fylm.ai – OK so I’ve sort of cheated with this one, in that these are two completely different (competing) tools for AI powered colour grading.
ColourLab’s tools are machine based with a stand alone app and plugins for Adobe Premiere, Final Cut Pro, DaVinci Resolve and Pomfort Live Grade while fylm.ai works in your web browser and processes everything on their cloud-based servers.
The benefit of AI powered grading is that you can let the algorithm do the heavy lifting of matching references, extracting looks, making different cameras match each other, creating look variations on the fly and a whole lot more.
Both are being consistently developed and improved over the past few years with fylm.ai recently updating to v1.5, which paves the way for iPad and mobile friendly versions.
Likewise, ColourLab’s product range is now at version 2.3, with their offering split into three main tools (Colour Lab AI, Look Designer and GrainLab) although they all work cohesively together, depending on what you’re doing.
Update: FreeLab Plugin – ColourLab.ai have released a new free plugin to ‘replace LUTS’ for Premiere and Resolve. Essentially, it is a ”free version of Colourlab’s film native colour engine, allowing you to balance cameras and apply consistent look across a variety of materials.”
Download it here to try it for yourself.
I’ve previously written more about both services in this post on Tools & Training for Pro Colorists.
ElevenLabs.io – A very affordable, and to date, the best sounding, speech to text AI voice generation platform I’ve heard. It also offers voice cloning based on a 1 minute sample and the results that I’ve heard are incredibly impressive.
The free account gives you 10,000 generated characters to play with, plus 5 cloned voices although you can only use it for non-commercial use and attribution is required.
However, the low-cost $5/month tier gives you 30,000 characters, 10 cloned voices and commercial use, not to mention API access, should you want it.
Unlike Altered Studio that charges by the downloaded minute, rather than the generation attempts, Elevenlabs charges per character generated.
Elevenlabs speech results are ideal for ebook, article and podcasts reads, although they are less ‘character driven’ than Altered Studio’s voices, the realism is much more believable.
In the video above I created an experimental episode of an issue of Cut/daily (my free weekly newsletter) which fit easily into the 10,000 character free quota, even with a few regeneration attempts, specific fixes and the like. All in all, incredibly easy to use and very impressed with the results.
Altered Studio – Actors in a box. That’s what Altered Studio can give you. Drop in your own line reading of some voice over or scripted lines into the system and let Altered Studio turn it into a complete multi-character production.
You can change your accent, add ‘gravitas and charisma’ with their 200+ voices, plus you can clone your own voice samples to be able to add text-to-speech additions to your edit.
Some of the voice samples in their demos sound more convincing than others but the ability to generate an entire table-reading, voice over or dramatic podcast from scratch with professional sounding voices is very compelling.
And while you can select different moods and stylistic approaches, I’m not sure there’s a way to ‘give direction’ to these AI voices in the same way that you can to a real actor – but at least you can create your first version, affordably.
Bonus – Swell for Transcripts, Timestamps and Social Media
I’ve been using Swell.ai to create timestamps for my YouTube videos, which can often get pretty long, but that’s only scratching the surface of this incredibly useful tool.
You can upload video and audio and have it generate:
- Complete transcripts
- Export subtitles (.srt/vtt)
- Shownotes
- Timestamps (Show notes and YouTube chapters)
- Summaries
- Titles
- Social media content (Twitter threads, blog posts, Newsletters, YouTube descriptions etc.)
- Long-form articles
- Chat-Agent
You can use the chat-agent to prompt Swell to do specific things, such as create a Twitter thread with a specific number of Tweets, or a summarise a portion of the interview etc.
Other cool features include watch folders for importing content into Swell from Dropbox, Google Drive, Apple podcast and other sources as well as outputting the results of Swell’s work back to similar locations.
Try Swell for yourself, for free here.
AI Video Editing and VFX Tools
In this section of the post I’ve curated a long list of AI/ML based tools that could come in handy as a video editor, colorist, sound designer or DIT.
For brevity, I’ve tried to keep things to the following formula:
Tool | Main Features | OS | Free Version? | Lowest monthly price
Runway.ml | Rotoscoping, motion tracking, audio clean up, transcription and subtitles, export alpha mattes, in-painting, text to video, video to video, text to image, image to image, text to 3D texture and more! | Browser-based | Free Tier | $15/month – SAVE 10% with discount code: ELWYNJX10
Topaz Labs | Still image and video – upscaling, denoise, sharpening, deinterlacing | Windows & Mac | Watermarked Free Trial | Topaz Photo AI – $199, Topaz Video AI – $299 – One-time fee.
Project Blink | Adobe’s AI-powered browser-based video editor with content search, text-based editing, transcripts, social clips, filler-word removal | Browser-based | Free Beta Trial
Adobe Firefly | Recently announced in beta, this collection of online tools from Adobe does a lot of similar things to Runway.ml but will eventually be embedded in Adobe’s ecosystem of apps and websites. Adobe are also trying to ‘train ethically’ | Browser-based | Free
Vidyo.ai | Instantly create social media subclips with captions, resizing, chapters | Browser-based | 3 hours Free Tier with 720p exports | $19.99/month
EditingTools.io | AI face detection and replacement (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits
AutoPod | AI powered multi-camera editor, shorts selection and silence removal plugins for Adobe Premiere Pro| Mac & PC | $29/month
Momento | AI generated social shorts selection and creation, media management, transcription and Chat-GPT trained article writing | Browser-based | $24.99/month
BigRoom.TV | Auto create 9:16 social media clips from 16:9 video with cuts and motion tracking | Browser-based with export to NLE | Free for now
Chat-GPT | The infamous chat bot that’s going ‘take over everything’ is actually useful for asking it for After Effects Expressions or DCTL code snippets in DaVinci Resolve | Browser-based | Free to use
CleanCut.ai | “Sifts out blurry or shaky clips, selects the optimal shots, and auto-assembles them into engaging sequences (based on text prompt), ready for your final touch-ups.” | NLE-based plugin | Currently in Beta
AI Colour Grading Tools
fylm.ai | AI grade extract, auto correct, colour matching, advanced technical matching, grade creation | Browser-based | Free Tier | $16/month
ColourLab.ai | Auto Color, shot matching, reference extract, ‘timeline intelligence’ and advanced professional features | Mac (Windows coming soon) | Free Trial | $14/month
Color.io | Online RAW still image grading built on ACES HDR colour management with “subtractive color modelling, physically accurate refraction and film density emulation.” Export your grades as 3D LUTs or custom Resolve DCTLs | Browser-based | Free Trial | $6.25/month
EditingTools.io | AI Colorise photos and video (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits
AI Audio Post Production Tools
LaLaL.ai | Extract vocals, specific instruments from music, also audio clean up | Browser-based | 10 min Free Trial | 300 min/£15 fee | Updated their algorithm recently
Altered Studio | Voice to voice actor generation, transcription, text to speech, performance enhancement| Browser, Mac & PC | Free Trial | £49+Tax/month
Riverside.fm | Online podcast/video interview studio with AI transcription, auto-social media clips, silence removal and more | Browser-based | 2 hour free Tier| $15/month |20% off coupon code: jonnyelwyn
Read my in-depth post on why Riverside is the best video interview tool here and why I use it for AI-generated notes here.
Elevenlabs.io | Online AI voice generator with text-to-speech tools, voice clone and translation and dubbing in 29 languages l | Browser based | Free Trial | $5/month for 30,000 characters.
Descript | Voice enhancement with Studio Sound, voice cloning and AI voices with Overdub, filler word removal | Browser-based | Free Tier | $12/month
Flawless.ai | Using AI and DeepFake techniques to deliver ‘immersive lip sync‘ for seamless digital reshoots and translation | Bespoke service per project
I’ve written more about DeepFakes in Post Production here.
Melody.ml | Create separate music stems (vocals, drums, bass etc.) from a single track | Browser-based | 2 free songs then $0.50 per song.
Vocalremover.org | Remove vocals, create split music stems, cut and join audio files, adjust the tempo and pitch of a song | Browser-based | Free
Adobe Enhance Podcast | Online podcast studio with text editing, remote recording, AI ‘Enhance Speech’ and AI Mic check | Browser-based | Free | Update Enhance is now in Premiere Pro 24 beta.
TimeBolt | Automatically remove silence and gaps in multi-track video and audio, more features than Recut| Mac & PC | Free Tier | $17/month
Recut | Automatically remove silence and gaps in video and audio with export to XML | Free Trial | Mac & Windows | $89 one-time fee with this discount link
Momento | AI generated social shorts selection and creation, media management, transcription and Chat-GPT trained article writing | Browser-based | $24.99/month
AI Transcription,Text Editing, Subtitles, Translation and Search Tools
A lot of tools are appearing which will transcribe your material to make it text searchable or text editable. This also opens up opportunities for metadata extraction, tagging, translation, subtitles and more.
Swell | Auto-generate transcripts, summaries, timestamps, show notes, social media content; blog posts, Twitter threads and even AI-chat bot with the content | Browser-based | Free Trial | Price per minute ($8.99/hr) | $29/month (5 hours)
Blanc | Translate your video to 47+ languages with AI generated subtitles or AI voice over and even lip-sync’d video | Browser-based | Free Trial | Price per minute | $10 free credit with code: JONNYELWYN
Descript | Transcription, text editing, voice enhancement, voice cloning, filler word removal, screen recording, podcast recording | Browser-based | Free Tier | $12/month
Nova.ai | Subtitles, translation, editing, social resizing and more | Browser-based | Free Tier | $10/month
EditingTools.io | AI subtitle translation into multiple languages (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits
Podium | From audio auto-generate shownotes, chapters, highlights, social media posts full transcripts and more | Browser-based | 3 hours free | $16/month
AI Media and Metadata Search
Momento | AI generated social shorts selection and creation, media management, transcription and Chat-GPT trained article writing | Browser-based | $24.99/month
Limecraft | “keeps track of assets on local storage and that uses AI transcription and image recognition to index video and to automate repetitive work.” | Cloud-based | Free Tier | €85/month
Add Context | Turn any YouTube playlist into a Chatbot as it learns your content: ‘ask questions, find exact moments’ | Browser-based | $4/month 10 hours of content
Kive.ai | AI search of collaborative asset management for visual references, mood boards etc | Browser-based | Free Tier | $12/month
Twelve Labs | Make your video internally searchable for objects, actions, people… | Browser-based | Free Tier
GrayMeta Curio | On-premises media search and metadata addition | Browser-based & Adobe Premiere Pro panel | Pricing by request
AI CGI & 3D Animation Tools
Wonder Studio | Drag and drop a CG character into a live action scene with AI-powered animation, lighting and composition. Export into Maya, Unreal, Blender for tweaking or export individual passes (Motion capture, Alpha mask, Clean plate, Camera track etc.) | Apply for Beta Access | Browser-based
AI Image & Video Creation Tools
These are most of the ones you’ve seen and heard of online already. Text prompt to image generation. Useful for generating random stuff in the style of different artists but also useful for creating background plates, storyboards and other visual assets.
Having played with these a bit text prompting does take a bit of practice but it doesn’t come with any of the joy, achievement, catharsis or skill that creating something ‘by hand’ conveys.
But the results are still remarkable!
Midjourney | Text to image generator in various artistic styles | Browser-based | 25 Free generations | $8/month
Kaliber.ai | Video Transformation tool with animated styling including selecting camera moves and storyboard functionality. Text to video, video to video and image to video creation. | Browser-based | Free beta | $5/month
DreamStudio | Stability AI’s text to image generator | Browser-based | 100 free credits (500 images) | $10 for 1000 credits (approx 5,000 images)
Runway.ml | Text to image, image to image, text to 3D texture and more! | Browser-based | Free Tier | $15/month
Synthesia | 85+ AI Avatars and 120+ language voice generation, video templates and more | Browser-based | £23/month
AI Image Enhancement Tools
Photoshop | Image generation, extension, augmentation on your desktop | Mac & PC | Part of Creative Cloud Subscription
LeiaPix | 2D image conversion to 3D Lightfield images. | Browser-based | Free Service
Nero ai | Image Upscaling, photo metadata tagging | Browser, Mac, PC | Free Online | Purchase credits.
AI Voice Over Generation Tools
Altered Studio | Voice to voice actor generation, transcription, text to speech, performance enhancement| Browser, Mac & PC | Free Trial | £49+Tax/month
Descript | Transcription, text editing, voice enhancement, voice cloning and AI voices, filler word removal, screen recording, podcast recording | Browser-based | Free Tier | $12/month
Blanc | Translate your video to 47+ languages with AI voice over and even lip-sync’d video | Browser-based | Free Trial | Price per minute | $10 free credit with code: JONNYELWYN
EditingTools.io | AI text to speech in multiple voices and languages (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits
Synthesia | 85+ AI Avatars and 120+ language voice generation, video templates and more | Browser-based | £23/month
AI Tools Already in Your Video Editing Software
AI Tools in DaVinci Resolve Studio
If you don’t already own the $295 Studio version of DaVinci Resolve, you’re missing out on one of the best deals in the Post Production world.
Not only is it an incredible piece of software housing at least 3 industry leading apps (Fusion, Fairlight and Resolve’s Colour Grading) but every single update has been free – so you’re not tied into a costly subscription!
Not to mention their Cloud-based team collaboration and remote workflows, new iPad version, constant update cycles and excellent hardware solutions. There’s just a lot to love!
Also Blackmagic Design have been actively developing and implementing some great AI driven features directly into DaVinci Resolve Studio, with more to come in the future.
You can read all about the Studio only features in DaVinci Resolve here, but here’s a short list of the current features powered by DaVinci Resolve’s own AI/ML ‘Neural Engine’:
- Facial recognition
- Magic mask for object isolation and tracking
- Object removal
- 3D Depth Maps
- Scene cut detection
- Smart reframing
- Speed warp retiming
- Super scale upscaling
- Auto colour
- Auto colour matching
Buy DaVinci Resolve Studio on Amazon Global Stores | or on B&H.com
AI Tools in Adobe Premiere Pro
While Adobe have to take care of a huge range of 40+ applications (so each app’s development seems to be a bit slower than Resolve’s relentless pace) they are also constantly pushing the boundaries of what’s possible through open beta’s and then later integrating those features into their Creative Cloud suite of apps.
If you’re a Premiere Pro editor like I am, then hopefully you’ve already had a chance to try out some of their current features powered by their AI/ML service Adobe Sensei but if not here’s a shortlist of some of the features in Adobe Premiere Pro currently powered by Adobe Sensei:
- Automatic Music Editing (Remix)
- Auto-ducking
- Morph Cut
- Colour Match
- Auto Reframe
- Content aware fill in After Effects
AI Creativity, Ethics and The End of Humanity
In this final section of this very long article, I wanted to share some insights on the ethics of AI creativity, where this all might be going and if we really are doomed by AI.
Kirby Ferguson’s intensely brilliant Everything is a Remix explores Artificial Creativity in the fourth part and final part of his recently re-made series.
There’s no going back
“I won’t lie and say that I know where this is going, but there’s no going back.“
Created for Runway.ml this short film explores the idea of ‘cultural ratcheting’ where “ideas that are distributed across a large network of minds can’t be lost, instead they accumulate and become more complex than any individual could grasp.” but now the disruption of AI is overtaking this uniquely human process.
AI Filmmaking is Not The Future. It’s a Grift
Patrick (H) Willems takes “AI filmmaking” to task in this extremely well researched video essay, responding to some of the viral Wes Anderson parody trailers created by Caleb Ward and arguing for the irreplaceable importance of human creativity.
The AI Dilemma
Presented at the Center for Humane Technology by Tristan Harris and Aza Raskin (creators of The Social Dilemma) explore the potentially disastrous negative effects of unleashing AI on an unsuspecting humanity.
Watch the presentation on Vimeo here.
Actually Using AI in Post-Production
At a recent LACPUG event Philip Hodgetts gives an overview of where we are at in terms of AI progress and where this is all going. Philip covers everything AI might be touching from script to screen today.
Don’t miss the amazing entirely AI generated commercial 26 minutes in.
One particularly interesting thought:
AI Transcription now breaks the paradigm of only being able to have 2/3 from fast, cheap, good. It breaks that paradigm that has fundamentally underpinned my entire career.
Philip Hodgetts
Michael Kammes follows up with a look at some AI tools that you can already use today.
He gives examples from:
- Text-to-Image (and video) – Kaiber
- Text-to-Speech via Voice Cloning – Elevenlabs
- Computer Vision – deriving meaningful information from digital media – GrayMeta’s Curio Anywhere
- Auto “Editing” – AutoPod.fm plugin for Premiere Pro
- Social Media Cutdowns – Momento.fm
Applied AI in Post-Production Part 2
In Part two of his talk, Michael Kammes continues demonstrating some of the existing AI-powered Post tools available today, including rolling your own home-brew AI.
- 01:00 Assistive color grading with Colourlab.io.
- 04:42 Host your own free, local Chatbot that looks at your documents with Anything-LLM.
- 08:20 Using OpenAI’s ChatGPT API key + cost examples
- 11:27 Expanding what an LLM “knows” with your documents, using Embeddings
- 12:12 Current easy use cases for AI
- 13:46 Online resources for models and learning more about AI
- 15:46 Why running LLMs locally is very tough (for now)
- 17:44 Cool sites and Github repositories to check out
[…] so many different kinds of AI tools available today, I’ll quickly define the scope of what we’re looking at […]