The Best AI Tools for Video and Audio Post Production

October 31, 2024
32 min read
1 Comment

The Best AI Tools for Video Editors

the best ai tools for video and audio post production

What are the best AI-powered tools for video and audio post-production that video editors, colorists and sound editors should be using in their workflows?

That’s the question I set out to answer in this article, yet with the explosion of “AI” based tools (really, it’s machine learning folks!) available today, there are a huge range of apps and services that you could be using, but which ones are actually worth your time and money?

Latest Update: November 2024

  • Updated shortlist of the best AI tools
  • Updated long list of AI tools in Post Production
  • Updated video model comparison section

The 8 Best AI Tools for Post Production shortlist should help you get straight to the good stuff, while the long list that comes after that will give you a sense of everything that’s out there, although I left a few dubious options out, that were just ‘not good.’

A Shortlist of the 8 Best AI Tools for Post-Production

I’ll be updating this list regularly as I find new and decent AI-based tools to add to it, but if you see something I’ve missed, do get in touch to let me know.

At the end of this article you’ll find a breakdown of some of the AI-powered tools already available on your timeline in DaVinci Resolve Studio and Adobe Premiere Pro.

If you’re after a list of tools that will actually help you get some editing done, such as speed up your rough cut, then check out this article on The Best AI Video Editing Tools.

Table of Contents

  • The Best AI Tools for Video Post Production
  • Frequently Asked Questions about AI in Post Production
  • AI Video Editing and VFX Tools
  • AI Colour Grading Tools
  • AI Audio Post-Production Tools
  • AI Transcription, Text Editing, Subtitles, Translation
  • AI Media and Metadata Search
  • AI CGI & 3D Animation Tools
  • AI Image & Video Creation Tools
  • AI Image Enhancement Tools
  • AI Voiceover Generation Tools
  • AI Tools Already in Your Video Editing Software
  • AI Creativity, Ethics and The End of Humanity
  • Actually Using AI in Post-Production
  • AI Video Generation Models Compared

The Best AI-Based Tools for Video Post Production

As there are so many tools on this list, I’ve curated a short list of my top 8 AI post-production tools up top, to save you from scrolling too far and to give you some more details on these specific services.

As I update this list, these might swap and change as new tools and services become available. I’ll also suggest how and why exactly a video editor like myself can use these AI tools in their workflow.

TOP 9 AI POST-PRODUCTION TOOLS

Strada.tech – Michael Cioni’s Strada.tech platform aims to leverage AI to solve many of Post Production’s core workflow problems. Having only just launched the company they are already rolling out a beta which does some incredibly impressive and really useful things such as:

Transfer all of your media data to the cloud and within the cloud into the Strada platform. This allows you to create virtual files where your footage is on Frame.io but your audio files are in Dropbox yet you can play them both back in sync in Strada.

Transcription provides the backbone of metadata search as well as text-based editing and translation. Meanwhile, the parallel processing of the cloud means that transcoded files are generated in minutes rather than hours.

Their Tag/Analyse view of presenting in-frame object metadata is the best and most intuitive design that I’ve seen!

I’m really excited about Strada for a couple of reasons: Michael’s track record at LightIron, Panavision and Frame.io, their creator-first development philosophy, the beautiful design aesthetic and mostly, that it’s a post-production-centric tool that solves real problems!

Sign up for the beta here.

Runway.ml – The Swiss Army knife of AI powered browser-based tools, Runway.ml has a huge range of useful features such as

  • Rotoscoping
  • Object removal
  • Text-to-video, video-to-video
  • Motion tracking
  • Colour grading
  • Audio noise removal
  • Transcription,
  • Text-to-image, image-to-image
  • and much more.

Imagine using this as your (temp) VFX tool for rapidly fixing shots, creating new assets or backgrounds and solving problems for your clients like you are able to do magic.

If all goes well you could complete the shot inside of Runway’s online editor or download elements like Alpha mattes, new green screen elements, background plates and more to finish the shot in your NLE of choice.

Update November 2024 – Runway’s video model is now Gen 3-Alpha, and includes the abilities to create vertical videos, edit time and speed, extend shots to up to 40 seconds and more.

Gen 3 Alpha Act-One is their latest tool which transforms a video reference into performance capture for animated generated videos, which you can see in action here.

Sign up for free to try Runway.ml for yourself

Adobe AI Tools – Premiere Pro, Adobe Enhance, Photoshop, Firefly Model

During NAB 2024, Adobe previewed new AI-powered tools coming soon to Adobe Premiere Pro, including the ability to:

  • Add frames you don’t have – extend a shot with generative AI
  • Add objects – Text prompt to video within a selection
  • Remove objects – logos, text etc with roto/tracking built-in
  • Replace objects – swap costumes, props and much more
  • Content Credentials – know if AI was used in a video edit
  • Leverage third-party video models – Pika, Sora and Runway
  • Add shots – Text prompt to b-roll with the above models

Update October 2024 – Generative Extend is now available in the Premiere Pro beta with the ability to extend the beginning or end of a clip by 2 seconds. It can also create 10-seconds of background audio atmos too.

Adobe Enhance Audio – While the ability to apply this audio clean-up tool is now in Premiere Pro’s Essential Sound Panel, I often prefer to use this online tool for a couple of reasons:

  • Offload compute to the cloud for a single, baked file
  • When applied as a clip effect you have to re-render after every change

There is a limited free version, but if you’re already a Creative Cloud subscriber, you can log in for premium features, including video uploads, bulk uploads, strength adjustment, and up to 4 hours of processing a day with files up to 1GB.

Adobe Firefly – This is the AI model at the heart of all of Adobe’s AI tools and has some unique features such as being safe for commercial use and ‘creator first’ with a couple of industry initiatives to stand up for “accountability, responsibility, and transparency in digital content.“

Adobe Photoshop – I also make use of Adobe’s AI tools in Photoshop for generatively extending images, selecting subjects and other simple tasks.

ElevenLabs.io – A very affordable, and to date, the best sounding, speech-to-text AI voice generation platform I’ve heard. It also offers voice cloning based on a 1 minute sample and the results that I’ve heard are incredibly impressive.

I’ve sometimes used this voice clone tool to fix client audio fix requests. The 1-minute sample training is usually enough to generate a passable copy within a few generations and with a bit of audio massaging such as adding EQ and reverb to help with the match.

One of the newest features is their vocal design studio that allows you to create a voice, based on a simple text prompt. Elevenlabs also has voice to voice dubbing, text-to-voice readers and conversational AI APIs. They are one of the most innovative AI start-ups out there.

You can also generate sound effects from a text-prompt. My main issue with this kind of service is that they are not specific sounds. So you can’t create the sound of a 57″ chevvy revving, just a ‘car revving’.

The free account gives you 10,000 generated characters to play with, plus 5 cloned voices although you can only use it for non-commercial use and attribution is required.

However, the low-cost $5/month tier gives you 30,000 characters, 10 cloned voices and commercial use, not to mention API access, should you want it.

Unlike Altered Studio that charges by the downloaded minute, rather than the generation attempts, Elevenlabs charges per character generated.

Elevenlabs speech results are ideal for ebook, article and podcasts reads, although they are less ‘character driven’ than Altered Studio’s voices, the realism is much more believable.

In the video above I created an experimental episode of an issue of Cut/daily (my free weekly newsletter) which fit easily into the 10,000 character free quota, even with a few regeneration attempts, specific fixes and the like. All in all, incredibly easy to use and very impressed with the results.

Riverside.fm is my go-to video call meeting platform, for podcast interviews, client meetings and general catch ups for two reasons:

  1. Pristine local recording for editable audio and video stems
  2. AI Transcripts for AI summaries.

If you record every call you can use the automatic AI transcription to feed the call transcript into Claude or ChatGPT to generate summaries, to do lists, feedback and generally use AI to ask questions of the text.

Riverside also has AI-generated show notes (summaries, title suggestions, chapter lists etc.), audio clean up via ‘magic audio’, text-to-speech in your voice plus limited lip sync too.

I use Riverside a lot and you should too. Save 20% on any Riverside.fm subscription with the discount code: jonnyelwyn

Artlist.io – I use Artlist for royalty-free production music for all of my projects, Artgrid for ungraded camera original stock footage, and now their AI-generated voiceovers.

These come in a range of accents, genders, and performance styles and sound convincing with a few tweaks and re-generations.

Again you’ll need to tweak these with EQ and reverb filters in your edit, especially if you’re trying to fit them into a specific audio environment. But Artlist also has an extensive sound effects library to help create this.

If you’re already an Artlist Max subscriber you’ll have AI voiceovers included in your plan, or it’s available solo for $9.99/month.

DaVinci Resolve Studio – May have less features than Runway but they do come with the huge benefit of being available right there in your timeline, no uploading or downloading required.

Also a one-time price of $295 for a lifetime of updates and usage it’s a pretty economical option.

DaVinci Resolve’s ‘Neural Engine’ features include facial recognition, magic mask, super scale, 3D depth maps, scene cut detection and a whole lot more (see the end of this article for more details!)

The Ripple Training video above covers these features:

  • AI Deinterlace
  • AI Super Scale
  • AI Audio Ducker
  • AI Detect Scene Cuts
  • AI Smart Reframe
  • AI Create Subtitles from Audio
  • AI Editing with Text
  • AI Intellitrack

Buy DaVinci Resolve Studio on Amazon Global Stores

ColourLab.ai & Fylm.ai – OK so I’ve sort of cheated with this one, in that these are two completely different (competing) tools for AI powered colour grading.

ColourLab’s tools are machine based with a stand alone app and plugins for Adobe Premiere, Final Cut Pro, DaVinci Resolve and Pomfort Live Grade while fylm.ai works in your web browser and processes everything on their cloud-based servers.

The benefit of AI powered grading is that you can let the algorithm do the heavy lifting of matching references, extracting looks, making different cameras match each other, creating look variations on the fly and a whole lot more.

Both are being consistently developed and improved over the past few years with fylm.ai recently updating to v1.5, which paves the way for iPad and mobile friendly versions.

Likewise, ColourLab’s product range is now at version 2.3, with their offering split into three main tools (Colour Lab AI, Look Designer and GrainLab) although they all work cohesively together, depending on what you’re doing.

Update: FreeLab Plugin – ColourLab.ai have released a new free plugin to ‘replace LUTS’ for Premiere and Resolve. Essentially, it is a ”free version of Colourlab’s film native colour engine, allowing you to balance cameras and apply consistent look across a variety of materials.”

Download it here to try it for yourself.

I’ve previously written more about both services in this post on Tools & Training for Pro Colorists.

FAQ about AI in Post-Production

With AI The Best UI Wins

My guess is that many of the services listed in this article are likely accessing the same AI and ML open source models underneath their browser-based UI. So, really, the race is on for whoever can develop the best user friendly interface, affordable pricing structure and win over enough paying users to hog market share and establish a sustainable business.

That said, some brands like Runway.ml, are actively developing their own proprietary models such as Stable Diffusion and have the VC-backed funds to develop new and existing features, market themselves well and play the long game.

Will AI Take My Job?

Whether you’re fearful of AI taking your job, or just want to get your day-to-day tasks done more efficiently, the AI toothpaste is already on its way out of the tube so we may as well make the most of it.

If you don’t, others will.

Personally, I’m not worried about creative video editors being relegated to the annals of history just yet, as the creative decision-making, client relationships, and cognitive processes involved in making excellent content aren’t going to be beat by AI—at least not just yet!

That said, obviously, AI editing tools can slap clips and audio together and create an ‘edit,’ but it’s not an apples-to-apples comparison.

What some of these tools excel at is removing the (human!) manual labour from complex tasks such as rotoscoping, transcription or asset tagging. Yet all of this work still needs to be managed, tweaked, corrected, edited and approved by a human.

AI Vs ML – Marketing Terms vs Technical Reality

To Nerd-splain for a minute, A.I. and M.L. are quite different things.

A.I. is mostly what you think of when you think of A.I.

Intelligent machines that can think and act upon their own objectives, usually involving world domination and the subjugation of the human race. They can do multiple things with dexterity.

Machine Learning, however is a branch of A.I. and far more constrained and it’s behind what most of these tools are doing.

Provide your image recognition algorithm with a large enough dataset of images, for example of cats, the computer program would be able to iteratively learn what a cat generally looks like based on patterns of pixels and so (eventually) predictively tag photos of cats with the metadata ‘cat’.

So there’s no need to start building your survival bunker just yet.

AI Video Editing and VFX Tools

In this section of the post I’ve curated a long list of AI/ML based tools that could come in handy as a video editor, colorist, sound designer or DIT.

For brevity, I’ve tried to keep things to the following formula:

Tool | Main Features | OS | Free Version? | Lowest monthly price

Runway.ml | Rotoscoping, motion tracking, audio clean up, transcription and subtitles, export alpha mattes, in-painting, text to video, video to video, text to image, image to image, text to 3D texture and more! | Browser-based | Free Tier | $15/month

Strada | Multi-cloud multi-cam dailies creation, metadata tagging and search, object recognition, face search | Browser-based | Public beta

Topaz Labs | Still image and video – upscaling, denoise, sharpening, deinterlacing | Windows & Mac | Watermarked Free Trial | Topaz Photo AI – $199, Topaz Video AI – $299 – One-time fee.

Project Blink | Adobe’s AI-powered browser-based video editor with content search, text-based editing, transcripts, social clips, filler-word removal | Browser-based | Free Beta Trial

Adobe Firefly | Recently announced in beta, this collection of online tools from Adobe does a lot of similar things to Runway.ml, but will eventually be embedded in Adobe’s ecosystem of apps and websites. Adobe are also trying to ‘train ethically’ | Browser-based | Free

Vidyo.ai | Instantly create social media subclips with captions, resizing, chapters | Browser-based | 3 hours Free Tier with 720p exports | $19.99/month

EditingTools.io | AI face detection and replacement (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits

AutoPod | AI powered multi-camera editor, shorts selection and silence removal plugins for Adobe Premiere Pro| Mac & PC | $29/month

Momento | AI generated social shorts selection and creation, media management, transcription and Chat-GPT trained article writing | Browser-based | $24.99/month

BigRoom.TV | Auto create 9:16 social media clips from 16:9 video with cuts and motion tracking | Browser-based with export to NLE | Free for now

Chat-GPT | The infamous chat bot that’s going ‘take over everything’ is actually useful for asking it for After Effects Expressions or DCTL code snippets in DaVinci Resolve | Browser-based | Free to use

CleanCut.ai | “Sifts out blurry or shaky clips, selects the optimal shots, and auto-assembles them into engaging sequences (based on text prompt), ready for your final touch-ups.” | NLE-based plugin | Currently in Beta

Boris FX | Boris Continuum suite gains AI tools such as Witness Protection ML, which auto-blurs out subjects faces or a similar tool for license plates. | NLE-based plugin | £30/month

Eddie.AI | Text-to-edit assistant editor with a sassy personality. Upload an interview and have Eddy cut it down and iterate on its edits, based on your feedback. | Read about my experiments with it here. | Browser-based | Currently in beta

Firecut.ai | Social media editing tool that removes silences, auto zooms, adds chapters and captions etc. | Premiere Pro Plugin | $11/month

AI Colour Grading Tools

fylm.ai | AI grade extract, auto correct, colour matching, advanced technical matching, grade creation | Browser-based | Free Tier | $16/month

ColourLab.ai | Auto Color, shot matching, reference extract, ‘timeline intelligence’ and advanced professional features | Mac (Windows coming soon) | Free Trial | $14/month

Color.io | Online RAW still image grading built on ACES HDR colour management with “subtractive color modelling, physically accurate refraction and film density emulation.” Export your grades as 3D LUTs or custom Resolve DCTLs | Browser-based | Free Trial | $6.25/month

EditingTools.io | AI Colorise photos and video (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits

AI Audio Post-Production Tools

LaLaL.ai | Extract vocals, specific instruments from music, also audio clean up | Browser-based | 10 min Free Trial | 300 min/£15 fee | Updated their algorithm recently

Riverside.fm | Online podcast/video interview studio with AI transcription, auto-social media clips, silence removal and more | Browser-based | 2 hour free Tier| $15/month |20% off coupon code: jonnyelwyn

Read my in-depth post on why Riverside is the best video interview tool here and why I use it for AI-generated notes here.

Adobe Enhance Podcast | Online podcast studio with text editing, remote recording, AI ‘Enhance Speech’ and AI Mic check | Browser-based | Free | Update Enhance is now in Premiere Pro 24 beta.

Elevenlabs.io | Online AI voice generator with text-to-speech tools, voice clone and translation and dubbing in 29 languages l | Browser based | Free Trial | $5/month for 30,000 characters.

Descript | Voice enhancement with Studio Sound, voice cloning and AI voices with Overdub, filler word removal | Browser-based | Free Tier | $12/month

Altered Studio | Voice to voice actor generation, transcription, text to speech, performance enhancement| Browser, Mac & PC | Free Trial | £49+Tax/month

Flawless.ai | Using AI and DeepFake techniques to deliver ‘immersive lip sync‘ for seamless digital reshoots and translation | Bespoke service per project

I’ve written more about DeepFakes in Post Production here.

Melody.ml | Create separate music stems (vocals, drums, bass etc.) from a single track | Browser-based | 2 free songs then $0.50 per song.

Vocalremover.org | Remove vocals, create split music stems, cut and join audio files, adjust the tempo and pitch of a song | Browser-based | Free

TimeBolt | Automatically remove silence and gaps in multi-track video and audio, more features than Recut| Mac & PC | Free Tier | $17/month

Recut | Automatically remove silence and gaps in video and audio with export to XML | Free Trial | Mac & Windows | $89 one-time fee with this discount link

Momento | AI generated social shorts selection and creation, media management, transcription and Chat-GPT trained article writing | Browser-based | $24.99/month

AI Transcription, Text Editing, Subtitles, Translation, Dubbing and Show Notes

A lot of tools are appearing which will transcribe your material to make it text searchable or text editable. This also opens up opportunities for metadata extraction, tagging, translation, subtitles and more.

Riverside.fm | Online interview recording studio with AI transcription, auto-social media clips, silence removal, audio polish, podcast show notes etc. | Browser-based | 2 hour free Tier| $15/month |20% off coupon code: jonnyelwyn

Swell | Auto-generate transcripts, summaries, timestamps, show notes, social media content; blog posts, Twitter threads and even AI-chat bot with the content | Browser-based | Free Trial | Price per minute ($8.99/hr) | $29/month (5 hours)

Blanc | Translate your video to 47+ languages with AI generated subtitles or AI voice over and even lip-sync’d video | Browser-based | Free Trial | Price per minute | $10 free credit with code: JONNYELWYN

Descript | Transcription, text editing, voice enhancement, voice cloning, filler word removal, screen recording, podcast recording | Browser-based | Free Tier | $12/month

Nova.ai | Subtitles, translation, editing, social resizing and more | Browser-based | Free Tier | $10/month

EditingTools.io | AI subtitle translation into multiple languages (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits

Podium | From audio auto-generate shownotes, chapters, highlights, social media posts full transcripts and more | Browser-based | 3 hours free | $16/month

Descript – Offers an ‘all-in-one’ solution for text-based audio and video editing which includes the ability to ‘overdub’ words and phrases with an AI generated version of the speaker.

Descript is already really popular with podcasters and anyone who doesn’t mind a jump cut in their video. It can automatically remove pauses and filler words (ums, ahs etc.) and allows you to edit your transcribed video like a text document.

It also features a comprehensive browser-based multi-track editing tool, very impressive ‘Studio Sound‘ noise reduction and audio rebuilding, transcription, caption creation and the ability to export to professional applications such as Premiere, Final Cut Pro and others.

The Free account includes 1 hour of transcription, one 720p watermark free export, ums & ahs removed, 1000 word overdub vocabulary and 10 minutes of Studio Sound per month.

Swell for Transcripts, Timestamps and Social Media

I’ve been using Swell.ai to create timestamps for my YouTube videos, which can often get pretty long, but that’s only scratching the surface of this incredibly useful tool.

You can upload video and audio and have it generate:

  • Complete transcripts
  • Export subtitles (.srt/vtt)
  • Shownotes
  • Timestamps (Show notes and YouTube chapters)
  • Summaries
  • Titles
  • Social media content (Twitter threads, blog posts, Newsletters, YouTube descriptions etc.)
  • Long-form articles
  • Chat-Agent

You can use the chat-agent to prompt Swell to do specific things, such as create a Twitter thread with a specific number of Tweets, or a summarise a portion of the interview etc.

Other cool features include watch folders for importing content into Swell from Dropbox, Google Drive, Apple podcast and other sources as well as outputting the results of Swell’s work back to similar locations.

Try Swell for yourself, for free here.

Useblanc.com – Blanc’s super power is its ability to convert one transcription into multiple languages, there by exponentially increasing the reach of the content to new audiences.

Currently this means you can create 47 other versions of your video with either translated subtitles, AI-generated voice over, or even AI-generated replacement lipsync.

One of the cool things that Useblanc does is to translate the original text accurately, but also in a way that mitigates vast differences in word-length or sentence structure between languages, in order to deliver the most natural sounding results.

You can get $10 worth of free credit to test Useblanc.com for yourself, with the promo code JONNYELWYN.

AI Media and Metadata Search

Strada | AI multi-cloud multicam dailies creation, metadata tagging and search, object recognition, face search | Browser-based | Public beta

Momento | AI generated social shorts selection and creation, media management, transcription and Chat-GPT trained article writing | Browser-based | $24.99/month

Limecraft | “keeps track of assets on local storage and that uses AI transcription and image recognition to index video and to automate repetitive work.” | Cloud-based | Free Tier | €85/month

Add Context | Turn any YouTube playlist into a Chatbot as it learns your content: ‘ask questions, find exact moments’ | Browser-based | $4/month 10 hours of content

Kive.ai | AI search of collaborative asset management for visual references, mood boards etc | Browser-based | Free Tier | $12/month

Twelve Labs | Make your video internally searchable for objects, actions, people… | Browser-based | Free Tier

GrayMeta Curio | On-premises media search and metadata addition | Browser-based & Adobe Premiere Pro panel | Pricing by request

AI CGI & 3D Animation Tools

Wonder Studio | Drag and drop a CG character into a live action scene with AI-powered animation, lighting and composition. Export into Maya, Unreal, Blender for tweaking or export individual passes (Motion capture, Alpha mask, Clean plate, Camera track etc.) | Apply for Beta Access | Browser-based

AI Image & Video Creation Tools

“What AI looks like, in abstract” – according to MidJourney

These are most of the ones you’ve seen and heard of online already. Text prompt to image generation. Useful for generating random stuff in the style of different artists but also useful for creating background plates, storyboards and other visual assets.

Having played with these a bit text prompting does take a bit of practice but it doesn’t come with any of the joy, achievement, catharsis or skill that creating something ‘by hand’ conveys.

But the results are still remarkable!

Midjourney | Text to image generator in various artistic styles | Browser-based | 25 Free generations | $8/month

Kaliber.ai | Video Transformation tool with animated styling including selecting camera moves and storyboard functionality. Text to video, video to video and image to video creation. | Browser-based | Free beta | $5/month

DreamStudio | Stability AI’s text to image generator | Browser-based | 100 free credits (500 images) | $10 for 1000 credits (approx 5,000 images)

Runway.ml | Text to image, image to image, text to 3D texture and more! | Browser-based | Free Tier | $15/month

Synthesia | 85+ AI Avatars and 120+ language voice generation, video templates and more | Browser-based | £23/month

AI Image Enhancement Tools

Photoshop | Image generation, extension, augmentation on your desktop | Mac & PC | Part of Creative Cloud Subscription

LeiaPix | 2D image conversion to 3D Lightfield images. | Browser-based | Free Service

Nero ai | Image Upscaling, photo metadata tagging | Browser, Mac, PC | Free Online | Purchase credits.

AI Voiceover Generation Tools

Artlist | Voice over generation tools in a range of bespoke voices | Browser based | No free trial | Voice over only $9.99/month

Elevenlabs.io | Online AI voice generator with text-to-speech tools, voice clone and translation and dubbing in 29 languages l | Browser based | Free Trial | $5/month for 30,000 characters.

Altered Studio | Voice to voice actor generation, transcription, text to speech, performance enhancement| Browser, Mac & PC | Free Trial | £49+Tax/month

Descript | Transcription, text editing, voice enhancement, voice cloning and AI voices, filler word removal, screen recording, podcast recording | Browser-based | Free Tier | $12/month

Blanc | Translate your video to 47+ languages with AI voice over and even lip-sync’d video | Browser-based | Free Trial | Price per minute | $10 free credit with code: JONNYELWYN

EditingTools.io | AI text to speech in multiple voices and languages (plus a ton of other great tools) | Browser-based | No free trial | Pay for per-usage credits

Synthesia | 85+ AI Avatars and 120+ language voice generation, video templates and more | Browser-based | £23/month

Altered Studio – Actors in a box. That’s what Altered Studio can give you. Drop in your own line reading of some voice over or scripted lines into the system and let Altered Studio turn it into a complete multi-character production.

You can change your accent, add ‘gravitas and charisma’ with their 200+ voices, plus you can clone your own voice samples to be able to add text-to-speech additions to your edit.

Some of the voice samples in their demos sound more convincing than others but the ability to generate an entire table-reading, voice over or dramatic podcast from scratch with professional sounding voices is very compelling.

And while you can select different moods and stylistic approaches, I’m not sure there’s a way to ‘give direction’ to these AI voices in the same way that you can to a real actor – but at least you can create your first version, affordably.

AI Tools Already in Your Video Editing Software

AI Tools in DaVinci Resolve Studio

If you don’t already own the $295 Studio version of DaVinci Resolve, you’re missing out on one of the best deals in the Post Production world.

Not only is it an incredible piece of software housing at least 3 industry leading apps (Fusion, Fairlight and Resolve’s Colour Grading) but every single update has been free – so you’re not tied into a costly subscription!

Not to mention their Cloud-based team collaboration and remote workflows, new iPad version, constant update cycles and excellent hardware solutions. There’s just a lot to love!

Also Blackmagic Design have been actively developing and implementing some great AI driven features directly into DaVinci Resolve Studio, with more to come in the future.

You can read all about the Studio only features in DaVinci Resolve here, but here’s a short list of the current features powered by DaVinci Resolve’s own AI/ML ‘Neural Engine’:

  • Facial recognition
  • Magic mask for object isolation and tracking
  • Object removal
  • 3D Depth Maps
  • Scene cut detection
  • Smart reframing
  • Speed warp retiming
  • Super scale upscaling
  • Auto colour
  • Auto colour matching

Buy DaVinci Resolve Studio on Amazon Global Stores | or on B&H.com

AI Tools in Adobe Premiere Pro

While Adobe have to take care of a huge range of 40+ applications (so each app’s development seems to be a bit slower than Resolve’s relentless pace) they are also constantly pushing the boundaries of what’s possible through open beta’s and then later integrating those features into their Creative Cloud suite of apps.

If you’re a Premiere Pro editor like I am, then hopefully you’ve already had a chance to try out some of their current features powered by their AI/ML service Adobe Sensei but if not here’s a shortlist of some of the features in Adobe Premiere Pro currently powered by Adobe Sensei:

  • Automatic Music Editing (Remix)
  • Auto-ducking
  • Morph Cut
  • Colour Match
  • Auto Reframe
  • Content aware fill in After Effects

AI Creativity, Ethics and The End of Humanity

In this final section of this very long article, I wanted to share some insights on the ethics of AI creativity, where this all might be going and if we really are doomed by AI.

Kirby Ferguson’s intensely brilliant Everything is a Remix explores Artificial Creativity in the fourth part and final part of his recently re-made series.

Watch all four parts here.

There’s no going back

“I won’t lie and say that I know where this is going, but there’s no going back.“

Created for Runway.ml this short film explores the idea of ‘cultural ratcheting’ where “ideas that are distributed across a large network of minds can’t be lost, instead they accumulate and become more complex than any individual could grasp.” but now the disruption of AI is overtaking this uniquely human process.

AI Filmmaking is Not The Future. It’s a Grift

Patrick (H) Willems takes “AI filmmaking” to task in this extremely well researched video essay, responding to some of the viral Wes Anderson parody trailers created by Caleb Ward and arguing for the irreplaceable importance of human creativity.

The AI Dilemma

Presented at the Center for Humane Technology by Tristan Harris and Aza Raskin (creators of The Social Dilemma) explore the potentially disastrous negative effects of unleashing AI on an unsuspecting humanity.

Watch the presentation on Vimeo here.

Making Viral Videos with AI – Dave Clark

Director and AI filmmaker Dave Clark has created some slick-looking teaser trailers and concept films in various different genres using a wide range of AI-powered filmmaking tools.

If you want to get a sense of what’s possible – spend a few minutes browsing his YouTube channel!

In this conversation with Every CEO Dan Shipper, Dave talks about his process and makes a short film from scratch.

World’s First AI Animated Feature Film

Apparently Where The Robots Grow is the world’s first AI animated feature film, created “with a combination of 3D, VFX and AI tools.” Costing just $8000/minute for it’s 87 minute runtime and made by Tom Paton’s AiMation Studios with a core team of 9 artists.

Lots of people asking (or just writing it somewhere on the internet), so I just want to reiterate… there isn’t a single frame of the movie that doesn’t have some form of GenAI in it. From MoCap, to shot clean up, to landscapes and all the way through to audio.

Actually Using AI in Post-Production

5 Things host Michael Kammes, showcases 5 AI Tools for Post Production that you may not have heard of including StoryToolKit AI, Twelve Labs, Curio Anywhere, CodeProject.ai, Pinokio.

These include AI models that work locally, include open-source solutions and allow you to select the kind of machine learning you want to use.

At a recent LACPUG event, Philip Hodgetts gave an overview of where we are in terms of AI progress and where this is all going. Philip covers everything AI might be touching from script to screen today.

Don’t miss the amazing, entirely AI-generated commercial 26 minutes in.

One particularly interesting thought:

AI Transcription now breaks the paradigm of only being able to have 2/3 from fast, cheap, good. It breaks that paradigm that has fundamentally underpinned my entire career.

Philip Hodgetts

Michael Kammes follows up with a look at some AI tools that you can already use today.

He gives examples from:

Applied AI in Post-Production Part 2

In Part two of his talk, Michael Kammes continues demonstrating some of the existing AI-powered Post tools available today, including rolling your own home-brew AI.

  • 01:00 Assistive color grading with Colourlab.io.
  • 04:42 Host your own free, local Chatbot that looks at your documents with Anything-LLM.
  • 08:20 Using OpenAI’s ChatGPT API key + cost examples
  • 11:27 Expanding what an LLM “knows” with your documents, using Embeddings
  • 12:12 Current easy use cases for AI
  • 13:46 Online resources for models and learning more about AI
  • 15:46 Why running LLMs locally is very tough (for now)
  • 17:44 Cool sites and Github repositories to check out

AI Video Generation Models Compared

When it comes to generating video clips from a text prompt or source image there are plenty of different models to choose from. The main contenders being:

  • OpenAI’s Sora
  • Pika Lab’s Pika
  • Runway’s Gen 3
  • Kling
  • Hailuo (Minimax)
  • Luma

Video generation app Krea claims to have all of the different video generations piped in via API to bring them all into one place – which might be handy if you’re trying to get the best results for a particular shot by testing all of them at once.

OpenAI (of ChatGPT fame) recently introduced their video model, Sora, which has some impressive-looking results in the video above. You can also watch some short films made using Sora here.

Pika

Pika now offers lipsync and sound effects as part of their video generation suite of tools, plus the ability to edit regions, reprompt generations and much more.

Runway Gen 3

One of Runway’s most useful tools is the motion brush, which allows you to add movement to still images both in terms of the camera and various layers of the image.

Kling

AI image and video generation from Kling.ai with an online video editor in browser coming soon. Some impressive results.

Lumalabs Dream Machine

LumaLabs latest video model, Dream Machine, let’s you created 1024 pixel generations with fluid cinematic camera moves and very active in-frame motion. It is also optimised for consistent character generation too.

Hailuo (minimax)

As far as I can tell Hailuo is the new name for the video generation platform from Minimax, which can create some stunning looking videos and strange effects at the same time. Paid and free tiers.

You may also like

1 Comment

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.