Transcribing AND translating videos with AI using Whisper tools

There are almost no requirements this time!

 

 

Required knowledge used through all the tutorial:

Opening a command prompt (CMD) window on a specific folder: This is super simple, just go to the address bar on your folder and write "cmd" to open a command prompt window on that specific direction:

Opening notepad: You probably know very well how to do this, but if not just press the windows start button and type "notepad", the application should appear there ready to open.

Getting the full location of a file: Another simple thing to do, just right click a file, select properties, go to the security tab and you will have the full path of a file to copy and paste:

 

 

Required Software:

  • 7zip
  • FFMPEG (existing in PATH, not just in a folder)
  • Visual C++ redistributables.

 

Make sure NONE of the software in this guide is installed on onedrive. Paths like, for example, C:\onedrive\user\whisper are no good. Let's start installing 7zip, which we will use to open compressed 7z files, if you already can open 7z files then ignore this part. Go to https://www.7-zip.org/download.html and download 7zip and download it, you'll need the one that says 64-bit Windows x64 and install it. Then once it is installed open it and go to options, 2 columns will appear, your username and all the PC users, click on your username column where it says 7z, like this gif:

You should be able to open 7z files with 7zip now. Now FFMPEG. FFMPEG is a collection of open source and free libraries to manipulate audio and video. It is arguably the ultimate tool to convert and manipulate audio and video. Go to https://www.gyan.dev/ffmpeg/builds/ and get the FULL ffmpeg 7z file. You need 7zip to open it.

Use 7zip to open it and put drag it to another folder on your PC, i prefer C: drive but it's not necessary. Rename it as "ffmpeg" to simplify things in the following steps:

Inside the "ffmpeg" folder there is a subfolder called "bin", in this folder are the ffmpeg executables, this is the folder you want to add to PATH.
Click the directory bar to copy the address, like this:

There are multiple ways to open the environment variables menu, the quick way is to press the start button and start typing "environment variable" it will automatically open the menu we want:

From there you have to choose "Environment variables", then on System variables select "Path" and press "Edit", finally you press "New" and add the path to FFMPEG.

There IS a small chance you won't have the "Edit" button enabled if enter this way, you can manually enter by going to the start menu, writing "Control Panel", then "System and security", then "System", followed by scrolling all the way to the bottom to "Advanced System settings" and clicking on it. Like here:

Either way, once that you put ffmpeg in the PATH, press ok on all the windows and sub-windows opened until they are all closed.
To check if you actually put ffmpeg folder on PATH, you can open a CMD window and write "PATH":

As you can see FFMPEG is there, so it was correctly added to path, if it's not there you either put the path to ffmpeg wrong or you didn't click "ok" to close all the windows related to environment variables.

Now Visual C++ Redistributables. Go to https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 , download both x86 and x64 versions of the Visual Studio 2015, 2017, 2019, and 2022 and install them:

And that be all for the pre-requisites.

 

 

Preparing the video we will use with Whisper AI with:

Whisper (or Faster-Whisper-XXL) do not generate subtitles from videos, they use the audio data to transcribe and translate. So we will use FFMPEG to extract the audio from a video in a format whisper can work with to create our subtitles For this example i will use this blender tutorial which is in chinese, it has no subtitles in either chinese or english.

CMD will have issues with foreign characters so i'll rename that video to cnvideo. Open a CMD window in the folder where your video file is and input the following FFMPEG command: ffmpeg.exe -i "cnvideo.mp4" -ar 16000 -acodec pcm_s16le "cnvideoaudio.wav"

Obviously you replace cnvideo.mp4 for your video file and cnvideoaudio for the output name you want for your audio. The format MUST be .wav though. In my case, the video was a mp4 but your video might be an .mkv. Make sure you input the correct format (right click, properties, security tab to see your video extension) The result will be a .wav audio file which we will use.

Now to use whisper!

 

Whisper.CPP and Faster-Whisper XXL:

Repetitions will be the most annoying issue you will run into. What happens here is that basically the AI gets stuck and repeats the same sentence for minutes, sometimes until the end of the video, like this:

It doesn't happen all the time, but if you run into this problem, there are some way to potentially solve it. First let's try whisper.CPP

 

 

WhisperCPP:

Go to https://github.com/ggerganov/whisper.cpp/releases/tag/v1.6.0 and download one of the relevant versions depending of if you work with GPU or with CPU:

Put the downloaded contents in a new folder called whisper. Then go to https://huggingface.co/ggerganov/whisper.cpp/tree/main and download ggml-large-v2.bin and ggml-large-v3.bin files:

You can put them anywhere, but it's preferable you put them on the whisper folder. These are the checkpoints (AI) that will do the transcription and translation.

Now we are ready to work with whisper! In the whisper folder, open a CMD window and write main.exe -help. This will bring the help commands and syntax to use with whisper

The command i will use is: main.exe -m C:\Plum\Whisper\ggml-large-v3.bin -l auto -tr -osrt cnvideo.wav -m is to tell whisper where is the checkpoint to use, in my case it's on C:\Plum\Whisper\ggml-large-v3.bin, try to not add space on your folders to avoid errors. Something like C:\my folder\some stuff here\Whisper would not be desirable due to the spaces on the folder names -l (that's an L not an i) is used to tell whisper in what language is your audio, if don't know you can put auto. -tr is used to tell whisper to translate into english, not just transcribe the audio. -osrt is used to tell whisper to generate a .srt subtitle file with what it translated. You can use -otxt for a text file, -ovtt for a vtt subtitle, etc cnvideo.wav is my audio file, obviously you put your file name there. If you are using CPU, write -ng too, to tell whisper to work with CPU instead of GPU. I am using a GPU so i didn't put it. When i press enter it will begin working:

"en": "english"
"zh": "chinese"
"de": "german"
"es": "spanish"
"ru": "russian"
"ko": "korean"
"fr": "french"
"ja": "japanese"
"pt": "portuguese"
"tr": "turkish"
"pl": "polish"
"ca": "catalan"
"nl": "dutch"
"ar": "arabic"
"sv": "swedish"
"it": "italian"
"id": "indonesian"
"hi": "hindi"
"fi": "finnish"
"vi": "vietnamese"
"he": "hebrew"
"uk": "ukrainian"
"el": "greek"
"ms": "malay"
"cs": "czech"
"ro": "romanian"
"da": "danish"
"hu": "hungarian"
"ta": "tamil"
"no": "norwegian"
"th": "thai"
"ur": "urdu"
"hr": "croatian"
"bg": "bulgarian"
"lt": "lithuanian"
"la": "latin"
"mi": "maori"
"ml": "malayalam"
"cy": "welsh"
"sk": "slovak"
"te": "telugu"
"fa": "persian"
"lv": "latvian"
"bn": "bengali"
"sr": "serbian"
"az": "azerbaijani"
"sl": "slovenian"
"kn": "kannada"
"et": "estonian"
"mk": "macedonian"
"br": "breton"
"eu": "basque"
"is": "icelandic"
"hy": "armenian"
"ne": "nepali"
"mn": "mongolian"
"bs": "bosnian"
"kk": "kazakh"
"sq": "albanian"
"sw": "swahili"
"gl": "galician"
"mr": "marathi"
"pa": "punjabi"
"si": "sinhala"
"km": "khmer"
"sn": "shona"
"yo": "yoruba"
"so": "somali"
"af": "afrikaans"
"oc": "occitan"
"ka": "georgian"
"be": "belarusian"
"tg": "tajik"
"sd": "sindhi"
"gu": "gujarati"
"am": "amharic"
"yi": "yiddish"
"lo": "lao"
"uz": "uzbek"
"fo": "faroese"
"ht": "haitian creole"
"ps": "pashto"
"tk": "turkmen"
"nn": "nynorsk"
"mt": "maltese"
"sa": "sanskrit"
"lb": "luxembourgish"
"my": "myanmar"
"bo": "tibetan"
"tl": "tagalog"
"mg": "malagasy"
"as": "assamese"
"tt": "tatar"
"haw": "hawaiian"
"ln": "lingala"
"ha": "hausa"
"ba": "bashkir"
"jw": "javanese"
"su": "sundanese"
"yue": "cantonese"

When running this, it IS likely you will run into repetitions. If that's the case, you might want to use large-v2 as the checkpoint instead of large-v3. If the issue persists, I have noticed that using CPU instead of GPU is a good way to avoid repetitions happening. HOWEVER, cpu is incredibly slow, so instead we will try Faster Whisper XXL which uses GPU and it is great at avoiding repetitions.

 

Faster-Whisper XXL:

Faster Whisper XXL is a great alternative when Whisper runs into repetitions no matter what. It is incredibly easy to use, just download from here a windows version (the latest) and put the contents in a folder: https://github.com/Purfview/whisper-standalone-win/releases/tag/Faster-Whisper-XXL

There is a single windows version for both GPU and CPU this time. In the folder where you put whisper XXL open a CMD window, write faster-whisper-xxl --help and press enter Unlike normal whisper we will use double - with XXL variant, so it's --help instead of -help. A LONG list of help commands will appear.

By default Faster-Whisper-XXL will try to work with Nvidia GPU, if it can't, it will automatically switch to CPU. Put your audio file on whisper XXL folder (for make things simpler) and run the following command. faster-whisper-xxl cnvideoaudio.wav --model large-v2 --model_dir C:\Plum\Whisper-Faster-XXL --task translate --language zh --output_format srt

cnvideo.wav is the audio you will work with. The FIRST time you run a model, Faster-Whisper-XXL will automatically download the files in the location given --model tells Faster-Whisper the model name you will use, i used large-v2 but you can also put large-v3. --model_dir tells Faster-Whisper where will the model will be, i chose C:\Plum\Whisper-Faster-XXL but obviously you choose your own folder. --task tells Faster-Whisper what to do, you can choose translate if want your subtitles translated into english or transcribe if just want to make subtitles. --language tells Faster-Whisper which language is your audio in. This time i specified zh for chinese. If don't know the language or want auto detection do not add --language, by default it will try to auto detect. --output_format tells Faster-Whisper which kind of output you want. I specified srt for srt subtitles but you can put vtt or txt too.

Let's try another syntax. faster-whisper-xxl cnvideoaudio.wav --model large-v3 --model_dir C:\Plum\Whisper-Faster-XXL --task translate --output_format vtt --device cpu

As you might have noticed, now I will use large-v3, and since it's the first time i use it, it will download it at C:\Plum\Whisper-Faster-XXL.

So BOTH large-v2 and large-v3 will be on the same folder. Now all you have to do is choose between --model large-v2 or --model large-v3.Anyway, what this is doing is translate to english, output the subtitles as vtt AND it will work with cpu rather than gpu.

Let's see the result, remember this is a chinese video:

The result is honestly incredible considering it took very little time to generate. The options in the software are the named the same as in the subtitles, so it's a 100% accurate, faithful translation!

Edit

Pub: 09 Jun 2024 18:17 UTC

Edit: 10 Jul 2024 15:36 UTC

Views: 709