Using videocr-PaddleOCR to OCR hardsubs

Disclaimer: I've only got into all this python business so I could run stable diffusion, so I generally have no idea what I'm writing about. If you run into any errors, google is your friend.

First, you'll need to download python. At the moment, videocr-PaddleOCR requires version 3.7 - 3.10
https://www.python.org/downloads/

Then, choose your system here.
https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/pip/windows-pip.html
If you're using Linux, you don't need any installation instructions from me, and there's no GPU versions for macOS, and I never learned to use conda, so we're talking Windows pip.
The GPU version requires corrects versions of CUDA and cuDNN installed. There's also no support for CUDA 12.1+ at the moment, so if you have the latest version installed, you'll have to either use docker (if you already know how to use it (I don't), otherwise it won't be worth your time) or downgrade your CUDA/cuDNN. Either way, the speed benefits of using GPU aren't THAT big. I did a benchmark for a full episode upscaled to 720, with the settings from the script below: CPU, i5-12400F - 6 minutes 15 seconds, GPU, RTX 3060 - 2 minutes 33 seconds. If you aren't going to OCR a 200 episodes show and don't have any experience installing software via command line, there's a good chance that you'll spend more time trying to make the GPU version work than you'd save on OCRing.
Whichever you choose, you'll get a line starting with "python -m pip install". Launch Command Prompt, paste the line into it and hit enter (google venv if you want to install everything into a single folder instead of globally). The Chinese servers are slow and time out often, so just wait and repeat the process if some files time out in the middle of downloading.

Hopefully, you managed to install it. Now go here and paste the line from the installation section into Command Prompt. Also, read the documentation.
https://github.com/devmaxxing/videocr-PaddleOCR

If the stars aligned, you will succeed. Now copy the following code and save it as whatevernameyouwant.py (replace x:\scanlation\preprocessed video\episode with full path and name of your video. Make sure that the first instance ends with your video's extension, and the second instance ends with .srt. You can use any folder you want for the .srt file). Change use_gpu=True to use_gpu=False if you want to run it on CPU. The crop settings are for 948x720 video, change them to the subs area of your video (check the example picture on videocr-PaddleOCR page).
These are the more or less optimal settings for HnK's Cosmic Baton Girl Comet-san release (with very thin pale yellow subs) and from before I discovered that using both English and Chinese at the same time radically improves things. Check the script at the bottom of this document, perhaps it will be more useful for your purposes.

1
2
3
4
5
6
7
from videocr import save_subtitles_to_file
if __name__ == '__main__':
    save_subtitles_to_file("x:\scanlation\preprocessed video\episode.mkv", "x:\scanlation\preprocessed video\episode.srt", lang='ch', #time_start='0:00', time_end='1:00',
     sim_threshold=98, conf_threshold=0, use_fullframe=False,
     det_model_dir=None, rec_model_dir=None, use_gpu=True,
     brightness_threshold=0, similar_image_threshold=100, similar_pixel_threshold=120, frames_to_skip=0,
     crop_x=1, crop_y=550, crop_width=946, crop_height=170)

Launch Command Prompt, run python "c:\path\to\the\folder\with\the\script\whatevernameyouwant.py"

That's it.

Tips:
If you're OCRing a 640x480 or smaller video, UPSCALE IT TO 720 FIRST, or the results will be much less accurate. Make sure to preserve the framerate. It will be a temporary file, so don't worry about filesize. Use crf 18, go with whatever codec your rig can handle fast, don't use a preset slower than medium. The accuracy at 1080 is actually lower than at 720, so don't overscale.

You can use either lang='en' for the English model or lang='ch', to use the Chinese-English model (which is stated to be slightly better for English language). The Chinese model seems to be much better at recognizing punctuation, but really likes to glue words together with the above settings, so it's up to you if you prefer to manually add periods or spaces. I think the English model can be a bit sloppy with timing, too, so I mostly used the Chinese before discovering the following:
The best results can be achieved by using the Chinese detection model together with the English recognition model. You need to have both models downloaded - just run a script once with lang='ch' and once with lang='en' (with det_model_dir=None, rec_model_dir=None), they'll be downloaded automatically. Now, replace det_model_dir=None, rec_model_dir=None in the script with det_model_dir="C:/Users/admin/.paddleocr/whl/det/ch/ch_PP-OCRv4_det_infer", rec_model_dir="C:/Users/admin/.paddleocr/whl/rec/en/en_PP-OCRv4_rec_infer" (change the path to whatever folder you have them installed on your system, make sure to use forward slashes / and not backslashes \. At the moment (March 2024), v4 is the latest one, just check the folder to see if it's still the case). When you're using both models, you'll have to set lang='en' or everything will be in Chinese.

If you want to experiment with parameters (you do. the settings in my script are tuned for the files I'm working on, and you could achieve better accuracy or speed, or get less trash), uncomment time_start and time_end (remove #), set them to a particularly problematic scene and play with settings until something changes to the better.
Also, If your episodes don't have pre-OP and post-ED segments, you can modify time_start and time_end to only OCR the video between OP and ED. This way you won't have to manually remove kilometers of junk caused by the Japanese on-screen text, and the time needed for OCR can be HALVED, an episode of Pastel Yumi takes me 1 minute 15 seconds on 3060.

Don't set frames_to_skip higher than 0. It would speed things up, but the timing will be off. And even if you set it to 0, the lines seem to always end 1 frame too early, which can be easily fixed with Aegisub, go to Timing->Shift Times and shift all rows by 1 frame forward, END TIMES ONLY (you need to have video loaded, otherwise the Frames option will be grayed out)

For the love of God, don't set crop_x to 0, or it will ocr the entire frame instead and you'll end up with tons of useless trash in subs.

Crop area can be tricky, set it too small and the results will be less accurate, set it too big and you'll get less punctuation marks recognized.

Ellipses are often recognized as "..", you can mass replace such cases in Notepad++: go to Search->Replace, set the Search Mode to "Regular expression", then paste this (?<!\.)\.\.(?!\.) into the "Find what" field and ... into the "Replace with" field.
Replace also \.\.\.\. with ... , (?<!,),\. with , , \?\. with ? and \!\. with !.
If the subs are missing many periods, consider replacing Dialogue.*[a-z]($) with $0. , this regular expression will find lines that end without any punctuation symbols. Unlike with ellipses, pressing "replace all" can be harmful, because some sentences can span for multiple lines and the lack of punctuation may be intentional (if such cases are rare in your subs, just replace all, manually removing 1-2 periods is faster than adding 10-20).
Replacing (?<!\.)\.(?=\\N[a-z]) with nothing will remove erroneous periods before the line break if the next part of the line isn't capitalized, but this will only work correctly if you set "Match case" flag. These last two expressions will only work after converting .srt to .ass with Aegisub.
Notepad++ allows you to replace multiple different values at once! To replace ALL above cases at once, replace ((?<!\.)\.\.(?!\.))|(\.\.\.\.)|((?<!,),\.)|(\?\.)|(\!\.)|(Dialogue.*[a-z]($))|((?<!\.)\.(?=\\N[a-z])) with (?1(...))(?2(...))(?3(,))(?4(?))(?5(!))(?6($&.))(?7()) (press "replace all" twice, because one of the expressions replaces the whole line and the changes made by other expressions will be lost for this line). If you want to build your own list, the syntax is Find: (FindA)|(FindB)|(FindC)... Replace: (?1(ReplaceA))(?2(ReplaceB))(?3(ReplaceC))... (in my experience, things start to get buggy if you go over 7 items, just use macros if you need more).
The settings should look like this:

Overall, after finetuning the settings for HnK's Cosmic Baton Girl Comet-san episodes, it was taking me about 1 hour of work to get the final version of subtitles. The chain is:
upscaling the 512x384 episode to 720p with nvenc on medium speed preset (36 seconds on 3060),
OCRing (2.5 minutes),
shifting end times by 1 frame forward and applying styles (30 seconds),
removing obvious bullshit (ch model produces lots with the above settings) and fixing glaring problems in Aegisub so I could have a smoother semi-livewatching session (5-10 minutes),
copypasting pre-timed pre-styled OP and ED lyrics with Notepad++ (I did it en masse, but for a single episode it would take under a minute),
roughly timing OP and ED to the original video (<1 minute),
semi-livewatching and editing (done by simply holding the right arrow button in Aegisub and releasing it whenever I spot a difference between the soft and hard subs (I temporarily changed the default style's vertical margin so the soft subs would appear right above the hard. 40-50 minutes)),
machine spellchecking the final script (1 minute),
loading the new, raw video in Aegisub and timing OP, ED and title cards (checking every line so they would align perfectly with the on-screen Japanese text, <5 minutes),
randomly checking if the timing for the subs in pre-OP, first half, second half and post-ED segments matches the audio of the raw file (sometimes it doesn't. 2 minutes), and that's it.
I'm OCRing it first and foremost for myself (I haven't watched the series before), so I'm also actually watching the episodes later. At that point, there's only rare punctuation problems left. If you aren't going to watch your release, just pay more attention at the semi-livewatching stage.

Below are the state-of-the-art settings I'm using to OCR Magical Idol Pastel Yumi, the subs are thankfully white (so I could raise brightness_threshold), and the video is already in 720. I'm getting almost no trash (apart from OP/ED) or glued together words, some punctuation marks aren't recognized properly but it can be mitigated by Notepad++ replacements I wrote above. The time to get the final version of subs is cut by a third, 40-45 minutes per episode is completely feasible.

1
2
3
4
5
6
7
from videocr import save_subtitles_to_file
if __name__ == '__main__':
    save_subtitles_to_file("v:\Video\Anime\Magical Idol Pastel Yumi [Web 720p]\[Dodgy] Magical Idol Pastel Yumi - 05 - Message From a Paper Airplane! [38B7B693].mp4", "v:\Video\Anime\Magical Idol Pastel Yumi [Web 720p]\[Dodgy] Magical Idol Pastel Yumi - 05 - Message From a Paper Airplane! [38B7B693].srt", lang='en', #time_start='19:18', time_end='20:25',
     sim_threshold=90, conf_threshold=0, use_fullframe=False,
     det_model_dir="C:/Users/admin/.paddleocr/whl/det/ch/ch_PP-OCRv4_det_infer", rec_model_dir="C:/Users/admin/.paddleocr/whl/rec/en/en_PP-OCRv4_rec_infer", use_gpu=True,
     brightness_threshold=235, similar_image_threshold=200, similar_pixel_threshold=220, frames_to_skip=0,
     crop_x=100, crop_y=560, crop_width=760, crop_height=160)
Edit Report
Pub: 28 Mar 2024 13:26 UTC
Edit: 02 Apr 2024 18:17 UTC
Views: 521