Mining on Steam Deck - Draft
if you've landed on this page, congratulations for discovering the link to something very much work in progress. Expect this to be barebones and change frequently till it's ready.
Introduction
This page documents how I'm immersing with Japanese games on the Steam Deck and adding vocab to Anki cards. If you're reading this, I assume you:
- are already familiar with the concept of sentence mining while learning Japanese
- have experience doing it on PC
- are just looking to see how to do it on the Steam Deck in portable mode with the default SteamOS
- don't mind using desktop mode and not gaming mode
If you're not familiar with the concept, it would be helpful to look at this introduction: http://learnjapanese.moe/guide/#mining
And if you're looking to set up mining workflow on PC (Windows/Linux) instead, refer to these guides, which I base my Deck mining workflow on:
rentry.co/mining
rentry.co/lazyXel
(Linux) https://github.com/eshrh/ames
(Linux) https://gist.github.com/xythh
You could for example, choose to run Windows instead (through VM, external drive, SD card, or unofficial dual boot), and use the Windows mining flow. You can skip to the controller config section if you choose this option.
A typical mining workflow will be:
- have Anki and a browser with texthook insertion page/Yomichan search page running in the background.
- Launch game, launch texthook program (Textractor or Agent), or launch an OCR program.
- Play the game, look up words you don't know with Yomichan in the browser, add to Anki cards.
- Optionally, take a screenshot of the game to save into Anki as additional context.
- (Not implemented, I can't find a good way to do it on the Deck yet) Optionally, replay the audio and record, save it to Anki for more context.
The next sections will go more into the details, and also some suggestions on mapping to controller buttons so you can do it all easily while holding the Steam Deck in your hands. (If you're using docked mode with keyboard+mouse, just use the desktop mining workflow.)
A note on the software used: I will only use software that can be installed without unlocking the read-only OS, so almost everything will be flatpaks or portable scripts.
Using a browser texthook page, Yomichan, and Anki
This is the same as desktop, really.
- pick a browser of your choice and install Yomichan extension (supports chromium-based browsers and Firefox)
- export your Yomichan profile from desktop, import it on the Deck to save setup effort.
- tweak Yomichan settings, I suggest:
- copy a profile, rename it to "Steam Deck"
- set the scan modifier key to none (show on hover)
- set the 2nd level scan modifier key to alt (useful for looking up words monolingual dictionaries)
- tweak the zoom level (I use 150% for Yomichan popup)
- use your preferred texthook page, eg. one of below. manga-ocr saves to clipboard instead of using websocket, pick accordingly.
- exSTATic - uses websocket, has fancy stat graphs
- Texthooker page using websocket
- Texthooker page using clipboard
- Yomichan search page (uses clipboard)
- zoom larger on the texthook page, around 125-175% for me usually
- install Anki from the Discover store, and install the Ankiconnect addon.
Launching Textractor
Steam VNs
Use SteamTinkerLaunch to launch Textractor or Agent in the same prefix
-
go more into the setup details
Non-Steam VNs
Use Lutris to:
- set up different wine prefixes (eg, with quartz, lavfilters, etc). Different games may need different prefixes. Refer to the VN wiki for more details
- install the VN
- launch Textractor with the same prefix
Launching Agent
Same as Textractor. I use the Windows version of Agent, there might be some performance hit. Theoretically you can install FRIDA server to run in proton + native Linux Agent instead, but I got an error and didn't continue pursuing it.
Get Agent here: https://github.com/0xDC00/agent
Watch the quickstart video linked to understand how to use it, you have to drag the crosshair icon to the game window, then select the script for the game, then click "attach".
You can see the list of games supported from the scripts list: https://github.com/0xDC00/scripts
Note: you can use Agent on supported emulator games, I tested out with my Emudeck Yuzu installation and it works the same as above.
OCR non-hookable games with manga-ocr
- follow Linux instructions to install manga-ocr. Skip the CUDA steps
- set up spectacle to capture rectangular region on hotkey trigger, save to a folder that manga-ocr will then scan
show my spectacle settings
- I use a shell script to start up both spectacle and manga-ocr
show my script. TODO: try rm the screenshot after ocr? Or user can clear the folder themselves
- click & drag for the screenshot, or drag the capture region over, and double click (mapped to a single right trackpad press in my config)
Mining
Map shortcuts to:
- mine word (Alt+E)
- save screenshot to card (see details below)
- replay Yomichan audio
How to save screenshots to Anki
- grab my shell script, save it somewhere (eg, my path !!!)
- make sure the script is executable (chmod/adjust file permissions in GUI !!!)
- I assume you installed Anki via flatpak; make sure Anki has access permission to the path it's saving the screenshots (instructions!!! flatseal)
- map the script to a keyboard shortcut (eg, xxx) - !!! need a keyboard, details
- map the keyboard shortcut to the Steam controller game config (need to do for each game)
- if you want to grab screenshot for non-Steam game, map to controller desktop config as well
Controller layout configs
- desktop
- VNs
- alt+tab
- advance text
- replay audio
- bring up backlog
- quick save, quick load
- other games using full controller: use the L4-5 and R4-5 back buttons to the fullest
- alt+tab for those hookable with agent
- additional mapping to grab screenshot for those using OCR
show my mappings for ATRI, Yakuza, and Sen no Kiseki. Links to the configs as well.
Deck-specific key learnings
- you'll want to use a physical keyboard and mouse during some of the setup (or you'll need to have loads of patience and experience typing while the on-screen keyboard obscures your view). You can use a cheap USB-C hub, use one of the remote desktop options, or connect bluetooth kb+m.
- the whole lookup and mining workflow is done in desktop mode. Anki doesn't work in gaming mode; I didn't try if just looking up words work.
- window mode for games are more stable than fullscreen since we have to alt+tab often. Use frameless window option if the game supports it.
- map right alt (not the left!) for alt+tab, for some reason some games ignore it when I use the left alt
- map alt & tab buttons separately for easier control over which window to switch to
- back buttons are good candidates to map for our window switching, OCR, and mining actions
- map single click and moving the cursor to separate trackpads for easier click and drag
- you can map double click by using extra command of type "start press" and put a delay
- disable mouse in the game if supported to avoid camera moving around or accidentally advancing text when grabbing a screenshot. Eg. Senpatcher for Sen no Kiseki
Targeted improvements (not implemented yet)
- save sentence audio to Anki
Extra: moving files to the Steam Deck
Options:
- SFTP (eg, WinSCP or FileZilla on Windows) - recommended as it was the easiest for me. example instructions
- KDE Connect (only for small files, it fails for files larger than a couple of GB for me)
- Warpinator (failed a few times on me)
Others I've heard of but didn't try:
- Syncthing
- sshfs
- deckMTP
- cloud services (eg, Google Drive, Dropbox, etc)
- USB drives