ACE-Step 1.5 LoRA Training Tutorial

Table of Contents

  1. Hardware Requirements
  2. Introduction and UI Selection
  3. Preparing Training Data
  4. Captioning
  5. Preprocessing
  6. Training Settings

Hardware Requirements

Side-Step is highly optimized and runs on hardware ranging from an RTX 3060 to an H100. Built-in presets configure these settings automatically based on your hardware.

VRAM Profiles

Profile VRAM Strategy
Comfortable 24 GB+ AdamW, Batch 2+, Rank 128
Standard 16-24 GB AdamW, Batch 1, Rank 64
Tight 12-16 GB AdamW8bit, Encoder offloading
Minimal 8-10 GB AdamW8bit, Offloading, Grad accumulation 8, Rank 16
  • Note: Gradient checkpointing is enabled by default, which reduces the baseline VRAM usage to ~7 GB before the optimizer state is even added.

Introduction and UI Selection

There are several options available for training ACE-Step 1.5 XL:

  • ACE-Step 1.5 Gradio: The original UI (found here). However, it tends to be unstable, slow, and less optimized than other methods.
  • AI Toolkit: A powerful alternative for those familiar with the ecosystem.
  • Side-Step: Found at koda-dernet/Side-Step.

This guide will focus on Side-Step, as it is currently the most lightweight, versatile, and robust version available.

Preparing Training Data

1. Sourcing Audio

For the best results, use high-quality audio files.

Quality Guidelines:

  • Aim for FLAC or 320 kbps MP3.
  • If the song is unavailable elsewhere, 192 kbps may be okay.

2. Upscaling & Enhancing (Optional)

If you are forced to use lower-quality files (such as those from YouTube), you can use the following tools to enhance musicality:

  • Matchering 2: An experimental tool that yields amazing results. It requires a "target" mix (a high-quality recording/FLAC from any artist in the same genre) to reference. This also works well for regular ACEStep 1.5 XL Turbo outputs.
  • Web Audio Mastering: Best for a quick fix-up when on a time crunch; the default settings are an effective way to upscale bad quality songs.

Captioning

ACE-Step 1.5 XL training requires a specific .json structure to organize metadata, captions, and lyrics. Below is an example of a single entry within the required dataset.json:

{
    "metadata": {
        "name": "miku_dataset",
        "custom_tag": "m1ku_d3c0",
        "tag_position": "prepend",
        "created_at": "2026-04-19T17:50:10.745225",
        "num_samples": 23,
        "all_instrumental": false,
        "genre_ratio": 0
    },
    "samples": [
        {
            "id": "f55b70b9",
            "audio_path": ".\\DECO 27\\01 ヴァンパイア.flac",
            "filename": "01 ヴァンパイア.flac",
            "caption": "Vibrant J-Pop track...",
            "genre": "J-Pop",
            "lyrics": "[Intro]\nあたしヴァンパイア...",
            "raw_lyrics": "",
            "formatted_lyrics": "[Intro]\nあたしヴァンパイア...",
            "bpm": 162,
            "keyscale": "F Minor",
            "timesignature": "4/4",
            "duration": 178,
            "language": "ja",
            "is_instrumental": false,
            "custom_tag": "m1ku_d3c0",
            "labeled": true,
            "prompt_override": null
        }
    ]
}

Automation Script (Gemini, Genius & BPM Detection)

To train ACE-Step 1.5 XL, you need a structured dataset.json. You can create this manually, or use Side-Step's built-in captioning options found within its settings (which use librosa for BPM detection, yielding strong results, though slightly less accurate than the Node analyzer based on my testing), or use a custom script. This tutorial focuses on the Custom Script method, which utilizes the Gemini API for high-level audio analysis/captioning and the Genius API/BeautifulSoup to fetch and sync lyrics automatically.

Note: The automation script has been updated. While earlier versions on this Rentry relied on VocalRemover for BPM, the current version utilizes a Node.js bridge to realtime-bpm-analyzer for highest accuracy, as VocalRemover results for BPM were highly inaccurate. KNOWN_SONGS is now handled automatically via the Genius API based on your ARTIST_HINT. For now, the key is still found with VocalRemover, as it tends to be more accurate.

1. Installation & Requirements

All files (Python script, .mjs script, and audio files) should be kept in the same directory.

Dependencies:

  1. Node.js: Ensure Node is installed on your system.
  2. Node Packages: In your training folder, run:
    npm install realtime-bpm-analyzer web-audio-api
    
  3. Python Packages:
    pip install librosa soundfile numpy google-genai requests beautifulsoup4 mutagen
    

2. BPM Detection Bridge (get_bpm.mjs)

Create a file named get_bpm.mjs in your script folder and paste the following:

// 1. Inject the Browser's Web Audio API into Node.js globally!
import 'web-audio-api/polyfill'; 

import fs from 'fs';
import { analyzeFullBuffer } from 'realtime-bpm-analyzer';

async function analyzeBPM() {
    const filePath = process.argv[2];
    if (!filePath) {
        console.error("No file provided");
        process.exit(1);
    }

    try {
        // Read the temporary WAV file Python created
        const fileBuffer = fs.readFileSync(filePath);

        // Use our new browser-like AudioContext to decode it
        const ctx = new AudioContext();
        const audioBuffer = await ctx.decodeAudioData(fileBuffer);

        // Run the analyzer!
        const tempos = await analyzeFullBuffer(audioBuffer);

        if (tempos && tempos.length > 0) {
            console.log("BPM_RESULT:" + Math.round(tempos[0].tempo));
        } else {
            console.log("BPM_RESULT:0");
        }
    } catch (err) {
        console.error("JS Error:", err);
        console.log("BPM_RESULT:0");
    }
}

analyzeBPM();

3. Main Captioning Script

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
import os
import json
import time
import uuid
import re
import sys
import shutil
import tempfile
import subprocess
import csv
from datetime import datetime
from difflib import SequenceMatcher

import requests
import librosa
import soundfile as sf
from google import genai
from google.genai import types

# --- CONFIGURATION ---
GEMINI_API_KEY = "..."       # Paste your Gemini key here
GENIUS_API_KEY = "..."       # Paste your Genius API token here

ARTIST_HINT = ""      # The artist's name on Genius E.G. DECO*27

DATASET_NAME = "miku_dataset"
CUSTOM_TAG = "m1ku_d3c0"
TAG_POSITION = "prepend"
GENRE_RATIO = 0
ALL_INSTRUMENTAL = False
MODEL_NAME = "gemini-3-flash-preview"
OUTPUT_JSON = "dataset.json"
CSV_METADATA_FILE = "key_bpm.csv"  # <-- Name of your https://vocalremover.org/key-bpm-finder CSV file (used for key detection)
AUDIO_EXTENSIONS = {".flac", ".wav", ".mp3", ".ogg"}

CAPTION_MAX_CHARS = 512

# Rate limit settings
SLEEP_BETWEEN_REQUESTS = 12
SLEEP_ON_429_DEFAULT = 60
SLEEP_ON_503 = 20

# --- KNOWN SONG LIST ---
# This will be populated automatically via Genius API based on the ARTIST_HINT
KNOWN_SONGS =[]

sys.stdout.reconfigure(encoding='utf-8')
client = genai.Client(api_key=GEMINI_API_KEY)

SYSTEM_PROMPT = """
You are an expert musicologist and dataset curator for the ACE-Step 1.5 AI model.
Analyze the audio and filename.

1. **filename**: Use for context.
2. **timesignature**: Detect time signature (e.g., "3/4").
3. **genre**: Specific style tag.
4. **language**: Detect the primary language of the vocals.
   - Use ISO 639-1 two-letter codes (e.g., "en", "ja", "es", "fr").
   - If instrumental or language is completely unrecognizable, use "unknown".
5. **caption**: RICH natural language description (Instruments, Emotion, Timbre, Mood).
   - IMPORTANT: The caption MUST be 512 characters or fewer. Be concise but descriptive.
   - If human sounds (shouts, whistles) exist, mention them.

IMPORTANT: Output STRICT valid JSON only. Do not add markdown blocks.
Output RAW JSON: {"caption": "...", "genre": "...", "timesignature": "...", "language": "ja"}
"""

def safe_print(text):
    try:
        print(text)
    except Exception:
        try:
            print(text.encode('utf-8', 'replace').decode('utf-8', 'replace'))
        except Exception:
            pass


def clean_json_text(text):
    if not text:
        return ""
    text = text.strip()
    if text.startswith("```json"):
        text = text[7:]
    if text.endswith("```"):
        text = text[:-3]
    match = re.search(r'\{[\s\S]*\}', text)
    if match:
        text = match.group(0)
    return text.strip()


def load_csv_metadata(csv_path):
    """Load Key mapping from VocalRemover's CSV file."""
    metadata = {}
    if not os.path.exists(csv_path):
        safe_print(f"Warning: CSV file '{csv_path}' not found. Keys will be left empty.")
        return metadata

    try:
        with open(csv_path, mode="r", encoding="utf-8-sig") as f:
            reader = csv.DictReader(f)
            for row in reader:
                file_name = row.get("File", "").strip()
                if file_name:
                    # Parse Key
                    key_str = row.get("Key", "").strip()
                    metadata[file_name] = {
                        "key": key_str
                    }
    except Exception as e:
        safe_print(f"Error reading CSV '{csv_path}': {e}")

    return metadata


# ---------------------------------------------------------------------------
# HYBRID BPM DETECTION (Python -> Node.js)
# ---------------------------------------------------------------------------

def get_bpm_via_javascript(filepath):
    """
    Decodes the audio using Python (librosa), passes a temporary WAV 
    to Node.js for realtime-bpm-analyzer processing, and returns the BPM.
    """
    safe_print("  -> [Python] Decoding audio to a safe temporary WAV format...")
    temp_wav = os.path.join(tempfile.gettempdir(), f"temp_bpm_track_{uuid.uuid4().hex[:8]}.wav")

    try:
        # Load downsampled audio to speed up processing
        y, sr = librosa.load(filepath, sr=44100, mono=True)
        sf.write(temp_wav, y, sr)

        # Ensure we are calling the script in the same directory
        script_dir = os.path.dirname(os.path.abspath(__file__)) if '__file__' in globals() else os.getcwd()
        js_path = os.path.join(script_dir, "get_bpm.mjs")

        if not os.path.exists(js_path):
            safe_print(f"  -> [Error] '{js_path}' not found! Returning BPM 0.")
            return 0

        safe_print("  ->[Node.js] Running realtime-bpm-analyzer...")
        result = subprocess.run(["node", js_path, temp_wav],
            capture_output=True,
            text=True,
            check=False
        )

        if result.stderr:
            safe_print(f"  -> JS Warning: {result.stderr.strip()}")

        output = result.stdout.strip()
        match = re.search(r'BPM_RESULT:(\d+)', output)
        if match:
            return int(match.group(1))

        return 0

    except Exception as e:
        safe_print(f"  -> Error bridging to Node.js: {e}")
        return 0

    finally:
        if os.path.exists(temp_wav):
            try:
                os.remove(temp_wav)
            except Exception:
                pass


def try_get_duration(filepath):
    try:
        from mutagen import File as MutagenFile
        audio = MutagenFile(filepath)
        if audio is not None and audio.info is not None:
            return int(audio.info.length)
    except ImportError:
        pass
    except Exception:
        pass
    return 0


def get_mime_type(filename):
    ext = os.path.splitext(filename)[1].lower()
    mime_map = {
        ".flac": "audio/flac",
        ".wav": "audio/wav",
        ".mp3": "audio/mpeg",
        ".ogg": "audio/ogg",
    }
    return mime_map.get(ext, "audio/flac")


def parse_retry_delay(error_str):
    try:
        match = re.search(r'retry[_ ]?(?:in|Delay)[\'"\s:]*(\d+)', error_str, re.IGNORECASE)
        if match:
            return int(match.group(1)) + 5
    except Exception:
        pass
    return SLEEP_ON_429_DEFAULT


def is_rate_limit_error(e):
    error_str = str(e)
    return ("429" in error_str or
            "RESOURCE_EXHAUSTED" in error_str or
            "rate limit" in error_str.lower() or
            "quota exceeded" in error_str.lower())


def is_server_error(e):
    error_str = str(e)
    return ("503" in error_str or
            "UNAVAILABLE" in error_str or
            "500" in error_str or
            "INTERNAL" in error_str)


def upload_with_temp_copy(full_audio_path):
    try:
        full_audio_path.encode('ascii')
        is_ascii = True
    except UnicodeEncodeError:
        is_ascii = False

    mime = get_mime_type(full_audio_path)

    if is_ascii:
        upload_result = client.files.upload(
            file=full_audio_path,
            config=types.UploadFileConfig(mime_type=mime)
        )
        return upload_result, None
    else:
        ext = os.path.splitext(full_audio_path)[1]
        temp_dir = tempfile.gettempdir()
        temp_name = f"gemini_upload_{uuid.uuid4().hex[:12]}{ext}"
        temp_path = os.path.join(temp_dir, temp_name)

        safe_print(f"  -> Non-ASCII path. Temp copy: {temp_name}")
        shutil.copy2(full_audio_path, temp_path)

        upload_result = client.files.upload(
            file=temp_path,
            config=types.UploadFileConfig(mime_type=mime)
        )
        return upload_result, temp_path


def scan_audio_files(directory):
    audio_files =[]
    for filename in os.listdir(directory):
        _, ext = os.path.splitext(filename)
        if ext.lower() in AUDIO_EXTENSIONS:
            audio_files.append(filename)
    audio_files.sort()
    return audio_files


# ---------------------------------------------------------------------------
# GENIUS LYRICS HELPERS
# ---------------------------------------------------------------------------

def fetch_known_songs(artist_hint: str):
    """Dynamically fetches all top songs for the given artist from Genius API."""
    global KNOWN_SONGS
    safe_print(f"\nFetching discography for '{artist_hint}' from Genius API...")

    if not GENIUS_API_KEY or GENIUS_API_KEY == "...":
        safe_print("  -> Warning: GENIUS_API_KEY not set. Skipping song fetch.\n")
        return

    headers = {"Authorization": f"Bearer {GENIUS_API_KEY}"}

    # 1. Get Artist ID from Genius
    try:
        resp = requests.get(
            "https://api.genius.com/search",
            headers=headers,
            params={"q": artist_hint},
            timeout=10
        )
        if resp.status_code != 200:
            safe_print(f"  -> Failed to search artist (HTTP {resp.status_code}).")
            return

        hits = resp.json().get("response", {}).get("hits",[])
        artist_id = None

        # Check exactly for the artist name
        for hit in hits:
            artist_info = hit.get("result", {}).get("primary_artist", {})
            if artist_hint.lower() in artist_info.get("name", "").lower():
                artist_id = artist_info.get("id")
                break

        # Fallback if strict string matching fails but hits exist
        if not artist_id and hits:
            artist_id = hits[0].get("result", {}).get("primary_artist", {}).get("id")

        if not artist_id:
            safe_print("  -> Could not determine Genius Artist ID. Skipping known songs fetch.\n")
            return

        safe_print(f"  -> Found Artist ID: {artist_id}. Pulling discography...")

        # 2. Paginate Songs to build the fuzzy-matching list
        page = 1
        fetched = 0
        while True:
            resp = requests.get(
                f"https://api.genius.com/artists/{artist_id}/songs",
                headers=headers,
                params={"per_page": 50, "page": page, "sort": "popularity"},
                timeout=10
            )
            if resp.status_code != 200:
                break

            data = resp.json().get("response", {})
            songs_data = data.get("songs",[])
            if not songs_data:
                break

            for song in songs_data:
                title = song.get("title", "")
                primary_artist = song.get("primary_artist", {}).get("name", "")
                KNOWN_SONGS.append((title, primary_artist))
                fetched += 1

            next_page = data.get("next_page")
            if not next_page:
                break
            page = next_page

        safe_print(f"  -> Successfully loaded {fetched} known songs into memory.\n")

    except Exception as e:
        safe_print(f"  -> Error fetching known songs: {e}\n")


def normalize_title(title: str) -> str:
    title = re.sub(r'^\d+[-–\s]+\d*[-–\s]*', '', title)
    title = re.sub(r'^\d+\s+', '', title)
    title = re.sub(r'\.(flac|wav|mp3|ogg)$', '', title, flags=re.IGNORECASE)
    title = re.sub(r'\s+', ' ', title).strip()
    return title


def extract_japanese(text: str) -> str:
    return ''.join(c for c in text if '\u3000' <= c <= '\u9fff' or '\uff00' <= c <= '\uffef')


def similarity(a: str, b: str) -> float:
    return SequenceMatcher(None, a.lower(), b.lower()).ratio()


def find_best_known_match(filename: str):
    norm = normalize_title(filename)
    jp_norm = extract_japanese(norm)

    best_score = 0.0
    best_entry = (None, None)

    for display_title, artist in KNOWN_SONGS:
        score = similarity(norm, display_title)
        jp_known = extract_japanese(display_title)
        if jp_norm and jp_known:
            jp_score = similarity(jp_norm, jp_known)
            score = max(score, jp_score)

        if score > best_score:
            best_score = score
            best_entry = (display_title, artist)

    if best_score >= 0.35:
        return best_entry, best_score
    return (None, None), best_score


def genius_search(query: str, artist_hint: str = ARTIST_HINT) -> dict | None:
    headers = {"Authorization": f"Bearer {GENIUS_API_KEY}"}
    params = {"q": query}
    try:
        resp = requests.get(
            "https://api.genius.com/search",
            headers=headers,
            params=params,
            timeout=10
        )
        if resp.status_code != 200:
            safe_print(f"  -> Genius search HTTP {resp.status_code}")
            return None
        hits = resp.json().get("response", {}).get("hits",[])
        if not hits:
            return None

        best_hit = None
        best_score = 0.0
        for hit in hits:
            result = hit.get("result", {})
            hit_title = result.get("title", "")
            hit_artist = result.get("primary_artist", {}).get("name", "")

            score = similarity(query, hit_title)
            if artist_hint.lower() in hit_artist.lower():
                score += 0.2

            if score > best_score:
                best_score = score
                best_hit = result

        if best_score >= 0.3:
            return best_hit
        return None
    except Exception as e:
        safe_print(f"  -> Genius search error: {e}")
        return None


def fetch_genius_lyrics(song_url: str) -> str:
    try:
        from bs4 import BeautifulSoup
    except ImportError:
        safe_print("  -> beautifulsoup4 not installed. Run: pip install beautifulsoup4")
        return ""

    try:
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                          "AppleWebKit/537.36 (KHTML, like Gecko) "
                          "Chrome/120.0.0.0 Safari/537.36"
        }
        resp = requests.get(song_url, headers=headers, timeout=15)
        if resp.status_code != 200:
            safe_print(f"  -> Genius page HTTP {resp.status_code}")
            return ""

        soup = BeautifulSoup(resp.text, "html.parser")
        containers = soup.find_all("div", {"data-lyrics-container": "true"})
        if not containers:
            safe_print("  -> No lyrics container found on Genius page.")
            return ""

        lines =[]
        for container in containers:
            for tag in container.find_all(["br", "a", "span", "i", "b", "em"]):
                if tag.name == "br":
                    tag.replace_with("\n")
            text = container.get_text(separator="\n")
            lines.append(text)

        raw = "\n".join(lines)
        raw = re.sub(r'\s*(\[[A-Za-z][A-Za-z0-9\s-]*\])', r'\n\n\1', raw)
        raw = re.sub(r'\n{3,}', '\n\n', raw).strip()
        return raw

    except Exception as e:
        safe_print(f"  -> Lyrics fetch error: {e}")
        return ""


def get_lyrics_for_file(filename: str) -> tuple[str, str]:
    norm = normalize_title(filename)
    safe_print(f"  -> Normalized title: '{norm}'")

    (matched_title, matched_artist), score = find_best_known_match(filename)
    if matched_title:
        safe_print(f"  -> Best known match: '{matched_title}' (score={score:.2f})")
        search_query = f"{matched_title} {matched_artist}"
    else:
        safe_print(f"  -> No known match found (best score={score:.2f}), using raw filename")
        search_query = norm

    hit = genius_search(search_query, artist_hint=ARTIST_HINT)

    if not hit:
        jp = extract_japanese(norm)
        if jp:
            safe_print(f"  -> Retrying Genius with Japanese only: '{jp}'")
            hit = genius_search(jp, artist_hint=ARTIST_HINT)

    if not hit:
        safe_print("  -> No Genius match found. Lyrics will be empty.")
        return "", ""

    song_url = hit.get("url", "")
    hit_title = hit.get("title", "")
    hit_artist = hit.get("primary_artist", {}).get("name", "")
    safe_print(f"  -> Genius match: '{hit_title}' by {hit_artist}")
    safe_print(f"  -> URL: {song_url}")

    raw_lyrics = fetch_genius_lyrics(song_url)
    if not raw_lyrics:
        return "", ""

    formatted = raw_lyrics.strip()
    return raw_lyrics, formatted


# ---------------------------------------------------------------------------
# DATASET HELPERS
# ---------------------------------------------------------------------------

def create_empty_sample(filename, directory):
    rel_path = os.path.join(".\\" + os.path.basename(directory), filename)
    full_path = os.path.join(directory, filename)
    duration = try_get_duration(full_path)

    return {
        "id": uuid.uuid4().hex[:8],
        "audio_path": rel_path,
        "filename": filename,
        "caption": "",
        "genre": "",
        "lyrics": "",
        "raw_lyrics": "",
        "formatted_lyrics": "",
        "bpm": 0,
        "keyscale": "",
        "timesignature": "",
        "duration": duration,
        "language": "unknown",
        "is_instrumental": ALL_INSTRUMENTAL,
        "custom_tag": CUSTOM_TAG,
        "labeled": False,
        "prompt_override": None
    }


def create_dataset_skeleton(directory, audio_files):
    samples =[create_empty_sample(f, directory) for f in audio_files]
    dataset = {
        "metadata": {
            "name": DATASET_NAME,
            "custom_tag": CUSTOM_TAG,
            "tag_position": TAG_POSITION,
            "created_at": datetime.utcnow().isoformat(),
            "num_samples": len(samples),
            "all_instrumental": ALL_INSTRUMENTAL,
            "genre_ratio": GENRE_RATIO,
        },
        "samples": samples
    }
    return dataset


def save_dataset(data):
    with open(OUTPUT_JSON, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=4, ensure_ascii=False)


# ---------------------------------------------------------------------------
# CORE PROCESSING
# ---------------------------------------------------------------------------

def process_sample(sample, directory, data, csv_metadata):
    filename = sample['filename']
    full_audio_path = os.path.join(directory, filename)

    if not os.path.exists(full_audio_path):
        safe_print(f"  Skipped: File not found")
        return False

    # --- Step 1: Automated BPM (Node.js) & Manual Key (CSV) ---
    safe_print(f"  [1/3] Extracting BPM via Node.js realtime-bpm-analyzer...")
    bpm = get_bpm_via_javascript(full_audio_path)
    sample['bpm'] = bpm

    csv_info = csv_metadata.get(filename, {})
    sample['keyscale'] = csv_info.get('key', "")

    # --- Step 2: Fetch lyrics from Genius ---
    safe_print("  [2/3] Fetching lyrics from Genius...")
    _, formatted_lyrics = get_lyrics_for_file(filename)

    # --- Step 3: Gemini for caption/genre/etc. ---
    max_retries = 3
    temp_file_path = None

    for attempt in range(max_retries):
        try:
            safe_print(f"[3/3] Uploading to Gemini (Attempt {attempt + 1}/{max_retries})...")
            upload_result, temp_file_path = upload_with_temp_copy(full_audio_path)

            wait_count = 0
            while upload_result.state.name == "PROCESSING":
                time.sleep(2)
                upload_result = client.files.get(name=upload_result.name)
                wait_count += 1
                if wait_count > 60:
                    raise TimeoutError("Upload processing timed out")

            if upload_result.state.name == "FAILED":
                safe_print("  -> Audio processing failed on Google server.")
                return False

            prompt = f"{SYSTEM_PROMPT}\n\nfilename: {filename}"

            response = client.models.generate_content(
                model=MODEL_NAME,
                contents=[
                    types.Content(
                        role="user",
                        parts=[
                            types.Part.from_uri(
                                file_uri=upload_result.uri,
                                mime_type=upload_result.mime_type),
                            types.Part.from_text(text=prompt)
                        ]
                    )
                ],
                config=types.GenerateContentConfig(
                    response_mime_type="application/json"
                )
            )

            if not response.text:
                safe_print("  -> Error: Model returned empty response.")
                raise ValueError("Empty response text")

            cleaned_text = clean_json_text(response.text)

            try:
                new_meta = json.loads(cleaned_text)
            except json.JSONDecodeError as je:
                safe_print(f"  -> JSON Parse Error: {je}")
                safe_print(f"  -> Raw text start: {cleaned_text[:80]}...")
                return False

            if isinstance(new_meta, list):
                if len(new_meta) == 0:
                    raise ValueError("Empty list from API")
                new_meta = new_meta[0]

            # Enforce 512-char caption limit
            caption = new_meta.get('caption', "")
            if len(caption) > CAPTION_MAX_CHARS:
                caption = caption[:CAPTION_MAX_CHARS].rsplit(' ', 1)[0]  # trim at word boundary

            sample['caption'] = caption
            sample['genre'] = new_meta.get('genre', "")
            sample['timesignature'] = new_meta.get('timesignature', "")

            # LYRICS ASSIGNMENT
            sample['lyrics'] = formatted_lyrics
            sample['raw_lyrics'] = ""
            sample['formatted_lyrics'] = formatted_lyrics 

            sample['prompt_override'] = None

            detected_lang = new_meta.get('language', 'unknown')
            if detected_lang and detected_lang.lower() != "unknown":
                sample['language'] = detected_lang.lower()[:2]
            else:
                sample['language'] = "unknown"

            if sample['duration'] == 0:
                sample['duration'] = try_get_duration(full_audio_path)

            sample['labeled'] = True

            safe_print(f"  -> Done! (Lang: {sample['language']}, BPM: {sample['bpm']}, Key: {sample['keyscale']}, "
                       f"Caption: {len(sample['caption'])} chars, "
                       f"Lyrics: {'yes' if sample['lyrics'] else 'none'})")

            save_dataset(data)
            return True

        except KeyboardInterrupt:
            safe_print("\n\n*** Interrupted by user. Saving... ***")
            save_dataset(data)
            sys.exit(0)

        except Exception as e:
            if is_rate_limit_error(e):
                retry_seconds = parse_retry_delay(str(e))
                safe_print(f"  -> Rate Limit. Sleeping {retry_seconds}s...")
                time.sleep(retry_seconds)
            elif is_server_error(e):
                safe_print(f"  -> Server Error (503). Sleeping {SLEEP_ON_503}s...")
                time.sleep(SLEEP_ON_503)
            else:
                safe_print(f"  -> FULL ERROR: {type(e).__name__}: {e}")
                break

        finally:
            if temp_file_path and os.path.exists(temp_file_path):
                try:
                    os.remove(temp_file_path)
                except Exception:
                    pass
                temp_file_path = None

    return False


def process_dataset(data, directory, csv_metadata):
    total = len(data['samples'])
    safe_print(f"Processing {total} samples...\n")

    processed_count = 0
    skipped_count = 0

    for i, sample in enumerate(data['samples']):
        if sample.get('labeled', False) and sample.get('language', 'unknown') != "unknown":
            skipped_count += 1
            continue

        safe_print(f"\n[{i + 1}/{total}] Processing: {sample['filename']}")

        success = process_sample(sample, directory, data, csv_metadata)

        if success:
            processed_count += 1
            time.sleep(SLEEP_BETWEEN_REQUESTS)
        else:
            safe_print("  -> Failed. Moving to next.")

    safe_print(f"\n{'=' * 50}")
    safe_print(f"Already done: {skipped_count}")
    safe_print(f"Newly processed: {processed_count}")
    safe_print(f"Total: {total}")
    safe_print(f"{'=' * 50}")


def main():
    directory = os.getcwd()
    safe_print(f"Working directory: {directory}")
    safe_print(f"Dataset: {DATASET_NAME} | Tag: {CUSTOM_TAG} | Caption max: {CAPTION_MAX_CHARS} chars\n")

    # Load keys from VocalRemover CSV
    csv_metadata = load_csv_metadata(CSV_METADATA_FILE)

    # PRE-LOAD DISC FROM GENIUS API
    if ARTIST_HINT:
        fetch_known_songs(ARTIST_HINT)

    if os.path.exists(OUTPUT_JSON):
        safe_print(f"Found existing '{OUTPUT_JSON}' — resuming.\n")
        try:
            with open(OUTPUT_JSON, "r", encoding="utf-8") as f:
                data = json.load(f)
        except json.JSONDecodeError:
            safe_print("ERROR: dataset.json is corrupted. Cannot resume.")
            return

        for s in data['samples']:
            if 'raw_lyrics' not in s:
                s['raw_lyrics'] = ""
            if 'formatted_lyrics' not in s:
                s['formatted_lyrics'] = s.get('lyrics', "")
            if 'prompt_override' not in s:
                s['prompt_override'] = None

        existing_filenames = {s['filename'] for s in data['samples']}
        all_audio = scan_audio_files(directory)
        new_files =[f for f in all_audio if f not in existing_filenames]

        if new_files:
            safe_print(f"Found {len(new_files)} new files. Adding them.")
            for nf in new_files:
                data['samples'].append(create_empty_sample(nf, directory))
            data['metadata']['num_samples'] = len(data['samples'])
            save_dataset(data)
    else:
        audio_files = scan_audio_files(directory)
        if not audio_files:
            safe_print("No audio files found.")
            return
        safe_print(f"Found {len(audio_files)} audio files. Creating dataset.\n")
        data = create_dataset_skeleton(directory, audio_files)
        save_dataset(data)

    process_dataset(data, directory, csv_metadata)
    safe_print("\nDONE.")


if __name__ == "__main__":
    main()

Important Captioning Notes

  • Gemini API Usage: This script uses the Gemini API to generate captions. While highly capable, captions may not be 100% accurate and often require manual tuning for the best results.
  • Artist LoRAs: If you are training a specific artist LoRA, modify the SYSTEM_PROMPT to explicitly include the artist's name in the description to reinforce the identity.
  • Language Tags: ACE-Step uses two-letter ISO tags. Note that Japanese is ja (not jp). Ensure your prompt explicitly instructs the model to use the correct tag for your target language to prevent training errors.
  • Local Alternatives: If you have the hardware, you can caption locally to avoid API limits.
    • acestep-captioner: The original model used to train the base ACE-Step model.
    • MOSS-Music-8B-Thinking: A newer, powerful alternative that uses "thinking" tokens to reason about the music for highly detailed captions.

Tagging and Metadata Configuration

When configuring your dataset.json or the automation script, pay close attention to how tags and lyrics are handled:

Custom Tag Options (tag_position)

The custom_tag (e.g., m1ku_d3c0) acts as the trigger word for your LoRA. You have three positioning options:

  • prepend (Recommended): Places the tag at the start of the caption (e.g., "m1ku_d3c0 Vibrant J-Pop..."). This preserves the descriptive caption and is the standard way to prompt the model.
  • append: Places the tag at the end of the caption.
  • replace: Replaces the entire caption with only the custom tag. This is generally not recommended as it loses descriptive detail.

Dataset Settings

  • all_instrumental: Keep this false unless your dataset consists entirely of instrumental tracks where the lyrics field is just [instrumental].
  • genre_ratio: Denotes the percentage of captions to be replaced by the genre tag during training. For example, 20 means 20% of your samples will use only the genre as the prompt. 0 is considered optimal, as it ensures the model sees the full detailed caption every time, and genre info is usually included in the caption anyway.

Captioning Tips

  • Length: Shorter is often better. Keep captions within the 512 character limit.
  • Style: While the script generates natural language, "Suno-style" (comma-separated tags) may perform well as it mimics how users naturally prompt the model.

Lyrics and Audio Metadata

Quality Control for Lyrics

The script uses the Genius API to pull lyrics, but manual verification is essential.

  • Cleanup: Genius often includes text before [Intro] or after [Outro], instruction tags in various languages, or may omit section tags entirely.
  • Missing Lyrics: If Genius does not have the lyrics, you should manually find them on other lyric hosting sites. Alternatively, you can use Gemini to transcribe the audio, but do so with extreme caution. The Gemini API often contains inaccuracies in transcriptions, and poor lyric quality directly hurts model performance.
  • Consistency: Ensure formatting (e.g., [Chorus], [Verse]) is consistent across your entire dataset. This should match how you intend to prompt the model later. Use an LLM to batch-reformat messy lyrics into a clean, consistent structure.
  • Redundancy: ACE-Step includes several fields (lyrics, raw_lyrics, formatted_lyrics). raw_lyrics can stay empty; providing the same text in lyrics and formatted_lyrics is sufficient.

Accuracy: BPM and Key

  • BPM Detection: This script uses realtime-bpm-analyzer via Node.js for high accuracy.
  • Key Detection: LLMs are inaccurate at identifying Key Scale. Use vocalremover.org's Key/BPM Finder, export the results as a .csv, and point the script to CSV_METADATA_FILE. The script will pull the Key from the CSV while using the Node Bridge for the BPM.

Genius API & Artist Hint

ARTIST_HINT is the complete artist name (including all spaces) found on the Genius website.
The KNOWN_SONGS list is now automatically populated by the script via the Genius API using your provided ARTIST_HINT. This ensures the fuzzy-matching engine can find the correct lyrics even if your filenames are stylized.

Preprocessing

Once your dataset.json and audio files are ready, you must preprocess them using Side-Step.

  1. Standard Preprocessing: Launch Side-Step and select the preprocessing options from the menu. When configuring the settings, ensure you set the following:
    • Model Selection: Always choose the same model you intend to train on. Selecting the Base XL model is highly recommended here.
    • Normalization: Choose Peak (-1.0 dBFS, no extra deps, matches ACE-Step). This levels your audio correctly for the model and prevents clipping.
  2. Preprocessing++ (Experimental): This is an advanced version of the pipeline that may yield stronger results, though it is currently experimental and less widely tested.
  3. Dataset Size: ACE-Step 1.5 XL is remarkably efficient with small datasets. Aim for 10–25 high-quality songs. Results vary by artist, so feel free to experiment with larger datasets if the initial results don't capture the nuances correctly.

Training Settings

Training ACE-Step 1.5 XL requires careful balance. Because memory consumption scales quadratically with song length, optimization is key.

Core Configuration

  • Model Selection: Always select the Base XL model in the Side-Step menu. Even if you intend to use the LoRA with the Turbo model later, you must train on the Base XL model.
  • Optimizer: Use AdamW. Avoid Prodigy, as it is unstable for the smaller datasets typically used in ACE-Step training.
  • Learning Rate (LR) & Rank:
    • Standard: 0.0001 LR with a rank of 64/128.
    • High Rank: 128/256 is possible, but you may need to lower the LR to avoid overfitting.
    • Alternative: Some users report success with a lower LR (0.00005) over long training runs (2,000–4,000 epochs) on non-XL versions.
  • Epochs & Checkpoints: Training is usually "done" between 200–800 epochs. Save checkpoints every 25 epochs; specific epochs can be noisy, so having a library of checkpoints allows you to pick the cleanest one.
    • Check your results every 50 epochs once you pass the 200 mark.
    • Generally, the "best" checkpoint is excellent, but you should test both that and regular epoch checkpoints to find the optimal version.
  • Determining Convergence: For a checkpoint to be "good," you are looking for the loss to have stabilized for several epochs. Keep in mind that since this is a diffusion model, a low loss (e.g., 0.1) does not necessarily mean convergence.
    • A slightly "overbaked" model—one that can reproduce some of the training songs given the exact same caption, metadata, and lyrics—is actually a good sign. It indicates the model has converged well, provided it can still generalize effectively when presented with different captions and lyrics.
  • Testing: Once the model is trained, it should be tested without the LM (Language Model) and with DiT only to achieve the best results.
    • Important Update: The previous recommendation to use the plain Turbo XL model for this step has changed. It is now highly recommended to use the merge-base-turbo-xl-ta-0.5.gguf (an equal blend of the Base XL and Turbo XL models). This merge significantly outperforms the plain Turbo XL model when using DiT-only prompting by enhancing three key areas: audio quality, song composition, and the overall creativity and capability of the model.

Performance & Optimization

  • Batch Size: A total batch size of 4 is the gold standard. On a 24 GB card (like a 3090/4090), set Batch Size 2 and Grad Accumulation 2. If you must use a batch size of 1 to fit VRAM, be aware that results may be less stable.
  • The "Chunk Duration" Trick: In Side-Step's advanced options, set Chunk Duration to 60 seconds.
    • This extracts a random window of the song during each pass rather than processing the full file.
    • Impact: This can reduce training time from 20+ hours down to just a few hours or minutes, as it bypasses the quadratic memory scaling.
    • Trade-off: Some users report slightly more repetition in generated outputs, though others have successfully replicated training data even with this setting enabled.

Cloud Training

ACE-Step 1.5 XL has very steep hardware requirements. Work is being done to optimize this. If you do not have a top-tier GPU (like a 5090 or 4090), consider using a cloud provider:

  • RunPod: Reliable paid GPU instances.
  • Modal: Offers $30 free monthly credits, which is often enough for a few LoRA training runs.
Edit

Pub: 01 May 2026 21:47 UTC

Edit: 08 May 2026 15:17 UTC

Views: 183