ACE-Step 1.5 LoRA Training Tutorial
Table of Contents
- Hardware Requirements
- Introduction and UI Selection
- Preparing Training Data
- Captioning
- Preprocessing
- Training Settings
Hardware Requirements
Side-Step is highly optimized and runs on hardware ranging from an RTX 3060 to an H100. Built-in presets configure these settings automatically based on your hardware.
VRAM Profiles
| Profile | VRAM | Strategy |
|---|---|---|
| Comfortable | 24 GB+ | AdamW, Batch 2+, Rank 128 |
| Standard | 16-24 GB | AdamW, Batch 1, Rank 64 |
| Tight | 12-16 GB | AdamW8bit, Encoder offloading |
| Minimal | 8-10 GB | AdamW8bit, Offloading, Grad accumulation 8, Rank 16 |
- Note: Gradient checkpointing is enabled by default, which reduces the baseline VRAM usage to ~7 GB before the optimizer state is even added.
Introduction and UI Selection
There are several options available for training ACE-Step 1.5 XL:
- ACE-Step 1.5 Gradio: The original UI (found here). However, it tends to be unstable, slow, and less optimized than other methods.
- AI Toolkit: A powerful alternative for those familiar with the ecosystem.
- Side-Step: Found at koda-dernet/Side-Step.
This guide will focus on Side-Step, as it is currently the most lightweight, versatile, and robust version available.
Preparing Training Data
1. Sourcing Audio
For the best results, use high-quality audio files.
- Free Sources: us.doubledouble.top, lucida.to, tidal.qqdl.site
- Paid/Advanced tools: qobuz-dl
Quality Guidelines:
- Aim for FLAC or 320 kbps MP3.
- If the song is unavailable elsewhere, 192 kbps may be okay.
2. Upscaling & Enhancing (Optional)
If you are forced to use lower-quality files (such as those from YouTube), you can use the following tools to enhance musicality:
- Matchering 2: An experimental tool that yields amazing results. It requires a "target" mix (a high-quality recording/FLAC from any artist in the same genre) to reference. This also works well for regular ACEStep 1.5 XL Turbo outputs.
- Web Audio Mastering: Best for a quick fix-up when on a time crunch; the default settings are an effective way to upscale bad quality songs.
Captioning
ACE-Step 1.5 XL training requires a specific .json structure to organize metadata, captions, and lyrics. Below is an example of a single entry within the required dataset.json:
Automation Script (Gemini, Genius & BPM Detection)
To train ACE-Step 1.5 XL, you need a structured dataset.json. You can create this manually, or use Side-Step's built-in captioning options found within its settings (which use librosa for BPM detection, yielding strong results, though slightly less accurate than the Node analyzer based on my testing), or use a custom script. This tutorial focuses on the Custom Script method, which utilizes the Gemini API for high-level audio analysis/captioning and the Genius API/BeautifulSoup to fetch and sync lyrics automatically.
Note: The automation script has been updated. While earlier versions on this Rentry relied on VocalRemover for BPM, the current version utilizes a Node.js bridge to realtime-bpm-analyzer for highest accuracy, as VocalRemover results for BPM were highly inaccurate. KNOWN_SONGS is now handled automatically via the Genius API based on your ARTIST_HINT. For now, the key is still found with VocalRemover, as it tends to be more accurate.
1. Installation & Requirements
All files (Python script, .mjs script, and audio files) should be kept in the same directory.
Dependencies:
- Node.js: Ensure Node is installed on your system.
- Node Packages: In your training folder, run:
- Python Packages:
2. BPM Detection Bridge (get_bpm.mjs)
Create a file named get_bpm.mjs in your script folder and paste the following:
3. Main Captioning Script
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 | import os
import json
import time
import uuid
import re
import sys
import shutil
import tempfile
import subprocess
import csv
from datetime import datetime
from difflib import SequenceMatcher
import requests
import librosa
import soundfile as sf
from google import genai
from google.genai import types
# --- CONFIGURATION ---
GEMINI_API_KEY = "..." # Paste your Gemini key here
GENIUS_API_KEY = "..." # Paste your Genius API token here
ARTIST_HINT = "" # The artist's name on Genius E.G. DECO*27
DATASET_NAME = "miku_dataset"
CUSTOM_TAG = "m1ku_d3c0"
TAG_POSITION = "prepend"
GENRE_RATIO = 0
ALL_INSTRUMENTAL = False
MODEL_NAME = "gemini-3-flash-preview"
OUTPUT_JSON = "dataset.json"
CSV_METADATA_FILE = "key_bpm.csv" # <-- Name of your https://vocalremover.org/key-bpm-finder CSV file (used for key detection)
AUDIO_EXTENSIONS = {".flac", ".wav", ".mp3", ".ogg"}
CAPTION_MAX_CHARS = 512
# Rate limit settings
SLEEP_BETWEEN_REQUESTS = 12
SLEEP_ON_429_DEFAULT = 60
SLEEP_ON_503 = 20
# --- KNOWN SONG LIST ---
# This will be populated automatically via Genius API based on the ARTIST_HINT
KNOWN_SONGS =[]
sys.stdout.reconfigure(encoding='utf-8')
client = genai.Client(api_key=GEMINI_API_KEY)
SYSTEM_PROMPT = """
You are an expert musicologist and dataset curator for the ACE-Step 1.5 AI model.
Analyze the audio and filename.
1. **filename**: Use for context.
2. **timesignature**: Detect time signature (e.g., "3/4").
3. **genre**: Specific style tag.
4. **language**: Detect the primary language of the vocals.
- Use ISO 639-1 two-letter codes (e.g., "en", "ja", "es", "fr").
- If instrumental or language is completely unrecognizable, use "unknown".
5. **caption**: RICH natural language description (Instruments, Emotion, Timbre, Mood).
- IMPORTANT: The caption MUST be 512 characters or fewer. Be concise but descriptive.
- If human sounds (shouts, whistles) exist, mention them.
IMPORTANT: Output STRICT valid JSON only. Do not add markdown blocks.
Output RAW JSON: {"caption": "...", "genre": "...", "timesignature": "...", "language": "ja"}
"""
def safe_print(text):
try:
print(text)
except Exception:
try:
print(text.encode('utf-8', 'replace').decode('utf-8', 'replace'))
except Exception:
pass
def clean_json_text(text):
if not text:
return ""
text = text.strip()
if text.startswith("```json"):
text = text[7:]
if text.endswith("```"):
text = text[:-3]
match = re.search(r'\{[\s\S]*\}', text)
if match:
text = match.group(0)
return text.strip()
def load_csv_metadata(csv_path):
"""Load Key mapping from VocalRemover's CSV file."""
metadata = {}
if not os.path.exists(csv_path):
safe_print(f"Warning: CSV file '{csv_path}' not found. Keys will be left empty.")
return metadata
try:
with open(csv_path, mode="r", encoding="utf-8-sig") as f:
reader = csv.DictReader(f)
for row in reader:
file_name = row.get("File", "").strip()
if file_name:
# Parse Key
key_str = row.get("Key", "").strip()
metadata[file_name] = {
"key": key_str
}
except Exception as e:
safe_print(f"Error reading CSV '{csv_path}': {e}")
return metadata
# ---------------------------------------------------------------------------
# HYBRID BPM DETECTION (Python -> Node.js)
# ---------------------------------------------------------------------------
def get_bpm_via_javascript(filepath):
"""
Decodes the audio using Python (librosa), passes a temporary WAV
to Node.js for realtime-bpm-analyzer processing, and returns the BPM.
"""
safe_print(" -> [Python] Decoding audio to a safe temporary WAV format...")
temp_wav = os.path.join(tempfile.gettempdir(), f"temp_bpm_track_{uuid.uuid4().hex[:8]}.wav")
try:
# Load downsampled audio to speed up processing
y, sr = librosa.load(filepath, sr=44100, mono=True)
sf.write(temp_wav, y, sr)
# Ensure we are calling the script in the same directory
script_dir = os.path.dirname(os.path.abspath(__file__)) if '__file__' in globals() else os.getcwd()
js_path = os.path.join(script_dir, "get_bpm.mjs")
if not os.path.exists(js_path):
safe_print(f" -> [Error] '{js_path}' not found! Returning BPM 0.")
return 0
safe_print(" ->[Node.js] Running realtime-bpm-analyzer...")
result = subprocess.run(["node", js_path, temp_wav],
capture_output=True,
text=True,
check=False
)
if result.stderr:
safe_print(f" -> JS Warning: {result.stderr.strip()}")
output = result.stdout.strip()
match = re.search(r'BPM_RESULT:(\d+)', output)
if match:
return int(match.group(1))
return 0
except Exception as e:
safe_print(f" -> Error bridging to Node.js: {e}")
return 0
finally:
if os.path.exists(temp_wav):
try:
os.remove(temp_wav)
except Exception:
pass
def try_get_duration(filepath):
try:
from mutagen import File as MutagenFile
audio = MutagenFile(filepath)
if audio is not None and audio.info is not None:
return int(audio.info.length)
except ImportError:
pass
except Exception:
pass
return 0
def get_mime_type(filename):
ext = os.path.splitext(filename)[1].lower()
mime_map = {
".flac": "audio/flac",
".wav": "audio/wav",
".mp3": "audio/mpeg",
".ogg": "audio/ogg",
}
return mime_map.get(ext, "audio/flac")
def parse_retry_delay(error_str):
try:
match = re.search(r'retry[_ ]?(?:in|Delay)[\'"\s:]*(\d+)', error_str, re.IGNORECASE)
if match:
return int(match.group(1)) + 5
except Exception:
pass
return SLEEP_ON_429_DEFAULT
def is_rate_limit_error(e):
error_str = str(e)
return ("429" in error_str or
"RESOURCE_EXHAUSTED" in error_str or
"rate limit" in error_str.lower() or
"quota exceeded" in error_str.lower())
def is_server_error(e):
error_str = str(e)
return ("503" in error_str or
"UNAVAILABLE" in error_str or
"500" in error_str or
"INTERNAL" in error_str)
def upload_with_temp_copy(full_audio_path):
try:
full_audio_path.encode('ascii')
is_ascii = True
except UnicodeEncodeError:
is_ascii = False
mime = get_mime_type(full_audio_path)
if is_ascii:
upload_result = client.files.upload(
file=full_audio_path,
config=types.UploadFileConfig(mime_type=mime)
)
return upload_result, None
else:
ext = os.path.splitext(full_audio_path)[1]
temp_dir = tempfile.gettempdir()
temp_name = f"gemini_upload_{uuid.uuid4().hex[:12]}{ext}"
temp_path = os.path.join(temp_dir, temp_name)
safe_print(f" -> Non-ASCII path. Temp copy: {temp_name}")
shutil.copy2(full_audio_path, temp_path)
upload_result = client.files.upload(
file=temp_path,
config=types.UploadFileConfig(mime_type=mime)
)
return upload_result, temp_path
def scan_audio_files(directory):
audio_files =[]
for filename in os.listdir(directory):
_, ext = os.path.splitext(filename)
if ext.lower() in AUDIO_EXTENSIONS:
audio_files.append(filename)
audio_files.sort()
return audio_files
# ---------------------------------------------------------------------------
# GENIUS LYRICS HELPERS
# ---------------------------------------------------------------------------
def fetch_known_songs(artist_hint: str):
"""Dynamically fetches all top songs for the given artist from Genius API."""
global KNOWN_SONGS
safe_print(f"\nFetching discography for '{artist_hint}' from Genius API...")
if not GENIUS_API_KEY or GENIUS_API_KEY == "...":
safe_print(" -> Warning: GENIUS_API_KEY not set. Skipping song fetch.\n")
return
headers = {"Authorization": f"Bearer {GENIUS_API_KEY}"}
# 1. Get Artist ID from Genius
try:
resp = requests.get(
"https://api.genius.com/search",
headers=headers,
params={"q": artist_hint},
timeout=10
)
if resp.status_code != 200:
safe_print(f" -> Failed to search artist (HTTP {resp.status_code}).")
return
hits = resp.json().get("response", {}).get("hits",[])
artist_id = None
# Check exactly for the artist name
for hit in hits:
artist_info = hit.get("result", {}).get("primary_artist", {})
if artist_hint.lower() in artist_info.get("name", "").lower():
artist_id = artist_info.get("id")
break
# Fallback if strict string matching fails but hits exist
if not artist_id and hits:
artist_id = hits[0].get("result", {}).get("primary_artist", {}).get("id")
if not artist_id:
safe_print(" -> Could not determine Genius Artist ID. Skipping known songs fetch.\n")
return
safe_print(f" -> Found Artist ID: {artist_id}. Pulling discography...")
# 2. Paginate Songs to build the fuzzy-matching list
page = 1
fetched = 0
while True:
resp = requests.get(
f"https://api.genius.com/artists/{artist_id}/songs",
headers=headers,
params={"per_page": 50, "page": page, "sort": "popularity"},
timeout=10
)
if resp.status_code != 200:
break
data = resp.json().get("response", {})
songs_data = data.get("songs",[])
if not songs_data:
break
for song in songs_data:
title = song.get("title", "")
primary_artist = song.get("primary_artist", {}).get("name", "")
KNOWN_SONGS.append((title, primary_artist))
fetched += 1
next_page = data.get("next_page")
if not next_page:
break
page = next_page
safe_print(f" -> Successfully loaded {fetched} known songs into memory.\n")
except Exception as e:
safe_print(f" -> Error fetching known songs: {e}\n")
def normalize_title(title: str) -> str:
title = re.sub(r'^\d+[-–\s]+\d*[-–\s]*', '', title)
title = re.sub(r'^\d+\s+', '', title)
title = re.sub(r'\.(flac|wav|mp3|ogg)$', '', title, flags=re.IGNORECASE)
title = re.sub(r'\s+', ' ', title).strip()
return title
def extract_japanese(text: str) -> str:
return ''.join(c for c in text if '\u3000' <= c <= '\u9fff' or '\uff00' <= c <= '\uffef')
def similarity(a: str, b: str) -> float:
return SequenceMatcher(None, a.lower(), b.lower()).ratio()
def find_best_known_match(filename: str):
norm = normalize_title(filename)
jp_norm = extract_japanese(norm)
best_score = 0.0
best_entry = (None, None)
for display_title, artist in KNOWN_SONGS:
score = similarity(norm, display_title)
jp_known = extract_japanese(display_title)
if jp_norm and jp_known:
jp_score = similarity(jp_norm, jp_known)
score = max(score, jp_score)
if score > best_score:
best_score = score
best_entry = (display_title, artist)
if best_score >= 0.35:
return best_entry, best_score
return (None, None), best_score
def genius_search(query: str, artist_hint: str = ARTIST_HINT) -> dict | None:
headers = {"Authorization": f"Bearer {GENIUS_API_KEY}"}
params = {"q": query}
try:
resp = requests.get(
"https://api.genius.com/search",
headers=headers,
params=params,
timeout=10
)
if resp.status_code != 200:
safe_print(f" -> Genius search HTTP {resp.status_code}")
return None
hits = resp.json().get("response", {}).get("hits",[])
if not hits:
return None
best_hit = None
best_score = 0.0
for hit in hits:
result = hit.get("result", {})
hit_title = result.get("title", "")
hit_artist = result.get("primary_artist", {}).get("name", "")
score = similarity(query, hit_title)
if artist_hint.lower() in hit_artist.lower():
score += 0.2
if score > best_score:
best_score = score
best_hit = result
if best_score >= 0.3:
return best_hit
return None
except Exception as e:
safe_print(f" -> Genius search error: {e}")
return None
def fetch_genius_lyrics(song_url: str) -> str:
try:
from bs4 import BeautifulSoup
except ImportError:
safe_print(" -> beautifulsoup4 not installed. Run: pip install beautifulsoup4")
return ""
try:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36"
}
resp = requests.get(song_url, headers=headers, timeout=15)
if resp.status_code != 200:
safe_print(f" -> Genius page HTTP {resp.status_code}")
return ""
soup = BeautifulSoup(resp.text, "html.parser")
containers = soup.find_all("div", {"data-lyrics-container": "true"})
if not containers:
safe_print(" -> No lyrics container found on Genius page.")
return ""
lines =[]
for container in containers:
for tag in container.find_all(["br", "a", "span", "i", "b", "em"]):
if tag.name == "br":
tag.replace_with("\n")
text = container.get_text(separator="\n")
lines.append(text)
raw = "\n".join(lines)
raw = re.sub(r'\s*(\[[A-Za-z][A-Za-z0-9\s-]*\])', r'\n\n\1', raw)
raw = re.sub(r'\n{3,}', '\n\n', raw).strip()
return raw
except Exception as e:
safe_print(f" -> Lyrics fetch error: {e}")
return ""
def get_lyrics_for_file(filename: str) -> tuple[str, str]:
norm = normalize_title(filename)
safe_print(f" -> Normalized title: '{norm}'")
(matched_title, matched_artist), score = find_best_known_match(filename)
if matched_title:
safe_print(f" -> Best known match: '{matched_title}' (score={score:.2f})")
search_query = f"{matched_title} {matched_artist}"
else:
safe_print(f" -> No known match found (best score={score:.2f}), using raw filename")
search_query = norm
hit = genius_search(search_query, artist_hint=ARTIST_HINT)
if not hit:
jp = extract_japanese(norm)
if jp:
safe_print(f" -> Retrying Genius with Japanese only: '{jp}'")
hit = genius_search(jp, artist_hint=ARTIST_HINT)
if not hit:
safe_print(" -> No Genius match found. Lyrics will be empty.")
return "", ""
song_url = hit.get("url", "")
hit_title = hit.get("title", "")
hit_artist = hit.get("primary_artist", {}).get("name", "")
safe_print(f" -> Genius match: '{hit_title}' by {hit_artist}")
safe_print(f" -> URL: {song_url}")
raw_lyrics = fetch_genius_lyrics(song_url)
if not raw_lyrics:
return "", ""
formatted = raw_lyrics.strip()
return raw_lyrics, formatted
# ---------------------------------------------------------------------------
# DATASET HELPERS
# ---------------------------------------------------------------------------
def create_empty_sample(filename, directory):
rel_path = os.path.join(".\\" + os.path.basename(directory), filename)
full_path = os.path.join(directory, filename)
duration = try_get_duration(full_path)
return {
"id": uuid.uuid4().hex[:8],
"audio_path": rel_path,
"filename": filename,
"caption": "",
"genre": "",
"lyrics": "",
"raw_lyrics": "",
"formatted_lyrics": "",
"bpm": 0,
"keyscale": "",
"timesignature": "",
"duration": duration,
"language": "unknown",
"is_instrumental": ALL_INSTRUMENTAL,
"custom_tag": CUSTOM_TAG,
"labeled": False,
"prompt_override": None
}
def create_dataset_skeleton(directory, audio_files):
samples =[create_empty_sample(f, directory) for f in audio_files]
dataset = {
"metadata": {
"name": DATASET_NAME,
"custom_tag": CUSTOM_TAG,
"tag_position": TAG_POSITION,
"created_at": datetime.utcnow().isoformat(),
"num_samples": len(samples),
"all_instrumental": ALL_INSTRUMENTAL,
"genre_ratio": GENRE_RATIO,
},
"samples": samples
}
return dataset
def save_dataset(data):
with open(OUTPUT_JSON, "w", encoding="utf-8") as f:
json.dump(data, f, indent=4, ensure_ascii=False)
# ---------------------------------------------------------------------------
# CORE PROCESSING
# ---------------------------------------------------------------------------
def process_sample(sample, directory, data, csv_metadata):
filename = sample['filename']
full_audio_path = os.path.join(directory, filename)
if not os.path.exists(full_audio_path):
safe_print(f" Skipped: File not found")
return False
# --- Step 1: Automated BPM (Node.js) & Manual Key (CSV) ---
safe_print(f" [1/3] Extracting BPM via Node.js realtime-bpm-analyzer...")
bpm = get_bpm_via_javascript(full_audio_path)
sample['bpm'] = bpm
csv_info = csv_metadata.get(filename, {})
sample['keyscale'] = csv_info.get('key', "")
# --- Step 2: Fetch lyrics from Genius ---
safe_print(" [2/3] Fetching lyrics from Genius...")
_, formatted_lyrics = get_lyrics_for_file(filename)
# --- Step 3: Gemini for caption/genre/etc. ---
max_retries = 3
temp_file_path = None
for attempt in range(max_retries):
try:
safe_print(f"[3/3] Uploading to Gemini (Attempt {attempt + 1}/{max_retries})...")
upload_result, temp_file_path = upload_with_temp_copy(full_audio_path)
wait_count = 0
while upload_result.state.name == "PROCESSING":
time.sleep(2)
upload_result = client.files.get(name=upload_result.name)
wait_count += 1
if wait_count > 60:
raise TimeoutError("Upload processing timed out")
if upload_result.state.name == "FAILED":
safe_print(" -> Audio processing failed on Google server.")
return False
prompt = f"{SYSTEM_PROMPT}\n\nfilename: {filename}"
response = client.models.generate_content(
model=MODEL_NAME,
contents=[
types.Content(
role="user",
parts=[
types.Part.from_uri(
file_uri=upload_result.uri,
mime_type=upload_result.mime_type),
types.Part.from_text(text=prompt)
]
)
],
config=types.GenerateContentConfig(
response_mime_type="application/json"
)
)
if not response.text:
safe_print(" -> Error: Model returned empty response.")
raise ValueError("Empty response text")
cleaned_text = clean_json_text(response.text)
try:
new_meta = json.loads(cleaned_text)
except json.JSONDecodeError as je:
safe_print(f" -> JSON Parse Error: {je}")
safe_print(f" -> Raw text start: {cleaned_text[:80]}...")
return False
if isinstance(new_meta, list):
if len(new_meta) == 0:
raise ValueError("Empty list from API")
new_meta = new_meta[0]
# Enforce 512-char caption limit
caption = new_meta.get('caption', "")
if len(caption) > CAPTION_MAX_CHARS:
caption = caption[:CAPTION_MAX_CHARS].rsplit(' ', 1)[0] # trim at word boundary
sample['caption'] = caption
sample['genre'] = new_meta.get('genre', "")
sample['timesignature'] = new_meta.get('timesignature', "")
# LYRICS ASSIGNMENT
sample['lyrics'] = formatted_lyrics
sample['raw_lyrics'] = ""
sample['formatted_lyrics'] = formatted_lyrics
sample['prompt_override'] = None
detected_lang = new_meta.get('language', 'unknown')
if detected_lang and detected_lang.lower() != "unknown":
sample['language'] = detected_lang.lower()[:2]
else:
sample['language'] = "unknown"
if sample['duration'] == 0:
sample['duration'] = try_get_duration(full_audio_path)
sample['labeled'] = True
safe_print(f" -> Done! (Lang: {sample['language']}, BPM: {sample['bpm']}, Key: {sample['keyscale']}, "
f"Caption: {len(sample['caption'])} chars, "
f"Lyrics: {'yes' if sample['lyrics'] else 'none'})")
save_dataset(data)
return True
except KeyboardInterrupt:
safe_print("\n\n*** Interrupted by user. Saving... ***")
save_dataset(data)
sys.exit(0)
except Exception as e:
if is_rate_limit_error(e):
retry_seconds = parse_retry_delay(str(e))
safe_print(f" -> Rate Limit. Sleeping {retry_seconds}s...")
time.sleep(retry_seconds)
elif is_server_error(e):
safe_print(f" -> Server Error (503). Sleeping {SLEEP_ON_503}s...")
time.sleep(SLEEP_ON_503)
else:
safe_print(f" -> FULL ERROR: {type(e).__name__}: {e}")
break
finally:
if temp_file_path and os.path.exists(temp_file_path):
try:
os.remove(temp_file_path)
except Exception:
pass
temp_file_path = None
return False
def process_dataset(data, directory, csv_metadata):
total = len(data['samples'])
safe_print(f"Processing {total} samples...\n")
processed_count = 0
skipped_count = 0
for i, sample in enumerate(data['samples']):
if sample.get('labeled', False) and sample.get('language', 'unknown') != "unknown":
skipped_count += 1
continue
safe_print(f"\n[{i + 1}/{total}] Processing: {sample['filename']}")
success = process_sample(sample, directory, data, csv_metadata)
if success:
processed_count += 1
time.sleep(SLEEP_BETWEEN_REQUESTS)
else:
safe_print(" -> Failed. Moving to next.")
safe_print(f"\n{'=' * 50}")
safe_print(f"Already done: {skipped_count}")
safe_print(f"Newly processed: {processed_count}")
safe_print(f"Total: {total}")
safe_print(f"{'=' * 50}")
def main():
directory = os.getcwd()
safe_print(f"Working directory: {directory}")
safe_print(f"Dataset: {DATASET_NAME} | Tag: {CUSTOM_TAG} | Caption max: {CAPTION_MAX_CHARS} chars\n")
# Load keys from VocalRemover CSV
csv_metadata = load_csv_metadata(CSV_METADATA_FILE)
# PRE-LOAD DISC FROM GENIUS API
if ARTIST_HINT:
fetch_known_songs(ARTIST_HINT)
if os.path.exists(OUTPUT_JSON):
safe_print(f"Found existing '{OUTPUT_JSON}' — resuming.\n")
try:
with open(OUTPUT_JSON, "r", encoding="utf-8") as f:
data = json.load(f)
except json.JSONDecodeError:
safe_print("ERROR: dataset.json is corrupted. Cannot resume.")
return
for s in data['samples']:
if 'raw_lyrics' not in s:
s['raw_lyrics'] = ""
if 'formatted_lyrics' not in s:
s['formatted_lyrics'] = s.get('lyrics', "")
if 'prompt_override' not in s:
s['prompt_override'] = None
existing_filenames = {s['filename'] for s in data['samples']}
all_audio = scan_audio_files(directory)
new_files =[f for f in all_audio if f not in existing_filenames]
if new_files:
safe_print(f"Found {len(new_files)} new files. Adding them.")
for nf in new_files:
data['samples'].append(create_empty_sample(nf, directory))
data['metadata']['num_samples'] = len(data['samples'])
save_dataset(data)
else:
audio_files = scan_audio_files(directory)
if not audio_files:
safe_print("No audio files found.")
return
safe_print(f"Found {len(audio_files)} audio files. Creating dataset.\n")
data = create_dataset_skeleton(directory, audio_files)
save_dataset(data)
process_dataset(data, directory, csv_metadata)
safe_print("\nDONE.")
if __name__ == "__main__":
main()
|
Important Captioning Notes
- Gemini API Usage: This script uses the Gemini API to generate captions. While highly capable, captions may not be 100% accurate and often require manual tuning for the best results.
- Artist LoRAs: If you are training a specific artist LoRA, modify the
SYSTEM_PROMPTto explicitly include the artist's name in the description to reinforce the identity. - Language Tags: ACE-Step uses two-letter ISO tags. Note that Japanese is
ja(notjp). Ensure your prompt explicitly instructs the model to use the correct tag for your target language to prevent training errors. - Local Alternatives: If you have the hardware, you can caption locally to avoid API limits.
- acestep-captioner: The original model used to train the base ACE-Step model.
- MOSS-Music-8B-Thinking: A newer, powerful alternative that uses "thinking" tokens to reason about the music for highly detailed captions.
Tagging and Metadata Configuration
When configuring your dataset.json or the automation script, pay close attention to how tags and lyrics are handled:
Custom Tag Options (tag_position)
The custom_tag (e.g., m1ku_d3c0) acts as the trigger word for your LoRA. You have three positioning options:
- prepend (Recommended): Places the tag at the start of the caption (e.g.,
"m1ku_d3c0 Vibrant J-Pop..."). This preserves the descriptive caption and is the standard way to prompt the model. - append: Places the tag at the end of the caption.
- replace: Replaces the entire caption with only the custom tag. This is generally not recommended as it loses descriptive detail.
Dataset Settings
- all_instrumental: Keep this
falseunless your dataset consists entirely of instrumental tracks where the lyrics field is just[instrumental]. - genre_ratio: Denotes the percentage of captions to be replaced by the
genretag during training. For example, 20 means 20% of your samples will use only the genre as the prompt. 0 is considered optimal, as it ensures the model sees the full detailed caption every time, and genre info is usually included in the caption anyway.
Captioning Tips
- Length: Shorter is often better. Keep captions within the 512 character limit.
- Style: While the script generates natural language, "Suno-style" (comma-separated tags) may perform well as it mimics how users naturally prompt the model.
Lyrics and Audio Metadata
Quality Control for Lyrics
The script uses the Genius API to pull lyrics, but manual verification is essential.
- Cleanup: Genius often includes text before
[Intro]or after[Outro], instruction tags in various languages, or may omit section tags entirely. - Missing Lyrics: If Genius does not have the lyrics, you should manually find them on other lyric hosting sites. Alternatively, you can use Gemini to transcribe the audio, but do so with extreme caution. The Gemini API often contains inaccuracies in transcriptions, and poor lyric quality directly hurts model performance.
- Consistency: Ensure formatting (e.g.,
[Chorus],[Verse]) is consistent across your entire dataset. This should match how you intend to prompt the model later. Use an LLM to batch-reformat messy lyrics into a clean, consistent structure. - Redundancy: ACE-Step includes several fields (
lyrics,raw_lyrics,formatted_lyrics).raw_lyricscan stay empty; providing the same text inlyricsandformatted_lyricsis sufficient.
Accuracy: BPM and Key
- BPM Detection: This script uses
realtime-bpm-analyzervia Node.js for high accuracy. - Key Detection: LLMs are inaccurate at identifying Key Scale. Use vocalremover.org's Key/BPM Finder, export the results as a
.csv, and point the script toCSV_METADATA_FILE. The script will pull the Key from the CSV while using the Node Bridge for the BPM.
Genius API & Artist Hint
ARTIST_HINT is the complete artist name (including all spaces) found on the Genius website.
The KNOWN_SONGS list is now automatically populated by the script via the Genius API using your provided ARTIST_HINT. This ensures the fuzzy-matching engine can find the correct lyrics even if your filenames are stylized.
Preprocessing
Once your dataset.json and audio files are ready, you must preprocess them using Side-Step.
- Standard Preprocessing: Launch Side-Step and select the preprocessing options from the menu. When configuring the settings, ensure you set the following:
- Model Selection: Always choose the same model you intend to train on. Selecting the Base XL model is highly recommended here.
- Normalization: Choose Peak (-1.0 dBFS, no extra deps, matches ACE-Step). This levels your audio correctly for the model and prevents clipping.
- Preprocessing++ (Experimental): This is an advanced version of the pipeline that may yield stronger results, though it is currently experimental and less widely tested.
- Dataset Size: ACE-Step 1.5 XL is remarkably efficient with small datasets. Aim for 10–25 high-quality songs. Results vary by artist, so feel free to experiment with larger datasets if the initial results don't capture the nuances correctly.
Training Settings
Training ACE-Step 1.5 XL requires careful balance. Because memory consumption scales quadratically with song length, optimization is key.
Core Configuration
- Model Selection: Always select the Base XL model in the Side-Step menu. Even if you intend to use the LoRA with the Turbo model later, you must train on the Base XL model.
- Optimizer: Use AdamW. Avoid Prodigy, as it is unstable for the smaller datasets typically used in ACE-Step training.
- Learning Rate (LR) & Rank:
- Standard: 0.0001 LR with a rank of 64/128.
- High Rank: 128/256 is possible, but you may need to lower the LR to avoid overfitting.
- Alternative: Some users report success with a lower LR (0.00005) over long training runs (2,000–4,000 epochs) on non-XL versions.
- Epochs & Checkpoints: Training is usually "done" between 200–800 epochs. Save checkpoints every 25 epochs; specific epochs can be noisy, so having a library of checkpoints allows you to pick the cleanest one.
- Check your results every 50 epochs once you pass the 200 mark.
- Generally, the "best" checkpoint is excellent, but you should test both that and regular epoch checkpoints to find the optimal version.
- Determining Convergence: For a checkpoint to be "good," you are looking for the loss to have stabilized for several epochs. Keep in mind that since this is a diffusion model, a low loss (e.g., 0.1) does not necessarily mean convergence.
- A slightly "overbaked" model—one that can reproduce some of the training songs given the exact same caption, metadata, and lyrics—is actually a good sign. It indicates the model has converged well, provided it can still generalize effectively when presented with different captions and lyrics.
- Testing: Once the model is trained, it should be tested without the LM (Language Model) and with DiT only to achieve the best results.
- Important Update: The previous recommendation to use the plain Turbo XL model for this step has changed. It is now highly recommended to use the merge-base-turbo-xl-ta-0.5.gguf (an equal blend of the Base XL and Turbo XL models). This merge significantly outperforms the plain Turbo XL model when using DiT-only prompting by enhancing three key areas: audio quality, song composition, and the overall creativity and capability of the model.
Performance & Optimization
- Batch Size: A total batch size of 4 is the gold standard. On a 24 GB card (like a 3090/4090), set Batch Size 2 and Grad Accumulation 2. If you must use a batch size of 1 to fit VRAM, be aware that results may be less stable.
- The "Chunk Duration" Trick: In Side-Step's advanced options, set Chunk Duration to 60 seconds.
- This extracts a random window of the song during each pass rather than processing the full file.
- Impact: This can reduce training time from 20+ hours down to just a few hours or minutes, as it bypasses the quadratic memory scaling.
- Trade-off: Some users report slightly more repetition in generated outputs, though others have successfully replicated training data even with this setting enabled.
Cloud Training
ACE-Step 1.5 XL has very steep hardware requirements. Work is being done to optimize this. If you do not have a top-tier GPU (like a 5090 or 4090), consider using a cloud provider:
- RunPod: Reliable paid GPU instances.
- Modal: Offers $30 free monthly credits, which is often enough for a few LoRA training runs.