Audio to MIDI per stem, for sampling and remixing

Audio to MIDI on a full mix gives you a mess. Four instruments, a vocal and a drum kit all competing for the same note detector, and what comes out is a cloud of notes nobody can use.

Audio to MIDI on a stem is a different story. One instrument, one line, one sensible transcription. So the real recipe for getting usable MIDI out of a record is two steps, not one: split first, then transcribe each stem separately.

Here is the workflow, what to expect from each stem, and what to do with the result.

Step one: split the stems

Both steps are hosted functions, so this is two calls. In Python:

import audial

audial.config.set_api_key("your_api_key")
audial.config.set_user_id("your_user_id")

stems = audial.stem_split(
    "track.mp3",
    stems=["vocals", "drums", "bass", "other"],
    results_folder="./out/stems",
)
for name, path in stems["files"]["files"].items():
    print(name, path)

Tempo and key are detected automatically during separation and tagged on every stem, which matters for the next step because the detected BPM is what you want the MIDI quantised against.

Step two: transcribe each stem

generate_midi takes a single path or a list of them, plus an optional BPM override:

from pathlib import Path

stem_dir = Path("./out/stems")
files = [str(p) for p in stem_dir.glob("*.mp3")]

midi = audial.generate_midi(files, bpm=124)
for name, path in midi["files"]["files"].items():
    print(name, path)

If you leave bpm out, the detected tempo is used. Pass it explicitly when you know the real tempo and the detection landed on a multiple or a half of it, which happens on sparse or heavily swung material.

Over REST it is the same thing with a repeatable files part:

curl -X POST "https://api.audialmusic.ai/api/functions/run/generate-midi" \
  -H "X-API-Key: your_api_key" -H "X-User-ID: your_user_id" \
  -F "userId=your_user_id" \
  -F "files=@/path/to/bass.mp3" \
  -F "files=@/path/to/other.mp3" \
  -F "bpm=124"

From an AI assistant with the Audial MCP server installed, the two steps are one sentence: "Split ~/Music/track.mp3 into stems, then convert the bass and other stems to MIDI."

What to expect from each stem

Bass. The most reliable transcription of the four, and usually the most useful. A bass line is monophonic, in a register where pitch detection is stable, and musically it is the thing you most often want to steal, rebuild or re-voice. If you transcribe one stem, transcribe this one.

Vocals. Good for toplines. The notes come back as a melodic line, which is exactly what you want for feeding a synth, a vocoder or a text-to-vocal render. Expect the transcription to be literal about slides, vibrato and ornaments, so a sustained note with a wide vibrato can arrive as several notes. Clean that up by hand or quantise and merge.

Other. This is the harmony stem: guitars, keys, pads, synths, strings. Polyphonic, so the transcription is messier, but this is where chords live and it is the stem to use for chord work. Expect extra notes and octave confusion on dense arrangements.

Drums. The awkward one, and the reason for the next section.

Why drum MIDI sometimes comes back empty

Transcription works by finding pitched note events: something starts, it has a pitch, it stops. A drum stem is mostly unpitched transients and noise, so there is often nothing for that process to lock onto, and the resulting MIDI file can contain no notes at all.

Drums are the usual case, but any stem can come back empty when there is too little sustained pitched material in it, for example the bass or vocal stem of a track that is mostly percussion. This is most common when the stem is sparse. A track with a kick, a clap and a closed hat and nothing else gives a transcriber almost no tonal content to work with. A dense live kit with toms and a ringing snare gives it more, and you may get notes, though they will not be mapped to a General MIDI drum map in any way you should trust.

So the honest guidance is: do not use audio to MIDI to extract a drum pattern. For drums, use one of these instead.

• Sample the hits. Run the sample pack function on the track and you get the kicks, snares and hats as one-shots, which is almost always what you actually wanted. Load them into a sampler and program the pattern yourself.
• Program against the grid. You already have the detected BPM, so the drum stem lined up in a DAW is a perfectly good visual reference for programming a pattern by ear in ten minutes.
• Use your DAW's transient detection. Most DAWs can slice a drum loop to transients and give you a MIDI clip triggering those slices. That is a sampler-based result, not a transcription, and it is the right tool for the job.

If you send a stem to the MIDI function and get an empty file back, nothing has failed. There was nothing pitched to find, and a DAW will refuse to import a MIDI file with no notes, which is usually how people notice.

Using the MIDI in a DAW

Set the project tempo first, from the detected BPM. Everything after this depends on it.

Import the MIDI onto an instrument track and park the matching audio stem on a muted track directly below it. You are going to be comparing the two constantly, and having them vertically adjacent makes that a single click.

Then the cleanup, in this order:

1. Delete the obvious wrong notes. On polyphonic stems there are usually a few, and they are easier to spot by ear than by eye. Solo the MIDI track and listen for the note that does not belong.
2. Fix octaves. Transcription on the other stem will sometimes place a note an octave out. Move it rather than deleting it.
3. Quantise lightly. Quantise to the grid at a conservative strength first. Hard quantising a transcription to 16ths destroys the feel that made you want the part.
4. Merge over-split notes. Long sustained notes often arrive as runs of short ones, especially on vocals. Join them.
5. Only then change the sound. Point the track at the instrument you actually want and keep editing from there.

For remixing specifically, the bass MIDI is the fastest route to a new arrangement that still feels like the original record. Take the bass MIDI, assign your own sub, and you have the harmonic skeleton without a single sample of the source in your project.

Chord detection ideas

There is no chord-detection endpoint, but the pieces to build one are all here, and the combination is more useful than a single chord label anyway.

Derive chords from the other stem MIDI. Group the notes into time windows, reduce each window to its pitch classes, and match against chord templates. A quarter-note or half-note window works for most material. The detected key from the analysis function tells you which spellings are plausible, which removes most of the ambiguity between enharmonic candidates.

Use the bass MIDI to pick the root. This is the single biggest improvement you can make to a naive chord detector. The bass note under a window is usually the root or the fifth, which resolves most inversion and rootless-voicing confusion in the harmony stem.

Segment first, then detect per section. Run the segmentation function to get section boundaries, then detect chords inside each section. Pop and dance music repeats harmony per section, so a progression detected over one chorus and then asserted across all choruses is more accurate than a window-by-window read of the whole track.

Write out a chord track. Once you have a progression with timings, emit it as a MIDI file of block chords and import it as a guide track. It is the fastest way to check whether your detection is right: you play it against the original and hear immediately where it is wrong.

Where to go next

The useful order is split, transcribe the melodic stems, sample the drums.

MIDI generation, the stem splitter and sample packs all run in the browser, and segmentation gives you the section boundaries for the chord work above. Parameters for every function are on the API reference.