Why Your AI Generates the Same Thing Twice (And Where It’s Hiding)

We pulled up the tracks Noema generated on both days. Different files, same mood, same tempo range, almost the same instrumentation.

Noema doesn't take a prompt. It reads your ambient signals — location, weather, time, motion — and generates a track from that. No text box, no "type what you want to hear."

That absence of a prompt is the whole pitch. You open the app and it just knows.

Except if 2 different days with similar signals keep producing tracks that feel the same, the pitch quietly breaks. The app stops reading your moment. It starts reading a bucket.

We didn't have a way to check this. We had a hunch and 2 screenshots.

Mode Collapse Hiding Behind Byte Uniqueness

Around the same time we ran into a paper called "Evaluating the Diversity of AI-Generated Content with Diversity Profiles." We're not going to pretend we fully absorbed a paper written for people training foundation models.

One idea stuck though. Diversity isn't "are these 2 outputs identical." It's whether your outputs spread across the feature space you actually care about, or collapse into a handful of modes that just look different on the surface.

That reframed the question we should've been asking from day one: how spread out are the tracks Noema actually produces for a given signal bucket? Byte-identical was never in question — audio generation guarantees that for free. The gap we cared about was mode collapse hiding behind that uniqueness.

Building the Entropy Audit

So we built a small audit.

Every generation already gets logged with its signal snapshot (location bucket, weather condition, time-of-day window, motion state) plus the model's own output metadata: tempo, key, instrumentation tags, a mood label. That data was already sitting in the database from the SSE pipeline we wrote about on Medium.

We just hadn't looked at it in aggregate.

We grouped generations by signal bucket and ran a spread check on the instrumentation tags:

function diversityScore(tracks: TrackMeta[]): number {
  const tagCounts = new Map<string, number>();
  for (const t of tracks) {
    for (const tag of t.instrumentationTags) {
      tagCounts.set(tag, (tagCounts.get(tag) ?? 0) + 1);
    }
  }
  const total = tracks.length;
  let entropy = 0;
  for (const count of tagCounts.values()) {
    const p = count / total;
    entropy -= p * Math.log2(p);
  }
  return entropy; // higher = more spread across tags
}

Nothing fancy. Shannon entropy over tag frequency, bucketed by signal condition. Enough to turn a hunch into a number.

Where Was the Bottleneck?

Some buckets looked healthy. "Morning, clear, walking" showed a wide spread of tempo and instrumentation across a few weeks of our own generations.

"Evening, any weather, stationary" was flat. Almost every track in that bucket landed in the same narrow tempo range with the same 2 or 3 instrumentation tags repeating. That's the bucket that produced our nearly-identical Tuesdays.

We assumed the fix was location. More granular geohash, tighter resolution, more distinct inputs to work with.

We added it. The diversity score for the evening bucket barely moved.

We were starving the model of the exact context we thought we were giving it.

Loosening that template — letting more of the raw signal survive into the prompt instead of flattening it into a category label — moved the entropy score for that bucket more than any amount of location precision did.

The Lesson for GenAI Builders

What we built is closer to a smoke test than a proper diversity eval, and that's fine. It caught something 2 random Tuesdays almost let slide.

If you're building anything that generates instead of retrieves, you probably have a diversity assumption sitting somewhere you haven't tested yet. Ours was hiding in a prompt template, not the model.

Worth checking where yours is hiding before a user notices it before you do.

Noema is an AI-powered music generation app for iOS. It reads your moment instead of asking you to describe it, and we'd rather catch it repeating itself than have you catch it first.