Actually it's more about chip color and formation for me. A bad sound in aluminum often means you're already way past the point of a dull tool. The finish goes bad way before the noise gets scary in softer metals. Relying only on sound seems like a good way to break an end mill.
Yeah but you still gotta listen for that change in tone, even if it's late. A sharp tool in aluminum has a clean, crisp sound. When it starts to get a high-pitched screech or a chattery rumble, that's your heads up to check everything else immediately. You're totally right that chip color and shape are the main things to watch. I just wouldn't turn the sound off completely, it's one more layer of info.