After some experiments, I am going to abandon the color space in the original post.
Problems:
1) Not enough pinks and browns. Detailed faces required interpolation between colors on the palette, and if I always need to do finer blending then it's easier to start with a smaller set of colors.
2) The colors mix differently at different saturation and lightness. I originally pulled red 207 and blue 104 out of the starting RGB set because each of these when mixed with a single other color component the results weren't visually very different. I ended up needed both of these components when using a mix of three components when trying to set up a smaller palette for specific images.
This tells me straightforward RGB mixing doesn't give the best results.
3) I need a better measure of how well colors will blend. Here's an example:

The pink and gray-pink are one step away from each other in the original color set. The middle row of gray-pink and yellow are two steps away, but they blend better together. The third row is the gray-pink and yellow that are one step apart on the original set.
Taking the gray-pink as a reference, and comparing the middle row yellow to the upper row pink, the pink is
Closer in RGB space (red differs by 23 vs blue differs by 72)
Closer in hue (pink differs by 7, yellow by 31)
Closer or equal in saturation (HSL vs HSV defn)
Only a little further in brightness (8% brighter for pink vs 4% darker for yellow)
The numbers suggest that the pink should be closer than the yellow to the gray-pink, but visually it has a higher contrast. This tells me the numbers are missing something, I'd like a better predictor of contrast that I can reverse engineer into a better defined color space.
Still to do:
2) Try a different set of starting colors, fewer colors or something that's not pure red, blue, green.
5) Figure out a better way to compare the colors with math
Feedback is always welcome.
Tourist