I like the flood fill testing, but I have an idea which might work better? Instead of testing horizontally and vertically like you do now (when most flood fills use horizontal or vertical spans) why not match the source pixel to ones in a target pattern? That would allow you to fill in custom dither pattens, as well as small patterns and potentially shapes that have previously been filled in with a pattern from a brush. Finding the XY offset for the pattern might be a bit tricky, as would trying to automatically find repeating patterns.
What do you think? 
It would certainly fix the 'invasiveness' problem inherent in 20-way filling.
(the 20way filling was deliberately 'a generic fill for most dithers' so the user could get a good result without needing to configure which patterns to search for; so 20-way fill actually copes with a lot of custom dither-patterns already.)
One of the things you didn't mention is that usually dither patterns come in groups, so matching to one of several same-size patterns is probably a more realistic use.
I've actually tried to do that before, but have a much better idea of how to do it now.
the alignment issues.. I think it requires a two-pass approach:
Preparation:
1. Generate a binary mask for the entire image, which is True where the pixel matches seed color.
2. Initialize a 'matched' array (uint8) to 0 (1 == pattern 0 matched , 2 == pattern 1 matched..)
3. Calculate hashes for each of the patterns being matched
Execution:
1. SCARY. But mainly, treat the current location as upper-left, then hash that rectangular area of the seed-mask and compare it to each stored hash. Don't try to match areas where >= half of the pixels have already been matched.
If it matches a hash, store the result in a temporary list, and keep searching with different offsets.Inspect each item in the list for best 'suitability', which tries to minimize the amount of neighbouring True pixels in the seed-mask, and mark the best area.
2. Try to match any remaining areas where one or more unvisited True pixels in the seed mask remain.
If the area is partially visited, prefer matching one of the already-matched patterns listed in the matched array.
(possibly need to support matching a partial pattern.. eg for the left side of the picture, if the leftmost 2x4 are unmatched and the next 4x4 is matched, need to match just the half of the pattern that fits on the picture.)
Obviously that isn't a floodfill; I find it a bit difficult to think how to constrain it in the usual floodfill way.
A variation on that scheme can be used to convert dither patterns -> some solid color.
Just for fun, I have a really simple dither-detection aka high-frequency noise detection algorithm that is relatively immune to XY offset.
1. Move in steps of 2 pixels along y,x axes
2. Calculate an average for the 4 2x2 squares at offsets (+0,+0), (+1,+0), (+0,+1), (+1,+1)
3. If the averages are all equal and the pixels are all equal, continue the loop at the next position.
4. Check those 4 averages against each other. each pixel is included in 3 of them, and
if the averages that a pixel is included in all match, that pixel is marked as dither.
Your GUI isn't really as big as I thought. What's that toolbox labelled 'icons 27x25'?
