Here's a minimal example with a very rapidly prototyped 8x8 tileset.
It mainly extends grass fringes.
There are two rule sets present.
One maps each tile to itself. This is important because isolated tiles will otherwise not be included on the output layer. These are 1x1 patterns.
The other recognizes the pattern 'grass top without a fringe' and outputs 'grass top with a fringe'. This is a 2x1 pattern.
Layers included are similar to the example set:
"Regions, Input_set, InputNot_set, Output_Ground"
And in broad summary, this is saying to look for input in a layer named set, and output a layer named Ground.
Tiles in Input_set are checked for their presence, tiles in InputNot_Set are checked for their absence. The combination of the two is how the rule 'X tile without Y tile' is expressed: I placed the lower grass tile in the lower cell of the 2x1 in the Input_set layer, and the fringe tile in the upper cell of the 2x1 in the InputNot_set layer, and finally both together in the Output_Ground layer. A variation of this could be used to remove fringes, for example (by including both together in the Input_set layer, none in the InputNot_set layer, and only the lower grass tile in the Ground layer.)
Regions expresses the area of pattern recognition, with careful isolation of each individual pattern. I made a very visible tile "R" for use on this layer, a practice adopted from the examples. Normally, you should map the Regions layer first, because all tiles on other layers should fall within the regions defined. Tiles outside those regions will be ignored (ie. have no contribution to the ruleset).
Regions do not need to be rectangular, although I think they do have to be contiguous. I later defined a rule
where the input regions are
#. .#
.# #.
and the output regions are
.# #.
.. ..
To automatically insert diagonal grass edges at platform edges. These are 8-directionally contiguous.
Finally, the rules.txt file lists which rule files to use. In this case it only references one. The result is that maps saved in that directory use those rules when 'Automap' is invoked. In general you should isolate each individual group of effects in a individual rule file, so you can control the order in which they are applied, which is a principle I broke here -- the 'map tiles to self' rules should really be in a separate file.
You can test it by loading ruletester.tmx and hitting Automap -- a Ground layer is generated which is identical to the input except it has longer grass fringes.
Before:
After:
Looking at your tileset, you could begin by making similar rules: You have a series of tiles with both dirt and grass, where the 'upper' grass tile to finish it is inferrable from the base tile, so you could make some 2x1 patterns to define that.
You could continue by making a few 3xN patterns for the dirt in the bottom-left corner, to take the generic dirt tile, detect that it is vertically isolated, and output the correct 'tapered column' tiles in the bottom left. I believe you can randomize this (pick randomly different dirt tiles for the 'mid' part of the column), but I haven't got experience with that yet. (EDIT: I got that working, following the same method for randomization listed on the wiki page)
