Even beyond just F1 issues, from a nature standpoint unlike animals can mate, but the result doesn't get very far in the gestation process. This is partly because animals are diploid, and the cross over event occurs not between the two animals, but between the two strands in both parents. Darwinbots switches this around since our bots are haploid, but then we have problems with unlike bots mating to produce a clone randomly of either parent. Which is clearly an artifact of our system and not something that mirrors real life.
Maybe something as simple as a % difference between the two parents' DNA. (ie: number of "zipped up" places / longest DNA length). If it's sufficiently low (ie: 5% or something), then the union fails. I'm not a fan of hard and fast constants like this, but I think we need something. It's one thing to mate with something and produce a sterile, stupid, hybrid. It's quite another to mate with something like a veg and produce a clone of yourself with veg status.