I was playing around with a similar idea, having Guards line the perimeter of the hive. If it is an enemy bot, it tries to kill it, if it's an ant from another hive, it shoots a memory shot that sends it flying backwards.
I'm moving away from the "good of the species" idea and towards the "good of the colony" idea, but that might make the whole thing unstable. I usually run a size 10 sim, and it doesn't take long for one single hive to swarm the map. If you ever ran it, you would see newborn bots (at least drones and workers) hurled backwards. Also, a newborn queen would get slaughtered. Maybe a baby queen overrides the guard's attack instinct?
How about this: A worker will sometimes steal food from the enemy hive. The enemy, if it sees the ant, becomes slightly more hostile and marks the ant. The marked ant may or may not be slaughtered by its own hive, depending on how hostile the hive is. When hostility reaches a certain point, it overrides the "good of the species" instinct and starts killing all ants from different hives.