I've been debating on writing something that basically null routes an IP address after a few failed logging attempts. To do this I would have have to write a log parser and monitor the file, which means dealing with log rotation and all that jazz. I found this: https://lwn.net/Articles/255781/
iptables -A INPUT -i eth0 -p tcp --dport 22 -m state \
--state NEW -m recent --set --name SSH
iptables -A INPUT -i eth0 -p tcp --dport 22 -m state \
--state NEW -m recent --update --seconds 60 --hitcount 6 \
--rttl --name SSH -j DROP