Here are some popular open source host filtering and IDS solutions that have been recommended by the Science DMZ community:
- IPTables: interface to install packet filtering rulesets. Note that use of IPTables may cause performance issues.
- OSSec: log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response
- Rkhunter: rootkit detection
- chkrootkit: rootkit detection
- Logcheck: log analysis
- Fail2ban: log analysis
- denyhosts: Blocks ssh attacks (tho you can also block ssh using router ACLs to all but trusted subnets)