It is a sad and known fact that having lots of ACLs in your SharePoint database has a negative effect on the performance of your site. What I did not know yet was that changes to the ACL will impact you incremental crawls.