Hi,
Okay so I have a Windows Server 2012 R2 box and Windows 10 Enterprise Oct 2018 update box, both utilising real IIS and the same version of the web site installed (MS .NET MVC, SQL Server etc..).
On our site we often get hit by bots that don't respect the robots.txt file and will spam the site with thousands of calls a minute. To prevent this we've added the following rewrite rule that should allow Googlebot and Bingbot but any other request whose user agent string mentions one of the terms in the rules expression will be immediately aborted.
<rewrite><rules><rule name="Bot Blocking" stopProcessing="true"><match url=".*" /><conditions logicalGrouping="MatchAll"><add input="{HTTP_USER_AGENT}" pattern="^((?!(GoogleBot|BingBot)).)*$" /><add input="{HTTP_USER_AGENT}" pattern="^.*(bot|crawler|kicks|scanner|spider).*$" /></conditions><action type="AbortRequest" /></rule></rules></rewrite>The problem is on the Windows Server 2012 R2 box, it is not aborting the requests when it mentions, for example, "crawler" in the user agent string. The Windows 10 IIS box works as expected.
I found this post https://stackoverflow.com/questions/20637592/iis-url-rewrite-works-one-old-server-but-not-on-new and added the rewrite portion from the answer to the applicationhost.config file it indicates inside the <system.webserver> element but the Server 2012 R2 box still fails to apply the rule correctly.
I'm at a total loss as to why there is a difference. Any ideas of why this isn't working or how to fix it would be very much appreciated.
Regards,
Peter