Opportunistic adversaries: On imminent threats to learning-based business automation

Abstract

False positives and negatives are inevitable in realworld classification problems. In general, machine-learningbased business process automation is still viable with reduced classification accuracy due to such false decisions, thanks to business models that replace human decision processes with automated decision processes covering the costs of introducing automation and the losses from rare mistakes by the automation with the profits from relatively large savings in humanfactor costs. However, under certain conditions, it is possible for attackers to outsmart a classifier at a reasonable cost and thus destroy the business model that the learner system depends on. Attackers may eventually detect the misclassification cases they can benefit from and try to create similar inputs that will be misclassified by the unaware learner system. We call adversaries of this type "opportunistic adversaries". This paper specifies the environmental patterns that can expose vulnerabilities to opportunistic adversaries and presents some likely business scenarios for these threats. Then we propose a countermeasure algorithm to detect such attacks based on change detection in the post-classification data distributions. Experimental results show that our algorithm has higher detection accuracy than other approaches based on outlier detection or change-point detection. © 2012 IEEE.