adversarial environment

An environment in which some actors are out to get you. Mistaking an adversarial environment for a random or cooperative environment is a common way to get yourself in trouble. Getting in the habit of asking “Is this an adversarial environment?” is probably a good first step.

Examples

• This is a common mistake in software development until you learn to think with a web application security mindset. It’s natural to build software for the people you’re trying to empower—who want your software to work correctly—but once software is out in the wild, people will eventually try to use it against you and your users.
• An example of the above was when in 2016, Microsoft released a chatbot that combined machine learning and natural language processing to “learn” from and emulate people on Twitter. They failed to consider that any public social network is an adversarial environment, and in this case bad actors fed offensive, hateful things to the chatbot which it proceeded to repeat publicly and had to be promptly shut down.
• Apparently, mistaking an adversarial environment for a random one bit Zillow to the tune of hundreds of millions of dollars

Adversarial Environments are Systems

They lend themselves to the Escalation system trap (e.g. arms races) in which adversaries waste resources competing for an edge against each other. When dealing with an intractable adversarial environment, consider the system structure (e.g. incentives, mental models) that gives rise to the opposition.

For example, natural language processing models like GPT-3 are growing adept at generating long-form text like essays, leading to concerns about plagiarism. The escalation “solution” is developing more sophisticated anti-plagiarism tools for detecting AI-generated essays, leading to an arms race pitting educators against an unknown subset of their students. If this isn’t the kind of educational environment we want, we might rethink the incentives that lead some students to value a credential itself (and risk losing it by plagiarizing) over the knowledge and experience the credential is supposed to represent.