Beyond 'Don't Be Evil'

Beyond 'Don't Be Evil'

Speakers: Petra Smith

Conference: CHCon

YouTube: https://www.youtube.com/watch?v=v-NFckqZ9wE

Overview

In her powerful CHCon talk, "Beyond 'Don't Be Evil'," Petra Smith delivers a sobering and urgent message about the escalating threat of authoritarianism and how technology, and those who build it, are inextricably linked to its rise and potential resistance. Moving past the simplistic and often insufficient ethical frameworks like Google's former motto, Smith challenges the tech community to recognize its profound responsibility in shaping a future that either enables or actively combats oppressive forces.

The talk provides a critical historical perspective, drawing parallels between current global trends and past eras of rising fascism, emphasizing that such movements rely heavily on technology for propaganda, surveillance, control, and the spread of misinformation. Smith argues that the harms inflicted by these technologies are not always the result of deliberate malice but frequently stem from indifference, obliviousness, or the pursuit of profit, disproportionately affecting already marginalized communities. This presentation is a call to action, urging technologists to adopt more robust ethical frameworks, embrace holistic threat modeling, and engage in collective resistance to safeguard human rights and democratic values.

Background

▶ Watch: The concerning rise of extremist voices (1:00)

Petra Smith begins by highlighting a deeply concerning global trend: the resurgence of extremist and authoritarian voices, amplified by mainstream media and populist leaders. She notes the historical "rhyming" with fascism, where demagogues gain traction by spreading lies, advocating violence against scapegoated groups, and demanding loyalty. This context sets the stage for her central argument: these movements critically depend on technology—and by extension, technologists—to maintain paranoia, spread propaganda, surveil populations, classify, control, and punish individuals.

Smith critically examines the tech industry's historical ethical stance, particularly Google's unofficial motto, "Don't Be Evil." She asserts that this phrase is deeply flawed because "evil" is subjective, difficult to define, and sets an impossibly high bar. More importantly, she argues, "don't be" is a passive injunction. Most technological harm, Smith contends, doesn't originate from deliberate malevolence but from indifference or obliviousness to the consequences, especially when those consequences disproportionately affect marginalized groups. She identifies three key areas where these harms manifest: the criminalization of specific populations, the escalation of online harassment into real-world violence against out-groups and dissenters, and the increase in technology-facilitated family and intimate partner violence fueled by societal shifts towards traditional gender roles. The talk frames these issues not as isolated technical glitches but as systemic challenges that require a proactive and ethically informed response from the tech community.

Key Findings

▶ Watch: Critique: 'Don't Be Evil' is too passive (4:55)

Petra Smith's talk unearths several critical findings that challenge conventional thinking within the tech industry:

Firstly, the pervasive motto "Don't Be Evil" is exposed as a passive, subjective, and ultimately insufficient ethical guideline for an industry whose creations have profound societal impact. Smith argues that true harm often arises from indifference or obliviousness, not deliberate malice, making a negative imperative like "don't be evil" inadequate for proactive prevention.

Secondly, she demonstrates that technology-driven harms are not evenly distributed, disproportionately affecting criminalized people, out-groups, and dissenters. These impacts often manifest in subtle, compounding ways that are not immediately apparent to those in positions of power or privilege.

Thirdly, Smith emphasizes that algorithmic decision-making, particularly in AI and machine learning, inherently carries biases derived from flawed input data, model design, and human operators. These biases compound, leading to amplified harm, as starkly illustrated by the failures of facial recognition systems.

Fourthly, the talk reveals that the common antidote of a "human in the loop" is not a panacea. Human operators, too, possess biases and can abuse systems if not properly constrained by robust safeguards and accountability mechanisms, as evidenced by internal misuse of police databases.

Finally, Smith highlights that designing for idealized user behavior is a dangerous oversight. Real-world user habits, such as password reuse and device sharing in intimate relationships, are frequently ignored, creating vulnerabilities that are then exploited in technology-facilitated abuse. The talk concludes that effective resistance requires not just technical solutions but collective action, political engagement, and a willingness to refuse to build harmful technologies.

Technical Deep Dive

▶ Watch: Using threat modeling for societal harms (8:20)

Smith proposes threat modeling as a fundamental tool for identifying and mitigating potential harms, but stresses the need to expand its scope beyond traditional security concerns. She advocates for Adam Shostack's four questions: "What are we working on?", "What could go wrong?", "What are we going to do about it?", and "Did we do a good job?". Crucially, she emphasizes the necessity of a diverse team for effective threat modeling, acknowledging the difficulty of achieving this given current tech layoff trends that disproportionately affect marginalized groups. Where internal diversity is lacking, organizations must seek external expertise or engage with affected communities through focus groups, ensuring their input is respected, transparently used, and fairly compensated.

A significant portion of the technical deep dive focuses on the inherent biases and compounding harms within algorithms, especially those driven by Artificial Intelligence (AI) and machine learning. Smith asserts that all decision-making, human or digital, has bias. In algorithms, these biases, limitations, and weaknesses disproportionately affect underrepresented groups. She explains how biased input data, combined with a biased model, and operated by biased humans, can compound to amplify harm significantly. The speaker mentions the emergence of benchmarking tools to independently assess model weaknesses, enabling better mitigation strategies.

Facial recognition systems serve as a prime example of these compounding biases. Smith categorizes these into:

• One-to-one verification: Like Face ID on a smartphone or time-carding systems.
• One-to-many identification: Matching individuals against a watchlist.
• Behavioral recognition: Systems claiming to detect shoplifting or deception during interviews.

She explains that computer vision, the foundation of many facial recognition systems, was often developed to detect contrast between light and dark. This fundamental design choice makes it inherently less effective on darker-skinned people. This technical limitation is then compounded by biased training data, which is often "cheaply and easily assembled" and therefore unrepresentative of diverse populations. The resulting system produces higher false positive rates for people of color, particularly women of color.

Smith cites the 2017 Metropolitan Police facial recognition trial in London as a stark illustration of these failures. The trial, conducted predominantly in diverse, non-white neighborhoods and at cultural events like the Notting Hill Carnival, used a model known to have high false positive rates for people of color. The police also expanded their watch list far beyond known criminals to include past offenders, associates (family members, cohabitants), and vulnerable people (youths, those with mental disabilities). The trial resulted in a staggering 98% false detection rate, yet was declared a success. This case highlights how technical flaws, biased data, and human operational biases combine to create a system that disproportionately targets and criminalizes marginalized communities.

The discussion then moves to the critical but often misunderstood role of the "human in the loop." Smith cautions that simply having a human review automated decisions is not a panacea. The quality of human intervention matters. She provides examples from New Zealand police: a new system allowing 111 operators to locate cell phones raised concerns about abuse, despite assurances that officers "know they've been told" not to misuse it. This was juxtaposed with reports of number plate reader lookups doubling in a year with no clear definition of "appropriate use," and a specific case of an officer misusing a rape complainant's file for personal gain. These incidents underscore that human behavior, including potential for abuse, must be anticipated and designed against, not merely assumed to be ethical.

Finally, Smith addresses technology-facilitated abuse (TFA), often overlooked because it leverages low-tech methods and off-the-shelf software (e.g., commercial location tracking apps, shared accounts) rather than traditional hacking. This abuse exploits the dynamics of intimate relationships, where initial trust (the "honeymoon phase") or boundary-testing ("red flag" for not sharing passwords) can lead to later exploitation. She highlights that many software designs assume all users in a household are trusted, making it difficult to detect or revoke access when relationships sour. This demonstrates a critical failure to design for real-world user behavior, where 70-80% of internet users reuse passwords and up to 80% of adults in relationships share device/account access, with young adults even viewing refusal as a "red flag." These behaviors, while discouraged by security professionals, are common and must be factored into system design to prevent deadly consequences.

Demo / Proof of Concept

▶ Watch: Addressing user behavior in technology-facilitated abuse (22:00)

While Petra Smith's talk did not include a live demonstration or a proof of concept developed by her, she highlighted a crucial piece of research that serves as an exemplary "proof of concept" for understanding and mitigating technology-facilitated abuse (TFA). She excitedly refers to a study (accessible via a QR code in her presentation, linking to an ACM publication) where researchers engaged individuals with lived experience of intimate partner violence.

These participants, who largely did not consider themselves technical, were asked to identify problems with smart home connected devices and propose solutions. Smith emphasizes that the insights garnered from these individuals were "really brilliant" and "a lot more practical than simply telling people, 'simply never share a password' or 'simply never share a device.'" The study revealed critical design flaws, such as smart home systems assuming all occupants are trusted users, making it difficult for victims to ascertain who has access or to revoke access from a no-longer-trusted partner. This research effectively demonstrates that involving end-users with direct experience of harm can yield invaluable, actionable insights for designing safer, more resilient technologies.

Defensive Implications

▶ Watch: The option to refuse harmful technology (30:50)

The defensive implications derived from Petra Smith's talk are multifaceted, extending beyond traditional technical safeguards to encompass ethical considerations, design principles, and collective action.

Firstly, organizations and development teams must adopt a holistic and expanded threat modeling approach. This means moving beyond solely technical vulnerabilities to consider the broader context of a system, including its human users, operational processes, and the societal impact of its inputs and outputs. Threat modeling teams should be as diverse as possible to identify harms that might be invisible to homogenous groups, and where internal diversity is lacking, external expert consultants or community focus groups should be engaged respectfully and transparently.

Secondly, a critical re-evaluation of AI and machine learning systems is paramount. Defenders must proactively seek out and understand the biases embedded within algorithms and their training data. This includes utilizing available benchmarking tools to assess model weaknesses and designing systems that account for these limitations, especially in high-stakes applications like facial recognition where compounding biases can lead to severe real-world harm and miscarriages of justice. Simply deploying an AI without understanding its inherent biases is irresponsible.

Thirdly, the notion of a "human in the loop" must be qualitatively assessed. It's insufficient to merely insert a human; systems must be designed to empower human operators to challenge information and prevent abuse, rather than passively accepting algorithmic outputs. This requires clear definitions of appropriate use, robust auditing mechanisms, and consequences for misuse, as exemplified by the New Zealand police's struggles with internal database abuse. Designers must anticipate and mitigate against potential human factors that could exacerbate harm, not just technical ones.

Fourthly, security design must acknowledge and accommodate real-world user behavior, rather than idealizing it. Given the prevalence of password reuse (70-80%) and device/account sharing in intimate relationships (up to 80%), security features must be built to function safely even under these common, albeit risky, scenarios. This includes making it easier for users to manage and revoke access to devices and accounts, especially in the context of smart home technologies where a default assumption of universal trust can be exploited for technology-facilitated abuse.

Finally, Smith emphasizes a broader, more societal defense: collective resistance. Individuals and organizations should actively vote, submit feedback on harmful legislation, contact elected representatives, and refuse to do business with companies that prioritize profit over ethical considerations, such as selling de-identified location data that can be re-identified (e.g., Grinder in 2019) or profiting from disinformation (e.g., platforms requiring 32 complaints to suspend an advertiser). Ultimately, technologists may need to consider the "extreme option" of refusing to build technologies that are inherently harmful or cannot be made safe, a stance that is most effective when organized and collective, as seen in 2018 with "FANG companies" resisting certain military contracts.

Key Takeaways

• "Don't Be Evil" is Insufficient: The tech industry needs to move beyond passive, subjective ethical mottos to proactive, responsible action, recognizing that most harm stems from indifference rather than deliberate malice.
• Embrace Holistic Threat Modeling: Extend threat modeling beyond technical vulnerabilities to include broader societal context, human factors, and the disproportionate impact on marginalized groups, leveraging diverse teams and external expertise.
• Address Algorithmic Bias Proactively: Critically evaluate AI/ML systems for inherent biases in design and training data, understanding how these biases compound to amplify harm, especially in facial recognition and other decision-making algorithms.
• Design for Real-World Human Behavior: Acknowledge common user behaviors like password reuse and device sharing in relationships, building safeguards against technology-facilitated abuse rather than assuming ideal user conduct.
• Challenge "Human in the Loop" Efficacy: Recognize that human oversight is not a panacea; systems must empower humans to challenge decisions and prevent misuse, with clear accountability for potential abuse by operators.
• Engage in Collective Resistance: Individuals and the tech community have a responsibility to push back against harmful laws, refuse to work for exploitative companies, and collectively refuse to build technologies that enable authoritarianism or cause undue harm.

About the Speaker(s)

Petra Smith is a seasoned speaker and participant at the CHCon (Christchurch Hackeron) conference, known for her engaging and sometimes "whimsical nonsense" presentations in previous years. However, in this particular talk, she adopts a more serious and urgent tone, driven by her deep concerns about current global trends. Smith brings a unique perspective that blends her technical expertise with a strong historical awareness, drawing parallels between contemporary events and past eras of rising authoritarianism. She is passionate about the ethical implications of technology and challenges her peers in the tech industry to embrace their responsibility in shaping a just future, advocating for proactive engagement and collective action.