Innovating Like an Optimist, Preparing Like a Pessimist: Ethical Speculation and the Legal Imagination<\/p>\n
by Casey Fiesler[1]<\/a><\/sup>*<\/p>\n \u201cScience fiction writers foresee the inevitable, and although problems and catastrophes may be inevitable, solutions are not.\u201d \u2013 Isaac Asimov<\/a>[2]<\/a><\/sup><\/p>\n <\/a><\/a> Introduction<\/p>\n We all tell stories\u2014to ourselves, and to others\u2014about the future. These stories typically draw us in two opposite directions: to an optimist utopia, where we imagine how things might be better than they are today, or to a pessimist dystopia where aggressive innovation leads to our destruction.<\/a>[3]<\/a><\/sup> In our current landscape of rapidly emerging technology, it is easy to jump to dystopic scenarios when we imagine the future\u2014and as individuals we often do.[4]<\/a><\/sup> Meanwhile, science fiction media like Black Mirror<\/em> also tell these stories for us, fast-forwarding technologies being developed to the point where the dream of that technology becomes a nightmare instead.<\/a>[5]<\/a><\/sup><\/p>\n Dystopias as a genre serve as cautionary tales that can warn us of what might lie ahead if we are not careful now.[6]<\/a><\/sup> These stories are particularly powerful in the context of unanticipated consequences<\/em>, where deliberate acts have effects that are unintended or unforeseen.<\/a>[7]<\/a><\/sup> By definition, negative consequences of this type are unforeseeable at the time a technology is designed\u2026 but what if they were not? Speculation<\/em> is the ability to imagine potential futures and alternatives.[8]<\/a><\/sup> And science fiction as a narrative genre of speculation can illuminate the likely social impact of change\u2014not just by criticizing na\u00efve optimism about the future, but also by providing a blueprint for a better one.<\/a>[9]<\/a><\/sup><\/p>\n For both pessimists and optimists, critique is not necessarily negative, but can be a testimonial towards how the world might be instead.[10]<\/a><\/sup> Creative speculation, as a method of ethical and legal foresight,<\/a>[11]<\/a><\/sup> can help us foresee potential consequences of emerging technologies. Subsequently, we may be able to use design, implementation, or regulation to mitigate negative outcomes. In fact, a number of scholars have called for multi-stakeholder and interdisciplinary approaches to regulation,<\/a>[12]<\/a><\/sup> and even pointed to the usefulness of science fiction and speculation.<\/a>[13]<\/a><\/sup> Moreover, much like issue-spotting and other traditional ways of \u201cthinking like a lawyer,\u201d<\/a>[14]<\/a><\/sup> creative speculation is a skill that can be practiced and taught.<\/p>\n In this essay, I begin by discussing the problem of unanticipated consequences in the design and regulation of emerging technology, pointing to the difficulty of foresight as an underlying cause. Next, I draw a line between the issue-spotting capabilities of \u201cthinking like a lawyer\u201d and cultivating foresight as a skill. Finally, I describe my own experiences with using science fiction and creative speculation in teaching ethics and policy, and argue for the usefulness of creative speculation as a tool for those who are designing, deploying, and regulating technology. I sometimes describe myself, in the context of technology, as an optimist who believes it is important to think like a pessimist. I believe that such tools can help us create the future that we want rather than the one that we fear.<\/p>\n <\/a><\/a> Unanticipated Consequences and the Challenge of Emerging Technology<\/p>\n Emerging technologies are often both high-risk and high-potential. They offer benefits to society, but with those benefits come ethical and regulatory quandaries. With this in mind, how do we simultaneously leverage an innovation\u2019s anticipated benefits while guarding against its potential harms? This question is particularly difficult to answer when we might not be able to understand the risk associated with a technology until it is suitably developed.[15]<\/a><\/sup> For example, rapid advancements in artificial intelligence have prompted alarm not just from the general public and regulators, but from the very leaders in the tech companies engaged in its development.<\/a>[16]<\/a><\/sup> Elon Musk called AI \u201cour biggest existential threat\u201d as he asked for regulatory oversight to make sure that \u201cwe don\u2019t do something very foolish.\u201d[17]<\/a><\/sup><\/p>\n While AI is not designed to produce negative consequences, it is<\/em> designed to produce the unforeseen. Artificial intelligence simulates human intelligence\u2014which means that by definition the actions it takes are not all hard-coded and known in advance.<\/a>[18]<\/a><\/sup> Even narrow AI (contrasted with general AI, still in the realm of science fiction), which is programmed to perform a specific<\/em> task, can have significant impacts on society even when applied carefully.<\/a>[19]<\/a><\/sup> The capability for AI to produce actions for which it is not directly programmed (and therefore, potentially unforeseen) is entirely intentional\u2014but the direct consequences, including the possibility of a loss of control of that AI\u2019s actions, might not be.[20]<\/a><\/sup> Even an AI agent with the seemingly harmless goal of making paperclips might have an unmitigated opportunity to effect change on the environment directly and negatively impact humans.[21]<\/a><\/sup> In other words, AI will inherently have unanticipated, if not unintended, consequences.<\/p>\n It is unsurprising, therefore, that the unforeseen aspects of AI have created ethical challenges. To address these challenges, we have seen a scrambling for AI ethics principles and guidelines from a huge variety of relevant actors\u2014from the government of Australia,[22]<\/a><\/sup> to the U.S. Department of Defense,[23]<\/a><\/sup> to Microsoft[24]<\/a><\/sup> and Google,[25]<\/a><\/sup> and even religious institutions.[26]<\/a><\/sup> Though these principles share some common features, they are still highly divergent on important matters of interpretation and application.[27]<\/a><\/sup><\/p>\n Additionally, AI raises unique legal challenges\u2014similar to those of the internet\u2014which actually resulted in a new subfield of law.[28]<\/a><\/sup> In fact, the unforeseeable poses a particularly vexing legal challenge: will legal systems choose to view the actions of some AI systems as unintended\/unanticipated, and if so, will system designers escape liability?[29]<\/a><\/sup> This type of quandary poses entirely new kinds of public risks.[30]<\/a><\/sup> Meanwhile, regulators are tasked not only with thinking about the potential consequences of the technology itself, but also about the possible consequences of regulation.[31]<\/a><\/sup><\/p>\n These ethical and legal challenges are largely created by uncertainty<\/em>, a common side effect of technological revolutions.[32]<\/a><\/sup> However, a question that often arises is whether there really was so much uncertainty, or were certain problems foreseeable? For example, consider the case of the Cambridge Analytica scandal, encompassing a number of large ethical, legal, and social issues including privacy violations and manipulation.<\/a>[33]<\/a><\/sup> Arguably, the use of personality traits by political campaigns to attempt to manipulate voters on Facebook could have been foreseeable. In fact, a 2013 paper revealed that undisclosed personality traits (e.g., introversion versus extroversion) could be accurately predicted by Facebook \u201clikes\u201d; the paper concluded by noting the \u201cconsiderable negative implications\u201d of the research.<\/a>[34]<\/a><\/sup> In the wake of the scandal, much public discourse shifted to the ethical responsibility of both technologists and platforms to anticipate potential problems associated with their technology.[35]<\/a><\/sup> Everyone involved in the design of technology should be looking for ethical warning signs, whether they can be inferred by existing data or are more speculative. \u201cI\u2019m just an engineer\u201d is no longer a valid excuse, and those engineers are expected to have considered the social implications of the technologies they create.[36]<\/a><\/sup> Even academic researchers have called for their community to \u201cwork much harder to address the downsides of our innovations\u201d without simply assuming that computing research will result in a net positive impact on the world.[37]<\/a><\/sup><\/p>\n Of course, this goal may fall into the \u201ceasier said than done\u201d category. When it comes to both the designers of technology and those who are tasked with regulating it, they cannot actually see the future. Foresight is difficult and fraught with pitfalls, including misunderstanding the potential of an emerging technology, misconceiving a scientific trajectory, or failing to predict pivotal events or innovations.[38]<\/a><\/sup> For example, few recognized how socially and commercially transformative the internet, or even fax technology, would be until it was widely adopted, nor were they prepared for the legal mischief that would follow.[39]<\/a><\/sup><\/p>\n According to Amara\u2019s Law, people tend to overestimate the impact of technology in the short term, but underestimate its impact in the long term.[40]<\/a><\/sup> In 2004, Facebook launched to students at Harvard,[41]<\/a><\/sup> and as its userbase grew, Facebook showed enough potential to attract investors\u2014but it is reasonable that at the time no one would have predicted that it might someday be so embedded in the social fabric of society that it could influence the course of elections.[42]<\/a><\/sup><\/p>\n According to sociologist Robert Merton\u2019s theory of unanticipated consequences, two of the major causes of negative outcomes when the relevant parties are well-intentioned are: (1) the inability to anticipate every eventuality, making incomplete analysis inevitable; and (2) errors in analysis that arise from methods or habits that may have worked in the past but do not apply to the current problem.[43]<\/a><\/sup> Both of these problems critically intersect with law\u2014not only with respect to a potential lack of foresight, but also because the law develops at a snail\u2019s pace compared to technology, and application of the law to new technologies often involves analogy and functional equivalence.<\/a>[44]<\/a><\/sup><\/p>\n Functional equivalence and the large role that analogy plays in case law make the perfect recipe for Merton\u2019s second challenge for unanticipated consequences, the application of habits that have worked in the past. In his discussion of the parallels between the regulatory challenges for robotics and the internet, one of the lessons that Ryan Calo draws from cyberlaw is that courts will look to how a new digital activity is \u201clike\u201d one for which there are already rules. For example, if a court is determining the appropriate Fourth Amendment protections for an email, they might ask whether an email is more like a postcard or a sealed letter.[45]<\/a><\/sup> Similarly, in a 2005 Supreme Court case, the court wrestled with whether a cable internet provider is more an \u201cinformation service\u201d than a \u201ctelecommunications service,\u201d with Justice Scalia\u2019s dissent arguing that it is analogous to a pizza delivery service.[46]<\/a><\/sup> A challenge then is that when the legal profession fails to keep in step with advancements in technology (due in part to a lack of technical knowledge), and therefore relies on less advanced technology for analogy, the application of the law may suffer in quality and subsequently result in undesirable consequences.[47]<\/a><\/sup><\/p>\n Though of course we will never be able to solve Merton\u2019s first challenge for unanticipated consequences by gaining the ability to anticipate every eventuality, ethical speculation and legal foresight can help create \u201cpathways into the unknown.\u201d[48]<\/a><\/sup> Asimov defined science fiction as the branch of literature that deals with \u201cthe reaction of human beings to changes in science and technology.\u201d[49]<\/a><\/sup> The introduction to Future Tense: Stories of Tomorrow<\/em>, a science fiction short story anthology, describes the power of science fiction to shape our reactions productively:<\/p>\n [T]he history of actual technological change \u2026 is always heterogeneous, ambivalent, growing out of and elaborating on our existing social structures and norms, cultures and values, and physical environments.\u2026 We get used to these changes quite quickly, and once we do, they become unremarkable, even invisible. A good science fiction story can help re-sensitize us by showing us people dangling over different technological precipices, or realizing their potential in once-unimaginable ways.[50]<\/a><\/sup><\/p>\n Perhaps optimists are more inclined to reimagine potential, and pessimists to dangle our possible futures over those precipices. Both are important. The idea is not to regulate now for the HAL-9000s, WALL-Es, or R2D2s that may or may not exist in any form in the future.[51]<\/a><\/sup> However, we can exercise the muscles of our imagination and avoid complacency over the changes around us.<\/p>\n I argue that the most important context for ethical speculation is as part of the design and implementation of new technology, as some small weapon against uncertainty. By the time we get to lawyers and lawmakers, it is often too late, since the regulation of disruptive technology tends to be reactive<\/em> to problems and challenges that arise out of uncertainty.<\/a>[52]<\/a><\/sup> As we consider speculation as part of education and design, however, there are lessons we can take not only from science fiction, but also from the legal imagination.<\/a>[53]<\/a><\/sup> Next, I consider how the characteristics of legal reasoning are useful for ethical speculation.<\/p>\n <\/a><\/a> Thinking Like a Lawyer\u2026 or a Science Fiction Writer<\/p>\n \u201cThinking like a lawyer\u201d is a skill one is supposed to learn in law school.[54]<\/a><\/sup> Traditionally this new way of thinking involves analytical skills, with a focus on thinking rhetorically in a problem-solving context, and in particular on the ability to inductively synthesize a legal principle from a series of cases and to analogize them to others.[55]<\/a><\/sup> One way that this skill finds its way into legal pedagogy is via \u201cissue-spotting\u201d exams that require perceiving the analogies between a fact pattern and a set of legal issues, standards, and precedents.<\/a>[56]<\/a><\/sup><\/p>\n I still remember the exam from my Torts class in the first year of law school. It began with a story (a \u201cfact pattern\u201d) that was about a page and a half long. The story ended with a plane crash, but prior to that there was a cast of potentially liable actors: a co-pilot who had had a drink before the flight, a pilot who was distracted by his affair with a flight attendant, an air traffic controller being trained on the job, a couple of rowdy passengers, the architects of a poorly lit runway, and a number of others I cannot remember. At the end of this story, there was a single prompt: \u201cDiscuss all possible torts claims.\u201d<\/p>\n Today, I teach a course on information ethics and policy, and the majority of my students are computer science and information science majors\u2014potential designers of the \u201cemerging technology\u201d of the future that one day we will find challenging to regulate. When it comes to teaching ethics\u2014a topic that very often does not have \u201cright\u201d answers\u2014issue-spotting is one of the most useful skills I can cultivate in my students. In fact, a recent analysis of syllabi from university tech ethics classes showed that variations on being able \u201cto recognize ethical issues in the world\u201d is one of the most common types of desirable learning outcomes.<\/a>[57]<\/a><\/sup><\/p>\n These fact patterns present some of the same ethical controversies that we see in the news every day\u2014for example, the behavioral microtargeting behind Cambridge Analytica.[58]<\/a><\/sup> Who were the bad actors in this scenario? What were the harms and were they foreseeable? How much did the design or business model of the platform contribute to those harms, and what responsibility might Facebook bear? What about the researchers who first determined that personal attributes are predictable from Facebook\u2019s collected data, and published a paper that noted the \u201cconsiderable negative implications\u201d of this finding?[59]<\/a><\/sup> This type of real-world fact pattern still boils down to a familiar question: \u201cDiscuss all possible ethical issues.\u201d<\/p>\n In addition to observational skills like issue-spotting, imagination also plays a critical role in legal reasoning because it fosters development of conceptual metaphors, which are more than just means of expression; they are also the \u201cimaginative means by which we receive the multiple relations of a complex world.\u201d[60]<\/a><\/sup> Like the philosophical concept of imagination, the legal imagination requires perceiving connections<\/em> between the general and the specific[61]<\/a><\/sup>\u2014or even the general and the speculative. When asking my students to imagine both the promise and the potential harms of the technology they might create, I am asking them to both extrapolate from the pitfalls of the past and to imagine uses and circumstances beyond their control. They must think now<\/em> about the consequences that they may not intend but that might, with a little imagination, be foreseeable. In recursively traveling between the general and the specific, we can choose among the possibilities and consider their moral consequences.[62]<\/a><\/sup><\/p>\n Arguments for interdisciplinarity around the regulation of technology often involve the ability to bring in greater technical expertise and to help alleviate multi-stakeholder tensions by having more people in the room from the start.[63]<\/a><\/sup> However, engaging multiple perspectives also has the opportunity to ramp up creative speculation. There have been arguments for engaging the public more with science fiction in order to increase capacity to think critically about our technological futures, as well as to promote science fiction writing<\/em> as a socially valuable profession with more direct interaction with scientists and technologists.[64]<\/a><\/sup> However, legal reasoning\u2014including issue-spotting, perceiving analogies, and extrapolation\u2014also provides a skillset that could be useful for technologists.<\/p>\n Perhaps we could create dream teams of technologists, lawyers, and science fiction writers to design and simultaneously consider the regulatory implications for the technologies of the future. However, in the interim, we can consider how creative speculation, like legal reasoning, can be cultivated as a skill.<\/p>\n <\/a><\/a> Teaching Creative Speculation<\/p>\n How best to teach ethics to computer science students or other technologists of tomorrow is an unsettled question, with a variety of pedagogical approaches represented even as the demand for such instruction continues to grow.[65]<\/a><\/sup> One approach, as exemplified in the course \u201cScience Fiction and Computer Ethics\u201d taught at University of Kentucky and University of Illinois, emphasizes \u201coffer[ing] students a way to cultivate their capacity for moral imagination\u201d through analyzing science fiction stories.[66]<\/a><\/sup> The instructors note that a key insight of this course was that \u201ca good technology ethics course teaches students how to think, not what to think, about their role in the development and deployment of technology, as no one can foresee the problems that will be faced in a future career.\u201d[67]<\/a><\/sup><\/p>\n I include analysis of science fiction texts and media in my own teaching, including stories like Cory Doctorow\u2019s \u201cScroogled\u201d[68]<\/a><\/sup> and Naomi Kritzer\u2019s \u201cCat Pictures Please\u201d[69]<\/a><\/sup> in conjunction with scholarly and news articles when covering surveillance and AI, respectively. I also have students write essays about an AI science fiction film of their choice; Ex Machina<\/em>, Her<\/em>, and Avengers: Age of Ultron <\/em>are particularly popular. However, some of my most successful teaching exercises have students not analyzing science fiction but creating<\/em> it, or engaging in further creative speculation around it. Next, I will discuss two such exercises that I first described in the online article \u201cBlack Mirror, Light Mirror\u201d[70]<\/a><\/sup> and have also taken on the road to try out in other classes and even beyond the classroom: the first an activity on speculative regulation, and the second an activity on imagining possible harms of future technologies.<\/p>\n <\/a><\/a> Speculative Regulation<\/p>\n The course I teach covers information\/technology policy in addition to ethics. I encourage students to use their legal imaginations, considering the intersection of metaphor and speculation. After we watched the Black Mirror<\/em> episode \u201cThe Entire History of You,\u201d[71]<\/a><\/sup> which takes place in a future in which every action we take is recorded (i.e., always-on lifelogging) and every memory accessible (even by others).<\/a>[72]<\/a><\/sup> When a student inquired whether this would put an end to crime, she followed up by asking if the police would have access to memories at all. Would it be an invasion of privacy? How might the Fourth Amendment apply? Would such a thing constitute unreasonable search? Someone else asked if your own memories could be used against you without your consent, or was that self-incrimination? The conversation then led us to a discussion about the FBI-Apple encryption dispute that concerned whether Apple could be compelled to unlock an encrypted iPhone,[73]<\/a><\/sup> and then I told them about the Supreme Court ruling in Katz v. United States<\/em>.[74]<\/a><\/sup><\/p>\n None of these regulatory or ethical issues came up in \u201cThe Entire History of You,\u201d which was much more concerned with the human and social consequences of the technology. However, this example highlights a feature we have established about science fiction; it can help us explore our present just as much as our future. The premise of this future technology served as a catalyst for discussing similar complexities we are grappling with today. Just as the creators of the iPhone were likely not thinking about how biometric keys might be used by law enforcement,[75]<\/a><\/sup> Alexander Graham Bell likely did not consider the legal privacy implications of the telephone.[76]<\/a><\/sup> Today\u2019s technology is yesterday\u2019s science fiction.<\/p>\n I use another Black Mirror<\/em> episode for a teaching exercise I call \u201cspeculative regulation.\u201d In \u201cBe Right Back,\u201d a young widow brings back her deceased husband first via a chatbot-like service, and eventually via an eerily lifelike robot recreation.[77]<\/a><\/sup> After watching the episode, class begins with the question: what regulations would exist in a world with this technology? If we could create robot versions of our deceased loved ones, what current laws might regulate this practice, or what new ones would be created?<\/p>\n Again, law can often be reactive in the face of new technology.[78]<\/a><\/sup> When Facebook was first launched, no one would have thought to create laws that would regulate the use of such platforms for disinformation campaigns, but after the Cambridge Analytica scandal, this seemed to be a given.[79]<\/a><\/sup> Because edge cases and counterfactuals are a critical part of legal analysis, the exercise continues with a series of hypotheticals[80]<\/a><\/sup> to shift the conversation and force students to find inconsistencies in their decisions and to follow the downstream effects of regulation. These hypotheticals raise a series of questions for the students to answer and ultimately make decisions. Is a robot inheritable property? Are there consequences for mistreatment of a robot? Who is liable for a robot\u2019s behavior? Who is responsible for its care? Can a robot hold a copyright (which nearly always leads to discussion of monkeys[81]<\/a><\/sup>)? Each decision shapes a set of laws (as well as, e.g., a Terms of Service for the robotics company) that in turn shape the social structure of the world that this fictional technology embodies.<\/p>\n The purpose of this exercise is not to think seriously about how we might regulate this technology; even if we can see the inspiration in current technologies designed around a digital afterlife,<\/a>[82]<\/a><\/sup> this is far future tech that might not ever come to pass. There are much more pressing matters for our regulatory structures to deal with right now than the potential rights or liabilities for eerily lifelike robots. However, the intended outcome<\/em> of this activity is to exercise the legal imagination, to learn to think through problems with creative speculation. Also\u2014it\u2019s fun. If students can get excited about thinking through the ethical and legal implications of some technology that someone else might<\/em> create a hundred years from now, they should be able to do the same with the technology that they are<\/em> creating right now. The next exercise takes students through an example of that process by giving them the opportunity to be science fiction writers.<\/p>\n <\/a><\/a> The Black Mirror Writers\u2019 Room<\/p>\n I think that one of the reasons Black Mirror<\/em> has been so successful is that it takes current technologies and pushes them just a step<\/em> further\u2014most often a foreseeable step, a plausible step. For example, the episode \u201cNosedive\u201d[83]<\/a><\/sup> features widespread adoption of a ratings-based social measurement tool with severe ramifications; a question like \u201cwhy would we agree with this?\u201d forces reflection about the role of social media and related technologies in our own lives.[84]<\/a><\/sup><\/p>\n \u201cNosedive\u201d also easily tees up conversations about surveillance (particularly as represented by the social credit system in China) and social media addiction and well-being. Similarly, \u201cThe Entire History of You\u201d takes on the ethical and normative implications of lifelogging and provokes memories of the failure of Google Glass. And even \u201cBe Right Back\u201d\u2014as far-fetched as it might seem\u2014begins (before the robot shows up) with a premise that is hardly science fiction at all; there has already been a tech company with the tagline \u201cwhen your heart stops beating you\u2019ll keep tweeting.\u201d[85]<\/a><\/sup><\/p>\n The common thread between these stories\u2014which anecdotally, my students count among their favorite episodes\u2014is that they take our current anxieties about technology and nudge them forward far enough to make a point, but close enough that you can still easily see the thread from here to there. They are cautionary tales not based on some distant future but based on where we might plausibly go based on the developments (and anxieties) of today.<\/p>\n Science fiction often starts with these same kinds of questions. Author Louisa Hall says that her novel Speak<\/em> began with imagining what legal, social, and corporate issues artificial intelligence might raise in the future.[86]<\/a><\/sup> Similarly, Annalee Newitz considered in her novel Autonomous <\/em>what the ACLU might think about robot rights.[87]<\/a><\/sup> And of course, Black Mirror<\/em> jumps straight to what might go most wrong with what the tech companies of today might have in development for tomorrow.