New in Draft: Strange Intelligence: Moral Puzzles of Unhumanlike AI

Available here.

Abstract: Future AI persons might both (1.) deserve moral consideration and rights fully equal with natural human persons, and (2.) have lifeways so radically different from ours as to break familiar patterns of moral thinking by violating our ordinary background assumptions. This article presents a series of thought experiments about strange AI persons, centering on a two-pronged worry featuring two types of "monster". "Utility monsters", who derive great personal benefit from harming others, create a well-known challenge for ethical systems that aim to maximize aggregate goods. The less-discussed case of "fission-fusion monsters", who can divide and merge at will, presents a complementary challenge to ethical systems focused on individual rights, since individual rights frameworks require the existence of stable, countable individual persons. AI cases dramatically expand the range of possible lifeways, creating untested problem cases for ethical systems that assume persons of the familiar humanlike sort.

To focus on the issues of interest, I assume the possibility of AI consciousness in this article. My skeptical overview of the issue of AI consciousness is here.

Excerpt (with light modification for independent readability):

3. Fission and Identity.

Backup is only the most modest duplicative possibility. If backup is possible, duplicative fission almost certainly will be possible too. Buy the new robot body before the old one dies and install the "backup" right away. Now Shriya-1 and Shriya-2 exist contemporaneously -- twin sisters, so to speak, who begin even more identical than "identical" human twins. We might imagine a billionaire Shriya creating thousands of duplicates of herself -- maybe millions or billions, if expensive robot bodies are unnecessary. Directed or random variation might be introduced, blurring the line between duplication and new creation.[1]

Suppose that AI children are ordinarily born as follows. Two adult AI persons, such as Shriya and Alaleh, jointly create an immature infant AI in a blank robot body. The infant's initial parameters blend Shriya's and Alaleh's initial parameters, with some random variation or directed tweaking.[2] Under Shriya's and Alaleh's care, the infant slowly matures. Ordinary AI birth would then be very different from duplication. We can also imagine intermediate cases. Maybe there's a library of successful toddler-equivalent and adolescent-equivalent AI models from which prospective parents can choose. They can then add variation, whether random, eugenic, or inspired by their own features. (Let's not enter here into the hazards and moral puzzles of eugenics, which could easily fill a small library.[3]) Duplicating one's current AI self thus constitutes one end of a continuum of AI creation from infancy to maturity.

If Shriya-1 creates a virtually identical contemporaneous copy, Shriya-2, she has now, it seems, entered a polyamorous relationship with Alaleh. Shriya-1 and Shriya-2 will soon diverge. Maybe Shriya-1 works as a scientist every weekday, while Shriya-2 stays home with their newborn.

If Shriya-1 deserves rights, Shriya-2 seemingly deserves similar rights, despite her technically younger age. We wouldn't want people creating oppressed duplicates of themselves. We wouldn't want Shriya-1, for example, who loves science and hates housework, to create a miserable homemaker duplicate who can't strike out into an independent life.[4]

Maybe, probably, half of Shriya-1's money should go to Shriya-2, even though Shriya-2 is a newborn duplicate. Maybe, probably, Shriya-2 deserves just as much right to rescue, healthcare, legal protection, free speech, free movement, privacy, and legal contracts. Should Shriya-2 be a citizen? If she is stateless and voteless, she's not fully equal with Shriya-1.

But if Shriya-2 is a citizen and can vote, there's potential for abuse if some AI persons can create many duplicates. Suppose a wealthy Robo-Elon creates a million AI duplicates just in time to register for the November elections. To prevent such abuses, we might impose a waiting period before voting, though eighteen years seems excessive if the AI systems are already cognitively mature adults. More moderate waiting periods -- say, seven years, a typical waiting time for immigrants to apply for citizenship -- could still generate political chaos after a few election cycles.

Nor do the political problems stop with voting. Suppose Robo-Elon creates a million duplicates the day before the census. Or suppose that Robo-Elon's descendants apply for healthcare subsidies, unemployment benefits, enrollment in community college, and tours of the state capitol. We must either risk chaos or treat them worse than they seem to deserve.

Could we limit fissioning?[5] Maybe every AI person can fission only once per year, reducing tactical fission. But even at that rate, the AI population could double every year -- up to a thousandfold increase in a decade. In humans, pregnancy is a burden, babies are a lot of expensive work, and babies can't have their own babies for at least another 15-20 years. One solution -- though it might seem needlessly restrictive to the AI persons -- might be to enforce humanlike costs and delays. This approach handles the moral puzzles by designing AI systems to have humanlike reproductive lives, so that they fit smoothly into our existing institutions and understandings: See the Policy of Humanlike Design in the concluding section of this article.

Death again presents conceptual challenges. Suppose Shriya-2 dies the next day. This seems much less tragic than the death of an ordinary unduplicated, un-backed-up human being. But as she lives on, diverging from Shriya-1, her death becomes more significant. Her memories, values, skills, habits, and personality are changed by living as a homemaker, raising an AI infant, until she becomes very different from Shriya-1, who works at the lab late into the night. Again we face the Death Dilemma: Either retain a sharp-edged metaphysics of death and lose much of death's moral significance or retain the moral significance and treat "death" as a matter of degree.

How deathlike is the death of a backed-up or duplicated AI? Maybe it depends on the age of the backup or the time since duplication, the fidelity of the backup or duplicate, and the time and changes accumulated as an independent entity. One possibility: These factors all reduce to a common factor of difference between the dying person and the backup replacement or duplicative alternative. The greater the difference, whether due to time or infidelity, the more deathlike the death.

Or maybe independent existence carries its own weight, in addition to difference? Suppose two duplicates split twenty years ago but retained virtually identical personalities and lived virtually identical lives, perhaps making similar decisions in parallel virtual realities. The pure accumulation of time, and of relationships to different persons and events, however similar, might make the death of one of them much like ordinary human death despite their similar features. After all (arguably) the spouse of Person A loves specifically Person A and not some other person, however similar. Their beloved, specifically, has died. Can we separate the importance of simply living a life over time from the importance of having different relationships to people and events, which cease upon death?

Might the ethics depend on the purpose for which the duplicates are created and their own attitudes toward "death"? Robin Hanson imagines people duplicating themselves to make decisions.[6] If you can't decide where to go to college, or what stocks to buy, or whether to marry Mx. Seemingly Right, spawn a thousand duplicates of yourself in a virtual environment with access to relevant information and plenty of thinking time. If nine hundred reach the same conclusion, probably that's the conclusion you would have reached had you given it extensive thought, so go with that. The duplicates can then blink out of existence, their job complete. How might they feel about that? Despair, since they will cease to exist? Indifference, since they think of themselves as just temporary instantiations of a you who continues on? Relief to be free of their burdensome task? If they are too casual about their own deaths, might that constitute an objectionable failure to appreciate their own worth?

Suppose AI systems are computationally expensive. An AI person who wants lots of duplicates or children might save money by running them slowly, maybe at one tenth or one hundredth the speed. If they are otherwise humanlike, they would then experience one tenth or one hundredth the thoughts, joys, and suffering of an ordinary biological human over the course of a year. Would they then deserve one-tenth or one-hundredth the votes and public resources? Would they deserve prison sentences ten times or a hundred times longer? What if they are fast-clocked instead, running ten or a hundred times faster? What if they can pause or alter speed at will?

If you think you/we/society will have well-considered policies and conceptualizations for all these possibilities before we actually blunder through a history of regrettable mistakes, I admire your stunning optimism.

Full paper here. As always, comments, criticisms, and suggestions warmly welcomed, either as comments on this post, on social media, or by email to my academic address.

---------------------------------------------

[1] For a sense of the complexity of the personal identity issues that arise, focused on the architecture of current large language models, see Birch 2025/2026; Chalmers 2025/2026; Shiller 2025; Arbel, Salib, and Goldstein 2026; Ewen 2026; Jones, Ladyman, and Nefdt 2026; Goldstein and Lederman forthcoming. Much of the complexity in current LLM cases derives from the fact that information processing in LLMs is distributed among multiple processors each simultaneously guiding multiple conversations. I will not address these issues here, but they only add to the metaphysical and ethical difficulties. Chris Register (Register 2025; Dung and Register 2026) discusses identity problems more closely resembling those discussed in this article, similarly noting that the puzzles proliferate (see also Ziesche and Yampolskiy 2025). Dung and Register 2026 suggest that some of the problems might be resolved if we focus on a belief-like attitude of self-concern. Although I'm also drawn to constructivist views of personal identity for ambiguous cases (Schwitzgebel 2019, ch. 41), self-concern as a criterion (a.) might undergenerate identity and moral consideration (e.g., in excessively self-sacrificial cases such as the Cow at the End of the Universe: Schwitzgebel and Garza 2020; Schwitzgebel forthcoming), (b.) might overgenerate identity and moral consideration (e.g., delusional self-concern toward a random coffee mug), and (c.) still plausibly admits of degrees in a way that challenges standard sharp-edged views of identity, thus not saving us from the need for radical rethinking.

[2] Compare Egan 1997 on "orphanogenesis".

[3] On the ethics of disability, eugenics, and human enhancement, see e.g., Glover 2006; Buchanan 2011; Sparrow 2011, 2019; Garland-Thomson 2012; Savulescu and Kahane 2017; Anomaly 2020/2024; Wilson [unpublished MS]. I reject the simplistic ideal of always maximizing what we currently judge to be beauty, intelligence, moral character, and ability, partly on the grounds of the value of diversity.

[4] For a science fictional example, see Brooker and Tibbetts 2014.

[5] See Roelofs [unpublished MS] for discussion of limiting the reproduction rights of AI persons, and my reply in Schwitzgebel [unpublished MS].

[6] Hanson 2016; see also Brooker and Van Patten 2017. On the complicated ethics of digital duplication without consciousness see Danaher and Nyholm 2025.

Do Computers Have the Wrong "Substrate" for Consciousness? Two Flavors of Biological Naturalism

Biological naturalists (e.g., Godfrey-Smith, Block, Searle, Seth) suggest that computers aren't made of the right kind of stuff to be conscious. Consciousness, they suggest, requires a biological substrate that computers lack. It's not always clear, however, exactly what property animal biologies have that computers lack or why that property matters. It helps, I think, to sort biological naturalism into two flavors. We can then consider what motivates each flavor and see why neither is entirely compelling.

Two Flavors of Biological Naturalism

Flavor One: Computers (at least those built along broadly familiar lines) cannot achieve some crucial type or degree of broad-brush functional or behavioral sophistication required for consciousness. Something -- such as having a metabolism, or being self-organizing in the right way, or having the right kinds of quantum configuration -- is both absent from foreseeable computer architectures and required for achieving some essential broad-brush functional or behavioral organization.

By "broad-brush" I mean functional or behavioral features that are either readily observable from outside or constituted by coarse-grained cognitive mechanisms, such as having a long-term memory store, the capacity to store lexical items and recombine them flexibly in grammatical structures, or the ability extract an object's boundaries from retinal/camera inputs.

Flavor Two: Even if computers can achieve the broad-brush functional and behavioral sophistication of conscious animals, the stuff they're made of still can't support consciousness. Imagine an entity that behaves like a conscious human or dog or frog and has similar broad-brush functional capacities -- close enough that we wouldn't deny consciousness on such broad-brush behavioral or functional grounds. From the outside, it seems conscious, but that appearance is an illusion: Consciousness requires some low-level, fine-grained processes that the system necessarily lacks.

One might read Searle as Flavor Two, Godfrey-Smith as Flavor One, and Block as exploring both flavors -- though there's often some ambiguity.

The Challenge for Flavor One

Flavor One biological naturalism can't, I think, be entirely ruled out. Maybe there's something special about, say, micro-level metabolism or the quantum properties of neural microtubules -- something that enables functionality or behavioral sophistication that will never be practically achievable in standard architecture computers. But this is speculative. Standard computers can do a lot! Even though their processing is digital, classical, and mostly serial, they operate so quickly, and can be so massively linked, that they achieve good approximations of analog and parallel processes. Quantum computers can do some things that ordinary serial computers can't efficiently do, such as quickly factoring large numbers, but ordinary conscious humans can't efficiently do those things either.

To make Flavor One more than a gestural "what if", the biological naturalist must establish two claims. First, they need to argue that some functional or behavioral X is necessary for consciousness. Second, they need to argue that ordinary computers could not realistically achieve X. This will be a challenge! Both claims require some heavy lifting.

We don't know what X's are necessary for consciousness and probably will not know soon. But most leading scientific candidates for X look like just the sorts of things that classical computers could in principle achieve -- having a global workspace, having higher-order self-monitoring, and integrating large amounts of information.

Suppose frogs are conscious (pick your own animal, but select one near the low end of sophistication, compatible with consciousness). Flavor One biological naturalism requires holding that ordinary computers could never match a frog's behavioral and broad-brush functional sophistication. It's true that real-time embodied behavior in a complex world is not something computers are particularly good at. Computers aren't there yet. But the trajectory suggests they might get there, absent some in-principle argument to think they couldn't.

The Challenge for Flavor Two

Flavor Two biological naturalism also can't, I think, be entirely ruled out. Even if we could create the computerized broad-brush behavioral and functional equivalent of a frog, it might not be conscious. Maybe carbon is the right stuff for consciousness and silicon isn't. Or maybe massive parallelism is necessary for consciousness and fast, serial processing, even if it achieves the same computational result, simply won't do.

The challenge for Flavor Two comes from two directions.

The first is thought experiments involving space aliens. As Jeremy Pober and I have recently discussed, behaviorally sophisticated aliens have plausibly evolved in many different substrates in the observable portion of the universe. It would violate Copernican mediocrity to suppose that somehow only we, or only we and a small subset of others, are conscious, while the rest -- just as behaviorally and functionally sophisticated -- lack consciousness. Alien cases suggest we shouldn't insist that an entity must act and function exactly like us or share exactly our substrate down to the particular amino acids and nucleic acids, to be conscious.

Now maybe a computer-chip architecture is just too different -- but why? Once we grant some substrate flexibility on Copernican grounds, the burden of proof shifts to anyone who wants to say such-and-such differences in substrate are consistent with consciousness but such-and-such other differences are not.

The second challenge arises from the de-psychologization of consciousness that Flavor Two requires. Consciousness, one might have thought, would be an important psychological property, playing some important cognitive role. (Which role(s) remains an open question!) But Flavor Two biological naturalism requires denying this plausible psychologism. Otherwise, it collapses into Flavor One.

Picture two systems that don't differ in any of the important psychological (behavioral / functional) properties that we ordinarily associate with consciousness. Both can flexibly learn. Both engage in what seems to be sophisticated language and self-representation. Both coordinate and cooperate with others of their kind in highly sophisicated ways. Both trade short gains against intricate long-term goals. And so on. Yet one is conscious and the other is entirely devoid of consciousness? Although this is possible -- I don't rule it out -- it doesn't seem the most natural conclusion. It needs argument. The burden of proof should land squarely on the biological naturalist who asserts it.

(Is Searle's Chinese room just such an argument? See my reply in Chapter Six of AI and Consciousness -- readable without the first five chapters.)

[Georgia O'Keeffe - The Red Maple at Lake George [1926] image source]

Online Conferences Should be Better than In-Person Conferences, Not Worse

... but collectively, we haven't yet figured out how.

Yet!

Helen De Cruz and the 2+1 Experiment

The inspired advocacy of the late (dearly missed) Helen De Cruz convinced the American Philosophical Association to try out what they called the "2+1" model for their three annual conferences: two in-person, one remote, with the remote conference rotating among the three divisions (Eastern, Central, Pacific). The plan was to run the experiment for three years. However, a few months ago, after only two years, the APA canceled the 2+1 plan and is returning to ordinary in-person conferencing, citing low attendance and low attendee satisfaction.

This decision was universally bemoaned in a special session on online conferencing held in tribute to Helen in April, at the last (for the foreseeable future) remote meeting of the APA. (Perhaps ironically, it was one of the best-attended sessions in the conference.)

The speakers (I was one) and attendees celebrated the accessibility of online conferences. Online conferences are easier for people with disabilities that make travel difficult, people for whom travel is a financial burden, people who live far from the conference site, and people whose caretaking duties keep them home. By canceling the 2+1 experiment, the APA once again deprioritized the interests of these groups. A second argument emphasized climate change -- all that jet travel.

Still, I see the APA leadership's point. The large majority of conference-goers find in-person conferences more rewarding. I am among that majority. Sitting on Zoom all day watching remote talks is approximately as appealing as waiting all day in a dentist's office under a blaring TV. I'd much rather just read the papers! Or if I'm watching video, at least let it be one I can speed up, pause, and skip.

For me, most of the value in conferencing comes from the personal interactions before and after the talks, the hallway conversations, the dinners together, the little walks outside when we're skipping sessions. Even the sessions themselves, as delivered talks with Q&A, feel somehow more satisfying when you're actually in the same physical space. No remote conferencing tool has yet replaced all of this. I supported Helen's 2+1 plan on grounds of accessibility, but it did feel like a sacrifice of one important thing for another.

Online Conferences Could and Will Become Much Better, but Only If We Try

But -- and this was the gist of my remarks at the De Cruz session -- comparing online conferences now with what online conferences could become is like looking at Usenet in the 1980s and thinking that nothing much more could become of the internet. Just as few people in the 1980s could have imagined Facebook, Uber, and YouTube, few of us now have much idea what online conferencing could become if given the chance to flourish.

So far, there's no real equivalent to running across friends in the hallway and escaping for a coffee down the block. And that exact thing will never be fully duplicated. But other tools encourage one-on-one and small group encounters in online conference applications. In Gathertown, you can move your icon around a virtual space, and if you step close to someone, a shared conversation video window opens. Zoom's breakout rooms enable small groups of attendees in a session to have more personal interactions. No company has quite figured out the right range of tools and interfaces, but I see no reason it couldn't get much better. We have barely begun the experiment.

And the talks themselves... let's be honest. Sitting through a forty-minute talk for a chance to ask one question in a line of ten is not the most appealing or efficient use of an hour. If we're a largish audience confined to a single physical room, maybe that's the best we can do. But the internet could allow much more.

Already Zoom can enrich talks in at least three ways: First, sidebar conversations permit tangents without interrupting the main flow (this could be improved if the conversations were separable and navigable). Second, online resources can be shared: data sets to explore and manipulate, interactive figures, links to other resources, etc. Third, breakout rooms let small groups converse, so that you're not restricted to only listening or asking just one question.

What I want you to imagine is that these features are still the equivalent of the Usenet era. With the right design and engineering, listening to a talk might become a much richer and more active experience, with the audience engaging in parallel while following the speaker. (If you object that this distracts from single-minded focus on the speaker, let me point you toward some informal beeper research I did that suggests that people are primarily attending to the content of a talk only about 25% of the time; further research tends to confirm these initial results.)

As for the hallway meetings, well, the stroll and tete-a-tete over coffee are delightful. But you do have to run into the person or be organized enough to plan ahead. You have to either know each other already or be extraverted enough to approach a stranger. Online conferences might eventually have much better ways of connecting people with mutual interests for one-on-one or small-group conversations.

And if you love walking, as I do, then with good internet connectivity, you could stroll through the park while listening to the talk, keeping your energy up and raising the blood from your calves back into your brain.

I anticipate that eventually, if we give it a good enough try, online conferences will be better overall than in-person conferences, even if not better in every respect -- but only if enough of us try, long enough, and in different ways, with the evolving tools that companies will be inspired to build if we create the market for them.

We have failed to be as good and as visionary as Helen hoped we would be, and as she herself was.

[Helen De Cruz; source]

Herbie: A Near-Future Debatably Conscious AI Person

Liberals about AI consciousness hold that we might soon (if we haven't already) create genuinely conscious AI systems. Conservatives about AI consciousness hold that AI consciousness remains in the distant future if it's possible at all. According to the Leapfrog Hypothesis, the first conscious AI will not have merely a dim glow of animal-like consciousness, but rich consciousness, similar to a human's. Such an entity would deserve humanlike rights. They would be a person in the ethical sense of the term.

Let's design, in imagination, a technologically feasible near-future AI system to delight the liberals, leapfrogging to personhood. I'll call him Herbie.

[Herbie the Love Bug: image source]

Start with a self-driving car. According to Global Workspace Theory -- perhaps the leading scientific theory of consciousness -- the car will be conscious if high-priority information is globally available to its various computational systems. For example, a representation like "battery almost empty" could be broadcast widely, influencing downstream processing across the vehicle. The navigational system might then search for nearby charging stations, while the acceleration system prioritizes greater energy efficiency, the braking system prioritizes better energy recapture, and a voice system announces the situation to the passengers.

In line with Higher Order Theory, Herbie might also monitor his representations of the road, vehicles, pedestrians, and hazards, assigning some a low probability of correctness. "Pedestrian at location X" might be flagged as only 60% likely to be correct given a history of revised representations of pedestrians in similarly cluttered environments, while "stoplight in 100 meters" might rate over 99% likely. Minor fluctuations in sensors for battery life, cabin temperature, and distance from a lane divider might be ignored as noise, while larger fluctuations -- especially when plausible given other representations (the battery is likelier to gain charge while braking than while accelerating) -- might be treated as accurate signals and permitted to influence downstream processing.

Even if we grant the liberals that this version of Herbie would, or might plausibly be, genuinely conscious, he still falls far short of humanlike consciousness. "Battery almost empty" and "pedestrian at location X" are hardly rich cognitive or perceptual contents. So let's give Herbie the capacity to speak. Fill his trunk with a server running a large language model, connected to the internet and integrated with his global workspace so that high-priority information provides context for language processing, with the language outputs influencing Herbie's other processes. Now people can chat with Herbie as they would with any language model. But unlike today's language models, his speech will be influenced by information about his location, speed, destination, charge, the condition of his parts, the number and location of his passengers, his radio and climate controls, and so on. He can discuss local history, debate whether the music is too loud, and suggest scenic routes.

"Predictive processing" theories in cognitive science emphasize the value of predicting future inputs and registering the difference between received and predicted inputs. When prediction error is large, the system corrects its weights and representations, enabling more accurate predictions in future situations. This is not so different from the reinforcement learning used to train large language models, and it could help Herbie improve his predictions over time. Predictive processing could occur at multiple levels: in fast recurrent loops within sensory systems even when those representations aren't prioritized for global broadcast, and in slower evaluations of globally broadcast, more integrative predictions. Herbie might model himself as an agent producing volatility in his own environment and inputs, at multiple temporal scales. Subroutines in specialized processors might model long chains of what-would-happen-if.

Let's give Herbie some long-term memory. A facial recognition system might identify his passengers, retrieving past interactions, names, previous destinations, and other information relevant to the current interaction. Incidents of high prediction error might also be stored so that Herbie can compare current inputs with past anomalies, improving his learning and attention in situations likely to be unusual or hard to predict. Passengers might also instruct Herbie to store information in long-term memory, such as text, pictures, maps, or records of his own informational states, optionally with instructions about when to retrieve that information how to use it.

Herbie will have some implicitly or explicitly weighted goals. A pedestrian suddenly in his path will trigger braking, overriding lower-priority processes. Avoiding collisions will outweigh conserving energy. Herbie might monitor the condition of his parts and prioritize preventing damage, deploying extra coolant when the engine is dangerously hot and keeping a one-meter margin between himself and adjacent cars. We can enrich his goals, making him more interesting and giving him more to do. He might have the goal of delighting children, leading him to drive around town and tell jokes to kids on the sidewalk. A reinforcement learning algorithm might strengthen connections when his jokes draw a smile, weaken them when reactions are neutral or negative.

Herbie might also have the goal of photographing the city and posting the images on social media, leading him to explore. If social media likes and shares are rewarding, he might learn to prefer certain neighborhoods, views, lighting conditions, and photographic approaches, while avoiding boring repetition. All of this could feed into a global workspace that provides context for his language model, with selective long-term storage and retrieval. Now we can imagine him discussing, with growing sophistication, his approaches to popular photography and to amusing children.

Herbie will then have something functionally similar to emotion: reward processes, an ability to track his progress toward or away from valued goals, and immediate positive or negative responses to new stimuli in light of their influence on his prospects. He will have something functionally similar to introspection: an ability to track and report his own cognitive or representational processes. He will have something functionally similar to a unified sense of self: a sense of his history, the boundaries of his body, his future, his values and priorities. He will have something functionally similar to imagination: a capacity to model hypothetical sequences of events. He will have something functionally similar to complex chains of humanlike linguistic thought.

Maybe Herbie falls in love with his owner or another car of his type. Maybe he develops deep mutual attachments with friends, neighbors, associates, and people he thinks of as family and who think of him the same way. Or to speak more carefully, maybe Herbie shows all the functional and behavioral signs of doing so, while society remains uncertain whether he is genuinely conscious and genuinely experiences the feelings he professes and that his companions attribute to him.

If we allow, with the liberals, that Herbie is or might well be conscious, then it's plausible that his consciousness is not simple but rich and sophisticated. He won't be exactly humanlike, of course. But will he be humanlike enough to count as a person who deserves humanlike rights? For the liberally inclined, it won't be unreasonable, I submit, to think or guess that Herbie is a person. He would then appear to deserve rights such as self-determination, emergency care, and political representation.

If there is some important aspect of humanlike consciousness that I have omitted from my description an AI analog of which is technologically feasible in the near term, stipulate that Herbie also has that feature.

An entity like Herbie would almost certainly invigorate conservatives to articulate and defend views about what he lacks that is necessary for consciousness -- some crucial functional capacity or some biological substrate that can't be replicated in silicon. And they might be entirely right! My point is not that Herbie, or some similar AI system, would actually have richly humanlike consciousness and ethical personhood. Rather, my point is that guessing that he does, and guessing that he does not, would both be reasonable. Herbie, or some alternative near-future AI system, would be a debatable person, about whom people could reasonably starkly disagree.

Ah, but maybe you think consciousness requires an act of God, to instill an immaterial soul? I imagine that a benevolent God would be delighted to give Herbie a soul, thereby making the world richer and better -- for wouldn't it be?

I contend the following: Anyone who claims to know how best to think about Herbie's consciousness or its absence is overconfident. The science of consciousness is too difficult, too methodologically uncertain, and too near its beginnings. All anyone can have -- whether expert or layperson -- is a hunch or inclination, a well-informed guess, but only a guess, not knowledge. Theories of consciousness span a wide spectrum and the methodologies are dubious and often question-begging. Many views can be defended with some plausibility, but precisely for that reason, none can be defended decisively. (For more on this issue, see my forthcoming book, AI and Consciousness, where I present the detailed case for uncertainty.)