I came up working in the Bay Area’s community technology center movements of the 1990s [which sought to expand access to computers and the early internet]. I was really excited about the possibilities of that work. But I also struggled to reconcile the utopianism of that moment with the reality of San Francisco in the 1990s. Public housing was being knocked down, the city was visibly whitening, and people were being displaced.

So I left San Francisco to escape a sort of personal, political crisis. I moved to a small town in upstate New York called Troy. And, coincidentally, I moved there just as the city decided to put all of their eggs in the basket of high-tech development.

Right. And that was the question that I started asking myself. There’s something here I can’t escape. How am I going to make sense of it?

I started doing community technology work in Troy, working primarily with poor and working-class adults out of a YWCA. A lot of people were coming out of the prison system or out of recovery. And they challenged me deeply in the ways I was thinking about technology.

There was this idea at the time that the major inequality issue in tech was the “digital divide” — you know, low-income people aren’t interacting with technology so they’re being left behind. But the people I was working with told me, “No. Listen. Technology is totally ubiquitous in our lives. I don’t know what you’re talking about.” They were seeing tons of technological innovation, just not where I was looking for it. They were seeing it in the criminal justice system, in the welfare office, and in low-wage employment.

I had this conversation with a woman who I called “Dorothy Allen” in my first book. We were talking about her EBT [Electronic Benefit Transfer] card. She told me, “Yeah. It’s more convenient. It’s great to not have to carry around paper [food] stamps anymore. But also, my caseworker uses it to track all of my purchases.” And I had this look on my face, a look of total shock. And Dorothy said, “Oh. You didn’t know that, did you?” I did not. She said, “You all” — meaning middle-class people like me — “You all should be paying attention to what happens to us, because they’re coming for you next.”

That conversation, and others like it, is where my interest in technology in the welfare system came from.

I think we tend to talk about these kinds of technological tools as if they arose outside of history. Like they just fell from the sky. One of the major arguments in the book is that these tools were more evolution than revolution. They’re very much in line with the kind of punitive policy, processes, and tools that came before them.

There’s two moments I’ll draw your attention to. First is the scientific charity movement of the 1870s, which was deeply informed by eugenics and the desire to “breed” out the moral inadequacies that produce indigence. It kicks off after the fall of Reconstruction and the violent reassertion of white supremacy that took hold then. You see the rise of Jim Crow, the exclusion of African Americans in public life, immigration restrictions that are based on scientific racism, and the involuntary sterilization of poor whites.

This all comes out of a moment where social service technologies are changing really fast; indiscriminate giving is being replaced by what was called “scientific giving.” The first “big data” set in the United States was the Eugenics Records Office in Cold Spring Harbor. It was the public arm of the eugenics movement. They sent scientists out into the world to collect these very detailed family trees that tried to track how poverty, “imbecility,” “depraved living” — all these words that they used at the time — were genetically carried.

The second moment is the rise of the “digital poorhouse,” which began in the late sixties, early seventies, really, in response to the successes of social movements in opening up public programs.

That’s right. In the mid to late sixties, into the early seventies, the national welfare rights movement was having extraordinary successes. Particularly legal successes. Supreme Court victories in 1968, ’69, and ’70 made it impossible to discriminate against folks receiving AFDC [Aid to Families with Dependent Children]; they enshrined due process rights for welfare recipients; and they prohibited “man-in-house” rules that allowed caseworkers to raid the homes of poor women to look for evidence of male cohabitants. These victories came toward the end of the sixties, and by 1971 you see these new technological tools implemented. It’s a backlash.

There’s this deep, deep, deep connection in the American mind between poverty and being a person of color, specifically being African American. You don’t get a welfare system like the one we have without using race. In each of these systems, race has been deployed as a narrative, as an organizing method.

Meanwhile, technology became a way of smuggling politics into the system without having an actual political conversation. In the 1970s, with the promise of increasing efficiency, you start seeing computerization to detect fraud and tighten eligibility rules. We think the real anti-welfare backlash started with Reagan. But the drop-off really began in the early seventies. That’s when they started culling the rolls — using these technical means. Around 1973, almost 50 percent of people living under the poverty line were receiving some kind of cash assistance. A decade later, it dropped to 30 percent. Now it’s less than ten.

At the same time the government is scrutinizing caseworkers more closely, cutting down on their discretion. As surveillance of recipients increased, so did surveillance of frontline workers. They introduced much more punitive processes, with much less human connection. And that also has continued in the systems we see today.

That was a really important part of telling the story. My intention in the work that I do is always to start with the folks who are getting the pointiest end of the stick. It’s harder, in many ways, than talking to the administrators or the policymakers. It requires me spending a lot of time with people, developing trust and developing understanding of their situations. But if we’re missing their voices, we’re missing a huge part of what this new regime of data analytics is about.

Another reason for including those stories is that I really see recipients of public services and frontline caseworkers as possible allies. And in fact, there have been a lot of historical moments where that collaboration has been really threatening to the status quo.

For example, one of the things that happens in the 1960s is the New York City welfare caseworkers strike: eight thousand caseworkers strike on behalf of recipients and their own working conditions. They say, “We’re not going back to work until you treat them better.” That’s a terrifying moment for the system.

I did welfare rights organizing for fifteen-plus years. One of the great challenges of the work is realizing that caseworkers are, indeed, the deliverers of the attitude of the system. And recipients often see them that way.

But many are also just one sickness, one period of bad luck away from being on public assistance themselves. And so there are a lot of horizontal lines there. It’s a difficult relationship, but they’re also natural allies in some important ways.

The three cases sort of ramp up in difficulty. People are very comfortable rejecting the first case [Indiana’s automated Medicaid eligibility system]. That’s clear black hat.

I can’t say what’s in his soul.

Feel free. But people struggle more with Los Angeles and Pittsburgh. And that was intentional. I was trying to step people through this process where it’s like, “Okay, bad intentions, bad outcome. We got it.” But I really didn’t want to leave people there. I thought that was too easy.

The LA chapter is frankly, really, about well-intentioned people, working hard in a situation involving an almost unbearable set of decisions. There are fifty-eight thousand homeless people, and there is just a handful of resources. And so the question that I’m trying to ask there is about the idea of triage. The algorithm is supposed to identify the most vulnerable, the most needy. But are we comfortable saying that we should triage people for their basic human rights?

One of the questions I’m trying to get at there is, are we using these technologies as empathy overrides? These decisions are too hard for humans to make. So, we’re letting computers make them for us.

We are creating these tools that just split up this pie that’s made of shit. We can create the most sophisticated tools to share out that pie, but it’s not going to change the fact that the pie is shit. I want a different pie. I don’t want a shit pie.

These are kids. We don’t want horrible things to happen to them. Caregivers do awful things to children in their care sometimes. And I believe — not everybody agrees with me — that the state has a responsibility to step in to protect children. I think the system we have now is deeply flawed, but I think it’s a reasonable thing for a state to do.

The thing that’s so challenging about that case is, in Allegheny County, they have done every single thing that progressive folks who talk about these algorithms have asked people to do. There is participatory design in the process. They are completely transparent. They publish everything. They’ve been working on racial disproportionality for a long time.

And so this is the case that I kind of think of as the worst best-case scenario. Because at the same time, I still believe the system is one of the most dangerous things I’ve ever seen.

Erin Dalton, the head of data analytics for Allegheny County, told me, “We definitely oversample the poor. All of the data systems we have are biased. We still think this data can be helpful in protecting kids.”

There’s something really deep there: it suggests we’re okay doing unethical things to poor people. You couldn’t really fill in that blank with any other group and still say, “But, we think it’s okay.”

CPS is a really complicated system, ethically, because it mixes these dual roles, of being the provider of last resort for poor families and being the agency that can investigate your parenting and remove your children. The great majority of investigations, 75 percent, are neglect, not emotional, sexual, or physical abuse. And neglect is very hard to distinguish from just the conditions of poverty.

I think most of the families experience the services they get with, at the very least, ambivalence. They feel like they are put in situations where they have to give up their children in order to get the resources they needed to care for their children. And once you’re in the system, there’s a ratcheting effect. The standard on your parenting is raised really high, and the resources aren’t sufficient to keep you at a place where you can comply with the expectations. It results in absolute tragedy too much of the time.

Even if CPS is functioning the way it should, and only taking children who are really in great danger of maltreatment out of their families, we’re also assuming that we have someplace better for them to go. I think many kids who have been through the foster care system would disagree.

That chapter is really saying, “If you do everything right, and you’re still living in a society that is deeply impacted by its hatred of the poor and its fear of precarity, you’re still going to create systems that punish, police, and profile.”

And that’s the most frustrating place that people get to. Because then they’re like, “Well then, what do we do?” And they want me to give them a ten-point plan for creating better technologies. And I’m like, “Well, what we do is, join in social movements, and end poverty everywhere, and forever.” And I’m sorry that that’s a frustrating answer. But it’s the real answer.

Part of my work is shifting the conversation from “privacy” to “self-determination.” Because privacy, frankly, just doesn’t work for the folks whose stories I tell.

Folks want some degree of privacy and resent intrusion, but it’s not the first thing they talk about. It’s not even the fifth thing they talk about. They want to be able to make the most consequential decisions in their life, for themselves: how they raise their kids, how they spend their money, where they live. I find that shifting the language in that way does make some of the pathways to solutions more clear, although the work is much bigger.

I think that the major political change that has to happen is shifting the public assistance system from something where the majority of time, money, and effort is spent trying to decide whether or not it’s your fault that you’re poor.

In the meantime, these technological systems aren’t going to stop, while we work this all out. And because we live in the system we live in, they’re going to continue to have these assumptions and analyses built into them: that poor folks are primarily fraudulent, lazy, addictive.

It’s a little bit silly, but I have this little Hippocratic oath for data designers in the book. There are two basic questions I encourage them to ask themselves. One is, does it increase self-determination of poor people? And two, if it was aimed at anyone but poor and working people, would you be able to do it? And if you answer “no” to either of those, don’t do it. You’re on the wrong side of history.