design software

Six Qualities of Benign Technology

Derek Meer

Derek Meer
Posted on 10 Oct 2020 · 4m read


This is a follow-up to the humane design article in which I attempt to enumerate the qualities I try to build into my projects.

Valuable, not Engaging

People use products with intention. Their intention depends on the value the product provides them, but it is never to spend all their time using the product. Yet this intention pervades modern software, since many products optimize for user engagement. Engagement cannot accurately measure value; we treat it like it can.

If engagement is the best we can do, we should do better. I'd like my software to determine user intentions, honor those intentions, and get out of the way. Users are customers, not resources; products are tools, not power plants.

Transparent, not Automated

Modern software requires automation. Without it, even small-scale platforms would overwhelm their maintainers. To deal with this pressing issue, we've developed ways to automate almost everything.

But do we need to? Automation without thorough testing can have side effects: fragile systems, overwhelming bills and poor transparency. This becomes a huge issue with deep learning "black boxes"; they make decisions that not even the maintainers can explain. Poor automation decisions without good explanations erode user trust, which is difficult to recoup.

Good products require trust, which requires transparency. To that end, my projects should not use "black box" automation. Any automation requires clear policies and up-to-date contacts in case of failure.

Sustainable, not Viral

Many products today follow a venture capitalist playbook: start fast, grow quickly, raise money, and exit. People take this route because of survivorship bias. Far more products fail using this method than succeed. Beyond that, it is unsustainable for more than a few years.

My products should forego this approach. They should grow slowly, yet sustainably. I don't intend to "exit"; I want to build products that people will use for decades, and I want to work on for decades. High growth means I re-evaluate potential harms. I don't want to have venture capital backing they intend to stick around long-term.

Thoughtful, not Trendy

People prefer short-term over long-term benefits — it's in our nature. Naturally, people tend to adopt trendy technology that solves their immediate problems before they've considered its long-term consequences. It might create accessibility issues, impose bad design decisions, or enable authoritarians to harm vulnerable populations.

I want my projects to temper technological progress with proper research. I want to develop new and interesting technology while ensuring it's used responsibly. I should talk to customers from diverse backgrounds, groups outside the initial target demographic. I should then publish this research for feedback from independent parties. If I cannot properly mitigate a technology's potential harms, I won't use it. If I move forward and see unintended consequences, I'll work to correct those and put measures in place to avoid repeating our mistakes.

Equitable, not Universal nor Targeted

Products tend to start life as targeted efforts. That is, they solve problems for a specific niche. As these products gain adoption, however, other groups may use them for unforeseen or even malicious purposes.

You could take the opposite route and build a universal product to treat all users equally. Unfortunately, one size does not fit all. Certain groups have special needs and contexts which universal products fail to address.

A balance between these two options is an equitable product. These products include features, accounting for how different groups might use them. I want to create equitable products by listening to diverse voices in product development and advocating for diversity on all projects. I believe all users deserve equitable treatment, and I work with others that recognize this.

Mindful, not Important

Many products today treat every event like an emergency. Pinging users every five minutes distracts them, reducing focus and productivity — things I value.

If my products notify users at all, they should only do so to gently remind them of their intentions, give them what they need to make good decisions, and encourage them to get on with their lives. They allow users to respond when convenient rather than immediately. They recognize their relative unimportance.