The Ethics of the Tools We Build
Systems that rank, predict or make decisions about people can cause harm even when nobody intends it. Applied ethics examines who is affected, which values guide the design, and how different moral frameworks judge the same technology.
What a learner can do afterwards
- Take one automated decision system and say who bears the cost when it is wrong
- Apply two of the three moral frameworks to the same technology case and compare their verdicts
- Explain a challenge in arguing for duties to future people or other species, without assuming those duties must be weaker
1 · Read
An error rate reports what happens. Calling a tool fair also needs a value judgment. A hiring filter trained on past decisions may repeat discrimination. Its history identifies a risk, not proof that every result is biased. Design choices, such as the score required to pass, can affect who gets an opportunity. Expressing a decision as maths does not settle whether it is fair.
Before launch, identify affected people, compare errors across relevant groups, and provide a challenge route. Define errors by the tool's task. If it only checks eligibility, rejecting someone who qualifies is a mistake; eligibility alone need not guarantee selection for a scarce place. A mistaken housing rejection can cost a rental opportunity without causing eviction. In the fictional chart, each group has 100 eligible applicants, and the bars count mistaken eligibility rejections.
Consequentialism judges choices by their consequences. A triage tool might save more lives overall yet miss urgent cases in one group more often. Compare expected benefits and harms for everyone affected, including trust and future effects. Counting only the extra lives saved leaves out other consequences. Different consequentialist theories may value outcomes differently.
Compare frameworks by applying each rule to the same options. A greatest-net-welfare rule selects the best expected balance of benefits and harms for everyone. A duty rule rejects options violating its stated requirement, such as fair treatment or informed consent. The rules may choose differently when the welfare rule favours an option that violates the duty. These are specified views, not fixed verdicts for every theory. Consent does not excuse every harm.
Future people cannot join today's decisions, and long-term effects may be uncertain. These difficulties concern representing their interests; they do not prove weaker duties. For other species, arguments may concern individual animals' suffering, species survival or ecological relationships. These are different grounds for protection. Inability to speak or consent does not by itself settle moral importance.
Identify affected people, compare group errors, and explain the consequences or duties behind your judgment. Difficulty representing future interests does not prove they matter less.
This lesson, quiz and flashcards were written for this dataset. Responsible Computing and Introduction to Philosophy informed the concept checks. The test data and trial comparison are fictional.
The readings and flashcards were written for this dataset. Linked videos are credited to Wireless Philosophy and Code.org.
More to read
What a challenge route needs to do
Imagine an applicant receives a rejection from a housing filter. The message gives no reason and no contact for asking questions. The applicant cannot tell whether the record contains a mistake or whether the rule itself needs examination.
A useful challenge route lets the person raise a concern and have it considered by someone able to change the decision. Merely adding a contact button does not show that review works. Ask what information the reviewer can examine and what happens when an error is found.
Individual review and group testing answer different questions. An appeal may reveal a mistaken record in one case. Comparing similar cases across groups may reveal a pattern that individual complaints miss. Both can matter even when nobody intended unfair treatment.
A human reviewer is not automatically fair either. The review process needs reasons and scrutiny, just as the automated decision does.
Two arguments about a software trial
Suppose a delivery company proposes a routing trial. Shorter journeys could save time, but unfamiliar routes could expose drivers to hazards. The drivers have not agreed to participate.
An outcome-based assessment considers the risks alongside the expected savings. It also asks about effects on trust and later working conditions. A claim about shorter journeys alone leaves relevant consequences unexamined.
A duty-based critic might argue that exposing people to experimental risks requires informed consent. That argument needs an account of the duty and why this trial falls under it. If drivers agree, other questions remain, including whether the risks are avoidable or excessive.
The approaches may reach the same verdict for different reasons. Both might reject this trial. To compare them, identify the consequences or duties doing the work in each argument. The framework's name alone is not a solution.
Whose interests are represented?
A decision about energy use, waste or land can affect people born later. Those people cannot attend today's meeting. Estimates of their circumstances may also be uncertain.
These difficulties do not show that future interests matter less. They raise questions about representation and prediction. An argument for protection might appeal to avoiding foreseeable harm or preserving conditions people will need to live.
Other living things raise further questions. Preventing an animal's suffering differs from protecting a species from extinction. Protecting ecological relationships is another concern. A proposal may help individual animals while failing to protect a habitat, or protect a species while imposing costs on some animals.
State what the argument aims to protect. Then separate the factual prediction from the reason that protection matters. Disagreement about what a tool will do is different from disagreement about what we owe those affected.
Explore more
2 · Watch
Take it off screen
Where it sits
Learn first
Optional connections
This opens up
Nothing builds on it yet.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.