Michelle De Mooy, CDT:
At the heart of what might be called the “unintended consequences” problem is that so many digital platforms were designed absent any user-centric values, such as privacy and security. Instead, they were built to accommodate growth and monetization, strategies that render human data disposable and interchangeable. Put simply, platforms are optimized for economic gain, which creates systems that prioritize maximum data collection while ignoring what might be in the best interest of users.
Ethical products and services cannot rely on “checking a box” for the use of customer data because such blanket consent ignores difficult questions about user expectations and the unique risks that data exposure might cause for one individual or group over another. Today’s notice and consent mechanisms have become compliance tools for avoiding liability.
Opacity is a common feature of design when it is detached from user-centric values. It is by design that it’s nearly impossible for most people to know what’s happening to their information in digital systems, no doubt because many people express intense discomfort when they learn how and why their data is used by companies. It’s telling that an entity like the U.S. military, where the privacy of information can be a matter of life or death, was seemingly blindsided by the implications of Strava’s heat map. Opacity permits company practices to become misaligned with user expectations, as well as societal and community norms.
When platforms collect and use customer information absent a foundation of ethics and user-centric values, they often develop products that run counter to their users’ interests and that will invariably produce backlash for the company. The data ecosystem has given commercial entities unprecedented power over individuals, whether it’s through exploiting our neural vulnerabilities to grab and hold our attention, filtering the content we see based on our past actions, or limiting our ability to make truly autonomous choices about uses of our data. Companies should balance this power dynamic by creating digital products that prioritize human rights, like privacy, over profit margins, and that make stewardship, not opacity or obfuscation, the most prominent design feature in their products.