š Attention is All You Have
Alice Girard Guittard:
Before recommendation algorithms were a thing, you had to decide what you would be doing on the computer.
You didnāt really have one big app that you could open and order it to entertain you. Instead, you had a few dozen of bookmarks to websites, each with a specific idea in mind. A site for video game news, that one website with lots of tutorials, a blog about anime that didnāt update often enough, a wiki about a TV show from the 90sā¦
Of course awful things existed on the web. We had Encyclopedia Dramatica and Rotten.com, but you actually had to put the effort to go there if you wanted. Nobody was going to put pictures of dead kids and far-right propaganda as a suggestion after a pancake recipe or a cat video.
The good thing is that this intentional internet is still around. It has just been a bit buried below the corporate web, but itās not very hard to find. After all youāre on this blog, so you probably already have a good idea about it.
I was lucky enough to grow up alongside the Internet. RFC 791āthe one that defines IPv4āwas published the same month I was born. When I was growing up, the only option was to be intentional when using the computer, so that's what I learned to do. I loved it.
By the time big and popular social media came around it was so different from my tried-and-true approach that it never really stuck. Sure, I've dabbled with social media over the years, but my use and engagement have always been low.
I prefer things like RSS feeds, newsletters, and Tumblr (which is, I guess, technically social media in its own unique way) over anything with an algorithmic feed. To put it another way, I prefer to select who has access to my time and attention instead of letting an algorithmic feed wash over me.
To be clear, I don't think algorithmic feeds are all bad. They can be quite useful for discovering new and interesting things, but you have to be careful about the algorithms you trust (which are few and far between these days) and use them in moderation. Given when and how I grew upāspending a lot of time on the computerāthat's easy for me, but it's not easy for a lot of people with different experience, like a lot of people younger than me. I mean, hell, a lot of those folks probably don't even realize there are other options.
That's a shame. A lot of people would benefit from being more intentional when using their computers, phones, and the Internet, but it's hard to demonstrate the benefits when the algorithms are so compelling. Sometimes you get lucky and the algorithm surfaces some great stuff, but those algorithms have their own agendas, and those agendas do not have your best interests at heart.
Being a passenger can be fun, but there's a lot of amazing stuff out there you'll never find if you don't spend some time in the driver's seat.
āļø More on Live Rewind and Siri Recap
If you missed it, please read my previous post about these features before continuing.
John Gruber at Daring Fireball regarding Live Rewind and Siri Recap:
Some people are objecting to the very existence of these features as privacy intrusive. I think theyāre pretty reasonably thought out. There are surveillance cameras all over the place now. I was walking home from dinner here in Philadelphia a few weeks ago with a friend visiting from out of town, talking about this, and he stopped midway between the restaurant and my house and asked me to count how many cameras we were on at that spot. I think we counted seven or eight. Who knows how many we missed? Iām not saying thatās good or bad, itās just the world today. So too itās soon going to be with personal devices recording audio. This is inevitable. Iām glad to see Apple ship features like this early, with pretty limited scope, to provide good examples before other companies ship devices that record and save audio continuously throughout the day. I think itās better for Apple to lead with privacy-minded versions of ambient transcripts and AI-generated conversation summaries than to follow after other products have hit the market with all-day recordings and whole-life transcripts.
Think back to Appleās announcement of AirTags. A lot of people had a perfectly reasonable āHey I donāt know if itās a good idea for a product like this to even existā initial reaction. But having Apple lead the way actually proved good. I suspect Live Rewind and Siri Recap will turn out like that.
"There are surveillance cameras all over the place now," and, "Iām not saying thatās good or bad, itās just the world today," line of thinking is a perfect example of why Apple shouldn't be helping to further normalize having surveillance devices everywhere.
I also sincerely hope that John is wrong about ubiquitous audio recording by personal devices being inevitable. There are a lot of ubiquitous bad things that once seemed inevitable or entrenched, but have nonetheless been greatly reduced or eliminated for the public good. Leaded gasoline, ozone-depleting chemicals, and smoking come to mind. We collectively decided we didn't want to all breathe tobacco smoke everywhere we went, and I hope we all collectively agree we don't want a bunch of watches listening to what we say everywhere, too. Just because we have to put up with something bad now doesn't mean we have to put up with it in the future. The world is what we make of it.
That said, like the examples above, we may have to live with the bad for a while until we decide to eliminate it. I would like to live in a world where Apple proudly and loudly refuses to create surveillance features like Live Rewind and Siri Recap, and then have the rest of the industry follow their example, but I recognize that's probably not the world we live in.
Apple building and releasing privacy- and security-focused versions of these featuresāas flawed as the specific implementations areāis the safe move. I agree with John that it may result in better implementations of similar features from other companies. A tangible release people can actually use, explore, and purchase is almost always going to be more compelling and persuasive (especially to competitors) than an ideological refusal to cross a line. If Apple tried to set a good example by refusing to build these features, the rest of the industry would probably ignore them and build (or, in some cases, continue to build) surveillance features that are privacy and security nightmares.
Probably, but not certainly.
At some point there will be a catalyst for change that nudges the industry in a better direction. Apple might be influential and powerful enough to be that catalyst. Yes, being the change you want to see in the world is a risk, and I see why Apple is choosing the safe path, but I'd rather see them draw on some of their courage and take a stand.
Back to the rest of John's thoughts on Live Rewind and Siri Recap:
Itās also interesting to think about why these two features are only on Apple Watch, not iPhone. I think itās because once an Apple Watch is on your wrist and unlocked, it remains a ātrusted deviceā until you take the watch off. An iPhone is only a trusted device while itās unlocked. Itās locked, and thus untrusted, while itās in your pocket or bag, or even just sitting on a desk. I leave my iPhone unattended in our kitchen all the time. Iām often not in the room for conversations that take place in the kitchen while my iPhone is there, sitting on the counter. I shouldnāt get summaries of those conversations. And, thinking more nefariously, I shouldnāt be able to hide my iPhone on or near someoneās desk, or in a conference room, or in a car, or anywhere else, to get Siri Recap summaries of conversations where Iām not present. Making this an Apple Watch exclusive feature avoids that. Thus, I think Apple Watch can be trusted to ālistenā ambiently, all the time while itās unlocked on the wearerās wrist, in ways that even an iPhone canāt.
That's a good callout, and something I didn't think about until now. If your iPhone could also listen that would indeed make Siri Recap far worse. Making the listening part of the puzzle exclusive to Apple Watch is another point in Apple's favor, but I still maintain that there shouldn't be a feature like this on any device in the first place.
š Features that Shouldn't Exist: Live Rewind and Siri Recap
Apple recently announced Audio Intelligence on Apple Watch, an umbrella term for several audio- and AI-related features. Some of these features provide clear benefits, utility, and have little to no downside, like Sound Recognition:
Sound Recognition detects important sounds like sirens, alarms, and doorbells, and notifies you ā even when your iPhone is not with you. This is especially useful for people who are deaf or hard of hearing.
Sound Recognition is new to Apple Watch, but has been available on iPhone since iOS 14. HomePods also have the ability to send an alert when they hear smoke and carbon monoxide alarms go off. Great, useful stuff that even has the potential to save lives.
Music Recognition is another nice feature:
Music Recognition with Shazam automatically detects the song playing in the background and displays the title and artist name in the Smart Stack, all without tapping the widget.
It's not going to save lives, but it's nice to have. Shazam has existed in various forms for many years, both before and after Apple bought it, and the new implementation on Apple Watch sounds handy and convenient.
The new Live Rewind and Siri Recap features, on the other hand, aren't clear-cut wins. These features do provide clear benefits and utility, but they also come with significant downsides.
Here's Apple overview of Live Rewind:
Live RewindĀ¹ recalls what was just said in the last 15 seconds by showing a text snippet on your Apple Watch when you double-press the Digital Crown, so you can catch something you missed or are less familiar with. You can ask Siri about the content of the snippet or simply save it to the Siri app.
And Siri Recap:
Siri RecapĀ¹ ambiently takes notes on your conversations throughout your day and creates high-level Apple Intelligenceāgenerated summaries that you can use later to jog your memory, so you can stay present in the moment. Your summaries auto-delete after 7 days if you choose not to save them.
The Ā¹ in both of those overviews points to this footnote:
- Siri Recap summaries and Live Rewind text snippets are generated using Apple Intelligence and may be incomplete or inaccurate. Important details may be omitted, misunderstood, or summarized incorrectly. When accuracy matters, confirm important information with the people involved or another reliable source. Siri Recap is designed to limit certain potentially sensitive content, such as financial information, government-assigned identifiers, and more, but automated filtering may not remove all sensitive information.
When you look at the more detailed privacy overview Apple provides for Audio Intelligence, it's clear they put a lot of effort into making these features very secure and private. They even created special hardware for it: the Secure Exclave. Audio captured by Apple Watch is only processed in the Secure Exclaves on Apple Watch and its paired iPhone, and that audio is not accessible to other parts of the system on either device. For Siri Recap, full transcriptions are condensed on-device, and only the condensed version is sent to Apple's Private Cloud Compute for summarization. Everything is end-to-end encrypted, including on-device encryption with periodically rotating encryption keys.
That's all great, and demonstrates a level of effort around privacy that no other company I'm aware of would even come close to if they implemented something similar. Unfortunately, it's not enough.
Apple Watch is a very popular product. Apple sells millions of them every year, and if you're looking for them when you're out and about you'll probably see several being worn by those around you. That kind of ubiquity imparts a certain social and civil responsibility on Apple, including the responsibility to not turn these devices into surveillance equipment.
It doesn't matter that everything is encrypted, secure, and only accessible by the Apple Watch's owner. What matters is that this kind of surveillance, at this scale, shouldn't be something the only big privacy-focused company in the tech industry should be offering to its users. Apple should be setting a better example for their peers and the world at large.
To put it another way: a device commonly worn by a significant fraction of the population should not be able to listen with the intent to create a recordābe it a verbatim or summarized recordāwithout providing the following:
- Clear indications to everyone in listening range that they're being listened to
- A reasonable way for them to opt-out of being listened to
Neither of these features are provided by Live Rewind or Siri Recap. Live Rewind comes closest; it emits an audible and visual indicator, but only when it's used:
How Live Rewind respects those around you
When activated, an audible chime plays from the speaker on your Apple Watch, even if your Apple Watch is on silent or you have headphones connected. A full-display animation and microphone indicator appear on the Apple Watch display, and a distinct double-press gesture is required to activate Live Rewind. This is to alert those nearby that the feature has been activated.
This only lets people know what they said was transcribed after the fact. It doesn't provide a way to opt-out of the transcription happening in the first place.
Letting people know it happened after the fact is better than nothing, but Siri Recap doesn't even do that:
How Siri Recap respects those around you
By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers. The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone.
While Siri Recap doesn't create a recording, it does create a record. It's a condensed, summarized record, but a record nonetheless. There's no way for those being listened to to know when that record is being made, how accurate that record is, and that's no way for them to opt-out.
The socially responsible way to implement Live Rewind and Siri Recap would include continuous visual and audible indicators when a device is listening. Those indicators would explain, via text on the screen and a clear speaking voice, that the device is listening. These indicators would also provide a way for everyone who sees or hears them to easily opt-out of being recorded.
For example, when Apple Watch is listening, the display could pulse red and have a, "I'm listening. If you don't want me to listen, say 'stop listening' at any time." And, because Apple Watch is often concealed by a sleeve, body, table, or other object, it should also audibly announce the same message periodically while it's listening.
The counterargument, of course, is that constant visual and repeating audible indicators as described above would be too annoying and impractical to implement. And that's true, they would.
The problem is that when Apple realized that impractical measuresālike those described aboveāare the only way to responsibly implement features like Live Rewind and Siri Recap, they didn't abort. Instead, Apple built those features anyway, and decided or figured out a way to justify too many social and ethical compromises. Apple decided it was okay to compromise on consent. Apple decided it was okay to limit opt-in and opt-out privileges to the owner of the device that's listening. Apple decided it's okay for the people being listened to by its device to not have opt-out or opt-in privileges (many of whom are, ironically, probably also Apple users).
They key difference between Live Rewind and Siri Recap versus other listening features like Sound Recognition, Music Recognition, and devices listening for wake words or phrases like, "Siri," or "Hey, Siri," is the intent to create a record of what happened, of what was said. Knowing a fire alarm went off, that a particular song was played, or that a wake word was used when no other details are recorded are very different applications of listening technology than writing full transcripts or producing detailed summaries.
Again, it doesn't matter if what's produced and saved is a verbatim recording, full transcript, or a summary. A detailed record of some kind is still being made, and it's being made surreptitiously by design.
If you're thinking that your own memory, in its own way, records and summarizes everything you experience, how is that different than Siri Recap? Apple is thinking along the same lines in the quote I shared above. Here it is again with emphasis added:
How Siri Recap respects those around you
By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers. The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone.
The implied reasoning is that Live Rewind and Siri Recap are nothing more than technological extensions meant to augment our inherent human functionality. They're designed in a way that lets them go just a little bit further than what we'd be capable of on our own. Live Rewind lets you peek back at the last 15 seconds if you were distracted. Siri Recap lets you stay focused instead of trying to manually take notes at the same time. So what's wrong with that?
There are two problems: the lack of disclosure and the precedent these features set.
It's one thing to know you're being listened to by a person, but it's another thing to have a device listening. People inherently know how other people work. We have millions of years of evolution to draw on when dealing with other people. It's also obvious when someone is or isn't taking notes about what's being said.
On the other hand, not everyone knows how Apple Watch works, or what its capabilities are. How much better or worse can Apple Watch microphones hear than a human? How will that change in the next model, or the one after that? Is the person sitting at the next table wearing the Apple Watch that listens all the time or an older model? How much of what was said will be dropped when the audio is condensed, or what will be incorrectly represented in the transcript or summary that's produced? What happens if Private Cloud Compute isn't as secure and impenetrable as Apple claims it is?
The lack of disclosure of the device's capabilities, and when it's listening, are huge omissions. By making these features available, Apple is essentially putting the onus on the owner of every Apple Watch with these capabilities to use them responsibly, let everyone around them know their device is listening, and provide them the option to opt-out. That's irresponsible because doing those things is impractical for the same reason the persistent visual and audible alerts Apple could have built are impractical. If you have Siri Recap enabled you're not going to announce it to the crowd every time you do things like enter a store or step on an airplane.
Then there's the precedent these features set. Appleāwho again is the only big privacy-focused company in techāis setting a really bad precedent at a really bad time by creating and releasing always-listening-to-everyone features like these. The existence of these features helps normalize persistent surveillance at exactly the point in our history when we should be doing the exact opposite.
Other tech companies certainly don't need permission or approval from Apple to build always-listening-or-watching features, but the existence of Live Rewind and Siri Recap make it easier for them to build and sell similar features. The potential customers of those companies will be more accustomed to and accepting of these kinds of features if Apple has them, too. This goes double for surveillance-focused companies and other surveillance apparatuses, like governments. Many of those entities want as many people as possible to be as comfortable as possible with constant surveillance, to the detriment of the public at large. A world where most people are used to and accepting of persistent surveillance is a very dangerous world to live in. By releasing Live Rewind and Siri Recap, Apple is helping us all take one big collective step closer to that world.
The other surprising thing about these features are the risks Apple is willing to take to implement them. One scenario is that there's a security issue or data breach surrounding these features, where sensitive transcripts or summaries end up in nefarious hands. I don't think that's very likely, but it's certainly possible.
A more likely scenario is that these features, once more people become aware of their existence, tarnish the reputation of Apple Watch and Apple itself. Today, most people who take note of someone wearing an Apple Watch probably do nothing more than think things like, "that's a cool band," or, "I see those everywhere, maybe I should learn more about them." In a world where a news story explains that new Apple Watches are listening to and summarizing everything they hear, people will start thinking, "I wonder if that thing is recording me right now?" Yes, they'll technically be wrong about it "recording" them, but that won't matter. What matters is their perception, and what they'll perceive is that they're being surveilled, and that Apple Watch is creepy. And they'll be right on both counts.
Sometimes risks are worth taking, but it surprises me that Apple is willing to risk so much potential reputational harmāboth for a specific product line and for their privacy-focused image as a wholeābecause they've decided that 15-second transcripts and LLM-generated summaries of your day that "may be incomplete or inaccurate" are somehow worth it.
Update: Marques Brownlee published a video titled The Apple Watch Has a Problem on Tuesday, September 22nd. After talking about some of the new fitness features, he spends the remainder of the video talking about Audio Intelligence and mentions many of the same concerns I did in this post. It's heartening to see someone of Marques' renown and influence call out the privacy and consent flaws of these features in public, and I hope more people with similar stature and followings do the same.