š Features that Shouldn't Exist: Live Rewind and Siri Recap
Apple recently announced Audio Intelligence on Apple Watch, an umbrella term for several audio- and AI-related features. Some of these features provide clear benefits, utility, and have little to no downside, like Sound Recognition:
Sound Recognition detects important sounds like sirens, alarms, and doorbells, and notifies you ā even when your iPhone is not with you. This is especially useful for people who are deaf or hard of hearing.
Sound Recognition is new to Apple Watch, but has been available on iPhone since iOS 14. HomePods also have the ability to send an alert when they hear smoke and carbon monoxide alarms go off. Great, useful stuff that even has the potential to save lives.
Music Recognition is another nice feature:
Music Recognition with Shazam automatically detects the song playing in the background and displays the title and artist name in the Smart Stack, all without tapping the widget.
It's not going to save lives, but it's nice to have. Shazam has existed in various forms for many years, both before and after Apple bought it, and the new implementation on Apple Watch sounds handy and convenient.
The new Live Rewind and Siri Recap features, on the other hand, aren't clear-cut wins. These features do provide clear benefits and utility, but they also come with significant downsides.
Here's Apple overview of Live Rewind:
Live RewindĀ¹ recalls what was just said in the last 15 seconds by showing a text snippet on your Apple Watch when you double-press the Digital Crown, so you can catch something you missed or are less familiar with. You can ask Siri about the content of the snippet or simply save it to the Siri app.
And Siri Recap:
Siri RecapĀ¹ ambiently takes notes on your conversations throughout your day and creates high-level Apple Intelligenceāgenerated summaries that you can use later to jog your memory, so you can stay present in the moment. Your summaries auto-delete after 7 days if you choose not to save them.
The Ā¹ in both of those overviews points to this footnote:
- Siri Recap summaries and Live Rewind text snippets are generated using Apple Intelligence and may be incomplete or inaccurate. Important details may be omitted, misunderstood, or summarized incorrectly. When accuracy matters, confirm important information with the people involved or another reliable source. Siri Recap is designed to limit certain potentially sensitive content, such as financial information, government-assigned identifiers, and more, but automated filtering may not remove all sensitive information.
When you look at the more detailed privacy overview Apple provides for Audio Intelligence, it's clear they put a lot of effort into making these features very secure and private. They even created special hardware for it: the Secure Exclave. Audio captured by Apple Watch is only processed in the Secure Exclaves on Apple Watch and its paired iPhone, and that audio is not accessible to other parts of the system on either device. For Siri Recap, full transcriptions are condensed on-device, and only the condensed version is sent to Apple's Private Cloud Compute for summarization. Everything is end-to-end encrypted, including on-device encryption with periodically rotating encryption keys.
That's all great, and demonstrates a level of effort around privacy that no other company I'm aware of would even come close to if they implemented something similar. Unfortunately, it's not enough.
Apple Watch is a very popular product. Apple sells millions of them every year, and if you're looking for them when you're out and about you'll probably see several being worn by those around you. That kind of ubiquity imparts a certain social and civil responsibility on Apple, including the responsibility to not turn these devices into surveillance equipment.
It doesn't matter that everything is encrypted, secure, and only accessible by the Apple Watch's owner. What matters is that this kind of surveillance, at this scale, shouldn't be something the only big privacy-focused company in the tech industry should be offering to its users. Apple should be setting a better example for their peers and the world at large.
To put it another way: a device commonly worn by a significant fraction of the population should not be able to listen with the intent to create a recordābe it a verbatim or summarized recordāwithout providing the following:
- Clear indications to everyone in listening range that they're being listened to
- A reasonable way for them to opt-out of being listened to
Neither of these features are provided by Live Rewind or Siri Recap. Live Rewind comes closest; it emits an audible and visual indicator, but only when it's used:
How Live Rewind respects those around you
When activated, an audible chime plays from the speaker on your Apple Watch, even if your Apple Watch is on silent or you have headphones connected. A full-display animation and microphone indicator appear on the Apple Watch display, and a distinct double-press gesture is required to activate Live Rewind. This is to alert those nearby that the feature has been activated.
This only lets people know what they said was transcribed after the fact. It doesn't provide a way to opt-out of the transcription happening in the first place.
Letting people know it happened after the fact is better than nothing, but Siri Recap doesn't even do that:
How Siri Recap respects those around you
By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers. The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone.
While Siri Recap doesn't create a recording, it does create a record. It's a condensed, summarized record, but a record nonetheless. There's no way for those being listened to to know when that record is being made, how accurate that record is, and that's no way for them to opt-out.
The socially responsible way to implement Live Rewind and Siri Recap would include continuous visual and audible indicators when a device is listening. Those indicators would explain, via text on the screen and a clear speaking voice, that the device is listening. These indicators would also provide a way for everyone who sees or hears them to easily opt-out of being recorded.
For example, when Apple Watch is listening, the display could pulse red and have a, "I'm listening. If you don't want me to listen, say 'stop listening' at any time." And, because Apple Watch is often concealed by a sleeve, body, table, or other object, it should also audibly announce the same message periodically while it's listening.
The counterargument, of course, is that constant visual and repeating audible indicators as described above would be too annoying and impractical to implement. And that's true, they would.
The problem is that when Apple realized that impractical measuresālike those described aboveāare the only way to responsibly implement features like Live Rewind and Siri Recap, they didn't abort. Instead, Apple built those features anyway, and decided or figured out a way to justify too many social and ethical compromises. Apple decided it was okay to compromise on consent. Apple decided it was okay to limit opt-in and opt-out privileges to the owner of the device that's listening. Apple decided it's okay for the people being listened to by its device to not have opt-out or opt-in privileges (many of whom are, ironically, probably also Apple users).
They key difference between Live Rewind and Siri Recap versus other listening features like Sound Recognition, Music Recognition, and devices listening for wake words or phrases like, "Siri," or "Hey, Siri," is the intent to create a record of what happened, of what was said. Knowing a fire alarm went off, that a particular song was played, or that a wake word was used when no other details are recorded are very different applications of listening technology than writing full transcripts or producing detailed summaries.
Again, it doesn't matter if what's produced and saved is a verbatim recording, full transcript, or a summary. A detailed record of some kind is still being made, and it's being made surreptitiously by design.
If you're thinking that your own memory, in its own way, records and summarizes everything you experience, how is that different than Siri Recap? Apple is thinking along the same lines in the quote I shared above. Here it is again with emphasis added:
How Siri Recap respects those around you
By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers. The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone.
The implied reasoning is that Live Rewind and Siri Recap are nothing more than technological extensions meant to augment our inherent human functionality. They're designed in a way that lets them go just a little bit further than what we'd be capable of on our own. Live Rewind lets you peek back at the last 15 seconds if you were distracted. Siri Recap lets you stay focused instead of trying to manually take notes at the same time. So what's wrong with that?
There are two problems: the lack of disclosure and the precedent these features set.
It's one thing to know you're being listened to by a person, but it's another thing to have a device listening. People inherently know how other people work. We have millions of years of evolution to draw on when dealing with other people. It's also obvious when someone is or isn't taking notes about what's being said.
On the other hand, not everyone knows how Apple Watch works, or what its capabilities are. How much better or worse can Apple Watch microphones hear than a human? How will that change in the next model, or the one after that? Is the person sitting at the next table wearing the Apple Watch that listens all the time or an older model? How much of what was said will be dropped when the audio is condensed, or what will be incorrectly represented in the transcript or summary that's produced? What happens if Private Cloud Compute isn't as secure and impenetrable as Apple claims it is?
The lack of disclosure of the device's capabilities, and when it's listening, are huge omissions. By making these features available, Apple is essentially putting the onus on the owner of every Apple Watch with these capabilities to use them responsibly, let everyone around them know their device is listening, and provide them the option to opt-out. That's irresponsible because doing those things is impractical for the same reason the persistent visual and audible alerts Apple could have built are impractical. If you have Siri Recap enabled you're not going to announce it to the crowd every time you do things like enter a store or step on an airplane.
Then there's the precedent these features set. Appleāwho again is the only big privacy-focused company in techāis setting a really bad precedent at a really bad time by creating and releasing always-listening-to-everyone features like these. The existence of these features helps normalize persistent surveillance at exactly the point in our history when we should be doing the exact opposite.
Other tech companies certainly don't need permission or approval from Apple to build always-listening-or-watching features, but the existence of Live Rewind and Siri Recap make it easier for them to build and sell similar features. The potential customers of those companies will be more accustomed to and accepting of these kinds of features if Apple has them, too. This goes double for surveillance-focused companies and other surveillance apparatuses, like governments. Many of those entities want as many people as possible to be as comfortable as possible with constant surveillance, to the detriment of the public at large. A world where most people are used to and accepting of persistent surveillance is a very dangerous world to live in. By releasing Live Rewind and Siri Recap, Apple is helping us all take one big collective step closer to that world.
The other surprising thing about these features are the risks Apple is willing to take to implement them. One scenario is that there's a security issue or data breach surrounding these features, where sensitive transcripts or summaries end up in nefarious hands. I don't think that's very likely, but it's certainly possible.
A more likely scenario is that these features, once more people become aware of their existence, tarnish the reputation of Apple Watch and Apple itself. Today, most people who take note of someone wearing an Apple Watch probably do nothing more than think things like, "that's a cool band," or, "I see those everywhere, maybe I should learn more about them." In a world where a news story explains that new Apple Watches are listening to and summarizing everything they hear, people will start thinking, "I wonder if that thing is recording me right now?" Yes, they'll technically be wrong about it "recording" them, but that won't matter. What matters is their perception, and what they'll perceive is that they're being surveilled, and that Apple Watch is creepy. And they'll be right on both counts.
Sometimes risks are worth taking, but it surprises me that Apple is willing to risk so much potential reputational harmāboth for a specific product line and for their privacy-focused image as a wholeābecause they've decided that 15-second transcripts and LLM-generated summaries of your day that "may be incomplete or inaccurate" are somehow worth it.