Andy Carvell joined SoundCloud in 2012, when the corporate was simply over 80 workers and 10 million month-to-month energetic customers.
The service now boasts over 150 million registered customers, and month-to-month actives within the excessive tens of thousands and thousands.
I just lately spoke with Andy as a part of a 1 hour interview protecting:
- How he introduced a web-first product to cell
- Exercise notifications, wealthy push, and different methods for driving cell development and retention
- Andy’s “Mobile Growth Stack” for 2017
You may watch the complete interview right here, and take a look at the truncated textual content model beneath.
Throughout Andy’s 4 and a half years at SoundCloud, he labored on a spread of cell development subjects, and in a bunch of various group configurations. He led a world group for some time, centered on working experiments in new markets, and did development work breaking floor in Latin American and different creating markets.
Andy’s final position simply earlier than leaving SoundCloud was working the person retention group there. He centered on driving retention of customers, each new customers and driving extra repeat and returning person. SoundCloud has notoriously excessive person retention, so I wished to dig in to seek out essentially the most helpful takeaways from the years of experiments that Andy instrumented from the group up.
The largest retention questions at SoundCloud
Round late 2014 to early 2015, the SoundCloud group recognized retention as an space of enchancment, particularly on the cell aspect. It really wasn’t too dangerous at SoundCloud, by business benchmarks, Andy informed me. However, retention can all the time be higher.
“When you can enhance retention a little bit bit, it is usually higher for long run development than simply rising acquisition a little bit bit.
The north star metric for SoundCloud was listening time, and we had some pretty aggressive targets as an organization, when it comes to rising listening time total. You may enhance listening time in two major keys: first by bringing individuals again for extra classes, or second by serving to them to attain longer listening classes.”
There have been plenty of alternative ways the SoundCloud group may transfer that key metric of listening time, however every little thing revolved round bringing individuals again and giving them extra causes to stay round and pay attention. In keeping with Andy, they had been laser-focused on constructing habits with customers in order that at any time when they had been listening, they might be listening on SoundCloud, relatively than on the radio, or with a special music streaming service.
A nuanced definition of “retention” at SoundCloud
Retention means various things for various firms, and SoundCloud was trying at a possible further nuance to retention on the cell aspect. At SoundCloud, says Andy, person retention was “listener retention.”
“We aimed at attempting to get individuals not simply to come back again, however to come back again and pay attention. We did not depend a person as retained if they simply got here again and relaunched the app once more, however really we wished them to come back again and pay attention for one more session.
The metric we had been attempting to maneuver was month-to-month energetic listeners, indicating month-on-month listener retention.”
A month-to-month energetic listener meant that that individual got here again and listened as soon as inside the month interval. Andy’s group had been additionally fascinated by rising each day energetic listeners, and attempting to extend the frequency of visits throughout every interval. To that finish, they designed initiatives that appeared at what may carry individuals again, not only one time in that month, however a number of instances in per week, and proceed to extend their frequency from there.
The place to begin for tackling retention points
There are various methods to start out in on a retention downside. Some groups could need to look at product points, whereas others need to leap into messaging and notifications. However Andy and the group at SoundCloud believed that the primary problem was to onboard customers properly sufficient that they had been in a position to expertise worth and perceive the product within the first session.
“From there, it was about giving customers a adequate cause to come back again and pay attention for one more session. However, retention actually begins with onboarding; if a person activated and had fun with Soundcloud within the first one or two classes, then the barrier for them to de-install the app or simply by no means to come back again was very low.”
Benchmarking retention in opposition to business requirements
Whenever you’re establishing retention experiments, it may be useful to know what you need to be aiming for. Andy’s group took a extra nuanced strategy to business benchmarks, trying at not simply business averages, however at particular classes.
“It’s significantly related for cell when you think about that within the app retailer, classes are so totally different and every class’s retention profiles is totally different. It isn’t all the time very significant to look at a blended business common, however relatively to attempt to look at class common benchmarks for these type of issues.”
Their group mined for information from suppliers like Modify, which publishes retention information by class, and protecting each class within the app retailer, and SimilarWeb, which gives information for each net and for Android. this as a service for each net and for Android.
What had been the “nice,” “good,” and “dangerous” benchmarks that SoundCloud stacked itself in opposition to?
“I will begin with a caveat that I can solely speak about person retention right here, not listener retention. It is vital clarification, as a result of these extra generic publications do not go right down to that stage of specificity — person versus listener — which is one thing that was a extremely vital, however inside, distinction for SoundCloud. So, on a person retention stage, trying at day one retention within the music and audio class, something above 35% is mostly thought of respectable.
Then month one retention, above 45-50% could be thought of good. Once more, that is throughout the entire class. At SoundCloud we wished to considerably over-index versus the class.”
Exercise notifications recognized as one key to retention — however with caveats
So, Andy’s group was centered on retention, and the large north star metric was listener time. There are a number of methods to affect that: you’ll be able to carry individuals again extra usually, or you’ll be able to deepen their engagement upon getting them again.
Their group was attempting to be within the prime 1% of those retention metrics, and recognized exercise notifications, or push notifications, as one thing that would majorly affect that.
“I believe it is pretty broadly acknowledged that push is an efficient channel for cell, in case you get it proper. Loads of firms do not get it proper. We additionally realized it was not a foregone conclusion, that we may simply begin sending some push notifications, and we might get a terrific outcome.
An enormous a part of why we selected this venture, relatively than for instance engaged on the product onboarding, which I believe may even have been very impactful, was that at the time, there have been no notifications being despatched out at all as a result of state of the SoundCloud iOS app at that point. It had been re-engineered and re-released a short while beforehand, and as a part of that re-engineering effort, there have been no push notifications arrange. There have been some electronic mail notifications, from a really previous legacy system, with no measurement round them. We figured they had been most likely driving some affect, however we hadn’t bought it instrumented.
We really even had some customers writing in to assist, saying, “Hey, the place are the push notifications?” We had been asking for permission, so we had been really getting numerous opt-ins, however we weren’t sending any notifications. To me, that was a terrific place to start out as a result of it was a clean canvas.”
It appeared like place for the group to start out, not solely as a result of it was low hanging fruit but additionally as a result of they might be going from zero to X, and the affect may very well be that rather more significant.
On Android, Andy explains, the scenario was a little bit extra complicated as a result of there have been notifications occurring there already. Nevertheless, these weren’t push notifications — they had been consumer aspect notifications. They appear precisely the identical to the person, however they had been being generated within the consumer.
Andy admits that that they had little or no measurement instrumented across the Android notifications however knew that they may most likely drive some further affect with push. The catch was that they’d should AB check it in opposition to the prevailing consumer aspect notifications, with further complexity round dealing with the rollout as much as manufacturing.
“We determined that what we begin with was constructing the backend notification service, and begin with iOS push as the primary channel. There now we have a extremely, a clear slate, and someplace the place we will actually set up the worth of push notifications, and the worth of the notifications that we’re constructing. Then we will look at bringing that very same system to Android, the place now we have an encumbered system that we would need to substitute, or at least benchmark in opposition to.”
Get My Growth Essays, One E-mail Per Put up.
Look ahead to destructive indicators when rolling out retention experiments
Along with the shortage of notifications on iOS (however a clear state), and the consumer aspect notifications on Android, there have been just a few different challenges that the group confronted.
First, any function the place you’re interrupting the person with a notification is inherently difficult as a result of that communication is probably annoying and undesirable — particularly in case you’re introducing new communications that didn’t exist earlier than.
“When you’re sending somebody a push notification, I all the time like to make use of the analogy, it is such as you’re coming as much as anyone and tapping them on the shoulder. You’d higher have one thing good to say.
When you’re going up and consistently tapping somebody on the shoulder, and significantly if you do not have something good to say, they are going to get fairly aggravated. In the perfect case, they are going to disable push notifications, and within the worst case, they are going to de-install the app and trash-talk you to all of their buddies.”
The SoundCloud group had been cautious about destructive indicators and destructive affect, and appeared for methods to gauge and measure them as they continued to roll out new notifications.
“A number of the destructive indicators we cared about had been opt-out charges, and destructive person sentiment. We’d ask the group group to actually preserve an eye fixed out on Twitter and these different type of social channels, and likewise of their electronic mail assist channels, to observe for individuals complaining about this stuff and tell us. We wished to have that destructive suggestions sign early, ideally earlier than customers had been opting out. Within the early days we did not have tremendous, granular notifications preferences the place we may see individuals opting out of particular courses of notifications. That was one thing we constructed as a stage two, exactly because of this.”
Retention shouldn’t be engagement
There’s one other attention-grabbing lesson right here. SoundCloud had respectable retention and had gotten that far already with out push notifications. The lesson is that retention is a extremely onerous factor to resolve for, and push notifications, or any sort of notifications, whether or not it’s electronic mail or one thing else, should not the reply to retention.
Engagement and retention are two phrases which might be actually inextricably linked. You may’t have retention if you do not have an enticing product.
“It actually begins with product. If the core product shouldn’t be delivering engagement and worth for the person, then nothing else you do round that when it comes to sending push notifications or emails or nice group assist, none of these issues is admittedly going to resolve that core traction downside that you’ll have, as a result of customers will not stick round simply since you’re sending them emails. They should get some actual worth from it first.”
One other manner to consider the distinction between retention and engagement is that this:
Retention is extra of a breadth metric: how many individuals come again inside a while interval?
Engagement is extra of a depth metric: how deep is that this expertise and the way concerned is the person with it?
You may’t have retention with out engagement, and it’s onerous to have engagement with no nice core product expertise, however each is a definite space to work on.
The SoundCloud experiment framework
Andy’s group had recognized a number of challenges, however had been already at thousands and thousands of customers at this level. Due to that scale and complexity, they weren’t going to say, “Okay, we’ve constructed these ten push notifications. Let’s push them stay. Let’s go.”
As a substitute, and throughout a number of phases, they constructed out a system to establish the appropriate push notifications to implement at the appropriate time. What had been the phases of that rollout and the way did they construct a scientific framework to establish, de-risk, and execute new retention experiments?
“When you’ve got an current person base of considerable measurement, there’s some stage of warning you should train. You need to just be sure you’re rolling stuff out that is going to have a optimistic affect.
To that finish, the very first thing we constructed was an experimental framework round these notifications that we may check not solely particular person notifications with a check group, and have a management group that was a holdout that by no means obtained that notification, but additionally that we may check combos of notifications collectively, to search for doable combinatorial results that we hadn’t thought of.
We outlined about 5 notifications to start out with, together with a “new monitor” notification, and a few social ones like “somebody adopted you” and “somebody favored your playlist.” The opposite preliminary notifications we examined had been round re-posting, which is one other core exercise within the app. Mainly, we had been listening for actions that had been occurring exterior of the person’s direct utilization of the app, however that they is perhaps fascinated by and that may carry them again into the app for one more listening session, for instance.
We outlined experimental teams for these notifications, one thing round 5% of the person base for every experiments, plus 5% for a management group. We had a giant group that weren’t receiving any notifications throughout this preliminary check interval, which was round two months lengthy. The primary month was all about constructing the core service, then we did some experimenting. Very MVP.”
Andy and I lined much more subjects within the full interview:
- Kicking off a development operate at SoundCloud 5:00
- The retention metrics that mattered at SoundCloud 6:45
- Retention benchmarks by class 9:00
- The distinction between retention and engagement 17:00
- Classes discovered in designing retention experimentation, and a few shocking elements that affected listener retention 19:30
- Breaking down month on month retention by section 23:00
- Different key proxy metrics that rolled as much as retention 29:15
- Push notification open charges vs attributed opens 30:00
- The person psychology behind push notifications 36:32
- The cell tech stack: construct or purchase? 39:00
- The (New) Mobile Growth Stack for 2017 42:10
- Andy’s findings on wealthy push notifications and the affect of emoji on open charges 47:00
Watch the complete interview:
Particular due to Andy Carvell for becoming a member of me for this interview. You may comply with Andy’s intensive writings on cell at The Mobile Growth Stack on Medium.