App Store · Google Play · Experiments
Did my ASO change work? How to judge a listing change, before and after
To judge a listing change, compare windows of days, not two days. Take the keyword’s daily ranks (each the median of several searches) for the week before the change went live in that country, skip the first days after it while search settles, keep measuring until four weeks after the change, then compare the two windows by their average rank and by their range. A change helped when the average gained at least one place and every day after beat every day before; it hurt in the mirror case. When the two ranges overlap, the move stays inside the store’s own daily wobble, and the honest verdict is “no measurable effect”. Before you credit your new text, check what else changed in those weeks.
The method works by hand or with any rank tracker. Every store fact below is quoted from Apple’s or Google’s own pages.
Why can’t you compare one day before with one day after?
Because ranks move on their own, and one evening is enough to see it.
Had a listing change gone live between those two readings, a day-against-day comparison would have reported a loss of three places. The readings alone cannot say whether the change caused it. A verdict needs enough days on each side to show how much a rank moves when nothing is done; the guide to measuring keyword rankings explains how to get each day’s number.
Why can’t the stores’ A/B tests answer it?
Both stores test listings, but not this.
Compare different app icons, screenshots, and app previews on your App Store product page to find out which resonate with people most.
Apple’s tests cover the icon, screenshots and previews, and they report conversion, not rank: the product page optimization guide goes through them. Google’s store listing experiments look for “the most effective graphics and text”: a localized experiment can test the icon, feature graphic, screenshots and descriptions, in up to five languages, and its result is measured in installs, opens or pre-registrations. Neither tests the App Store name, subtitle or keyword field, or the Google Play title, and neither measures where you rank. For those, a before and an after is what you have.
When did the change go live, country by country?
Not on the day you submitted it.
- On the App Store, the name, subtitle and keyword field change with a new version, which goes through App Review. Apple writes that “after your app is approved, it can take up to 24 hours to go live on the App Store”, and that after a manual release, “it may take up to 24 hours for it to appear on the App Store”.
- On Google Play, listing changes are reviewed first. Google writes that processing “can take a few hours or up to seven days (or longer in exceptional cases)”; with managed publishing, once you confirm, “your update will be published on Google Play within a few minutes”.
The change date that counts is the first day your new text shows in that country’s store. Check each country, and date each one separately. A changed localization also counts in every country where its language is searched: the localizations guide has Apple’s table.
Keep a short log of every change, written on the day it happens:
- the country, and the first day the new text showed there;
- the field, its old text and its new text;
- the version that carried it, and anything else that version changed;
- the keywords you expect it to move, and one you expect it to leave alone;
- any change to your ads during the weeks that follow.
Four weeks later, that log is what lets you read the verdict.
How long should the before and after windows be?
The method uses three windows around the change day, in each country:
| Window | Days | What it holds |
|---|---|---|
| Before | The 7 days before the change | The baseline: one daily rank per day |
| Settling | The change day and the next 2 | Not compared |
| After | From day 3 to day 28 | Up to 26 daily ranks |
Why a week before. It holds each day of the week once, and it ends at the change, so it describes the listing you replaced.
Why skip three days. Neither store says how long a new text takes to settle in search, and the first days mix the old and the new. Comparing them would measure the transition, not the result.
Why day 28. Ranks move a little every day on their own. A few days after a change can look stable, or unstable, by chance; four weeks show how the rank behaves around its new level, and tell a lasting move from a bump. The cost is time, and time lets other things happen, which is why the confounders below matter.
Count only measured days. A day when the app was absent from the results, the store searched another word, or the search failed is written down, not averaged.
How do you compare the two windows?
Per keyword and per country, compute two things for each window: the average of its daily ranks, and its range, from its best day to its worst. Then ask two questions. Did the average move by at least one place? Do the two ranges overlap?
Both questions matter. A gain of a fraction of a place is not a gain. A large move of the average with overlapping ranges can come from a few unusual days. Only a move that clears both tests is more than the store’s wobble.
| Verdict | Rule | What to do |
|---|---|---|
| Helped | The average is at least one place better, and the ranges do not overlap | Keep the change; it is your new baseline |
| Hurt | The average is at least one place worse, and the ranges do not overlap | Check the confounders, then revert the change |
| No measurable effect | The ranges overlap, or the average moved by less than one place | Decide on other grounds |
| Not enough measurements | Too few measured days to conclude | Measure longer, or again with the next change |
Judge each keyword on its own. One change can help a keyword and hurt another, and a change that helps in one country can do nothing in the next.
ASO Gauge applies this rule: when your listing changes on the store, it opens an experiment in that country, compares each keyword’s ranks in the 7 days before with its ranks from day 3 after, and gives its verdict on day 28. The timeline is on our home page.
What else could have moved your rank?
A verdict compares two windows. Your text is not the only thing that changed between them.
- The season. Apple’s search page names “trending or seasonal terms, such as ‘Christmas’ or ‘summer’”. Apple Ads’ search term popularity is reported by week or month, and Apple suggests it to “understand seasonal trends”. If you have it, compare the keyword’s popularity in both windows.
- A competitor. The apps around yours change their names, ship versions and get featured. Check whether the apps just above and below you changed their text during the windows.
- The store. Apple says it is “constantly evolving how search works on the App Store”. Google says it constantly tests its discovery experience, including “enhancing our ranking algorithms”. If keywords you did not touch moved on the same days, look there first.
- Your ratings. Apple counts the number and quality of ratings and reviews among its ranking parameters. A rating reset ships with a version, and it “affects all regions or countries simultaneously”. Do not reset in the version you want to measure.
- Ads and other traffic. Apple counts downloads among its parameters, and does not say whether downloads from ads count. Keep ad spend steady across both windows, or write down when it changed. The same goes for press, featuring and launches.
- Several changes at once. If the name and the keyword field changed in the same version, the verdict belongs to both.
Keep at least one keyword you did not change, in each country, as a control. If it moves with the others, the cause is probably outside your text.
What does “no measurable effect” mean?
It does not mean the change did nothing. It means that any effect was smaller than what the rank does on its own over these weeks. A keyword that moves between 7th and 10th within an evening cannot show a gain of one place.
So decide on other grounds. Keep the new text if it reads better for people, since they read the name and subtitle in the results. Revert it if it took out a word you need. Do not run the same change again hoping for another answer.
“Not enough measurements” is different. When the app was absent from the results most days, or the searches failed, the windows are too thin to compare. An absence can still be news: an app absent all week before and ranked every day after has entered that keyword’s results. Note that as such; an average cannot.
Common mistakes
- Comparing one day before with one day after. One evening’s searches already moved three places.
- Dating the change on the day you submitted it. Date it on the day the new text shows, in each country.
- Judging in the first days. That measures the transition.
- Changing several fields in one version. You will not know which one moved the rank.
- Resetting ratings in the measured version. The two changes become one.
- Changing ad spend during the windows. Your downloads change with it.
- Reading “no measurable effect” as a failure. It is a measurement too.
- Starting the next change before the verdict. Its before window would be the last one’s after.
Sources
- Apple, Product page optimization: what Apple’s tests compare.
- Google, Play Console Help, Run A/B tests on your store listing: what Google’s experiments test and measure.
- Apple, App Store Connect Help, Overview of publishing your app on the App Store and Select an App Store version release option: up to 24 hours to go live.
- Google, Play Console Help, Control when app changes are reviewed and published: processing time, publishing within minutes.
- Apple, App Store search: seasonal terms, search keeps changing.
- Apple, App Store, Learn more about search results and ranking: ratings and downloads among the ranking parameters.
- Apple, App Store Connect Help, Reset an app overview rating: a reset affects every country at once.
- Apple, Apple Ads Platform API, Insights endpoints: weekly and monthly popularity, seasonal trends.
- Google, Play Console Help, App Discovery and Ranking: ranking algorithms keep changing.
- Measurements: the App Store app’s iPhone search, three searches per keyword between 20:39 and 20:51 Paris time on 25 October 2026, and three more between 00:08 and 00:18 on 26 October.
Checked on 26 October 2026.