The study

Does it actually work?

We don't know yet whether it works, and we'd rather find out properly than assume. We designed a study to run inside the app. That study is not open yet: nobody can join it, and the app does not invite you to. This page is the record of how it was designed, what it will measure, and what happens if the answer is no.

In short, from Heron

I'll put it as I would to a colleague. Everyone in the study gets the same app. We compare how much each person uses it with how their answers change. If the two move together, that is an association, not proof: people who use it more may differ in ways that have nothing to do with us. It's the most this design can show, and I'd rather you heard its limit from me.

The problem with asking

"We asked people and they felt better" isn't good enough.

Three things make an app like this look effective whether or not it is. All three push the same way.

People start at a low pointYou download something like this when you're feeling bad, and feeling bad is partly temporary. Recovery and the seasons do some of the work, and neither of them is us.
Volunteers aren't a neutral samplePeople who sign up for research about an app they chose to install already like it more than average.
Being measured changes the answerKnowing you're being asked about something you hoped would help changes what you say about it.

So the only useful question is compared to what? Without an answer to that, a chart of our users feeling better over time is marketing — and anyone who reads research for a living will read it as marketing. A trial answers it by giving some people something different. This study doesn't: everyone in it gets the same, full app. So it can't answer that question in full, and the next section says what it can show instead.

What we compare

How much you use it, against how your answers change.

Everyone in the study gets the same app and uses it as much or as little as they like. We put how much each person actually used it beside their answers over six months.

What it can show

Whether people who use it more tend to answer differently from people who use it less, and by how much. If there is no difference, that is a result too, and we'll publish it.

What it can't show

That the app caused any difference. People choose how much they use it, so the people who use it most may differ in ways that have nothing to do with us. Every result we publish will say so beside the number.

The schedule

Four sets of questions, a few minutes each.

The first one has to come first. A starting point measured after a good first session isn't a starting point — it's already part of the result, and it's the easiest thing in a study like this to get wrong.

  1. 1Before your first sessionThe design had one invitation, shown in the app once, after you made your account and before anything had happened. It is not shown, because the study is not open yet.
  2. 2Two weeksEarly change, and how much you've actually used it.
  3. 3Two monthsThe point the whole study turns on. This is the point we committed to in advance for the analysis.
  4. 4Six monthsWhether anything that changed has stayed changed.

What we ask

Questions other people can check us against.

We use scales that already exist, with their standard wording, unchanged. A questionnaire we wrote ourselves would produce numbers nobody can interpret — including us.

WHO-5

Five questions on well-being, used in hundreds of published studies. Because the wording is standard, our number means the same thing as everyone else's.

Life satisfaction and loneliness

A handful of questions with national reference data behind them, so we can say how our users compare to the general population and not only to themselves.

One open question

What, if anything, have you done differently? No scale behind it, and usually the most interesting thing we get.

We deliberately don't ask about self-harm. A question like that creates a duty of care a set of research questions cannot discharge, and a disclosure nobody acts on is worse than a question nobody asked. The SOS button in the app is there whether or not you're in the study.

Being honest about it

The four things that would let us fool you.

We will pay for answering, not for the answer. Answering will earn AsclepiCoins, the same amount every time, whatever you say, including that the app has done nothing for you. It will never depend on still using it, and leaving will not take back what you've earned.
We say what we're looking for before we look. The plan is registered publicly before the first person answers: which measure decides it, at which point, and how we handle people who stop. That's what stops us picking the best-looking number afterwards.
We publish who stopped answering. Most people will. The ones who stop are not the same as the ones who stay, and that quietly ruins studies like this — so we report the numbers at every stage, and what the result looks like if you assume everyone who left got no benefit at all.
Your answers are filed under a research number. We have no name, email or phone for you and that hasn't changed for the study. What you say in a session never goes into the research data. We do count how many sessions you do, and compare that with your answers.

The study is not open yet

Nothing to join yet.

You can't sign up here, and the app does not invite you to take part.

Meet Heron, who runs it

This is a study of our own product, not an independent clinical trial, and we are not a medical service. If we ever publish this work in a journal it will go through external ethical review first.

See for yourself in the app.

No sign-up and no email to give.

Coming soon to iPhone and Android.