Kim Montgomery

Written by

Kim Montgomery

28 April 2021

News & Features

Analytics User Research Report: What You Said

We recently undertook a piece of reach to understand what our users want, need, and feel about the analytics available within Novlr. Now the results are in.

First, the conclusion:

Looking through the results of this research, I was initially disheartened to find that on a few key questions there was no single consensus on how you expect or want things to work. I was gutted. I really wanted the picture to be clear. Far from being a failure, this was in fact *exactly* what I needed to see because it clarifies for me this: our users all need different things, and we are going to need the stats and analytics to work in a way that is flexible and customisable by users. More detail below.

Who filled it in and where did they come from?

In an ideal world, to really understand who uses the analytics and how, we would have every single one of our users fill out the form, but we have no interest in forced form-filling! So we have to try and get responses in whatever unobtrusive ways we can.

We posted the research form on the blog, included it in a newsletter and sent it to Novlr users, in-app, over a 2-week period. It was shown to 552 users in-app, opened by 275, and clicked by 180. We received 172 responses through the form in total.

What did they say?

How much are analytics being used?

Most respondents used Analytics (86.6%), and 67% of those who use it looked at them at least once per writing session!

This is more frequent use than I had anticipated! However, it is safe to assume that those that chose to fill out the form were those most invested in how they work and so could be over-represented by heavier analytics users.

Which stats are used the most?

These were your most used stats from most-used to least. (These were checkboxes, so users could select as many of their most frequent as they liked).

  1. Total words written today (89%)
  2. Total words written this month (78%)
  3. Average words per day (58%)
  4. Time spent today (46%)
  5. Total words written this year (44%)
  6. Per-novel words today (in the Novels tab) (31%)
  7. Time spent this month (30%)
  8. Time spent this year (22%)
  9. Per-novel words this month (in the Novels tab) (21%)
  10. Per-novel words this year (in the Novels tab) (15%)

What did you expect to be included in the “words written” figures?

This is the one I had really hoped for a consensus on! The team and I have, over the years, fought over what should and shouldn’t be included – tried to guess what people want – and could never agree. We could have saved a lot of time by asking you guys sooner to find that: yes….no one agrees!

On the first three, there is majority, but not quite enough to say “everyone wants these included/excluded”. And on whether or not words that you wrote but then deleted should be included, it was a lot more undecided!

Off the back of this, I feel like we have to try and figure out a way for you to have checkbox filters for all four of these options (but especially that last one) that you can include or exclude at your will. It will be a big overhaul required in the background, but it seems clear to me that people want different things and we need to give the flexibility to change these as you want them. Further cemented by the answer to this question:

In those free text answers, there were at least 5 other suggestions of how this could or should be calculated. Similar situation with this question:

Did you feel like you can rely on the stats?

This is really important to me. If you don’t feel like the stats are correct, what is the point in having them? Overall, the picture is not too bad – lots of room for improvement though. However, knowing now that not everybody knows exactly what is and isn’t being calculated in the ‘words written’ totals, goes some way to explaining why some might feel their stats are ‘wrong’ (if they assume words in their notes are counted, and they are not, for example).

Other key takeaways

1.Most of you want a graph to be able to track stats through time

2. 70% of respondents set goals in Novlr. Of those, 70% had a good or very good experience of goal setting and tracking. 27% felt it was fine. And 3% had a poor or very poor experience.

3. Streaks need work. You had great suggestions about seeing historical streaks on a calendar and tracking things like the longest streak ever.

4. A number of you suggested having “average words per chapter”.

5. A number of you suggested having an estimated novel page count (we have had this suggested a number of times before and will endeavour to include something like this that is an average across different novel formats).

Our next steps are to spec out a new architecture for the statistics in the backend that allows us to do as much of the above as possible. We want to write them in a way that allows us to do as much of this and future-proofs against doing even more interesting things in the future (weekly/monthly/quarterly reports in your emails for example). After that we will look at what and how we show it. Then next – we build it.

Thank you so much for helping to inform this crucial stage of the planning process. I couldn’t have asked for better engagement and this is has entirely shaped how we will build stats in the future.