Forums

Bugs The 'percentage correct' display is missing.

I feel like I'm complaining all the time, but I wish the site worked the way it did when I started using it three months ago. I do prefer it to LingQ and some other things I've tried, and I would really like to to work long enough for me to finish a couple of the lists I'm using.

you don't get the session stats pop-up at the end of a watering? I've been using that to track my error rate.

I only get it if I complete a set, which in this case is now 50...it used to be off to the side. I currently have ~300 words in the Mandarin Comprehensive queue and I can't do them all at once. Usually I take a break when I start to fail, but now I won't know when that is. Also, I have enough words, that I sometimes have to do them as fast as possible, or I won't have time in my day to complete them. So, I used the display as an important indicator. If my percentage was 50% or something, I would slow down a bit.

I used it similarly and I'm a bit saddened to see it go. I actually preferred seeing all the data as I went rather than this new page that (was, but now isn't) redundant.

It's not as useful as seeing it as it progresses and it's an extra step between sessions now as well.

It may be a psych thing. Some people could be sensitive to their error rate. In essence, the error rate will cause them to make more errors. Or, they may not stay as long on the website. Some people may not care about the error rate and not change the level of their participation with the website.

Remember - memrise is playing with our minds a little. The introduction of quotes and tips did change our (memrise users) behavior. (see the thread on the quotes, particularly BenW's reply)

When I go jogging, I use Nike+ on my iPod to track my progress. Before that I used a watch. However, I would simply not be able to run the way I do without ways of measuring my progress.

The developers may be thinking they've removed a distraction, but I now find it more difficult to complete the tests.

I signed up for LinQ today. My plan was to maintain my Memrise practice with a fixed number of words once I completed the lists I'm using and then start using integrated techniques. I'm starting to think that might not be possible though. Its quite possible that I'm a statistical outlier and that the site isn't really designed for someone like myself. That would definitely explain why I tend to complain so much. Its also possible that they don't know how many users will abandon the site when they make various changes.

"Remember - memrise is playing with our minds a little." If they're running experiment that's fine, but they should keep in mind that their results will eventually be tainted from weeding out people who abandon the site along the way. What they could be left with is a sample of people who don't use the site very much.

Another analogy--without clear progress data, it feels like playing a basketball game where no one knows the score. Sure every play should matter regardless of the score, but that's not how it works in practice.

@carl_a - I don't think you are a statistical outlier. Memrise needs people like us to complain about features that we don't think add value to our learning experience and to request features that we think will add value to our learning experience. I think memrise is playing around with things and seeing how it affects people. If enough people complain about this "feature", I'm pretty sure they will roll back to the previous version. Remember how quickly the blue box disappeared.

I'm still evaluating whether or not I like not knowing my error rate as I work through a session.
Here are a few of my thoughts: 1. how does knowledge of my error rate affect my behavior while watering (the observer effect - physics but I guess this is also present in any field that studies behavior) - i.e. do I, when I know I am failing, subconsciously choose to fail the rest of the test or do I struggle through

1. (this is closely related to the first one and it could just be semantics that cause me to differentiate these two (1 &2)) how does the lack of information affect how I use the memrise system - i.e. do I go and learn more words, do I increase my time on the site, do I review more, do I go faster or slower answering the questions

2. how realistic is this to real life scenarios - where I won't be "graded" (I won't be able to tell my error rate when I am speaking or writing - until I am corrected by someone.)

For what it's worth, I have a spreadsheet in which I track my results. I use the pop-up to note down my score and I mentally guesstimate how much time each watering session will take based on the number of plants I need to water.

sorry I pressed the enter key too quickly

@carl_a - "If they're running experiment that's fine, but they should keep in mind that their results will eventually be tainted from weeding out people who abandon the site along the way."

I agree with you. Maybe memrise could create some sort of system by which people can volunteer to be test subjects. While that might create some "self-selection" issues, it might be a way to not unnecessarily irritate large groups of people.

I too thought it had gone, but elements of the progress bar can now be seen creeping across the top of the screen - look for the green bit at the very top of the page.

From a usability point of view I find this more confusing as (i) I don't know what the bar means, (ii) green is not particularly distinctive in the context of the rest of the site, and (iii) it loses data that were previous available about performance - although it's a moot point whether or not this motivates or demotivates people. For me it doesn't really matter, I have a relatively unscientific way to record progress i.e. I see if I can remember things...

Glad Memrise is always trying to improve. Glad that Memrise will listen to its users and return or change things again when necessary.

Just to state this clearly: The percentage correct indicator functions for me as a motivational gauge. At the end of the session I don't actually care how I did and I try to click back on the /gardens/ bookmark before its even displayed.

That green bar creeping along the top is just a visual progress of that watering session. If you get 100% of your answers wrong you'll still see the green go across and eventually reach the end.

At first I thought it was a "rating" bar of some sorts so I started getting words wrong on purpose to check.

I find it the new system really demotivating. I push myself to do a little better and try to get a certain percentage/number of points.

It's also harder to pace myself. I like to know how many I have left to do if I'm memorising with a short amount of time.

This change seems to be an example of breaking something that works. Would be good if as much visible effort was made on fixing the things that are broken.

@carl_a When I read your 1st post I smiled as you sounded just like me...if I see something I don't like I complain, not for the heck of it but because I found an inconvenience. But this time I will not complain, really. I am so fed up with Memrise!

We are told by Ben that major bugs will get priority, oh yeah? New feature coming out "no more stats" and thin progress bar most didn't even spot yet the Spanish Patrol still doesn't work after almost 2 months, the dictionary search doesn't work, edit right all messed up, duplicates by the 1000's, etc. But did you notice the new white circle on the colorful top left link to the Main Page and the new colors of the Pause button? How adorable and such a necessary change that it couldn't wait.

My friends we can complain all we want, we are not the voice they listen to, us old users are a minority, I bet we don't constitute %5 of the Users table. "Traffic" is the main goal and they do get traffic with new users, if it looks cool they'll talk positively about the site (no matter if most don't stay for a month).

I just don't see it as if Memrise has the same priorities as we have. I am really sorry to say that I have lost confidence in this site. I cannot speak for others but you all know that I am not the only one, by far.

@jennifer Remember - memrise is playing with our minds a little. The introduction of quotes and tips did change our (memrise users) behavior. (see the thread on the quotes, particularly BenW's reply) I would be curious to see some clear and solid metrics on this one. My experience is that most time the page loads to fast for me to even have time to read the quote. How did it change my behavior? Hum...I'm trying to read faster :0)

@revan I am 100% behind your statement. And I can't wait to read the reasoning behind this new "feature".

I am very sorry that you all dislike this change so much - as I posted on another thread:

"The fact that people (myself included) enjoyed looking at the stats screen during learning sessions was a bit of a double edged sword: it is good that they enjoyed it, but that could become a distraction, and coule lead to people giving up on sessions that were going badly etc. Having to wait to the end to see the stats is an extra motivator that seems to make people more likely to finish a learning session, and then more likely to go on and learn more. We will be monitoring the metrics on this, but that seems to be the situation at the moment, and that is good news for the amount of learning that people are doing!"

But clearly you don't feel that this is happening for you. Is the % correct the single most important number that you would like to see (I think it is for me). We certainly do listen to you, and I will see what we can do about getting that stat put back in to the learning sessions, at least on a test basis to get some firm figures on its effect.

@sfrenchie, in terms of the bug fixing, the bugs that you mention are all top priority, and the members of the team who can work on them are working on them flat out. The front end design changes are not implemented by the database engineers who are working on the bugs in the database.

Best wishes and thank you, as ever, for you feedback,

Ben

Necessary changes on test page: - Show % correct (great motivation) - Show % completed in the form x/x or x of x - Smaller progress bar (as it was) changing color with rate of failure - In case of failure carry the wrongfully typed word to the definition page

Thanks

I'm with sfrenchie; great ideas. However, the screen real estate is now blank, so its not like it was swallowed up in some kind of design trade-off. How about putting the data back with a toggle to display it or not. The default could be off for novice users. By the time people reach a million points or so, they will figure out whether they want this displayed.

Sorry Ben but I have to analyse your statements and it seems the decision for the new modification was based on wind.

"...was a bit of a double edged sword" can't please everybody until you opt for a personal settings system.

"...but that could become a distraction", could? Ah...but not sure? "...and coule lead to people giving up on sessions..." any analysis was done on that or is it just a guess?

"Having to wait to the end to see the stats is an extra motivator" no, I don't agree. This time you should use the conditional tense.

"...that seems to make people more likely to finish a learning session, and then more likely to go on and learn more." based on what observation or stats? The feature was just removed, how could you have metrics on the new state?

"...and that is good news for the amount of learning that people are doing!" I don't understand this statement. It's good news for the learning that most serious users are upset?

Ben thank you for the update on the bugs issue. So could you please give us some ETA? I suppose those guys have proposed modifications diagrams to the team leader and a plan of implementation and a timeline were drawn?

sfrenchie, I think there's value in giving the Memrise team the benefit of the doubt, instead of rejecting a new change straight away.

This change has just been rolled out - surely it's worth trying for a week or so before jumping to conclusions? We can't afford to be conservative about a site that's still in its beta, or we'll never get anywhere.

More features does not mean something is better - simplicity and clarity are very valuable in a product. Having done my sessions today with the new feature, despite being initially against it, I've rapidly come round to the 'push to the end of the session' point of view.

Instead of constantly having one eye on my score (which would often get me down as well as motivating me), I can now finish the session and then have a think about that session. I like this better, and don't want it changed back.

Talafar, I'm not trying to impose my personal ideas here and I am glad you let us know of your feelings and experience. That is what the forum is for.

@sfrenchie we ran an AB test on this, running the new version concurrently with the old version for different subsets of users. From that we gained enough evidence that this change was be positive to the amount that people were learning to warrant making it live to a wider audience.

Now we have done that, and are gathering feedback from a wider range of users, like you. By combining this feedback with the statistical evidence we will make the decision on whether to keep the new version or not. So we are going to be monitoring the metrics, but we are also going to be listening to feedback; I should have made that clearer in the post above; I wrote that response originally on another thread where the strength of feeling on this point was less clear, and so I didn't make it clear enough that we would be taking account of it. Apologies for that, having read this thread I should have re-written that at once.

I hope that makes it a bit clearer,

Best wishes

Ben

How I think the layout should be:

The top bar should be a judge of how poor/good you are doing going from Red-->Orange-->Yellow-->Green. As this seems to be what everyone was thinking it was - so it is a logical step to just "go along" with that idea.

The data displayed at the end of a lesson should return to the left-hand side. It was redundant to show it at the end, it should retain the information on the left and just show your gardens in the center (as if you skipped over this information).

Allow users to collapse/expand the side navigation (where this information is kept) and store this choice in a cookie or similar fashion.

This gets rid of that extra 'distracting' information while allowing those of us whom it helps to leave it there.

I could come up with some "quick-fix" images to express my idea more clearly or an example of how it can be neatly implemented into the Memrise interface if needed. But your designer has a knack of his own for minimalism so I don't think that will be needed. ;)

As always Ben, thanks for your feedback and diplomacy with fussy customers ;0)

I agree with everything that sfrenchie's said. I feel like I'm playing basketball blind. I have no idea where the court is, what the score is or how many shots I've made. It may be exciting to finish and find out that you actually made some shots but you don't feel any adrenaline or a push to do better at the end of the game. (I have NO idea where that analogy came from, I hate basketball :P)

LOL Robodl95, I like the analogy. Mine is going through a school trimester and you are not told of any result until the end (when it's too late). I actually seldom look at the % success, I just glance at it once in a while. In my bad days when I near 50% it gives me a kick since I will not accept a score under P. At times I see 80-85% and that motivates me to pay attention, maybe to not answer too fast in order to "beat the score".

The status bar at the top of the screen sucks. Bring back the old stats display.

I'm loving the changes, because they let me focus on words instead of eye the scoreboard all the time. Surely, from time to time it was nice to be energized by the amount of points you were gaining, but I think (and feel) that this simplified screen is better for learning efficiency!

I was missing the small panel too. May I suggest placing it back where it was but adding a small "-" to the top right so whoever doesn't like it can collapse the panel?

@marijndejong: Agreed. Or have it hidden by default but those of us who like it can pop it up?

sfrenchie: (Commenting on your suggestions a few posts back for what is needed) - Yes to all, especially the xx of xx , which is new - the little progress bar was a useful quick unconscious guide, but sometimes you just need to know how far you've come / how far you still have to go.

I remember once walking up a steep hill somewhere in Austria and being overtaken (just) by a keen cyclist, gleaming with sweat and red in the face, who shouted to me "wie viel, oh wie viel?". He really needed to know how far it was to the top. I couldn't help due to non-comprehension and being completely unfamiliar with the road. He could have used a progress bar or a mileage check. And sometimes in tests it is really helpful to know how far you've got.

PS and OT: I'm not sure his German was much better than mine - when I stopped for lunch my little dictionary suggested he should have asked "wie weit?". But he was long gone by then.

Ben:

Two points:

1. I understand that developers come in different flavours, but could a tiny part of the front-end engineers' effort be applied to ameliorating one or two of the big problems?

The obvious example is the Patrol tab. I believe it works in some languages, but it doesn't in Spanish and (so I read) in quite a few others. Could the front-end guys hide it where it doesn't work? And also where the user doesn't have permission.

1. Of course it doesn't make sense to put changes by team A on hold while team B are working on unrelated changes, even if Team B's changes have a higher priority. But from the longterm user perspective it does feel a bit like deckshairs being rearranged while the ship wallow, fiddles being played while the city burns, etc.

So your explanation wold have been helpful in advance - "our database engineers are working hard on problem B but this isn't going to happen quickly. Meanwhile our front end engineers will continue to experiment with improvements in the look and feel of the software - as part of our testing strategy, some of you will see these changes, some of you won't." And then "We've found from our tests that ongoing display of testing statistics actually makes it more difficult for most of our users to learn, so we're going to make some changes here in the near future"

Of course we'd then grumble at the announcement as well as when the change happened, but at least we wouldn't feel so much like mushrooms (in the dark, etc). memrise needs to manage expectations better - and that will help with reputation management.

@revans, both excellent points.

1 - with hindsight this should certainly have happened. It was certainly not intended that the patrol would languish unfinished for this long. I will discuss with the team now whether we should be hiding the patrol where it doesn't work, or whether the fix now really is imminent.

2 - You are right, of course, managing expectations is something that we need to do much better at. This is an excellent idea for how to go about that, and we are discussing how best to build this into into the workflow. I will keep you updated on this.

Best wishes

Ben

Thanks Ben.

I imagine the majority of memrise users probably rarely glance at the forums and the What's New stuff, but if you could give some advance warning as part of the normal development cycle that would be much appreciated by those of us who do.

Well I like the new layout, and I'd like to echo everything in TL-Talafar's last post. A lot of people seem to be saying what's 'necessary', when in fact all they're saying is what they personally favour. Which is fair enough, but that doesn't mean they're necessary.

I know how far through a watering session I am because a) I know how many words I'm about to water before I start and b) the new progress bar - although I agree it would be great to have red and yellow sections for wrong and partly wrong asnwers, instead of being solid green.

I also know roughly how well I'm doing because... well, beause I'm the one doing it and I know whether I'm getting lots right or if I'm making a bit of a dog's dinner of it. I won't know the exact percentage, but I don't need to. With the new system I sometimes think I'm doing worse than I am and am pleasantly surprised by the end percentage, which is a nice surprise.

Not every change is going to be a good one, I think everyone acknowledges that. But equally people can get stuck in a rut and develop a fixed way of doing things - and not necessarily for the better. A change can be a good, if initially uncomfortable, way of jolting you out of those behaviours.

And because it doesn't seem to get mentioned very often: don't forget that we are still in beta here. It says it on most pages. If some people can't cope with the implications of that maybe they should come back in six months.

@SoapDistant:

1. Your mileage clearly varies from mine. So it would be good to have the option.

2. At least one of your "some people" have worked on beta software many times before. Sadly memrise is being managed like an alpha site but without the control and communication. My concern is that memrise won't make it through beta and into a successful IPO if it doesn't get its collective act together.

3. If I go away and come back in six months my gardens will be even more overgrown than they are now and the neighbours will complain.

I was a software developer for many years. One thing I quickly learned was that changes annoy users. Especially if you take something away, even if 80% of users are happy about it 20% will not be and you don't want to p*ss off 20% of your users. So you really should always consider giving people the option of doing the things the new way or the old way. Now I know this is supposed to be a beta site, but how many thousand people are hooked on it daily? Our betas were with a small sample of users who knew they were beta testing. They gave us feedback on the software after which we made changes we thought were appropriate from their comments and then a new version was issued. After a few iterations, when everyone was happy, it would be rolled out to the userbase at large. But generally no existing facilities would be taken away for reasons already given. I agree with revans that the site is really at the alpha stage - it is being continually developed, but there are thousands using it already and IMO that just won't work. Perhaps it would be better to have a stable site and a separate development site that users can try out and feedback comments?
Let me finish by saying I actually think the site is great and we users are hugely fortunate that people are willing to give up so much time to its creation and development - many thanks to all concerned.

Jespa

Yes. And yes again.

Maybe the infrastructure has got quite complex and that may be one of the reasons why memrise are reluctant to set up an alpha site - but it really would help their image if it could be done, and there are clearly a lot of people out here who would be willing to become alpha testers, even if it involved some formality of response instead of just our current casual grumbling.

@SoapDistant, I am very glad that you are enjoying the new layout, and thank you for getting involved in this discussion - it is good to be reminded that there are differing needs and preferences that we need to balance. At the moment I think that the most likely scenario is that we will allow you to toggle the display on and off according to preference, which would hopefully suit everyone.

I do also think that revans' criticisms are fair though: we have not been doing a good enough job about communicating what developments we are testing and working on, and explaining new features in advance. We are in Beta, and that is a learning process; with all of your help, we are learning to do these things better.

Thanks,

Ben

One thing on this site that is NOT in beta is Ben Whateley. The memrise experience would be much worse without Ben.

Haha, thank you, that is very kind of you to say!

What I would love to see is a statistics display with an aggregate for the day, not just for one watering session.

I'm working on lots of different courses, so that means several different watering sessions a day. If I'm doing badly with my watering, I don't want to learn new words that day, but if I've done well, I'd like to learn something new. Now I just have to guess how well I've done in total.

A final thought after reading the many thoughtful comments: Memrise will probably be monetized in one of two ways, either by advertising/sponsorship or by selling subscriptions/add-ons. If its the latter, they will put themselves in the same class as the game maker Zynga, which would be appropriate given the 'gamified' Farmville-ish nature of the site. However, only about 3% of Zynga game players ever pay for games or content, leaving 97% of the userbase as free riders. The people who pay are generally those who are obsessed with the games and pay all the time... the same as the people who cared enough to post in this thread. Of course Memrise needs to build the user base, gain critical mass, and reach what author Malcolm Gadwell calls 'the tipping point'. However, if their user churn is too high, they could find that there are no people left willing to pay. I pay for LinQ and I would pay for Memrise as well, insofar as it continues to be something I can use.

I've considered making a new account with fewer words so that I wouldn't be so hypercritical about the UI. However, that would basically be moving back to being more of a casual user.

It's good to have a stats summary at the end of a watering session but like several others I also feel the loss of the cumulative stats as the session progresses. It introduces an element of competing against myself and, as others have said, it helps me to recognise when I'm flagging and should stop and resume when I'm fresher. Please bring it back!

I for one would vote against the new progress bar indicator during a testing session. Call me a stick-in-the-mud if you like, but I think the previous indication was far better UI design. Having said that, I really do love the site and how well it works.

I know you like to try new things to find improvements and so I wanted to give this a fair shot, which I think I have now, but I also GREATLY prefer the old way. I like seeing the stats, but in practical use, I find it very irritating that now I have ANOTHER screen to click through at the end, which I have to WAIT for (seconds add up). This makes the process slower, which is no improvement for me. I also like seeing my percentage correct right as I go along - it's a motivator for me. But even more, I miss seeing the number of words left in the set. Now I have no way of knowing where I am & how many to go. I feel it was more user friendly before. Please, please put it back the way it was?