Trust highlighting in Wikipedia articles

[Erik Moeller]

The University of Santa Cruz/California has an interesting demo up
that computes author trust based on whether users' edits are
kept/improved or reverted. It then highlights passages of the text
according to the computed reputation of the author who added them:

http://trust.cse.ucsc.edu/
http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random

NB, this is still very experimental, but it seems promising.

Luca de Alfaro, who did most of this work, will also be presenting at Wikimania.

[John Erling Blad]

This is described at
http://meta.wikimedia.org/wiki/User:Agtfjott/Article_quality_and_user_creditability

The first work on such a system is an article at (From 12. mar 2006)
http://no.wikipedia.org/wiki/Bruker:Agtfjott/Artikkelkvalitet_og_brukeretterrettelighet

This article is now deleted.
Links to the article at meta are posted both here, at #wikipedia,
#wikimedia and several other places so it can hardly be an unknown
approach to anyone.

It is also discussed to set it up as a project together with Norwegian
Computing Center
http://en.wikipedia.org/wiki/Norwegian_Computing_Center

John E

Post by Erik Moeller
The University of Santa Cruz/California has an interesting demo up
that computes author trust based on whether users' edits are
kept/improved or reverted. It then highlights passages of the text
http://trust.cse.ucsc.edu/
http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at Wikimania.

[John Erling Blad]

From a very fast reading of the paper at

http://www.soe.ucsc.edu/~luca/papers/07/wikiwww2007.pdf

I found the following problems with the paper, and some problems not
covered at all
* It does not take into account the problems introduced by dyslectic
users. Such users can contribute a lot of info, yet the info they
contribute can be completely rewritten.
* User credits/reputation is unbound and therefore very difficult to use
for decisions, should an user be allowed to edit a specific article or not?
* Collaborations are not taken into account. This is very important as
users working together usually makes very good contributions.
* Users having expertise in one area, which reflects in the
contributions. This will have impact on the user reputation at similar
articles, ie within the same category.
* Credits/Reputation is not only given as a function of an users own
edits, but also as a result by the quality of the article he or she
edits. The quality of the article is again given by the reputation of
the users editing them, thereby creating a highly interconnected system.
To anly take user contributions and reverts into account is to simple.
* When credits/reputation is used for decisions like excluding an user
from editing an article the decision will reinforce its own decision, so
after making the decision it will become harder and harder to get over
the limit to be allowed to edit an article.
* If both user credits/reputation and article quality are in use there
is easily introduced an instability which results in a situation whereby
an user are given less credits for editing an article than the article
itself are given quality points, leading to a situation where the user
is blocked from editing his own article. This situation is most likely
to happen when there is introduced a non-linear growth factor in
credits/reputation or quality.

It could be more that I didn't notice.

John E

This is described at
http://meta.wikimedia.org/wiki/User:Agtfjott/Article_quality_and_user_creditability
The first work on such a system is an article at (From 12. mar 2006)
http://no.wikipedia.org/wiki/Bruker:Agtfjott/Artikkelkvalitet_og_brukeretterrettelighet
This article is now deleted.
Links to the article at meta are posted both here, at #wikipedia,
#wikimedia and several other places so it can hardly be an unknown
approach to anyone.
It is also discussed to set it up as a project together with Norwegian
Computing Center
http://en.wikipedia.org/wiki/Norwegian_Computing_Center
John E

Post by Erik Moeller
The University of Santa Cruz/California has an interesting demo up
that computes author trust based on whether users' edits are
kept/improved or reverted. It then highlights passages of the text
http://trust.cse.ucsc.edu/
http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at

Wikimania.

[John Erling Blad]

I'll crosspost this to admins at no.wp

John E

Post by John Erling Blad
This is described at
http://meta.wikimedia.org/wiki/User:Agtfjott/Article_quality_and_user_creditability
The first work on such a system is an article at (From 12. mar 2006)
http://no.wikipedia.org/wiki/Bruker:Agtfjott/Artikkelkvalitet_og_brukeretterrettelighet
This article is now deleted.
Links to the article at meta are posted both here, at #wikipedia,
#wikimedia and several other places so it can hardly be an unknown
approach to anyone.
It is also discussed to set it up as a project together with Norwegian
Computing Center
http://en.wikipedia.org/wiki/Norwegian_Computing_Center
John E

Post by Erik Moeller
The University of Santa Cruz/California has an interesting demo up
that computes author trust based on whether users' edits are
kept/improved or reverted. It then highlights passages of the text
http://trust.cse.ucsc.edu/
http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at

Wikimania.

[Daniel Mayer]

Post by Erik Moeller
The University of Santa Cruz/California has an interesting demo up
that computes author trust based on whether users' edits are
kept/improved or reverted. It then highlights passages of the text
http://trust.cse.ucsc.edu/
http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at Wikimania.

Wow - that is even better than the idea of user-driven trust metrics. On top of that, it
encourages activity we want to encourage; creating content that lasts and is built on. It also
looks like they might have a better diff algorithm than we do.

-- mav




____________________________________________________________________________________
Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/

[Luca de Alfaro]

Thanks for the comments! Yes, we very much wanted a system that

- does not change the day-to-day Wikipedia experience (it worked so
well so far, let's not change what worked; would people be put off, or
strange behaviors encouraged, by user-to-user ratings?),
- encourages lasting content, but does not punish people whose content
is rewritten/improved: people are mainly punished for reversions or partial
reversions.
- does not display reputation associated with authors (newbies to the
Wikipedia provide a good share of the factual content, as they include many
domain experts, so it's important not to put them off)
- but still gives useful information to visitors on the trust of text
(and lots more can be done, e.g., getting on request the last high
trust version, ...)

As you point out, getting text diff to work is not trivial, and it took us a
long time to get something we liked; we had to write it from scratch... the
idea is given in the WWW07 paper: a greedy algorithm, that matches longest
substrings first, giving however a bias in favor of substrings that occur in
the same relative position in the pages. Moreover, we keep track not only
of the text present in a page (the "live" text), but also of the text that
used to be present, but has been deleted (the "dead" text). If you don't do
this, reverts (and partial reverts) are not dealt with correctly.
We think that even better can be done, in fact (everything can always be
improved), but we haven't had a chance yet.

Luca

Post by Erik Moeller

Post by Erik Moeller
The University of Santa Cruz/California has an interesting demo up
that computes author trust based on whether users' edits are
kept/improved or reverted. It then highlights passages of the text
http://trust.cse.ucsc.edu/
http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at

Wikimania.
Wow - that is even better than the idea of user-driven trust metrics. On top of that, it
encourages activity we want to encourage; creating content that lasts and
is built on. It also
looks like they might have a better diff algorithm than we do.
-- mav
____________________________________________________________________________________
Park yourself in front of a world of choices in alternative vehicles.
Visit the Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/
_______________________________________________
Wikiquality-l mailing list
http://lists.wikimedia.org/mailman/listinfo/wikiquality-l

[Daniel Mayer]

Post by Luca de Alfaro
Thanks for the comments! Yes, we very much wanted a system that
- does not change the day-to-day Wikipedia experience (it worked so
well so far, let's not change what worked; would people be put off, or
strange behaviors encouraged, by user-to-user ratings?),
- encourages lasting content, but does not punish people whose content
is rewritten/improved: people are mainly punished for reversions or partial
reversions.
- does not display reputation associated with authors (newbies to the
Wikipedia provide a good share of the factual content, as they include many
domain experts, so it's important not to put them off)
- but still gives useful information to visitors on the trust of text
(and lots more can be done, e.g., getting on request the last high
trust version, ...)
As you point out, getting text diff to work is not trivial, and it took us a
long time to get something we liked; we had to write it from scratch... the
idea is given in the WWW07 paper: a greedy algorithm, that matches longest
substrings first, giving however a bias in favor of substrings that occur in
the same relative position in the pages. Moreover, we keep track not only
of the text present in a page (the "live" text), but also of the text that
used to be present, but has been deleted (the "dead" text). If you don't do
this, reverts (and partial reverts) are not dealt with correctly.
We think that even better can be done, in fact (everything can always be
improved), but we haven't had a chance yet.

Please continue your work! :) Also, will UCSC allow you to license your work under the GPL or a
compatible license? It would be a shame if your work could not be incorporated into MediaWiki one
day.

Oh - I'm sure you already thought of this, but please make sure this tool only works on the main
namespace since that is where all the articles (content) hang out. Your analysis will be skewed if
it looks at non-article content since much of which tends to be preserved as is and without
modification for years.

I also don’t want to reward those who do almost all their editing outside of articles by giving
their edits an incorrectly higher rank. We really need to encourage editing of content, not
arguing on talk and policy pages.

-- mav



____________________________________________________________________________________
Luggage? GPS? Comic books?
Check out fitting gifts for grads at Yahoo! Search
http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=bz

[Luca de Alfaro]

Dear All,

We plan to release the code under the GPL (I assume this is fine for
MediaWiki; otherwise, we can release it under what they wish if there is
interest into an integration). We haven't done a release yet mainly because
the code is in flux (we just finished implementing trust; we will now tinker
with it a bit until we are satisfied). Note however, that the code is
currently written to batch-process .xml dumps, where all the revisions of a
page are given sequentially. The reputation we use is well-suited to become
an on-line system (the amount of computation to do for every edit is small),
but that would require some code modifications, since the order of
presentation of the revisions would be different (every time someone edits).

The main thing that would be needed to implement this as an on-line
(real-time) system in WikiMedia is to store, for each page, a little bit of
extra information in some auxiliary DB table (such as trust and version of
origin for each word of the last revision; reputation value for each
author).
The information would be much smaller than the total history size; it would
be essentially proportional to the size of the last revision (so the storage
requirements would not change much).

To implement this system with minimal disruption to the current Wikipedia
servers, one could make sure that every time the main Mediawiki servers get
an edit, they forward a copy of the edit to another server, which runs the
reputation/trust stuff, and which can then serve the colored pages to
interested visitors. This way, one could bring the trust/reputation system
online with minimal disruption to the Wikipedia servers (another advantage
of not requiring user-to-user ratings).

As for the namespaces, yes, we know, we just analyze the main name space (or
at least we try -- let me know if you find some article in the demo that
does not belong there).

Best, Luca

Post by Daniel Mayer

Post by Luca de Alfaro
Thanks for the comments! Yes, we very much wanted a system that
- does not change the day-to-day Wikipedia experience (it worked so
well so far, let's not change what worked; would people be put off,

or

Post by Luca de Alfaro
strange behaviors encouraged, by user-to-user ratings?),
- encourages lasting content, but does not punish people whose

content

Post by Luca de Alfaro
is rewritten/improved: people are mainly punished for reversions or

partial

Post by Luca de Alfaro
reversions.
- does not display reputation associated with authors (newbies to the
Wikipedia provide a good share of the factual content, as they

include many

Post by Luca de Alfaro
domain experts, so it's important not to put them off)
- but still gives useful information to visitors on the trust of text
(and lots more can be done, e.g., getting on request the last high
trust version, ...)
As you point out, getting text diff to work is not trivial, and it took

us a

Post by Luca de Alfaro
long time to get something we liked; we had to write it from scratch...

the

Post by Luca de Alfaro
idea is given in the WWW07 paper: a greedy algorithm, that matches

longest

Post by Luca de Alfaro
substrings first, giving however a bias in favor of substrings that

occur in

Post by Luca de Alfaro
the same relative position in the pages. Moreover, we keep track not

only

Post by Luca de Alfaro
of the text present in a page (the "live" text), but also of the text

that

Post by Luca de Alfaro
used to be present, but has been deleted (the "dead" text). If you

don't do

Post by Luca de Alfaro
this, reverts (and partial reverts) are not dealt with correctly.
We think that even better can be done, in fact (everything can always be
improved), but we haven't had a chance yet.

Please continue your work! :) Also, will UCSC allow you to license your
work under the GPL or a
compatible license? It would be a shame if your work could not be
incorporated into MediaWiki one
day.
Oh - I'm sure you already thought of this, but please make sure this tool
only works on the main
namespace since that is where all the articles (content) hang out. Your
analysis will be skewed if
it looks at non-article content since much of which tends to be preserved as is and without
modification for years.
I also don't want to reward those who do almost all their editing outside
of articles by giving
their edits an incorrectly higher rank. We really need to encourage editing of content, not
arguing on talk and policy pages.
-- mav
____________________________________________________________________________________
Luggage? GPS? Comic books?
Check out fitting gifts for grads at Yahoo! Search
http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=bz