A very efficient stop on bumping the trust is to only bump the trust
metrics when actual content is added. When I experimented with similar
the entropy of weird looking iw-links would bump the metrics. To adjust
and was once more susceptible for sock puppetry. This should be avoidable.
exists in a vocabulary, and every other word are given a flat rating.
This seems to work well.
Post by Aaron SchulzHaving looked at several pages there, socks don't seem much of a
problem. Either it would take several very old "high-trusted" socks to
edit or many many new socks to get an article to slight orange/white
levels. It takes a good 5-7 edits of mainly average users to get out
of deep orange into near white.
What does bother me just a little is that all kinds minor
grammar/spelling fixing, tagging, categorizing, ect...edits seem to be
made for articles. Actually, when I wrote a JS regexp based history
stats tool, I noticed this a while ago. Some random IP/new users adds
a chuck of text or starts and article, then some users and bots make
from 5-10 really minor edits (style/category/tagging stuff), and then
occasionally some actual content edits are made. The Article Trust
code keeps bumping up the trust for pages. Looking at sample pages
there, it doesn't appear to be enough to get vandalism to near-white,
which is good. My only worry is the trollish user that adds
POV-vandalism and subtle vandalism (switching dates) will have it's
trust get too because bunch of users make maintenance edits
afterwards. Much of our main user base just does maintenance, so they
tend to have high "reputation" trust and make many such edits.
I think what would help, behind excluding bots (which is really a
no-brainer in my opinion) is to add some heuristics that devalue the
trust increase if someone is just make very small edits to the upper
or lower extremities of a page. This would catch a lot of
tag/category/interwiki link maintenance and stop it from bumping the
trust to much in the page's middle.
Still it is pretty good at picking up on the garbage, so it looks
promising. I'd be interesting in knowing what would happen if the
newest revision with all 7+ trust was marked as a stable version for
each page. Would that be good enough to pretty much be vandal-free?
-Aaron Schulz
------------------------------------------------------------------------
Date: Wed, 19 Dec 2007 19:44:14 -0800
Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
Daniel is making some very good points.
* Sock puppets
* People who split an edit into many smaller ones, done with
sock puppets or not, in order to raise the trust of text.
We think we know how to fix or at least mitigate both problems.
This is why I say that a "real-time" system that colors revisions
as they are made is a couple of months (I hope) away. The
challenge is not so much to reorganize the code to work from
wikipedia dumps to real-time edits. The challenge for us is to
analyze, implement, and quantify the performance of versions of
the algorithms that are resistant to attack. For those of you who
have checked our papers, you would have seen that not only we
propose algorithms, but we do extensive performance studies on how
good the algorithms are. We will want to do the same for the
algorithms for fighting sock puppets.
About the proposal by Daniel: time alone does not cover our full set of concerns.
I can every day use identity A to erase some good text, and
identity B to put it back in. Then, the reputation of B would grow
a bit every day, even though B did not do much effort.
We are thinking of some other solutions... but please forgive us
for keeping this to ourselves a little bit longer... we would like
to have a chance to do a full study before shooting our mouths off...
Luca
Hello Luca,
Post by Luca de Alfaro1. The trust coloring rightly colored orange (low-trust) some
unreliable content,
Yes I was lost in translation. ;-)
Post by Luca de Alfaro2. and the Wikipedia people were quick in reverting it.
Yes.
Post by Luca de AlfaroNote that we also highlight as low trust text that is by
anonymous
Post by Luca de Alfarocontributors. The text will then gain trust as it is revised.
One possible weakness came into my mind after I also read your paper. Your
algorithm is perhapes a bit vulnerable to "sock puppets". Imagine person A
with one account and person B with two accounts. Both have a medium
reputation value for their accounts. User A edits an article
with his account
4 times. All 4 subsequent edits are taken together and the article has a
maximum trust value according to the user's reputation. User B
makes as well
4 edits to an article but switches between his accounts and thus "reviews"
his own edits. If I understand your algorithm correctly the sock puppeted
article is trusted more than the other one.
Quite some time ago I reflected how to avoid incentives for sock puppets in
http://meta.wikimedia.org/wiki/Meritokratischer_Review (sadly
in German ;-).
The system described there differs from your approach but the
idea on how to
avoid incentives for sock puppets without even knowing who a sock puppet is
could perhapes adapted to your system.
The basic idea for a sock puppet proof metric is is that a
person has only a
limited amount of time for editing (I don't consider bots cause they are
easily detectable by humans). A single person needs the same
time for e.g. 4
edits (in the following I assume each edit has the same length in bytes)
regardless how much accounts are used but two different people with each 2
edits only need half of the (imaginary) time (you don't need to measure any
time untits at all).
So the maximum possible reliability person B can apply to the
article with its
two accounts (let us say each acount has 2 edits = 4 total edits) has to be
the same as the one which is possible with person A's single account (4
edits). So in general two accounts with each X edits should
never be able to
add more trust to an article than one person with 2*X edits
(note: edit count
number is only for illustration, you can take another appropriate
contribution unit).
Post by Luca de AlfaroAbout 2, I am very glad that bad edits are quickly reverted;
this is the
Post by Luca de Alfarowhole reason Wikipedia has worked up to now.
Still, it might be easier for editors to find content to
check via the
Post by Luca de Alfarocoloring, rather than by staring at diffs.
That's certainly true for articles not on your watchlist (or bad edits that
were forgotten and are still the latest version).
Post by Luca de Alfaro- Finding when flagged revisions are out of date (there
may be a new
Post by Luca de Alfarohigh-trust version later)
Well as I said I'd love to see flagged revisions and your
system combined (in
a way described by my previous mail). An automated system
probably always has
some weaknesses some clever people can abuse but it is very fast, while a
hand crafted system depends on the speed of individual persons but is much
harder to fool.
Post by Luca de AlfaroBTW, as the method is language-independent, we look forward
to doing the
Post by Luca de Alfarosame for wikipedias in other languages.
Good to know. :-)
Arnomane
_______________________________________________
Wikiquality-l mailing list
http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
------------------------------------------------------------------------
The best games are on Xbox 360. Click here for a special offer on an
Xbox 360 Console. Get it now!
<http://www.xbox.com/en-US/hardware/wheretobuy/>
------------------------------------------------------------------------
_______________________________________________
Wikiquality-l mailing list
http://lists.wikimedia.org/mailman/listinfo/wikiquality-l