Appendix (Paper ID: 3382)

1 Proof of Lemma 1 satisfies

j = P( iI(j) wij zij < 0|qj = 1)
P
Proof. Define ij = wij zij , where wij = 2pij 1 is
the weight of peer i grading assignment j. Let j+ = = P+ ( iI(j) wij zij E [ iI(j) wij zij ] < j )
P P
E+ ( iI(j) wij zij ) =
P P
E(zij |qj = 1) = (2)
P iI(j) wijP
( )2
j
w (1p +(1)(1p )) = iI(j) wij (2pij 1)
2
P
iI(j) ij ij ij 2
iI(j) wij
be the expected value aggregated by WMV when the true 1e .
0 1
value of j is 1 and peer
P is reliability pij {pi , pi }. Finally, we have the expected accuracy rate of WMV on j, j
Similarly, let j = iI(j) E(pij zij |qj = 1) = satisfies
j = P( qj = qj )
P
iI(j) wij (2pij 1) as the expected value if the true
value of j is 1 and is reliability is pij . Let j+ = = P(qj = 1, qj = 1) + P(qj = 1, qj = 1)
P P
P( iI(j) wij zij > 0|qj = 1) be the expected accuracy rate = P(qj = 1)P( iI(j) wij zij > 0) (3)
of WMV if the true value of j is 1. This accuracy rate satisfies P
+ P(qj = 1)P( iI(j) wij zij < 0)
j+ = P( iI(j) wij zij 0|qj = 1) = P(qj = 1)j+ + P(qj = 1)j
P

Substituting j+ by Ineq.(1) and j by Ineq.(2), we have

P
P( iI(j) wij zij = 0|qj = 1)
= P( iI(j) wij zij E+ [ iI(j) wij zij ] > j+ )
P P +
( )2
j
2
P
+
2( )2 (1) 2
iI(j) wij
j j P(qj = 1)(1 e )
2
P
iI(j) (wij (wij ))
( )2
1e j
2
P
+
( )2
2
iI(j) wij (4)
j + P(qj = 1)(1 e )
2
P
2
iI(j) wij 1))2
P
=1e . ( w (2p
P ij ij2
iI(j)

The inequality holds because of the Hoeffding inequality that =1e

2
iI(j) wij .
X X The last equality hods because = (j )2 =
(j+ )2
P( wij zij E+ [ wij zij ] > j+ ) P 2
iI(j) iI(j)
( iI(j) wij (2pij 1)) . Thus, the expected error rate
X X P(qj 6= qj ) of each assignment j by the WMV aggregation
= 1 P( wij zij E+ [ wij zij ] j+ ) method satisfies
iI(j) iI(j) ( iI(j) wij (2pij 1))2
P

2 iI(j) w2
P
+
2( )2
j Pe (j) = P(qj 6= qj ) e ij . (5)
P 2
1e iI(j) (bi ai )
Replacing the weight wij by 2pij 1, we have
( iI(j) (2pij 1))2
P
where wij distributes in the range [ai , bi ] = [wij , wij ] uni-
Pe (j) = P(qj 6= qj ) e 2 . (6)
formly [Hoeffding, 1963]1 . Similarly, we can derive that the
expected accuracy rate of WMV if the true value of j is -1 And the expected accuracy rate P(qj = qj ) of each assign-
ment j by the WMV aggregation method satisfies

We will buy additional pages to include the appendix in the final Pa (j) = P(qj = 1 Pe (j) (7)
version if the paper is accepted. P 2
( iI(j) (2pij 1))
1
To make the Hoeffding inequality holds, the value of j+ 0
and in the latter weight setting, we will ensure this non-negative
= qj ) 1 e 2 . (8)
condition holds.
2 Proof of Theorem 3 j H, Pue (j, y)=Pue (j, x)), where each assignment in H
Submodular Property: Let S1 and S2 denote two diligent is graded diligently by the peers under both of x and y.

peer-assignment pair subsets, where S1S2 . For any diligen- Ineq.(10) holds because 1) j H, Pue (jH , x) Pue (j, x)
u
t peer-assignment grading (i , j ) S / 2 , S1 =S1 (i , j ) and and xj yj and 2) and j L, Pe (jL , x) Pue (j, x) and

S2 =S2 (i , j ), then, we have we proceed to compute xj yj . Eq.(11) holds because Pue (jL , y)Pue (jH

, y), where

the assignment jH that is fully diligently graded by the peers
n (S1 ) n (S1 ) (n (S2 ) n (S2 )) has a smaller expected error rate than expected error rate of
S
= Pue (j , I(j , S1 )\{i }) (1xSj 1 )(1xj 1 )Pue (j , {i })

the assignment jL that might be partially diligently graded

by the peers. Formally, denote jH s expected error rate of
S
Pue (j , I(j , S2 )\{i }) (1xSj 2 )(1xj 2 )Pue (j , {i })


(2p1 1)2
P
0.5
fully diligent grading as Pue (jH , x)=e iI(j )
H
i
,
S
= Pue (j , I(j , S1 )\{i }) (1xSj 1 )(1xj 1 )Pue (j , {i })
 where |I(jH )|=l, and jL s expected error rate of partially
(2p1i 1)2
P
0.5 0
Pue (j , I(j , S2 \S1 )\{i }) (1xSj 2 ) diligent grading as Pue (jL , x)=e iI (j )
L , where
0 0 0

S
I (jL ) I(jL ) and |I (jL )|=l l. Because each assign-
(1xj 2 )Pue (j , {i })


ment is randomly allocated to the peers with probability n1 ,

S then we have jH s expected error ratePof fully diligent grad-
(1 xSj 1 ) (1 xj 1 )Pue (j , {i }) l 1 2
ed by l peers is Pue (jH
, x)=e0.5 n iI (2pij 1) and jL s
Pue (j , I(j , S2 )\I(j , S1 ) \ {i }) 0
expected error rate of fully diligent graded by l peers is
S 0
(1xSj 2 )(1xj 2 )Pue (j , {i })


l
P 1 2 0
Pue (jL , x)=e0.5 n iI (2pi 1) . Because l l, we have
(1xSj 1 )(1x
S1
)Pue (j , {i })

Pue (jL , x)Pue (jH , x).
j P
Case 2), If K > j j . Formally, resort the assignments
S
(1xSj 2 )(1xj 2 )Pue (j , {i })


in decreasing order of their error rate under fully diligen-
S S t grading, i.e., Pue (1) Pue (2) . . . Pue (n), let x={x1 =
= (xSj 2 xSj 1 )Pue (j , {i })(xj 2 xj 1 ) 1 + xr1 , x2 = 2 + xr1 , . . . , xn = nP + xrn } denote Algorith-
S S S S
(xSj 2 xSj 1 )(xj 2 xj 1 ) = (xj 1 xSj 1 )(xj 2 xSj 2 )0.
r r
m 1s policy, where xj =max{K 1q<j xrq , 1 j } and
K r =K j j is the remaining budget after diligent elic-
P

3 Proof of Theorem 5 itation

P stage and y={y1 , y2 , . . . , yn }, where yj [0, 1] and
y
j j =K denote any other policy. Record the index k, where
Proof.
P We further split P the proof into two cases: 1) K = j [1, k], xj yj and j (k, n], xj < yj . Then, we have
j j and 2) K > j j .
P the aggregate accuracy rate difference between x and y is
Case 1), If K= j j . Formally, let x={x1 =
1 , x2 = 2 , . . . , xn = n } denote Algorithm n (x) n (y))
P 1s policy and
y={y1 , y2 , . . . , yn }, where yj [0, 1] and j yj =K as another


= 1jn (1 (1 xj )Pue (j, x)) (1 (1 yj )Pue (j, y))
P
policy. Then, we have (x)(y). We split the assignments

= 1jk (1 (1 xj )Pue (j, x)) (1 (1 yj )Pue (j, y))
P
J into two disjoint groups, H and L, where H = {j|yj xj }

and L = {j|yj < xj }. Denote jH = arg maxjH Pue (j, y)


+ k<jn (1 (1 xj )Pue (j, x)) (1 (1 yj )Pue (j, y))
P
as the assignment that has the largest error rate and jL =
1jk (xj yj )Pue (j, x) + k<jn xj yj )Pue (j, x)
P P
arg minjH Pue (j, y) as the assignment that has the smallest
error rate under x. Then, the aggregate accuracy rate differ- (12)
ence between x and y is u u
P P
Pe (k, x) 1jk (xj yj ) + Pe (k, x) kjn (xj yj )
n (x) n (y) (13)
u
 
= jJ (1 (1 xj )Pue (j, x)) (1 (1 yj )Pue (j, y))
P P
= Pe (k, x) 1jn (xj yj ) = 0
= jH ((1 yj )Pue (j, y) (1 xj )Pue (j, x))
P
Eq.(12) holds because under the fully diligent policy x, for
+ jL ((1 yj )Pue (j, y) (1 xj )Pue (j, x)) each assignment j has the smallest error rate that that under
P
the alternative policy y, i.e., Peu (j, x) Peu (j, y). Ineq.(13)
jH (xj yj ) Pue (j, y) + jL (xj yj ) Pue (j, y)
P P
holds because j k, Pue (j, x) Pue (k, x) and xj yj , and
j > k, Pue (j, x) < Pue (k, x) and xj < yj .
| {z } | {z }
0 0
(9)

Pue (jH
P
, y) jH (xj yj ) + Pue (jL , y)
P
yj ) References
jL (xj
(10) [Hoeffding, 1963] Wassily Hoeffding. Probability inequali-
ties for sums of bounded random variables. Journal of the
Pue (jL , y)
P
jJ (xj yj ) = 0 (11)
American Statistical Association, 58(301):1330, 1963.
Ineq.(9) holds because 1) j L, Pue (j, y) Pue (j, x))
where each assignment in x has a larger spot-checking prob-
ability, resulting a smaller error rate than that in y and 2)