03 Arithmeding PDF

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no.
1
Arithmetic coding
Elias coding
Arithmetic coding as finite-precision Elias
coding
Redundancy due to finite precision
Multiplication-free arithmetic coding
Context-adaptive coding

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2
Elias coding
Entropy coding algorithm for sequences of
symbols x with general (conditional) probabilities
Representation of x by a subinterval of the unit
interval [0,1)
Width of the subinterval is approximately equal to
the probability f
X
(x)
Subinterval for x can be determined by recursive
subdivision algorithm
Represent x by shortest binary fraction in the
subinterval
Subinterval of width f
X
(x) is guaranteed to contain
one number that can be represented by L binary
digits, with

( )
f
X
x
0.1111
0.111
0.1101
0.11
0.1011
0.101
0.1001
0.1
0.0111
0.011
0.0101
0.01
0.0011
0.001
0.0001
0
0
1
( )
2
log L f ~
X
x
Example: Elias coding of memoryless binary source
( ) ( )
1 3
0 1
4 4
X X
f f = =
0
1
1
4
n
c
n n
c a +
0
1
4
0
1
1
16
4
16
1
7
64
16
64
0
37
256
28
256
1
121
1024
148
1024
1 1
Example: Elias coding of memoryless ternary source
Courtesy: T. Wiegand

f
X
a
( )
= f
X
b
( )
=
1
4
f
X
c
( )
=
1
2
Elias coding: choose binary fraction in subinterval
Uniquely identify interval [c
n
,c
n
+a
n
) by a binary fraction

Length of bit-string

Bit-string

bits
0. [ , )
n
n n n
L
bbbbbb b c c a e +

L
n
= log
2
1
a
n
(
(
(
+1= h
X
0:n1
x
0:n1
( )
(
(
+1
2 2 1
n n
L L
n n n
c c c
(
= + >

Successive decoding
Elias bit string can be decoded symbol by symbol, starting with the
first symbol
For each symbol n=0,1,2, . . .
Calculate the intervals [c
n
,c
n
+a
n
) corresponding to all possible x
n
Determine the interval for which
Emit the corresponding x
n

)
,
n n n n
c c c a e +
Elias coding for memoryless source
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
+
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
i
X i X j
j
X K
F f o o
o o o o
=
=
Elias coding for changing probabilities
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n
n
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
+
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
n n
n
i
X i X j
j
X K
F f o o
o o o o
=
=
Elias coding for Markov random processes
Elias coding algorithm for Markov-p sources
{ }
( )
( )
: 1
: 1
0 0
1 | : 1
1 | : 1
Initialize 0 and 1.
For each 0,1,...
Set min ,
Update ,
Update ,
n n p n
n n p n
n n X n n p n
n n n X n n p n
c a
n
p p n
a a f x
c c a F x
'
'
' +
' +
= =
=
'
=
+
X
X
x
x
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative conditional
distribution (excluding
current symbol)
Up to p
previous
symbols
Arithmetic coding
Elias coding not practical for long symbol strings:
required arithmetic precision grows with string length
Finite precision implementations: arithmetic coding
Widely used in modern image and video compression algorithms:
JBIG, JPEG, JPEG-2000, H.263, H.264/AVC

n
x
Arithmetic
coder
Arithmetic
decoder
n
x
( )
PMF
n
X n
f x
( )
PMF
n
X n
f x
bit stream c
Finite precision for arithmetic coding
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
+
N-bit precision is
permissible, if
rounding down
P-bit approximation of
probabilities
At most N+P LSBs change,
except for a possible carry
affecting MSBs.
Each bit can be affected by
at most one carry.


Represent by
state variable C
n

Represent by
state variable r
n

Send
Finite precision for arithmetic coding (cont.)
Output code string

Maximum value of r
n
can be limited by bit stuffing

MSBs that will 1 bits LSBs that
no longer change represent lower that might be
interval boundary for affected by a
further computation carry
. 01111111 1
n
n
r N P
c xxxxxxxxx xxx cccccccc c
+ +
=
Inefficiency due to interval rounding
Recall: Subinterval of width f
X
(x) is guaranteed to contain one
number that can be represented by L
n
binary digits, with

Hence, rounding one interval value a
n+1
increases bit string by

L
n
~ log
2
f
X
0:n1
x
0:n1
( )

log
2
a
n
f
X
x
n
( )
a
n+1
< log
2
2
N1
+1
2
N1
= log
2
1+ 2
1N
( )
~
1
ln2
2
1N
( )
( )
( )
( ) ( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
2 2 2
log 1 log log 1
( )
1

ln 2
0
X X
X X
X X
X
X
X X
X X
x x
X X
f x f x
E f x E f x E
f x f x
H X
f x
E
f x
f x f x
E f x f x
f x f x
(
| | (
| | | | A A
(
+ = + ( | ( | |

| |
|
( (
\ . \ .
\ .

=
( A
~
(

( A A
= = A =
(

Limited precision probabilities
Efficiency loss
provided that rounded
probabilities still add to 1
Example: inefficiency for binary source
due to representing smaller probability by 2
-P

log
2
p
( )
r
e
l
a
t
i
v
e

b
i
t
-
r
a
t
e

i
n
c
r
e
a
s
e

Interval width is normalized to by binary shift after
each symbol
Approximation, with :

Corresponds to approximating of p(L) within a factor of 2:

Factor o can be lumped into external probabilities
Optimum o depends somewhat on data, typically

JBIG
Multiplication-free binary arithmetic coding
2 3
or 0.708 or
3 4
o o o = = =
( )
( )
( ) ( )
( ) (1 ( )) ( )
Ap L p L p L
Ap M A p L A p L A p L
o
o
'
~ =
'
= ~ =
1/ 2 1 A s <
p(L) s p(M)
( ) ( )
p L p L
A
o
~
MPS-LPS switch
Problem: interval width for symbol M can become negative

Solution: switch the role of M and L, if

Problem: interval for symbol M can be smaller than interval for L if

Solution: Conditional exchange of symbols, when problem occurs
Improves compression efficiency, implemented in JPEG and JBIG
( ) ( )
1
0 if ,1
2
A p L p L A

|
' '
< > e
|
( )
1
2
p L
'
>
( ) ( ) ( ) )
1 1
4 2
1
,
2
p L A p L p L A
' ' '
> > e
Sufficiency of binary coders

Consider r.v. equivalent to a r.v.

No loss by conditionally encoding bits (in any order)

Supply arithmetic coder with symbol/probability pairs

Total number of (conditional) probabilities to be estimated per symbol x

. . . same as with K-ary encoding of x
{ }
0,1, , 2 1
K
X e
0
1
1 K
B
B
B

| |
|
|
=
|
|
|
\ .
B
MSB
LSB
( ) ( )
( ) ( ) ( )
0 1 0 1 0 1 2
| | , ,
K K
H X H
H B H B B H B B B B

=
= + + +
B
( ) ( ) ( ) ( )
0: 1
0 0 1 1 1 1 | 0: 1
, , , , , , with 0,
k k
K K k B k
b p b p b p p f

=
B
b
1
1 2 2 2 1
K K
+ + + =
Consider stationary binary Markov-k process with probabilities

Context labeling function, enumerating all possible context vectors

Backward probability estimation separately for each context

Adaptive probability estimation
} {
n
X
( ) ( )
0: 1 ( ): 1
| 0: 1 | ( ): 1
0, 0,
k k n n k n
X k X n k n
f f

=
X X
x x
Context vector,
2
k
possible
combinations
( ) { } { }
: 0,1 0,1, , 2 1
k
k
e x x
| |
( )
( )
( )
( )
( )
0 ( ): 1
0
0 ( ): 1 1 ( ): 1
,
, ,
n k n
n k n n k n
C n
p n
C n C n

(
+ A

=
( (
+ A + + A

x
x x
Count of zeroes that
have previously
occured in that context
Probability
of symbol
x[n]=0
Bias for low counts,
typically A=1
Adaptive probability estimation (cont.)
Tracking changing statistics, compromise between
Rapid adaptation (small past sample counts)
Accurate estimates (large past sample counts)
Scaled count estimation algorithm

{ } { }
| |
0 1
1 1
0 0
0 1 min 0 1 max
0 1
0 1
0
0
0 1
Initialize 0
For 0,1,
If 1
C 1
else
C 1
If min C , or max C ,
C ; C
2 2
Estimate
2
n
C C
n
x
C
C
C C C C
C C
C
p n
C C
= =
=
=
+
+
> >
( (

( (

+ A
=
+ + A
Context adaptive coding
2-d extension of Markov model

Markov conditional independence property

N
+ n
causal half-plane
| |
x n
Coding
context

N+ n

f
X n
X
N
+n
X n
, X
N
+n
( )
= f
X n
X
N+n
X n
, X
N+n
( )
Context adaptive coding (cont.)
Coding context vector can be mapped into context label directly
without loss
Feasible for binary images
Example: JBIG uses 10 binary pixels as context label
For 8-bit images, number of different contexts
Context might have to be clustered
Combine with prediction

256
N

Reading
W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr., R. B.
Arps, An overview of the basic principles of the Q-Coder
adaptive binary arithmetic coder, IBM J. Res. Develop., vol.
32, no. 6, November 1988.
Witten, Radford, Neal, Cleary, Arithmetic Coding for Data
Compression Communications of the ACM, vol. 30, no. 6,
pp. 520-540, June 1987.
Moffat, Neal, Witten, Arithmetic Coding Revisited, ACM
Transactions on Information Systems, vol. 16, vo. 3, pp.
256294, July 1998.

03 Arithmeding PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

03 Arithmeding PDF

Uploaded by

Copyright:

Available Formats

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no.

You might also like