You are on page 1of 137

Introduction to Ichip

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0


10Gbps PFE design
Lookup Memory
32MB RLDRAM

IIF/Wo Memory
32MB RLDRAM

Packet Memory and


Notification Memory
1GB DDR2

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0


I3.0 is being used (Reno use I2.0)
ADPC2(atlas-re0 vty)# show ichip 0 internal
(0x4b2321c8)

ichip : 0x4092f390

(0x4092f390)

which_ichip : 0

(0x4092f394)

version : 3

(0x4092f3bc)

debug_flags : 0x00000000

(0x4092f3cc)

ichip_initialized : TRUE

(0x4092f3d0)

ichip_real : TRUE

(0x4092f3d4)

run_periodics : TRUE

(0x4092f930)
(0x4092f3b0)

pio_base : 0xc0000000
pic_count : 1

Both use the same packet forwarding logic

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0


I3.0 supports 48 fabric destinations instead of
16 in I2.0
Narrower and faster HSL2 (3.7Gbps vs.
3.11Gbps)
WRR behavior difference in bonus region

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0


Ifi (Fabric Input)
This block receives the cells from the fabric, put them into
correct order and send them to Iwi for further processing.
Iwi (Wan Input)
This block receives the frames from the Wan interface
(NPEZ) and the cells from the Ifi, phrase them correctly,
perform the cellification and pass the cells to the Ipktwr
block.
Ipktwr (Packet Writer)
This block receives the cells from the Iwi block, write them
into the data packet memory via Im, create the indirect cell
if needed and construct a notification cell to Iiif

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0


Iiif (IIF index lookup)
This is a new functional block on our routing platform.
Instead of using channel lookup to determine the IIF, we
have used a compressed JTREE for IIF lookup. The table is
stored on a RLDRAM which is accessible via Isr block.
After the IIF has been determined, the complete notification
is formed and it will be forwarded to the next block (Irlkp).
Isr (Route / IIF lookup Memory Access)
This is a block being used to access the RLDRAM on the
chip. The RLDRAM has been used to store the routing
table, interface lookup table, nexthop database and
accounting.
Irlkp (Route Lookup)
This is where we perform route lookup. Also, this is where
we perform the BA and MF classification (except fixed
forwarding class assignment which is provided by Iiif).
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0


Im (Packet Data and Notification Memory Buffer)
This block is used to access the data packet and
notification memory.
Ip (Host Packet)
This is the block to handle the host traffic. It works with
Atlas Ichip Assistant (A-IA) FPGA for host bound packets.
Imq (Scheduling and Queuing)
This is where we performs the packet scheduling,
queuing and RED drop on the PFE.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Ichip Overview I3.0


Ipktrd (Packet Reader)
This block is used to extract the cells back from the Im.
Iwo (Wan Output)
This block is where we reassemble the packet from the
data cells and perform the L2 and L3 header
manipulations. It will access the RLDRAM via the Isr block
to extract the L2rw information extraction and perform
accounting.
Ifo (Fabric Output)
This is the block to send out the cells towards the egress
PFE via 4 fabric planes.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

Layer 3 Forwarding

VLAN id lookup on Ezchip (L2 service number)


Cellification from Iwi
Packet data write via Ipktwr
IIF lookup from Ichip (IIF index)
Route lookup on Ichip
Egress queue on Imq
Packet data read via Ipktrd
Packet reassemble and L23 hdr construction on Iwo
Egress cookie tagged with L2 service number
Service table lookup on Ezchip

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

10

Layer 3 Forwarding NPEZ


Ethernet frame would be handled by the build-in MAC
chip on Ezchip
VC lookup is done on the NPEZ
VLAN id is the key to determine the L2 service number
(l2sn)
The port number and service number would be used by
the Ichip (Iiif block) to determine the IIF index

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

11

Layer 3 Forwarding NPEZ


# set interfaces ge-0/0/0 unit 0 vlan-?
Possible completions:
vlan-id
Virtual LAN identifier value for 802.1q VLAN tags
vlan-id-range
Virtual LAN identifier range of form vid1-vid2
> vlan-tags
IEEE 802.1q tags
# set interfaces ge-0/0/0 unit 0 vlan-tags inner?
Possible completions:
inner-range
[tpid.]vid1-vid2, tpid format is 0xNNNN and is optional

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

12

Layer 3 Forwarding NPEZ


Dual tagged with inner-vlan-range
ge-2/0/7 {
stacked-vlan-tagging;
encapsulation extended-vlan-ccc;
unit 100 {
vlan-tags outer 100 inner-range 1-4094;
family ccc;
}
}
port
ovlan
ivlan
<--------Key-------->
0x000007 0x0064 0x0400
0xffffff 0xffff 0xfc00
0x000007 0x0064 0x0200
0xffffff 0xffff 0xfe00
0x000007 0x0064 0x0100
0xffffff 0xffff 0xff00
0x000007 0x0064 0x0080
0xffffff 0xffff 0xff80
0x000007 0x0064 0x0040
0xffffff 0xffff 0xffc0
.....
Copyright 2007 Juniper Networks, Inc.

ctrl
cond fc
bd
l2st l2sn l2iif
stp_idx
<-----------------Result------------------------------->
0x0b00 0x04 0x00 0x0007 0x07 0x0002 0x000000 0x0000
<-- 1024 to 2047
0x0b00 0x04 0x00 0x0007 0x07 0x0002 0x000000 0x0000
<-- 512 to 1023
0x0b00 0x04 0x00 0x0007 0x07 0x0002 0x000000 0x0000
<-- 256 to 511
0x0b00 0x04 0x00 0x0007 0x07 0x0002 0x000000 0x0000
<-- 128 to 255
0x0b00 0x04 0x00 0x0007 0x07 0x0002 0x000000 0x0000
<-- 64 to 127
Proprietary and Confidential

www.juniper.net

13

Layer 3 Forwarding NPEZ


Dual tagged
Logical interface ge-2/1/0.100 (Index 142) (SNMP ifIndex 180) (Generation 498)
Flags: SNMP-Traps 0x4000 VLAN-Tag [ 0x8100.200 0x8100.100 ] In(pop) Out(push
0x8100.200) Encapsulation: Extended-VLAN-VPLS
Logical interface ge-2/1/0.101 (Index 149) (SNMP ifIndex 181) (Generation 499)
Flags: SNMP-Traps 0x4000 VLAN-Tag [ 0x8100.200 0x8100.101 ] In(pop) Out(push
0x8100.200) Encapsulation: Extended-VLAN-VPLS
ADPC2(atlas-re0 vty)# show npez 2 vc-mt
port
ovlan
ivlan
ctrl
cond fc
bd
l2st l2sn l2iif
stp_idx
<--------Key-------->
<-----------------Result------------------------------->
0x000000 0x00c8 0x0064
0x3302 0x24 0x34 0x0014 0x03 0x0008 0x0000bc 0x0001
0xffffff 0xffff 0xffff
0x000000 0x00c8 0x0065
0x3302 0x24 0x34 0x0015 0x03 0x0009 0x0000bd 0x0001
0xffffff 0xffff 0xffff
ADPC2(atlas-re0 vty)# show npez 2 vc-ht
port epoch ovlan ivlan
ctrl cond
fc
bd
l2st l2sn l2iif
stp_idx
<-----------Key----------> <------------------------------Result------------------->
0
0
0x00c8 0x0064
0x3302 0x24 0x34 0x0014 0x03 0x0008 0x0000bc 0x0001
0
0
0x00c8 0x0065
0x3302 0x24 0x34 0x0015 0x03 0x0009 0x0000bd 0x0001
.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

14

Layer 3 Forwarding NPEZ


Single tagged
lab@atlas-re0> show interfaces ge-2/2/0
.....
Logical interface ge-2/2/0.100 (Index
Flags: SNMP-Traps 0x4000 VLAN-Tag [
.....
Logical interface ge-2/2/0.101 (Index
Flags: SNMP-Traps 0x4000 VLAN-Tag [

extensive
154) (SNMP ifIndex 188) (Generation 490)
0x8100.100 ] Encapsulation: ENET2
155) (SNMP ifIndex 189) (Generation 491)
0x8100.101 ] Encapsulation: ENET2

ADPC2(atlas-re0 vty)# show npez 1 vc-dt


total number of entries: 12
port
vlan
<-----Key---->
0
0x0064
0
0x0065
.....

ctrl
cond
fc
bd
l2st l2sn l2iif
stp_idx
<------------------------------Result-------------------->
0x0b00 0x04 0x15
0x0000 0x03 0x000a 0x000000 0x0000
0x0b00 0x04 0x15
0x0000 0x03 0x000b 0x000000 0x0000

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

15

Layer 3 Forwarding NPEZ


MAC Statistics
ADPC2(atlas-re0 vty)# show xeth-pic 1 port 0 stats
Stats for ifd : ge-2/1/0
Description
Rx Stats
Packet
446407
Reject
31
Reject DA
0
Reject SA
0
TX Pad
0
Packet Error
0
Octets
34351612
Packets
446407
Unicasts
399073
Broadcasts
46
Multicasts
47288
CRC/Align Errros
0
Undersize Pkts
0
Oversize Pkts
0
64 Octets Pkts
0
65-127 Octets Pkts
444984
128-255 Octets Pkts
1172
256-511 Octets Pkts
1
512-1023 Octets Pkts
0
1024-1518 Octets Pkts
187

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

Tx Stats
480497
0
0
0
0
0
36662411
480497
437037
86
43374
0
0
0
0
0
0
0
0
0

www.juniper.net

16

Layer 3 Forwarding NPEZ


(cont...)
1024-1518 Octets Pkts
MAC Control Frames
Unsupported Opcode
Pause Frames
OK Frames
OK Octets
In Range Length Errors
Out of Range Length Field
Code Violations
VLAN Tagged Frames
Ether Fragments
Jabber
Fifo Underrun
Packets
Bytes
Multicast Packets
Packet Rate
Byte Rate
Error
Replicated multicast pkts
Replicated multicast bytes
Replicated multicast pkts rate
Replicated multicast bytes rate
ADPC2(atlas-re0 vty)#
Copyright 2007 Juniper Networks, Inc.

187
0
0
0
446407
34351612
0
0
0
0
0
0
0
446407
34351612
47288
19
1495
0
46084
4428391
1
87
Proprietary and Confidential

0
0
0
0
0
0
0
0
0
0
0
0
0
480497
36662411
43374
24
2095
0
0
0
0
0
www.juniper.net

17

Layer 3 Forwarding Iwi


Packet with the NPEZ
RX cookie would be
sent to the Iwi block
8 x l23 ucode engine
for Wan and 8 x l23
ucode engine for
Fabric
Iwi (Wan Input)
This block receives the
frames from the Wan
interface (NPEZ) and the
cells from the Ifi, phrase them
correctly, perform the
cellification and pass the cells
to the Ipktwr block.
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

18

Layer 3 Forwarding Iwi


In Atlas, one single WAN Ichip stream would be used to
handle all ingress traffic.
On the egress direction, each IFD (or GE port) would
have its own WAN stream for QoS purpose.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

19

Layer 3 Forwarding Iwi

(cont.....)

ADPC2(atlas-re0 vty)# show ichip 1 ifd


ICHIP 1:
Initialized, Version 3,
STREAM 32 (wan stream 0) has 11 IFDs.
IFD 154: ge-2/2/0
IFD 155: ge-2/2/1
IFD 156: ge-2/2/2
IFD 157: ge-2/2/3
IFD 158: ge-2/2/4
IFD 159: ge-2/2/5
IFD 160: ge-2/2/6
IFD 161: ge-2/2/7
IFD 162: ge-2/2/8
IFD 163: ge-2/2/9
IFD 164: lc-2/2/0
.....

Ingress

ADPC0(calypso-re0 vty)# show ichip 0 ifd


ICHIP 0:
Initialized, Version 3,
STREAM 32 (wan stream 0) has 2 IFDs.
IFD 129: xe-0/0/0
IFD 130: lc-0/0/0
ADPC0(calypso-re0 vty)#

STREAM 33 (wan stream

1) has 2 IFDs.

IFD 154: ge-2/2/0


IFD 164: lc-2/2/0
STREAM 34 (wan stream

2) has 1 IFDs.

IFD 155: ge-2/2/1


STREAM 35 (wan stream

3) has 1 IFDs.

IFD 156: ge-2/2/2


STREAM 36 (wan stream

4) has 1 IFDs.

IFD 157: ge-2/2/3


STREAM 37 (wan stream
IFD 158: ge-2/2/4
STREAM 38 (wan stream

5) has 1 IFDs.

Egress

6) has 1 IFDs.

IFD 159: ge-2/2/5


STREAM 39 (wan stream

7) has 1 IFDs.

IFD 160: ge-2/2/6


STREAM 40 (wan stream

8) has 1 IFDs.

IFD 161: ge-2/2/7


STREAM 41 (wan stream

9) has 1 IFDs.

IFD 162: ge-2/2/8


STREAM 42 (wan stream 10) has 1 IFDs.
IFD 163: ge-2/2/9

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

20

Layer 3 Forwarding Iwi


Not the same case on the other Ichip platforms.
RFEB4(Jocelyn_RE0 vty)# show ichip 0 ifd
ICHIP 0:
Initialized, Version 2,
STREAM 32 (wan stream 0) has 2 IFDs.
IFD 169: at-4/0/0
IFD 170: at-4/0/1
STREAM 36 (wan stream 4) has 1 IFDs.
IFD 171: ge-4/1/0
STREAM 40 (wan stream 8) has 1 IFDs.
IFD 172: so-4/2/0
STREAM 41 (wan stream 9) has 1 IFDs.
IFD 173: so-4/2/1
STREAM 42 (wan stream 10) has 1 IFDs.
IFD 174: so-4/2/2
STREAM 43 (wan stream 11) has 1 IFDs.
IFD 175: so-4/2/3
RFEB4(Jocelyn_RE0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

21

Layer 3 Forwarding Iwi


Key Extraction
FTTL Check and initialization
Cellification

Iwi does not add overhead to the


cell data.The notification and the
cell data are sent out on
separate ports to packet writer

Adding 2 bytes CRC

if the LC size is <= 62 bytes

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

22

Layer 3 Forwarding Iwi


Per stream statistics (WAN)
ADPC2(atlas-re0 vty)# show ichip 1 wi winq statistics 0
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------WINQ Counter
582219
25
45
(BYTE)
48360659
2088
14283
ADPC2(atlas-re0 vty)# show ichip 1 wi winq errors 0
Counter Name
Total
---------------------- ---------------Drop
0
Drop Error
0
Abort
0
Abort Error
0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

23

Layer 3 Forwarding Iwi


Per stream statistics (WAN)
ADPC2(atlas-re0 vty)# show ichip 1 wi seg errors wan_stream 0
Counter Name
Total
---------------------- ---------------Timeout
0
Packet Header
0
Ucode Discard
0
TTL
20079
Checksum
0
Packet Len
0
Giant
0
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

24

Layer 3 Forwarding Ipktwr


Packet Writer block
Write cells received
from Iwi into the data
packet memory via Im
block

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

25

Layer 3 Forwarding Ipktwr


Construct a notification cell to Iiif block
Receive data from Iwi and Ip blocks (Host packet)
Spray the cells across the 12 memory bank via Im
block (768MB data memory buffer)
Create the indirect cell if needed (>= 6 cells)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

26

Layer 3 Forwarding Ipktwr


ADPC2(atlas-re0 vty)# show ichip 1 ipktwr statistics
Pktwr counters:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------pkt_drop
0
0
0
General Counters
----------------Counter Name
Total
---------------------- ---------------off_ovf
0
ntfq_drop
0
ntfb_drop
0
icellq_drop
0
icellb_drop
0
dcellq_drop
0
dcellb_drop
0
.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

27

Layer 3 Forwarding Ipktwr


ADPC2(atlas-re0 vty)# show ichip 1 ipktwr statistics bank
Bank Counter
-----------Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------bank[00] dc_wr_req
207556
7
42
bank[00] ic_wr_req
43
0
1
......
bank[10] dc_wr_req
207557
7
42
bank[10] ic_wr_req
41
0
1
bank[11] dc_wr_req
207558
7
41
bank[11] ic_wr_req
40
0
1
bank[12] dc_wr_req
0
0
0
bank[12] ic_wr_req
0
0
0
bank[13] dc_wr_req
0
0
0
bank[13] ic_wr_req
0
0
0
bank[14] dc_wr_req
0
0
0
bank[14] ic_wr_req
0
0
0
bank[15] dc_wr_req
0
0
0
bank[15] ic_wr_req
0
0
0
ADPC2(atlas-re0 vty)#
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

28

Layer 3 Forwarding Im
Mange the packet and
notification memory
4 DIMMs (16 banks) of
memory
12 banks are used for
data storage
4 banks are used for
notification cell storage
(DRAMQ)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

29

Layer 3 Forwarding Im
ADPC2(atlas-re0 vty)# show ichip 1 m dimm_presence
dimm0 size 256 MBytes
dimm1 size 256 MBytes
dimm2 size 256 MBytes
dimm3 size 256 MBytes
sdram device size 512 Mbit
sdram device type DDR-II
ADPC2(atlas-re0 vty)# show ichip 1 m errors dimm
<number>
select dimm number
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

30

Layer 3 Forwarding Im
ADPC2(atlas-re0 vty)# show ichip 1 m errors dimm 0
Error interrupts : dimm 0
hbit_err_e
0
hbit_err_o
0
ecc_err
0
unknown
0
ECC error status bits detected by polling: dimm 0
uncorrectabl err bit set 0 times
correctable err bit set 0 times
Memory errors: dimm 0
ECC uncorr. error count
ECC corr. error count
ECC currnet error status:
ECC syndrome
ECC err address
ECC err data_63_32
ECC err data_31_0
Heart beat:
err hbit_cnt_sel
err hbit_cnt_val
Copyright 2007 Juniper Networks, Inc.

0
0
none
0x00000000
0x00000000 (bank 0 cell 0)
0x00000000
0x00000000
0x00000000
0
Proprietary and Confidential

www.juniper.net

31

Layer 3 Forwarding Iiif


iif indexes are stored in
a Compressed JTREE
format.
The lookup table is
stored in the RLDRAM
which is accessible
thru Isr block
iif index lookup would
be performed like route
lookup
No more channel
lookup limitation we
have on Martini and
Gimlet chipsets.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

32

Layer 3 Forwarding Iiif


Incoming notif from Ipktwr
is divided into 3 parts
IIF key data is used for IIF
lookup
Keepalive Q is for the Host
injected packets with KA
flag
No_IIF Q is for the packets
from the Ifi or no IIF lookup
is required
Result is combined with the
original Notif hdr data and
Route lookup key data
Perform fixed classification

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

33

Layer 3 Forwarding Iiif


Per-stream configuration
ADPC2(atlas-re0 vty)# show ichip 1 iif wan_stream 0
Stream 32
====================
stream type: wan
initialized: yes
sst register
:0xa000006c 0x00000000 0x00080080
(in-core shadow:0xa000006c 0x00000000 0x00080080)
enable: yes tt: 0 aen: 1 srw: 0 l30w_sel: 0 l30w_en: 0 l30w: 0
default_iif: 108 (0x6c)
qs: 0 qs_sel: 0 dp: 0 dp_sel: 0 smp: 0 smp_sel: 0 pm: 0 pm_sel: 0
st_inst_ptr: 0x000100 (byte-offset 0x00000100)
bits: 0 op: branch bit_offset: 0
iif overwrite disable (0x00000000): no
.....
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

34

Layer 3 Forwarding Iiif


Per ifl accounting
IFD
154

ge-2/2/0

Ethernet

0x0000000000008000

2 Up

IFL
145
146
156

ge-2/2/0.100
ge-2/2/0.101
ge-2/2/0.32767

VLAN Tagged
VLAN Tagged
VLAN Tagged

Ethernet
Ethernet
Ethernet

0x000000000000c000
0x000000000000c000
0x000000000000c000

ADPC2(atlas-re0 vty)# show iif ifd 154


ge-2/2/0, stream 0, chip 1, lookup idx 2
Stream default 0xa000006c, srw 0x00000000, iif 108
Lookup tree:
idx chan_type key_len level tree_type stream refcnt
----- ------------- ------------2
8
3
1
1
0
0
Chan1 Chan2
iif
Pkt_cnt
pkt_rate
byte_cnt
continue
------ ------ ----- ------------- -------- ------------- -------0x0
n/a
156
0
0
0 0x40db17d0
0xa
n/a
145
14420
1
1145617
0xb
n/a
146
203052
23
15153259

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

35

Layer 3 Forwarding Iiif


Fixed classification
ge-2/2/0 {
unit 101 {
forwarding-class assured-forwarding;
}
}
ADPC2(atlas-re0 vty)# show iif ifd 154 stat
ge-2/2/0, stream 0, chip 1, lookup idx 2
Stream default 0xa000006c, srw 0x00000000, iif 108
Lookup tree:
idx chan_type key_len level tree_type stream
----- ------------- --------2
8
3
1
1
0
Chan1 Chan2
iif
enable
QN
cnt_addr
------ ------ ----- ------ ---- ----------0x0
n/a
156
yes
dis 0x4af218e0
0xa
n/a
145
yes
dis 0x4af218b8
0xb
n/a
146
yes
2
0x4af21908
total channel entries entries 3

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

refcnt
----0

www.juniper.net

36

Layer 3 Forwarding Iiif


iif lookup
Using the Layer 2 Service Number (l2sn) within the EZchip RX cookie
bytes as a key
For non-Atlas platform, IIF lookup is done by looking at the vlan
tags/DLCI values/Cookie indexetc
145
146

ge-2/2/0.100
ge-2/2/0.101

VLAN Tagged
VLAN Tagged

Ethernet
Ethernet

0x000000000000c000
0x000000000000c000

ADPC2(atlas-re0 vty)# show npez 1 vc-dt port 0


total number of entries: 3
port
vlan
<-----Key---->
0
0x0064
0
0x0065
0
0x1001

ctrl
cond
fc
bd
l2st l2sn l2iif
stp_idx
<------------------------------Result-------------------->
0x0b00 0x04 0x15
0x0000 0x03 0x000a 0x000000 0x0000
0x0b00 0x04 0x15
0x0000 0x03 0x000b 0x000000 0x0000
0x0b00 0x00 0x00
0x0000 0x00 0x0000 0x000000 0x0000

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

37

Layer 3 Forwarding Iiif


iif lookup
ADPC2(atlas-re0 vty)# show ichip 1 iif sram_decoded wan_stream 0
start decoding with search start instruction
=============================================
>> search start instruction :00080080 key_pointer:0 bits
br_base:word offset 0x200, 256-way br, bit offset:0
depth 1 branch idx 0, word_offset 0x000200:0003f994
key_pointer:0 bits
br_base:word offset 0xfe, 2-way br, bit offset:20
depth 2 branch idx 0, word_offset 0x0000fe:000de718
key_pointer:0 bits
act_base:word offset 0x378) r/c/p:111 bit offset:24
prefix length 24 bits
1 prefix words 0x00000000
key pointer move: 3 bytes
continuation instruction: 0x00040080
frw: b000009c ena:1 tt:0 aen:1 iif:156
srw: 00000000
instruction : 0x00040080 key_pointer:24 bits
br_base:word offset 0x100, 256-way br, bit offset:0
stop decoding at lookup depth 3
.....
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

38

Layer 3 Forwarding Iiif


(cont...)
depth 2 branch idx 1, word_offset 0x0000ff:00021197
key_pointer:0 bits
br_base:word offset 0x84, 2-way br, bit offset:23
depth 3 branch idx 0, word_offset 0x000084:000c0518
key_pointer:0 bits
act_base:word offset 0x300) r/c/p:101 bit offset:24
prefix length 24 bits
1 prefix words 0x00000a00
key pointer move: 3 bytes
frw: b0000091 ena:1 tt:0 aen:1 iif:145
srw: 00000000
repeat until mismatch or no bits in key buffer
depth 3 branch idx 1, word_offset 0x000085:000c2518
key_pointer:0 bits
act_base:word offset 0x308) r/c/p:101 bit offset:24
prefix length 24 bits
1 prefix words 0x00000b00
key pointer move: 3 bytes
frw: b0000092 ena:1 tt:0 aen:1 iif:146
srw: 00000000
repeat until mismatch or no bits in key buffer
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

39

Layer 3 Forwarding Iiif


Per-stream Lookup Statistics
ADPC2(atlas-re0 vty)# show ichip 1 iif statistics wan_stream 0
Traffic stats:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------wan_strm[00] total
724289
25
45
(BYTE)
54309380
1917
13980
wan_strm[00] default
0
0
0
(BYTE)
0
0
0
Error counters:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------wan_strm[00] uaiif
0
0
0
wan_strm[00] tmo
0
0
0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

40

Layer 3 Forwarding Iiif


Global Lookup Statistics
ADPC2(atlas-re0 vty)# show ichip 1 iif statistics
Traffic stats:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------GFAB_BCNTR
27223571
960
16375
KA_PCNTR
300547
10
16
KA_BCNTR
29769716
1019
1822
Discard counters:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------WAN_DROP_CNTR
0
0
0
FAB_DROP_CNTR
0
0
0
KA_DROP_CNTR
0
0
0
HOST_DROP_CNTR
0
0
0
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

41

Layer 3 Forwarding Irlkp


13 Key Engines
Perform BA table lookup
and MF classification
Perform route lookup
Jtree and firewall filter are
stored on the RLDRAM
which is accessible thru
Isr block
Host packet notification
would be sent to Ip block
Transit traffic notification
is queued in Imq block

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

42

Layer 3 Forwarding Irlkp


Segment 0 is for Routes 16MB
Segment 1 is for filters/policers 16MB
Cannot be shared unless its in enhanced memroy mode
ADPC2(atlas-re0 vty)# show jtree 1 memory
Jtree memory segment 0 (Context: 0x453cd18c)

(cont...)

-------------------------------------------

Jtree memory segment 1 (Context: 0x4544ed8c)

Memory Statistics:

-------------------------------------------

16777216 bytes total

Memory Statistics:

1373904 bytes used

16777216 bytes total

15396976 bytes available (15395840 bytes from free pages)


6048 bytes wasted

12319112 bytes available (12317696 bytes from free pages)

288 bytes unusable

1008 bytes wasted

32768 pages total

32 bytes unusable

2692 pages used (2586 pages used in page alloc)


6 pages partially used
30070 pages free (max contiguous = 30070)
.....

4457064 bytes used

32768 pages total


8706 pages used (8706 pages used in page alloc)
4 pages partially used
24058 pages free (max contiguous = 24058)
Context: 0x433c311c
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

43

Layer 3 Forwarding Irlkp


Ir block statistics
ADPC2(atlas-re0 vty)# show ichip 1 r counters
Traffic stats:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------rcp_input_ucast

1404078

45

64

(BYTE)

111884214

3630

15621

rcp_output_ucast

1335194

43

61

(BYTE)

106668714

3503

15202

rcp_input_mcast

92378

54

(BYTE)

8592088

303

15973

rcp_output_mcast

92378

54

(BYTE)

8592088

303

15973

.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

44

Layer 3 Forwarding Irlkp


(cont...)
Discarded Packets:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------JTREE_TRUNC_KEY

NEXHOP_TRUNC_KEY

ITABLE_TRUNC_KEY

TIMEOUT

IIF_CHECK

12

SRAM_PAR_ERR

EXEC_STACK_OFLOW

SID_ZERO

85

FW_DROP

POLICER_DROP

INDEX_NBITS

FW_ILLEGAL_INSTR

RCP_INPUT_MCAST_DROP

RCP_UBR_CNTR

.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

45

Layer 3 Forwarding Irlkp


(Cont...)
HNP discards:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------Q0_DCS_CNTR

Q1_DCS_CNTR

Q2_DCS_CNTR

Q3_DCS_CNTR

Q4_DCS_CNTR

Q5_DCS_CNTR

Q6_DCS_CNTR

Q7_DCS_CNTR

.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

46

Layer 3 Forwarding Irlkp


(Cont...)
HNP ages:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------Q0_AGE_CNTR

Q1_AGE_CNTR

Q2_AGE_CNTR

Q3_AGE_CNTR

Q4_AGE_CNTR

Q5_AGE_CNTR

Q6_AGE_CNTR

Q7_AGE_CNTR

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

47

Layer 3 Forwarding Irlkp


Ir block error statistics
ADPC2(atlas-re0 vty)# show ichip 1 r errors
MLP errors:
0 Parity Errors For Port0
0 Parity Errors For Port1
0 Data Buffer Overflow
0 Data Buffer Underflow
0 Mcast List Request Buffer Overflow
0 Indir Mcast List Request Buffer Overflow
0 Mcast Indirect Errors
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

48

Layer 3 Forwarding Irlkp


Ir block Hash key configuration
ADPC2(atlas-re0 vty)# show ichip 1 r configuration ricp keyhash wan proto
<number>

show r ricp key hash wan configuration proto (0 ..7)

ADPC2(atlas-re0 vty)# show ichip 1 r configuration ricp keyhash wan proto 0


Ir Hash Byte Count: 17
Ir Hash Byte Selection:
Hash Word

Source Byte

Nibble Mask

----------

-----------

-----------

0x88090a10

himask

0x88090a10

nomask

0x88090a10

10

nomask

0x88090a10

16

nomask

0x11121314

17

nomask

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

49

Layer 3 Forwarding Irlkp


Ir block Hash key configuration
* Field Name

Bit Range

Summary

* mask 0

31:30

nibble mask for select 0

* select0

29:24

source byte select 0

* mask 1

23:22

nibble mask for select 1

* select1

21:16

source byte select 1

* mask 2

15:14

nibble mask for select 2

* select2

13:8

source byte select 2

* mask 3

7:6

nibble mask for select 3

* select3

5:0

source byte select 3

typedef enum nibble_mask_def {


NIBBLE_MASK_DEF_NOMASK = 0,
NIBBLE_MASK_DEF_LOMASK = 1, (Mask bit [3:0] of byte to zero)
NIBBLE_MASK_DEF_HIMASK = 2, (Mask bit [7:4] of byte to zero)
NIBBLE_MASK_DEF_ALLMASK = 3 (Mask all bits in byte to zero)
} nibble_mask_def;

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

50

Layer 3 Forwarding Irlkp


Ir block Hash key configuration
Ir Hash Byte Selection:
Hash Word

Source Byte

Nibble Mask

----------

-----------

-----------

0x88090a0d

himask = b'(10)(001000) (00)(001001)

0x88090a0d

nomask

0x88090a0d

10

nomask

(00)(001010) (00)(001101)

|---------------------------------------------------------------|
|

rewrite 0

rewrite 1

rewrite 2

rewrite 3

|[0-31]

|---------------------------------------------------------------|
|

|pol|

|L|

|sample sn[7:0] |res| DP|

QS |3|

L3OW

|1:0|1:0|[2:0]|N|

[5:0]

|E|C|

|T|O|rewr |

|C|E| PT

|E|P| ite |[32-63]

|T| |[2:0]| | | X

|---------------------------------------------------------------|
|

iif[19:0]

| pkt_len[15:8] |

|---------------------------------------------------------------|

8th to 10th bytes

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

51

Layer 3 Forwarding Isr


This Block is a memory
controller for the RLDRAM
4 x 288Mbit RLDRAM part
(128MB in total)
JTREE data structure is
replicated to two RLDRAM
parts for higher performance
Effective memory size is
32MB for Ir, 8MB for Iiif and
24MB for Iwo

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

52

Layer 3 Forwarding Isr


RLDRAM Partition
ADPC2(atlas-re0 vty)# show ichip
sr partition information:
r partition size
0x02000000
r partition base
0x30000000
iif partition size 0x00800000
iif partition base 0x18000000
wo partition size 0x01800000
wo partition base 0x18800000

2 isr partition
32MB
8MB
24MB

WO sram block partition (offset to start of WO sram)


mit/mcast base
0x00000000
l2desc base
0x00800000
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

53

Layer 3 Forwarding Isr


RLDRAM Access Statistics
ADPC2(atlas-re0 vty)# show ichip 2 isr stats
RLDRAM accessing statistics
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------R1_PLC_WR
0
0
0
R1_PLC_RD
0
0
0
R1_ACC_WR
1217797
0
335104
R1_ACC_RD
284904
0
31
R1_SRM_RD
6933036548849
178630500
198728991
R2_PIO_WR
1199351
0
335104
R2_PIO_RD
0
0
0
R2_SRM_RD
6810030540404
178666166
198753016
MLP_SRM_RD
0
0
0
IIFWO1_ACC_WR
9359
0
10
IIFWO1_ACC_RD
596198
0
25
IIFWO1_SRM_RD
20086203
21000
52666
IIFWO2_PIO_WR
9284
0
10
IIFWO2_PIO_RD
0
0
0
IIFWO2_SRM_RD
10222923
10700
27033
ADPC2(atlas-re0 vty)#
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

54

Layer 3 Forwarding Isr


RLDRAM Replication

R1 part is used to store the firewall counters / policer counters


JTREE is replicated in both R1 and R2 parts
JTREE unicast route lookup can be done in either R1 and R2 parts
Firewall / policer counters would only be read/updated in R1 part
Multicast lists would only be read/updated in R2 part
Similar on IIFWO1 and IIFWO2 parts. (Iiif / nexthop stats update
only happens on part 1)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

55

Layer 3 Forwarding Isr


RLDRAM Error Statistics
ADPC2(atlas-re0 vty)# show ichip 2 isr errors
0 rkp_acc_err
0 r1_par_cnt
0 r1_par_adr
0 r2_par_cnt
0 r2_par_adr
0 iifwo_acc_err
0 iifwo1_par_cnt
0x00000000 iifwo1_par_adr (hashed 0x00000000)
0 iifwo2_par_cnt
0x00000000 iifwo2_par_adr (hashed 0x00000000)

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

56

Layer 3 Forwarding Imq


Queuing the notification for
both Wan and Fabric traffic
Both Head and Tail RED are
available
One cell can store 3
notifications
Maximum 32 Wan streams
Maximum 48 Fabric streams
8 queues per Wan stream
2 queues per Fabric stream
4 Priority, 4 DPs
Per DP counter
128 32-bit RED drop profiles

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

57

Layer 3 Forwarding Imq


8 Priority Levels

4 in positive credit regions (High, Medium-High, Medium-Low, Low)


4 in negative credit regions (High, Medium-High, Medium-Low,
Low)

WRR behavior in bonus region I3.0


DRAMQ and Data buffer size follow BUF allocation in order
to control the latency
Mad is supported

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

58

Layer 3 Forwarding Imq


Example: Mas and DRAMQ size Assignment
schedulers {
be {
transmit-rate percent 40;
buffer-size percent 25;
}
ef {
transmit-rate percent 30;
buffer-size percent 25;
}
af {
transmit-rate percent 20;
buffer-size percent 25;
}
nc {
transmit-rate percent 10;
buffer-size percent 25;
priority high;
}

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

59

Layer 3 Forwarding Imq


ADPC0(mx480a-re0 vty)# show ichip 0 imq config stream wan_stream 1 qos
Stream 33:
----------prv_mode
-------queue 0
2
queue 1
2
queue 2
2
queue 3
2
queue 4
2
queue 5
2
queue 6
2
queue 7
2

rate_ctrl
--------false
false
false
false
false
false
false
false

bo_prio
------0
0
0
3
0
0
0
0

Stream 33: bonus details


------------------------bo_prio
bo_rate(%)
----------------queue 0
0
40
queue 1
0
30
queue 2
0
20
queue 3
3
10
queue 4
0
0
queue 5
0
0
queue 6
0
0
queue 7
0
0

Copyright 2007 Juniper Networks, Inc.

dq_prio
------0
0
0
3
0
0
0
0

bo_weight
--------2633
1975
1317
659
0
0
0
0

dis_dq
-----false
false
false
false
false
false
false
false

weight
-------2633
1975
1317
659
0
0
0
0

clamp
-------10532
7900
5268
2636
0
0
0
0

mas
-------24576
24576
19748
10094
444
444
444
444

tx_rate(%) delay_bw(%)
--------- ----------40
25
30
25
20
25
10
25
0
0
0
0
0
0
0
0

bo_clamp
-------10532
7900
5268
2636
0
0
0
0

Proprietary and Confidential

www.juniper.net

60

Layer 3 Forwarding Imq


ADPC0(mx480a-re0 vty)# show ichip 0 imq config stream wan_stream 1 mem
stream 33
queue
queue
queue
queue
queue
queue
queue
queue

0
1
2
3
4
5
6
7

HNQ range(size)
------------------0x0600:060b(0x000c)
0x060c:0617(0x000c)
0x0618:0623(0x000c)
0x0624:062f(0x000c)
0x0630:063b(0x000c)
0x063c:0647(0x000c)
0x0648:0653(0x000c)
0x0654:065f(0x000c)

RAMQ range(size)
----------------------0x380000:381f6b(0x001f6c)
0x381f6c:383ed7(0x001f6c)
0x383ed8:385e43(0x001f6c)
0x385e44:387daf(0x001f6c)
0x387db0:387e43(0x000094)
0x387e44:387ed7(0x000094)
0x387ed8:387f6b(0x000094)
0x387f6c:387fff(0x000094)

mas
-------24576
24576
19748
10094
444
444
444
444

ADPC0(mx480a-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

61

Layer 3 Forwarding Imq


Queue Utilization (Mu/Mad)
ADPC0(mx480a-re0 vty)# show ichip 0 imq config stream wan_stream 1 mu_mad
Stream 33 mas/mu/mad/hnq_ptr info:
queue
-------

mas
--------

mu

mad

--------

--------

hnq_ptr(0x)
---------------

24576

000600:000600

24576

00060c:00060c

19748

000618:000618

10094

000624:000624

444

000630:000630

444

00063c:00063c

444

000648:000648

444

000654:000654

ADPC0(mx480a-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

62

Layer 3 Forwarding Imq


Queue Utilization
ADPC0(mx480a-re0 vty)# ... ichip 0 imq config stream wan_stream 1 empty

stream 33

TNQ empty

HNQ empty

DRAMQ empty

---------

---------

-----------

queue

true

true

true

queue

true

true

true

queue

true

true

true

queue

true

true

true

queue

true

true

true

queue

true

true

true

queue

true

true

true

queue

true

true

true

ADPC0(mx480a-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

63

Layer 3 Forwarding Imq


RED Profile
ADPC9(atlas-re0 vty)# show ichip 0 imq config red profile 0
COS profile ID: 65534
RED physical profile id: 0, profile data(in HEX):
0x0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff
ADPC9(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

64

Layer 3 Forwarding Imq


Imq Global Statistics
ADPC9(atlas-re0 vty)# show ichip 0 imq stats
Imq global stats:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------OUT_PKT_TOTAL_WAN
1210187
24
37
(BYTE)
65673855
1316
13724
OUT_PKT_TOTAL_FAB
4
0
0
(BYTE)
312
0
21
RDROP_WAN_PKT
0
0
0
RDROP_FAB_PKT
0
0
0

ADPC9(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

65

Layer 3 Forwarding Imq


Per stream statistics
ADPC2(Atlas-RE0 vty)# show ichip 0 imq stats wan_stream 1
Stream 33 IN_OFFLOAD_PKT_BYTE_CNT:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------queue 0
0
0
0
(BYTE)
0
0
0
queue 1
0
0
0
(BYTE)
0
0
0
queue 2
0
0
0
(BYTE)
0
0
0
queue 3
0
0
0
(BYTE)
0
0
0
queue 4
0
0
0
(BYTE)
0
0
0
queue 5
0
0
0
(BYTE)
0
0
0
queue 6
0
0
0
(BYTE)
0
0
0
queue 7
0
0
0
(BYTE)
0
0
0
.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

66

Layer 3 Forwarding Imq


Per stream statistics
(Cont...)
stream 33 OUT_PKT_Q:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------queue 0
0
0
0
(BYTE)
0
0
0
queue 1
0
0
0
(BYTE)
0
0
0
queue 2
0
0
0
(BYTE)
0
0
0
queue 3
0
0
0
(BYTE)
0
0
0
queue 4
0
0
0
(BYTE)
0
0
0
queue 5
0
0
0
(BYTE)
0
0
0
queue 6
0
0
0
(BYTE)
0
0
0
queue 7
0
0
0
(BYTE)
0
0
0
ADPC2(Atlas-RE0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

67

Layer 3 Forwarding Imq


Per queue statistics
ADPC2(atlas-re0 vty)# show ichip 2 imq stats wan_stream 0 queue 0
physical queue 0:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------OFFLOAD_PKT_BYTE

(BYTE)

OUT_PKT_Q

(BYTE)

TD_PKT_BYTE non-red

(BYTE)

TD_PKT_BYTE red dp00

(BYTE)

TD_PKT_BYTE red dp01

(BYTE)

TD_PKT_BYTE red dp10

(BYTE)

TD_PKT_BYTE red dp11

(BYTE)

.......

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

68

Layer 3 Forwarding Imq


Per queue counter
(cont.....)
RDROP_PKT_Q DP00

(BYTE)

RDROP_PKT_Q DP01

(BYTE)

RDROP_PKT_Q DP10

(BYTE)

RDROP_PKT_Q DP11

(BYTE)

ECN_MARKED_PKT_Q DP00

ECN_MARKED_PKT_Q DP01

ECN_MARKED_PKT_Q DP10

ECN_MARKED_PKT_Q DP11

Age status: no age


ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

69

Layer 3 Forwarding Imq


Age Mask
ADPC5(calypso-re0 vty)# show ichip 2 imq config age_mask
Streams configued in AGE_MASK registers:
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
Age counter:
WAN queue
counter
----------------0
0
1
0
2
0
3
0
4
0
5
0
6
0
7
0
FAB queue
counter
----------------0
0
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

70

Layer 3 Forwarding Imq


Priority Mapping

48 PFE destinations x 2 priority levels = 96 queues


ADPC2(atlas-re0 vty)# show ichip 2 imq config fab sid_qs
FAB qs sid/qs mapping table:
input SID output SID
--------- ---------64
64
65
65
66
66
67
67
.............
107
107
108
108
109
109
110
110
111
111

priority mapping (q0..q7)


------------------------lo lo lo lo lo lo lo lo
lo lo lo lo lo lo lo lo
lo lo lo lo lo lo lo lo
lo lo lo lo lo lo lo lo
lo
lo
lo
lo
lo

lo
lo
lo
lo
lo

lo
lo
lo
lo
lo

lo
lo
lo
lo
lo

lo
lo
lo
lo
lo

lo
lo
lo
lo
lo

lo
lo
lo
lo
lo

lo
lo
lo
lo
lo

fab stream(lo/hi)
----------------0/1
2/3
4/5
6/7
86/87
88/89
90/91
92/93
94/95

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

71

Layer 3 Forwarding Imq


Two queues per FAB destination (High and Low)
ADPC0(mx480a-re0 vty)# show ichip 2 imq stats fab_stream 0
Stream 64 IN_OFFLOAD_PKT_BYTE_CNT:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------queue 0
0
0
0
(BYTE)
0
0
0
queue 1
0
0
0
(BYTE)
0
0
0
stream 64 OUT_PKT_Q:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------queue 0
0
0
0
(BYTE)
0
0
0
queue 1
0
0
0
(BYTE)
0
0
0
ADPC0(mx480a-re0 vty)#
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

72

Layer 3 Forwarding Ipktrd


Packet Reader block
Retrieve data back from the
Im block according to the
notification received from the
Imq / Ip block
Data goes to either Iwo (Wan)
or Ifo (Fabric)
Send cell buffer reservation
request to Ifo for fabric data
Back pressure to Imq if the
stream is congested

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

73

Layer 3 Forwarding Ipktrd


Per data bank statistics
ADPC2(atlas-re0 vty)# show ichip 2 ipktrd statistics bank
<carriage return>

Completes command

<number>

bank number (0 to 11)

ADPC2(atlas-re0 vty)# show ichip 2 ipktrd statistics bank 0


Bank Statistics
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------wan_bank[00] icrd

11

hnf_bank[00] icrd

20

wan_bank[00] dcrd

122612

31

hnf_bank[00] dcrd

124782

37

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

74

Layer 3 Forwarding Ipktrd


Per stream error statistics
ADPC2(atlas-re0 vty)# show ichip 2 ipktrd errors
all

show all stream error counters

fab_stream

show fab stream error counters

hst_stream

show host stream error counters

wan_stream

show wan stream error counters

ADPC2(atlas-re0 vty)# show ichip 2 ipktrd errors wan_stream


<number>

wan stream 0..31

ADPC2(atlas-re0 vty)# show ichip 2 ipktrd errors fab_stream


<number>

Copyright 2007 Juniper Networks, Inc.

fab stream 0..31 or 95

Proprietary and Confidential

www.juniper.net

75

Layer 3 Forwarding Ipktrd


Per stream error statistics
ADPC2(atlas-re0 vty)# show ichip 2 ipktrd errors wan_stream 0
Cell-error Counters:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------wan_strm[00] ecc

wan_strm[00] err

wan_strm[00] nt_age

wan_strm[00] ic_age

wan_strm[00] dc_age

General error Counters:


Counter Name

Total

---------------------- ---------------wan_strm[00] byc_oe

wan_strm[00] byc_ue

wan_strm[00] prq_oe

wan_strm[00] crq_oe

wan_strm[00] icb_oe

ADPC2(atlas-re0 vty)#
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

76

Layer 3 Forwarding Ipktrd


Per stream status
ADPC2(atlas-re0 vty)# show ichip 2 ipktrd qstatus
WAN Queue Status
WAN_PRQ_MPTY -

0xffffffff (Packet Read Queue empty)

WAN_ICB_MPTY -

0xffffffff (Icell buffer Queue empty)

WAN_PRD_DONE -

0xffffffff (Packet Read Done Queue)

FABRIC Queue Status


FAB_PRQ_MPTY[0] -

0xffffffff

FAB_PRQ_MPTY[1] -

0xffffffff

FAB_PRQ_MPTY[2] -

0xffffffff

FAB_ICB_MPTY[0] -

0xffffffff

FAB_ICB_MPTY[1] -

0xffffffff

FAB_ICB_MPTY[2] -

0xffffffff

FAB_PRD_DONE[0] -

0xffffffff

FAB_PRD_DONE[1] -

0xffffffff

FAB_PRD_DONE[2] -

0xffffffff

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

77

Layer 3 Forwarding Iwo


Receive notification from
Ipktrd and data cells from Im
L23 ucode engine performs
IPv4 fragmentation and
redirect check
Byte and packet counts per
descriptor
Build L2/L3 header
Packet CRC check

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

78

Layer 3 Forwarding Iwo


16MB RLDRAM allocated for L2_desc
ADPC2(atlas-re0 vty)# show ichip 2 wo configuration desrd
mit/mt desc base : 0x00800000
l2/tag desc base : 0x01000000
l2/tag QW mode
: false

The maximum size of the L2 descriptor data is 16 words


ADPC2(atlas-re0 vty)# show nhdb id 831 extensive
ID
Type
Interface
Next Hop Addr
Protocol
----- -------- ------------- --------------- ---------831
Unicast ge-2/2/0.100
IPv4->MPLS
.....
L2 Program:
Address: 0x00250030
L2 microcode: iwo_l2_common_encap_hdlr
Template size: 1 word
.....
Tag Descriptor (0x00000118):
.....
Tag Labels:
0x1bd901ff: Label 114064, TTL 255
L2 Descriptor (0x00000270):
Raw format: 0x583011a0
Accounting: Disabled
L2 words: 6

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

Encap
-----------Ethernet

MTU
---4512

www.juniper.net

79

Layer 3 Forwarding Iwo


Global Statistics
ADPC2(atlas-re0 vty)# show ichip 2 wo statistics
Iwo Statistics:
Summary

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------input packets

1174507

34

70

output packets

615431

17

44

output bytes

58442506

1716

16617

total drops

559076

16

25

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

80

Layer 3 Forwarding Iwo


Similar to Lchip
Notification and first
one/two cells go thru the
DESRD and HDRF blocks
Data goes to the Data
Buffer block

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

81

Layer 3 Forwarding Iwo


Per functional block statistics
ADPC2(atlas-re0 vty)# show ichip 2 wo statistics ip
Iwo Input Processor Statistics:
Summary
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------input packets
1174899
34
70
output packets
1174899
34
70
total drops
0
0
0
ADPC2(atlas-re0 vty)# show ichip 2 wo statistics desrd
Iwo Descriptor Reader Statistics:
Summary
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------input packets
1174641
37
70
output packets
1174641
37
70
total drops
0
0
0
ADPC2(atlas-re0 vty)# show ichip 2 wo statistics hdrf
Iwo Header Fifo Statistics:
Summary
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------output packets
615567
18
44
output bytes
58455523
1746
16617
total drops
559203
17
25
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

82

Layer 3 Forwarding Iwo


Per Wan stream statistics
ADPC2(atlas-re0 vty)# show ichip 2 wo statistics wan_stream 1
Iwo Statistics:
Summary

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------global drops

565214

17

25

input packets

622188

19

43

output packets

56974

17

output bytes

6445344

206

9884

total drops

Stream(1):

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

83

Layer 3 Forwarding Iwo


Per Wan stream statistics
ADPC2(atlas_re0 vty)# show ichip 2 wo statistics ip wan_stream 0
Iwo Input Processor Statistics:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------Stream(0):
input packets
0
0
0
output packets
0
0
0
ssmcst packets
0
0
0
fragmented packets
0
0
0
input drops
0
0
0
output drops
0
0
0
> 2 cell crc drops
0
0
0
<= 2 cell crc drops
0
0
0
ADPC2(atlas_re0 vty)#

if (log_errors && is_crcerr) {


syslog(LOG_ERR, "ICHIP(%u): New crc errors in WO IP stream_id %u, "
"%s\n", ichip->which_ichip, stream_id, __FUNCTION__);

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

84

Layer 3 Forwarding Iwo


SPI interface statistics
ADPC9(atlas_re0 vty)# show ichip 0 spi4 status
I-Chip Spi4 Link Status:
Sink(Rx):
status

: up

pll lock

: locked

training state : trained


dip4 status

: passing

total errors

: 0

Source(Tx):
status

: up

training state : trained


dip2 status

: passing

total errors

: 0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

85

Layer 3 Forwarding Iwo


SPI interface statistics
ADPC9(atlas_re0 vty)# show ichip 0 spi4 errors
I-Chip Spi4 Polling Period with Errors:
Sink(Rx):
dip4 errors
: 0
loss of sync
: 0
sop min drop
: 0
filter eop drop
: 0
filter dip drop
: 0
filter mtu drop
: 0
filter ovr drop
: 0
unpkr fifo ovr
: 0
unpkr non mod16
: 0
unpkr fifo limit
: 0
bad ctx
: 0
payload ctl
: 0
eop ctl
: 0
filter sop drop
: 0
....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

86

Layer 3 Forwarding Iwo


SPI interface statistics
(Cont.....)

Source(Tx):
dip2 errors
framing errors
unexpected 11 errors
loss of sync
client fifo ovr
pkr fifo ovr
fifo underun

:
:
:
:
:
:
:

0
0
0
0
0
0
0

ADPC9(atlas_re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

87

Layer 3 Forwarding Iwo


L2 Descriptor can be shared
Reduce the L2 descriptor usage
Per TAG descriptor statistics can be maintained

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

88

Layer 3 Forwarding Iwo


TAG descriptor with statistics enabled
ADPC2(atlas-re0 vty)# show nhdb id 831 extensive
ID
Type
Interface
Next Hop Addr
Protocol
----- -------- ------------- --------------- ---------831
Unicast ge-2/2/0.100
IPv4->MPLS
.....
Tag Descriptor (0x00000118):
Raw format: 0x8cc00138 (No swap, TTL copy)
Accounting: Enabled
Packet count
: 0
HW Byte count
: 0
Adjusted Byte count: 0
Number of tags: 1
L2 des ptr: 0x00400270
Tag Labels:
0x1bd901ff: Label 114064, TTL 255
.....
L2 des ptr: 0x00400270
Tag Labels:
0x1bd901ff: Label 114064, TTL 255
L2 Descriptor (0x00000270):
Raw format: 0x583011a0
Accounting: Disabled
.....
NOTE: This is shared L2 descriptor (ref_count: 4)
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

Encap
-----------Ethernet

MTU
---4512

www.juniper.net

89

Layer 3 Forwarding Iwo


ADPC2(atlas-re0 vty)# show nhdb id 835 extensive
ID
Type
Interface
Next Hop Addr
Protocol
----- -------- ------------- --------------- ---------835
Unicast ge-2/2/0.100
IPv4->MPLS
.....
Tag Descriptor (0x00000148):
Raw format: 0x8cc00138 (No swap, TTL copy)
Accounting: Enabled
.....
Packet count
: 0
HW Byte count
: 0
Adjusted Byte count: 0
Number of tags: 1
L2 des ptr: 0x00400270
Tag Labels:
0x1bda01ff: Label 114080, TTL 255
L2 Descriptor (0x00000270):
Raw format: 0x583011a0
Accounting: Disabled
.....
NOTE: This is shared L2 descriptor (ref_count: 4)

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

Encap
-----------Ethernet

MTU
---4512

www.juniper.net

90

Layer 3 Forwarding Iwo


ADPC2(atlas-re0 vty)# show topology route ip ifl 91
.....
Topology: ge-2/2/0.100
Flavor: oh-iff (5), Refcount 5, Flags 0x1
Addr: 0x4aea9248, Next: 0x4099ecd8, Context 0x4a7c539c
Link 0: 00000000:00000000, Offset 16, Next: 00000000:00000000
Link 1: 00000000:00000000, Offset 16, Next: 00000000:00000000
Link 2: 00000000:00000000, Offset 16, Next: 00000000:00000000
Link 3: 00000000:00000000, Offset 16, Next: 00000000:00000000
Topology Neighbors:
nh(Unicast,835)-> ge-2/2/0.100-> o-root(IPv4)
nh(Unicast,831)-+
nh(Unicast,799)-+
nh(Unicast,795)-+
nh(Unicast,745)-+
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

91

Layer 3 Forwarding Iwo


ADPC2(atlas-re0 vty)# show l2rw share_l2_des
Chip 0:
Total shared L2 des count:0
Chip 1:
[0] L2 des: chip:1, L2 des addr:0x400270, chunk:4, type:shared,
ref_count:4
Key: L2 des info:0x583011a0, ifl:91, tunnel_id:805306372
L2 des data: 2.640002a.906920.3bf08100.648847.c6.0.0.0.0.0.0.0.
Tag des data: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.
NH prefix: a.1.1.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.
Linked list (Tag des addr): 0x40016c, 0x400166, 0x400118, 0x400148,
List count: actual:4, traversed items:4

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

92

Layer 3 Forwarding NPEZ


According to the L2_desc, the corresponding service number
would be included on the TX cookie to NPEZ
Egress function can be 802.1p rewrite, vlan-tag rewrite..etc
ADPC9(Atlas_re0 vty)# show nhdb id 542 extensive
ID

Type

Interface

Next Hop Addr

Protocol

Encap

MTU

-----

--------

-------------

---------------

----------

------------

----

542

Unicast

xe-9/2/0.53

216.246.102.102

IPv4

Ethernet

9194

.....
L2 Descriptor (0x000001d4):
Raw format: 0x5a2023ea
Accounting: Enabled
Packet count

: 0

HW Byte count

: 0

Adjusted Byte count: 0


L2 words: 6
Template ID: 32
L2 MTU: 9194
Rewrite word 1: 0x00000005
.....
Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

93

Layer 3 Forwarding NPEZ


ADPC9(Atlas_re0 vty)# show npez 2 servid-tbl serv-id 0x5 port 0
--- Service id table entry --Key Parameters
service id
5
port num
0
Result Parameters
dot1p table index
0
stp index
0
native inner vlan
0
native inner tpid
0
native inner vlan present
0
native outer vlan present
0
valid vlan range min
53
valid vlan range max
53
inner vlan range check
0
outer vlan range check
0
iif type
2
policing enable
0
tm profile id
0
dmac lookup enable
0
send to host
0
stats index
0x7ffe4000
token bucket index
0x00000000
port mac index
0
First TM scheduler number
0
Second TM scheduler number
0
QN bits table
0x00000000

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

94

Layer 3 Forwarding Ifo


Include Stream Flow
Engine(SFE), Fabric Transmit
Logic(FTX) and HST macros
Stream Flow Engine (SFE)
handles most of the Ifo
functions
FTX generates the HSL2 cells
HST manages the HST
interfaces

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

95

Layer 3 Forwarding Ifo

SFE sends the cell buffer reservation grant to Ipktrd, Ipktrd sends
the cell read request to Im
Data cells would come from the Im

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

96

Layer 3 Forwarding Ifo


SFE does the cell reordering.
Insert the notification data (12B) back to the first data
cell.
Generate 2 bytes CRC if it doesnt generate an extra cell

Packet CRC check before sending the cells over the


fabric planes. Flag the LC_E if the check is failed.

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

97

Layer 3 Forwarding Ifo


Per fabric stream statistics

Two queues per FAB destination ([0,1] [High , Low])


ADPC2(atlas-re0 vty)# show ichip 2 f statistics fab_stream 0
Traffic stats:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------fi_burst_cnt_rcv[00]
0
0
0
fi_rord_cnt_shift[00]
0
fi_rord_timeout[00]
0
fi_rord_cnt_apdone[00]
0
fi_rord_cnt_rstall[00]
0
fo_good_pkts[00]
0
0
0
fo_cells_tx[00]
0
0
0
fo_err_pkts[00]
0
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

98

Layer 3 Forwarding Ifi


Ifi (Fabric Input)
Receiving side for the Ifo
This block receives the cells
from the fabric, put them into
correct order and send them
to Iwi for further processing
It also performs packet CRC
check

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

99

Layer 3 Forwarding Ifi


Ifi block will flag the last cell as LC_E for CRC error
packet and indicate that this packet is bad
ADPC2(Atlas.RE1 vty)# show ichip 0 registers fi pkt_chksm
FI_PKT_CHKSM Registers:
(0xc0202010)
fi.pkt_chksm.dis: 0x00000000
ADPC2(Atlas.RE1 vty)#

Packet would be dropped in Ipktwr block


ADPC2(Atlas.RE1 vty)# show ichip 0 ipktwr statistics
Pktwr counters:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------pkt_drop
0
0
0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

100

Layer 3 Forwarding Ifi


CRC check helps finding the faulty module

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

101

Layer 3 Forwarding Ifi


Per stream statistics (Fabric)
ADPC2(atlas-re0 vty)# show ichip 1 f statistics fab_stream 16
Traffic stats:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------fi_burst_cnt_rcv[16]
40570
1
125
fi_rord_cnt_shift[16]
0
fi_rord_timeout[16]
0
fi_rord_cnt_apdone[16]
0
fi_rord_cnt_rstall[16]
0
fo_good_pkts[16]
0
0
0
fo_cells_tx[16]
0
0
0
fo_err_pkts[16]
0
ADPC2(atlas-re0 vty)# show ichip 1 wi finq statistics 16
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------FINQ Counter
19371
0
26
(BYTE)
2014643
82
7920

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

102

Layer 3 Forwarding Ifi


Statistics summary (Fabric)
ADPC2(re0.xr1.ord1 vty)# show ichip 0 f statistics
Discard counters:
Counter Name
Total
Rate
Peak Rate
---------------------- ---------------- -------------- -------------FI_BURST_CNT_FC_DROP
0
0
0
FI_BURST_CNT_NFC_DROP
0
0
0
FI_RORD_CNT_FC_DROP
0
0
0
FI_RORD_CNT_NFC_DROP
0
0
0
ADPC2(re0.xr1.ord1 vty)#

Fabric Drop
cell_stats->fabric_drops
cell_stats->fabric_drops
cell_stats->fabric_drops
cell_stats->fabric_drops

Copyright 2007 Juniper Networks, Inc.

+=
+=
+=
+=

f_stats->fi_cell_fc_drop.total;
f_stats->fi_cell_nfc_drop.total;
f_stats->fi_rord_fc_drop.total;
f_stats->fi_rord_nfc_drop.total;

Proprietary and Confidential

www.juniper.net

103

Layer 3 Forwarding Statistics Collection


Useful commands to collect the info for a specific
wan/fabric stream (From Iwi to Iwo)
# show ichip 0 statistics
# show ichip 0 statistics fab_stream 0
# show ichip 0 statistics wan_stream 0

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

104

Layer 3 Forwarding Statistics Collection


ADPC2(atlas_re0 vty)# show ichip 0 statistics wan_stream 0
Counters for stream 32 of ichip 0
Iwi:
----------Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------WINQ Counter

(BYTE)

Counter Name

Total

Rate

Peak Rate

Iiif BLOCK:
----------Traffic stats:
---------------------- ---------------- -------------- -------------wan_strm[00] total

(BYTE)

wan_strm[00] default

(BYTE)

Counter Name

Total

Rate

Peak Rate

Error counters:
---------------------- ---------------- -------------- -------------wan_strm[00] uaiif
wan_strm[00] tmo

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

105

Layer 3 Forwarding Statistics Collection


(cont...)
Imq Block:
--------Stream 32 IN_OFFLOAD_PKT_BYTE_CNT:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------queue 0

(BYTE)

queue 1

(BYTE)

queue 2

(BYTE)

queue 3

(BYTE)

queue 4

(BYTE)

queue 5

(BYTE)

......

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

106

Layer 3 Forwarding Statistics Collection


(cont...)
queue 6

(BYTE)

queue 7

(BYTE)

Total

Rate

Peak Rate

stream 32 OUT_PKT_Q:
Counter Name

---------------------- ---------------- -------------- --------------

Copyright 2007 Juniper Networks, Inc.

queue 0

(BYTE)

queue 1

(BYTE)

queue 2

(BYTE)

queue 3

(BYTE)

queue 4

(BYTE)

queue 5

(BYTE)

Proprietary and Confidential

www.juniper.net

107

Layer 3 Forwarding Statistics Collection


(cont...)
queue 6

(BYTE)

queue 7

(BYTE)

Total

Rate

Peak Rate

Ipkt BLOCK:
--------Cell-error Counters:
Counter Name

---------------------- ---------------- -------------- -------------wan_strm[00] ecc

wan_strm[00] err

wan_strm[00] nt_age

wan_strm[00] ic_age

wan_strm[00] dc_age

.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

108

Layer 3 Forwarding Statistics Collection


(cont...)
General error Counters:
Counter Name

Total

---------------------- ---------------wan_strm[00] byc_oe

wan_strm[00] byc_ue

wan_strm[00] prq_oe

wan_strm[00] crq_oe

wan_strm[00] icb_oe

Iwo:
----------Iwo Statistics:
Summary

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------global drops

input packets

output packets

output bytes

total drops

Stream(0):

ADPC2(atlas_re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

109

Tunnel Interface
No Tunnel PIC for Atlas platform
Tunneling interface might be required (vt- / gr- / ip- /
mt- / lt- / pe- / pd- ...etc)
Convert / create a tunnel interface on the PFE but
looping the packet back from the NPEZ
Two types of tunnel interface - 1Gbps or 10Gbps

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

110

Tunnel Interface
Two types of tunneling interface can be created
(1Gbps vs 10Gbps)
On 40-GE DPC

Considered as port 10 on the PFE.


All 10 ports GE + 1 tunnel interface share the bandwidth
between NPEZ and Ichip
Maximum bandwidth is 1Gbps

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

111

Tunnel Interface
1Gbps Tunnel Interface
[edit]
lab@atlas-re0# show chassis
fpc 2 {
pic 3 {
tunnel-services {
bandwidth 1g;
}
}
}
lab@atlas-re0> show interfaces terse | match 2/3/10
gr-2/3/10
up
up
ip-2/3/10
up
up
lt-2/3/10
up
up
mt-2/3/10
up
up
pd-2/3/10
up
up
pe-2/3/10
up
up
vt-2/3/10
up
up

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

112

Tunnel Interface
On 4-XGE DPC

The XGE port would be changed to tunnel interface


It occupies whole 10Gbps bandwidth between NPEZ and
Ichip
No more XGE port can be used on that PFE

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

113

Tunnel Interface
10Gbps Tunnel Interface
[edit]
lab@atlas-re0# show chassis
fpc 9 {
pic 1 {
tunnel-services {
bandwidth 10g;
}
}
}
lab@atlas-re0> show interfaces terse | match 9/1/0
gr-9/1/0
up
up
ip-9/1/0
up
up
lc-9/1/0
up
up
lc-9/1/0.32769
up
up
vpls
lt-9/1/0
up
up
mt-9/1/0
up
up
pd-9/1/0
up
up
pe-9/1/0
up
up
vt-9/1/0
up
up

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

114

Exception Traffic
Traffic to / from the PFE interface and need attention
from the PFE PPC / RE
Routing update, ping, telnet, ssh...etc
Traceroute, ICMP frag, ICMP redirect....etc
Notification path: Irlkp <-> Ip <-> A-IA FPGA <-> PFE
PPC <-> RE

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

115

Exception Traffic Ip
Handle Host traffic
Provide 2Gbps bandwidth
between host CPU and Ichip
via A-IA FPGA
Handle interrupts from
different ASICs

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

116

Exception Traffic Ip
Notifications are received from the Irlkp block
ADPC2(atlas-re0 vty)# show ichip 2 p counters
Traffic stats:
Counter Name

Total

Rate

Peak Rate

---------------------- ---------------- -------------- -------------Read Req RX

538120

15

39

Read Resp TX

538120

15

39

Read Resp Segment TX

662584

19

245

Write Req RX

569899

15

27

Write Req Segment RX

1138248

31

115

Write Resp TX

569899

15

27

Write Resp Segment TX

MTU Redirect TX

Age Check TX

Flow Control RX

Flow Control TX

PIO Read RX

176272811

5388

5727

PIO Read TX

176272811

5388

5727

....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

117

Exception Traffic Ip
(cont....)
Notification Q[0] TX

30504

Notification Q[1] TX

2726

Notification Q[2] TX

102022

16

Notification Q[3] TX

402868

11

14

Notification Q[4] TX

Notification Q[5] TX

Notification Q[6] TX

Notification Q[7] TX

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

118

Exception Traffic Ip
Flow control is being used for the Host packet between Ip and AIA FPGA
FC status of the transmitter on the IA bus (1 means stop sending /
receiving)
ADPC2(atlas-re0 vty)# show ichip 2 registers p
P Registers:
(0xd0c00000)
(0xd0c00008)
(0xd0c00014)
(0xd0c00018)
(0xd0c0001c)
(0xd0c00020)
(0xd0c0002c)
(0xd0c00030)
(0xd0c00034)
Stop TX notif to A-IA when set
(0xd0c00038)
Stop pkt read/write from A-IA
(0xd0c0003c)

Copyright 2007 Juniper Networks, Inc.

p.rev:
p.ctl_to:
p.send_ptype_support:
p.recv_ptype_support:
p.err_check:
p.scratch:
p.err_check_en:
p.link_status:
p.train:

0x01000004
0x00ffffff
0x33f00000
0xc4400000
0xa9400000
0x12345678
0x09400000
0x81000000
0x0f0f0000

p.flow_cont_tx: 0x00000000
p.flow_cont_rx: 0x00000000

Proprietary and Confidential

www.juniper.net

119

Exception Traffic Ip

p.flow_cont_tx: 0x00000000
ia_bus_fc_mtured 12 1'h0 Stop sending MTU/Redirect packets
ia_bus_fc_read_resp 8 1'h0 Stop sending read response packets
ia_bus_fc_notif_q7 7 1'h0 Stop sending notification for queue 7
ia_bus_fc_notif_q6 6 1'h0 Stop sending notification for queue 6
ia_bus_fc_notif_q5 5 1'h0 Stop sending notification for queue 5
ia_bus_fc_notif_q4 4 1'h0 Stop sending notification for queue 4
ia_bus_fc_notif_q3 3 1'h0 Stop sending notification for queue 3
ia_bus_fc_notif_q2 2 1'h0 Stop sending notification for queue 2
ia_bus_fc_notif_q1 1 1'h0 Stop sending notification for queue 1
ia_bus_fc_notif_q0 0 1'h0 Stop sending notification for queue 0

p.flow_cont_rx: 0x00000000
ia_bus_fc_read 10 1'h0 Stop receiving read packets
ia_bus_fc_write 9 1'h0 Stop receiving write packets

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

120

Exception Traffic Ip
When the lookup result is with 1<=SID<=7, HNQ
classification is based on the NH_Type
* catg

queue

type

*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*

0
1
2
2
3
3
0
1
0
4
4
5
7
7
7
7
7
7
7
7

Bypass host notification high


Bypass host notification low
STP high priorirty (send to processor)
STP low priority (send to processor)
IIF mismatch diag notification
Non iif mismatch diag notifcation
Key Lookup host, high priority
Key Lookup host, low priority
HLUP, diagnostics only
Options, high priority
Options, low priority
TTL Exceeded
Sample class 7
Sample class 6
Sample class 5
Sample class 4
Sample class 3
Sample class 2
Sample class 1
Sample class 0

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

121

Exception Traffic Ip
When lookup result is with 8<=SID<=15, HNQ classification
is done in SID-to-Queue Table
All goes to HNQ6 now. For example, RNH_REJECT will have SID
8 which goes to HNQ6.
ADPC2(atlas-re0 vty)# show ichip 2 registers r rcp_smp
R rcp smp Registers:
.....
(0xd0570850)
r.rcp_smp.hnq_map.sid_to_hnq: 0x66666666

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

122

Exception Traffic Ip
Aggregate the interrupts and notify the Host
ADPC2(atlas-re0 vty)# show ichip 0 p internal
(

Address)

vector

count

(0x40b960e8)

isr[ICHIP_P] : 0x40211d00

(0x40b960ec)

isr[ICHIP_SR_R] : 0x00000000

(0x40b960f0)

isr[ICHIP_SR_IIF] : 0x00000000

(0x40b960f4)

isr[ICHIP_SR_WO] : 0x00000000

(0x40b960f8)

isr[ICHIP_M] : 0x401e52f8

(0x40b960fc)

isr[ICHIP_MQ] : 0x00000000

(0x40b96100)

isr[ICHIP_BD] : 0x401aae10

(0x40b96104)

isr[ICHIP_WO] : 0x401ada38

(0x40b96108)

isr[ICHIP_IF] : 0x00000000

(0x40b9610c)

isr[ICHIP_FI] : 0x00000000

(0x40b96110)

isr[ICHIP_FO] : 0x00000000

(0x40b96114)

isr[ICHIP_R] : 0x40211bc0

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

123

Exception Traffic A-IA FPGA

Atlas Ichip Assistant ASIC (A-IA)


Communication between ASIC and PFE PPC
Statistics collection (nexthop / fw / intf)
Host bound traffic
Interrupt handling

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

124

Exception Traffic A-IA FPGA


PCI bus to communicate with PPC
IA Bus to communicate with Ichip
One A-IA ASIC serves 2 Ichips
ADPC2(atlas-re0 vty)# show ia map
ia-index

ij-index

pfe-index

--------

--------

---------

---------------------------------

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

125

Exception Traffic A-IA FPGA


Packet to Host

A-IA examines the heads of the notification queues


WRR serves all the queues equally
Packet is extracted from Ipktrd block via Ip
A-IA moves the packet to the free buffer descriptor ring buffer
(FDRB) and mark the buffer is used
Raise polling request for CPU to read the packet

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

126

Exception Traffic A-IA FPGA


Packet from Host

CPU writes the packet into the Write Packet Descriptor Ring
Buffer (WPDRB)
Change the ownership of the buffer to CPU -> FPGA
A-IA polls the top descriptor and read the packet out
Packet is broken down into 64 bytes cells and transmit over
to the Ip block, then to Ipktwr block
Notification are transmitted with the first data cell

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

127

Exception Traffic A-IA FPGA


Flow control status between A-IA and Ip
ADPC2(atlas-re0 vty)# show ia 0 registers rxif 0
rxif IA[0][0]
(0xc3000800)
training_csr
(0xc3000804)
training_wait_thresh
(0xc3000808)
training_seq_thresh
(0xc300080c)
ia_bus_in_data_0
(0xc3000810)
ia_bus_in_data_1
FC received from Ip. Stop packet write to Ip
(0xc3000814)
fc_stat

:
:
:
:
:

ADPC2(atlas-re0 vty)# show ia 0 registers txif 0


txif IA[0][0]
(0xc3000a00)
pio_ctl
(0xc3000a04)
pio_data
(0xc3000a08)
pio_timeout_val
(0xc3000a0c)
pio_page
FC sent to Ip. Stop incoming notif from Ip
(0xc3000a10)
fc_stat

:
:
:
:

Copyright 2007 Juniper Networks, Inc.

0x00000004
0x80008000
0x00000810
0x00005c00
0x00005c00

: 0x00000000

0x00000000
0x00000000
0x000001ff
0x00000020

: 0x00000000

Proprietary and Confidential

www.juniper.net

128

Exception Traffic A-IA FPGA


Requests to IA FPGA would be under throttling protection
ADPC2(atlas-re0 vty)# show throttle
Throttle Statistics:
ID

Epoch(ms)

E/D(ms)

State

Name

--

----------

-----------

--------

----

12

4/8

Enabled

IA 0 Notif Q 0

12

4/8

Enabled

IA 0 Notif Q 1

12

4/8

Enabled

IA 0 Notif Q 2

12

4/8

Enabled

IA 0 Notif Q 3

12

4/8

Enabled

IA 0 Notif Q 4

12

4/8

Enabled

IA 0 Notif Q 5

12

4/8

Enabled

IA 0 Notif Q 6

12

4/8

Enabled

IA 0 Notif Q 7

12

4/8

Enabled

IA 0 Notif Q 8

12

4/8

Enabled

IA 1 Notif Q 0

10

12

4/8

Enabled

IA 1 Notif Q 1

11

12

4/8

Enabled

IA 1 Notif Q 2

.....

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

129

Exception Traffic A-IA FPGA


Statistics collection
ADPC2(atlas-re0 vty)# show ia 0 statistics applications
IA Stats Appl Info:
Num

Stats Appl

---

----------

nh

iif

dfw

ADPC2(atlas-re0 vty)# show nhdb id 831 extensive


ID

Type

Interface

Next Hop Addr

Protocol

Encap

MTU

-----

--------

-------------

---------------

----------

------------

----

831

Unicast

ge-2/2/0.100

IPv4->MPLS

Ethernet

4512

.....
IA Counter Address: 0x4a8ceb70

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

130

Exception Traffic A-IA FPGA


NH statistics collection
ADPC2(atlas-re0 vty)# show ia 0 statistics counter 0x4a8ceb70
IA Stats Acct Entry Info:
Index

Packets

Bytes

Diff Packets

Diff Bytes

-----

--------------

------------------

--------------

------------------

19

Error Count: 0
Counter Type: acct_entry_with_opt_counter
Sram Address: 0x09000468
ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

131

Exception Traffic A-IA FPGA


IIF Statistics collection
ADPC2(atlas-re0 vty)# show ia 0 statistics acctentry 1 0
IA Stats Acct Entry Info:
Index Packets
Bytes
Diff Packets
----- -------------- ------------------ -------------0
0
0
0
1
0
0
0
2
0
0
0
3
0
0
0
4
0
0
0
5
0
0
0
6
0
0
0
7
0
0
0
8
0
0
0
9
0
0
0
10
0
0
0
11
0
0
0
12
0
0
0
13
0
0
0
14
6
654
0
15
939288
58929129
31
16
5
500
0
17
91913
8086664
6

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

Diff Bytes
-----------------0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1929
0
568

www.juniper.net

132

Exception Traffic A-IA FPGA


Interrupt management
ADPC2(atlas-re0 vty)# show fpc interrupts
V En Lv Src
Count Handler
Arg
-- -- -- ---- ---------- -------- -------0 Y 6 I/00
0 40037200 00000000
2 Y 7 E/10
11468141 40036e78 00000000
3 Y 6 I/03
0 40037cc4 00000000
4 Y 5 I/26
58259 4002d80c 00000000
5 Y 2 I/13
1550502 40161ae0 40a9e8a4
6 Y 2 I/14
1678903 40161b0c 40a9e8a4
7 Y 2 I/18
0 40161b38 40a9e8a4
11 Y 2 I/15
56842 40161ae0 40aa0958
12 Y 2 I/16
170211 40161b0c 40aa0958
13 Y 2 I/17
0 40161b38 40aa0958
19 Y 1 E/03
400341 404fb020 00000000
20 Y 1 E/04
1935277 404fb020 00000001
21 Y 1 E/05
1 404fb04c 00000000
22 Y 1 E/06
5 404fb04c 00000001
23 Y 1 E/07
159 404fb04c 00000002
24 Y 1 E/08
1 404fb04c 00000003

Copyright 2007 Juniper Networks, Inc.

Stack
-------409123dc
409a4640
409a8ea0
409a6e80
40aa4610
40aa4610
40aa4610
40aa4610
40aa4610
40aa4610
40b55780
40b55780
40b55780
40b55780
40b55780
40b55780

Proprietary and Confidential

Flags
----E0
L0
L1C
L1
L1
L1
L1
L1
L1
L1
L0
L0
L0
L0
L0
L0

Name
----------Unsupported
Clock Tick
Errors
Console
TSEC0/TX
TSEC0/RX
TSEC0/ERR
TSEC1/TX
TSEC1/RX
TSEC1/ERR
ia0
ia1
pfe0
pfe1
pfe2
pfe3

www.juniper.net

133

Exception Traffic A-IA FPGA

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

134

Exception Traffic A-IA FPGA


PIO path for the PPC to ASIC
ADPC2(atlas-re0 vty)# show ia 0 pio status
IA_0: IJ 0 pio timeout

FALSE

IA_0: IJ 0 pio in progress FALSE


IA_0: IJ 0 pio indirect

FALSE

IA_0: IJ 1 pio timeout

FALSE

IA_0: IJ 1 pio in progress FALSE


IA_0: IJ 1 pio indirect

FALSE

ADPC2(atlas-re0 vty)#

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

135

Thank You !!!

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

136

Copyright 2007 Juniper Networks, Inc.

Proprietary and Confidential

www.juniper.net

137

You might also like