Fixing JUNOS next-hop resolution failures


I’ve recently seen a number of instances where a Juniper router will simply not forward packets correctly, often onto a “LAN” (Ethernet with lots of hosts). Looking at the routing table seems to show that everything is fine, and pinging the host from the router works, but transit traffic appears to just disappear into a black hole. This can be really annoying and hard to debug.

These are my quick notes on troubleshooting and fixing this. Note that the examples has been edited slightly to make it easier to read.

Scenario / description of issue

Traffic coming in on xe-1/0/1.0 destined for and on xe-1/0/0.0 is being dropped. It isn’t a firewall problem, and packets are d efinitely arriving at the router (trust me on this).

Pinging from the router works just fine:

wkumari@r2> ping count 1
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=2.085 ms

--- ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.085/2.085/2.085/0.000 ms

and that is indeed the correct ARP entry (confirmed on the device being pinged).

wkumari@r2> show arp no-resolve hostname
MAC Address       Address         Interface         Flags
aa:bb:bb:dd:11:7c  xe-1/0/0.0               none


But, pinging from the “outside” doesn’t work:

 ping -c 2
PING ( 56(84) bytes of data.

--- ping statistics ---
2 packets transmitted, 0 received, +1 errors, 100% packet loss, time 1002ms

Looking at the routing table, everything is fine.

wkumari@r2> show route

inet.0: 813779 destinations, 1632343 routes (813775 active, 3 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both   *[Direct/0] 2d 12:34:56
                    > via xe-1/0/0.0
                    [Static/5] 2d 12:35:14
                    [BGP/170] 2d 12:33:03, MED 0, localpref 200, from
                      AS path: I, validation-state: unverified
                    > to via xe-1/0/0.0

and looking in the forwarding table everything also looks fine:

wkumari@r2> show route forwarding-table destination
Routing table: default.inet
Destination        Type RtRef Next hop           Type Index    NhRef Netif  dest     0 aa:bb:bb:dd:11:7c  ucst      723     1 xe-1/0/0.0



But, transit traffic still just disappears into a black hole…

Cause / troubleshooting

Under some set of cases, the router and forwarding table will both be correctly programmed, complete with the next-hop, but the PFE isn’t correctly programmed.


wkumari@r2# run show pfe route ip table index 0 prefix

Slot 2
================ fpc2 ================

IPv4 Route Table 0, default.0, 0x80000:
Destination                       NH IP Addr      Type     NH ID Interface
--------------------------------- --------------- -------- ----- ---------                                        Hold   631 RT-ifl 65543 xe-1/0/0.0 ifl 65543

The Next Hop IP Address in the PFE is stuck in Hold state. This means that, when the PFE is trying to forward the packet, it doesn’t have a next-hop address to send it, and it (for some reason) isn’t initializing a resolution. The address can be pinged from the router the router itself has the next-hop resolved correctly (see above) and doesn’t need to pass through the PFE.


This can be fixed by rebooting the router (erm…), or, more easily, by making the router redo the PFE configuration and resolution:

wkumari@r2# set route retain next-hop
wkumari@r2# commit
wkumari@r2# delete route
wkumari@r2# commit

and now, looking in the PFE again, we see that the Next Hop IP Address is now resolved, and everything works correctly again:

wkumari@r2# show pfe route ip table index 0 prefix

Slot 2
================ fpc2 ================

IPv4 Route Table 0, default.0, 0x80000:
Destination                       NH IP Addr      Type     NH ID Interface
--------------------------------- --------------- -------- ----- ---------             Unicast   631 RT-ifl 65543 xe-1/0/0.0 ifl 65543

and from the outside:

PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=44 time=56.667 ms
64 bytes from icmp_seq=1 ttl=44 time=56.503 ms
64 bytes from icmp_seq=2 ttl=44 time=56.648 ms
64 bytes from icmp_seq=3 ttl=44 time=62.501 ms
--- ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 56.503/58.080/62.501/2.553 ms