Reply
Highlighted
Member
Posts: 204
Registered: ‎06-18-2013
Kudos: 101
Solutions: 4

Kernel Panic/Crash

[ Edited ]

My EdgeRouter POE has been working great for months on end; last night it reset at least 3 times unexpectedly.. The watchdog timer brought everything back up within a few mins.

 

here is what I found while logged in:

 

Message from syslogd@edge-router at Mar 13 04:28:14 ...
 kernel:skbuff: skb_under_panic: text:ffffffff804a1f80 len:109 put:-4 head:800000041b51c000 data:800000031b51c146 tail:0x1b3 end:0xe80 dev:eth1
Mar 13 04:28:14 edge-router kernel: skbuff: skb_under_panic: text:ffffffff804a1f80 len:109 put:-4 head:800000041b51c000 data:800000031b51c146 tail:0x1b3 end:0xe80 dev:eth1
Mar 13 04:28:14 edge-router kernel: Kernel bug detected[#1]:
Write failed: Broken pipe

 

$ show version
Version:      v1.6.0
Build ID:     4716006
Build on:     10/31/14 17:31
Copyright:    2012-2014 Ubiquiti Networks, Inc.
HW model:     EdgeRouter PoE 5-Port
HW S/N:       24A43C05B50D
Uptime:       09:42:01 up  5:13,  1 user,  load average: 0.03, 0.02, 0.05

 

I havent made any changes, dont know why its suddenly crashing.

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3092
Solutions: 945
Contributions: 16

Re: Kernel Panic/Crash

Can you post your config file?

EdgeMAX Router Software Development
Member
Posts: 204
Registered: ‎06-18-2013
Kudos: 101
Solutions: 4

Re: Kernel Panic/Crash

[ Edited ]

Its too big to put in the post (Message cannot exceed 20,000 characters.)

 

http://pastebin.com/r5b9yWxt 

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3092
Solutions: 945
Contributions: 16

Re: Kernel Panic/Crash

I have seen a similar crash that I'm working on trying to reproduce.  I suspect it's related to vlan offload and I see that you are using vlans with vlan offload.  Could you try disabling vlan offload if it happens again?

EdgeMAX Router Software Development
Member
Posts: 204
Registered: ‎06-18-2013
Kudos: 101
Solutions: 4

Re: Kernel Panic/Crash

[ Edited ]

yeah if it starts reoccouring I will try disabling vlan offload; but that wouldent work as a perminent solution.. as you can see I have alot of vlans and turnning it off greatly reduces throughput. 

 

Its odd that it was happening over and over lastnight, once it was not even up for 5mins before it went down again.. loads would have been absolutely minimal during this time, at least compared to now when its been up all morning with people actualy doing stuff on the network.

 

All the log entries up to the kernel panic were IPv6 Traffic, and this would have been the most recent thing I done to the router... was enable IPv6 on my Comcast Business, shortly after 1.6.0 was released.

 

Spoiler

Mar 13 04:27:15 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:05c0:1400:000b:0000:0000:0000:360b DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=49 FLOWLBL=0 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:16 edge-router radvd[4404]: our AdvOtherConfigFlag on eth1 doesn't agree with fe80::eafc:afff:fee8:ec13

Mar 13 04:27:19 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2002:d56f:b295:000b:e9a8:dc89:96a3:0546 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=186 TC=32 HOPLIMIT=107 FLOWLBL=0 PROTO=UDP SPT=49001 DPT=50667 LEN=146

Mar 13 04:27:25 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:08a0:fe09:9901:ad68:2e87:0f82:1b1f DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=42 FLOWLBL=915042 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:31 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:41d0:0008:9263:0000:0000:0000:0001 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=50 FLOWLBL=0 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:35 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:05c0:1400:000b:0000:0000:0000:360b DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=49 FLOWLBL=0 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:38 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:0318:2015:0023:c81d:4374:e3ba:a619 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=46 FLOWLBL=878871 PROTO=UDP SPT=62584 DPT=50667 LEN=105

Mar 13 04:27:39 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:41d0:000a:216d:0000:0000:0000:0001 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=50 FLOWLBL=0 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:39 edge-router radvd[4404]: our AdvOtherConfigFlag on eth1 doesn't agree with fe80::eafc:afff:fee8:ec13

Mar 13 04:27:42 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:08a0:fe09:9901:ad68:2e87:0f82:1b1f DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=42 FLOWLBL=915042 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:47 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:41d0:0008:9263:0000:0000:0000:0001 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=50 FLOWLBL=0 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:51 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2a01:4200:deda:059c:ac96:1fc0:ea1a:2b5a DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=186 TC=32 HOPLIMIT=42 FLOWLBL=0 PROTO=UDP SPT=49001 DPT=50667 LEN=146

Mar 13 04:27:54 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:05c0:1400:000b:0000:0000:0000:360b DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=49 FLOWLBL=0 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:58 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:41d0:000a:216d:0000:0000:0000:0001 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=50 FLOWLBL=0 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:27:58 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:0318:2015:0023:c81d:4374:e3ba:a619 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=46 FLOWLBL=878871 PROTO=UDP SPT=62584 DPT=50667 LEN=105

Mar 13 04:27:59 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2001:08a0:fe09:9901:ad68:2e87:0f82:1b1f DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=42 FLOWLBL=915042 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:28:00 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2a02:8431:005b:a700:2ac6:8eff:fe35:26be DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=47 FLOWLBL=0 PROTO=UDP SPT=32768 DPT=50667 LEN=105

Mar 13 04:28:00 edge-router radvd[4404]: our AdvOtherConfigFlag on eth1 doesn't agree with fe80::eafc:afff:fee8:ec13

Mar 13 04:28:02 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2a02:8420:498e:7000:45e4:6992:abc4:aa3f DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=47 FLOWLBL=393217 PROTO=UDP SPT=51413 DPT=50667 LEN=105

Mar 13 04:28:05 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2a02:0238:f014:0812:0000:0000:0000:0001 DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=145 TC=32 HOPLIMIT=46 FLOWLBL=0 PROTO=UDP SPT=59451 DPT=50667 LEN=105

Mar 13 04:28:10 edge-router kernel: [WANv6_LOCAL-default-D]IN=eth0 OUT= MAC=33:33:00:00:00:fb:24:a4:3c:05:b5:0e:86:dd src=2601:0001:8d01:5ef0:26a4:3cff:fe05:b50e DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=83 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=5353 DPT=5353 LEN=43

Mar 13 04:28:10 edge-router kernel: [WANv6_LOCAL-default-D]IN=eth0 OUT= MAC= src=2601:0001:8d01:5e00:0000:0000:0000:0001 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=83 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=5353 DPT=5353 LEN=43

Mar 13 04:28:11 edge-router kernel: [WANv6_IN-default-D]IN=eth0 OUT=eth1.419 MAC=24:a4:3c:05:b5:0d:e8:fc:af:e8:ec:13:86:dd src=2a01:4200:deda:059c:ac96:1fc0:ea1a:2b5a DST=2601:0001:8d01:5ef2:b1b9:87b6:1d0d:0da6 LEN=186 TC=32 HOPLIMIT=42 FLOWLBL=0 PROTO=UDP SPT=49001 DPT=50667 LEN=146

 

Previous Employee
Posts: 10,504
Registered: ‎06-09-2011
Kudos: 3092
Solutions: 945
Contributions: 16

Re: Kernel Panic/Crash

Yes, disabling offload certainly isn't the solution.  That router that I have seen this occur on is now running a debug kernel, but it hasn't crashed again yet.  Oddly enough the last time it did occur it happened 4 times in the middle of the night when there should be very little traffic going on.  It seems to be some kind of traffic that triggers it.

EdgeMAX Router Software Development
Member
Posts: 204
Registered: ‎06-18-2013
Kudos: 101
Solutions: 4

Re: Kernel Panic/Crash

[ Edited ]

I checked some logs and it too seems that mine went down a total of 4 times last night.. so thats common.

 

Maby it was some external packet? I cant imagine at this time there was anything the least bit abnormal going on; only packets it normally sees all night long.. most all vlan traffic across subnets would have been video streams from security cameras at this time; phones, tablets, and workstations were not in use.

 

Pretty much all my routable ip's and port-forwards talk to services connected via eth1, there are absolutely no windows machines or mobile devices connected through this interface.

 

The last time I had random crashes (over a year ago) they seemed to go away when I enabled statefull packet inspection on my cable modem, I suspected it was the result of a spoof-attack, dns attack, or agressive scan.. I dont think thats still enabled, I got a new modem since then.

Reply