You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm trying to use netmap with my X520 and a capture software that uses multithreaded capture to get the data (And this software uses multithread). If i use the default values, i get this result:
The problem is that netmap is being asked to allocate an insane number of buffers, and so it fails.
Why are you setting ring_size to such a huge value?
What does "ethtool -g" say (e.g. the ring sizes of your physical card)?
Why are you setting admode to 1? It should be 0, since you don't have a native netmap adapter.
I want to create more than 18 pipes when using lb application. After 18 (for example lb -i p2p2 -p 20), I got this error:261.135269 nm_open [847] NIOCREGIF failed: Cannot allocate memory p2p2{18
261.135276 main [456] cannot open netmap:p2p2{18
It seems that there is some limitation. There is a macro NM_MAXPIPES 64, but I can't make more than 18. It seems here is not the limitation. My question is that if I want to create more than 18 pipes, eg 100 pipes, where should I change to make it work?
Dmesg shows the error:
[88943.459956] 112.380834 [ 840] netmap_obj_malloc no more netmap_ring objects
[88943.471745] ixgbe 0000:07:00.1 p2p2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[88943.481829] 112.402705 [1736] netmap_mem2_rings_create Cannot allocate TX_ring
[88943.852137] ixgbe 0000:07:00.1 p2p2: detected SFP+: 4
[88944.091797] ixgbe 0000:07:00.1 p2p2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Thanks
Hi all. Netmap preallocates rings and buffers from the kernel, then distributes them as required to NICs, VALE ports, pipes etc. You are running out of preallocated rings and/or buffers. In /sys/module/netmap/parameters there are a few variables that you can change to tune the preallocation (beware: the new values will be picked only when no application is using netmap). The one you are interested in are ring_num and buf_num. ring_size and buf_size, on the other end, set the size in bytes of each individual ring and buffer, respectively, which is probably not what you want. Note that the preallocation may partially fail: ring_curr_num and buf_curr_num will tell you how many rings and buffers have been actually preallocated. Each pipe needs 8 rings, with 4 of them actually unused and essentially empty (an unfortunate compatibility constraint that I hope to remove someday). You also need enough buffers to fill the 4 true rings of each pipe.
Thanks so much for your guidance! My aim is to run the load distributor, then it can distribute the traffic to other apps (I have tried lb + pkt-gen rx), then test the throughput and latency. I use MoonGen test the throughput and latency. If so, I have to make the apps can send the traffic to MoonGen which runs on another machine. lb + pkt-gen rx will not work which I can only test the throughput from the netmap machine. I thought lb + bridge (bridge: I thought one port will be the lb pipe output, the other port will be physical port p2p2. I would change the bridge code slightly to make different bridge apps write the packets to different queues) will be a good match. But if even I directly run bridge, the
zhangwei1984@nimbnode21:~/netmap/examples$ sudo ./bridge -i p2p2
./bridge built May 24 2016 14:05:05
929.111807 main [225] same interface, endpoint 0 goes to host
929.111829 main [233] cannot open p2p2^
Dmesg does not show useful information:
[67449.503730] ixgbe 0000:07:00.1: irq 134 for MSI/MSI-X
[67449.503817] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count = 24, Tx Queue count = 24
[67449.503950] ixgbe 0000:07:00.1: PCI Express bandwidth of 32GT/s available
[67449.503953] ixgbe 0000:07:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[67449.504039] ixgbe 0000:07:00.1: MAC: 2, PHY: 9, SFP+: 4, PBA No: E68793-006
[67449.504042] ixgbe 0000:07:00.1: 90:e2:ba:5b:eb:75
[67449.505774] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
[67449.505791] net eth0: netmap queues/slots: TX 24/512, RX 24/512
[67449.526876] systemd-udevd[30350]: renamed network interface eth0 to p2p2
[67466.093158] ixgbe 0000:07:00.1: registered PHC device on p2p2
[67466.196289] IPv6: ADDRCONF(NETDEV_UP): p2p2: link is not ready
[67466.260125] ixgbe 0000:07:00.1 p2p2: detected SFP+: 4
[67466.499301] ixgbe 0000:07:00.1 p2p2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[67466.499459] IPv6: ADDRCONF(NETDEV_CHANGE): p2p2: link becomes ready
If run the command
sudo ./bridge -i netmap:p2p2
The server will crash. Can you give some guidance how to achieve my goal: lb + apps, which apps can send the packets to the physical port, then MoonGen server can receives the packets measure the latency.
How can I do the math with (ring|buf)_(size|num) relationed to number of capture threads? I read the comment you wrote about number of ring/buf size, but I did not undestand how pragmatically I can get these numbers. Can you explain this?
I've encountered this problem a year ago and asked for help in #139, this helped me, I've increased value of ring_size and error is gone.
Today I've update netmap to latest master source and changing value of ring_size didn't help me.
Followed the instructions given by @giuseppelettieri , I changed the buf_num from 163840 to 1000000 to adapt it to my application on top. But it failed since I observed buf_cur_num became 0. I'm wondering if there is a legal range that I can play with buf_num?
Addition: Just noticed that the change of buf_num took effect finally. However, when I ran my application on top of netmap, the address sanitizer reported:
==21093==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f09d744d882 bp 0x000000001000 sp 0x62100001c538 T0)
Is this a sign that the modification of buf_num didn't actually work? or I missed some steps before I can make sure that the buf_num is actually changed.
I've encountered this problem with the latest code, the linux kernel version of my host is 18.04, and it's a virtual machine in the cloud. I only use netmap.ko, and nm_open failed. Aug 31 13:48:42 VZ202106150501 kernel: [ 2658.237950] 922.350033 [1017] netmap_obj_malloc netmap_ring request size 65792 too large Aug 31 13:48:42 VZ202106150501 kernel: [ 2658.239627] 922.351709 [1918] netmap_mem2_rings_create Cannot allocate RX_ring Aug 31 13:48:42 VZ202106150501 aiwan.sh[18885]: 922.353520 nm_open [946] NIOCREGIF failed: Cannot allocate memory netmap:eth0
I've changed ring_size, ring_num parameters and whatever I do it didn't work.
Then I build netmap.ko on the host running netmap application instead of coyping netmap.ko from other host with the same kernel version, and the error above is gone.
So netmap.ko has something to do with the compiling host, hope it helps.
Activity
leleobhz commentedon May 24, 2016
Append: With verbose:
vmaffione commentedon May 24, 2016
The problem is that netmap is being asked to allocate an insane number of buffers, and so it fails.
Why are you setting ring_size to such a huge value?
What does "ethtool -g" say (e.g. the ring sizes of your physical card)?
Why are you setting admode to 1? It should be 0, since you don't have a native netmap adapter.
leleobhz commentedon May 24, 2016
@vmaffione - the application is who asks for these number of buffers?
Just a test to see if someone got fixed. I did this test before try to read the source code to find what happens.
I guessed its about the mode the interface is set, not about the interface being a native netmap adapter. Changed this.
And with the configuration pasted before, the dmesg goes this way:
What can I do now? May this be a issue with my application and the way it instantiate the packet filter?
zhangwei1984 commentedon May 25, 2016
I have the same problem. #185
Hi,
I want to create more than 18 pipes when using lb application. After 18 (for example lb -i p2p2 -p 20), I got this error:261.135269 nm_open [847] NIOCREGIF failed: Cannot allocate memory p2p2{18
261.135276 main [456] cannot open netmap:p2p2{18
It seems that there is some limitation. There is a macro NM_MAXPIPES 64, but I can't make more than 18. It seems here is not the limitation. My question is that if I want to create more than 18 pipes, eg 100 pipes, where should I change to make it work?
From the error "Cannot allocate memory", inside function netmap_get_pipe_na()
here will allocate memory which is related to the number of pipes
https://github.com/luigirizzo/netmap/blob/master/sys/dev/netmap/netmap_pipe.c#L102.
But I do not think it was caused by here.
Dmesg shows the error:
[88943.459956] 112.380834 [ 840] netmap_obj_malloc no more netmap_ring objects
[88943.471745] ixgbe 0000:07:00.1 p2p2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[88943.481829] 112.402705 [1736] netmap_mem2_rings_create Cannot allocate TX_ring
[88943.852137] ixgbe 0000:07:00.1 p2p2: detected SFP+: 4
[88944.091797] ixgbe 0000:07:00.1 p2p2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Thanks
giuseppelettieri commentedon May 25, 2016
Hi all. Netmap preallocates rings and buffers from the kernel, then distributes them as required to NICs, VALE ports, pipes etc. You are running out of preallocated rings and/or buffers. In /sys/module/netmap/parameters there are a few variables that you can change to tune the preallocation (beware: the new values will be picked only when no application is using netmap). The one you are interested in are ring_num and buf_num. ring_size and buf_size, on the other end, set the size in bytes of each individual ring and buffer, respectively, which is probably not what you want. Note that the preallocation may partially fail: ring_curr_num and buf_curr_num will tell you how many rings and buffers have been actually preallocated. Each pipe needs 8 rings, with 4 of them actually unused and essentially empty (an unfortunate compatibility constraint that I hope to remove someday). You also need enough buffers to fill the 4 true rings of each pipe.
zhangwei1984 commentedon May 27, 2016
Thanks so much for your guidance! My aim is to run the load distributor, then it can distribute the traffic to other apps (I have tried lb + pkt-gen rx), then test the throughput and latency. I use MoonGen test the throughput and latency. If so, I have to make the apps can send the traffic to MoonGen which runs on another machine. lb + pkt-gen rx will not work which I can only test the throughput from the netmap machine. I thought lb + bridge (bridge: I thought one port will be the lb pipe output, the other port will be physical port p2p2. I would change the bridge code slightly to make different bridge apps write the packets to different queues) will be a good match. But if even I directly run bridge, the
Dmesg does not show useful information:
If run the command
The server will crash. Can you give some guidance how to achieve my goal: lb + apps, which apps can send the packets to the physical port, then MoonGen server can receives the packets measure the latency.
leleobhz commentedon May 30, 2016
Hello @giuseppelettieri
How can I do the math with (ring|buf)_(size|num) relationed to number of capture threads? I read the comment you wrote about number of ring/buf size, but I did not undestand how pragmatically I can get these numbers. Can you explain this?
Thanks!
gureedo commentedon Jan 11, 2017
Looks like option
ring_sizedoesn't work.I've encountered this problem a year ago and asked for help in #139, this helped me, I've increased value of
ring_sizeand error is gone.Today I've update netmap to latest master source and changing value of
ring_sizedidn't help me.vmaffione commentedon Sep 16, 2017
It looks more like a misconfiguration. The
ring_sizeis for physical NICs. Try to changepriv_ring_sizeif you are using VALE ports.gureedo commentedon Sep 18, 2017
It was a bug in master branch. Not actual anymore.
Winters123 commentedon Mar 23, 2020
Followed the instructions given by @giuseppelettieri , I changed the
buf_numfrom 163840 to 1000000 to adapt it to my application on top. But it failed since I observedbuf_cur_numbecame 0. I'm wondering if there is a legal range that I can play withbuf_num?Winters123 commentedon Mar 23, 2020
Addition: Just noticed that the change of
buf_numtook effect finally. However, when I ran my application on top of netmap, theaddress sanitizerreported:Is this a sign that the modification of
buf_numdidn't actually work? or I missed some steps before I can make sure that thebuf_numis actually changed.Cheers,
davidmpc commentedon Sep 14, 2021
I've encountered this problem with the latest code, the linux kernel version of my host is 18.04, and it's a virtual machine in the cloud. I only use netmap.ko, and nm_open failed.
Aug 31 13:48:42 VZ202106150501 kernel: [ 2658.237950] 922.350033 [1017] netmap_obj_malloc netmap_ring request size 65792 too large Aug 31 13:48:42 VZ202106150501 kernel: [ 2658.239627] 922.351709 [1918] netmap_mem2_rings_create Cannot allocate RX_ring Aug 31 13:48:42 VZ202106150501 aiwan.sh[18885]: 922.353520 nm_open [946] NIOCREGIF failed: Cannot allocate memory netmap:eth0I've changed ring_size, ring_num parameters and whatever I do it didn't work.
Then I build netmap.ko on the host running netmap application instead of coyping netmap.ko from other host with the same kernel version, and the error above is gone.
So netmap.ko has something to do with the compiling host, hope it helps.