joy/android_kernel_sony_sdm845
Watch
1
Fork
You've already forked android_kernel_sony_sdm845
 0
Code Issues Pull requests Projects Releases Wiki Activity

[NET]: Make NAPI polling independent of struct net_device objects.

Browse source 
Several devices have multiple independant RX queues per net
device, and some have a single interrupt doorbell for several
queues.

In either case, it's easier to support layouts like that if the
structure representing the poll is independant from the net
device itself.

The signature of the ->poll() call back goes from:

	int foo_poll(struct net_device *dev, int *budget)

to

	int foo_poll(struct napi_struct *napi, int budget)

The caller is returned the number of RX packets processed (or
the number of "NAPI credits" consumed if you want to get
abstract).  The callee no longer messes around bumping
dev->quota, *budget, etc. because that is all handled in the
caller upon return.

The napi_struct is to be embedded in the device driver private data
structures.

Furthermore, it is the driver's responsibility to disable all NAPI
instances in it's ->stop() device close handler.  Since the
napi_struct is privatized into the driver's private data structures,
only the driver knows how to get at all of the napi_struct instances
it may have per-device.

With lots of help and suggestions from Rusty Russell, Roland Dreier,
Michael Chan, Jeff Garzik, and Jamal Hadi Salim.

Bug fixes from Thomas Graf, Roland Dreier, Peter Zijlstra,
Joseph Fannin, Scott Wood, Hans J. Koch, and Michael Chan.

[ Ported to current tree and all drivers converted.  Integrated
  Stephen's follow-on kerneldoc additions, and restored poll_list
  handling to the old style to fix mutual exclusion issues.  -DaveM ]

Signed-off-by: Stephen Hemminger <shemminger@linux-foundation.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
This commit is contained in:
Stephen Hemminger 2007-10-03 16:41:36 -07:00 • committed by David S. Miller
commit bea3348eef
91 changed files with 1758 additions and 2375 deletions
Download patch file Download diff file Expand all files Collapse all files

12
Documentation/DocBook/kernel-api.tmpl
View file

@ -240,17 +240,23 @@ X!Ilib/string.c
<sect1><title>Driver Support</title>
!Enet/core/dev.c
!Enet/ethernet/eth.c
!Enet/sched/sch_generic.c
!Iinclude/linux/etherdevice.h
!Iinclude/linux/netdevice.h
</sect1>
<sect1><title>PHY Support</title>
!Edrivers/net/phy/phy.c
!Idrivers/net/phy/phy.c
!Edrivers/net/phy/phy_device.c
!Idrivers/net/phy/phy_device.c
!Edrivers/net/phy/mdio_bus.c
!Idrivers/net/phy/mdio_bus.c
<!-- FIXME: Removed for now since no structured comments in source
X!Enet/core/wireless.c
-->
</sect1>
<!-- FIXME: Removed for now since no structured comments in source
<sect1><title>Wireless</title>
X!Enet/core/wireless.c
</sect1>
-->
<sect1><title>Synchronous PPP</title>
!Edrivers/net/wan/syncppp.c
</sect1>

766
Documentation/networking/NAPI_HOWTO.txt
View file

@ -1,766 +0,0 @@
HISTORY:
February 16/2002 -- revision 0.2.1:
COR typo corrected
February 10/2002 -- revision 0.2:
some spell checking ;->
January 12/2002 -- revision 0.1
This is still work in progress so may change.
To keep up to date please watch this space.
Introduction to NAPI
====================
NAPI is a proven (www.cyberus.ca/~hadi/usenix-paper.tgz) technique
to improve network performance on Linux. For more details please
read that paper.
NAPI provides a "inherent mitigation" which is bound by system capacity
as can be seen from the following data collected by Robert on Gigabit
ethernet (e1000):
Psize Ipps Tput Rxint Txint Done Ndone
---------------------------------------------------------------
60 890000 409362 17 27622 7 6823
128 758150 464364 21 9301 10 7738
256 445632 774646 42 15507 21 12906
512 232666 994445 241292 19147 241192 1062
1024 119061 1000003 872519 19258 872511 0
1440 85193 1000003 946576 19505 946569 0
Legend:
"Ipps" stands for input packets per second.
"Tput" == packets out of total 1M that made it out.
"txint" == transmit completion interrupts seen
"Done" == The number of times that the poll() managed to pull all
packets out of the rx ring. Note from this that the lower the
load the more we could clean up the rxring
"Ndone" == is the converse of "Done". Note again, that the higher
the load the more times we couldn't clean up the rxring.
Observe that:
when the NIC receives 890Kpackets/sec only 17 rx interrupts are generated.
The system cant handle the processing at 1 interrupt/packet at that load level.
At lower rates on the other hand, rx interrupts go up and therefore the
interrupt/packet ratio goes up (as observable from that table). So there is
possibility that under low enough input, you get one poll call for each
input packet caused by a single interrupt each time. And if the system
cant handle interrupt per packet ratio of 1, then it will just have to
chug along ....
0) Prerequisites:
==================
A driver MAY continue using the old 2.4 technique for interfacing
to the network stack and not benefit from the NAPI changes.
NAPI additions to the kernel do not break backward compatibility.
NAPI, however, requires the following features to be available:
A) DMA ring or enough RAM to store packets in software devices.
B) Ability to turn off interrupts or maybe events that send packets up
the stack.
NAPI processes packet events in what is known as dev->poll() method.
Typically, only packet receive events are processed in dev->poll().
The rest of the events MAY be processed by the regular interrupt handler
to reduce processing latency (justified also because there are not that
many of them).
Note, however, NAPI does not enforce that dev->poll() only processes
receive events.
Tests with the tulip driver indicated slightly increased latency if
all of the interrupt handler is moved to dev->poll(). Also MII handling
gets a little trickier.
The example used in this document is to move the receive processing only
to dev->poll(); this is shown with the patch for the tulip driver.
For an example of code that moves all the interrupt driver to
dev->poll() look at the ported e1000 code.
There are caveats that might force you to go with moving everything to
dev->poll(). Different NICs work differently depending on their status/event
acknowledgement setup.
There are two types of event register ACK mechanisms.
I) what is known as Clear-on-read (COR).
when you read the status/event register, it clears everything!
The natsemi and sunbmac NICs are known to do this.
In this case your only choice is to move all to dev->poll()
II) Clear-on-write (COW)
i) you clear the status by writing a 1 in the bit-location you want.
These are the majority of the NICs and work the best with NAPI.
Put only receive events in dev->poll(); leave the rest in
the old interrupt handler.
ii) whatever you write in the status register clears every thing ;->
Cant seem to find any supported by Linux which do this. If
someone knows such a chip email us please.
Move all to dev->poll()
C) Ability to detect new work correctly.
NAPI works by shutting down event interrupts when there's work and
turning them on when there's none.
New packets might show up in the small window while interrupts were being
re-enabled (refer to appendix 2). A packet might sneak in during the period
we are enabling interrupts. We only get to know about such a packet when the
next new packet arrives and generates an interrupt.
Essentially, there is a small window of opportunity for a race condition
which for clarity we'll refer to as the "rotting packet".
This is a very important topic and appendix 2 is dedicated for more
discussion.
Locking rules and environmental guarantees
==========================================
-Guarantee: Only one CPU at any time can call dev->poll(); this is because
only one CPU can pick the initial interrupt and hence the initial
netif_rx_schedule(dev);
- The core layer invokes devices to send packets in a round robin format.
This implies receive is totally lockless because of the guarantee that only
one CPU is executing it.
- contention can only be the result of some other CPU accessing the rx
ring. This happens only in close() and suspend() (when these methods
try to clean the rx ring);
****guarantee: driver authors need not worry about this; synchronization
is taken care for them by the top net layer.
-local interrupts are enabled (if you dont move all to dev->poll()). For
example link/MII and txcomplete continue functioning just same old way.
This improves the latency of processing these events. It is also assumed that
the receive interrupt is the largest cause of noise. Note this might not
always be true.
[according to Manfred Spraul, the winbond insists on sending one
txmitcomplete interrupt for each packet (although this can be mitigated)].
For these broken drivers, move all to dev->poll().
For the rest of this text, we'll assume that dev->poll() only
processes receive events.
new methods introduce by NAPI
=============================
a) netif_rx_schedule(dev)
Called by an IRQ handler to schedule a poll for device
b) netif_rx_schedule_prep(dev)
puts the device in a state which allows for it to be added to the
CPU polling list if it is up and running. You can look at this as
the first half of netif_rx_schedule(dev) above; the second half
being c) below.
c) __netif_rx_schedule(dev)
Add device to the poll list for this CPU; assuming that _prep above
has already been called and returned 1.
d) netif_rx_reschedule(dev, undo)
Called to reschedule polling for device specifically for some
deficient hardware. Read Appendix 2 for more details.
e) netif_rx_complete(dev)
Remove interface from the CPU poll list: it must be in the poll list
on current cpu. This primitive is called by dev->poll(), when
it completes its work. The device cannot be out of poll list at this
call, if it is then clearly it is a BUG(). You'll know ;->
All of the above methods are used below, so keep reading for clarity.
Device driver changes to be made when porting NAPI
==================================================
Below we describe what kind of changes are required for NAPI to work.
1) introduction of dev->poll() method
=====================================
This is the method that is invoked by the network core when it requests
for new packets from the driver. A driver is allowed to send upto
dev->quota packets by the current CPU before yielding to the network
subsystem (so other devices can also get opportunity to send to the stack).
dev->poll() prototype looks as follows:
int my_poll(struct net_device *dev, int *budget)
budget is the remaining number of packets the network subsystem on the
current CPU can send up the stack before yielding to other system tasks.
*Each driver is responsible for decrementing budget by the total number of
packets sent.
Total number of packets cannot exceed dev->quota.
dev->poll() method is invoked by the top layer, the driver just sends if it
can to the stack the packet quantity requested.
more on dev->poll() below after the interrupt changes are explained.
2) registering dev->poll() method
===================================
dev->poll should be set in the dev->probe() method.
e.g:
dev->open = my_open;
.
.
/* two new additions */
/* first register my poll method */
dev->poll = my_poll;
/* next register my weight/quanta; can be overridden in /proc */
dev->weight = 16;
.
.
dev->stop = my_close;
3) scheduling dev->poll()
=============================
This involves modifying the interrupt handler and the code
path which takes the packet off the NIC and sends them to the
stack.
it's important at this point to introduce the classical D Becker
interrupt processor:
------------------
static irqreturn_t
netdevice_interrupt(int irq, void *dev_id, struct pt_regs *regs)
{
struct net_device *dev = (struct net_device *)dev_instance;
struct my_private *tp = (struct my_private *)dev->priv;
int work_count = my_work_count;
status = read_interrupt_status_reg();
if (status == 0)
return IRQ_NONE; /* Shared IRQ: not us */
if (status == 0xffff)
return IRQ_HANDLED; /* Hot unplug */
if (status & error)
do_some_error_handling()
do {
acknowledge_ints_ASAP();
if (status & link_interrupt) {
spin_lock(&tp->link_lock);
do_some_link_stat_stuff();
spin_lock(&tp->link_lock);
}
if (status & rx_interrupt) {
receive_packets(dev);
}
if (status & rx_nobufs) {
make_rx_buffs_avail();
}
if (status & tx_related) {
spin_lock(&tp->lock);
tx_ring_free(dev);
if (tx_died)
restart_tx();
spin_unlock(&tp->lock);
}
status = read_interrupt_status_reg();
} while (!(status & error) || more_work_to_be_done);
return IRQ_HANDLED;
}
----------------------------------------------------------------------
We now change this to what is shown below to NAPI-enable it:
----------------------------------------------------------------------
static irqreturn_t
netdevice_interrupt(int irq, void *dev_id, struct pt_regs *regs)
{
struct net_device *dev = (struct net_device *)dev_instance;
struct my_private *tp = (struct my_private *)dev->priv;
status = read_interrupt_status_reg();
if (status == 0)
return IRQ_NONE; /* Shared IRQ: not us */
if (status == 0xffff)
return IRQ_HANDLED; /* Hot unplug */
if (status & error)
do_some_error_handling();
do {
/************************ start note *********************************/
acknowledge_ints_ASAP(); // dont ack rx and rxnobuff here
/************************ end note *********************************/
if (status & link_interrupt) {
spin_lock(&tp->link_lock);
do_some_link_stat_stuff();
spin_unlock(&tp->link_lock);
}
/************************ start note *********************************/
if (status & rx_interrupt || (status & rx_nobuffs)) {
if (netif_rx_schedule_prep(dev)) {
/* disable interrupts caused
* by arriving packets */
disable_rx_and_rxnobuff_ints();
/* tell system we have work to be done. */
__netif_rx_schedule(dev);
} else {
printk("driver bug! interrupt while in poll\n");
/* FIX by disabling interrupts */
disable_rx_and_rxnobuff_ints();
}
}
/************************ end note note *********************************/
if (status & tx_related) {
spin_lock(&tp->lock);
tx_ring_free(dev);
if (tx_died)
restart_tx();
spin_unlock(&tp->lock);
}
status = read_interrupt_status_reg();
/************************ start note *********************************/
} while (!(status & error) || more_work_to_be_done(status));
/************************ end note note *********************************/
return IRQ_HANDLED;
}
---------------------------------------------------------------------
We note several things from above:
I) Any interrupt source which is caused by arriving packets is now
turned off when it occurs. Depending on the hardware, there could be
several reasons that arriving packets would cause interrupts; these are the
interrupt sources we wish to avoid. The two common ones are a) a packet
arriving (rxint) b) a packet arriving and finding no DMA buffers available
(rxnobuff) .
This means also acknowledge_ints_ASAP() will not clear the status
register for those two items above; clearing is done in the place where
proper work is done within NAPI; at the poll() and refill_rx_ring()
discussed further below.
netif_rx_schedule_prep() returns 1 if device is in running state and
gets successfully added to the core poll list. If we get a zero value
we can _almost_ assume are already added to the list (instead of not running.
Logic based on the fact that you shouldn't get interrupt if not running)
We rectify this by disabling rx and rxnobuf interrupts.
II) that receive_packets(dev) and make_rx_buffs_avail() may have disappeared.
These functionalities are still around actually......
infact, receive_packets(dev) is very close to my_poll() and
make_rx_buffs_avail() is invoked from my_poll()
4) converting receive_packets() to dev->poll()
===============================================
We need to convert the classical D Becker receive_packets(dev) to my_poll()
First the typical receive_packets() below:
-------------------------------------------------------------------
/* this is called by interrupt handler */
static void receive_packets (struct net_device *dev)
{
struct my_private *tp = (struct my_private *)dev->priv;
rx_ring = tp->rx_ring;
cur_rx = tp->cur_rx;
int entry = cur_rx % RX_RING_SIZE;
int received = 0;
int rx_work_limit = tp->dirty_rx + RX_RING_SIZE - tp->cur_rx;
while (rx_ring_not_empty) {
u32 rx_status;
unsigned int rx_size;
unsigned int pkt_size;
struct sk_buff *skb;
/* read size+status of next frame from DMA ring buffer */
/* the number 16 and 4 are just examples */
rx_status = le32_to_cpu (*(u32 *) (rx_ring + ring_offset));
rx_size = rx_status >> 16;
pkt_size = rx_size - 4;
/* process errors */
if ((rx_size > (MAX_ETH_FRAME_SIZE+4)) ||
(!(rx_status & RxStatusOK))) {
netdrv_rx_err (rx_status, dev, tp, ioaddr);
return;
}
if (--rx_work_limit < 0)
break;
/* grab a skb */
skb = dev_alloc_skb (pkt_size + 2);
if (skb) {
.
.
netif_rx (skb);
.
.
} else { /* OOM */
/*seems very driver specific ... some just pass
whatever is on the ring already. */
}
/* move to the next skb on the ring */
entry = (++tp->cur_rx) % RX_RING_SIZE;
received++ ;
}
/* store current ring pointer state */
tp->cur_rx = cur_rx;
/* Refill the Rx ring buffers if they are needed */
refill_rx_ring();
.
.
}
-------------------------------------------------------------------
We change it to a new one below; note the additional parameter in
the call.
-------------------------------------------------------------------
/* this is called by the network core */
static int my_poll (struct net_device *dev, int *budget)
{
struct my_private *tp = (struct my_private *)dev->priv;
rx_ring = tp->rx_ring;
cur_rx = tp->cur_rx;
int entry = cur_rx % RX_BUF_LEN;
/* maximum packets to send to the stack */
/************************ note note *********************************/
int rx_work_limit = dev->quota;
/************************ end note note *********************************/
do { // outer beginning loop starts here
clear_rx_status_register_bit();
while (rx_ring_not_empty) {
u32 rx_status;
unsigned int rx_size;
unsigned int pkt_size;
struct sk_buff *skb;
/* read size+status of next frame from DMA ring buffer */
/* the number 16 and 4 are just examples */
rx_status = le32_to_cpu (*(u32 *) (rx_ring + ring_offset));
rx_size = rx_status >> 16;
pkt_size = rx_size - 4;
/* process errors */
if ((rx_size > (MAX_ETH_FRAME_SIZE+4)) ||
(!(rx_status & RxStatusOK))) {
netdrv_rx_err (rx_status, dev, tp, ioaddr);
return 1;
}
/************************ note note *********************************/
if (--rx_work_limit < 0) { /* we got packets, but no quota */
/* store current ring pointer state */
tp->cur_rx = cur_rx;
/* Refill the Rx ring buffers if they are needed */
refill_rx_ring(dev);
goto not_done;
}
/********************** end note **********************************/
/* grab a skb */
skb = dev_alloc_skb (pkt_size + 2);
if (skb) {
.
.
/************************ note note *********************************/
netif_receive_skb (skb);
/********************** end note **********************************/
.
.
} else { /* OOM */
/*seems very driver specific ... common is just pass
whatever is on the ring already. */
}
/* move to the next skb on the ring */
entry = (++tp->cur_rx) % RX_RING_SIZE;
received++ ;
}
/* store current ring pointer state */
tp->cur_rx = cur_rx;
/* Refill the Rx ring buffers if they are needed */
refill_rx_ring(dev);
/* no packets on ring; but new ones can arrive since we last
checked */
status = read_interrupt_status_reg();
if (rx status is not set) {
/* If something arrives in this narrow window,
an interrupt will be generated */
goto done;
}
/* done! at least that's what it looks like ;->
if new packets came in after our last check on status bits
they'll be caught by the while check and we go back and clear them
since we havent exceeded our quota */
} while (rx_status_is_set);
done:
/************************ note note *********************************/
dev->quota -= received;
*budget -= received;
/* If RX ring is not full we are out of memory. */
if (tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL)
goto oom;
/* we are happy/done, no more packets on ring; put us back
to where we can start processing interrupts again */
netif_rx_complete(dev);
enable_rx_and_rxnobuf_ints();
/* The last op happens after poll completion. Which means the following:
* 1. it can race with disabling irqs in irq handler (which are done to
* schedule polls)
* 2. it can race with dis/enabling irqs in other poll threads
* 3. if an irq raised after the beginning of the outer beginning
* loop (marked in the code above), it will be immediately
* triggered here.
*
* Summarizing: the logic may result in some redundant irqs both
* due to races in masking and due to too late acking of already
* processed irqs. The good news: no events are ever lost.
*/
return 0; /* done */
not_done:
if (tp->cur_rx - tp->dirty_rx > RX_RING_SIZE/2 ||
tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL)
refill_rx_ring(dev);
if (!received) {
printk("received==0\n");
received = 1;
}
dev->quota -= received;
*budget -= received;
return 1; /* not_done */
oom:
/* Start timer, stop polling, but do not enable rx interrupts. */
start_poll_timer(dev);
return 0; /* we'll take it from here so tell core "done"*/
/************************ End note note *********************************/
}
-------------------------------------------------------------------
From above we note that:
0) rx_work_limit = dev->quota
1) refill_rx_ring() is in charge of clearing the bit for rxnobuff when
it does the work.
2) We have a done and not_done state.
3) instead of netif_rx() we call netif_receive_skb() to pass the skb.
4) we have a new way of handling oom condition
5) A new outer for (;;) loop has been added. This serves the purpose of
ensuring that if a new packet has come in, after we are all set and done,
and we have not exceeded our quota that we continue sending packets up.
-----------------------------------------------------------
Poll timer code will need to do the following:
a)
if (tp->cur_rx - tp->dirty_rx > RX_RING_SIZE/2 ||
tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL)
refill_rx_ring(dev);
/* If RX ring is not full we are still out of memory.
Restart the timer again. Else we re-add ourselves
to the master poll list.
*/
if (tp->rx_buffers[tp->dirty_rx % RX_RING_SIZE].skb == NULL)
restart_timer();
else netif_rx_schedule(dev); /* we are back on the poll list */
5) dev->close() and dev->suspend() issues
==========================================
The driver writer needn't worry about this; the top net layer takes
care of it.
6) Adding new Stats to /proc
=============================
In order to debug some of the new features, we introduce new stats
that need to be collected.
TODO: Fill this later.
APPENDIX 1: discussion on using ethernet HW FC
==============================================
Most chips with FC only send a pause packet when they run out of Rx buffers.
Since packets are pulled off the DMA ring by a softirq in NAPI,
if the system is slow in grabbing them and we have a high input
rate (faster than the system's capacity to remove packets), then theoretically
there will only be one rx interrupt for all packets during a given packetstorm.
Under low load, we might have a single interrupt per packet.
FC should be programmed to apply in the case when the system cant pull out
packets fast enough i.e send a pause only when you run out of rx buffers.
Note FC in itself is a good solution but we have found it to not be
much of a commodity feature (both in NICs and switches) and hence falls
under the same category as using NIC based mitigation. Also, experiments
indicate that it's much harder to resolve the resource allocation
issue (aka lazy receiving that NAPI offers) and hence quantify its usefulness
proved harder. In any case, FC works even better with NAPI but is not
necessary.
APPENDIX 2: the "rotting packet" race-window avoidance scheme
=============================================================
There are two types of associations seen here
1) status/int which honors level triggered IRQ
If a status bit for receive or rxnobuff is set and the corresponding
interrupt-enable bit is not on, then no interrupts will be generated. However,
as soon as the "interrupt-enable" bit is unmasked, an immediate interrupt is
generated. [assuming the status bit was not turned off].
Generally the concept of level triggered IRQs in association with a status and
interrupt-enable CSR register set is used to avoid the race.
If we take the example of the tulip:
"pending work" is indicated by the status bit(CSR5 in tulip).
the corresponding interrupt bit (CSR7 in tulip) might be turned off (but
the CSR5 will continue to be turned on with new packet arrivals even if
we clear it the first time)
Very important is the fact that if we turn on the interrupt bit on when
status is set that an immediate irq is triggered.
If we cleared the rx ring and proclaimed there was "no more work
to be done" and then went on to do a few other things; then when we enable
interrupts, there is a possibility that a new packet might sneak in during
this phase. It helps to look at the pseudo code for the tulip poll
routine:
--------------------------
do {
ACK;
while (ring_is_not_empty()) {
work-work-work
if quota is exceeded: exit, no touching irq status/mask
}
/* No packets, but new can arrive while we are doing this*/
CSR5 := read
if (CSR5 is not set) {
/* If something arrives in this narrow window here,
* where the comments are ;-> irq will be generated */
unmask irqs;
exit poll;
}
} while (rx_status_is_set);
------------------------
CSR5 bit of interest is only the rx status.
If you look at the last if statement:
you just finished grabbing all the packets from the rx ring .. you check if
status bit says there are more packets just in ... it says none; you then
enable rx interrupts again; if a new packet just came in during this check,
we are counting that CSR5 will be set in that small window of opportunity
and that by re-enabling interrupts, we would actually trigger an interrupt
to register the new packet for processing.
[The above description nay be very verbose, if you have better wording
that will make this more understandable, please suggest it.]
2) non-capable hardware
These do not generally respect level triggered IRQs. Normally,
irqs may be lost while being masked and the only way to leave poll is to do
a double check for new input after netif_rx_complete() is invoked
and re-enable polling (after seeing this new input).
Sample code:
---------
.
.
restart_poll:
while (ring_is_not_empty()) {
work-work-work
if quota is exceeded: exit, not touching irq status/mask
}
.
.
.
enable_rx_interrupts()
netif_rx_complete(dev);
if (ring_has_new_packet() && netif_rx_reschedule(dev, received)) {
disable_rx_and_rxnobufs()
goto restart_poll
} while (rx_status_is_set);
---------
Basically netif_rx_complete() removes us from the poll list, but because a
new packet which will never be caught due to the possibility of a race
might come in, we attempt to re-add ourselves to the poll list.
APPENDIX 3: Scheduling issues.
==============================
As seen NAPI moves processing to softirq level. Linux uses the ksoftirqd as the
general solution to schedule softirq's to run before next interrupt and by putting
them under scheduler control. Also this prevents consecutive softirq's from
monopolize the CPU. This also have the effect that the priority of ksoftirq needs
to be considered when running very CPU-intensive applications and networking to
get the proper balance of softirq/user balance. Increasing ksoftirq priority to 0
(eventually more) is reported cure problems with low network performance at high
CPU load.
Most used processes in a GIGE router:
USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND
root 3 0.2 0.0 0 0 ? RWN Aug 15 602:00 (ksoftirqd_CPU0)
root 232 0.0 7.9 41400 40884 ? S Aug 15 74:12 gated
--------------------------------------------------------------------
relevant sites:
==================
ftp://robur.slu.se/pub/Linux/net-development/NAPI/
--------------------------------------------------------------------
TODO: Write net-skeleton.c driver.
-------------------------------------------------------------
Authors:
========
Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Jamal Hadi Salim <hadi@cyberus.ca>
Robert Olsson <Robert.Olsson@data.slu.se>
Acknowledgements:
================
People who made this document better:
Lennert Buytenhek <buytenh@gnu.org>
Andrew Morton <akpm@zip.com.au>
Manfred Spraul <manfred@colorfullife.com>
Donald Becker <becker@scyld.com>
Jeff Garzik <jgarzik@pobox.com>

12
Documentation/networking/netdevices.txt
View file

@ -95,9 +95,13 @@ dev->set_multicast_list:
Synchronization: netif_tx_lock spinlock.
Context: BHs disabled
dev->poll:
Synchronization: __LINK_STATE_RX_SCHED bit in dev->state. See
dev_close code and comments in net/core/dev.c for more info.
struct napi_struct synchronization rules
========================================
napi->poll:
Synchronization: NAPI_STATE_SCHED bit in napi->state. Device
driver's dev->close method will invoke napi_disable() on
all NAPI instances which will do a sleeping poll on the
NAPI_STATE_SCHED napi->state bit, waiting for all pending
NAPI activity to cease.
Context: softirq
will be called with interrupts disabled by netconsole.

4
drivers/infiniband/ulp/ipoib/ipoib.h
View file

@ -228,6 +228,8 @@ struct ipoib_dev_priv {
struct net_device *dev;
struct napi_struct napi;
unsigned long flags;
struct mutex mcast_mutex;
@ -351,7 +353,7 @@ extern struct workqueue_struct *ipoib_workqueue;
/* functions */
int ipoib_poll(struct net_device *dev, int *budget);
int ipoib_poll(struct napi_struct *napi, int budget);
void ipoib_ib_completion(struct ib_cq *cq, void *dev_ptr);
struct ipoib_ah *ipoib_create_ah(struct net_device *dev,

43
drivers/infiniband/ulp/ipoib/ipoib_ib.c
View file

@ -281,63 +281,58 @@ static void ipoib_ib_handle_tx_wc(struct net_device *dev, struct ib_wc *wc)
wc->status, wr_id, wc->vendor_err);
}
int ipoib_poll(struct net_device *dev, int *budget)
int ipoib_poll(struct napi_struct *napi, int budget)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
int max = min(*budget, dev->quota);
struct ipoib_dev_priv *priv = container_of(napi, struct ipoib_dev_priv, napi);
struct net_device *dev = priv->dev;
int done;
int t;
int empty;
int n, i;
done = 0;
empty = 0;
while (max) {
poll_more:
while (done < budget) {
int max = (budget - done);
t = min(IPOIB_NUM_WC, max);
n = ib_poll_cq(priv->cq, t, priv->ibwc);
for (i = 0; i < n; ++i) {
for (i = 0; i < n; i++) {
struct ib_wc *wc = priv->ibwc + i;
if (wc->wr_id & IPOIB_CM_OP_SRQ) {
++done;
--max;
ipoib_cm_handle_rx_wc(dev, wc);
} else if (wc->wr_id & IPOIB_OP_RECV) {
++done;
--max;
ipoib_ib_handle_rx_wc(dev, wc);
} else
ipoib_ib_handle_tx_wc(dev, wc);
}
if (n != t) {
empty = 1;
if (n != t)
break;
}
}
dev->quota -= done;
*budget -= done;
if (empty) {
netif_rx_complete(dev);
if (done < budget) {
netif_rx_complete(dev, napi);
if (unlikely(ib_req_notify_cq(priv->cq,
IB_CQ_NEXT_COMP |
IB_CQ_REPORT_MISSED_EVENTS)) &&
netif_rx_reschedule(dev, 0))
return 1;
return 0;
netif_rx_reschedule(dev, napi))
goto poll_more;
}
return 1;
return done;
}
void ipoib_ib_completion(struct ib_cq *cq, void *dev_ptr)
{
netif_rx_schedule(dev_ptr);
struct net_device *dev = dev_ptr;
struct ipoib_dev_priv *priv = netdev_priv(dev);
netif_rx_schedule(dev, &priv->napi);
}
static inline int post_send(struct ipoib_dev_priv *priv,
@ -577,7 +572,6 @@ int ipoib_ib_dev_stop(struct net_device *dev, int flush)
int i;
clear_bit(IPOIB_FLAG_INITIALIZED, &priv->flags);
netif_poll_disable(dev);
ipoib_cm_dev_stop(dev);
@ -660,7 +654,6 @@ timeout:
msleep(1);
}
netif_poll_enable(dev);
ib_req_notify_cq(priv->cq, IB_CQ_NEXT_COMP);
return 0;

11
drivers/infiniband/ulp/ipoib/ipoib_main.c
View file

@ -98,16 +98,20 @@ int ipoib_open(struct net_device *dev)
ipoib_dbg(priv, "bringing up interface\n");
napi_enable(&priv->napi);
set_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags);
if (ipoib_pkey_dev_delay_open(dev))
return 0;
if (ipoib_ib_dev_open(dev))
if (ipoib_ib_dev_open(dev)) {
napi_disable(&priv->napi);
return -EINVAL;
}
if (ipoib_ib_dev_up(dev)) {
ipoib_ib_dev_stop(dev, 1);
napi_disable(&priv->napi);
return -EINVAL;
}
@ -140,6 +144,7 @@ static int ipoib_stop(struct net_device *dev)
ipoib_dbg(priv, "stopping interface\n");
clear_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags);
napi_disable(&priv->napi);
netif_stop_queue(dev);
@ -948,8 +953,8 @@ static void ipoib_setup(struct net_device *dev)
dev->hard_header = ipoib_hard_header;
dev->set_multicast_list = ipoib_set_mcast_list;
dev->neigh_setup = ipoib_neigh_setup_dev;
dev->poll = ipoib_poll;
dev->weight = 100;
netif_napi_add(dev, &priv->napi, ipoib_poll, 100);
dev->watchdog_timeo = HZ;

41
drivers/net/8139cp.c
View file

@ -334,6 +334,8 @@ struct cp_private {
spinlock_t lock;
u32 msg_enable;
struct napi_struct napi;
struct pci_dev *pdev;
u32 rx_config;
u16 cpcmd;
@ -501,12 +503,12 @@ static inline unsigned int cp_rx_csum_ok (u32 status)
return 0;
}
static int cp_rx_poll (struct net_device *dev, int *budget)
static int cp_rx_poll(struct napi_struct *napi, int budget)
{
struct cp_private *cp = netdev_priv(dev);
unsigned rx_tail = cp->rx_tail;
unsigned rx_work = dev->quota;
unsigned rx;
struct cp_private *cp = container_of(napi, struct cp_private, napi);
struct net_device *dev = cp->dev;
unsigned int rx_tail = cp->rx_tail;
int rx;
rx_status_loop:
rx = 0;
@ -588,33 +590,28 @@ rx_next:
desc->opts1 = cpu_to_le32(DescOwn | cp->rx_buf_sz);
rx_tail = NEXT_RX(rx_tail);
if (!rx_work--)
if (rx >= budget)
break;
}
cp->rx_tail = rx_tail;
dev->quota -= rx;
*budget -= rx;
/* if we did not reach work limit, then we're done with
* this round of polling
*/
if (rx_work) {
if (rx < budget) {
unsigned long flags;
if (cpr16(IntrStatus) & cp_rx_intr_mask)
goto rx_status_loop;
local_irq_save(flags);
spin_lock_irqsave(&cp->lock, flags);
cpw16_f(IntrMask, cp_intr_mask);
__netif_rx_complete(dev);
local_irq_restore(flags);
return 0; /* done */
__netif_rx_complete(dev, napi);
spin_unlock_irqrestore(&cp->lock, flags);
}
return 1; /* not done */
return rx;
}
static irqreturn_t cp_interrupt (int irq, void *dev_instance)
@ -647,9 +644,9 @@ static irqreturn_t cp_interrupt (int irq, void *dev_instance)
}
if (status & (RxOK | RxErr | RxEmpty | RxFIFOOvr))
if (netif_rx_schedule_prep(dev)) {
if (netif_rx_schedule_prep(dev, &cp->napi)) {
cpw16_f(IntrMask, cp_norx_intr_mask);
__netif_rx_schedule(dev);
__netif_rx_schedule(dev, &cp->napi);
}
if (status & (TxOK | TxErr | TxEmpty | SWInt))
@ -1175,6 +1172,8 @@ static int cp_open (struct net_device *dev)
if (rc)
return rc;
napi_enable(&cp->napi);
cp_init_hw(cp);
rc = request_irq(dev->irq, cp_interrupt, IRQF_SHARED, dev->name, dev);
@ -1188,6 +1187,7 @@ static int cp_open (struct net_device *dev)
return 0;
err_out_hw:
napi_disable(&cp->napi);
cp_stop_hw(cp);
cp_free_rings(cp);
return rc;
@ -1198,6 +1198,8 @@ static int cp_close (struct net_device *dev)
struct cp_private *cp = netdev_priv(dev);
unsigned long flags;
napi_disable(&cp->napi);
if (netif_msg_ifdown(cp))
printk(KERN_DEBUG "%s: disabling interface\n", dev->name);
@ -1933,11 +1935,10 @@ static int cp_init_one (struct pci_dev *pdev, const struct pci_device_id *ent)
dev->hard_start_xmit = cp_start_xmit;
dev->get_stats = cp_get_stats;
dev->do_ioctl = cp_ioctl;
dev->poll = cp_rx_poll;
#ifdef CONFIG_NET_POLL_CONTROLLER
dev->poll_controller = cp_poll_controller;
#endif
dev->weight = 16; /* arbitrary? from NAPI_HOWTO.txt. */
netif_napi_add(dev, &cp->napi, cp_rx_poll, 16);
#ifdef BROKEN
dev->change_mtu = cp_change_mtu;
#endif

48
drivers/net/8139too.c
View file

@ -573,6 +573,8 @@ struct rtl8139_private {
int drv_flags;
struct pci_dev *pci_dev;
u32 msg_enable;
struct napi_struct napi;
struct net_device *dev;
struct net_device_stats stats;
unsigned char *rx_ring;
unsigned int cur_rx; /* Index into the Rx buffer of next Rx pkt. */
@ -625,10 +627,10 @@ static void rtl8139_tx_timeout (struct net_device *dev);
static void rtl8139_init_ring (struct net_device *dev);
static int rtl8139_start_xmit (struct sk_buff *skb,
struct net_device *dev);
static int rtl8139_poll(struct net_device *dev, int *budget);
#ifdef CONFIG_NET_POLL_CONTROLLER
static void rtl8139_poll_controller(struct net_device *dev);
#endif
static int rtl8139_poll(struct napi_struct *napi, int budget);
static irqreturn_t rtl8139_interrupt (int irq, void *dev_instance);
static int rtl8139_close (struct net_device *dev);
static int netdev_ioctl (struct net_device *dev, struct ifreq *rq, int cmd);
@ -963,6 +965,7 @@ static int __devinit rtl8139_init_one (struct pci_dev *pdev,
assert (dev != NULL);
tp = netdev_priv(dev);
tp->dev = dev;
ioaddr = tp->mmio_addr;
assert (ioaddr != NULL);
@ -976,8 +979,7 @@ static int __devinit rtl8139_init_one (struct pci_dev *pdev,
/* The Rtl8139-specific entries in the device structure. */
dev->open = rtl8139_open;
dev->hard_start_xmit = rtl8139_start_xmit;
dev->poll = rtl8139_poll;
dev->weight = 64;
netif_napi_add(dev, &tp->napi, rtl8139_poll, 64);
dev->stop = rtl8139_close;
dev->get_stats = rtl8139_get_stats;
dev->set_multicast_list = rtl8139_set_rx_mode;
@ -1332,6 +1334,8 @@ static int rtl8139_open (struct net_device *dev)
}
napi_enable(&tp->napi);
tp->mii.full_duplex = tp->mii.force_media;
tp->tx_flag = (TX_FIFO_THRESH << 11) & 0x003f0000;
@ -2103,39 +2107,32 @@ static void rtl8139_weird_interrupt (struct net_device *dev,
}
}
static int rtl8139_poll(struct net_device *dev, int *budget)
static int rtl8139_poll(struct napi_struct *napi, int budget)
{
struct rtl8139_private *tp = netdev_priv(dev);
struct rtl8139_private *tp = container_of(napi, struct rtl8139_private, napi);
struct net_device *dev = tp->dev;
void __iomem *ioaddr = tp->mmio_addr;
int orig_budget = min(*budget, dev->quota);
int done = 1;
int work_done;
spin_lock(&tp->rx_lock);
if (likely(RTL_R16(IntrStatus) & RxAckBits)) {
int work_done;
work_done = 0;
if (likely(RTL_R16(IntrStatus) & RxAckBits))
work_done += rtl8139_rx(dev, tp, budget);
work_done = rtl8139_rx(dev, tp, orig_budget);
if (likely(work_done > 0)) {
*budget -= work_done;
dev->quota -= work_done;
done = (work_done < orig_budget);
}
}
if (done) {
if (work_done < budget) {
unsigned long flags;
/*
* Order is important since data can get interrupted
* again when we think we are done.
*/
local_irq_save(flags);
spin_lock_irqsave(&tp->lock, flags);
RTL_W16_F(IntrMask, rtl8139_intr_mask);
__netif_rx_complete(dev);
local_irq_restore(flags);
__netif_rx_complete(dev, napi);
spin_unlock_irqrestore(&tp->lock, flags);
}
spin_unlock(&tp->rx_lock);
return !done;
return work_done;
}
/* The interrupt handler does all of the Rx thread work and cleans up
@ -2180,9 +2177,9 @@ static irqreturn_t rtl8139_interrupt (int irq, void *dev_instance)
/* Receive packets are processed by poll routine.
If not running start it now. */
if (status & RxAckBits){
if (netif_rx_schedule_prep(dev)) {
if (netif_rx_schedule_prep(dev, &tp->napi)) {
RTL_W16_F (IntrMask, rtl8139_norx_intr_mask);
__netif_rx_schedule (dev);
__netif_rx_schedule(dev, &tp->napi);
}
}
@ -2223,7 +2220,8 @@ static int rtl8139_close (struct net_device *dev)
void __iomem *ioaddr = tp->mmio_addr;
unsigned long flags;
netif_stop_queue (dev);
netif_stop_queue(dev);
napi_disable(&tp->napi);
if (netif_msg_ifdown(tp))
printk(KERN_DEBUG "%s: Shutting down ethercard, status was 0x%4.4x.\n",

30
drivers/net/amd8111e.c
View file

@ -723,9 +723,10 @@ static int amd8111e_tx(struct net_device *dev)
#ifdef CONFIG_AMD8111E_NAPI
/* This function handles the driver receive operation in polling mode */
static int amd8111e_rx_poll(struct net_device *dev, int * budget)
static int amd8111e_rx_poll(struct napi_struct *napi, int budget)
{
struct amd8111e_priv *lp = netdev_priv(dev);
struct amd8111e_priv *lp = container_of(napi, struct amd8111e_priv, napi);
struct net_device *dev = lp->amd8111e_net_dev;
int rx_index = lp->rx_idx & RX_RING_DR_MOD_MASK;
void __iomem *mmio = lp->mmio;
struct sk_buff *skb,*new_skb;
@ -737,7 +738,7 @@ static int amd8111e_rx_poll(struct net_device *dev, int * budget)
#if AMD8111E_VLAN_TAG_USED
short vtag;
#endif
int rx_pkt_limit = dev->quota;
int rx_pkt_limit = budget;
unsigned long flags;
do{
@ -838,21 +839,14 @@ static int amd8111e_rx_poll(struct net_device *dev, int * budget)
} while(intr0 & RINT0);
/* Receive descriptor is empty now */
dev->quota -= num_rx_pkt;
*budget -= num_rx_pkt;
spin_lock_irqsave(&lp->lock, flags);
netif_rx_complete(dev);
__netif_rx_complete(dev, napi);
writel(VAL0|RINTEN0, mmio + INTEN0);
writel(VAL2 | RDMD0, mmio + CMD0);
spin_unlock_irqrestore(&lp->lock, flags);
return 0;
rx_not_empty:
/* Do not call a netif_rx_complete */
dev->quota -= num_rx_pkt;
*budget -= num_rx_pkt;
return 1;
return num_rx_pkt;
}
#else
@ -1287,11 +1281,11 @@ static irqreturn_t amd8111e_interrupt(int irq, void *dev_id)
/* Check if Receive Interrupt has occurred. */
#ifdef CONFIG_AMD8111E_NAPI
if(intr0 & RINT0){
if(netif_rx_schedule_prep(dev)){
if(netif_rx_schedule_prep(dev, &lp->napi)){
/* Disable receive interupts */
writel(RINTEN0, mmio + INTEN0);
/* Schedule a polling routine */
__netif_rx_schedule(dev);
__netif_rx_schedule(dev, &lp->napi);
}
else if (intren0 & RINTEN0) {
printk("************Driver bug! \
@ -1345,6 +1339,8 @@ static int amd8111e_close(struct net_device * dev)
struct amd8111e_priv *lp = netdev_priv(dev);
netif_stop_queue(dev);
napi_disable(&lp->napi);
spin_lock_irq(&lp->lock);
amd8111e_disable_interrupt(lp);
@ -1375,12 +1371,15 @@ static int amd8111e_open(struct net_device * dev )
dev->name, dev))
return -EAGAIN;
napi_enable(&lp->napi);
spin_lock_irq(&lp->lock);
amd8111e_init_hw_default(lp);
if(amd8111e_restart(dev)){
spin_unlock_irq(&lp->lock);
napi_disable(&lp->napi);
if (dev->irq)
free_irq(dev->irq, dev);
return -ENOMEM;
@ -2031,8 +2030,7 @@ static int __devinit amd8111e_probe_one(struct pci_dev *pdev,
dev->tx_timeout = amd8111e_tx_timeout;
dev->watchdog_timeo = AMD8111E_TX_TIMEOUT;
#ifdef CONFIG_AMD8111E_NAPI
dev->poll = amd8111e_rx_poll;
dev->weight = 32;
netif_napi_add(dev, &lp->napi, amd8111e_rx_poll, 32);
#endif
#ifdef CONFIG_NET_POLL_CONTROLLER
dev->poll_controller = amd8111e_poll;

2
drivers/net/amd8111e.h
View file

@ -763,6 +763,8 @@ struct amd8111e_priv{
/* Reg memory mapped address */
void __iomem *mmio;
struct napi_struct napi;
spinlock_t lock; /* Guard lock */
unsigned long rx_idx, tx_idx; /* The next free ring entry */
unsigned long tx_complete_idx;

70
drivers/net/arm/ep93xx_eth.c
View file

@ -169,6 +169,9 @@ struct ep93xx_priv
spinlock_t tx_pending_lock;
unsigned int tx_pending;
struct net_device *dev;
struct napi_struct napi;
struct net_device_stats stats;
struct mii_if_info mii;
@ -190,15 +193,11 @@ static struct net_device_stats *ep93xx_get_stats(struct net_device *dev)
return &(ep->stats);
}
static int ep93xx_rx(struct net_device *dev, int *budget)
static int ep93xx_rx(struct net_device *dev, int processed, int budget)
{
struct ep93xx_priv *ep = netdev_priv(dev);
int rx_done;
int processed;
rx_done = 0;
processed = 0;
while (*budget > 0) {
while (processed < budget) {
int entry;
struct ep93xx_rstat *rstat;
u32 rstat0;
@ -211,10 +210,8 @@ static int ep93xx_rx(struct net_device *dev, int *budget)
rstat0 = rstat->rstat0;
rstat1 = rstat->rstat1;
if (!(rstat0 & RSTAT0_RFP) || !(rstat1 & RSTAT1_RFP)) {
rx_done = 1;
if (!(rstat0 & RSTAT0_RFP) || !(rstat1 & RSTAT1_RFP))
break;
}
rstat->rstat0 = 0;
rstat->rstat1 = 0;
@ -275,8 +272,6 @@ static int ep93xx_rx(struct net_device *dev, int *budget)
err:
ep->rx_pointer = (entry + 1) & (RX_QUEUE_ENTRIES - 1);
processed++;
dev->quota--;
(*budget)--;
}
if (processed) {
@ -284,7 +279,7 @@ err:
wrw(ep, REG_RXSTSENQ, processed);
}
return !rx_done;
return processed;
}
static int ep93xx_have_more_rx(struct ep93xx_priv *ep)
@ -293,36 +288,32 @@ static int ep93xx_have_more_rx(struct ep93xx_priv *ep)
return !!((rstat->rstat0 & RSTAT0_RFP) && (rstat->rstat1 & RSTAT1_RFP));
}
static int ep93xx_poll(struct net_device *dev, int *budget)
static int ep93xx_poll(struct napi_struct *napi, int budget)
{
struct ep93xx_priv *ep = netdev_priv(dev);
/*
* @@@ Have to stop polling if device is downed while we
* are polling.
*/
struct ep93xx_priv *ep = container_of(napi, struct ep93xx_priv, napi);
struct net_device *dev = ep->dev;
int rx = 0;
poll_some_more:
if (ep93xx_rx(dev, budget))
return 1;
rx = ep93xx_rx(dev, rx, budget);
if (rx < budget) {
int more = 0;
netif_rx_complete(dev);
spin_lock_irq(&ep->rx_lock);
wrl(ep, REG_INTEN, REG_INTEN_TX | REG_INTEN_RX);
if (ep93xx_have_more_rx(ep)) {
wrl(ep, REG_INTEN, REG_INTEN_TX);
wrl(ep, REG_INTSTSP, REG_INTSTS_RX);
spin_lock_irq(&ep->rx_lock);
__netif_rx_complete(dev, napi);
wrl(ep, REG_INTEN, REG_INTEN_TX | REG_INTEN_RX);
if (ep93xx_have_more_rx(ep)) {
wrl(ep, REG_INTEN, REG_INTEN_TX);
wrl(ep, REG_INTSTSP, REG_INTSTS_RX);
more = 1;
}
spin_unlock_irq(&ep->rx_lock);
if (netif_rx_reschedule(dev, 0))
if (more && netif_rx_reschedule(dev, napi))
goto poll_some_more;
return 0;
}
spin_unlock_irq(&ep->rx_lock);
return 0;
return rx;
}
static int ep93xx_xmit(struct sk_buff *skb, struct net_device *dev)
@ -426,9 +417,9 @@ static irqreturn_t ep93xx_irq(int irq, void *dev_id)
if (status & REG_INTSTS_RX) {
spin_lock(&ep->rx_lock);
if (likely(__netif_rx_schedule_prep(dev))) {
if (likely(__netif_rx_schedule_prep(dev, &ep->napi))) {
wrl(ep, REG_INTEN, REG_INTEN_TX);
__netif_rx_schedule(dev);
__netif_rx_schedule(dev, &ep->napi);
}
spin_unlock(&ep->rx_lock);
}
@ -648,7 +639,10 @@ static int ep93xx_open(struct net_device *dev)
dev->dev_addr[4], dev->dev_addr[5]);
}
napi_enable(&ep->napi);
if (ep93xx_start_hw(dev)) {
napi_disable(&ep->napi);
ep93xx_free_buffers(ep);
return -EIO;
}
@ -662,6 +656,7 @@ static int ep93xx_open(struct net_device *dev)
err = request_irq(ep->irq, ep93xx_irq, IRQF_SHARED, dev->name, dev);
if (err) {
napi_disable(&ep->napi);
ep93xx_stop_hw(dev);
ep93xx_free_buffers(ep);
return err;
@ -678,6 +673,7 @@ static int ep93xx_close(struct net_device *dev)
{
struct ep93xx_priv *ep = netdev_priv(dev);
napi_disable(&ep->napi);
netif_stop_queue(dev);
wrl(ep, REG_GIINTMSK, 0);
@ -788,14 +784,12 @@ struct net_device *ep93xx_dev_alloc(struct ep93xx_eth_data *data)
dev->get_stats = ep93xx_get_stats;
dev->ethtool_ops = &ep93xx_ethtool_ops;
dev->poll = ep93xx_poll;
dev->hard_start_xmit = ep93xx_xmit;
dev->open = ep93xx_open;
dev->stop = ep93xx_close;
dev->do_ioctl = ep93xx_ioctl;
dev->features |= NETIF_F_SG | NETIF_F_HW_CSUM;
dev->weight = 64;
return dev;
}
@ -847,6 +841,8 @@ static int ep93xx_eth_probe(struct platform_device *pdev)
goto err_out;
}
ep = netdev_priv(dev);
ep->dev = dev;
netif_napi_add(dev, &ep->napi, ep93xx_poll, 64);
platform_set_drvdata(pdev, dev);

48
drivers/net/b44.c
View file

@ -848,10 +848,11 @@ static int b44_rx(struct b44 *bp, int budget)
return received;
}
static int b44_poll(struct net_device *netdev, int *budget)
static int b44_poll(struct napi_struct *napi, int budget)
{
struct b44 *bp = netdev_priv(netdev);
int done;
struct b44 *bp = container_of(napi, struct b44, napi);
struct net_device *netdev = bp->dev;
int work_done;
spin_lock_irq(&bp->lock);
@ -862,22 +863,9 @@ static int b44_poll(struct net_device *netdev, int *budget)
}
spin_unlock_irq(&bp->lock);
done = 1;
if (bp->istat & ISTAT_RX) {
int orig_budget = *budget;
int work_done;
if (orig_budget > netdev->quota)
orig_budget = netdev->quota;
work_done = b44_rx(bp, orig_budget);
*budget -= work_done;
netdev->quota -= work_done;
if (work_done >= orig_budget)
done = 0;
}
work_done = 0;
if (bp->istat & ISTAT_RX)
work_done += b44_rx(bp, budget);
if (bp->istat & ISTAT_ERRORS) {
unsigned long flags;
@ -888,15 +876,15 @@ static int b44_poll(struct net_device *netdev, int *budget)
b44_init_hw(bp, B44_FULL_RESET_SKIP_PHY);
netif_wake_queue(bp->dev);
spin_unlock_irqrestore(&bp->lock, flags);
done = 1;
work_done = 0;
}
if (done) {
netif_rx_complete(netdev);
if (work_done < budget) {
netif_rx_complete(netdev, napi);
b44_enable_ints(bp);
}
return (done ? 0 : 1);
return work_done;
}
static irqreturn_t b44_interrupt(int irq, void *dev_id)
@ -924,13 +912,13 @@ static irqreturn_t b44_interrupt(int irq, void *dev_id)
goto irq_ack;
}
if (netif_rx_schedule_prep(dev)) {
if (netif_rx_schedule_prep(dev, &bp->napi)) {
/* NOTE: These writes are posted by the readback of
* the ISTAT register below.
*/
bp->istat = istat;
__b44_disable_ints(bp);
__netif_rx_schedule(dev);
__netif_rx_schedule(dev, &bp->napi);
} else {
printk(KERN_ERR PFX "%s: Error, poll already scheduled\n",
dev->name);
@ -1420,6 +1408,8 @@ static int b44_open(struct net_device *dev)
if (err)
goto out;
napi_enable(&bp->napi);
b44_init_rings(bp);
b44_init_hw(bp, B44_FULL_RESET);
@ -1427,6 +1417,7 @@ static int b44_open(struct net_device *dev)
err = request_irq(dev->irq, b44_interrupt, IRQF_SHARED, dev->name, dev);
if (unlikely(err < 0)) {
napi_disable(&bp->napi);
b44_chip_reset(bp);
b44_free_rings(bp);
b44_free_consistent(bp);
@ -1609,7 +1600,7 @@ static int b44_close(struct net_device *dev)
netif_stop_queue(dev);
netif_poll_disable(dev);
napi_disable(&bp->napi);
del_timer_sync(&bp->timer);
@ -1626,8 +1617,6 @@ static int b44_close(struct net_device *dev)
free_irq(dev->irq, dev);
netif_poll_enable(dev);
if (bp->flags & B44_FLAG_WOL_ENABLE) {
b44_init_hw(bp, B44_PARTIAL_RESET);
b44_setup_wol(bp);
@ -2194,8 +2183,7 @@ static int __devinit b44_init_one(struct pci_dev *pdev,
dev->set_mac_address = b44_set_mac_addr;
dev->do_ioctl = b44_ioctl;
dev->tx_timeout = b44_tx_timeout;
dev->poll = b44_poll;
dev->weight = 64;
netif_napi_add(dev, &bp->napi, b44_poll, 64);
dev->watchdog_timeo = B44_TX_TIMEOUT;
#ifdef CONFIG_NET_POLL_CONTROLLER
dev->poll_controller = b44_poll_controller;

2
drivers/net/b44.h
View file

@ -423,6 +423,8 @@ struct b44 {
struct ring_info *rx_buffers;
struct ring_info *tx_buffers;
struct napi_struct napi;
u32 dma_offset;
u32 flags;
#define B44_FLAG_B0_ANDLATER 0x00000001

47
drivers/net/bnx2.c
View file

@ -428,7 +428,7 @@ bnx2_netif_stop(struct bnx2 *bp)
{
bnx2_disable_int_sync(bp);
if (netif_running(bp->dev)) {
netif_poll_disable(bp->dev);
napi_disable(&bp->napi);
netif_tx_disable(bp->dev);
bp->dev->trans_start = jiffies; /* prevent tx timeout */
}
@ -440,7 +440,7 @@ bnx2_netif_start(struct bnx2 *bp)
if (atomic_dec_and_test(&bp->intr_sem)) {
if (netif_running(bp->dev)) {
netif_wake_queue(bp->dev);
netif_poll_enable(bp->dev);
napi_enable(&bp->napi);
bnx2_enable_int(bp);
}
}
@ -2551,7 +2551,7 @@ bnx2_msi(int irq, void *dev_instance)
if (unlikely(atomic_read(&bp->intr_sem) != 0))
return IRQ_HANDLED;
netif_rx_schedule(dev);
netif_rx_schedule(dev, &bp->napi);
return IRQ_HANDLED;
}
@ -2568,7 +2568,7 @@ bnx2_msi_1shot(int irq, void *dev_instance)
if (unlikely(atomic_read(&bp->intr_sem) != 0))
return IRQ_HANDLED;
netif_rx_schedule(dev);
netif_rx_schedule(dev, &bp->napi);
return IRQ_HANDLED;
}
@ -2604,9 +2604,9 @@ bnx2_interrupt(int irq, void *dev_instance)
if (unlikely(atomic_read(&bp->intr_sem) != 0))
return IRQ_HANDLED;
if (netif_rx_schedule_prep(dev)) {
if (netif_rx_schedule_prep(dev, &bp->napi)) {
bp->last_status_idx = sblk->status_idx;
__netif_rx_schedule(dev);
__netif_rx_schedule(dev, &bp->napi);
}
return IRQ_HANDLED;
@ -2632,12 +2632,14 @@ bnx2_has_work(struct bnx2 *bp)
}
static int
bnx2_poll(struct net_device *dev, int *budget)
bnx2_poll(struct napi_struct *napi, int budget)
{
struct bnx2 *bp = netdev_priv(dev);
struct bnx2 *bp = container_of(napi, struct bnx2, napi);
struct net_device *dev = bp->dev;
struct status_block *sblk = bp->status_blk;
u32 status_attn_bits = sblk->status_attn_bits;
u32 status_attn_bits_ack = sblk->status_attn_bits_ack;
int work_done = 0;
if ((status_attn_bits & STATUS_ATTN_EVENTS) !=
(status_attn_bits_ack & STATUS_ATTN_EVENTS)) {
@ -2655,23 +2657,14 @@ bnx2_poll(struct net_device *dev, int *budget)
if (bp->status_blk->status_tx_quick_consumer_index0 != bp->hw_tx_cons)
bnx2_tx_int(bp);
if (bp->status_blk->status_rx_quick_consumer_index0 != bp->hw_rx_cons) {
int orig_budget = *budget;
int work_done;
if (orig_budget > dev->quota)
orig_budget = dev->quota;
work_done = bnx2_rx_int(bp, orig_budget);
*budget -= work_done;
dev->quota -= work_done;
}
if (bp->status_blk->status_rx_quick_consumer_index0 != bp->hw_rx_cons)
work_done = bnx2_rx_int(bp, budget);
bp->last_status_idx = bp->status_blk->status_idx;
rmb();
if (!bnx2_has_work(bp)) {
netif_rx_complete(dev);
netif_rx_complete(dev, napi);
if (likely(bp->flags & USING_MSI_FLAG)) {
REG_WR(bp, BNX2_PCICFG_INT_ACK_CMD,
BNX2_PCICFG_INT_ACK_CMD_INDEX_VALID |
@ -2686,10 +2679,9 @@ bnx2_poll(struct net_device *dev, int *budget)
REG_WR(bp, BNX2_PCICFG_INT_ACK_CMD,
BNX2_PCICFG_INT_ACK_CMD_INDEX_VALID |
bp->last_status_idx);
return 0;
}
return 1;
return work_done;
}
/* Called with rtnl_lock from vlan functions and also netif_tx_lock
@ -5039,6 +5031,8 @@ bnx2_open(struct net_device *dev)
if (rc)
return rc;
napi_enable(&bp->napi);
if ((bp->flags & MSI_CAP_FLAG) && !disable_msi) {
if (pci_enable_msi(bp->pdev) == 0) {
bp->flags |= USING_MSI_FLAG;
@ -5049,6 +5043,7 @@ bnx2_open(struct net_device *dev)
rc = bnx2_request_irq(bp);
if (rc) {
napi_disable(&bp->napi);
bnx2_free_mem(bp);
return rc;
}
@ -5056,6 +5051,7 @@ bnx2_open(struct net_device *dev)
rc = bnx2_init_nic(bp);
if (rc) {
napi_disable(&bp->napi);
bnx2_free_irq(bp);
bnx2_free_skbs(bp);
bnx2_free_mem(bp);
@ -5088,6 +5084,7 @@ bnx2_open(struct net_device *dev)
rc = bnx2_request_irq(bp);
if (rc) {
napi_disable(&bp->napi);
bnx2_free_skbs(bp);
bnx2_free_mem(bp);
del_timer_sync(&bp->timer);
@ -5301,7 +5298,8 @@ bnx2_close(struct net_device *dev)
while (bp->in_reset_task)
msleep(1);
bnx2_netif_stop(bp);
bnx2_disable_int_sync(bp);
napi_disable(&bp->napi);
del_timer_sync(&bp->timer);
if (bp->flags & NO_WOL_FLAG)
reset_code = BNX2_DRV_MSG_CODE_UNLOAD_LNK_DN;
@ -6858,11 +6856,10 @@ bnx2_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
#ifdef BCM_VLAN
dev->vlan_rx_register = bnx2_vlan_rx_register;
#endif
dev->poll = bnx2_poll;
dev->ethtool_ops = &bnx2_ethtool_ops;
dev->weight = 64;
bp = netdev_priv(dev);
netif_napi_add(dev, &bp->napi, bnx2_poll, 64);
#if defined(HAVE_POLL_CONTROLLER) || defined(CONFIG_NET_POLL_CONTROLLER)
dev->poll_controller = poll_bnx2;

2
drivers/net/bnx2.h
View file

@ -6473,6 +6473,8 @@ struct bnx2 {
struct net_device *dev;
struct pci_dev *pdev;
struct napi_struct napi;
atomic_t intr_sem;
struct status_block *status_blk;

40
drivers/net/cassini.c
View file

@ -2485,7 +2485,7 @@ static irqreturn_t cas_interruptN(int irq, void *dev_id)
if (status & INTR_RX_DONE_ALT) { /* handle rx separately */
#ifdef USE_NAPI
cas_mask_intr(cp);
netif_rx_schedule(dev);
netif_rx_schedule(dev, &cp->napi);
#else
cas_rx_ringN(cp, ring, 0);
#endif
@ -2536,7 +2536,7 @@ static irqreturn_t cas_interrupt1(int irq, void *dev_id)
if (status & INTR_RX_DONE_ALT) { /* handle rx separately */
#ifdef USE_NAPI
cas_mask_intr(cp);
netif_rx_schedule(dev);
netif_rx_schedule(dev, &cp->napi);
#else
cas_rx_ringN(cp, 1, 0);
#endif
@ -2592,7 +2592,7 @@ static irqreturn_t cas_interrupt(int irq, void *dev_id)
if (status & INTR_RX_DONE) {
#ifdef USE_NAPI
cas_mask_intr(cp);
netif_rx_schedule(dev);
netif_rx_schedule(dev, &cp->napi);
#else
cas_rx_ringN(cp, 0, 0);
#endif
@ -2607,9 +2607,10 @@ static irqreturn_t cas_interrupt(int irq, void *dev_id)
#ifdef USE_NAPI
static int cas_poll(struct net_device *dev, int *budget)
static int cas_poll(struct napi_struct *napi, int budget)
{
struct cas *cp = netdev_priv(dev);
struct cas *cp = container_of(napi, struct cas, napi);
struct net_device *dev = cp->dev;
int i, enable_intr, todo, credits;
u32 status = readl(cp->regs + REG_INTR_STATUS);
unsigned long flags;
@ -2620,20 +2621,18 @@ static int cas_poll(struct net_device *dev, int *budget)
/* NAPI rx packets. we spread the credits across all of the
* rxc rings
*/
todo = min(*budget, dev->quota);
/* to make sure we're fair with the work we loop through each
*
* to make sure we're fair with the work we loop through each
* ring N_RX_COMP_RING times with a request of
* todo / N_RX_COMP_RINGS
* budget / N_RX_COMP_RINGS
*/
enable_intr = 1;
credits = 0;
for (i = 0; i < N_RX_COMP_RINGS; i++) {
int j;
for (j = 0; j < N_RX_COMP_RINGS; j++) {
credits += cas_rx_ringN(cp, j, todo / N_RX_COMP_RINGS);
if (credits >= todo) {
credits += cas_rx_ringN(cp, j, budget / N_RX_COMP_RINGS);
if (credits >= budget) {
enable_intr = 0;
goto rx_comp;
}
@ -2641,9 +2640,6 @@ static int cas_poll(struct net_device *dev, int *budget)
}
rx_comp:
*budget -= credits;
dev->quota -= credits;
/* final rx completion */
spin_lock_irqsave(&cp->lock, flags);
if (status)
@ -2674,11 +2670,10 @@ rx_comp:
#endif
spin_unlock_irqrestore(&cp->lock, flags);
if (enable_intr) {
netif_rx_complete(dev);
netif_rx_complete(dev, napi);
cas_unmask_intr(cp);
return 0;
}
return 1;
return credits;
}
#endif
@ -4351,6 +4346,9 @@ static int cas_open(struct net_device *dev)
goto err_spare;
}
#ifdef USE_NAPI
napi_enable(&cp->napi);
#endif
/* init hw */
cas_lock_all_save(cp, flags);
cas_clean_rings(cp);
@ -4376,6 +4374,9 @@ static int cas_close(struct net_device *dev)
unsigned long flags;
struct cas *cp = netdev_priv(dev);
#ifdef USE_NAPI
napi_enable(&cp->napi);
#endif
/* Make sure we don't get distracted by suspend/resume */
mutex_lock(&cp->pm_mutex);
@ -5062,8 +5063,7 @@ static int __devinit cas_init_one(struct pci_dev *pdev,
dev->watchdog_timeo = CAS_TX_TIMEOUT;
dev->change_mtu = cas_change_mtu;
#ifdef USE_NAPI
dev->poll = cas_poll;
dev->weight = 64;
netif_napi_add(dev, &cp->napi, cas_poll, 64);
#endif
#ifdef CONFIG_NET_POLL_CONTROLLER
dev->poll_controller = cas_netpoll;

2
drivers/net/cassini.h
View file

@ -4280,6 +4280,8 @@ struct cas {
int rx_cur[N_RX_COMP_RINGS], rx_new[N_RX_COMP_RINGS];
int rx_last[N_RX_DESC_RINGS];
struct napi_struct napi;
/* Set when chip is actually in operational state
* (ie. not power managed) */
int hw_running;

1
drivers/net/chelsio/common.h
View file

@ -278,6 +278,7 @@ struct adapter {
struct peespi *espi;
struct petp *tp;
struct napi_struct napi;
struct port_info port[MAX_NPORTS];
struct delayed_work stats_update_task;
struct timer_list stats_update_timer;

9
drivers/net/chelsio/cxgb2.c
View file

@ -255,8 +255,11 @@ static int cxgb_open(struct net_device *dev)
struct adapter *adapter = dev->priv;
int other_ports = adapter->open_device_map & PORT_MASK;
if (!adapter->open_device_map && (err = cxgb_up(adapter)) < 0)
napi_enable(&adapter->napi);
if (!adapter->open_device_map && (err = cxgb_up(adapter)) < 0) {
napi_disable(&adapter->napi);
return err;
}
__set_bit(dev->if_port, &adapter->open_device_map);
link_start(&adapter->port[dev->if_port]);
@ -274,6 +277,7 @@ static int cxgb_close(struct net_device *dev)
struct cmac *mac = p->mac;
netif_stop_queue(dev);
napi_disable(&adapter->napi);
mac->ops->disable(mac, MAC_DIRECTION_TX | MAC_DIRECTION_RX);
netif_carrier_off(dev);
@ -1113,8 +1117,7 @@ static int __devinit init_one(struct pci_dev *pdev,
netdev->poll_controller = t1_netpoll;
#endif
#ifdef CONFIG_CHELSIO_T1_NAPI
netdev->weight = 64;
netdev->poll = t1_poll;
netif_napi_add(netdev, &adapter->napi, t1_poll, 64);
#endif
SET_ETHTOOL_OPS(netdev, &t1_ethtool_ops);

29
drivers/net/chelsio/sge.c
View file

@ -1620,23 +1620,20 @@ static int process_pure_responses(struct adapter *adapter)
* or protection from interrupts as data interrupts are off at this point and
* other adapter interrupts do not interfere.
*/
int t1_poll(struct net_device *dev, int *budget)
int t1_poll(struct napi_struct *napi, int budget)
{
struct adapter *adapter = dev->priv;
struct adapter *adapter = container_of(napi, struct adapter, napi);
struct net_device *dev = adapter->port[0].dev;
int work_done;
work_done = process_responses(adapter, min(*budget, dev->quota));
*budget -= work_done;
dev->quota -= work_done;
if (unlikely(responses_pending(adapter)))
return 1;
netif_rx_complete(dev);
writel(adapter->sge->respQ.cidx, adapter->regs + A_SG_SLEEPING);
return 0;
work_done = process_responses(adapter, budget);
if (likely(!responses_pending(adapter))) {
netif_rx_complete(dev, napi);
writel(adapter->sge->respQ.cidx,
adapter->regs + A_SG_SLEEPING);
}
return work_done;
}
/*
@ -1653,13 +1650,13 @@ irqreturn_t t1_interrupt(int irq, void *data)
writel(F_PL_INTR_SGE_DATA, adapter->regs + A_PL_CAUSE);
if (__netif_rx_schedule_prep(dev)) {
if (napi_schedule_prep(&adapter->napi)) {
if (process_pure_responses(adapter))
__netif_rx_schedule(dev);
__netif_rx_schedule(dev, &adapter->napi);
else {
/* no data, no NAPI needed */
writel(sge->respQ.cidx, adapter->regs + A_SG_SLEEPING);
netif_poll_enable(dev); /* undo schedule_prep */
napi_enable(&adapter->napi); /* undo schedule_prep */
}
}
return IRQ_HANDLED;

Some files were not shown because too many files have changed in this diff Show more