From: DAUBIGNE Sebastien - BOR (sebastien.daubigne@atosorigin.com)
Date: Mon Mar 08 2004 - 12:17:31 EST
Hi,
We have an IBM Shark (ESS 2105-800) connected via 8 Emulex LP9802 FC-AL
adapters (lpfc) to our Solaris 8 server, multipathed via VxVM3.2/DMP.
In its HBA configuration guide, IBM recommends to set :
- Solaris "sd_max_throttle" to ( 256 / max_number_of_LUN_per_adapter ).
- lpfc "lun-queue-depth" to 30 (default value).
As I have 16 LUN per adapter, "sd_max_throttle"=16 is less than
"lun-queue-depth".
"sd_max_throttle" is the maximum number of commands the sd driver will send
to a LUN before queuing.
"lun-queue-depth" is the maximum number of commands the lpfc driver will
send to a LUN before queuing.
The lpfc queuing algorithm is more clever than sd one, because it can
decrease the active queue depth when queue_full conditions occur, then
increase it when there has not been any queue_full since a certain amount of
time (all this tunable via dqfull-throttle-up-time/dqfull-throttle-up-inc).
As the lpfc driver sends the command after the sd driver and each one has
itw own queuing parameter, I wondered if I could just let "sd_max_throttle"
to its default value (256), and let lpfc driver manage the queuing.
So I made a typical load test, and the winner is sd_max_throttle to the
default value (256) : 20% better throughput than the recommended value (16)
!!
Now the funny thing with theses parameters is that if we set it too high,
the ESS may send QUEUE_FULL conditions, making the server retry commands,
which would definitely affect throughput.
That's why I wonder if anyone did play with both
lun-queue-depth/sd_max_throttle, and got good results leaving
sd_max_throttle to its default value, letting lpfc manage the queue with its
more clever algorithm.
Is it safe to limit the active queue size at the lpfc level only, or should
I also limit the sd level to make sure not to get queue_full conditions ?
Thanks for your feedback. I will summarize.
-- Sebastien DAUBIGNE sebastien.daubigne@atosorigin.com - +33(0)5.57.26.56.36 AtosOrigin - Infogerance/ERP/Pessac _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:28:12 EDT