Ang: Re: [Stgt-devel] Re: [Iscsitarget-devel] stgt a new version of iscsi target?

[Vladislav Bolkhovitin]

Mike Christie wrote:
>>> There is still memory and scatterlist allocations. If we are not 
>>> going to allocate all the memory for a command buffer and request 
>>> with GFP_ATOMIC (and can then run from the the HW interrupt or soft 
>>> irq) we have to pass that on to a thread. I guess there is 
>>> disagreement whether that part is a feature or bad use of GFP_ATOMIC 
>>> though so... But I just mean to say there could be a little more to do.
>>
>>
>>
>> Actually, there is the way to allocate sg vectors with buffers in SIRQ 
>> and not with GFP_ATOMIC. This is the second major improvement, which 
>> is pending in scst. I called it sgv_pool. This is a new allocator in 
>> the kernel similar to mem_pool, but it contains *complete* sg-vectors 
>> of some size with data buffers (pages). Initiator sends data requests 
>> usually with some fixed size, like 128K. After a data command 
>> completed, its sg vector will not be immediately freed, but will be 
>> kept in 
> 
> 
> We considered this, but what did you decide is the upper limit size for 
> the pool? Is it dynmaic? We also wanted something that the SCSI ULDs 
> could use for their allocations which could go up to 6 MB.

Why do you think it needs any upper limit size? Would you like also any 
upper limits on sizes of the page or slab caches? I don't see any 
difference between them and sgv_pool. Let it allocate and keep unused as 
much memory as possible until it would be notified about memory pressure 
and ask to free some. Similarly as slab does.

BTW, from our experience, initiators tend to send commands without limit 
and in case if actual SCSI device isn't so fast to serve all of them 
with the rate, which they arrive, incoming queued commands very quickly 
eat all system memory with obvious consequences. So, there must be some 
kind of IO-throttling, when after some watermark the initiators start 
receiving TASK QUEUE FULL, which calm them down. I implemented very 
stupid one: don't let more than 32 commands per session. This let scst 
survive under extreme loads. You can think about something smarter  :-)

Vlad