[stgt] iser Issue, cannot allocate memory

[Roi Dayan]

Hi John,

128MB is a forced minimum in the code for iser to work properly.

You can ignore the shmget rdma pool failure message, 
It means it failed to allocate huge-pages and will revert to normal memory allocation.
Can you show more from the log after it?


Thanks,
Roi



-----Original Message-----
From: John Francis [mailto:john at recompute.com.au] 
Sent: Tuesday, July 23, 2013 3:17 PM
To: Roi Dayan
Cc: stgt at vger.kernel.org
Subject: Re: iser Issue, cannot allocate memory

On 23 July 2013 16:59, Roi Dayan <roid at mellanox.com> wrote:
> Hi John,
>
> Can you try to limit the memory pool iser is trying to allocate?
> i.e.:
> tgtd --iser pool_sz_mb=100

Okay, I was running an older version of tgtd that didn't support that option. Thanks to Roi's advice I've upgraded to tgtd 1.0.37 and started up the server both with the --ser pool_sz_mb=100 and without.

With the limit, I get:
tgtd: iser_alloc_pool(573) shmget rdma pool sz:134217728 failed

Without it, I get:
tgtd: iser_alloc_pool(573) shmget rdma pool sz:1073741824 failed

It's funny, that it looks like the minimum it tries to allocate is 128 MB, even it I use pool_sz_mb=50, it still tries to allocate 128 MB.

Could this be a kernel.shmmax or kernel.shmall issue?

>
> Thanks,
> Roi
>
>
> -----Original Message-----
> From: stgt-owner at vger.kernel.org [mailto:stgt-owner at vger.kernel.org] 
> On Behalf Of John Francis
> Sent: Tuesday, July 23, 2013 4:32 AM
> To: stgt at vger.kernel.org
> Subject: iser Issue, cannot allocate memory
>
> Hi,
>
> I have an iser target configured. When I try to log in to that target 
> from another machine, it doesn't work, and the following appears in
> /var/log/message:
>
> 10:54:33 stor1 tgtd: iser_device_init(3204) iser_init_rdma_buf_pool 
> failed
> 10:54:37 stor1 tgtd: iser_init_rdma_buf_pool(589) ibv_reg_mr failed, Cannot allocate memory ...
>
> I've tried fiddling with various limits but to no avail. I'm struggling to figure out if the memory limit being hit is card related of host related.
>
> Using some Mellonox cards with ib_mthca module. IPoIB seems to be working fine. It should be noted that this error occurs on initial log in to the target.
>
> --
> Kind regards,
>
> John Francis
>
> www.recompute.com.au
> 6/38 Canterbury Rd Bankstown NSW 2200
> T: (02) 9709 5775 ext 9150
>
> Sign up to Hit Refresh, our newsletter with specials, discounts and tips, or follow us on Twitter @recompute_au.
> --
> To unsubscribe from this list: send the line "unsubscribe stgt" in the 
> body of a message to majordomo at vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html



--
Kind regards,

John Francis

www.recompute.com.au
6/38 Canterbury Rd Bankstown NSW 2200
T: (02) 9709 5775 ext 9150

Sign up to Hit Refresh, our newsletter with specials, discounts and tips, or follow us on Twitter @recompute_au.
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html