[stgt] iser Issue, cannot allocate memory

[John Francis]

Hi Roi,

I deleted the targets and performed discovery and it works fine now.
Even without pool_sz_mb=100.

Thanks for your great tip about installing the latest version from
source. I'll summarise that here for anyone else that is interested.
This is on CentOS 6.4 (i.e. RHEL 6.4 binary equivalent).

Tip: build the RPM on a management workstation running the same distro
as your production servers, then just distribute the binary RPM.

(I've left out $PS1 to make for easier copy and paste, this is on the
management machine as normal (non-root) user)

sudo yum groupinstall Development\ tools
sudo yum install librdmacm-devel git \
     docbook-style-xsl libaio-devel
mkdir sources
cd sources/
git clone https://github.com/fujita/tgt.git
cd tgt/
make rpm
scp rpmtop/RPMS/x86_64/scsi-target-utils-*.rpm my_storage_server:.

On the server, make sure you remove the scsi-target-utils package
before proceeding with the install.

Note: the newer version of tgtd lets you place your target
configurations in /etc/tgt/conf.d/ so if you have existing
definitions, you'll need to place them in snippet conf files in there.

On 23 July 2013 23:14, Roi Dayan <roid at mellanox.com> wrote:
> Hi John,
>
> 128MB is a forced minimum in the code for iser to work properly.
>
> You can ignore the shmget rdma pool failure message,
> It means it failed to allocate huge-pages and will revert to normal memory allocation.
> Can you show more from the log after it?
>
>
> Thanks,
> Roi
>
>
>
> -----Original Message-----
> From: John Francis [mailto:john at recompute.com.au]
> Sent: Tuesday, July 23, 2013 3:17 PM
> To: Roi Dayan
> Cc: stgt at vger.kernel.org
> Subject: Re: iser Issue, cannot allocate memory
>
> On 23 July 2013 16:59, Roi Dayan <roid at mellanox.com> wrote:
>> Hi John,
>>
>> Can you try to limit the memory pool iser is trying to allocate?
>> i.e.:
>> tgtd --iser pool_sz_mb=100
>
> Okay, I was running an older version of tgtd that didn't support that option. Thanks to Roi's advice I've upgraded to tgtd 1.0.37 and started up the server both with the --ser pool_sz_mb=100 and without.
>
> With the limit, I get:
> tgtd: iser_alloc_pool(573) shmget rdma pool sz:134217728 failed
>
> Without it, I get:
> tgtd: iser_alloc_pool(573) shmget rdma pool sz:1073741824 failed
>
> It's funny, that it looks like the minimum it tries to allocate is 128 MB, even it I use pool_sz_mb=50, it still tries to allocate 128 MB.
>
> Could this be a kernel.shmmax or kernel.shmall issue?
>
>>
>> Thanks,
>> Roi
>>
>>
>> -----Original Message-----
>> From: stgt-owner at vger.kernel.org [mailto:stgt-owner at vger.kernel.org]
>> On Behalf Of John Francis
>> Sent: Tuesday, July 23, 2013 4:32 AM
>> To: stgt at vger.kernel.org
>> Subject: iser Issue, cannot allocate memory
>>
>> Hi,
>>
>> I have an iser target configured. When I try to log in to that target
>> from another machine, it doesn't work, and the following appears in
>> /var/log/message:
>>
>> 10:54:33 stor1 tgtd: iser_device_init(3204) iser_init_rdma_buf_pool
>> failed
>> 10:54:37 stor1 tgtd: iser_init_rdma_buf_pool(589) ibv_reg_mr failed, Cannot allocate memory ...
>>
>> I've tried fiddling with various limits but to no avail. I'm struggling to figure out if the memory limit being hit is card related of host related.
>>
>> Using some Mellonox cards with ib_mthca module. IPoIB seems to be working fine. It should be noted that this error occurs on initial log in to the target.
>>
>> --
>> Kind regards,
>>
>> John Francis
>>
>> www.recompute.com.au
>> 6/38 Canterbury Rd Bankstown NSW 2200
>> T: (02) 9709 5775 ext 9150
>>
>> Sign up to Hit Refresh, our newsletter with specials, discounts and tips, or follow us on Twitter @recompute_au.
>> --
>> To unsubscribe from this list: send the line "unsubscribe stgt" in the
>> body of a message to majordomo at vger.kernel.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Kind regards,
>
> John Francis
>
> www.recompute.com.au
> 6/38 Canterbury Rd Bankstown NSW 2200
> T: (02) 9709 5775 ext 9150
>
> Sign up to Hit Refresh, our newsletter with specials, discounts and tips, or follow us on Twitter @recompute_au.



-- 
Kind regards,

John Francis

www.recompute.com.au
6/38 Canterbury Rd Bankstown NSW 2200
T: (02) 9709 5775 ext 9150

Sign up to Hit Refresh, our newsletter with specials, discounts and
tips, or follow us on Twitter @recompute_au.
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html