[Stgt-devel] iSER - one too many rubber duckies in the mempool

[Robin Humble]

Hi,

with >=2 iSER clients I'm hitting these quite a lot:
  tgtd: iscsi_rdma_malloc(1619) free list empty
  tgtd: iscsi_rdma_alloc_data_buf(1647) free list empty
     (the fn name depends on the tgtd version)
after which the initiator node is pretty much toast :-/

the code for this is in iscsi/iscsi_rdma.c -->

  static void *iscsi_rdma_alloc_data_buf(struct iscsi_connection *conn, size_t sz)
  {       
  ...
          if (list_empty(&dev->mempool_free)) {
                  /* XXX: take slow path: allocate and register */
                  eprintf("free list empty\n");
                  exit(1);
          }
  ...

which looks like an OO(rdma)M fallback that's just unimplemented at
the moment?

as a workaround I boosted:
  static int mempool_num = 192;
to 1920 which let 2 clients survive, but not the 15 or 100 that I'd
ideally like.

is dynamically adding more entries to the mempool the solution, or a
separate list of non-mempool rdma bufs, or just telling the initiator to
backoff for a while?

cheers,
robin