[stgt] [PATCH 2/2] dynamically link libibverbs and librdma

[Pete Wyckoff]

dorons at voltaire.com wrote on Wed, 08 Oct 2008 11:19 +0200:
> Pete Wyckoff wrote:
> > dorons at Voltaire.COM wrote on Thu, 02 Oct 2008 13:30 +0300:
> >> dynamically link libibverbs and librdma for using
> >> stgt without having userspace IB (e.g tcp mode).
> > 
> > A couple of comments.
> > 
> >> @@ -1,4 +1,6 @@
> >>  mandir = /usr/share/man
> >> +libdir = /usr/lib64
> >> +bindir = /usr/sbin
> > 
> > Fine for most 64-bit machines, not sure if we are worried about others.
> > 
> >>  tgtd: $(TGTD_OBJS)
> >> -       $(CC) $^ -o $@ $(LIBS)
> >> +       $(CC) -Xlinker -E $^ -o $@ $(LIBS)
> > 
> > You may need to do something like:
> > 
> >     $(LD) -o $@ -Wl,-whole-archive libtgtrdma.so -Wl,-no-whole-archive $(LIBS)
> > 
> > to make sure the constructor functions in the .so get called properly.
> 
> I've tried to use:
> $(CC) -o $@ -Wl,-whole-archive libtgtrdma.so -Wl,-no-whole-archive $(LIBS)
> but I received the following errors:
> libtgtrdma.so: undefined reference to `ibv_ack_cq_events'
> libtgtrdma.so: undefined reference to `rdma_reject'
...

LIBS had better contain -libverbs and -lrdmacm.

> >> +$(SO_LIBS): $(ISER_OBJS)
> >> +       rm -f $@ $(SO_NAME)
> >> +       $(LD) -shared -soname $(SO_LIBS) -o $(SO_LIBS) $(ISER_OBJS)
> >> +       ln -s $(SO_LIBS) $(SO_NAME)
> >> +
> > 
> > You must put -lrdmacm -libverbs on this $(LD) line.  I learned that
> > the hard way once or twice.  Else the symbol versions don't work out
> > at runtime.
> > 
> 
> As far as I've seen it works without the -lrdmacm -libverbs.
> Adding this causes the rpm be depended on libibverbs and librdmacm
> and this is exactly what we wanted to avoid from.

I have been explicitly bitten by this problem in the context of
these IB libraries.  Read the definitive reference
    http://people.redhat.com/drepper/dsohowto.pdf
for the reasons why.  In short, if the linker creating the DSO
doesn't have access to all the symbols, it cannot assign versions.
Thus at runtime, when your DSO is pulled into the application,
versions will be assigned.  The rules of symbol versioning will
cause the _earliest_ ones to be picked, or in our case the
IBVERBS_1.0 ones, which is definitely not what you want.  It causes
errors that are very difficult to debug.

Please send out your updated patch.  I'll see if I can fix it up.

		-- Pete
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html