Andys Dynamic Link Library routines

Download from http://www.nyangau.org/dll/dll.zip.

Introduction

Many operating systems provide a form of dynamic linking or shared libraries.

There are two basic modes of operation :-

  1. Run time dynamic linking to libraries whose names are fixed at application compile/link time.
  2. Run time dynamic linking to libraries whose names are determined at application run time.

The first case isn't particularly interesting, as far as this document is concerned. All it really provides is a way for applications to share executable code and data. The second case is much more interesting as it allows for peices of programs to be supplied and/or developed seperately and later glued together. From here on onwards, we shall only discuss the latter case.

As an application developer, if you enable your application so that it can load a user specified DLL and call specific entrypoints within it, you allow your clients to extend the capabilities of your program to their needs.

Andys Binary Folding Editor, or BE for short, is a good example of an application which can have its power significantly enhanced by the user providing their own DLL(s). BE is an editor which normally edits files, but by providing a DLL, BE can edit data in any address space the user chooses. This can include the memory space of an embedded processor, perhaps via a JTAG debug connector, effectively turning BE into the data navigation half of a debugger!

As a subsystem developer, if you package up your subsystem so that it is a DLL (or a regular statically linked library), you enable other programs to take advantage of your code.

The manner in which dynamic linking / shared-libraries are supported in each operating system varys wildly. So it can be quite difficult to write platform independant code which supports dynamic linking.

The set of routines in explained in this document allow you to write DLLs and applications in a largely platform independant way. The mechanics of how DLLs are implemented on each platform is concealed from the programmer. However, as application developers may want other parties to write DLLs which can be loaded by them (on assorted platforms), sufficient information on how to do it is given in this document. Sample makefiles are included.

Operating systems

How are DLLs implemented on the various supported platforms :-

16 bit DOS
DLLs are actually a form of TSR. When the application 'loads' the DLL, it does so by running it, thus leaving its executable code and data still resident. The DLL is found by looking in the current directory, and along the PATH environment variable for a file called DLLNAME.DLL. The Microsoft C 6.00 C compiler can be used to compile DLLs and programs which load them. Only the Large memory model is supported.
32 bit DOS
This is the CauseWay DOS protect mode extender environment. These DLL routines make use of the CauseWay LoadLibrary and GetProcAddress routines. The DLL is found by looking in the current directory, and along the PATH environment variable for a file called DLLNAME.DLL. The Watcom C/C++ (version 10.6 or later) and CauseWay DOS extender (version 1.3 or later) can be used to compile DLLs and programs which load them. Despite using a Watcom compiler, I personally still use the Microsoft / IBM NMAKE (rather than WMAKE), as I prefer it.
16 bit OS/2
Here we use the DosLoadModule and DosGetProcAddr APIs. OS/2 looks along the LIBPATH special environment variable for DLLNAME.DLL. You may need to add the current directory to the LIBPATH, or ensure your DLLs are in a directory on the LIBPATH. The Microsoft C 6.00 C compiler can be used to compile DLLs and programs which use them. These programs and DLLs should be compiled multi-threaded, in accordance with the sample makefile supplied here.
32 bit OS/2
Here we use the DosLoadModule and DosQueryProcAddr APIs. OS/2 looks along the LIBPATH special environment variable for DLLNAME.DLL. Pretty much every 32 bit OS/2 system I've seen has had the current directory in the LIBPATH. DLLNAME should be no more than 8 characters long. The IBM VisualAge C++ compiler (version 3.0, with fixpaks CTC306, CTD302 and CTU304, or later recommended) can be used to compile DLLs and programs which load them. DLLs and programs which use them should be compiled multi-threaded.
Win32
Here we use the Win32 LoadLibrary and GetProcAddress APIs. The MS Visual C++ compiler (version 4.2 or 5.0, or later) can be used to compile DLLs and programs which load them. DLLs and programs which use them should be compiled multi-threaded.
AIX
AIX is particularly complicated. First, the operating system supports a proprietry loading system known as load and unload. However, when using C++, one must use the loadAndInit and terminateAndUnload equivelents, to ensure static constructors and destructors are called, and unfortunately these routines are a part of the xlC compiler product, and so you can't use them if you are using the gcc/g++ compilers. Such shared libraries are named dllname. These calls are all you get in AIX 4.1.5 and before. On AIX 4.2.1 and later, standard support via dlopen is provided, but the instant you use it, you render your program unusable on earlier versions of AIX. When using dlopen, the shared libraries are named dllname.so and are found by looking along the LIBPATH environment variable. dlopen can be used from xlc/xlC or from gcc/g++. In this code, we favor the use of dlopen, but include legacy support for load style operation. One day, I'll drop support for AIX 4.1.5 and earlier, and when that day comes, I'll delete the load stuff from here.
Linux
Under Linux, the dlopen family of routines are used. The library should be named dllname.so. Linux searches along the LD_LIBRARY_PATH environment variable, in directories listed in the /etc/ld.so.cache file, and in the /usr/lib and /lib directories. Rather irritatingly, the current directory isn't automatically searched, and it isn't in most peoples LD_LIBRARY_PATH by default. Remember to set this up. The gcc C compiler (or g++ C++ compiler) (version 2.7.2.3 or later) can be used to compile DLLs or programs which load them.
HP-UX
Under HP-UX the shl_load family of routines are used. The library should be named dllname.sl. HP-UX searches along the SHLIB_PATH environment variable. The standard cc C compiler (or aCC C++ compiler) can be used to compile DLLs and programs which load them.
SunOS
Under SunOS, the dlopen family of routines are used. The library should be named dllname.so. SunOS searches along the LD_LIBRARY_PATH environment variable (and may look elsewhere). Rather irritatingly, the current directory isn't automatically searched, and it isn't in most peoples LD_LIBRARY_PATH by default. Remember to set this up. The cc C compiler (or CC C++ compiler) can be used to compile DLLs or programs which load them.
Cygwin
This is like Linux and SunOS, except that the library is named with a .dll extension.
NetWare
Under NetWare a 'DLL' is actually another NLM named with an .ndl extension.

I reserve the right to switch-over-to or preferrably add-support-for the dlopen style of dynamic linking on those UNIXes which don't currently support it widely enough, when they do.

Although this document lists one suitable compiler for each platform, other compilers may work too (with suitably changed makefiles). I list the combinations I use, and therefore know to work.

DLL implementation

When implementing a DLL, you are obliged to include the DLL implementors header :-

#include "dllimpl.h"

This essentially defines the calling conventions of the entrypoints to the DLL. These vary somewhat from operating system to operating system. Clearly it is possible to have DLL entrypoints whose calling conventions vary from those dictated by the DLL code documented here. However, deviate far from this document, and things may not work.

Entrypoints should be declared according to the following template :-

DLLEXPORT return_type DLLENTRY function(args)
  {
  ... implementation
  }

On 32 bit DOS (via CauseWay DOS extender), you are obliged to have a dummy main function in your code :-

#ifdef DOS32
int main(int term) { term=term; return 0; }
#endif

Unfortunately, 16 bit DOS and AIX don't support dynamic linking well, and so you are also obliged to include a table of entrypoints, in the following form :-

#if defined(xxDOS) || defined(AIX_LOAD)
DLL_EXPORT * __start(void)
  {
  static DLL_EXPORT exports[] =
    {
    (DLL_EP) function1, "function1",
    (DLL_EP) function2, "function2",
    ... more
    0                 , 0          ,
    };
  return exports;
  }
#endif

On NetWare, you need to have the following (assuming the DLL is called libname) :-

#ifdef NW
extern void _dll_term(void);
DLLEXPORT void DLLENTRY _libname_term(void) { _dll_term(); }
#endif

DLL usage

When writing a program which dynamically loads and uses a DLL, you are obliged to include the DLL users header :-

#include "dlluser.h"

As well as providing access to the calling convention information, this provides calls to load and unload a library of choice, and also to find an entrypoint of a given name within it :-

DLL   *dll_load(const char *name);
void   dll_unload(DLL *dll);
DLL_EP dll_entrypoint(DLL *dll, const char *name, const char *args);

A program to dynamically load and use a DLL can be as simple as :-

#include <stdio.h>
#include "dlluser.h"

int main(int argc, char *argv[])
  {
  DLL *dll;
  DLL_EP ep;
  int (DLL_EP_PTR binop_func)(int, int);
  int result;
  if ( argc < 3 )
    return 1; /* Insufficient command line arguments */
  if ( (dll = dll_load(argv[1])) == NULL )
    return 2; /* Cannot load named DLL */
  if ( (ep = dll_entrypoint(dll, argv[2], "8")) == NULL )
    { dll_unload(dll); return 3; } /* DLL does not contain named entrypoint */
  binop_func = (int (DLL_EP_PTR)(int, int)) ep;
  result = (*binop_func)(2,3);
  printf("%s(2,3)=%d\n", argv[2], result);
  dll_unload(dll);
  return 0;
  }

The example program is fed the name of the DLL and the entrypoint. It is assumed the entrypoint is a function which takes two numbers, does something to them (perhaps it adds them, multiplies them, whatever), and returns a result.

Note the need to do the dll_unload along the 'DLL does not contain named entrypoint' error path. The operating system will not always clean up properly (ie: 16 bit DOS).

The 3rd argument to dll_entrypoint is the number of bytes the entrypoint has as its parameters. This is an artifact of the Windows NT __stdcall calling convention. Usually this is simply 4 * the number of parameters (as a string). The value isn't used on any other platforms (yet).

The DLL_EP_PTR malarky is an attempt to handle the fact that function entrypoints can have a number of calling conventions on some systems (typically PC platforms), and the syntax of specifying the calling convention varies. On some compilers the calling convention modifier is written before the *, and on others, its afterwards. eg:

int (*           multiply)(int, int);      /* UNIX */
int (__stdcall * multiply)(int, int);      /* Windows, MS Visual C++ */
int (* _System   multiply)(int, int);      /* OS/2, IBM VisualAge C/C++ */

     \_________/
          \______ == DLL_EP_PTR

DLLs which use DLLs

Can a DLL use another DLLs? I guess so. I see no reason why not. I've never tried it.

Compiling and linking

The sample makefiles build a DLL and a program which loads a DLL.

16 bit DOS

CFLAGS =	-AL -Gs -W3 -FPa -DxxDOS -nologo
LFLAGS =	/NOE /NOD /EXEPACK /BATCH /NOLOGO

all:		libname.dll progname.exe

libname.dll:	libname.obj dllimpl.lib
		link $(LFLAGS) /STACK:4096 @<<
			libname
			$@
			nul
			dllimpl llibcar;
<<

libname.obj:	libname.c dllimpl.h
		cl -c $(CFLAGS) -Aw $*.c

progname.exe:	progname.obj dlluser.lib
		link $(LFLAGS) /STACK:4096 @<<
			progname
			$@
			nul
			dlluser llibcar;
<<

progname.obj:	progname.c dlluser.h dllimpl.h
		cl -c $(CFLAGS) $*.c

You may notice the need to link to dllimpl.lib. This library provides main and includes the logic to turn the executable which is output from the link step, into a TSR, capable of being invoked by the dlluser.lib calling code.

Note also, that the 16 bit DOS example uses the alternate floating point support. Floating point support does not work properly in the DLL (aka TSR) environment, because part of the process of terminating and staying resident is to terminate the floating point support in the C run time.

Finally, note the -Aw DS!=SS switch in the DLL compile.

32 bit DOS

CW =		directory where CauseWay files are

CFLAGS =	-bt=DOS -dDOS32 -oit -4r -s -w3 -zp4 -mf -zq -fr -zq

all:		libname.dll progname.exe

libname.dll:	libname.obj
		wlink @<<
			System CWDLLR
			Name $@
			File libname.obj
			Option Quiet
<<

libname.obj:	libname.c dllimpl.h
		wcc386 $(CFLAGS) -bd $*.c

progname.exe:	progname.obj dlluser.lib
		wlink @<<
			System CauseWay
			Name $@
			File progname.obj
			Library dlluser.lib
			LibFile $(CW)\dllfunc.obj,$(CW)\cwapi.obj
			Option Quiet
<<

progname.obj:	progname.c dlluser.h dllimpl.h
		wcc386 $(CFLAGS) $*.c

Remember, this is a NMAKE file, its probably not suitable for WMAKE.

16 bit OS/2

CFLAGS =	-c -Gs -W3 -DO16 -nologo
LFLAGS =	/NOD /NOI /EXEPACK /BATCH /NOLOGO

all:		libname.dll progname.exe

libname.dll:	libname.obj libname.def
		link $(LFLAGS) @<<
			libname
			$@
			nul
			llibcdll os2
			libname;
<<

libname.obj:	libname.c dllimpl.h
		cl $(CFLAGS) -ML $*.c

progname.exe:	progname.obj dlluser.lib
		link $(LFLAGS) /STACK:4096 /PMTYPE:VIO @<<
			progname
			$@
			nul
			dlluser llibcmt os2;
<<

progname.obj:	progname.c dlluser.h
		cl $(CFLAGS) -MT $*.c

We also need a module definition file libname.def :-

LIBRARY LIBNAME INITINSTANCE

PROTMODE

DATA MULTIPLE NONSHARED

EXPORTS
  ENTRYPOINT1
  ENTRYPOINT2

When compiling 16 bit DLLs using the C run time, there are two choices :-

First, have the program and the DLLs use a single common C run time in a DLL, which is typically called CRTLIB.DLL. This is problematical, especially when many programs exist in the system which do this, all of which wanting their version of the C run time.

The second approach - the one recommended here - is to have the DLL link with LLIBCDLL.LIB, which is a C run time for binding into a DLL. This has the side effect of turning the code multi-threaded, as this library is designed for multi-threaded code. It also has the side effect of making file sizes bigger, but thats not really an issue nowadays. In the above makefile, the calling program links with LLIBCMT.LIB as it is multi-threaded too. Just because a program can be multi-threaded, and is linked as such, it doesn't mean that it has to use any extra threads.

32 bit OS/2

CWARNS =	/W3 /Wcmp+cnd+dcl+ord+par+use+
CFLAGS =	/DOS2 /G4 /Gd-m+ $(CWARNS) /Q+ /Ft-
LFLAGS =	/NOI /NOLOGO

all:		libname.dll progname.exe

libname.dll:	libname.obj libname.def
		ilink $(LFLAGS) /OUT:$@ $**

libname.obj:	libname.c dllimpl.h
		icc /C+ $(CFLAGS) /Ge- $*.c

progname.exe:	progname.obj
		ilink $(LFLAGS) /PMTYPE:VIO /BASE:0x10000 /EXEPACK /OUT:$@ $**

progname.obj:	progname.c dlluser.h dllimpl.h
		icc /C+ $(CFLAGS) /Ge+ $*.c

We also need a module definition file libname.def :-

LIBRARY libname INITINSTANCE TERMINSTANCE

DATA MULTIPLE NONSHARED READWRITE

CODE PRELOAD EXECUTEREAD

EXPORTS
  entrypoint1
  entrypoint2

Win32

CFLAGS =	/DWIN32 /G4 /Gs /Oit /MT /nologo /W3 /WX
LFLAGS =	/NOLOGO /BATCH /INCREMENTAL:NO

all:		libname.dll progname.exe

libname.dll:	libname.obj
		link $(LFLAGS) /DLL $** /OUT:$@

libname.obj:	libname.c dllimpl.h
		cl /c $(CFLAGS) $*.c

progname.exe:	progname.obj
		link $(LFLAGS) $** /OUT:$@

progname.obj:	progname.c dlluser.h dllimpl.h
		cl /c $(CFLAGS) $*.c

AIX

CFLAGS =  	-DUNIX -DAIX -DAIX_DLOPEN

all:		libname.so progname

libname.so:	libname.o
		gcc -shared -Wl,-bE:libname.exp
		chmod a-x $@

libname.o:	libname.c dllimpl.h
		gcc -c $(CFLAGS) -fPIC $*.c

progname:	progname.o
		gcc -o $@ progname.o -ldl

progname.o:	progname.c dlluser.h dllimpl.h
		gcc -c $(CFLAGS) $*.c

This is a makefile suitable for use on AIX 4.2.1 or later. The shared library is created dlopen style, and the program only handles such shared libraries.

Linux

CFLAGS =	-DUNIX -DLINUX

all:		libname.so progname

libname.so:	libname.o
		gcc -shared -o $@ libname.o
		chmod a-x $@

libname.o:	libname.c dllimpl.h
		gcc -c $(CFLAGS) -fPIC $*.c

progname:	progname.o
		gcc -o $@ progname.o -ldl

progname.o:	progname.c dlluser.h dllimpl.h
		gcc -c $(CFLAGS) $*.c

-fPIC says generate position independant code. -shared says link as a shared library.

HP-UX

CFLAGS =	-DUNIX -DHP

all:		libname.sl progname

libname.sl:	libname.o
		cc -b -o $@ libname.o

libname.o:	libname.c dllimpl.h
		cc -c $(CFLAGS) +z $*.c

progname:	progname.o
		cc -o $@ -Wl,+s progname.o -ldld

progname.o:	progname.c dlluser.h dllimpl.h
		cc -c $(CFLAGS) $*.c

+z says compile small position independant code. -Wl,+s enables shared library loading to look along the SHLIB_PATH environment variable.

SunOS

Using the GNU toolchain :-

CFLAGS =	-DUNIX -DSUN

all:		libname.so progname

libname.so:	libname.o
		gcc -shared -o $@ libname.o

libname.o:	libname.c dllimpl.h
		gcc -c $(CFLAGS) -shared $*.c

progname:	progname.o
		gcc -o $@ progname.o

progname.o:	progname.c dlluser.h dllimpl.h
		gcc -c $(CFLAGS) $*.c

Or if using the standard (Forte) Toolchain :-

CFLAGS =	-DUNIX -DSUN

all:		libname.so progname

libname.so:	libname.o
		cc -G -Kpic -o $@ libname.o

libname.o:	libname.c dllimpl.h
		cc -c $(CFLAGS) -Kpic $*.c

progname:	progname.o
		cc -o $@ progname.o -ldl

progname.o:	progname.c dlluser.h dllimpl.h
		cc -c $(CFLAGS) $*.c

-fPIC says generate position independant code. -shared says link as a shared library.

Cygwin

CFLAGS =	-DUNIX -DCYGWIN

all:		libname.so progname

libname.dll:	libname.o
		gcc -shared -o $@ libname.o

libname.o:	libname.c dllimpl.h
		gcc -c $(CFLAGS) -shared $*.c

progname:	progname.o
		gcc -o $@ progname.o

progname.o:	progname.c dlluser.h dllimpl.h
		gcc -c $(CFLAGS) $*.c

NetWare

#
# Dynamic Link Library
# NetWare
# Watcom C/C++ 10.6
#

CFLAGS =	/s /fpi87 /mf /zfp /zgp /zl /zq \
		/4s /fpd /wx /bt=NETWARE /DNW

.c.obj:
		wcc386 $(CFLAGS) $*.c


all:		libname.ndl progname.nlm

# Sample DLL

libname.ndl:	libname.obj dllimpl.obj
		wlink @<<
			Format Novell NLM '$@'
			Name $@
			Option Quiet
			Option Map
			Option ScreenName 'System Console'
			Option ThreadName '$@'
			Debug Novell
			Module clib, mathlib
			File libname.obj
			Library dllimpl.lib
			Library $(WATCOM)\lib386\math387s.lib
			Library $(WATCOM)\lib386\noemu387.lib
			Library $(WATCOM)\lib386\netware\clib3s.lib
			Import @$(WATCOM)\novi\clib.imp
			Export _libname_term
			Export entrypoint1, entrypoint2
<<

libname.obj:	libname.c dllimpl.h

# Sample program using a DLL

progname.nlm:	progname.obj dlluser.lib
		wlink @<<
			Format Novell NLM '$@'
			Name $@
			Option Quiet
			Option Map
			Option ScreenName '$@'
			Option ThreadName '$@'
			Option Stack = 0x1c000
			Debug Novell
			Module clib, mathlib
			File progname.obj
			Library dlluser.lib
			Library $(WATCOM)\lib386\math387s.lib
			Library $(WATCOM)\lib386\noemu387.lib
			Library $(WATCOM)\lib386\netware\clib3s.lib
			Import @$(WATCOM)\novi\clib.imp
<<

progname.obj:	progname.c dlluser.h dllimpl.h

C++ considerations

The template code above assumes the use of C (rather than C++). On those platforms that allow it, C++ will work, providing care is taken to avoid (or cope with) the name mangling.

Another reason to avoid C++ interfaces is that the name mangling applied varies from compiler to compiler.

Obviously, there would need to be changes to the makefiles to invoke C++ compilers and linkers.

On 32 bit DOS, when compiling a C++ DLL, the following line must be added to the wlink script :-

			Library %watcom%\lib386\plib3r.lib

If you don't explicitly reference plib3r.lib and the C++ code uses operator new then its multithreaded equivelent gets dragged in (which causes link problems, as DOS is supposed to be single threaded).

Combinations of C and C++ should be thought through carefully :-

Some makefile info

Just in case it isn't obvious :-

$@ means the target of the rule (ie: what is being made).

$* means 'the first filename, without file extension, after the colon in the make rule'.

$** means all the filenames to the right of the colon (ie: all the files required to make the target). Doesn't seem to work on UNIX versions of make.

The sample UNIX makefile supplied with this DLL package is suitable for Linux, AIX, HP-UX and SunOS. However, UNIX makefiles have no standard convention for ifdefs. The supplied makefile uses a subset of the GNU make style of ifdefs. So you can just type :-

make HP=1

By default, the makefiles assume LINUX.

Those systems without GNU make can use the minimal Make Pre-Processor (called mpp) to create a makefile suitable for them. This script is available from my home page. eg:

mpp makefile SUN > makefile.SUN
make -f makefile.SUN

Writing cross-platform DLLs

Writing DLLs that work, or work identically, on a wide variety of platforms is not always a straightforward task.

The main gotcha is that in some environments, the executable program and DLLs each have their own 'run time' (eg: 16 bit DOS, 32 bit DOS, 16 bit OS/2, 32 bit OS/2, Win32). Typically this run time is the data used by the C and C++ run time libraries. In other systems, the C run time can be independant from the the program. There may be a single instance of C run time data for the whole program (eg: Linux with libc.so).

So this means you can get into trouble if you design your DLL interfaces such that a DLL mallocs and the program frees it (or vice-versa). This can be quite difficult to avoid if you are using C++, with new and delete. Beware DLL interfaces with objects which can delete themselves. Watch for this on 32 bit OS/2 and Win32.

Similarly, file handles opened by an application may not be readable by code in a DLL (or vice-versa). This is true on Win32. So if a DLL is required to do file I/O, then it must either do the open, or the application must deliver a file handle, and callback functions which invoke the applications versions of read and write etc.. Similarly for fopen level I/O, and also for C++ streams.

In some environments, floating point support in a DLL may not work (16 bit DOS), or may be inefficient (16 bit OS/2).

Irritations

The fact that all UNIXes don't support dlopen, means I must code up various similar but different implementations for each UNIX.

The fact that despite AIX 4.2.1 and later support dlopen, earlier versions such as AIX 4.1.5 don't.

Why does the Win32 __stdcall calling convention need to mangle its function names?

Copying

This DLL module, including its source code and sample programs are public domain. Caveat Emptor.


This documentation was written by the DLL module author, Andy Key
andy.z.key@googlemail.com