Resource Managers

• Introduction
• What is a resource manager?
• Resource manager architecture
• Summary

Introduction

To give QNX Neutrino a great degree of flexibility, to minimize the runtime memory requirements of the final system, and to cope with the wide variety of devices that may be found in a custom embedded system, the OS allows user-written processes to act as resource managers that can be started and stopped dynamically.

Resource managers are typically responsible for presenting an interface to various types of devices. This may involve managing actual hardware devices (like serial ports, parallel ports, network cards, and disk drives) or virtual devices (like /dev/null, a network filesystem, and pseudo-ttys).

In other operating systems, this functionality is traditionally associated with device drivers. But unlike device drivers, resource managers don't require any special arrangements with the kernel. In fact, a resource manager looks just like any other user-level program.

What is a resource manager?

Since QNX Neutrino is a distributed, microkernel OS with virtually all nonkernel functionality provided by user-installable programs, a clean and well-defined interface is required between client programs and resource managers. All resource manager functions are documented; there's no “magic” or private interface between the kernel and a resource manager.

In fact, a resource manager is basically a user-level server program that accepts messages from other programs and, optionally, communicates with hardware. Again, the power and flexibility of our native IPC services allow the resource manager to be decoupled from the OS.

The binding between the resource manager and the client programs that use the associated resource is done through a flexible mechanism called pathname space mapping.

In pathname space mapping, an association is made between a pathname and a resource manager. The resource manager sets up this pathname space mapping by informing the process manager that it is the one responsible for handling requests at (or below, in the case of filesystems), a certain mountpoint. This allows the process manager to associate services (i.e. functions provided by resource managers) with pathnames.

For example, a serial port may be managed by a resource manager called devc-ser*, but the actual resource may be called /dev/ser1 in the pathname space. Therefore, when a program requests serial port services, it typically does so by opening a serial port — in this case /dev/ser1.

Why write a resource manager?

Here are a few reasons why you may want to write a resource manager:

• The client API is POSIX.

  The API for communicating with the resource manager is for the most part, POSIX. All C programmers are familiar with the open(), read(), and write() functions. Training costs are minimized, and so is the need to document the interface to your server.

• You can reduce the number of interface types.

  If you have many server processes, writing each server as a resource manager keeps the number of different interfaces that clients need to use to a minimum.

  For example, suppose you have a team of programmers building your overall application, and each programmer is writing one or more servers for that application. These programmers may work directly for your company, or they may belong to partner companies who are developing addon hardware for your modular platform.

  If the servers are resource managers, then the interface to all of those servers is the POSIX functions: open(), read(), write(), and whatever else makes sense. For control-type messages that don't fit into a read/write model, there's devctl() (although devctl() isn't POSIX).

• Command-line utilities can communicate with resource managers.

  Since the API for communicating with a resource manager is the POSIX set of functions, and since standard POSIX utilities use this API, you can use the utilities for communicating with the resource managers.

  For instance, suppose a resource manager registers the name /proc/my_stats. If you open this name and read from it, the resource manager responds with a body of text that describes its statistics.

  The cat utility takes the name of a file and opens the file, reads from it, and displays whatever it reads to standard output (typically the screen). As a result, you could type:

  cat /proc/my_stats

  and the resource manager would respond with the appropriate statistics.

  You could also use command-line utilities for a robot-arm driver. The driver could register the name, /dev/robot/arm/angle, and interpret any writes to this device as the angle to set the robot arm to. To test the driver from the command line, you'd type:

  echo 87 >/dev/robot/arm/angle

  The echo utility opens /dev/robot/arm/angle and writes the string (“87”) to it. The driver handles the write by setting the robot arm to 87 degrees. Note that this was accomplished without writing a special tester program.

  Another example would be names such as /dev/robot/registers/r1, r2,.... Reading from these names returns the contents of the corresponding registers; writing to these names sets the corresponding registers to the given values.

  Even if all of your other IPC is done via some non-POSIX API, it's still worth having one thread written as a resource manager for responding to reads and writes for doing things as shown above.

The types of resource managers

Depending on how much work you want to do yourself in order to present a proper POSIX filesystem to the client, you can break resource managers into two types:

• Device resource managers
• Filesystem resource managers

Device resource managers

Device resource managers create only single-file entries in the filesystem, each of which is registered with the process manager. Each name usually represents a single device. These resource managers typically rely on the resource-manager library to do most of the work in presenting a POSIX device to the user.

For example, a serial port driver registers names such as /dev/ser1 and /dev/ser2. When the user does ls -l /dev, the library does the necessary handling to respond to the resulting _IO_STAT messages with the proper information. The person who writes the serial port driver can concentrate instead on the details of managing the serial port hardware.

Filesystem resource managers

Filesystem resource managers register a mountpoint with the process manager. A mountpoint is the portion of the path that's registered with the process manager. The remaining parts of the path are managed by the filesystem resource manager. For example, when a filesystem resource manager attaches a mountpoint at /mount, and the path /mount/home/thomasf is examined:

/mount/

Identifies the mountpoint that's managed by the process manager.

home/thomasf

Identifies the remaining part that's to be managed by the filesystem resource manager.

Here are some examples of using filesystem resource managers:

• flash filesystem drivers (although the source code for flash drivers — available on our Foundry27 community website — takes care of these details)
• a tar filesystem process that presents the contents of a tar file as a filesystem that the user can cd into and ls from
• a mailbox-management process that registers the name /mailboxes and manages individual mailboxes that look like directories, and files that contain the actual messages.

Communication via native IPC

Once a resource manager has established its pathname prefix, it will receive messages whenever any client program tries to do an open(), read(), write(), etc. on that pathname. For example, after devc-ser* has taken over the pathname /dev/ser1, and a client program executes:

fd = open ("/dev/ser1", O_RDONLY);

the client's C library will construct an io_open message, which it then sends to the devc-ser* resource manager via IPC.

Some time later, when the client program executes:

read (fd, buf, BUFSIZ);

the client's C library constructs an io_read message, which is then sent to the resource manager.

A key point is that all communications between the client program and the resource manager are done through native IPC messaging. This allows for a number of unique features:

• A well-defined interface to application programs. In a development environment, this allows a very clean division of labor for the implementation of the client side and the resource manager side.
• A simple interface to the resource manager. Since all interactions with the resource manager go through native IPC, and there are no special “back door” hooks or arrangements with the OS, the writer of a resource manager can focus on the task at hand, rather than worry about all the special considerations needed in other operating systems.
• Free network transparency. Since the underlying native IPC messaging mechanism is inherently network-distributed without any additional effort required by the client or server (resource manager), programs can seamlessly access resources on other nodes in the network without even being aware that they're going over a network.

---

All QNX Neutrino device drivers and filesystems are implemented as resource managers. This means that everything that a “native” QNX Neutrino device driver or filesystem can do, a user-written resource manager can do as well.

---

Consider FTP filesystems, for instance. Here a resource manager would take over a portion of the pathname space (e.g. /ftp) and allow users to cd into FTP sites to get files. For example, cd /ftp/rtfm.mit.edu/pub would connect to the FTP site rtfm.mit.edu and change directory to /pub. After that point, the user could open, edit, or copy files.

Application-specific filesystems would be another example of a user-written resource manager. Given an application that makes extensive use of disk-based files, a custom tailored filesystem can be written that works with that application and delivers superior performance.

The possibilities for custom resource managers are limited only by the application developer's imagination.

Resource manager architecture

Here is the heart of a resource manager:

initialize the dispatch interface
register the pathname with the process manager
DO forever
    receive a message
    SWITCH on the type of message
        CASE io_open:
            perform io_open processing
            ENDCASE
        CASE io_read:
            perform io_read processing
            ENDCASE
        CASE io_write:
            perform io_write processing
            ENDCASE
        .   // etc. handle all other messages
        .   // that may occur, performing
        .   // processing as appropriate
    ENDSWITCH
ENDDO

The architecture contains three parts:

1. A channel is created so that client programs can connect to the resource manager to send it messages.
2. The pathname (or pathnames) that the resource manager is going to be responsible for is registered with the process manager, so that it can resolve open requests for that particular pathname to this resource manager.
3. Messages are received and processed.

This message-processing structure (the switch/case, above) is required for each and every resource manager. However, we provide a set of convenient library functions to handle this functionality (and other key functionality as well).

Message types

Architecturally, there are two categories of messages that a resource manager will receive:

• connect messages
• I/O messages.

A connect message is issued by the client to perform an operation based on a pathname (e.g. an io_open message). This may involve performing operations such as permission checks (does the client have the correct permission to open this device?) and setting up a context for that request.

An I/O message is one that relies upon this context (created between the client and the resource manager) to perform subsequent processing of I/O messages (e.g. io_read).

There are good reasons for this design. It would be inefficient to pass the full pathname for each and every read() request, for example. The io_open handler can also perform tasks that we want done only once (e.g. permission checks), rather than with each I/O message. Also, when the read() has read 4096 bytes from a disk file, there may be another 20 megabytes still waiting to be read. Therefore, the read() function would need to have some context information telling it the position within the file it's reading from.

The resource manager shared library

In a custom embedded system, part of the design effort may be spent writing a resource manager, because there may not be an off-the-shelf driver available for the custom hardware component in the system.

Our resource manager shared library makes this task relatively simple.

Automatic default message handling

If there are functions that the resource manager doesn't want to handle for some reason (e.g. a digital-to-analog converter doesn't support a function such as lseek(), or the software doesn't require it), the shared library will conveniently supply default actions.

There are two levels of default actions:

• The first level simply returns ENOSYS to the client application, informing it that that particular function is not supported.
• The second level (i.e. the iofunc_*() shared library) allows a resource manager to automatically handle various functions.

For more information on default actions, see the section on “Second-level default message handling” in this chapter.

open(), dup(), and close()

Another convenient service that the resource manager shared library provides is the automatic handling of dup() messages.

Suppose that the client program executed code that eventually ended up performing:

fd = open ("/dev/device", O_RDONLY);
...
fd2 = dup (fd);
...
fd3 = dup (fd);
...
close (fd3);
...
close (fd2);
...
close (fd);

The client would generate an io_open message for the first open(), and then two io_dup messages for the two dup() calls. Then, when the client executed the close() calls, three io_close messages would be generated.

Since the dup() functions generate duplicates of the file descriptors, new context information should not be allocated for each one. When the io_close messages arrive, because no new context has been allocated for each dup(), no release of the memory by each io_close message should occur either! (If it did, the first close would wipe out the context.)

The resource manager shared library provides default handlers that keep track of the open(), dup(), and close() messages and perform work only for the last close (i.e. the third io_close message in the example above).

Multiple thread handling

One of the salient features of QNX Neutrino is the ability to use threads. By using multiple threads, a resource manager can be structured so that several threads are waiting for messages and then simultaneously handling them.

This thread management is another convenient function provided by the resource manager shared library. Besides keeping track of both the number of threads created and the number of threads waiting, the library also takes care of maintaining the optimal number of threads.

Dispatch functions

The OS provides a set of dispatch_* functions that:

• allow a common blocking point for managers and clients that need to support multiple message types (e.g. a resource manager could handle its own private message range).
• provide a flexible interface for message types that isn't tied to the resource manager (for clean handling of private messages and pulse codes)
• decouple the blocking and handler code from threads. You can implement the resource manager event loop in your main code. This decoupling also makes for easier debugging, because you can put a breakpoint between the block function and the handler function.

For more information, see the Resource Managers chapter of Getting Started with QNX Neutrino, and the Writing a Resource Manager guide.

Combine messages

In order to conserve network bandwidth and to provide support for atomic operations, the OS supports combine messages. A combine message is constructed by the client's C library and consists of a number of I/O and/or connect messages packaged together into one.

For example, the function readblock() allows a thread to atomically perform an lseek() and read() operation. This is done in the client library by combining the io_lseek and io_read messages into one. When the resource manager shared library receives the message, it will process both the io_lseek and io_read messages, effectively making that readblock() function behave atomically.

Combine messages are also useful for the stat() function. A stat() call can be implemented in the client's library as an open(), fstat(), and close(). Instead of generating three separate messages (one for each of the component functions), the library puts them together into one contiguous combine message. This boosts performance, especially over a networked connection, and also simplifies the resource manager, which doesn't need a connect function to handle stat().

The resource manager shared library takes care of the issues associated with breaking out the individual components of the combine message and passing them to the various handler functions supplied. Again, this minimizes the effort associated with writing a resource manager.

Second-level default message handling

Since a large number of the messages received by a resource manager deal with a common set of attributes, the OS provides another level of default handling. This second level, called the iofunc_*() shared library, allows a resource manager to handle functions like stat(), chmod(), chown(), lseek(), etc. automatically, without the programmer having to write additional code. As an added benefit, these iofunc_*() default handlers implement the POSIX semantics for the messages, again offloading work from the programmer.

Three main structures need to be considered:

• context
• attributes structure
• mount structure

A resource manager is responsible for three data structures.

The first data structure, the context, has already been discussed (see the section on “Message types”). It holds data used on a per-open basis, such as the current position into a file (the lseek() offset).

Since a resource manager may be responsible for more than one device (e.g. devc-ser* may be responsible for /dev/ser1, /dev/ser2, /dev/ser3, etc.), the attributes structure holds data on a per-device basis. The attributes structure contains such items as the user and group ID of the owner of the device, the last modification time, etc.

For filesystem (block I/O device) managers, one more structure is used. This is the mount structure, which contains data items that are global to the entire mount device.

When a number of client programs have opened various devices on a particular resource, the data structures may look like this:

Multiple clients opening various devices.

The iofunc_*() default functions operate on the assumption that the programmer has used the default definitions for the context block and the attributes structures. This is a safe assumption for two reasons:

1. The default context and attribute structures contain sufficient information for most applications.
2. If the default structures don't hold enough information, they can be encapsulated within the structures that you've defined.

By definition, the default structures must be the first members of their respective superstructures, allowing clean and simple access to the requisite base members by the iofunc_*() default functions:

Encapsulation.

The library contains iofunc_*() default handlers for these client functions:

Summary

By supporting pathname space mapping, by having a well-defined interface to resource managers, and by providing a set of libraries for common resource manager functions, QNX Neutrino offers the developer unprecedented flexibility and simplicity in developing “drivers” for new hardware — a critical feature for many embedded systems.

For more details on developing a resource manager, see the Resource Managers chapter of Getting Started with QNX Neutrino, and the Writing a Resource Manager guide.