L2 RTAPI Application Programming Guide

The L2 Realtime API Application Programming Guide : What can the RTAPI do for you?

Overview.

The L2 Real Time API (L2 RTAPI) is a multithreaded API that provides an interface for an application to send messages in real time to the L2 diagnosis and recovery library. It is the responsibility of the RTAPI to provide a thin interface which maintains responsiveness to further commands while a queue of prior inputs to L2 is being processed. The RTAPI provides a framework for introducing real time (versus L2's discrete event-based timing) systematically for the time-based injection of commands, observations, faults, etc. Both of these capabilites are necessary when L2 is running in conjunction with real hardware systems, because observations may continue to stream in while commands may take of the order of seconds to actually be completed (both on the hardware and within the L2 processing algorithms).

The RTAPI provides a framework for important application modules such as telemetry output (inferred transitions, typically) and diagnostic triggering. Default autonomous diagnostic (and eventually recovery) policies are implemented while allowing for complete customization as necessary for the application. Through the use of inherited interfaces, new implementation objects may be defined with no functional changes required in other parts of code. Similar techniques allow for extensive debugging and verification during application testing, with compile-time switches for lean runtime builds.

The RTAPI provides good encapsulation of the application-L2 interface layer in that the application code is shielded from the details of the L2 public methods as well as platform threading specifics, while an internal queue keeps interprocess communication low.

The RTAPI is currently supported on both Unix (posix) and Win32 with plans to port to VxWorks.

How an application uses the L2 RTAPI.

See Section 3 for further details.

Realtime functionality encapsulation.

Interfaces specified: L2_rtapi, Reporter, L2_queue, timeout handlers, model time mapping enumeration

See the reference manual and Appendix A.

Cross platform multithreaded L2 interface.

The realtime api provides an interface for posting messages to the L2 engine that will run in a separate thread of execution. This maintains reactivity of the system as a whole while long computations complete. As implied, the api knows about time and thus commands may be scheduled for execution after some amount of clock time has passed. The api also provides for the specification of runtime autonomous diagnostic policies. Defaults are provided. See Sections 3 and 4.

Timeout handling for commands and observations.

See sections 3 and 4.

Runtime diagnosis triggering.

See Sections 3 and 4.

Realtime debugging output.

Equivalent L2 scripts are generated for debugging.
Dribble files.

See Sections 2 and 3.

Extensibility.

Custom telemetry output.
Custom timeout handlers for observations and commands.
Debugging

See Section 4 for details.

The L2 Realtime API Application Programming Guide : Building and using the sample application

Getting the L2 source.

Rather than building the debug command line test, however, you will build a special RTAPI test application.

Building the sample application under:

Solaris.

Linux.

Win32.

Using the sample application to test model scenarios.

But does the sample application actually do? First, it instantiates an L2 engine and read in the "cb" model files. In addition to the normal "cb.xmpl", "cb.ini", and "cb.hrn" the application relies upon two include files for realtime-specific model information: "enumapi.h" and "time_delays.h". The first is used to specify enumerations corresponding to the model observables and commands. It is generated automatically by the api_gen tool described in more detail below. The second file must be coded by handed and contains arrays which define timeout durations (in seconds!) for each command and observation in the enumerated api.

After that, the application read in a simulated input sequence from "scenario.rt". The inputs are executed as specified, including real time delays to simulate an actual system. Corresponding RTAPI functions are invoked in order. That's it. Any other I/O takes place within the RTAPI or L2.

Understanding the provided test scenario.

The first line of "test1.scr" (and, hence "scenario.rt") indicates initiation of an "on" command to the number thirteen circuit breaker. The command is issued with an associated timeout, as indicated by use of the keyword 'cmdt' rather than 'cmd'. The length of this timeout is read in from the "time_delays.h" file. The second line indicates an observation that the number eight led is on. Finally, the third line is a 'wait' command (of duration one second), specific to the RTAPI test environment that causes the main task to sleep for a specified amount of time. This command is present only for the purpose of simulating real time delays between messages and doesn't result in any messages being placed upon the L2 queue.

During execution of the test program, the circuit breaker eight command is initiated with a call to "queue_command_and_start_time(unsigned int cmd, unsigned int cmd_index)" on an L2_rtapi object. This function places the circuit breaker eight command on the L2 queue and spawns a thread using the default command timeout handler function. This timeout handler thread is put to sleep for the specified timeout of ten seconds. Upon waking, the default timeout handler function runs, placing a request for a "find_candidates" on the L2_queue. While the thread is sleeping (before the handler runs), the observation is processed by the main thread. This results in a call to "queue_observation(unsigned int obs, unsigned int obs_index)" using through the L2_rtapi object. Thus the L2_queue ends up with the command, the observation, and the find candidates in first-to-last order. Obviously, this is entirely a function of the specified timing. If the observation had been preceding by a 'wait' which caused the test program to simulate an interval of fifteen seconds between the command being issued and the observation being recorded, the diagnosis would have been triggered before the observation and resulted in a conflict. These results may be verified in the L2 RTAPI output files.

Using api_gen and api_scr.

The api_scr tool takes a realtime simulation script written by a user and translates the commands into corresponding integer values for the RTAPI test program. The api_scr tool is built via the Makefile in the 'src/liv_utils/api_scr' directory. The executable takes one command line argument, the name of the input script. Output is "scenario.rt". Allowed scripting keywords are: "cmd" - issue a command with no timeout, "obs" - record an observation with no timeout, "cmdt" - issue a command and start a timeout, "obst" - record and observation and start a timeout, "fc" - initiate a diagnosis, "report" - report the current system state, and "wait" - simulate a real time delay. All observation and command functions require the name string and domain value. E.g. "obst cbAndLed.led8.ledState on". "fc" requires no arguments, while "wait" needs an integer number of seconds. The input script's final line must be "end". See the sample in "src/test/scenario.scr".

Real time scenario testing sample summary.

Files needed: "<model>.ini", "<model>.xmpl", "<model>.hrn", "enumapi.h", "time_delays.h", "scenario.rt", and, optionally, "scenario.scr".

Modifying the sample app to use different Livingstone objects and reporting.

The L2 Realtime API Application Programming Guide : Application programming with the RTAPI

Application architecture and platform support.

There are many issues governing the application's realtime needs, including platform capabilities. Currently, the RTAPI is implemented under Win32 and Posix/Unix assuming L2 is run in a separate process from that of the software generating observations and commands. Thus monitors and avionics might reside in one process, posting messages to an IPC queue. The IPC messages are retrieved by the L2 process and passed on to the L2 engine via the RTAPI running in the same process as L2. This is depicted graphically below.

Figure 2. A possible application architecture with L2 RTAPI.

For the operating systems that implement processes to run in separate memory address spaces (Win32 and Unix), IPC is relatively slow. Thus, this design is efficient when a single monitor or command message may result in multiple messages sent to the L2 engine (that is the case for RTAPI) since a process internal queue will be much faster. The L2 RTAPI design assumes that the RTAPI and the L2 engine can both directly address the same memory space, i.e. they share pointers. It should be noted that while most Unix operating systems provide mechanisms for placing structures in memory blocks which are directly accessible crosspieces (Win32 may, as well), Unix and Win32 are not flight OS's, in general. According to the VxWorks 5.4 Programmer's Guide: "all code executes in a common address space" so that, in fact, the current design more closely reflects the runtime environment to be encountered under VxWorks. VxWorks also provides OS level memory pooling and other functionality. A reasonable alternative application architecture and use of the RTAPI is pictured below. Note that now the L2_queue interface wraps an intertask queue provided by VxWorks.

Figure 3. Alternative architecture, assuming single address space.

Again, while this architecture may be implemented under Unix by wrapping an IPC queue with the L2_queue interface, specifying certain objects to be placed in global memory, and enabling interprocess event signaling, the utility of such effort at this point is questionable given that VxWorks is the typical realtime OS. Given the significant platform differences, the important thing is that the interfaces and logic remain the same through the testing and flight phases.

Another useful feature provided by VxWorks is the watchdog timer. Watchdog timers are interrupt timers which execute callback functions after a specified time delay has elapsed. They have much lower overhead than tasks. The timeouts on VxWorks, then, would utilize watchdog timers with the timeout handlers specified as callback functions. This contrasts the the Win32/Posix implementations in which timers execute in spawned threads.

While there may be other minor issues, it is expected that the port to VxWorks will be relatively straightforward. If the reader desires this port from the author, email spepke@ptolemy.arc.nasa.gov and your wish may become reality.

Below is a table summarizing the current (Win32 and Unix) and expected (VxWorks) RTAPI platform implementation differences.

Feature Implementation	Win32	Unix	VxWorks
L2 execution	Win32 thread	pthread	task
timeouts	Win32 sleep	unix sleep	watchdog timers
memory address space	shared access	shared access	shared access
L2_queue	in-process	in-process	ITC global
memory pooling	custom	custom	OS supplied
event signaling	in-process	in-process	inter-task

Figure 4. Current (Unix and Win32) and expected (VxWorks) platform differences.

Preparing model files (apigen).

Application software communicating with the L2 RTAPI must have knowledge of the enumeration api. All commands and observations are indexed according to this api, which mirrors the L2 engine's internal indexing. Make the application in the 'src/liv_utils/api_gen' directory. Provide the model name (prefix for the .ini, .hrn, and .xmpl files) as a command line argument, e.g. "api_gen cb". This outputs the header file "enumapi.h" for use by the RTAPI and the application. Note that this header file name is hard coded into the RTAPI. The RTAPI looks for this header file in the project include directories.

Building and running with the RTAPI.

Since the L2 RTAPI is multithreaded, only applications built with the appropriate multithreading flags set can use it. Under Solaris, the necessary flag is "-mt". No flag is required under Linux, provided a threading library is linked against. In addition to linking in the realtime_api library, the application must link the pthread library under posix, i.e. -lpthread.

In preparation for using the api, the calling application must instantiate three objects: a Livingstone engine, a Reporter object, and an L2_rtapi object. References to the L2_engine and Reporter object are given as arguments to the L2_rtapi constructor. Although, it might seem awkward, this design allows the application programmer maximum flexibility in terms of engine parameters and output. For details on this, please see the next section on customization.

pseudocode or code sample -- note necessary includes, library linkage

Once the L2_rtapi is instantiated, all L2 messages should be sent via the L2_rtapi public interface. Note that it is the responsibility of the calling application to clean up any memory allocation for the Livingstone and Reporter objects.

Testing.

In the absence of a full simulation, testing can be done using input scripts similar to that used for the sample application in the previous section. During any phase of development, RTAPI functionality can be verified by utilizing either the provided debugging objects or coding custom debug types. For details on this, please see the next section on customization.

The L2 Realtime API Application Programming Guide : Extending the API functionality

Custom output.

        struct ReporterInterface
        {
               virtual void open()=0;
               virtual void close()=0;

               virtual ReporterInterface & operator()()=0;
               virtual ReporterInterface & operator()(livingstone_message_calls fcn)=0;
               virtual ReporterInterface & operator()(livingstone_message_calls fcn, unsigned int var_index, unsigned int val_index)=0;
               virtual ReporterInterface & operator()(livingstone_message_calls fcn, unsigned int var_index, unsigned int val_index, unsigned int         time_delay)=0;
        };

Timeout handlers for observations and commands.

Debug classes.

The Reporter Interface methods are invoked by the LivingstoneDispatcher object upon processing of a message. You can define any debug reporting object you like, provided it implements the ReporterInterface, and pass it to the L2_rtapi object for use.

L2_rtapi_debug is derived from L2_rtapi to provide dribble output. L2_rtapi does not define any abstract interfaces, however, which means L2_rtapi_debug is unrestricted. Thus you may view L2_rtapi_debug giving useful additional output while also providing an example of how you might provide additional debugging functionality. You could choose to redefine L2_rtapi_debug or simply derive from it in order to define added functionality.

The L2 Realtime API Application Programming Guide

The L2 Realtime API Application Programming Guide : Preface

Related documentation.

Questions and comments.