The layout of the OpenTrack sources

The OpenTrack sources are stored on ot6:/u1/ot/build. As of 13 April 1994 the most recent version of the sources is OT V3.0.2. To inspect them, first mount the ot area with the osfmount command, then give the following form of the mount command, first creating /project/ot/build if it doesn't exist:

osfmount ot mount ot6:/u1/ot/build /project/ot/build

For archive purposes, there is a directory ot6:/u1/OT_pre_V3.0 containing the sources for version 2 of OpenTrack, which is the version used in creating the CRs in the dce_inactive database on snoopy.

OpenTrack sources and builds are performed with ODE, so the layout of the directories should be familiar.

The bin directory contains the main functions for the OT clients and servers.

bin/ot.c
enter and update client
bin/ot_bugs.c
report generating client
bin/otcl.c
client used in writing OT TCL scripts
bin/ode2ot.c
client which updates CRs based on bsubmit.log information
bin/otAfterBoot.c
program which cleans up locked CR files. It should be called in /etc/rc (or equivalent)
bin/otBlankTemplate.c
program which generates a blank template file
bin/otCreateHeadHistFile.c
program which generates a combined HeadHistFile of header and history information for each CR.
bin/otCreateHeadDB.c
program which generates the headdb file only for speed
bin/otSplitHeadHistFile.c
program which splits the HeadHistFile built by otCreateHeadHistFile into headdb and histdb files read by otKwikPix (otSlowPix) and written by otqm
bin/otqm.c
the OpenTrack queue manager which reads header data files in the otdqueue directory (as deposited by otd) and adds their information to headdb and histdb
bin/otd.c
the OpenTrack daemon which accepts incoming connections and speaks to clients in an OT-enhanced TCL language, executing enter/update and handling queries
bin/otKwikPix.c
the OpenTrack storing server, which is accessed by otqm and otd via a UNIX socket. It stores header information from the headdb file and accepts enter/update information from otqm.
bin/otk.c
This is just for fun: an OT TCL image with the Tk toolkit linked in. It could be the basis for a Tk UI for OT.

The lib directory contains three libraries used by OpenTrack and one unused library.

lib/libdp
This is the TCL distributed programming library, a public-domain library which provides RPC-like mechanisms to execute TCL commands remotely. Changes made to this library for OT purposes are marked with the comment "OT CHANGE". The purpose of the changes was to allow the ot, ot_bugs or otcl client to communicate via a pipe as well as a socket. This is required for the odexmcli support. See otConn() [lib/libot/otNetwork.c].

The lib/libot directory contains all the general OT routines.

libot/otCLI.c
These routines are specific to ot and ot_bugs: they parse command-line arguments.
libot/otCompare.c
These routines are type-specific comparison routines called by the query and validation routines.
libot/otConvert.c
These routines convert an OT template to a TCL code fragment for passing client/server or vice versa, return a string contain a file's contents, and generate a 'control' command for queries requests to the otd and otKwikPix/otSlowPix server.
libot/otDelete.c
This routine implements the OT TCL delete command.
libot/otEnter.c
These routines implement the OT TCL enter command (server-side).
libot/otError.c
This file contains the error message array as well as routines used to manipulate them. The codes for the error messages are defined in ot.h.
libot/otGlobals.c
This file contains global variables.
libot/otInit.c
These routines are mostly initialization for common structures like control blocks.
libot/otKwik.c
These routines read and write "kwik strings", which is the data format used in the headdb and histdb files.
libot/otMetaTempl.c
This routine reads the information in the project metatemplate into an internal data structure.
libot/otNetwork.c
These routines sit on top of the Tcl-DP routines for sending TCL commands to the server and receiving a return string and status code (for client applications like ot and otcl). The otMakeServer() routine is called by otd and calls the TCL-DP routine Tdp_ReceiveRPC().
libot/otNotify.c
The otNotify() routine handles notification, at present on the client-side.
libot/otProject.c
The routine in this file reads the project definition file and stores it in a global data structure.
libot/otQuery.c
This file contains the otProcessCR() routine for evaluating a CR given a search criterion and returning true/false, and a set of formatting routines for printing.
libot/otRCS.c
These routines invoke the RCS tools set for checking out and checking in.
libot/otSummary.c
This routines are used in printing summary information.
libot/otTcl.c
This contains all the routines associated with TCL support in OT, including most command bindings and critical TCL supporting routines such as cmdField() for evaluating and assigning header field values, cmdNote() for assigning note values, and many others.
libot/otTemplate.c
This file contains all the routines for reading templates from a file into an internal representation (the OTTemplate structure).
libot/otUpdate.c
This file contains all the routines which are called by the server in updating a CR.
libot/otUtil.c
This contains a number of utility routines.
libot/otValidate.c
This contains routines associated with validation, including a general routine, an array of type definitions and type-specific validation routines.
libot/ot.h
Contains typedefs for all the routines, as well as structure definitions.
libot/otInt.h
The "OT Internals" header file, which defines private structures and function interfaces.
lib/libtcl
This is TCL version 7.0. There are no OT-specific changes to TCL although there were some mild changes to make TCL build under ODE.
lib/libtk
This is Tk version 3.0.
man/man1
This contains the sources for man pages for user commands.
man/man5
This contains one manual page, ot_defs.5, which defines the file format for the project definition file and the project metatemplate file.

How to build OpenTrack

OpenTrack is at present built on the following platforms:

The i386 and mips builds are completely straightforward: see the ODE documentation for relevant information. The Sparc and HP-UX environments require more customization.

Here is a series of steps for the i386 or mips builds:

Sparc platform running Sun/OS

The sparc platform basically uses a Motif build procedure: that is to say, the make command and not the build command is used. To build on sparc, follow these steps.

HP9000 running HP/UX

Debugging the OT server

If you wish to step through the OT server code in searching for a bug, hooks are in place to make this easier. Follow these steps.

Architecture description

OpenTrack is an set of utilities which provides simple database functionality without the aid of a relational database package. It stores data in a simple RFC-802 like format (header followed by text) with some added information for type and validation. It uses RCS to maintain archives of each CR or record. Header and history information is extracted and placed in summary files for quick access. A storing server runs in the background for quick access of header information. OpenTrack is a client-server application which uses an enriched version of TCL as the application protocol.

Data flow

Data entry

This is a description of the flow of data during an enter operation. For purposes of illustration the motif project will be used.

Update operations are in general quite similar to the enter operation except that the client sends an 'update xxx' command to the server, where xxx is the number associated with the CR to update. This causes the server to check out the file containing the CR and read its content into an internal representation. If the return value of the 'update' command is successful (null string) then the client will send a 'tclString' command (actually a 'tclString orig' command) to the server, which will cause the server to send as result a TCL program which when interpreted by the client, causes the current CR buffer on the client side to be assigned the current header, note and history information as stored in the server current CR buffer.

Queries are performed by the client as follows. As in the case of enter and update, once the server host machine is identified, it sends a command of the form project set motif to the server machine to initialize various internal project-specific buffers. It then sends a 'control' command of the form control { xxx }, where xxx is a series of label/value pairs. For example, the following sets the server's query context such that only short reports should be sent, and the tclString to use in queries should by "[stat open]" (print CRs for whom the TCL evaluation of this returns '1').

control { {fullText 0} {tclString {[stat open]}} }

Here is a list of the labels recognized by OT for control lists:

skipHeader
Do not generate a header line for short reports (value 0 or 1).
count
Generate a count of CRs instead of the usual report (value 0 or 1).
summary
Generate a summary of CRs instead of the usual report (value 0 or 1).
tabSeparate
Separate short report fields by tabs instead of spaces (value 0 or 1).
fullText
Send the full text of the CR and not a short report (value 0 or 1).
historyLines
If set to '1', successful completion of query requires that the history information be available. If this is so then the otd daemon will connect to otSlowPix rather than otKwikPix, in which history information is not available.
quiet
Do not print the CR even if it matches the tclString search criterion.
layout
A list of the fields to print in a short report.
keyInText
Print all CRs with this keyword in the text of the notes somewhere.
nsens
If this has a value, then print only the notes whose note sensitivity field matches it.
tclFile
At the moment this hook is ignored. It could someday be used to read in a one-time-only file of TCL procedures.
tclOtrc
At the moment this hook is ignored. It could someday be used to read in a user's file of TCL procedures.
tclString
A string which is evaluated once per CR examined by the otKwikPix (or otSlowPix) server and whose return string is either 1 (a match - print the CR) or 0 (do not print the CR). If the user on the client side used ot_bugs and had an -s argument (e.g., -s stat=open) then this expression will be translated into one of the above TCL forms and sent over as a value to the tclString list in this control command.

After the control command is sent to the server, the next command depends on what kind of report is requested.

Configuration files

The primary configuration files for an OT project are the project definition file and the project metatemplate file. The parsing code for these is in lib/libot in otProject.c and otMetaTempl.c. The manual page ot_def.5 describes them both. The ASCII graphic below shows the general appearance of a project base directory.

base directory (e.g., /project/ot/motif) +-----------+-----------+---------+---------+-------+------+-----(--+--)... | | | | | | | project project otKwikPix queue next CR headdb d00 d01 ... definition metatemplate socket directory number histdb | file file +----+--(-+-)... | | | motif.def motif.template.def otdqueue number d00 d01 d02 ... | +--+--+ CRs: RCS c000001 | c000002 c000001,v c000003 c000002,v ... ...

CR files

The CR data files are stored in directories under the project directory. The general format of these files is described in the manual page ot.1. The routines in lib/libot/otTemplate.c parse this format.

kwik format

The header and history information in the headdb and histdb files illustrates the use of kwik strings. There were two motivations for the kwik string format.

In general the kwik string format involves separating the header fields and history fields with special control characters (\013 through \017 as defined in otInt.c). See otKwik.c for details on the format of the strings. What is most important is that in general the order of the fields in the kwik string for a project follow the order of the fields in the metatemplate and there is no tagging in it: thus for a metatemplate with two fields, "Engineer:" and "Status:", the kwik string would look like

pnh<SEP>open However, since part of the original requirements for OpenTrack was that users need to be able to add fields not in the metatemplate, kwik strings can also contain 'whole fields' (that is, user-contributed fields not in the metatemplate). If the template is Engineer: Status: and the user edits the template to look like the following: Engineer: pnh Status: open General topic: text utilities then the kwik string for this will look something like the following: pnh<SEP>open<OTHERSEP>General topic: text utilities This design of the kwik string was to preserve the performance gain of the untagged information, while making it possible to reflect information not corresponding to metatemplate fields.

The history kwik strings are more complicated because there are more field separators but this implementation is very straight-forward.

It would have been interesting to design the header information as TCL lists rather than kwik strings, but at the time it seemed as though the TCL list parsing routines would not be fast enough. This could be revisited someday.

Application protocol

OpenTrack uses the TCL Distributed Processing library, which emulates a kind of RPC mechanism so individual TCL commands can be executed on a server. The message formats are not documented but can be deduced by reading lib/libdp/dpnetwork.c (see Tdp_PacketSend()) and lib/libdp/dprpc.c (see Tdp_RPC() and Tdp_ProcessRPCMessages()).

The client-side routine otRemoteTcl() (in lib/libot/otNetwork.c) passes a command string (e.g., project set motif) to the RPC mechanism via the routine Tdp_RPC(). The RPC routines in lib/libdp tag the message as an RPC message and pass it to the 'packet level', which is a layer of routines which encapsulates the RPC message as a TCL-DP packet with a message format of magic number, length field (32 bits) and (null-terminated) command string. When the server (otd) returns either a TCL result string or an error status and error message, the packet is parsed by the packet layer (Tdp_PacketReceive() in libdp/dpnetwork.c) and the resulting message is passed up to the RPC layer, which removes either the 'return' token or the 'error' token from the front of the message.

The most important aspects of the protocol have been discussed in the Data flow section. Here are some other salient points of the protocol:

Note that the file on each server machine /project/ot/serverV3.log keeps a copy of everything sent by the client to the server. This can be used to get a sense of the protocol. Other ways to become more familiar with it is to examine instances of its use, as in the otmig script in util/scripts.

Process structure

OpenTrack is a client-server application. The server also communicates with other processes in performing enter, update and query operations.

Client and server

The OT client is one of ot, ot_bugs or otcl. The otcl program may be used to write scripts. See otmig in util/scripts for an example of one of these scripts.

The otd server is exec'd by /etc/inetd like most UNIX servers. The only unusual aspect of the connect procedure is that in lieu of using the getserverport...() routine or hardcoding the server port for otd, the current OT clients check the 'server port' value in the project definition file for the project whose server they are connecting to.

Note that while the ODEXM server protocol is somewhat like rsh in that it must execute child processes to execute commands requested from a client, the otd daemon is not like rsh: in general it does not execute processes to execute the commands, but rather the support is built in to the otd binary.

Like INN, OpenTrack can use a storing server (otKwikPix) which keeps an in-core copy of the header data at any one time. Note that this is just a performance enhancer and that if the otKwikPix server is unavailable (i.e. the OT server otd cannot connect to the UNIX socket on which otKwikPix is listening) then otd will fork and exec /etc/otSlowPix, which will provide the same information. The interactions between these processes is represented in the ASCII graphic below.

+----------------------------------+ | ot - OT client | | (also ot_bugs, otcl) | +---------------+------------------+ client | TCP-IP socket | ------------------------------------------------------+-------------------- | +------------+ server +---------------+------------------+ | | | | | otKwikPix | | otd - OT daemon | | | | | |stores +----+ <-------+daemon handles Tcl-DP protocol | |header | | / | from client | |information | | / | - connects to otKwikPix for query| |as array of | | / | - forks/execs otqm for enters,upd| |OTTemplate | | / +---------------+------------------+ |structures | | / | (otqm is exec'd on | | | / | enter/update only) +------------+ | / +---------------v------------------+ |/ | ot queue manager (otqm) | o<---------------+ | UNIX socket | - reads header, history info | (e.g. /project/ot/motif/socket) | deposited by otd in otdqueue | on which otKwikPix listens after | - merges info in headdb,histdb | initialization for | - connects to otKwikPix to | - incoming connect from otd for queries | refresh stored information | - incoming connect from otqm for +----------------------------------+ refreshing stored info

OT queue manager

The queue manager otqm exists in order to update the headdb and histdb databases. These files are summaries in kwik string format of the header and history information in the CRs, such that the first line in the file represents the nth CR, followed by the (n-1)th and so down to the end. This scheme was chosen because it offers simplicity and the fastest speed when accessing the whole file, as would be done by otSlowPix if otKwikPix were not running. Updating one line during a commit operation in one of these files, which can run many megabytes in size, is time-consuming and thus must be done asynchronously by a separate process, otqm. The otd server deposits a queue item file in the otdqueue directory. The name of the file starts with 'qi' and the file itself is composed of two lines: the first is the header kwik string and the second is the history kwik string.

The queue manager otqm is forked by otd (in cmdCommit(), see otTcl.c). otqm searches the directory for qi files and when it finds one, it integrates the kwik strings it contains into headdb and histdb and attempts to connect to otKwikPix to refresh its cache (via a 'delta' command intercepted by otKwikPix - see the sources). otqm uses the file otqm.lok immediately under the base directory to prevent other otqm processes from being exec'd during this operation (cf. lpd). When all the qi files have been processed otqm exits.

otKwikPix and otSlowPix

The otKwikPix server should be started either manually or in a system rc file on startup. Each otKwikPix process can only handle information for one project. On startup it initializes an array of OTTemplate structures by reading the data from the headdb file. It then listens on a UNIX socket under the project base directory for connects from one of two places:

The otd server must choose whether to connect to otKwikPix or to fork and exec otSlowPix as a child process. Since the otKwikPix server does not store the history information (because the memory requirements become truly exhorbitant), otSlowPix must be consultant if the user wishes to use history information during a query. This is indicated somewhere in the begin condition of the query procedure by giving a command such as

control { {historyLines 1} } The manual page ot_tcl.1 has an example of the use of the control command to accomplish this (see "History Pseudofields"). If the historyLines property is 'on' for this query, then otd will choose otSlowPix.

The other condition under which otSlowPix is selected is if otKwikPix is unreachable or not running.

Both otSlowPix and otKwikPix return the same information for the same requests. The otd server always sends a control command to determine the query.

With regard to the sources, the relationship between otKwikPix and otSlowPix is similar to that between ex and vi; otSlowPix is a link to otKwikPix. The binary inspects the value of argv[0] to determine which image was intended and determines the method of access. If the command name is 'otKwikPix', it reads all information into an array of OTTemplate and waits for connects. If the command name is 'otSlowPix', it reads each entry from headdb and if necessary from histdb one at a time and evaluates the search criteria (TCL procedure).