The layout of the OpenTrack sources
The OpenTrack sources are stored on ot6:/u1/ot/build.
As of 13 April 1994 the most recent version of the sources is OT V3.0.2.
To inspect them,
first mount the ot area with the
osfmount
command, then give the following
form of the
mount
command, first creating
/project/ot/build if it doesn't exist:
osfmount ot
mount ot6:/u1/ot/build /project/ot/build
For archive purposes, there is a directory
ot6:/u1/OT_pre_V3.0
containing the sources for version 2 of OpenTrack, which is the version
used in creating the CRs in the dce_inactive database on snoopy.
OpenTrack sources and builds are performed with ODE, so the layout of the
directories should be familiar.
The bin directory contains the main functions for the OT clients and
servers.
- bin/ot.c
- enter and update client
- bin/ot_bugs.c
- report generating client
- bin/otcl.c
- client used in writing OT TCL scripts
- bin/ode2ot.c
- client which updates CRs based on
bsubmit.log information
- bin/otAfterBoot.c
- program which cleans up locked CR files. It should be called in
/etc/rc (or equivalent)
- bin/otBlankTemplate.c
- program which generates a blank template file
- bin/otCreateHeadHistFile.c
- program which generates a combined
HeadHistFile of header and history information for each CR.
- bin/otCreateHeadDB.c
- program which generates the headdb file only for speed
- bin/otSplitHeadHistFile.c
- program which splits the HeadHistFile built by
otCreateHeadHistFile into
headdb and histdb files read by otKwikPix (otSlowPix)
and written by otqm
- bin/otqm.c
- the OpenTrack queue manager which reads header data files in the otdqueue
directory (as deposited by otd) and adds their information to
headdb and histdb
- bin/otd.c
- the OpenTrack daemon which accepts incoming connections and speaks
to clients in an OT-enhanced TCL language, executing enter/update and
handling queries
- bin/otKwikPix.c
- the OpenTrack storing server, which is accessed by
otqm and otd
via a UNIX socket. It stores header information from the
headdb file and accepts enter/update information from otqm.
- bin/otk.c
- This is just for fun: an OT TCL image with the Tk toolkit
linked in. It could be the basis for a Tk UI for OT.
The lib directory contains three libraries used by OpenTrack and
one unused library.
- lib/libdp
- This is the TCL distributed programming library, a public-domain
library which provides RPC-like mechanisms to execute TCL commands remotely.
Changes made to this library for OT purposes are marked with the comment
"OT CHANGE". The purpose of the changes was to allow the ot, ot_bugs or
otcl client to communicate via a pipe as well as a socket. This is
required for the odexmcli support. See otConn() [lib/libot/otNetwork.c].
The lib/libot directory contains all the general OT routines.
- libot/otCLI.c
- These routines are specific to ot and ot_bugs: they parse command-line
arguments.
- libot/otCompare.c
- These routines are type-specific comparison routines called by the
query and validation routines.
- libot/otConvert.c
- These routines convert an OT template to a TCL code fragment for passing
client/server or vice versa, return a string contain a file's contents,
and generate a 'control' command for queries requests to the otd
and otKwikPix/otSlowPix server.
- libot/otDelete.c
- This routine implements the OT TCL delete command.
- libot/otEnter.c
- These routines implement the OT TCL enter command (server-side).
- libot/otError.c
- This file contains the error message array as well as routines used to
manipulate them. The codes for the error messages are defined in ot.h.
- libot/otGlobals.c
- This file contains global variables.
- libot/otInit.c
- These routines are mostly initialization for common structures like
control blocks.
- libot/otKwik.c
- These routines read and write "kwik strings", which is the data format
used in the
headdb
and
histdb
files.
- libot/otMetaTempl.c
- This routine reads the information in the project metatemplate into an
internal data structure.
- libot/otNetwork.c
- These routines sit on top of the Tcl-DP routines for sending TCL commands
to the server and receiving a return string and status code (for client
applications like ot and otcl).
The otMakeServer() routine is called by otd and calls the TCL-DP routine
Tdp_ReceiveRPC().
- libot/otNotify.c
- The otNotify() routine handles notification, at present on the client-side.
- libot/otProject.c
- The routine in this file reads the project definition file and stores
it in a global data structure.
- libot/otQuery.c
- This file contains the otProcessCR() routine for evaluating a CR
given a search criterion and returning true/false, and a set of formatting
routines for printing.
- libot/otRCS.c
- These routines invoke the RCS tools set for checking out and checking in.
- libot/otSummary.c
- This routines are used in printing summary information.
- libot/otTcl.c
- This contains all the routines associated with TCL support in OT,
including most command bindings and critical TCL supporting routines
such as cmdField() for evaluating and assigning header field values,
cmdNote() for assigning note values, and many others.
- libot/otTemplate.c
- This file contains all the routines for reading templates from a
file into an internal representation (the OTTemplate structure).
- libot/otUpdate.c
- This file contains all the routines which are called by the server
in updating a CR.
- libot/otUtil.c
- This contains a number of utility routines.
- libot/otValidate.c
- This contains routines associated with validation, including a
general routine, an array of type definitions and type-specific validation
routines.
- libot/ot.h
- Contains typedefs for all the routines, as well as structure
definitions.
- libot/otInt.h
- The "OT Internals" header file, which defines private
structures and function interfaces.
- lib/libtcl
- This is TCL version 7.0. There are no OT-specific changes to
TCL although there were some mild changes to make TCL build under ODE.
- lib/libtk
- This is Tk version 3.0.
- man/man1
- This contains the sources for man pages for user commands.
- man/man5
- This contains one manual page, ot_defs.5, which defines the file
format for the project definition file and the project metatemplate file.
How to build OpenTrack
OpenTrack is at present built on the following platforms:
- i386 running OSF/1
- mips running ULTRIX
- sparc running Sun/OS
- HP9000 running HP/UX
The i386 and mips builds are completely straightforward: see the ODE
documentation for relevant information. The Sparc and HP-UX environments
require more customization.
Here is a series of steps for the i386 or mips builds:
- Log into the i386 or mips workstation and osfmount ot and ode and then the
sources from ot6 with the following commands.
osfmount ode
osfmount ot
mount server5:/u6/release/ode2.1 /project/ode/tools/ode2.1
mount ot6:/u1/ot/build /project/ot/build
- Put /project/ode/tools/ode2.1/(platform)/bin at the
beginning of your command search path.
- Create a sandbox against
/project/ot/build/ot3.0.2.
- To build the OT objects, give the command
make -r
Sparc platform running Sun/OS
The sparc platform basically uses a Motif build procedure: that is to say,
the make command and not the build command is used.
To build on sparc, follow these steps.
- Log into malaga or betelgeuse and osfmount ode and ot and then the sources
from ot6 with the following commands. You will need the superuser password
before you can execute these: you can obtain it from someone in the Motif
group.
osfmount ode
osfmount ot
mount server5:/u6/release/ode2.1 /project/ode/tools/ode2.1
mount ot6:/u1/ot/build /project/ot/build
- Give these commands
setenv SHARED_LIBRARY_FORMAT on
set path=(/project/ode/tools/ode2.1/sparc_sunos/bin \
/project/ode/tools/ode2.1/sparc_sunos/lib /usr/local/bin \
/usr/ucb $path)
- Create a sandbox against
/project/ot/build/ot3.0.2.
- To build the OT objects, give the command
make -r
HP9000 running HP/UX
- Log into europa or magnolia and osfmount ot, then the sources from
ot6 with the following commands. You will need the superuser password before
you can execute these: you can obtain it from someone in the Motif group.
osfmount ode
osfmount ot
mount ot6:/u1/ot/build /project/ot/build
- Give these commands
setenv SHARED_LIBRARY_FORMAT on
set path=(/usr/ode/bin /usr/local/bin /usr/ucb $path)
- Create a sandbox against
/project/ot/build/ot3.0.2.
- To build the OT objects, give the following commands to
set up the environment and initiate a build (build rather than
make is used on the HP platform):
setenv OPT_LEVEL "-g"
setenv USER
setenv context hp700_hpux
setenv HOME
setenv EXPORT_USING_TAR on
setenv SHARED_LIBRARY_FORMAT on
set path=(/usr/ode/bin /usr/local/bin $path /usr/bin)
build
Debugging the OT server
If you wish to step through the OT server code in searching for a bug,
hooks are in place to make this easier. Follow these steps.
- Set the environmental variable OT_PORT to a server port
number (for example, 9000).
- Run the otd server under gdb.
% setenv OT_PORT 9000
% gdb ./otd
[messages]
[gdb prompt]
- Now, set the environmental variable OT_PORT to the same
server port in the window or shell from which the test client will be
executed.
% setenv OT_PORT 9000
% ./ot_bugs [rest of arguments]
Of course whatever project is selected as the target must have a
'server host' value identical to that of the system on which the server
is to be run (under gdb).
- The OT_PORT variable overrides the default binding port of
both the client and server.
- When you are finished debugging, unset the variable
(unsetenv OT_PORT).
Architecture description
OpenTrack is an set of utilities which provides simple database functionality
without the aid of a relational database package. It stores data in a
simple RFC-802 like format (header followed by text) with some added
information for type and validation. It uses RCS to maintain archives
of each CR or record. Header and history information is extracted and placed
in summary files for quick access. A storing server runs in the background
for quick access of header information. OpenTrack is a client-server
application which uses an enriched version of TCL as the application protocol.
Data flow
Data entry
This is a description of the flow of data during an enter operation.
For purposes of illustration the motif project will be used.
- On an enter operation, the client reads the project definition file
and project metatemplate file and initializes certain internal control blocks.
- Using the information in the OTMetatemplate structure of the OTProject
structure of the main control block, it builds a structure of type OTTemplate
which is blank. That means that the values are blank except for default
values in the project metatemplate file or in environment variables in the
user's environment.
- The blank template is written to a file which the user may edit.
This step is omitted if a file containing the CR to enter is given to the
ot command.
- After the CR is written out to a file, the client initiates a connection
to the server for the motif project (as determined by the
server host and
server port
values in the project definition file).
These should cause a connection to the project server machine and the
/etc/otd program.
The client sends the command project set motif
to initialize the server's internal structures for that project.
It sends an 'enter' command to the server and translates the CR to a
TCL program which it sends to the server, followed by a 'commit' command.
- When the server receives the 'enter' command, it initializes an 'original'
and a current OTTemplate structure (so that it may determine if in fact only
a 'blank' CR has been sent by the client). When it receives the CR in
TCL form, roughly for each command it assigns a value to a field in the
header, or it assigns the note. The history line information cannot be
assigned on the server side so it is safe from corruption by a client.
- When the server receives the 'commit' command, it writes the current
CR to a file and moves it to the appropriate directory based on the pattern
baseDirectory/dxx/dyy/cxxyyzz, where xxyyzz is the
number for the CR which the server obtains from the number file under the
base directory. The server subsequently increments the number in the number
file, so the number in the file always represents the
next number available.
- The server then uses the ci command to check the CR into the archive
immediately under the baseDirectory/dxx/dyy directory.
- The server builds a temporary file with the header and history information
to store for quick access (see otBuildQueueItem() in
otUtil.c)
and deposits it in the queue directory otdqueue.
The name of the temporary file begins with the prefix qi and
is in kwik format (as are headdb and histdb).
- The server forks and execs the otqm queue manager.
- The queue manager reads through the qi files in
otdqueue, copies their information into the headdb and
histdb files, and deletes the qi files.
- The queue manager connects to the otKwikPix storing server for this
project, if one is running, and sends a delta command with
the information to update (see qmRefreshKwikPix() in
otqm.c). The use of the otKwikPix server is
optional.
- When the client is done it sends the 'serverExit' command to the server
to bring down the session.
Update operations are in general quite similar to the enter operation
except that the client sends an 'update xxx' command to the server, where
xxx is the number associated with the CR to update. This causes
the server to check out the file containing the CR and read its content
into an internal representation. If the return value of the 'update' command
is successful (null string) then the client will send a 'tclString' command
(actually a 'tclString orig' command)
to the server, which will cause the server to send as result a TCL program
which when interpreted by the client, causes the current CR buffer on the
client side to be assigned the current header, note and history information
as stored in the server current CR buffer.
Queries are performed by the client as follows. As in the case of enter
and update, once the server host machine is identified, it sends a command
of the form project set motif to the server machine to initialize
various internal project-specific buffers. It then sends a 'control' command
of the form control { xxx }, where xxx is a series of
label/value pairs. For example, the following sets the server's query context
such that only short reports should be sent, and the tclString to use in
queries should by "[stat open]" (print CRs for whom the TCL evaluation of this
returns '1').
control { {fullText 0} {tclString {[stat open]}} }
Here is a list of the labels recognized by OT for control lists:
- skipHeader
- Do not generate a header line for short reports (value 0 or 1).
- count
- Generate a count of CRs instead of the usual report (value 0 or 1).
- summary
- Generate a summary of CRs instead of the usual report (value 0 or 1).
- tabSeparate
- Separate short report fields by tabs instead of spaces (value 0 or 1).
- fullText
- Send the full text of the CR and not a short report (value 0 or 1).
- historyLines
- If set to '1', successful completion of query requires that the history
information be available. If this is so then the otd daemon will
connect to otSlowPix rather than otKwikPix, in which history information is
not available.
- quiet
- Do not print the CR even if it matches the tclString search criterion.
- layout
- A list of the fields to print in a short report.
- keyInText
- Print all CRs with this keyword in the text of the notes somewhere.
- nsens
- If this has a value, then print only the notes whose note sensitivity
field matches it.
- tclFile
- At the moment this hook is ignored. It could someday be used to read
in a one-time-only file of TCL procedures.
- tclOtrc
- At the moment this hook is ignored. It could someday be used to read
in a user's file of TCL procedures.
- tclString
- A string which is evaluated once per CR examined by the otKwikPix (or
otSlowPix) server and whose return string is either 1 (a match - print the
CR) or 0 (do not print the CR). If the user on the client side used ot_bugs
and had an -s argument (e.g., -s stat=open) then this expression will be
translated into one of the above TCL forms and sent over as a value to the
tclString list in this control command.
After the control command is sent to the server, the next command
depends on what kind of report is requested.
- If a single CR is requested, then the 'query n' command is given,
where n is the number of the CR to fetch.
- Otherwise the 'kwikQuery' command
is sent. The kwikQuery command returns data by writing data back directly
to the socket rather than using the TCL-DP layer to return data as return
strings (with tag-length-value style protocol). This was a performance
enhancement.
Configuration files
The primary configuration files for an OT project are the project definition
file and the project metatemplate file. The parsing code for these is in
lib/libot in otProject.c and otMetaTempl.c. The manual page ot_def.5
describes them both. The ASCII graphic below shows the general appearance
of a project base directory.
base directory
(e.g., /project/ot/motif)
+-----------+-----------+---------+---------+-------+------+-----(--+--)...
| | | | | | |
project project otKwikPix queue next CR headdb d00 d01 ...
definition metatemplate socket directory number histdb |
file file +----+--(-+-)...
| | |
motif.def motif.template.def otdqueue number d00 d01 d02 ...
|
+--+--+
CRs: RCS
c000001 |
c000002 c000001,v
c000003 c000002,v
... ...
CR files
The CR data files are stored in directories under the project directory.
The general format of these files is described in the manual page ot.1.
The routines in lib/libot/otTemplate.c parse this format.
kwik format
The header and history information in the headdb and
histdb files illustrates the use of kwik strings. There were
two motivations for the kwik string format.
- The otKwikPix storing server needed to be initialized quickly on startup.
If it had read the total header of each CR file in turn in the data tree it
would have taken about twenty minutes to start otKwikPix.
- All projects do not need the overhead or performance advantage of a
storing server, but they can benefit from optimized access to header
information, even if that is through a file search in which all the header
information is condensed and placed in a file for rapid access.
In general the kwik string format involves separating the header fields
and history fields with special control characters (\013 through \017
as defined in otInt.c). See otKwik.c for details on the
format of the strings. What is most important is that in general the order
of the fields in the kwik string for a project follow the order of the
fields in the metatemplate and there is no tagging in it: thus for a
metatemplate with two fields, "Engineer:" and "Status:", the kwik string
would look like
pnhopen
However, since part of the original requirements for OpenTrack was that
users need to be able to add fields not in the metatemplate, kwik strings
can also contain 'whole fields' (that is, user-contributed fields not in the
metatemplate). If the template is
Engineer:
Status:
and the user edits the template to look like the following:
Engineer: pnh
Status: open
General topic: text utilities
then the kwik string for this will look something like the following:
pnhopenGeneral topic: text utilities
This design of the kwik string was to preserve the performance gain of
the untagged information, while making it possible to reflect information
not corresponding to metatemplate fields.
The history kwik strings are more complicated because there are more
field separators but this implementation is very straight-forward.
It would have been interesting to design the header information
as TCL lists rather than kwik strings, but at the time it seemed as though
the TCL list parsing routines would not be fast enough. This could be
revisited someday.
Application protocol
OpenTrack uses the TCL Distributed Processing library, which emulates a kind
of RPC mechanism so individual TCL commands can be executed on a server.
The message formats are not documented but can be deduced by reading
lib/libdp/dpnetwork.c (see Tdp_PacketSend()) and lib/libdp/dprpc.c
(see Tdp_RPC() and Tdp_ProcessRPCMessages()).
The client-side routine otRemoteTcl() (in lib/libot/otNetwork.c) passes
a command string (e.g., project set motif) to the RPC mechanism via the
routine Tdp_RPC(). The RPC routines in lib/libdp tag the message as an
RPC message and pass it to the 'packet level', which is a layer of routines
which encapsulates the RPC message as a TCL-DP packet with a message format of
magic number, length field (32 bits) and (null-terminated) command string.
When the server (otd) returns either a TCL result string or an error status
and error message, the packet is parsed by the packet layer
(Tdp_PacketReceive() in libdp/dpnetwork.c) and the resulting message is
passed up to the RPC layer, which removes either the 'return' token or the
'error' token from the front of the message.
The most important aspects of the protocol have been discussed in the
Data flow section. Here are some other salient points of the protocol:
- At the beginning of a session, the client may enhance the project search
path of the server by sending a command like the following to the server:
set env(OT_DBPATH) /project/x
This would cause the server to search in the directory /project/x to find
a project, if it could not be found in /project/ot first. This illustrates
the most characteristic aspect of this protocol. Its commands are those of
the default TCL set with application-specific extensions. As a consequence,
it can be extended by adding commands in lib/libot/otTcl.c.
- When the 'kwikQuery' command is given, the server no longer uses
the TCL-DP routines for sending return string back to the client. Instead
it finds the socket connected to the client and sends query data back
directly as a stream. This was a technique employed to enhance
performance.
- Unlike the 'kwikQuery' command, the query command never circumvents
the TCL-DP protocol. It always returns a result and error code.
It has three forms.
- With a number argument ('query 4'), it returns a formatted version of
the CR.
- With no argument, it initiates a query and returns a header string
appropriate as the first line of a short report.
- With 'next' as argument (e.g. 'query next'), the next group of reports
is sent back as a return string.
- The client brings down the server with the 'serverExit' request.
Note that the file on each server machine
/project/ot/serverV3.log keeps a copy of everything sent by the
client to the server. This can be used to get a sense of the protocol.
Other ways to become more familiar with it is to examine instances of its
use, as in the otmig script in util/scripts.
Process structure
OpenTrack is a client-server application. The server also communicates with
other processes in performing enter, update and query operations.
Client and server
The OT client is one of ot, ot_bugs or otcl. The otcl program may be used
to write scripts. See otmig in util/scripts for an example of
one of these scripts.
The otd server is exec'd by /etc/inetd like most UNIX servers.
The only unusual aspect of the connect procedure is that in lieu of using
the getserverport...() routine or hardcoding the server port for
otd, the current OT clients check the 'server port' value in the project
definition file for the project whose server they are connecting to.
Note that while the ODEXM server protocol is somewhat like rsh in
that it must execute child processes to execute commands requested from
a client, the otd daemon is not like rsh: in general it does not execute
processes to execute the commands, but rather the support is built in to
the otd binary.
Like INN, OpenTrack can use a storing server (otKwikPix) which keeps an
in-core copy of the header data at any one time. Note that this is just a
performance enhancer and that if the otKwikPix server is unavailable
(i.e. the OT server otd cannot connect to the UNIX socket on which otKwikPix
is listening) then otd will fork and exec /etc/otSlowPix, which will provide
the same information. The interactions between these processes is
represented in the ASCII graphic below.
+----------------------------------+
| ot - OT client |
| (also ot_bugs, otcl) |
+---------------+------------------+
client | TCP-IP socket
|
------------------------------------------------------+--------------------
|
+------------+ server +---------------+------------------+
| | | |
| otKwikPix | | otd - OT daemon |
| | | |
|stores +----+ <-------+daemon handles Tcl-DP protocol |
|header | | / | from client |
|information | | / | - connects to otKwikPix for query|
|as array of | | / | - forks/execs otqm for enters,upd|
|OTTemplate | | / +---------------+------------------+
|structures | | / | (otqm is exec'd on
| | | / | enter/update only)
+------------+ | / +---------------v------------------+
|/ | ot queue manager (otqm) |
o<---------------+ |
UNIX socket | - reads header, history info |
(e.g. /project/ot/motif/socket) | deposited by otd in otdqueue |
on which otKwikPix listens after | - merges info in headdb,histdb |
initialization for | - connects to otKwikPix to |
- incoming connect from otd for queries | refresh stored information |
- incoming connect from otqm for +----------------------------------+
refreshing stored info
OT queue manager
The queue manager otqm exists in order to update the headdb and
histdb databases. These files are summaries in kwik string
format of the header and history information in the CRs, such that the first
line in the file represents the nth CR, followed by the (n-1)th and so
down to the end. This scheme was chosen because it offers simplicity and
the fastest speed when accessing the whole file, as would be done by
otSlowPix if otKwikPix were not running. Updating one line during a commit
operation in one of these files, which can run many megabytes in size, is
time-consuming and thus must be done asynchronously by a separate process,
otqm. The otd server deposits a queue item file in the otdqueue directory.
The name of the file starts with 'qi' and the file itself is composed of
two lines: the first is the header kwik string and the second is the history
kwik string.
The queue manager otqm is forked by otd (in cmdCommit(), see
otTcl.c). otqm searches the directory for qi files and when it
finds one, it integrates the kwik strings it contains into headdb
and histdb and attempts to connect to otKwikPix to refresh its
cache (via a 'delta' command intercepted by otKwikPix - see the sources).
otqm uses the file otqm.lok immediately under the base directory
to prevent other otqm processes from being exec'd during this operation
(cf. lpd). When all the qi files have been processed otqm exits.
otKwikPix and otSlowPix
The otKwikPix server should be started either manually or in a
system rc file on startup. Each otKwikPix process can only handle
information for one project. On startup it initializes an array of OTTemplate
structures by reading the data from the headdb file.
It then listens on a UNIX socket under the project base directory for
connects from one of two places:
- From otd for queries
- From otqm, to update stored CR information, or add new CR header
information
The otd server must choose whether to connect to otKwikPix or to fork
and exec otSlowPix as a child process. Since the otKwikPix server does
not store the history information (because the memory requirements become
truly exhorbitant), otSlowPix must be consultant if the user wishes to
use history information during a query. This is indicated somewhere in
the begin condition of the query procedure by giving a command such as
control { {historyLines 1} }
The manual page ot_tcl.1 has an example of the use of the control
command to accomplish this (see "History Pseudofields").
If the historyLines property is 'on' for this query, then otd will choose
otSlowPix.
The other condition under which otSlowPix is selected is if otKwikPix is
unreachable or not running.
Both otSlowPix and otKwikPix return the same information for the same
requests. The otd server always sends a control command to determine the
query.
- If a short report is requested and there is no keyword to match (-k in
ot_bugs arguments), the matching CRs are formatted when
they are inspected and the search criteria satisfied, and the report is
returned to otd and thence to the client.
- If a full report is requested or there is a keyword to match, then
the numbers of the CRs are returned to otd, which then reads the full
CR from the dxx/dyy directory, searches for a keyword if there is a keyword
to match, formats it in the case of a match and sends the data to the
client.
With regard to the sources, the relationship between otKwikPix and otSlowPix
is similar to that between ex and vi; otSlowPix is a link to otKwikPix.
The binary inspects the value of argv[0] to determine which image was
intended and determines the method of access. If the command name
is 'otKwikPix', it reads all information into an array of OTTemplate and
waits for connects. If the command name is 'otSlowPix', it reads each
entry from headdb and if necessary from histdb one
at a time and evaluates the search criteria (TCL procedure).