Ganymede Release 1.0.12 March 1, 2004 CHANGES --------------------------------------------- -------------------- Changes from 1.0.11 to 1.0.12 ------------------- RELEASE DATE: March 1, 2004 1. [CLIENT] Properly update StringSelector count display on manual item entry The StringSelector widget used to manipulate vectors of strings and object references in the client was not properly updating the item count when an item was manually entered through the text box. 2. [SERVER] Fixed regression in object event label display I had neglected to edit the objectEventCustom.java class to cohere with the change described in CHANGES 1.0.11#8. 3. [SERVER] Fixed exception vulnerability in admin console, scheduler code We had an occasion in which the Ganymede scheduler terminated during the server's normal operations, as a result of cascading failure in the admin console notification system. The changeStatus() method in serverAdminProxy was not properly checking the done flag, so a NullPointerException was thrown where the GanymedeAdmin code was only expecting the possibility of a RemoteException. The NullPointerException was being thrown up through the Ganymede.debug() call frame, which led to the scheduler being killed. I have fixed the serverAdminProxy.changeStatus() method so that the NullPointerException will not re-occur, and I have fixed the Ganymede scheduler so that nothing but an actual interrupt on the scheduler thread will cause the scheduler to terminate. 4. [SERVER] Turned off extraneous debug output from XML processing logic I had accidentally left the debug flag enabled in the xmlobject and xmlfield classes, which caused a lot of extra output to be displayed when processing XML files through the xmlclient. 5. [SERVER] Removed 1.4-only StringBuffer method usage from DBEditObject.java It turns out that java.lang.StringBuffer gains a method overload that executes special code in the event that the parameter passed to append is itself a StringBuffer. When I compiled the Ganymede code with a 1.4.1 JDK, code that previously was interpreted to have an implicit toString() method call now uses the optimized StringBuffer call, which doesn't exist in JDK 1.3 and earlier. Putting the toString() in explicitly forces javac to use the String parameter method, which is present in all versions of the JDK. This fix should make it possible to run the Ganymede server on JDK 1.3.1 or earlier again. Thanks to Maura Mathieu (maura.mathieu@sdis66.fr) and Kent Wick (kent.wick@mhmr.state.tx.us) for reporting this. 6. [SERVER SCRIPTS] Fixed RMIGC definition in runServer for ksh Per Kent Wick, the RMIGC definition in the runServer templates has been surrounded by single quotes. 7. [SERVER] Preparations for logging to PostGreSQL The Ganymede log classes have been reworked in preparation for supporting SQL databases for log event storage and retrieval. 1.0.12 is not SQL-ready, quite, but I've done some refactoring to allow for a variety of DBLogController implementations. I've implemented a DBLogPostGreSQLController class for logging to PostGreSQL with an appropriate schema, but I'm not including the appropriate support files necessary for initializing the database and what-not. Next release if all works out well. 8. [SERVER, CLIENT] Reworked client and server to support direct login In all previous versions of Ganymede, the login process is a two-step dance.. first the client (GUI or xmlclient) calls up the appropriate login method on the server, passing a reference to a callback object hosted by the client, then the server uses that callback object to query the client or xmlclient for username and password. I initially did this (6-7 years ago) to throw a (vanishingly small) roadblock into the path of people trying to packet sniff the login process, and so that people wanting to do a replay attack would have to emulate the full RMI protocol. This was really always complete silliness, of course. Despite that, I had no real motivation to rework the login process until David Cummins reported that he was unable to use the Ganymede client from his firewalled desktop systems. Given the increasing prevalence of defensive firewalling on the local system level, it seemed like it was time to rejigger the Ganymede login process. Users of a locally firewalled system will now be able to log in and use Ganymede, but the experience will still be second rate.. the server won't be able to send any build status messages to the client, and if the server needs to timeout or force disconnect the client, the client won't get the message. NOTE: One side effect of this change is that certain methods in the Ganymede server are now marked as throwing a 'NotLoggedInException' condition. If you have custom code that calls any of these methods, you may see build problems when you go to rebuild your code for use with Ganymede 1.0.12. Surrounding the offending calls with an appropriate throw..catch clause should be all you need to do to make your code compatible with 1.0.12. 9. [CLIENT] Improvements to text messages in dialogs The common composite dialog class used in the Ganymede client and admin console has been improved to support adaptive word wrapping and scrollbars. This change was motivated by the client's new ability to report remote exception traces in a dialog. 10. [CLIENT/CONSOLE/SERVER] Reworked RMI API to be system firewall transparent This is an extension of the 1.0.12#8 work. It used to be that the server used RMI calls to the client and admin console to send notification for build status and force off (on the client) and all of the data displays on the admin console. The problem with this is that it required the client/console to open a socket and wait for an RMI call from the server to send update information. I have now reworked the server, client, and admin console so that all transmission of data from the server to the client is done by having the client/console run a polling thread which repeatedly calls an asynchronous message port on the server. Since the data is flowing through the result of a forward RMI call, no socket needs to be opened up for listening on the client/console system, and full functionality is achieved even in the presence of a client-side system/personal firewall. This change also speeds up login and noticeably reduces the memory loading on the server. A win/win/win deal. 11. [CLIENT] Fixed gclient.getObjectHandle() The getObjectHandle() method in the client's gclient class didn't properly deal with an empty list back from the GanymedeSession's queryInvids() method. This could result in an exception being thrown rather than a dialog being shown in certain circumstances in the client in which a user tries to edit an object reference that he doesn't have permission to edit or view. 12. [SERVER] Improved namespace conflict message The DBField class in several places had code to report namespace conflicts. Created a common method to generate a more detailed error dialog which informs the user what object/field is conflicting. 13. [SERVER] Cleaned up DBObject.getTypeName()/getTypeDesc() The DBObject class had two methods, getTypeName() and getTypeDesc() which did exactly the same thing. Embarassing. I went through and changed all instances of DBObject.getTypeDesc() to DBObject.getTypeName() calls, in keeping with the db_object interface. 14. [CLIENT] Added row count indicator to query window In response to a so-simple-it's-brilliant suggestion by David Cummins, I have tweaked the Ganymede client to display a count of the rows returned in the title bar of the client's query results window. 15. [CLIENT] Various tweaks for better behavior under JDK 1.5 beta I've made some slight tweaks to the client and console's GUI layout to improve behavior when running under the JDK 1.5 beta. This includes modifications to the JNLP launch files in webforms so that the user will be given the option of using versions 1.2 through 1.5 of the Java runtime. -------------------- Changes from 1.0.10 to 1.0.11 ------------------- RELEASE DATE: December 9, 2002 1. [SERVER] Put safety interlock in DBEditSet.addNewObject() The DBEditSet class had a vulnerability that could conceivably cause transaction corruption if an adopter wrote custom code that attempted to pull new objects in to the transaction in a custom DBEditObject classes commitPhase2() method. Now addNewObject() will throw a RuntimeException() if someone attempts to pull new objects into a transaction during the transaction's commit() process, causing the transaction to cleanly abort. 2. [CLIENT] Made the Ganymede login applet show the server name Added information on the login box about the location of the Ganymede server that the client has bound to. 3. [SERVER] Added new ganymede.subjectprefix property to the server.properties file The server now supports a new server property, entitled 'ganymede.subjectprefix'. If this property is set to a string that begins and ends with single quotation marks, the text included between the single quote marks will be used as the identifying subject prefix for email sent by the Ganymede server. If no such server property is defined, the server will use 'Ganymede: ' as the subject prefix. 4. [SERVER] Fixed GanymedeSession addResultRow(), internalQuery() to honor editableOnly query filtering The query logic in GanymedeSession has always been grotesquely over complicated, and it turns out that there has been a significant error in the addResultRow() method, which was neglecting to filter out non-editable objects from queries that specifically request that non-editable objects be filtered, in the case that GanymedeSession.internalQuery() was used to execute the query. In addition, the addResultRow() logic in the old non-supergash, non-internal case was confused and confusing, so I cleaned that up a bit as well. I have swept through my code looking for instances of internalQuery() usage, and I have made sure that all queries supplied to internalQuery() in non-supergash sessions explicitly turn off the editableOnly restriction. This involved some changes in the schemas/gasharl/custom_src code. 5. [SERVER] Modified DBObject.checkRequiredFields() to check all fields The checkRequiredFields() method in DBObject was only checking the custom-specified fields, rather than all fields. This made it impossible for a DBEditObject plugin's fieldRequired() method to enforce field requirements for the expiration and removal fields. 6. [SERVER] Optimized XMLReader and serverAdminProxy slightly Reworked both XMLReader and serverAdminProxy classes to use an efficient circular buffer rather than continually doing System.arraycopy operations within a Vector. Won't make any noticeable difference, but after seeing Doug Lea's optimized concurrency structures (at http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html), I felt like tinkering. 7. [SERVER/XML] Fixed feedback on transaction failure in the xml subsystem The Ganymede server's XML transaction logic was faulty, and would attempt to report a successful commit even in the circumstance where the commit-time transaction verification logic caused the transaction to be rejected. Now, an xmlclient transaction that could not complete due to consistency problems during transaction commit will no longer report a confusing mix of success and failure messages. 8. [SERVER] Changed plug-in API for object labels The Ganymede object model has always had a rather poorly thought out implementation of object labeling, with two separate means of expressing object labels. The first was by designated label field, the second was by calculated label field using the DBEditObject plugin's getLabelHook() method. There were a number of problems with this. First, the Ganymede query engine completely ignored getLabelHook()-derived labels, and if an object type had both a designated label field and a getLabelHook() method in a DBEditObject plugin, the getLabelHook() was used for many but not all things.. querying on an object's label simply ignored the getLabelHook() method's output and searched on a label field if defined, or simply failed if no label field was assigned. Now, the Ganymede query engine will check the new useLabelHook() method for a plugin, and if that method returns true, the label field will be ignored in queries, in favor of the results from getLabelHook(). To prevent things from being too confusing, the DBObject getLabel() method will now also consult the new useLabelHook() method, and if useLabelHook() does not return true, the getLabelHook() method will simply not be used under any circumstances. 9. [SERVER] runServer now specifies less frequent full-heap GC Since 1.2, Sun's HotSpot JVM (server-mode) has incorporated generational garbage collection, which allows the JVM to perform limited-scope garbage collection passes over recently created objects relatively frequently, while avoiding full heap garbage collection passes unless and until the full heap as a whole is filled. This allows the CPU loading due to garbage collection to be dramatically reduced, relative to a more naive full-heap garbage collector. Unfortunately, the Ganymede server has in the past not been benefitting greatly from generational garbage collection, due to a peculiarity of the RMI system. By default, RMI programs that access remote objects (as the Ganymede server does to asynchronously send messages to the admin console and to the gui clients) do full-heap garbage collection on a periodic basis (once a minute) in order to quickly release any remote RMI objects. This extra garbage collection activity is useless for the Ganymede server, as by design it will never relinquish references to the few remote objects it talks to until such time as the client or admin console disconnects/quits, whereupon the issue of client-side local garbage collection becomes moot. I've recently become aware of an RMI system tuning parameter for recent JVM's which allows this RMI full-heap gc interval to be tuned. I have edited the runServer script in the Ganymede distribution to set this tuning parameter to do full-heap gc once an hour. Previously, our Ganymede server was taking around .5 seconds every minute doing the full-heap GC cycle, now this overhead is only done once an hour, at the cost of a rather greater proportion of the server's allocated heap size 'in-use' on average. All of this is documented in the new runServer script, and this parameter can be tweaked as you like. See http://java.sun.com/docs/hotspot/gc/ for complete details. 10. [CLIENT] Several small improvements to the Ganymede user interface You can now hit enter after typing a search value in the query box and the query will be submitted for you. This should save people from having to mouse down to the 'submit' button for each query. You will still need to manually mouse down to the 'add choices' box if you want to add an additional search constraint. In addition, if a query returns no results, you'll now have the option to pop the query box right back up, revise your query, and resubmit. The GUI widget used for changing passwords in Ganymede will now provide a quick dialog box telling you that you didn't type the password the same way twice, rather than silently clearing the fields and having you try again without any guidance as to why. Thanks to Marcus Walker and Tom Embleton for these suggestions.. I hope the changes I made help out. 11. [SERVER] Improved logging for newly created objects I slightly reworked the commit_createLogEvent() method in DBEditSet so that newly created objects will use DBObject.getPrintString() to create a full summary of the newly created object, as is done when such objects are deleted, rather than using the DBEditObject.diff() method. This has the advantage that newly created objects which contain embedded objects will include full information on the embedded objects. 12. [SERVER] Reworked ownership inheritance The support in GanymedeSession's recursePersonaMatch() method for recursing up the ownership tree to find matching personas was broken in the case when an owner group contained no members. In such cases, the upwards recursion stopped inappropriately. This was a pretty major logical error, and one which has left support for the Ganymede owner group hierarchy broken for quite a long time. Upon looking at the code, I found a lot of gross stuff in the whole owner group permission handling logic. I went in and committed major surgery on the owner group code. The code is now simpler and less prone to security problems due to inappropriate caching. In addition, there may be a noticeable decrease in the time taken when a new object is created, as the owner group list calculation code is far more efficient now. 13. [SERVER] Tweaked object creation ownership behavior Previously, the Ganymede server would complain any time someone tried to create an object with more than one default owner selected. Ganymede will now allow this, but will complain if no owner list has been manually selected in the interactive case. Ganymede will now provide a warning if it had to pick an owner group at random to put a newly created object in when run with a non-interactive client. 14. [CLIENT] Added menu options to view object window, cleaned dialogs The view object window now has several menu options to allow editing, deleting, cloning, and inactivating of objects from the view object window. Cleaned up several dialogs in the code. -------------------- Changes from 1.0.9 to 1.0.10 ------------------- RELEASE DATE: April 2, 2002 1. [SERVER] Consistency checks should not be forced if 'enableOversight' not set We were having some problems with the nightly expiration task failing due to a consistency check failure unrelated to the actual expirations at hand. This was due to the default DBEditObject commitPhase1() method calling consistencyCheck() on each object involved in the expiration transaction, despite the expiration task's having elected to turn off enableOversight. Now, if the GanymedeSession's enableOversight flag is turned off, the default commitPhase1() method won't bother to run consistencyCheck(), and will simply return a positive success value. This comes in handy, as the nightly GanymedeExpirationTask runs with enableOversight turned off, so that we can allow object expiration/removal that would break a required field or other consistency guarantee if need be, rather than having the nightly task be blocked from committing its changes if anything would lead to a slight inconsistency. Note as always that the default DBEditObject commitPhase1() method can be overridden with custom logic if it is not desireable to have any consistency checks skipped in response to the enableOversight flag being reset. See the next item for more. 2. [SERVER] Made default transaction commit-time consistency check smarter Previously, the default DBEditObject.commitPhase1() method rejected a commit if the object's consistencyCheck() method failed. Now, commitPhase1() goes a step further, and only returns a failure code if the object's consistencyCheck() fails, and the consistencyCheck() succeeds for the original version of the object going into the transaction. That is, if a user is processing a transaction and happens to involve in his working set an object which was previously inconsistent, we won't hold it against him and force him to correct the previously existing inconsistency. Instead, we will depend on the GanymedeValidationTask to detect such conditions for a knowledgeable admin to fix, see the next item. 3. [SERVER] Improved GanymedeValidationTask to run consistencyCheck() The GanymedeValidationTask scans through the database checking objects for consistency. Previously, however, the only check this task was performing was to see if any required fields were missing. Now, the GanymedeValidationTask will also call consistencyCheck() on each object, allowing inconsistencies that have creeped in from transactions run with enableOversight turned off to be easily detected on demand. 4. [SERVER] Turned off passive password capture Turned off passive password capture. Ganymede stores passwords in a variety of hashed forms, which are strings that can be used for validating a password but not which can't reveal the actual text of the password without the use of a brute-force encryption search. Plaintext is never saved on disk unless the password field is specifically configured to do so in the Ganymede schema. Things like sync'ing passwords to Windows NT which require plaintext passwords have to obtain them from temporary storage in the Ganymede server. Formerly, whenever a user logged into Ganymede, Ganymede would temporarily acquire and retain the plaintext used to login to Ganymede in RAM only. This plaintext could then be passed to Windows NT for synchronization. In this way, users would not necessarily have to change their passwords in order for their account to be sync'ed. Unfortunately, it turns out that the password identity validation is not perfect. A UNIX password hashed using the traditional Unix crypt() routine is only significant to 8 characters. If a user logged into Ganymede for the first time since the Ganymede server was restarted with a mistyped version of their password that was correct to 8 characters but then diverged in the ninth, Ganymede would remember the incorrect plaintext, and insist on seeing it again before allowing the user to login until such time as the server was restarted, and the plaintext password information was lost. Worse, such carelessly entered passwords could get propagated to our Windows NT domain controller in their erroneous form, and Windows NT's password hashing is significant to 14 characters. This is not a reasonable or tenable situation, so the passive plaintext password capture logic has been disabled. 5. [SERVER] Better error message on link permission failure The Ganymede server will now explicitly report a permissions problem when an attempt is made to link to an uneditable object. 6. [SERVER] Re-enabled optional mailout event logging I had disabled logging of mailout events, to keep the Ganymede log from being flooded by non-critical email sent out by the Ganymede server. Unfortunately, the inevitable question arose here about whether Ganymede did or did not send out an email message warning about an upcoming account inactivation. Having a log of all advisory email sent out by Ganymede would have helped clarify the situation, so this function is being added back to the Ganymede server as an optional logging facility. Now, if the Ganymede server's ganymede.properties file defines a ganymede.maillog property, the file named in that property will have all mailout events logged to it. If the ganymede.maillog property is not set, no mail events will be logged on disk. By default, the server's ganymede.properties file is installed with no ganymede.maillog property defined. If you are using a schema kit that sends out a lot of mailout events, you may only want to enable mail event logging for short periods as a diagnostic. As with all properties defined in the ganymede.properties file, the server must be stopped and restarted for any changes in the properties file to take effect. 7. [SERVER] Improved logging/email for embedded object modifications All informational logging of changes made to embedded objects now includes the name of the top-level object which holds the embedded object. For our usage, this makes it possible to tell what system was involved when an interface definition was changed. 8. [SERVER] Clean ups and optimizations to the admin console handling I found that there was some really ugly code in the GanymedeAdmin class that handles communications to and from attached admin consoles. I substantially reworked this class for better encapsulation, and while doing so wound up removing an extra RMI call that the server was attempting to make to each admin console for every line of log information sent to the console. The serverAdminProxy was eliminating a lot of these calls if lines of logging information were being sent rapidly to the admin consoles, but there was no need to have the server attempt to send so much unnecessary data to the attached admin consoles. 9. [SERVER] Shutdown sequence refinements In the process of doing work for #8, above, I introduced a class cast exception into some of the admin console code that resulted in the server getting stuck during shutdown. In fixing this, I revised the shutdown sequence to be more verbose and to ensure that arbitrary exceptions thrown late in the server shutdown process won't prevent the server from exiting. 10. [CLIENT] Cleanups to the GUI calendar widget I did a cleanup job on the JpanelCalendar class, to make the thing easier to read and maintain, and to improve its layout in the face of window sizing. When viewing, rather than editing, objects' expiration and removal date fields, the client will now show a full-size, display-only calendar rather than the cheap text label that it used to show in that circumstance. When editing these fields, the old label showing what the expiration/removal field was set to in the calendar panel can now be used to jump the calendar back to the calendar page containing the selected time. 11. [SERVER/XML] Liberalized the server's behavior with xmlclient deletions The server was previously rejecting any XML transaction that attempted to delete a non-existent object. Now, the server will merely transmit a warning to the xmlclient that an attempt to delete a non-existent object was attempted, without completely rejecting the valid deletion commands. This change was made in response to a comment by Gaurav Bhargava at Villanova University. 12. [SERVER] Fixed slight inconsistency in object link logging The Ganymede server's logging and email notification logic had a small inconsistency in it. When logging changes to invid (object reference) fields, the Ganymede server logged old values removed using the label for those objects that pertained at the end of the transaction in question. If an object's name changed during the transaction due to other editing, something like the following would be logged: objectchanged ------------- I.P. Network ATL - Outgoing Inet Access Only, <267:96> was modified. Fields changed: Records Deleted: [10.5.16.132 - 08:00:20:9a:16:f2] objectchanged ------------- I.P. Network ADG/OMG - Outgoing Inet Access Only, <267:79> was modified. Fields changed: Records Added: [10.5.16.132 - 08:00:20:9a:16:f2] objectchanged ------------- System corona's Embedded System Interface, '[10.5.16.132 - 08:00:20:9a:16:f2]', <265:970> was modified. Fields changed: I.P. Address Old: 10.8.16.186 New: 10.5.16.132 Ethernet Info Old: 8-0-20-9a-16-f2 New: 08:00:20:9a:16:f2 I.P. Network Old: ATL - Outgoing Inet Access Only New: ADG/OMG - Outgoing Inet Access Only In this case, the 'Records Deleted' part of the 'ATL - Outgoing Inet Access Only' objectchanged log entry is showing the new name for the interface that was deleted, even though when that interface was in the ATL network, it had a different name. This has been changed so that whenever an object is removed from an invid field, the removal will be logged using the object's name as of the start of the transaction. This would make the above logging sequence look like this: objectchanged ------------- I.P. Network ATL - Outgoing Inet Access Only, <267:96> was modified. Fields changed: Records Deleted: [10.8.16.186 - 8-0-20-9a-16-f2] objectchanged ------------- I.P. Network ADG/OMG - Outgoing Inet Access Only, <267:79> was modified. Fields changed: Records Added: [10.5.16.132 - 08:00:20:9a:16:f2] objectchanged ------------- System corona's Embedded System Interface, '[10.5.16.132 - 08:00:20:9a:16:f2]', <265:970> was modified. Fields changed: I.P. Address Old: 10.8.16.186 New: 10.5.16.132 Ethernet Info Old: 8-0-20-9a-16-f2 New: 08:00:20:9a:16:f2 I.P. Network Old: ATL - Outgoing Inet Access Only New: ADG/OMG - Outgoing Inet Access Only This misfeature was reported by Gil Kloepfer, here at ARL. 13. [CLIENT] Put warning dialog up when editing inactivated objects A good number of admins here at ARL have ran into problems when they tried to manually edit inactivated users, rather than using the reactivation function that is available on the object tree right-click menu for inactivated objects. The problem is that most admins here just set the password and commit the transaction, without realizing that they need to clear the Removal Date field. Our server is configured to provide a 'reactivate' wizard when users use the right-click reactivation function, but users too often don't realize or remember that that function exists. In response to this, I have implemented a dialog box which will pop up if an admin attempts to edit an inactivated object, prompting the user to attempt to use the server's reactivation wizard rather than directly editing the object. This is pretty clean, but this will put more pressure on schema kit developers to insure that if they want an object type to be able to be inactivated rather than simply removed, they really will need to provide a reactivation function. 14. [CLIENT] Synchronized the transaction cancel button to avoid exceptions The gui client now synchronizes its cancel button, to ensure that all object windows (which load data from the server on their own threads) have stopped loading before the client tells the server it wants its transaction cancelled. Previously, some of the client's loading threads could briefly race, continuing to try and call data loading methods on the server after the client's main GUI thread had already cancelled the transaction, which could cause the server to throw an exception down to the client. This change just serializes transaction cancellation on the client side a bit better. 15. [CLIENT, ADMIN CONSOLE] Miscellaneous GUI cleanups, improvements Quite a varied collection of small improvements in the GUI client and the admin console. In the client, the history panel now uses the OneTouchExpandable option in its splitter panel, and sets its initial size so all the historical fields are visible. The expiration and removal tabs now show the appropriate icon from the tree when they are present. This will hopefully make it less likely that users will miss noticing that these tabs are present when editing certain user accounts. The client's vertical splitter supports the OneTouchExpandable option as well, making it possible to entirely hide the tree if more space is needed for showing object windows. The admin console's layout has changed very slightly, with a splitter bar added between the log panel and the bottom status tables panel, and the 'clear log' button removed and put in the Control menu, where it won't take up screen real estate. 16. [SERVER] Better exception handling in checkpointed code The object linking code was not properly catching exceptions that might be thrown by plug-in code, and so certain object linking and unlinking operations could leave checkpoints on the server's checkpoint stack. Due to the thread identity checking logic in the DBEditSet checkpointing routines, this could cause a session on the server to refuse any further checkpointing, causing the user to have to cancel the transaction. Now, if any user-level code throws exceptions during object linking or unlinking, the checkpoint stack will properly be cleared. This is in response to a bug report by Gil Kloepfer. 17. [CLIENT] Fixed the clients' failure to terminate on window closure The gui client and admin console were not properly terminating the JVM when the clients were run as applications rather than applets, and the user closed the window using the operating system controls rather than the 'quit' button. The login box will now properly terminate the JVM if the close window button is used. This is in response to a bug report by Gil Kloepfer. 18. [SERVER] Cleanups and refinements to DBEditSet class The transaction commit() method code in DBEditSet has been radically reworked. The biggest part of this reworking is a simple refactoring to make the code far simpler to read, but a number of reliability improvements were made in the process, and a few extremely unlikely yet nonetheless possible security race conditions were fixed. In addition, a lot of DBEditSet member fields have been made private, and special accessors have been added and method calls altered where needed to allow for interaction with other classes. 19. [SERVER] Slight API change for DBEditObject subclasses Previously, the DBEditObject class held a 'committing' boolean flag that needed to be set and cleared in specific methods in order to coordinate field locking during the transaction commit process. The problem is that access to this boolean flag was unsynchronized, making it possible in theory for a system running the Ganymede server with multiple threads on multiple CPU's to allow changes to be made to fields after the commit process has started for a given object. This race condition is extremely unlikely, as it would depend on a system having a non-coherent memory access model in the absence of specific object synchronization, but the Java Language Specification does allow for such implementations. In any case, the committing boolean has been replaced by an arlut.csd.Util.booleanSemaphore object, and made private. Subclasses of DBEditObject which override commitPhase1() and/or release() should now use the DBEditObject setCommitting() method to set or clear that flag. None of the custom object plugins or schema kits that we have provided overrided those two methods, so I think it'll be unlikely that this API change will affect people, but if you are not able to recompile your schema kit and/or plugins with this version of the server, you may need to look at this issue. 20. [SERVER] Shutdown Expedited The GanymedeBuilderTask base class will now inhibit any builder tasks from running at all once a server shutdown command has been issued. Previously, the server shutdown process only blocked phase 2 builds. This change makes it so that the server won't bother trying to write out data files that it won't do anything with before shutdown. 21. [SERVER, ADMIN CONSOLE] Added support for Apache md5 hash format The Ganymede PasswordDBFile class now has support for working with passwords hashed using the variant Md5Crypt() algorithm that the Apache web server supports for its htpasswd files. Implementing this support involved changes to the Ganymede database file format (now up to file format 2.4), along with changes to the RMI interface definition for the schema editor and for the password field interface. Because of these changes, Ganymede 1.0.10 will not work properly with an admin console or client from Ganymede 1.0.9 or earlier. -------------------- Changes from 1.0.8 to 1.0.9 ------------------- RELEASE DATE: December 4, 2001 1. [SERVER] Fixed xml parser error introduced in 1.0.8 The server side XML parsing code in Ganymede 1.0.8 had a major bug. The 'addIfNotPresent' vector field mode was made the default, but support for handling the addIfNotPresent mode was not implemented for invid vector fields. Loading data into invid vector fields silently failed. This has been fixed, and the xml file processing now works properly for all kinds of vector fields. In addition, the server's xml code now reports whenever it sees an element that it is not able to resolve during the data consolidation phase. Apologies to all and thanks to Gaurav Bhargava at Villanova University for reporting this brown paper bag bug. -------------------- Changes from 1.0.7 to 1.0.8 ------------------- RELEASE DATE: November 16, 2001 1. [SERVER] Fixed Invid, String, IP array fields disk i/o to handle large arrays The InvidDBField, StringDBField, and IPDBField classes support array values for object pointeers, strings, and IP addresses, respectively. Prior to this release, the length of these arrays were limited to 32k entries, as a 16 bit signed short was used to store the array length on disk. This has now been changed to a 32 bit signed int, so that Ganymede's disk format can now handle 2 billion items per array. That doesn't mean that the server will be able to behave well with 2 billion items in a String field, but there at least won't be an artificial cap of 32k items anymore. This will be especially valuable in that owner groups will no longer be limited to 32k items per group. This is an incompatible change with versions 1.0.7 and earlier of the Ganymede server. ganymede.db and journal files written by Ganymede 1.0.8 and later won't be readable by Ganymede 1.0.7 and earlier. 2. [SERVER] Fixed scheduler to allow enabling/disabling of on demand tasks The Ganymede scheduler code to handle task enabling/disabling from the admin console had a bug that prevented it from being able to handle on demand tasks. This has now been fixed, and any kind of task should be able to be disabled. This can be useful if you want to commit changes to the Ganymede database without having the server trigger the builder tasks. 3. [CLIENT] Fixed exception in timed out querybox The querybox code was improperly retaining a reference to a client-side cache structure, making the re-use of the querybox impossible after the client was forcibly timed out. The querybox now takes care to get an updated reference to the caching structures it needs, so that re-use of the client querybox after a forced time out will not cause an exception to be thrown. 4. [SERVER/XML] Improved password importing support in the XML system The server's xmlfield code was missing support for processing the "lanman" and "ntmd4" XML attributes for the element in the Ganymede XML file format when loading data. This support has now been added, and it should now be possible to write scripts to load user accounts, passwords included, from a Samba password file. In addition, the XML client can now simultaneously load passwords in several hash formats, so if you already have a Samba file with LANMAN and NTMD4 password hashes along with your UNIX passwd file's crypt and md5 crypt passwords, you can load all four hashes into accounts in Ganymede at once. Ganymede obviously can't verify that the hashes match in this case, but if you already have been synchronizing passwords to different hash formats, this will let Ganymede support your accounts without requiring that your current passwords be available in plaintext. This entails a slightly incompatible change in the RMI API between the client and the server, so the Ganymede 1.0.8 and later xmlclients should not be used with a 1.0.7 or earlier server. 5. [CLIENT/SERVER] Fixed query filtering in Ganymede client and server The Ganymede client has for a long time had a (nonworking) 'Filter Query' menu option that was intended to allow users in many owner groups (including supergash) to set a filter so that only objects owned by a given list of owner groups would appear in the tree. It works now, but it's a touch inelegant. Ultimately, the answer to overloading in the tree is to force a hierarchical data representation, but until such time as we can rework the Ganymede data model that deeply, you can apply and clear owner filters using the "Set Owner Filter" menu item from the client's File menu, and the tree and object choice lists in objects being edited will be dynamically updated. Voila, 'supergash' level power while still being able to 'drill down' on a reasonable subset of the server's data. This entails a slightly incompatible change in the RMI API between the client and the server, so the Ganymede 1.0.8 and later clients should not be used with a 1.0.7 or earlier server. 6. [SERVER] Fixed nullPointerException throw on empty regexp string Fixed a couple of places in DBQueryHandler that could result in a null or empty string being passed down to the gnu.regexp.re constructor. 7. [XML/SERVER] Improved semantics of XML vector field handling Previously, the default mode for vector fields in Ganymede's XML support was "add". That is, the following XML snippet will cause "jonabbey" to be added to the Aliases field in the broccol user object. This is true even if "jonabbey" was already listed in the Aliases field. Now, a new vector field mode has been added, "addIfNotPresent". This mode will now be the default when a vector handling mode is not specified. In other words, the above snippet is equivalent to Note that "add" mode for Invid fields was always handled as if it were "addIfNotPresent", as the InvidDBField.addElements() method automatically filters out duplicates. I have updated the XML docs to reflect this change. In addition, the mode can now properly be used to remove redundant copies of strings from a vector string field.. this did not work properly before. In other words, if we have an object with the following values then will properly remove one of the "jonabbey" strings. Before, the XML field handling logic simply saw that the "jonabbey" string was already in the vector (at least) once, and so didn't bother to remove the redundant copy. 8. [ADMIN CONSOLE/SERVER] More fixes to prevent memory pinning with schema editor The admin console's schema editing code has continued to cause memory on the server to be pinned after the schema editor window has been closed. I've added a great deal of manual object dissolution and reference clearing to prevent memory pinning in the server. More importantly, the admin console's schema editor code now properly handles clearing the server-side schema editor context if the window is closed through the window system's controls rather than through the 'commit' or 'cancel' buttons. -------------------- Changes from 1.0.6 to 1.0.7 ------------------- RELEASE DATE: October 11, 2001 1. [SERVER] Deleted objects now have all of their fields logged The Ganymede server has not been logging the status of objects when they are deleted, which has made it in some cases impossible to determine what was in an object when it was deleted. In theory, today, all objects created in the server will be logged with their initial state as well as all changes made, but at ARL we had a great deal of objects that were imported using the old 'directLoader' hack, without any logging done. The real advantage to this for other users will not so much be for scanning the log file as it will be to make the object deletion email more informative. 2. [SERVER] Better error message on field creation failure The server now gives a better error message if a user tries to link an object to a (previously undefined) field that the user does not have permission to create. 3. [SERVER] Made sessions smarter about refusing to edit objects during commit The DBSession class has been modified to disallow editing of new objects during a transaction's commit or abort process. This is to help make sure that any code during the commitPhase1() and commitPhase2() DBEditObject customization methods that seeks to bring new objects into the transaction after the commit has already started will trigger a runtimeException, rather than silently confusing the commit logic. This change shouldn't have any effect on any code provided with the Ganymede distribution or the userKit, but might force an error in custom code if the customizers were doing some sort of dangerous activities during the commit process. 4. [SERVER] Amended wizard shutdown process The GanymediatorWizard class had a loophole that would cause wizards to not be unregistered from a GanymedeSession if the wizard's state had been forced to DONE outside of the GanymediatorWizard.unregister() method. This has been fixed so that any time a wizard goes through returns a success() or fail() result it will be unregistered. This fix took care of a bug in the laboratory's GASHARL schema kit operations, but is very unlikely to be affecting anyone out there unless you've done just the right (wrong) kind of custom schema coding. 5. [SERVER] Put loopback binding check into server startup The server now checks to make sure that it is not being bound to a loopback address on startup, which would prevent clients from being able to communicate with the server from remote systems. The server will now generate a message of the form ** Error ** Both the system hostname (hostname) and the ganymede.serverhost definition (hostname) resolve to the 127.0.0.1 loopback address The Ganymede server must have an externally accessible IP address or else clients will not be able to communicate with the Ganymede server from other than localhost. If you really want to be only usable for localhost, edit the runServer script to use the -forcelocalhost option. if the server is not able to figure out a hostname that will resolve to a network-accessible IP address. RedHat Linux 7 systems are particularly prone to this, as the default configuration for such systems includes the system's hostname in /etc/hosts as mapping to 127.0.0.1, and the system 'hostname' function returns by default a non-fully-qualified hostname. The combination of these two conditions resulted in the Ganymede server registering itself with the rmiregistry process as being on IP address 127.0.0.1. This change was in response to a problem report from Steve Lemons (steve.lemons@arrisi.com). 6. [SERVER] Objects in the process of being deleted may not be linked to The server's object binding logic had a bug which would allow an object which was being deleted to be improperly linked into an Invid field in another object. This occurred in a bit of custom code in the GASHARL schema kit where unlinking a remote object in a given context resulted in the object being deleted by the custom server-side code. The client, however, showed the unlinked object as a valid choice to be added back into the link field. If the user tried to put the (now deleted) object back into the field, the link was made, even though the object was marked for deletion when the transaction committed. This resulted in an Invid field containing an Invid pointer to an object which no longer existed, which is a bug. The server's InvidDBField logic is now smart enough to forbid linking to an object which has been marked for deletion by the transaction. This was a very obscure integrity violation vulnerability. The server and client have always cooperated in other ways to prevent this condition from arising. It was only due to some odd custom coding in the GASHARL schema kit that this vulnerability was revealed. 7. [CLIENT] Optimized out redundant network calls in the GUI client The Ganymede GUI client has been optimized to remove a great many redundant RMI calls when viewing objects on the server. The client can now display objects with only about half the RMI calls that the client used previously. This will speed up the client noticeably on slow network links, and will reduce loading on the server slightly. -------------------- Changes from 1.0.5 to 1.0.6 ------------------- RELEASE DATE: August 15, 2001 1. [SERVER, ADMIN CONSOLE] Fixed delayed memory release for schema editing The Ganymede server wasn't properly clearing references to a bunch of temporary data structures when a schema edit was cancelled by the admin console. This had the effect of increasing our server's memory footprint by a couple hundred kilobytes or so until such time as the admin console was stopped and restarted. 2. [WEBACCESS] Fixed ganypass.pl to allow < and > chars in passwords The ganypass.pl script, which provides a CGI interface to the xmlclient for changing users' passwords, was improperly failing to convert < and > to the XML entity escapes < and >, which could cause the server's XML parser to reject password changes that involved those characters. 3. [SERVER] Fixed several journal processing bugs Yowtch. Several nasty bugs that were lurking here. None of these bugs caused any problems if the server was shut down cleanly and restarted, but if the server was abnormally terminated, the following bugs caused problems during transaction recovery on startup. Firstly, the server had a bug that prevented successful startup if the server was killed without getting a chance to consolidate the database into a new ganymede.db file, and if one of the transactions in the journal file involved an deleted object that had one or more namespace-constrained database fields. This was due to an attempt to use Ganymede.internalSession before that variable was defined. This has been fixed by making the DBNameSpaceHandle.getField() method look up objects directly out of the DBStore tables without using a GanymedeSession, if needed. Thanks to Gaurav Bhargava at Villanova University for reporting this first bug. Secondly, there was a bad bug that caused transaction processing in the journal handling code to repeatedly process object records in the journal. The journal processing code was not clearing the entry vector at the end of each transaction's processing, which caused the server to do a lot of repetitive work and to generate misleading warnings about 'objects already deleted' in the object deletion case. Ugh. Thirdly, there was a bug, subtle in causation but HUMONGOUSLY AWFUL in effect, which caused the server to *eat* changed scalar fields when processing journal entries. This bug has been in Ganymede since 1.0pre1, when I moved field storage in DBObject to a packed hash as a memory optimization. The replaceField() code that edited the packed hash to replace fields with the versions noted in the journal transaction entries silently failed if it could not find the field it was looking for, causing the changed fields to be simply dropped on the floor. That's what I get for recklessly optimizing my data structures. Data loss bugs in Ganymede have been rare, thankfully, but this was an ugly one, and it bit me when I was testing the fixes to the first two journal bugs. My apologies if it bit anyone else out there. The good news is that I've cleaned up the journal loading code a fair bit, and that I have significantly improved the debug instrumentation in this section of the code. -------------------- Changes from 1.04 to 1.0.5 ------------------- RELEASE DATE: July 27, 2001 1. [DISTRIB] Fixed a number of bugs for JDK 1.4 compatibility Fixed a few places that assumed SimpleTimeZone.getDefault() would return an object of class SimpleTimeZone. Fixed the jar packaging to satisfy Java 1.4's slightly pickier serialization logic. Because of these changes, the Ganymede 1.0.5 server will not be compatible with clients from 1.04 or earlier. 2. [DISTRIB] Added support for using Java Web Start to launch clients Sun's excellent Java Web Start provides the best way yet to launch the Ganymede clients on Windows. I have made changes to installWeb to support the Java Web Start option in the Ganymede launcher. 3. [CLIENT] Fixed java.security.AccessControlException in client On some versions of the JDK, the client, when running as an applet, was refusing to allow changes to the animated build status icon in response to RMI callbacks from the server. This robbed the user of valuable feedback and resulted in java.security.AccessControlException messages being thrown on the server in response to the client notification call. I have added a new 'SecurityLaunderThread' to the client, which waits for build status messages from the server and updates the build status icon. Since the SecurityLaunderThread was started by the client and not by the RMI system, it has proper permissions to play with the Swing Event Queue, and everything works properly. 4. [SERVER, CLIENT] Reworked permissions handling for built-in fields The Ganymede server previously did not separately track permission to edit the "built-in" fields. That is, the permissions editor in the client did not allow per-field permissions to be set for the standard fields that are automatically present in every object, including the Owner List, Expiration Date, Removal Date, and Notes fields. Instead of allowing access controls for these fields to be set in the standard permission matrices in the Role objects, all of these fields were accessible with the same permission as the containing object itself. If a user had permission to edit an object, that user automatically had permission to edit these four system fields. This turned out to be a security hole, because there are many cases in which it is desirable to allow an object to be minimally editable by non-privileged users (so that end users can change their own passwords, for instance), where such users may not be trusted to edit the owner list, notes, expiration date, and removal date fields. The client's permissions editor and the server now allow explicit setting of permissions for the Owner List, Notes, Removal, and Expiration Fields. The four 'historical' fields used to track object creation and modification timestamps and admins are never allowed to be edited by the user, and so are still left out of the permissions editor. In the process of doing this work, a few potential vulnerabilities that might have been exploitable by a rewritten Ganymede client were closed. The Ganymede philosophy is that the server should not trust the client for anything, and a few security assumptions have now been made explicit. The server now makes several checks and forbids the granting of privileges that would be liable to break the Ganymede security model. For instance, non-privileged end users will never be allowed to edit the owner list field for objects, regardless of whatever permission for the Owner List field might be granted in the server's Default role. 5. [CLIENT] Improved GUI cleanup Miklos Muller reported that the client was leaving the permissions editor up if the server forced a client disconnect due to a timeout. I looked into things, and the code for handling resource cleanup on window close in the client was still very messy, despite Bug fix #11 for release 1.0, below. Things are a good bit cleaner in the client, now, and all popup permission editor windows should now be put away properly when the window that the permission editor was launched from is closed. 6. [DISTRIB] Changed numbering scheme I realized that if I have more than 6 more 1.0x releases, I'll suddenly be at 1.1, which I don't want to do by accident. So, we're going from 1.04 to 1.0.5. 7. [DOCUMENTATION] Added an upgrade guide, improved XML docs I've written a short upgrade guide that explains how to go about upgrading the server, schema kit, and clients when a new version of Ganymede comes out. I've also done a fair amount of rework of the xmlclient documentation. 8. [SERVER] Made the server use a default address for email if necessary The Ganymede server typically sends mail out for actions taken by admins on behalf of the admin's email address. That is, Admin Persona objects have an 'email address' field, and when the Ganymede server sends out mail in response to that admin's actions, mail from Ganymede is sent with the admin's address used as the 'From:' header. Previously, if Ganymede could not determine an email address to use for an admin, it would try to send mail out with a null return address, which caused the mail attempt to fail. Martin Vogt reported this bug, and the proper fix for it, back in March of 2000, but I didn't realize how likely this problem was to affect new users until I did some testing of a fresh install on a new system. 9. [SERVER] Fixed field/object permission interaction Previously, the GanymedeSession.getPerm() method filtered per-field permission bits against the permission bits for the object as a whole. This makes sense for the view and edit bits, as you can't view or edit a field if you can't view or edit the object it is contained in, but it does not make sense for the create bit, as it might well be appropriate to be able to create a new field in a pre-existing object that you can edit, even if you don't have the right to create a new object of the type. The server is now smart enough to avoid filtering the create bit for a field against the create bit for its object. -------------------- Changes from 1.03 to 1.04 ------------------- RELEASE DATE: July 9, 2001 1. [SERVER] Fixed NullPointerException thrown in XML Login path The Ganymede server's xmlLogin() method was throwing a NullPointerException if a user submitted an XML file through the XML client with an invalid username and/or password. The server now avoids having that exception thrown by explicitly checking the success of the username/password login before trying to construct a GanymedeXMLSession. 2. [CLIENT] Fixed date reversion in the pop-up calendar GUI component I fixed the pop-up calendar's reversion handling for when the user attempts to change the date/time in a date field and chooses a date that is out of range. 3. [CLIENT] Enhanced list GUI boxes in the client The client's JstringListBox and StringSelector components have had their behavior enhanced, with a 'select all' button added at the top of the StringSelector, and with the JstringListBox keeping all items added selected, to allow easy undo of item moves. The JstringListBox is now capable of supporting item dragging in the list. 4. [CLIENT] Enhanced query box The query box now uses a pair of StringSelectors to represent the list of field choices rather than a bunch of checkboxes. 5. [SERVER] Fixed namespace constraint handling during schema editing Previously, it was possible during schema editing to set a namespace constraint on a field that contained data in violation of that constraint. Now the schema editor will not allow any schema edits to be made that would allow a namespace constraint to be placed on a field that contains data in violation of such a uniqueness constraint. With this change, it should now be impossible to put the server into a state where fields hold data in contradiction to an expressed uniqueness constraint. Thanks to Gaurav Bhargava at Villanova University for reporting this bug. -------------------- Changes from 1.02 to 1.03 ------------------- RELEASE DATE: June 22, 2001 1. [SERVER] Fixed non-event related mail sending in DBLog The DBLog class had an error, effectively, that prevented 'mailout' class mail from being sent, although it was logged. Added a 'sendMail' method to DBLog to simplify the sending of mail by plug-in code and custom tasks. 2. [SERVER] Fixed startup default permissions for non-supergash users Miklos Muller, mmuller@lbcons.net, reported a bug that caused Ganymede to not allow non-supergash users to login to a freshly initialized Ganymede server. Editing the Default Role object's Default Access Bits field, works around the problem, but unless this is done, an exception will be thrown in GanymedeSession.resetDefaultPerms(). I have corrected the GanymedeSession resaultDefaultPerms() method so that it will not freak out when it doesn't find any definition for the Default Access Bits field in the Default Role object. This bug was avoided by users who used the userKit's loader.pl script to initialize the Ganymede database. 3. [SERVER] Fixed infinite loop bug in XML parser The XML parsing code had a bug that would leave it looping infinitely if the XML stream was closed before completion of parsing, either due to an incomplete XML file being fed to the xmlclient, or through the xmlclient dying suddenly. This didn't break the server, but it did take up a lot of processor time, and kept all of the memory associated with an XML session pinned in the server's heap. 4. [SERVER] Fixed default label field for persona objects The persona object schema definition was improperly defaulted to use the older 'name' field as the label, rather than the composite 'label' field. This could cause errors when trying to login with a newly created persona. This bug was reported by Miklos Muller as well. -------------------- Changes from 1.01 to 1.02 ------------------- RELEASE DATE: June 7, 2001 1. [SERVER] Fixed an incompatibility with JDK 1.1 on the server The server code included a use of a generic 'add()' method on the java.util.Vector class that didn't get added until Java 1.2. Replaced it with addElement(). I'm going to have to put up some sort of nightly build directory until I can get network CVS access going, putting out new releases like this is silly. ;-) -------------------- Changes from 1.0 to 1.01 ------------------- RELEASE DATE: June 7, 2001 1. [SERVER] Fixed an IllegalArgumentException with empty namespaces Michael Jung (mikej@confluenttech.com) reported an exception thrown during startup. It turned out to be due to some of the checkpoint code trying to deal with an empty namespace. Easily fixed, so I did. -------------------- Changes from 1.0pre1 to 1.0 ------------------- RELEASE DATE: May 31, 2001 1.[SERVER] Made server do better server-side cleanup for stuck clients The RMI system has the ability to get into a state where a TCP/IP handling thread waits indefinitely for communication from a client. Prior to this fix, the server would wait forever for a dedicated thread to try and notify the client that it is being forced off before handling the server-side client logout housekeeping. Now, the forceOff() method in GanymedeSession will expunge the client from the server on the server-side before notifying the client that it has been disconnected. Note that this fix may still leave those dedicated background threads stuck in the JVM trying to finish reading from a socket from the (presumably dead) client. I can't do much about this other than wait for Sun to fix their RMI implementation so that RMI reads on a socket will eventually time out. In my experience, threads hanging due to a partial socket read is reasonably rare, so I'm not overly worried about hanging disconnect threads clogging up the server. 2. [SERVER, ADMIN CONSOLE] Added support for memory status tracking The server and admin console now collaborate to display the amount of free memory and the total amount of memory taken up by the server, to be updated once a minute by a new Memory Status Updater task. 3. [SERVER] Added the garbage collection task back in I added back in the on-demand garbage collection task, which is an interesting thing to play with now that the admin console shows memory usage on the server. It looks like System.gc() doesn't do much of anything under the HotSpot server VM. Not too surprising, as the HotSpot server VM is already optimized for continual, incremental garbage collection, but having the ability to launch a garbage collection task from the admin console's task monitor may be helpful on some VM's. 4. [SERVER] Cleaned up setLastEvent() usage in GanymedeSession I cleaned up usage of the diagnostic setLastEvent() stuff in GanymedeSession. Viewing or editing objects that contain embedded objects will no longer cause notification for the embedded stuff to be sent to the client. In addition, all setLastEvent()-style notification for non-interactive sessions has been turned off, to reduce the overhead during XML bulk-loading. 5. [CLIENT] Informative dialogs are no longer modal The Ganymede GUI client was occasionally freezing up on start-up if the user attempted to bring up a toolbar dialog at the same time the client was working in the background to bring up the MOTD dialog. This was a result of the HTML messageDialog class used for the MOTD, Credits, and 'About Ganymede' dialogs being configured to lock out any other GUI interactions until those dialogs were released. I don't think there was ever any compelling reason to make these dialogs modal, so I have made them not be so. There's now a small risk that a user might ignore the MOTD dialog without dismissing it, but that's a far less serious outcome than having the client lock up. Thanks to Tom Embleton for bringing this back to my attention. 6. [SERVER] Memory leaks fixed Once I had the admin console tracking memory usage, I quickly saw that the Ganymede server was leaking memory over time. At ARL, our server's heap was starting out at about 10 megs, and after 8-9 days, the heap had nearly doubled in size. I ran the Ganymede server through a profiler for a couple of weeks and identified several places where bits of code were holding on to references to things that were no longer needed. Some of these were really obscure.. things I would *never* have found were it not for the assistance of the profiler. After a month or so of analysis and code work, the Ganymede server's memory usage has been brought under far better control. I've not yet had a chance to run the newly updated server for long enough to judge whether the heap usage is truly stable over the space of weeks and months, but I can say that the server's memory growth rate appears to have dropped to less than a tenth of what it was before, at worst. 7. [SERVER] File descriptor leak fixed The arlut.csd.ganymede.util.zipIt class was neglecting to close files that were being read in for adding to a zip file. This caused the Ganymede server to run out of file descriptors during nightly archives. Apparently this problem was masked on earlier versions of the JVM by the FileInputStream's finalize() method doing clean-up. Since finalize() is not guaranteed to ever be called, moving the Ganymede server to a new JVM made file descriptor starvation start appearing. I have now fixed the code so that finalization is no longer required for proper resource release, and everything is working well. 8. [SERVER] Added cryptographic hash support for Samba The server now includes classes to support both forms of password hashing used in Samba's encrypted password file. This will simplify support of Samba in userKit 1.0, as the NTSambaBuilderTask will now be able to emit a file with all of the plaintext passwords pre-hashed, rather than depending on a hard-to-install Expect script to drive smbpasswd to insert updates into the Samba password file. 9. [SERVER/ADMIN CONSOLE] Added a 'force rebuild' menu item to admin console The admin console now has the ability to force a mandatory rebuild of all external data. Previously, it was possible to manually run a builder task from the task monitor table through the right-click popup menu, but that builder task would typically not actually write out data if the object bases in Ganymede that contained the data had not been changed since the last run of that builder task. Now the admin console has a 'force build' option under its Control menu which forces the running of all builder tasks registered in the system, and that forces those builder tasks to treat all object bases as having been dirtied since the last run of the builder tasks. This allows an admin to force a dump of all external data files generated by the builder tasks, and the consequent execution of all external update scripts. 10. [ADMIN CONSOLE] Improved network error handling The admin console would previously force a disconnect from the server in the event that any network operation returned a RemoteException. Unfortunately, a RemoteException may be returned even if the network connection remains good. Any exception that occurs on the server in response to an RMI call results in a RemoteException. Having the admin console do a force disconnect on any error was a bit harsh. Now, the admin console will append the stack trace information for any RMI remote exception to the console's log area, where it may be seen. 11. [CLIENT] Cleaned up resource retention on the Ganymede GUI client The Ganymede client was doing very little in the way of cleaning up allocated structures. So little, in fact, that it was retaining references to remotely exported fields from the server, even if the client had been logged out. The client has now had some extensive work done to disassemble object structures when appropriate. This will very significantly reduce memory loading on the client. 12. [CLIENT] Made the client's query window non-modal The client's query window's modality was causing the JpopUpCalendar to lock up when someone tried to edit the date and time of a time field search. Making the query window non-modal takes care of this. In addition, I reworked the way that threading is handled for the 'waiting for query' animation and table window display to make sure rendering operations for them are handled on the GUI thread, which should help avoid some exception traces that were occasionally being thrown on the client. 13. [GUI, SCHEMA EDITOR] Fixed a race condition in the tree node drag code The schema editor provides support for re-ordering fields within an object, or objects within an object category folder. The tree GUI component had a race condition, however. Depending on the rate at which the Java GUI thread produced mouse motion events and the rate at which the user moved the mouse, node dragging could result in the wrong node being moved. This of course, made the schema editor amazingly frustrating to try and use if conditions allowed this bug to be expressed. This bug has been around for years, transiently appearing and disappearing according to the speed of GUI thread processing on the JVM on which the schema editor was run. I am *very* happy to have it fixed at last. 14. [SERVER] Fixed transient logic errors in permission editing There were a few places in the Ganymede server where the Ganymede logic was invalidly depending on the post-serialization initialization of transient fields in objects passed over RMI. This is not in keeping with the way that Java serialization actually works, and in one instance this logic flaw resulted in changes made to permission matrices by editing Role objects not taking effect until the server was stopped and restarted. Nasty, nasty bug, and a pain to track down. I've fixed this, and audited all of the server code to insure that there are no remaining attempted dependencies on post-serialization initialization of transient fields. 15. [SERVER XML] Fixed missing attribute error reporting in XML parser The Ganymede server's XML parsing code is designed to report to the xmlclient a record of the last 30 XML elements parsed prior to encountering an XML validity error (such as an invalid character, say). Unfortunately, the class that represents XML entities (such as '') had a bug in it's toString() method that caused every other attribute to be skipped over when regenerating a string representation for the node. This was a small bug, but made it maddeningly difficult to track down syntax errors in XML files created for use with Ganymede, if the odd numbered attributes in the nodes were not unique enough to identify position in the XML input file. 16. [CLIENT] Tweaked the Ganymede client's GUI I have tweaked a number of aspects of the Ganymede client's GUI to make using the Ganymede client a bit more convenient, mostly in the way of reducing the default length of fields in the object viewing and editing windows. 17. [SERVER] Fixed DBNameSpace rollback()/unmark()/commit() bug The server had a vulnerability in the DBNameSpace class that would cause the commit() method to fail in circumstances where a namespace rollback() operation had been followed by an unmark() operation on a value previously allocated in that namespace. The DBNameSpace.commit() method could in this circumstance throw a NullPointerException at a point in the transaction commit process that would leave the transaction in an unfinished state, causing confusion and mayhem all around. This bug only manifested one time in the year and a half that Ganymede has been in production here at ARL:UT, and appears to have manifested at a time of heavy contention for resources in the namespace, such that multiple transactions were trying to allocate namespace-guarded unique values concurrently. In response to this, the DBNameSpace.java code was largely rewritten. The rewrite fixed this problem, and also significantly reduced the memory requirements for unique value namespace handling. 18. [CLIENT, ADMIN CONSOLE] GUI tweaks for 1.0 Added 'About Ganymede' dialog to Admin Console. Added keyboard accelerators to Admin Console menus. Tweaked menu names in Admin Console to allow unique and intuitive accelerators. Tweaked the messageDialog class so the client's MOTD can be dismissed by hitting spacebar. Tweaked the menus in object windows in the client, added keyboard accelerators to menus. Did some more cleanup work on the tree GUI component. When you drag a node in the schema editor, the node you are dragging will now be highlighted in blue during the drag operation, in addition to the icon drag that the tree has had. Made right-clicking on nodes select the node before popping up the pop-up menu, as a visual reference point. 19. [SERVER] Fixed DBObject synchronization The server ran into a deadlock race condition where logins could not be processed while a database dump was in progress at a certain point. This arose out of the conversion to a packed array hash structure for field storage in the DBObject class within the last year. There were a number of places in which I was sloppy with nested monitor synchronization. I have thoroughly reworked the DBObject synchronization logic to avoid all such problems. -------------------- Changes from 0.99.9 to 1.0pre1 ------------------- RELEASE DATE: January 30, 2001 1. [SERVER] Reworked server synchronization logic The GanymedeServer.login() method was vulnerable to a nested monitor deadlock due to a section being synchronized on both the server's GanymedeServer object and its DBStore object. Basically, I was getting way too fancy in synchronizing on various server objects instead of defining a proper semaphore class and using that. I added the loginSemaphore class, which takes care of everything without my having to worry about n^2 different possible code paths. I modified the admin console to report the login semaphore count for each log entry.. this makes it easy to see at a glance how many users are logged on for each log entry, and to insure that the semaphore count is incremented and decremented properly under all circumstances. In the process of all this, I realized that I was unnecessarily doing a notifyAll() every time I did a synchronized operation on the server's DBStore object.. I had misunderstood when it was necessary to do a notify when I originally wrote a lot of this code. The sweepInvids() and checkInvids() methods have been fixed so that they are no longer synchronized on GanymedeServer but they do now establish a dump lock on the server's database, to insure that the object walk is done safely. 2. [SERVER] Added getBackLinks() method to the DBObject class Martin Schneider reported that he needed to be able to get the Vector of Invid's pointing to a given object through asymmetric links, in the same way that was possible by using the old backlinks field. To support this, I added a getBackLinks() method to the DBObject class that directly consults the DBStore's backPointers hash tree. 3. [ADMIN CONSOLE] Fixed admin console confusion with multiple logins The admin console had a bug that would cause the console user table to become confused if more than one user with the same persona name was logged in at the same time on the Ganymede server. This problem only happened when an admin logged into the server with admin privileges more than one at a time. Logging in without admin privileges did not cause this problem, because the server assigns unique usernames to logins, but the server does not guarantee unique persona names. The server now reports to the admin console each login's unique user name, which the admin console uses for login tracking, but the admin console will continue to display the user's (possibly non-unique) persona name. This one has bugged me for awhile, it's good to have this fixed. This change (and change 5 below) change some of the semantics of the admin console's operations, so you should not use an older version of the Ganymede admin console with a 1.0pre1 server. 4. [CLIENT] Right clicking on a selected JTable row no longer de-selects the row This one has bothered me for a long time, too. I changed the baseTable, rowTable, and tableCanvas classes in src/JTable so that right-clicks on a table row won't ever deselect that row if already selected. Left-clicking on a selected row in the rowTable will still deselect the row. 5. [ADMIN CONSOLE] Threaded out server-to-console communications The Ganymede server has always communicated synchronously with any attached admin consoles, which made the server effectively block if the admin console became non-responsive without explicitly throwing a RemoteException. I noticed today that the admin console was blocking the server from processing logins/logouts while my window manager on X was waiting for me to place a new window, which of course is not acceptable. The Ganymede server now queues up notification events for the admin console, and uses a separate thread for each admin console for actually handling the server-to-console communications, using the new serverAdminProxy class. The Ganymede server will no longer sit around waiting for the admin console when something happens, with the obvious benefits resulting in speed and reliability on the server. As a side benefit of the event queueing model, I've implemented event coalescing and event replacement, so that if the Ganymede server is generating admin console events faster than the console update thread is getting them to the admin console, the server will combine and/or drop old events, so that the server won't waste time sending events to the console that are already out of date. The admin console was changed a very little bit to support event coalescing; the changeStatus() method in arlut.csd.ganymede.iAdmin now depends on the server prefixing each status line with a time stamp, rather than trying to time stamp each event from the server itself. Good stuff, and very clean on the server. 6. [ADMIN CONSOLE] Fixed admin console shutdown process I fixed some code in the server's GanymedeAdmin class so that an admin console will stay open if a shutdown attempt by it was refused for some reason. Also fixed a bug in the admin console that made it fail to really attempt to shutdown the server if it tried once and failed. 7. [SERVER] Cleaned up some locking and transaction logic I went through and did a code review of all of the server's transaction and locking classes, and removed a few bits of extraneous stuff, fixed a couple of mistakes that could conceivably cause problems on edge cases, and extensively commented code that I hadn't really touched for a couple of years so that if I dive into it in 2002 I won't have to figure it all out from scratch again. These changes probably won't be noticeable in operation unless I broke something. The DBLock code itself has been reworked a fair bit. The DBLock code no longer synchronizes directly on the Ganymede server's DBStore object, but instead uses a DBLockSync object held in the DBStore object. This as well will help prevent deadlock problems, by limiting the scope of synchronization on the DBStore object even further. 8. [SERVER] Made the database dumper task less timid Due in part to the increased confidence I have in the server's thread safety with the new login semaphore and my code review, I made the database dumper task go ahead and do a database consolidation even if users are logged into the server. Previously, the database consolidation task could be postponed indefinitely if users were logged in each time the dump task fired, leading to a bigger journal file and longer startup times if the server machine was rebooted. Now the database dumper task will go ahead and do the dump, even if it has to wait for transaction commits to finish so it can assert a DBDumpLock on the database. 9. [SERVER/CLIENT] Added bulk add/remove methods to DBField The Ganymede client has long supported moving large blocks of choices from the unselected panel to the selected panel in vector invid and string fields, using shift-click to do multiple selection. Unfortunately, actually doing this with large blocks has been *excruciatingly* slow to perform, as each item moved was a separate round trip network call. For Invid fields, things were even worse, as each single item moved involved a separate transaction checkpoint to make sure that all bind and unbind operations for the element add/delete were done with transactional consistency. The db_field interface now has explicit addElements() and deleteElements() methods that support bulk adds or deletes. The DBField and InvidDBField classes have had these methods added as well. For InvidDBField particularly, the speed up from using the bulk add/delete methods is dramatic. In the process of adding addElements() and deleteElements() on the server, I wound up significantly increasing the efficiency of even non-bulk vector adds where there is an enumerated list of choices that any adds have to be chosen from. In the process of supporting such bulk transfers, I modified a good bit of the client and the client's StringSelector GUI component. Everywhere the client uses the StringSelector can now handle bulk moves in one operation. Making this change has the potential to complicate custom DBEditObject subclasses, which now have to be able to respond in wizardHook() to both singleton and bulk adds and deletes, and which have to provide separate finalizeAddElements() and finalizeDeleteElements() in addition to the old finalizeAddElement() and finalizeDeleteElement() methods if the customizer wishes to take special action when items are added to or removed from a vector field. 10. [CLIENT] Cleaned up the Objects Owned panel in the Owner Group window Despite the fact that the 'Objects Owned' panel in the Owner Group window uses the same GUI component that the normal 'General' panel uses, it was not possible to right-click on entries in order to view or edit objects listed. This has been fixed, and it should now be possible to right-click on any object listing in the client and get at least a view/edit menu. The object type listing pulldown menu in the Objects Owned panel will now be sorted in the client. Turned debugging off in the Objects Owned panel. 11. [CLIENT] Added threading to adminHistory panel When I added a thread-out to the generic history panel in the client (change 0.99.9#10), I neglected to do the same for the admin history panel that is present when viewing and/or editing an admin persona object. The admin history panel is now threaded so that it won't block the GUI refresh during long waits for a history summary from the server. 12. [SERVER] Fixed journal handling on start up The DBStore load() method failed to consolidate the journal into a clean ganymede.db dump if the journal was not in a fully consistent state. It would print an error message and then skip the normal start-up dump. Now, the error message is still printed, but since the journal loading code will properly cope with incomplete or malformed transaction records in the journal, the normal startup process is followed thereafter. 13. [SERVER/CLIENT] The client now displays whether the server is doing a build The server will now notify the client when it starts and stops performing a build. The client displays this information in the panel in the very bottom left corner, where the 'Status: ' text used to be. This is an asynchronous message sent by the server using a new serverClientProxy class that has its own background thread for queueing and sending asynchronous RMI calls to the client. The mechanism used to send this message is generic; it will be easy to extend this mechanism if I want to add more generic server notification features to the client. This involved a change to several server classes, including the GanymedeBuilderTask base class. Any task not subclassed from GanymedeBuilderTask will not trigger this status update. 14. [SERVER] Took out deprecated thread stop() operations The Ganymede Server's background scheduler used the deprecated java.lang.Thread.stop() method call to shut down running tasks when the server is shut down. The stop() method is deprecated in 1.2 as unsafe, however, so the scheduler now uses the interrupted() method instead. The scheduler and all tasks included with Ganymede 1.0pre1 are responsive to the interrupted() call, but custom background tasks written for use with the Ganymede scheduler may need to be reworked to respond to the interrupt call. The GanymedeBuilderTask base class will check for an interrupt between phase1 and phase2 of a build, but it will be up to a builder task writer to check java.lang.Thread.currentThread.isInterrupted() during the two phases if they want to be able to quickly interrupt their builder task. With this change and a couple of unrelated deprecation fixes in the JDataComponent and JTree packages, the Ganymede server and client should now build with no deprecation warnings under Java 1.2 and later. 15. [SERVER/CLIENT] Added regexp support for IP fields in Query mechanism The Ganymede server and client did not previously support regexp matching for IP fields. The client now provides a 'matching' option in the query box for performing regexp searches on IP address fields. Gil Kloepfer (gil@arlut.utexas.edu) pointed this out. 16. [SERVER] Did a basic reworking of ganymede.db and DBStore structures I've been laying the groundwork for support for XML database export/import, and towards this end, I have evolved the ganymede.db database format and the server's DBStore data structures in a direction that may make it easier in the future to load data from an XML file, and to make the server code simpler. The ganymede.db file no longer includes definitions for the mandatory fields in the schema section. The server now implicitly creates these field definitions whenever an object type definition is read from the ganymede.db file or created with the schema editor. This not only makes the schema editor simpler, but it also will lessen bulk and possibility for error when schema importation from an XML file is supported, as the XML file would only need to specify custom object and field types. And while I was mucking around with the file format, I modified server code and the ganymede.db file format to make keeping track of the order of fields within an object and object types within a category simpler, and more robust. There were also a good number of RMI interface changes to simplify schema editing logic. Signficant changes were made to Base, BaseField, Category, FieldTemplate, and SchemaEdit, and the classes that implement these interfaces. A number of effects of these changes are discussed in change 1.0pre1#17, below. These changes in the server are wrapped in file version checks, so Ganymede 1.0pre1 should work fine with ganymede.db files from older versions of the Ganymede server. When 1.0pre1 dumps out the ganymede.db file, however, it will be dumped in its new form. 17. [SERVER/ADMIN CONSOLE] Made schema editing more robust The server now supports schema editing better. Schema changes that would break the database are prevented (at least, much more than before). The server now provides feedback to the admin console (with error dialogs, and the like) if an attempted change would cause problems. A lot of the interface between the graphical schema editor and the server has been simplified and regularized, to depend less on serialization of server-specific data structures. The schema editing system now uses String labels to identify objects to be manipulated in many cases. A number of operations during schema editing that were performed as an interaction between the graphical schema editor and the server are now done locally on the server, to reduce performance problems. 18. [SERVER] The server now resolves client ip addresses to host names When a user or admin console logs into the Ganymede server, the server will now do a reverse DNS lookup to get the fully qualified name of the system that the user is logging in from. The admin console will show the fully qualified domain name for attached clients. 19. [SERVER] Transaction email can be turned off with the starttransaction event Formerly, the server would always send a transaction log to the admin who commited a transaction, as well as sending email to all owner groups that own any objects affected by the transaction. Now, this email can be turned on or off by editing the 'starttransaction' system event object. If the 'Event Mail' checkbox in the starttransaction object is not checked, no transaction logs will ever be mailed. If it is checked, then the 'Cc: Admin' and 'Cc: Owner Groups' checkboxes are consulted to determine whether the admin and the owner groups of any affected objects should be notified. In addition, any email addresses added to the 'Mail List' vector field will receive a complete copy of any transaction logs generated during the server's operation. By default, all three checkboxes are turned on and mail will be sent out in the same fashion that it was in 0.99.9 and earlier. Users with pre-existing Ganymede databases will have to edit this system event object in order to restore the previous mail behavior. 20. [SERVER] Fixed QueryResult.append() The arlut.csd.ganymede.QueryResult class's append() method did not properly update its invidHash and labelHash tables when multiple QueryResult objects were coalesced. This caused the appended QueryResult object on the server to act as if it did not contain Invid's that it did, which could result in a permissions violation report when an InvidDBField tried to verify an Invid in an addElement operation. This only had an affect when the DBEditObject.obtainChoiceList() method used the QueryResult.append() method to concatenate query results, which the gasharl schema kit's emailListCustom class did for its Internal Members field. Thanks to Andrew Helyer (helyer@arlut.utexas.edu) for reporting this. 21. [CLIENT,SERVER] Fixed object description mail from the client Fixed the client's object mail feature. Added getSummaryDescription() to the DBObject class and the db_object interface, which is now used for sending an email description of an object from the client. This feature is activated by using the 'Mail To...' menu item under the 'File' menu in the client's object view/edit windows. 22. [SERVER] Fixed role cloning The PermissionMatrixDBField class had a slight error in its copyFieldTo() method which broke role cloning. This was supposed to work, but it didn't. Thanks to Pug Bainter (pug@pug.net) for reporting this. 23. [SERVER] Changed client naming for concurrent use Previously, if a user logged into the Ganymede server with more than one client at a time but the same username, the Ganymede server would report the second account as user2 in the logs and the admin console. This can be confusing for situations where both user and user2 actually exist as valid user accounts, so the Ganymede server logic has been changed so that if 'user' is logged in twice at the same time, the second login will be reported as 'user[2]' rather than 'user2'. Thanks to Pug for complaining about this. 24. [SCHEMA] Fixed gash/gasharl schema system interface deletion bug Pug caught a bug in the gash/gasharl schema kit, by doing something I would never have been able to make myself do. He created a system, set the room to a room with a single network, edited the system's single interface, set the network, then deleted the interface. In doing so, he discovered that the gasharl schema kit didn't keep track of things properly on interface deletion, and that the network that was chosen became unavailable as a choice for that system when an interface was created again with the 'add interface' button. I've fixed this, so now you can delete and add interfaces all you want without losing the ability to choose a previously chosen network. 25. [CLIENT] Fixed default owner dialog logic Previously, the ganymede client could get confused about the proper settings for the default owner dialog (the dialog that pops up to ask you what owner group you want to put a soon-to-be created object into) if you change personas. It would be too painful right now to explain the precise situation in which the ganymede client wouldn't do the right thing, but it doesn't anymore. It does the right thing. This is another one of those things that's been bugging me for a long time, and I'm glad to see it fixed. 26. [SERVER] Fixed case-sensitive login handling The GanymedeServer login process was mishandling login authentication, by canonicalizing user login to lower case, but using a case-sensitive comparison to find matching admin privileges. GanymedeServer.login() now uses a case-insensitive query to look up a user's login privileges. This is another thing that Pug caught. 27. [SERVER] Fixed mail handling for persona names Another fix for a Pug bug. Pug reported that attempting to log into a server with a fully qualified persona name, like 'pug:admin', and messing up on the password would cause the server to attempt to send mail to 'pug:admin@mailserver.com', which his mail system ultimately resolved to be equivalent to 'admin@mailserver.com'. The DBLog code that handles system event mail will now truncate at the colon, so a bad login for 'pug:admin' will be sent to 'pug@mailserver.com', as it should. 28. [PASSWORD CLIENT] Fixed PasswordClient compatibility for 1.2 The PasswordClient class used by the changePassword program attempted to set an RMISecurityManager, which is not allowed under Java 1.2. As a result, running changePassword on a 1.2 JVM would cause a security exception to be thrown. Sun really makes it hard to be both 1.1 and 1.2 compatible, some times. Fortunately, the Ganymede clients work acceptably well with the default security manager, since we package all of the RMI stub classes with the client and password jar files. This was yet another fine bug report by Pug. 29. [NIS PASSWORD CLIENT] Fixed rpcpass non-termination errors There were certain conditions in which the rpcpass client proxy would not properly exit the JVM, but would instead linger until manually killed. Now, the client will always terminate if something goes wrong, rather than sticking around. This time, for real. 30. [SERVER] The dump task now reports how many transactions it is dumping The dump task, which typically runs every couple of hours to consolidate the transaction journal into the ganymede.db file, will now announce the number of transactions in the journal to the log and admin console when it runs. 31. [SERVER] Fixed supergash admin persona renaming in DBStore.initializeObjects(). The Ganymede server wasn't properly renaming the supergash admin persona on startup if the ganymede.properties file specified an alternate name for the supergash persona. I've fixed this.. the server will now always reset the supergash persona name to that specified in the server's ganymede.properties file. Thanks to Alex Robinson (Alex_robinson@bigfoot.com) for reporting this. 32. [SERVER] Improved object cloning with partial failure on vector copy I happened to notice that when I attempted to clone a user who belonged to one or more groups that I did not have permission to put into a new user, Ganymede did not react well. That is, I got a dialog saying that certain items could not be cloned, and when I looked at my newly cloned object, several of the fields were just totally blank. I've now modified DBField.addElements() and InvidDBField.addElements() so that they are able to handle partial copies, and so that the DBField.copyFieldTo() method takes advantage of this to make object cloning more robust. 33. [CLIENT] Added a fabulous new XML client When installing the client, a new text-mode client program called xmlclient is installed. xmlclient is able to bulk load data into the Ganymede server from an XML file. Loading data through the xmlclient is less efficient than the old custom runDirectLoader hacks that data loaded depended on before, but it is far more versatile and lets you load in any kind of data that you can specify in your Ganymede schema definition with nothing more than a Perl script and this client. Ultimately, I want to make the xmlclient able to not only load arbitrary Ganymede data from an XML file, but for it also to be able to connect to an empty Ganymede server and establish a custom schema from an XML file. Supporting this and converting all of the existing schema kits to this method of server initialization is the big goal left for Ganymede 1.0. 34. [SERVER] Made the server less picky about setting owner groups ahead of time The Ganymede server now allows non-interactive clients (clients that have turned wizards off) to create objects without having previously selected one or more owner groups to be the default owners for newly created groups. This was only an issue if the user that was creating objects belonged to more than one owner group, such as supergash. Now, if supergash creates objects through a non-interactive client, the objects will be created with null (supergash) ownership. If a non-supergash user that belongs to multiple owner groups creates objects through a non-interactive client, the objects will be owned by the first owner group found by the server. All of this was done to simplify things for XML bulk loading. 35. [SERVER] Made the server not complain about redundant invid loading I'm finding that I need to make a number of tweaks to the server so that it behaves in a reasonable way in the XML client context. In many ways, the server was designed around interactive usage, and a number of issues are coming up with the batch mode XML client. In this case, the server previously complained if an object reference (invid) was added to an vector object reference field that already contained that object reference. Now, the server silently discards redundant invid additions. This allows the XML client to have a symmetric invid relationship in which both sides of the relationship show the link without having the server reject the second object's invid links as redundant. 36. [SERVER] Improved error reporting on object editing contention The Ganymede server previously returned a misleading error message if during editing the user attempted to cause an object to be linked to an object that was already checked out for editing by another user on the Ganymede server. Now, if a link to an object cannot be established because the object is already checked out for editing by another user, the server will report the error properly, and will identify the user that has the object checked out for editing. 37. [SERVER, CLIENT] Fixed the deferred shutdown option The 'shutdown the server when all users log off' mechanism was broken, and didn't actually shutdown the server even when all active users logged off. This has been fixed. Additionally, the Ganymede ClientBase class has been reworked slightly so that if the server is in deferred shutdown mode, the login failure message will explain this, rather than the generic 'bad username/password?' message. 38. [SERVER] Added deleteAllElements() to the db_field interface I added the deleteAllElements() method to the server's DBField class and db_field interface, to allow the new XML client to remove all values in a Vector DBField without having to do an additional roundtrip to the server to get the list of elements currently in the field. 39. [SERVER] InvidDBField.createNewEmbedded() now includes a db_object ref Another change to support the XML client efficiently. The InvidDBField createNewEmbedded() method now includes a db_object reference as well as a serialized Invid. This removes the need for the XML client to make an additional round trip request to the server in order to get an editing handle on a newly created object. 40. [SERVER] Added support for high speed log scanning The Ganymede server has always had support for scanning its own log file to produce an object summary report for the Ganymede client. Unfortunately, Java is rather inefficient at text file parsing, and the log scanning code converts each line of the log file scanned into a DBLogEvent object. This conversion made coding convenient, but made scanning a very large log file much much slower than a simple grep would be. The Ganymede server now supports an external Perl script which prefilters the log file on request, so the Ganymede server only needs to process those lines that match the object being reported on. This makes the object history feature much much faster, as the server doesn't have to process unrelated log entries. If an object (such as an owner group) actually does have a very large number of lines in the log file, parsing and reporting on what might be megabytes of log data will still be very slow, however. At some point, the log scanning code should probably be written to avoid the object conversion step in favor of simply identifying matching lines through textual means and appending the text of such lines to the object history report. The external helper program is an optional acceleration.. if the 'ganymede.loghelper' property isn't defined in the server's ganymede.properties file, then log scanning will be performed in the old-fashioned style. 41. [CLIENT] Fixed persona creation in user object panel The Ganymede client allows new admin personae to be created in two different ways. In 0.99.9, however, creating a new admin persona from within the user editing window threw an exception. It is now possible to create new admin personae either directly through the client's tree by creating a new 'Admin Persona' object, or by editing a user object and using the Personae tab to create a new persona. 42. [SERVER] Optimized critical database lookup code. This one's embarassing. A long time ago I heavily tuned and optimized the Ganymede server's database hash structures. A job well done, I said to myself. Unfortunately, the DBSession.viewDBObject() method which everything in the server actually uses to pull objects out of the server was doing some extremely dumb things. In particular, before ever looking in the highly optimized database structures, viewDBObject() looked to see if the object being sought was already checked out for editing by the session's transaction. Unfortunately, the DBEditSet transaction class tracked objects checked out for editing in a linear Vector rather than a hashing structure, so as more and more objects got checked out by a session, *all* database access operations on behalf of that session got slower and slower and slower. This turns out to be the reason why the Ganymede server's bulk loading was so very slow. Now this transactional data structure has been converted to a Hashtable, and viewDBObject() has been extensively optimized so that it will only attempt to find an object in the transaction's hashtable if an object has not yet been committed into the main database structures. All transactional object view, edit, delete, inactivate, and clone operations will involve *much* less work as transactions get bigger. I knew there was a reason I hadn't called it 1.0 yet.. 43. [SERVER] Schema Editor dumps database after making schema changes A number of people, including Christoph Litauer (litauer@uni-koblenz.de), reported problems resulting from the server not consolidating its on-disk database after schema changes were committed. The problems arose whenever changes were made to the server's schema in the schema editor after which transactions were committed to the journal file before the server was abnormally terminated, leaving a mix of object data in the journal file from after the schema change, which is not itself recorded in the journal file. The server will now dump its database to disk and consolidate the journal file when schema changes are committed, so there is no opportunity for a mismatch between the schema in the ganymede.db file and the object change records recorded in the server's journal file. 44. [SERVER] Object Event Mail Is Now Coalesced Another change in response to bulk loading with the XML client. It used to be that people who requested mail notification when certain operations happened to certain kinds of events (such as when a user was created or deleted, say) will now get a single message with a summary of all events of that type during a transaction rather than a single message for each event. We had a user who was subscribed to system creation objects and was getting hundreds of messages all at once every time I ran my bulk loader test. Now he'll just get an easily deleted summary. In addition, when these events do happen one at a time, the message subject in the mail sent out by Ganymede will now include the name of the object operated on. 45. [SERVER] Expiration/Removal dates may not be set to times past Jeff Sims (jsims@arlut.utexas.edu) reported that Ganymede erroneously allowed a user's Expiration date to be set to an already passed date. The admin intended to set the Expiration some months in the future, across a year boundary, but neglected to change the year, causing Ganymede to expire the object at the next run of the Expiration Task. Ganymede will now forbid any changes to an object's Expiration or Removal date that would set either date to a time earlier than the Ganymede server's notion of current time, as a safeguard against this sort of mishap. 46. [SERVER/CLIENT] Server now has soft timeout option The Ganymede server has always had the ability to time out users that were idle in the Ganymede client. Tom Embleton (embleton@arlut.utexas.edu) expressed dissatisfaction with this, for the reason that having to login again from scratch forced him to readjust his windows and was a pain in general. In response, I implemented a soft timeout option that can be turned on by activating the 'ganymede.softtimeout' property in the server's ganymede.properties file. If this option is chosen, the server will send a time out message to the client, which will respond by downshifting from any admin persona to the base user login and presenting a mandatory persona selection dialog which will insist on being given a proper password in order to let the user continue work. A soft timeout will still close any transactions open at the time of the server timeout, but it will be faster to enter the user's password into the persona dialog and get back to work than to login from scratch again. In addition, the server's two time limits (for idle sessions with no objects checked out and for idle sessions with objects checked out for editing) are now configurable in the server's ganymede.properties file. 47. [SERVER] Added shutdown semaphore to keep shutdown from breaking builds I have added an additional semaphore to the server so that when an admin initiates a shutdown of the server from the admin console, the server will wait for any builder tasks waiting on phase 2 (execution of external scripts for NIS/DNS/etc. builds) to complete before shutting down. This is to insure that a builder task in phase 2 won't interrupt an external build script or program, leaving a network service in an inconsistent state. 48. [SERVER] Fixed namespace-optimized lookups on IP fields The server has special logic in GanymedeSession for optimizing query lookups on namespace-indexed data fields. That logic was not properly handling IP address fields when given an IP address in string form rather than in byte array form. Now, the query logic will properly convert IP strings to byte array form for optimized namespace lookups. 49. [SERVER] Fixed queries on embedded objects The GanymedeSession queryDispatch() method will now properly perform a search on an embedded object if a Query is submitted whose requested returnType is equal to the searched-for type. My initial decision to include embedded objects in the Ganymede design has made *so* many things painfully complex in the whole system. Sigh. This is another bug that showed up in development and testing of the XML client. 50. [SERVER] Optimized transaction commit The server code had a potentially quite expensive checkpoint() operation in the DBEditSet.commit() method. This checkpoint operation was there to allow the server to set the historical fields (creation and/or modification date and user identity) before the commitPhase1() methods were called while still being able to do a rollback() if a commitPhase1() method rejected the transaction commit. This checkpoint and rollback pair could be enormously expensive if the transaction gets real big, as it can with the xmlclient. The commit() method now forces the new date and user identity information into the transaction's edited objects in a manner that bypasses the object value freeze put in place during an early point in transaction commit. This change will make committing large transactions a good deal faster, and will make failed commit attempts even faster than that. 51. [SERVER] Improved startup sequence The server now intelligently checks to make sure that no server is already running at the same host and port, rather than simply ignoring the warning from the RMI rebind process. The server will now decline to start up if another server is answering the phone at the same address. 52. [SERVER] Fixed PermissionMatrixDBField loading The server had a bug that would prevent the database from loading if a role had a permission matrix defined with no entries defined in it. 53. [INSTALL] Install scripts now check to make sure jars were built All of the install scripts now double check to make sure the jar files were made during the system build. 54. [SERVER] Reworked checkpoint handling for non-interactive transactions I have simplified how non-interactive transactions handle checkpoints.. basically, the DBEditSet class now automatically ignores checkpoint activity for non-interactive transactions. Any rollback request made during a non-interactive transaction will result in the transaction committing hari-kari at commit time. By putting all the logic for this in DBEditSet, I've simplified a fair bit of code in InvidDBField, DBEditObject, and DBSession. I should have done this before. 55. [CLIENT] Added count indicator to StringSelector The vector field GUI component now includes a count number, to make it easy to see how many items are in the field. 56. [CLIENT] Windows now kept within bounds It turns out that the default Swing window resize/move logic made it possible to resize/pull windows above the top of the inner desktop, such that the title bar was no longer accessible. Richard Mach here at ARL:UT graciously pointed out this problem, and it turned out to be pretty easy to fix. The Ganymede client will now block attempts to resize or pull a window out of the visibility area to the top and left. 57. [CLIENT] Fixed owner panel on view windows The right-menu popup menus weren't enabled on view object windows in the owner panel. Now, you can view any object and be able to right-click on the items in the owner list to directly view or edit the owner groups that own that object. 58. [SERVER] Fixed object cloning in previously broken case The object cloning logic was not previously excluding object linkage (invid) fields that were involved in a one-to-one or many-to-one relationship. The upshot of this was that when a user here cloned an I.P. network, the cloned network had duplicated not only a list of rooms for the network to be present in, but also the systems, thereby unlinking those systems from the original in the process. This caused much confusion until the cause was detected. Thanks again to Gil Kloepfer for pointing this out. 59. [SERVER] Fixed rollback for object deletes This was a really critical bug. The result of this bug was that if an object deletion was attempted, but was refused due to consistency problems, the system would rollback everything *except* the object's to-be-deleted status. This meant that if the user went on to commit a transaction after unsuccessfully deleting an object, that object would be deleted anyway. Worse, all the objects that should have had their links to the object undone as part of the deletion operation will retain their links to the deleted object, ruining the symmetrical pointer integrity constraint of the Ganymede database. This bug and #58 above manifested as part of the same incident, so thanks must go again to Gil Kloepfer for demonstrating a really ugly problem. This bug has been in Ganymede since 0.99.8, lurking quietly for over 10 months. The fix was to move one line of code in DBSession up about three lines. Amazing how these little touches matter. 60. [SERVER] Serialized checkpoint logic, removed client access to checkpoint() In examining the checkpoint and rollback code, I realized that there was nothing in the system that would prevent multiple threads from attempting to assert and rollback checkpoints on the same transaction. This meant that it was in principle possible for the server to interleave checkpoint operations, leading to unpredictable and incorrect results if multiple threads simultaneously attempted to do things like object creation, deletion, and linking within a single session's transaction. I don't believe this does ever happen, in practice, but if it ever did it would be confusing and a pain to debug. To avoid this, I put in a system whereby only one thread at a time is allowed to have a checkpoint on any given transaction. If another thread attempts to checkpoint a transaction while it is already checkpointed by a thread, the thread that is attempting to checkpoint will block until the earlier checkpoint is popped or rolled back. This scheme will guarantee safety of checkpoint operations at the cost of an mildly increased risk of deadlock. To mitigate the deadlock risk, I removed the ability for remote clients to access the checkpoint() and rollback() methods in the GanymedeSession class, and I put a deadlock timeout on the checkpoint block, so if a checkpoint() attempt blocks for more than 60 seconds, an exception will be thrown and the checkpointing thread will let go. The loss of client access to checkpoint and rollback doesn't seem to be an unreasonable thing to do, as no client code that I have yet seen has done anything with the intra-transactional checkpoints. If someone convinces me that it is critical that remote clients be able to access intra-transactional checkpoints, I'll need to rework the anti-interleaving code to deal with the fact that subsequent RMI calls from a single client are not guaranteed to be handled by the same thread on the server. 61. [SERVER] Fixed deadlock vulnerabilities Previously, the GanymedeServer object used Ganymede.internalSession to process database queries for user and admin console logins. The problem was that this was prone to deadlock with some other uses of Ganymede.internalSession in the Ganymede server. In particular, if anyone ever attempted to login at the same precise moment that a transaction was being committed, the whole server locked up. Now, the Ganymede server uses a distinct GanymedeSession object to handle logins. Since GanymedeServer has its own synchronization at a higher level, it is no longer possible to get nested monitor deadlocks from logins. In addition, since the Ganymede.internalSession object is often used by random Ganymede server code to look up things in the database, I partially de-synchronized DBSession.viewDBObject() so that a DBSession that does not have an open transaction asssociated with it will not incur a synchronization penalty/deadlock opportunity merely to retrieve a reference to an object from the database. 62. [SERVER, CLIENT] Added -dump{schema|data} to the xmlclient, removed decodeDB The xmlclient is now able to emit a copy of the server's schema definition in xml format, using xmlclient -dumpschema, and to dump the server's database in its entirety in xml format, using xmlclient -dumpdata.. Due to this, I have removed the old hackish decodeDB and permdecodeDB front ends to the Ganymede server. 63. [SERVER, ADMIN CONSOLE] Added runEmbeddedTest, runEmbeddedSweep Bug 59 above, the deletion rollback bug, allowed some inconsistencies into our database, wherein embedded objects were left in the database when their owners had been deleted. I added test features to the server and the admin console to detect and repair this condition. 64. [CLIENT] Made Client check for MOTD message in background The client now runs its check for a new MOTD in a background thread, so as not to lock up the GUI thread initially. This will get rid of the ugly lack of refresh that was visible when the user dismissed the select persona dialog. 65. [CLIENT, ADMIN CONSOLE] Cleaned up login boxes, console shutdown dialog I finally went back and got the admin console and gui client to put the focus on the username box, so people can just start typing their name when the clients come up. I reworked the admin console shutdown dialog so it just uses one dialog box for the various shutdown modes. 66. [INSTALL] Removed installClient2, made installServer autodetect java version A bunch of changes to the installation process. installClient2 is now gone, installClient is now smart enough to autodetect what version of Java is being used and to configure the client install appropriately. Ditto the installServer script. I cleaned up the gasharl schema kit's ganymede.schema file to take care of some inconsistencies that have been in that file for a long while. 67. [CLIENT] Tweaked right-menu clicks in StringSelector to select and popup Mike Mulvaney (ex-Ganymede GUI guy) filed a bug report indicating that clicking the right mouse button should select the item that the pop-up menu is being applied to. He was right, and I so fixed it. 68. [SERVER SECURITY] Fixed a loophole in the object permission system The Ganymede system allows admins and users to be given editing privileges to objects that are not considered to be owned by the admin or user. This is an unusual circumstance, usually only done when the customizer wants objects to appear in the client's object tree but doesn't want to allow certain (or even any) fields to be modified. The problem was that the built-in fields, including the owner list field, do not typically have their own access permissions tracked, independently from the object's access permissions. This allowed any user who could edit an object to grant themselves ownership of the object, which is bad. The server now refuses to grant edit privilege to an object's owner list field unless the user already owns the object, or is running with supergash-level privileges. 69. [SERVER, XMLCLIENT] Added XML-based schema editing through the xmlclient The xmlclient can now be used to perform schema edits on a running Ganymede server (so long as no users are logged in, and as long as the schema edits do not result in an incompatibility that will break loaded data). This makes is possible to get rid of the old directLoader crud in the schema kits. 70. [SERVER, ADMIN CONSOLE] Fixed exceptions on drag/drop in schema editor Fixed bug #13 in bugzilla. 71. [SERVER] Added TaskOptionStrings to Task definition The built-in object type, 'Task' now supports an optional vector string field, which will be used in 1.0 to represesnt builder task options for schema kits that support more than one transaction-time builder task. 72. [CLIENT, SCHEMA EDITOR] Fixed tree's pop-up menus on Linux The GUI tree component was still using old-style AWT pop-up menus, which wasn't working properly on Linux using Java 1.3. Since almost everything in the Ganymede client and admin console are based on Swing, switching the pop-ups to Swing should keep things consistent and working on various platforms. 73. [ADMIN CONSOLE] Fixed console deadlocks on Linux I was being sloppy with my GUI thread handling in the admin console.. updates from the server were not being painted on the GUI thread, which was leading to swift lockups when running the admin console on the Linux 1.3 JDK. I fixed this, the admin console should run fine in any Java environment now. 74. [SERVER] Reworked query engine to increase concurrency The Ganymede server no longer has to block threads from doing most queries on the server while a transaction has portions of the database locked for updates. The server now makes an interal snapshot vector of the objects in an objectbase when transactions are committed, and most queries are run against these snapshots. The Ganymede server will still lock the database on a standard GanymedeSession query if the query spans more than one object type. In practice, this means that unless you or your code attempts to do a query that involves embedded object values (as in searching for a system using a give i.p. address held in an embedded interface object), the query will never block. For this reason, it is now safe to take advantage of the Ganymede query mechanisms during all stages of transaction commit in DBEditObject subclasses. The main tradeoffs for doing this are a slight increase in memory usage (4 bytes per object in the database), to maintain a second copy of references to all objects in the database, and a slight increase in transaction commit time, as the new snapshot is constructed. Another, more subtle trade-off is that support for this level of concurrency in the server might be hard to maintain if the Ganymede server is ever reworked to move away from its in-memory database. 75. [XML] Simplified the XML schema for integer and boolean fields The XML file format used to require boolean field values to be expressed as or , and integer numeric values as and the like, which seemed unnecessarily wordy, since boolean and numeric fields are always scalar. Now you can do things like 1005 and rather than and The Ganymede XML loader will continue to recognize the older style for backwards compatibility.. 76. [SERVER] Memory optimizations I made a number of optimizations to memory usage in the server, which should cut the amount of per-field overhead roughly in half. Heap required for objects in the database has been reduced by at least 10-20 bytes plus 10 bytes per field in each object. Fields in the database now have only 14 bytes of overhead each, plus whatever overhead your Java VM needs to track garbage collection for two objects. To accomplish this, I reduced the number of variables held in the DBField class to an absolute minimum and restructured DBObject so it uses a simple array with packed hashtable access logic rather than a separate hashtable. Custom DBEditObject subclasses that directly manipulate the old fields FieldTable variable will need to be reworked in order to work with the new field storage structure. In addition, the server will now create a lot fewer temporary Enumeration objects during runtime, and a lot fewer PermEntry objects, both of which will have some impact on reducing memory usage. Schema edit commits will be faster, but otherwise the overall speed of the server will probably not be much changed. 77. [SERVER, CLIENT] More Memory Optimizations, History Panel revisions I reduced the size of the creator info and modifier info strings so that they will only include the name of the user/admin and the name of the client used to create/modify the object. This will save about 100 bytes from each object created right off. In addition, since the time is no longer included in these fields, the automatic interning that the server does to eliminate redundant strings will reduce the overhead for these data elements by a very, very large percentage. To compensate for the creation and modification time no longer being included in the history string fields, I have modified the client to display the history date fields with hour, minute, and second as well as date. 78. [SERVER] Memory, security fix The new system for handling DBField/DBObjectBaseField association revealed a longstanding bug in DBObject that prevented Ganymede from properly de-linking fields from their previous DBEditObject when a transaction was committed. This had the effect of preventing the last DBEditObject constructed for a particular database object from being garbage collected unless and until the object was actually deleted from the database. A small memory leak. This bug also allowed clients the possibility of at least attempting to continue to perform edit operations on fields after a transaction was committed and the editing objects were checked back into the server's database hashes. Both conditions have been solved, which will reduce the server's active working set somewhat during runtime and which will protect the server against malicious clients. 79. [SERVER] Fixed logging of newly created Role objects The PermissionMatrixDBField wasn't properly reporting the value of a newly created Permission Matrix in the log file. 80. [SERVER] Fixed permissions xml parsing Fixed the xml parser's handling of permission matrix field objects. 81. [SERVER] Redfined DBEditObject.canRemove() to allow dialog return Modified the canRemove() method in the DBEditObject class to allow a complete ReturnVal to be returned rather than the old boolean. -------------------- Changes from 0.99.8 to 0.99.9 ------------------- RELEASE DATE: January 13, 2000 1. [SCHEMA] Fixed gasharl schema errors a. The gasharl schema kit had a bug which made it impossible to edit a system if that system was connected to a room that had no networks listed. b. The gasharl GASHBuilderTask code is designed to handle netgroups that overflow the 1024 byte NIS line length limit, but the code that handled netgroup folding did so very inefficiently, putting only one netgroup entry per line after the first. 2. [SERVER] Fixed race condition in Qstmp stopThreaded() method The Qstmp shutdown process could throw a NullPointerException in stopThreaded() if thread timing worked out just wrong. This has been caught, and this race condition will no longer be able to prevent clean server shut-down. 3. [SERVER] Fixed various deadlock possibilities in server a. The 3 DBLock subclasses (DBReadLock, DBWriteLock, DBDumpLock) did not properly call notifyAll() on the DBStore object if they threw an exception in their establish() method. Now, all synchronized blocks in the DBLock subclasses use try..finally to guarantee that other threads waiting on DBStore will be woken up, even if something unexpected happens. The DBReadLock.establish() code had code to throw an exception if a session that possessed a write or dump lock tried to obtain a readlock at the same time. Unfortunately, the internal sessions used for the Ganymede builder tasks, expiration task, and warning task were given the same identifier as the standard 'internal' session used to process queries in the login process, so if a login was attempted during a builder task's phase1 execution, or during portions of the expire or warning task, an exception could be thrown, possibly causing threadlock if other threads had wait()'ed at that point. All this should be fixed now. b. There are several ways a user can be forced off of the Ganymede server.. the idle timer, the admin console, and the RMI unreferenced object mechanism. Previously, various essential objects in the server could get locked if the client refused to respond to the server's disconnect command. Now the client force-disconnect code is better coordinated so only one thread in the server will ever attempt to force the user to disconnect, and that thread will itself spawn a background thread to force the user to disconnect. As a result, although the server may wind up with a single dangling thread per client if the RMI system blocks trying to call the client, nothing in the server will deadlock because of it. As a result of this change, knocking a user off the system by the admin console is no longer a synchronous operation, so it may take a few seconds before the admin console updates to show the results of a user kill operation. c. The admin console code in the server had a number of problems that could lead to a deadlock that blocked any admin console operations (including the stopServer script). The server code that handles communications to/from the admin consoles have been significantly reworked so as to not be vulnerable to a threadlock condition, and some redundant operations have been pruned to reduce overhead. A number of possible race conditions have been closed. This will make the server faster and safer when admin consoles are attached, but the admin console updating is still generally synchronous. Every admin console connected to the Ganymede server will slow th