
                               COSH.EXE

             A COM Shell for TCL running on Microsoft NT

                                  by

            David Shepherd at ICR GmbH, Ueberlingen, Germany



COSH is a shell programm for TCL like WISH - in fact it *is* WISH with
the addition of a COM interface called ITclScript. Perhaps you could call 
COSH the "COM version" of WISH. 

For anyone involved in distributed processing on Microsoft NT systems, COM
is the first word in the vocabulary. It turns out that COM and TCL can work 
together quite happily and can actually compliment each other in a couple of 
important respects. Some points to note about COSH are:


    COSH differs from WISH in that it can be remotely started 
    on any machine via a call to the WIN32 API "CoCreateInstance". 
    It does this by implementing a COM "Class Factory"

    You can write in scripts at the COSH console, or you can pass in 
    scripts for evaluation via the ITclScript interface. This lets you
    remote start plain-vanilla COM shells and then pass in a bootstrap
    script for evaluation. This makes COSH a good wrapper programm for 
    server-style scripts running in a distributed environment under NT.

    There is a separate version of COSH (coshd.exe) which is built on 
    the standard IDispatch interface, so you can start and call into 
    COSH instances from COM automation clients such as Visual Basic and 
    Delphi. Thus in Visual Basic you could, for example, start a COM 
    shell instance on any machine, then construct a Tcl script in a 
    VB string variable and cause it to be evaluated in the remote COSH 
    interpreter.

    When programming components designed for things like  Microsoft 
    Transaction Server or Active Directory Services, Microsoft has 
    mandated that these be packaged as COM components. COSH (or an 
    equivalent in-proc server, aka DLL) can function as a generic COM 
    component and thus can leverage Tcl code directly into the Microsoft 
    distributed processing mechanisms.

    It is also easy to install and use.



COSH Installation
==================

COSH needs TCL 8.0 to run and VC++ 5.0 to compile

The following files need to be installed on any machine which will run COSH. 
On the server side they are:

  -  COSH.EXE         The TCL shell programm (like WISH)
  -  COSHPS.DLL       A "Proxy/Stub" DLL used to remote client calls to COSH
  -  COSHD.EXE        Same as COSH, but built on IDispatch
  -  COSHD.TLB        A Type Library needed by the IDispatch interface
  -  COSHREG.TCL      Registration script

You can install these files to any directory you like. Copy the files there
and then run the script COSHREG.TCL from that directory. This will create
the relevant registry entries needed by COM to remote start the shells.

In addition there is on the client side, as a demonstration

  -  COSHCLIENT.DLL   A TCL extension DLL demonstrating a COSH client
                      which calls the ITclScript interface directly (vtable).
                      See below for details. Also note that the "eval" function
                      in ITclScript can be called by any COM automation client
                      via the standard IDispatch interface in coshd.exe.



Why bother with COM?
=====================

Microsoft has gone all out over COM and has successively built larger portions of
COM and OLE into the operating system itself. For anyone working on programming
distributed systems on NT, COM is *the* method of choice to get servers and
clients connected and components managed, because you are using native OS services 
to do it.

But while COM of itself, being a relatively elegant solution for component inter-
operability, is a Good Thing, Microsoft has built a services superstructure 
on top of COM (OLE, ActiveX et al) of enormous complexity. If, as a software
developer, you subscribe solely to the Microsoft Way, you already have your work
cut out for you - its reputation for being hard to learn and use is thoroughly 
deserved.

Fortunately, COM/OLE is not an all or nothing proposition. You can also say "as
much COM as necessary and as little as possible". 


A distributed scenario
=======================

COSH comes out of a project for a system of distributed server processes which has
some similarity to workflow systems. A requirement of the system is that it can be
controlled from a single point-of-administration. This central administration program 
can create and monitor a number of server processes on various machines, can perform 
load balancing among them etc etc. The servers read from and write to message queues 
and can be arranged into "processing networks".

This product runs on Windows-NT and has a modular component architecture, so COM is
the first thing which springs to mind. However, we also have a significant investment in 
TCL scripting technology and, seeing the amazing flexibilty which it affords us, we
want to retain TCL as far as possible as the glueing technology of our products. While we
do not track each and every Microsoft technology as and when they are released, we 
definitely want to use such services as DCOM, Transaction Server, Message Queue, Active 
Directory and so on. On the other hand, we like to package our components as TCL extension 
DLLs and scripts - a nice, easy, effective and lo-tech way of doing things. What to do?

In the end, the idea of a COM-capable TCL shell distilled out of what we
wanted to do. We would install on each machine in the network (1) TCL/TK plus a
number of support packages and (2) a generic TCL shell which can be remotely
activated and can accept scripts for evaluation from its client. The "client" in
this case is a central administration and monitor programm, displayed as a 
"Game Board". This program creates COM shells as needed according to its load
balancing rules and bootstraps them as server processes by pumping scripts for 
evaluation into them at load time via the ITclScript interface. This is nice
because all the server scripts can be kept in one place.


What is the current status of COSH?
=======================================

On a scale where 0.1 represents "almost useless" and 1.0 is "production quality"
COSH is about at 0.70. Plenty still to do, but on the other hand you can do
at the moment about 90% of all things you need to do with it.


The ITclScript interface
=========================

Disregarding for a moment the standard COM functions such as QueryInterface,
AddRef and the like, this interface contains exactly one function, called "eval". 
It is prototyped in IDL like this:

    HRESULT eval ([in] BSTR script, [out] BSTR* result, 
                  [out] long* retcode, [out] BSTR* errorinfo);


It is pretty obvious what this function does. It takes an arbitrary script
and evaluates it in its host interpreter, returning the result, the status
code (TCL_OK, TCL_ERROR etc) and maybe the errorInfo stack trace.





The CoshClient Extension DLL
==============================

A TCL extension DLL called CoshClient.dll is part of the distribution.
It demonstrates how to remote start, terminate and feed scripts to a 
COSH program. It makes use of the vtable interface in COSH.EXE and uses
COSHPS.DLL to marshall the call parameters across process boundaries. It
instantiates an object of type COMSHELL. It implements the command:

     comshell ?machine-id?

When called without the machine-id argument, the command attempts
to create a COSH instance on the local machine, as specified in the 
relevant registry entries. When machine-id is specified in UNC
or IP form, it tries to create a remote instance, using the remote
registry. 

If successful, the command returns a token (cosh000, cosh001 etc) which 
is itself a command. Use this command as follows:

     cosh000 eval script

     Synchronously evaluate $script in the COM shell. If an error
     occurs on the remote machine, it is propagated to the local
     machine and the remote "errorInfo" call trace is added to the
     local one.


     cosh000 release ?-lockserver?

     Release our reference on the interface. This will normally cause
     the server to shut down. Add option -lockserver to keep the
     server running. (Performs an additional AddRef on the interface).
     The server shuts down by internally calling the "exit" command.
     The token is thereafter invalid.


     cosh000 getptr

     Get the actual pointer value of the Interface. It is returned
     formatted with "%08X". Formatted back to a 32-bit value, you
     could pass this pointer on to other COM aware programs and
     reference the COM shell from there. (That programm would need
     to call AddRef on the interface pointer)


ITclScriptDisp Dual Interface
==================================
COSHD.EXE implements the same function "eval" as described above, but
builds on the standard IDispatch interface instead of using a custom
proxy/stub DLL to do the call marshalling. If you wish to connect to
a COM shell via this interface, you must instantiate an object of type
COMSHELLD.


COSH Activation options
========================

At present COSH is hardcoded as a "Single Use" server. That means each
time an interface is instantiated with a call to CoCreateInstance, a new
and independent COSH is started. This precludes connecting to an already
running server.

In the source code there is a define (REGISTER_ACTIVE) which will activate
code which enters the current instance of the COSH server in the Running
Object Table (ROT). Any client which from then on requests an instance of
ITclScript on that machine will receive a pointer to that running instance. 

One nice thing here is that COM will serialize multiple client calls so that 
no problems arise with the non-multithreadedness of TCL. 

If you need to mix those models - ie sometimes connect to a running instance
and sometimes create a new one on a single machine, then it looks at first
glance as if you are out of luck. This is because, when you connect via
COM, you are connecting to an *interface*, and that interface has only
one definition on any one machine (which you can find in the registry under 
HKEY_CLASSES_ROOT\\Interface\\<Interface_ID>). I suspect, however, that it 
will not be too difficult to find some kind of workaround for that.


The call to "eval" in ITclScript is synchronous
=================================================

Of its nature, COM only deals in synchronous calls. When you start to need
asynchronous calls, callback notification, exception propagation and what
have you, things start to get very hairy very quickly when you have to do
things the Microsoft Way. You need to start creating interfaces which, although
defined in the server, are actually implemented in the client. (Or was it the
other way around?) Your client has more or less no choice than to run multithreaded, 
which creates a whole new dimension of hassles and snags for what are, for the 
most part, otherwise relatively simple server and client programms.

Thus there is no provision for callback mechanisms in the ITclScript interface 
- all calls are synchronous and that's all there is to it.


However, TCL would not be TCL if there were no way around that.

I would like to mention in this connection a mechanism we haved used to
achieve the same end. It involves using not COM as the RPC mechanism but the
TCL-DP package. Our problem was, from a single point-of-administration to
start and monitor various servers in a distributed system, preferably without
resorting to the use of launch daemons. We use an ITclScript client to remotely 
start the COSH shells on the various machines. Then that client programm passes 
in the scripts necessary to bootstrap the servers via the ITclScript interface. 

The client code looks like this:


  set csh [comshell 156.98.0.61]   ;# start remote comshell and return token to it

  #====== read the server bootstrap script from local file

  set f [open server_script.tcl r]
  set script [read $f]
  close $f

  #======= pump server script into comshell using token from above as command

  $csh eval $script



Next we tell the comshell to create a DP server socket on any available TCP port
and to return the port and ip-address needed to connect to it, to which the client 
then connects.


   #====== start dp server in the remote COM shell
 
   set svr_addr [

       $csh eval {      ;# eval this script on the remote machine

            package require dp                               ;# needs dp from Zeno
            set connect [dp_MakeRPCServer]                   ;# make server on any port
            lappend retval [fconfigure $connect -myIpAddr]
            lappend retval [fconfigure $connect -myport]

       }
    ]

    #========= retrieve server address and connect

    set ip [lindex $svr_addr 0]
    set port [lindex $svr_addr 1]
    set sock [dp_MakeRPCClient $ip $port]


At this point we have two different connections open to the COM shell - a DP socket
and the COM connection. The two connections compliment each other in the sense that,
while in DP you can only call already existing procs in the server (either synchronously
or asynchronously), the ITclScript interface allows you to evaluate abitrary scripts in
the server, (albeit only synchronously). In practice, you can create the procs via ITclScript, 
which you then call asynchronously via DP.

If you don't need to evaluate any more scripts in the server but only calls its procs,
then you can release your reference on the ITclScript interface. Normally this would 
have the effect of terminating the server, but by calling "AddRef" on the interface you 
can lock it into memory. The code below does this (see the CoshClient code).

    #====== lock server and release our COM interface

    $csh release -lockserver     ;# release our reference with addref



You can then do asynchonous RPCs using the dp_RDO command together with the 
-callback option.



DCOM is Not Easy
====================

The thing I like most about COSH is that it keeps my COM involvement to an absolute
minimum. I don't need to slavishly follow the Microsoft Way and learn the arcane details
of the IConnectionPointContainer or IParseDisplayNames interfaces. One interface with 
one function is about as basic as you can get in COM - and there are good reasons to 
keep everything very straightforward when working with DCOM.

DCOM is definitely wierd. For one thing, when you remote start a process on another
machine, it has no window. It only shows up in the Task List. Programms with windows,
such as TK, still function OK though. I don't know why this is or if you can (or should 
want to) do anything about it. If you start it on the local machine though, it has a 
window, but only if you specify the UNC name of the computer or no name at all - if it 
is started locally via its ip address, it again has no window.

I have experienced other wierdnesses too. For instance, I have two machines networked. 
I install COSH server and client on both. Client A with Server B works fine, but not
Server A with Client B. (I suspect "Service Pack" discrepancies).

Or, a Dispatch interface which works perfectly well on the local machine will give
strange error messages when called remotely ("Stub received corrupt data" and the like).
And they told me DCOM was just COM "with a long wire" - don't you believe it!

Sometimes COM seems to hang for about 5-10 seconds or so before completing a remote 
function call - again, don't know why.

As a COM server, COSH participates in the NT Security schemes. This may cause unexpected
hiccups during development - it certainly did for me. Beware of security concerns, 
permissions and the like 


The code in COSH has been hacked from different sources - maybe there are bugs, although
everything seems straightforward enough. Maybe there are just kinks in my network setup - 
it is not very homogeneous. Maybe everything will be fine in NT 5.0. Who knows? DCOM can 
be a real pain. If you experience DCOM problems - well I am as much in the dark as you. 

First test locally, then remotely. You can connect to a running instance of COSH with
a debugger.


I recommend "Professional DCOM programming" by Richard Grimes (Wrox Press)
as a DCOM-manual and a good introduction to COM in general.



What's to do?
===============

- DLL inproc version

  An in-process version of COSH might be quite useful when integrating TCL extension
  DLLs into services like Microsoft Transaction Server. Still have to figure out a good
  way of getting the bootstrap scripts evaluated in the DLL at load time.


- Combine COSH.EXE and COSHD.EXE into one executable

  Strictly speaking, only one interface (a dual interface) should be necessary to
  enable both IDispatch and vtable bindings. However there always seemed to be a gotcha
  where that wouldn't work and in the end I gave up and made two separate versions.
  For instance, the WIN32 API "LoadRegTypeLib" insists on overwriting registry
  entries which effectively disables vtable binding. This causes at minimum a
  performance hit and at worst hard-to-debug crashes.


- command line switches ROT, safe interp etc

  COSH could benefit from a couple of command line switches which influence the startup
  behavoir - Safe interp or not. Enter the instance in the Running Object Table etc.
  Also specify security attributes.


- Auto package installer

  Might be interesting to give COSH the ability to accept arbitrary data, which it writes
  to file at the remote end. This could be the basis of a mechanism which can remotely
  install DLLs and scripts on the remote host if they're not there already. 

- "Security"

  The thought of evaluating any script on any machine is probably a nightmare for those who
  are security conscious. Still, as a COM server COSH is covered by the "NT security blanket"
  - whatever that means - and TCL itself offers some handles for implementing security policies
  through safe interpreters. Also TCL-DP from Zeno, which we use extensively in connection with
  COSH, offers further ways of limiting the scope for mischief. If anybody could offer advice
  about which mix of security mechanisms offer the best coverage, I would be pleased to hear
  about it.
  

- more friendly COM error reporting

- testing testing testing



Good Luck!



--
o David Shepherd
o Independent Software Author
o T/F (+49) 7557-91015
o Email: dshepherd@t-online.de
o http://home.t-online.de/home/dshepherd


