## mxProxy - Generic Proxy Wrapper Type for Python

Access Protocol : Cleanup Protocol : Implicit Access : Weak References : Interface : Examples : Structure : Support : Download : Copyright & License : History : Home | Version 2.0.3

### Introduction

A while ago, I ran into a security problem within Python: there was no apparent way to hide attributes from unauthorized access other than running in an restricted environment and using the standard lib's Bastion object wrapper.

Since I couldn't run the code in restricted mode (e.g. it does assignments to __class__ and __dict__ in a few vital places), the only reasonable way seemed building a new type that hides a few attributes in the C implementation. These attributes are not accessible from within Python. Of course, a debugger will give you access and some special extension module that knows about the internal structure of the types could too. But for my purpose this level security suffices.

Instead of creating a special Bastion like type I figured that it would be a good idea to provide a more generic and extendable version of a Proxy object. The object itself doesn't provide too much funtionality. It basically hides the wrapped object and provides controlled access to its attributes.

Another subject that comes up every now and then in the Python community is that of weak references, i.e. references to objects that don't prevent them to be garbage collected. Since Python uses a reference count garbage collection scheme, circular references cause memory leakage as soon as the reference to the cycle is dropped (the involved objects are not garbage collected because their reference count never drops to zero).

Proxy objects can use two flavors of references starting with version 0.2: strong references which act just like any other reference you use to the object in Python and weak references. The `WeakProxy()` constructor must be used for the latter. Details are described below.

#### Access Protocol

To control access to the wrapped objects attributes (these also include its methods), the Proxy provides two features:

1. Access is only granted if the attributes name appears in a special interface list of accessible names.

2. Public access can be further controlled if the object provides the methods `__public_getattr__` and/or `__public_setattr__`.

The access scheme first checks the interface list. If given, only attributes with names appearing in it can be set or fetched. All other accesses are cancelled by raising an `AccessError` exception (which is a subclass of the standard `AttributeError`).

The presence of the interface list also indicates whether methods that are to be returned to the requesting object should be Proxy-wrapped on-the-fly or not. Wrapped methods only have the callable slot enabled, thus inhibiting access to the enclosed object reference.

In case an attribute it to be fetched, the Proxy checks the availability of a `__public_getattr__` method. If found, this method is called with the name as parameter and whatever this method returns is returned to the requesting object. Otherwise, the Proxy uses the standard getattr()-functionality to fetch the attribute.

Setting attributes is done in a similar way: the Proxy checks the availability of a `__public_setattr__` method. If found, this method is called with parameters name and value. Otherwise, the standard setattr() functionality is used.

#### Cleanup Protocol

When dealing with large object systems circular references are sometimes introduced. Since these reference loops cannot automatically be broken by the system, the user has to provide means of breaking the circles at the proper places. One such place is the object destructor of objects that explicitly contain such references.

The Proxy object can be used to provide an indirect pointer into such a circle of objects that reference each other. When the system deletes the Proxy object, its destructor tries to find a special method `__cleanup__` in the wrapped object and calls it before continuing the destruction. This allows the object to break the reference circles, e.g. by performing a `self.__dict__.clear()` or something similar.

Errors raised in `__cleanup__` calls are ignored. Warnings are printed to stderr in case Python is run in verbose mode (python -v) and tracebacks printed if run in debug mode (python -d; note that printing tracebacks can sometimes cause core dumps e.g. during finalization of the interpreter).

#### Implicit Access

On many occasions object attributes are not explicitly accessed by e.g. using `object.attribute`, but indirect through builtin functions or the interpreter itself. This poses a problem to the Proxy, since there are different ways to reach an objects implementation. For Python instances all important hooks are reachable via `getattr()`, e.g. __len__ and __add__. This is different for extension and builtin types: they use a system of slots for providing access to their data.

As of version 0.2, Proxy objects also allow managing the type slot interface for most applications. This means that you can transparently use the Proxy to wrap builtin types such as lists or tuples and use the Proxy just like you would use the referenced builtin object itself.

Since Proxies are about access management, you can also restrict access to these slots. For simplicity, the names you have to use for different slots are exactly those you would define for Python classes, e.g. the length slot is named '__len__', the sequence and mapping get item slot are grouped under the name '__getitem__'. These names have to be explicitly stated in the interface list you pass to the Proxy constructor when creating the proxy in case you do define an interface list. If you don't specify such a list, no direct interface restriction is applied.

These type slot names are defined:

Generic:
  __call__, __hash__, __str__, __cmp__
Mapping:
  __getitem__, __setitem__, __len__
Sequence:
  __add__, __repeat__, __getitem__, __getslice__ __setitem__, __getitem__
Number:
  __add__, __sub__, __mul__, __div__ __mod__, __divmod__, __pow__, __neg__, __pos__, __abs__, __true__ __invert__, __lshift__, __rshift__, __and__, __str__, __or__, __int__ __long__, __float__
  Note: number coercion does not yet work, so most of these are currently useless !

Omissions currently are: __coerce__, __hex__ and __oct__. __repr__ is handled by the Proxy object itself since it would otherwise possibly expose address information about the underlying object.

Proxies also work transparently for Python instances. To achieve this, another proxy-like object which is a real Python instance is to be put in front of the Proxy object. Access to this instance then gets translated into getattr-calls by the interpreter, these calls are filtered through the Proxy and the wrapped instance object's attribute then gets accessed in the usual way. The result is passed back to the requesting object.

#### Weak References

Proxy objects work in two modes: keeping a strong or a weak reference to the object. The `Proxy()` constructor returns a Proxy object using a strong reference, the `WeakProxy()` constructor one using a weak reference.

Weak references are called weak because they don't keep the object alive by incrementing the reference count on the referenced object. Since objects get garbage collected when this reference count falls down to 0, a weak reference can become invalid at any time. The mxProxy implementation raises a `LostReferenceError` in case a weak reference to such an object is used.

This may sound like a pretty flaky feature at first, but the main pro argument for these weak references is that you can build up circular references without having to fear about them not being properly garbage collected. Using strong references (which do increment the reference count and thus keep the object alive as long as the reference is around) would produce cycles in the referencing scheme which the Python garbage collection (GC) mechanism cannot automatically break causing the cycle to become unreachable from the Python namespaces: a severe memory leak.

Weak references in mxProxy work by using a global dictionary of all objects handled through weak reference Proxies. This dictionary is checked prior to every action on a weak Proxy and after its deletion. You can also force a check by calling the `checkweakrefs()` anytime you like, e.g. at regular intervals.

The dictionary holds a strong reference to the object keeping it alive until the next check is done. During the check all handled objects are inspected to see if their reference count has gone down to 1 (phantom objects: only the dictionary references them). If this is the case, all weak proxies are marked defunct and the object is removed from the dictionary causing it to be garbage collected by the Python GC mechanism. All subsequent actions on the weak references to this object will then cause a `LostReferenceError` exception to be raised.

### Interface

#### Proxy Constructors

These constructors are available in the package:

`Proxy(object,interface=None,passobj=None)`
  Returns a new Proxy instance that wraps object.

  interface can be given as sequence of strings and/or objects with __name__ attribute or as dictionary with string keys (only the keys are currently used) and tells the Proxy to only allow access to these names. If not given or None, no filtering is done by the Proxy (see above on how access is managed).

  passobj can be provided to retrieve the wrapped object from the Proxy at a later point using the `.proxy_object()` method.

`InstanceProxy(object,interface=None,passobj=None)`
  Same as above, except that a Python instance wraps the Proxy object.

  This makes the Proxy transparent for access to wrapped Python instances, meaning that the Proxy will act as if it were the wrapped object itself (with the added features mentioned above).

`CachingInstanceProxy(object,interface=None,passobj=None)`
  Same as InstanceProxy(), except that a read cache is used by the Proxy which caches all queried attributes in the Proxy instance's dictionary.

  Note that this may introduce circular references if not used properly. Cached attributes are not looked up in the wrapped instance after the first lookup -- if their value changes, this won't be noticed by objects that access the object through this wrapper.

`SelectiveCachingInstanceProxy(object,interface=None,passobj=None)`
  Same as InstanceProxy(), except that a read cache is used by the Proxy which caches certain queried attributes in the Proxy instance's dictionary depending on their type.

  The cached types are defined by the `.proxy_cacheable_types` attribute. It defaults to only cache Python methods.

`MethodCachingProxy(object,interface=None,passobj=None)`
  Alias for SelectiveCachingInstanceProxy().

`ReadonlyInstanceProxy(object,interface=None,passobj=None)`
  Same as InstanceProxy, except that write access will result in an `AccessError` being raised

`ProxyFactory(Class,interface=None)`
  Returns a factory object for producing Class instances that are automatically wrapped using Proxy-instances.

  interface is passed to the Proxy constructor, pass-objects are not supported.

`InstanceProxyFactory(Class,interface=None)`
  Returns a factory object for producing Class instances that are automatically wrapped using InstanceProxy-instances.

`WeakProxy(object,interface=None,passobj=None)`
  Returns a new weak referencing Proxy instance that points to object.

  interface can be given as sequence of strings and/or objects with __name__ attribute or as dictionary with string keys (only the keys are currently used) and tells the Proxy to only allow access to these names. If not given or None, no filtering is done by the Proxy (see above on how access is managed).

  passobj can be provided to retrieve the wrapped object from the Proxy at a later point using the `.proxy_object()` method.

  For details on weak references and how they work, see the above discussion.

#### Proxy Instance Methods

A `Proxy` instance `proxy` defines these methods in addition to the ones available through restricted access to the wrapped object:

`proxy_object(passobj)`
  Returns the wrapped object provided the given passobj is identical to the one used for creating the Proxy.

  Simple equality is not enough -- it has to be the same object.

`proxy_getattr(name)`
  Same as getattr(proxy,name).

`proxy_setattr(name,value)`
  Same as setattr(proxy,name,value).

`proxy_defunct()`
  Return 1 iff the referenced object has already been garbage collected.

Note that all attribute names starting with '`proxy_`' are interpreted as being Proxy attributes and are not passed to the wrapped object. Access to these attributes is not subject to the restrictions explained above.

#### Proxy Instance Variables

`Proxy` instances do not provide any instance variables themselves. They do provide restricted access to the variables of the wrapped object though.

Note that all attribute names starting with '`proxy_`' are interpreted as being Proxy attributes and are not passed to the wrapped object. Also, access to these attributes is not subject to the restrictions explained above.

#### Functions

These functions are available:

`checkweakrefs()`
  Calling this function causes the global dictionary used for managing weak references to be checked for phantom objects. If such objects are found, they are garbage collected during the call and weak referencing Proxies pointing to them are defunct.

  Weak references cause the objects to stay alive until either a proxy is used on them (causing an exception), a proxy referencing them is deleted or this function is called. To ensure the timely garbage collection of the objects, call this function at regular intervals.

`initweakrefs()`
  For internal use only:
  Initializes or reinitializes the weak reference implementation. This first forces a finalization of the previous state if the implementation has already been used and then starts again with a clean weak reference dictionary.

`finalizeweakrefs()`
  For internal use only:
  Forces finalization of the weak reference implementation. Subsequent usage of weak references will cause errors to be raised.

  Calling this function after finalization is not an error.

#### Constants

These constants are available:

`AccessError`
  Exception object used for access specific errors that the module raises. It is a subclass of exceptions.AttributeError.

### Examples of Use

Here is a very simple one:

    import Proxy

    class DataRecord:
        a = 2
        b = 3
        # Make read-only:
        def __public_setattr__(self,what,to):
    	  raise Proxy.AccessError,'read-only'
        # Cleanup protocol
        def __cleanup__(self):
    	  print 'cleaning up',self

    o = DataRecord()

    # Wrap the instance:
    p = Proxy.InstanceProxy(o,('a',))
    # Remove o from the accessible system:
    del o

    print 'Read p.a through Proxy:',p.a

    # This will cause an exception, because the object is read-only
    p.a = 3

    # This will cause an exception, because no access is given to .b
    print p.b

    # Clear all traces of the provious exceptions (they might contain
    # references to p) by issuing another one. Note that not doing
    # so will cause the following 'del p' to not destroy the final
    # reference to p... (don't ask me why).
    1/0

    # Deleting the Proxy will also delete the wrapped object, if there
    # is no other reference to it in the system. It will invoke
    # the __cleanup__ method in that case.
    del p

    # If you want to have the wrapping done automatically, you can use
    # the InstanceProxyFactory:
    DataRecord = Proxy.InstanceProxyFactory(DataRecord,('a',))

    # This gives the same behaviour...
    p = DataRecord()
    print p.a
    p.a = 3
    print p.b

More examples will eventually appear in the Examples subdirectory of the package.

### Package Structure

    [Proxy]
           Doc/
           Examples/
           [mxProxy]
           Proxy.py

Entries enclosed in brackets are packages (i.e. they are directories that include a `__init__.py` file). Ones without brackets are just simple subdirectories that are not accessible via `import`.

The package imports all symbols from the Proxy sub module which in turn imports the extension module, so you only need to '`import Proxy`' to start working.

### Support

eGenix.com is providing commercial support for this package. If you are interested in receiving information about this service please see the eGenix.com Support Conditions.

### Copyright & License

© 1998-2000, Copyright by Marc-André Lemburg; All Rights Reserved. mailto: mal@lemburg.com

© 2000-2001, Copyright by eGenix.com Software GmbH, Langenfeld, Germany; All Rights Reserved. mailto: info@egenix.com

This software is covered by the eGenix.com Public License Agreement. The text of the license is also included as file "LICENSE" in the package's main directory.

By downloading, copying, installing or otherwise using the software, you agree to be bound by the terms and conditions of the eGenix.com Public License Agreement.

### History & Future

Things that still need to be done:

- Get number coercion to work. This may not be possible in general due to deficiencies in Python internal coercion mechanism.

- Provide a callback hook for weak referencing proxies which gets called when the referenced object dies.

- Implement getattr-translators that map __XXX__ style names to type slot calls.

- Make automatic method and function wrapping (and proxy object hiding) optional.

There were no significant changes between 2.0.0 and 2.0.3.

Changes from 0.2.0 to 2.0.0:

- Fixed a bug in the finalization of the weak reference implemenation. Thanks to Michael Muller for pointing me to this quirk. Finalization of the weak ref dict previously occurred in an atexit() function, but at this time, the Python interpreter is already gone, so __del__ hooks would cause a fatal error due to a missing thread state. The implementation now uses explicit function APIs to handle finalization at module cleanup time.

- Removed __weakref reference to the weak ref dict from the module interface.

- Added initweakrefs() and finalizeweakrefs().

- AccessError is now a subclass of exceptions.AttributeError to enhance integration with the getattr() protocol.

- Exceptions raised during .__cleanup__() are only reported in debug or verbose mode (python -v or -d).

- AccessErrors now contain the name of the attribute to which access was denied (in case the name is a string).

- Moved the package under a new top-level package 'mx'. It is part of the eGenix.com mx BASE distribution.

Changes from 0.1.0 to 0.2.0:

- AccessError is a subclass of exceptions.StandardError.

- Fixed a bug where a proxy without interface would not wrap bound and unbound methods.

- Added CachingInstanceProxy.

- Added weak referencing mechanism.

- Added support for nearly all low level type slots.

Version 0.1.0 was the intial release.

© 1999, Copyright by Marc-André Lemburg; All Rights Reserved. mailto: mal@lemburg.com