Can't use non-default directory service

[tylerkeating@…]

Hi, After a recent update, I'm now getting the following error when trying to run the server.

Traceback (most recent call last):

File "/Users/admin/Developer/Collaboration/Twisted/bin/twistd", line 21, in <module>
run()

File "/Users/admin/Developer/Collaboration/Twisted/twisted/scripts/twistd.py", line 27, in run
app.run(runApp, ServerOptions)

File "/Users/admin/Developer/Collaboration/Twisted/twisted/application/app.py", line 374, in run
config.parseOptions()

File "/Users/admin/Developer/Collaboration/Twisted/twisted/application/app.py", line 354, in parseOptions
usage.Options.parseOptions(self, options)

File "/Users/admin/Developer/Collaboration/Twisted/twisted/python/usage.py", line 189, in parseOptions
self.subOptions.parseOptions(rest)

File "/Users/admin/Developer/Collaboration/Twisted/twisted/python/usage.py", line 199, in parseOptions
self.postOptions()

File "/Users/admin/Developer/Collaboration/CalendarServer/twistedcaldav/tap.py", line 127, in postOptions
parseConfig(self['config'])

File "/Users/admin/Developer/Collaboration/CalendarServer/twistedcaldav/config.py", line 342, in parseConfig
config.loadConfig(configFile)

File "/Users/admin/Developer/Collaboration/CalendarServer/twistedcaldav/config.py", line 242, in loadConfig
self.update(configDict)

File "/Users/admin/Developer/Collaboration/CalendarServer/twistedcaldav/config.py", line 186, in update
self._data["DirectoryService"]["params"] = copy.deepcopy(serviceDefaultParams[dsType])

KeyError: 'twistedcaldav.directory.myapp.MyAppDirectoryService'

Where 'twistedcaldav.directory.myapp.MyAppDirectoryService' is my custom directory service, which obviously isn't in the list of serviceDefaultParams. Since serviceDefaultParams only includes XML and Open directory services, you get the same error trying to use the Basic, Digest or SQL directory services too. There is something wrong with how the update method is parsing directory services.

- Tyler

[wsanchez@…]

Whoops, that's no good.

[wsanchez@…]

Fixed in r1493.

[wsanchez@…]

Note ticket #138 which requests that we make this extensible.

[tylerkeating@…]

There is still a bug in the code and using anything other than the XML or OpenDirectory directory service will result in a crash. To simulate it, you only need to uncomment the default Apache-style directory service included in caldavd-text.plist.

Just to get it running on my end, I've moved the offending clauses under the new conditional checking for a default directory service and attached the file (see previous attachment). Please have a look at it, it's a minor change.

[callan@…]

Having written a (hackish) solution to this for my own uses and come across dreid in IRC I've cleaned it up and attach a patch to allow this to be extensible. The basic premise is to load the class using the Twisted reflection utilities, look for a defaultParams class variable and inject it into the defaultServiceParams module variable in twistedcaldav.config. It also provides sanity checks for class loading and altogether missing default configuration parameters. For example:

class MyDirectoryService(DirectoryService):
    defaultParams = {
        'constructorVariable1': 'James Bond 007'
        'constructorVariable2': 'Ian Fleming'
    }
    ...

If preferable I can also do it using a class method:

class MyDirectoryService(DirectoryService):
    @classmethod
    def getDefaultParams(klass):
        return {
            'constructorVariable1': 'James Bond 007'
            'constructorVariable2': 'Ian Fleming'
        }
    ...

Comments? This should also resolve #138.

-Chris (kagon)

[callan@…]

Patch to allow directory service implementations to declare their default configuration options.

[callan@…]

After further discussions with dreid the band-aid style patch idea was thrown out and a new slightly more ambitious idea was put forward. This new patch should ensure all existing DirectoryService instances follow the extensibility paradigm discussed above and were realized by the initial patch. The new patch includes the following changes:

• Removal of serviceDefaultParams altogether
• Changes to twistedcaldav.config.Config.update() to expect all default configuration options to be loaded at runtime
• Addition of defaultParams to the IDirectoryService interface
• Extension of twistedcaldav.test.test_config to encompass the new logic
• Changes to twistedcaldav.test.test_config to allow a little more modularity and code reuse

[callan@…]

More ambitious patch extending the extensible configuration logic throught

[callan@…]

With the patch applied all the tests pass apart from those I cannot run due to being on Linux:

callan@shadowblade ~/CalendarServer $ trial twistedcaldav.test.test_config
Running 22 tests.
twistedcaldav.test.test_config
  ConfigTests
    testCopiesDefaults ...                                                 [OK]
    testDefaults ...                                                       [OK]
    testDirectoryService_badParam ...                                      [OK]
    testDirectoryService_newBadParam ...                                   [OK]
    testDirectoryService_newParam ...                                      [OK]
    testDirectoryService_newParamOD ...                                 [ERROR]
    testDirectoryService_newType ...                                       [OK]
    testDirectoryService_newTypeOD ...                                  [ERROR]
    testDirectoryService_noChange ...                                      [OK]
    testDirectoryService_sameType ...                                      [OK]
    testDirectoryService_twoNewParams ...                                  [OK]
    testDirectoryService_unknownType ...                                   [OK]
    testLoadConfig ...                                                     [OK]
    testMerge ...                                                          [OK]
    testMergeDefaults ...                                                  [OK]
    testReloading ...                                                      [OK]
    testScoping ...                                                        [OK]
    testSetAttr ...                                                        [OK]
    testSetDefaults ...                                                    [OK]
    testUpdateAndReload ...                                                [OK]
    testUpdateDefaults ...                                                 [OK]
    testUpdating ...                                                       [OK]

===============================================================================
[ERROR]: twistedcaldav.test.test_config.ConfigTests.testDirectoryService_newParamOD

Traceback (most recent call last):
  File "/home/callan/CalendarServer/twistedcaldav/test/test_config.py", line 249, in testDirectoryService_newParamOD
    config.update({"DirectoryService": {"type": OPEN_DIRECTORY_SERVICE}})
  File "/home/callan/CalendarServer/twistedcaldav/config.py", line 204, in update
    dsClass = _loadDirectoryService(dsType)
  File "/home/callan/CalendarServer/twistedcaldav/config.py", line 35, in _loadDirectoryService
    raise ConfigurationError(
twistedcaldav.config.ConfigurationError: Unable to load directory service twistedcaldav.directory.appleopendirectory.OpenDirectoryService
===============================================================================
[ERROR]: twistedcaldav.test.test_config.ConfigTests.testDirectoryService_newTypeOD

Traceback (most recent call last):
  File "/home/callan/CalendarServer/twistedcaldav/test/test_config.py", line 244, in testDirectoryService_newTypeOD
    config.update({"DirectoryService": {"type": OPEN_DIRECTORY_SERVICE}})
  File "/home/callan/CalendarServer/twistedcaldav/config.py", line 204, in update
    dsClass = _loadDirectoryService(dsType)
  File "/home/callan/CalendarServer/twistedcaldav/config.py", line 35, in _loadDirectoryService
    raise ConfigurationError(
twistedcaldav.config.ConfigurationError: Unable to load directory service twistedcaldav.directory.appleopendirectory.OpenDirectoryService
-------------------------------------------------------------------------------
Ran 22 tests in 0.105s

FAILED (errors=2, successes=20)

[callan@…]

After being reminded via e-mail by several people of the existence of other DirectoryService instances such as the ApacheDirectoryService I've made sure each of these has defaultParams and a test to ensure that it actually loads correctly.

[dreid@…]

config.py:33 You no longer have the possibility of an AttributeError

What if the values of one of my parameters is a dictionary, do we want to be able to merge the defaults with what is specified in the config file? Does this work already?

[callan@…]

config.py:33 You no longer have the possibility of an AttributeError

Untrue unfortunately, and I've already been bitten by a misspelling that raised one. For example:

callan@shadowblade ~ $ python
Python 2.5.1 (r251:54863, Nov  1 2007, 13:11:14) 
[GCC 4.1.2 (Gentoo 4.1.2 p1.0.2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from twisted.python.reflect import namedClass
>>> class Test(object):
...     pass
... 
>>> namedClass("__main__.Testz")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.5/site-packages/twisted/python/reflect.py", line 344, in namedObject
    return getattr(module, classSplit[-1])
AttributeError: 'module' object has no attribute 'Testz'
>>> namedClass("__main__.Test")
<class '__main__.Test'>

What if the values of one of my parameters is a dictionary, do we want to be able to merge the defaults with what is specified in the config file? Does this work already?

This should already work as a result of the recursive nature of the _mergeData() function. I'll try and whip up a test case.

[callan@…]

Updated patch with test cases covering the possibility of dictionary based parameters

[callan@…]

Further dictionary test fixes