Thread: python-ldap as replication client

Hello dear list members,

Please allow me to thank you and shortly introduce myself as I'm new to the 
list; My name is Jeroen van Meeuwen, I'm Dutch and I live in the United 
Kingdom, where I work for a Swiss company in Free Software groupware 
solutions, Kolab Systems.

I'm looking to implement LDAP_CONTROL_SYNC(*) capabilities to python-ldap's 
ldap.controls, and while I do have some experience in several areas, 
admittedly compared to you I'm probably the most under-qualified programmer to 
actually do it.

That said, I first wanted to ask whether something like python-ldap becoming a 
replication client (through server controls) was feasible in your opinion(s). 
I think RFC 3928[1] is the corresponding standard. Another standard was 
proposed in RFC 4533[2] but that one bounced in favor of the former.

Thanks in advance,

Kind regards,

Jeroen van Meeuwen

[1] http://tools.ietf.org/html/rfc3928
[2] http://tools.ietf.org/html/rfc4533

-- 
Senior Engineer, Kolab Systems AG

e: van...@ko...
t: +316 42 801 403
w: http://www.kolabsys.com

pgp: 9342 BF08

Jeroen van Meeuwen (Kolab Systems) wrote:
> I'm looking to implement LDAP_CONTROL_SYNC(*) capabilities to
> python-ldap's ldap.controls, and while I do have some experience in
> several areas, admittedly compared to you I'm probably the most
> under-qualified programmer to actually do it.

You're always welcome to send demo code and get it commented here.

> That said, I first wanted to ask whether something like python-ldap
> becoming a replication client (through server controls) was feasible in
> your opinion(s).

No matter which sync protocol you implement it's very likely that you need
python-LDAP from CVS HEAD (will be python 2.4) since this version contains
code to extract response controls from intermediate responses. Beware that
this may still be subject of API changes especially regarding ldap.controls
and ldap.extop.

Some additional ASN.1 work for encoding/decoding controls is needed too. I'm
currently using pyasn1.sf.net for that which is outside python-ldap.

> I think RFC 3928[1] is the corresponding standard.
> Another standard was proposed in RFC 4533[2] but that one bounced in
> favor of the former.

Which sync protocol standard suits your needs depends on the LDAP server your
application is talking to.

If you use the OpenLDAP server the OpenLDAP developers strongly recommend
syncrepl. There were already some people here implementing syncrepl (RFC 4533)
based on python-ldap.

Personally I'm currently using LDAP persistent search retrieving data from a
Novell eDirectory server since this is the control this server supports.

Other LDAP servers have other sync controls, e.g. MS AD implemented the
proprietary DirSync control, etc.

Ciao, Michael.

Michael Ströder wrote:
> Jeroen van Meeuwen (Kolab Systems) wrote:
> > I'm looking to implement LDAP_CONTROL_SYNC(*) capabilities to
> > python-ldap's ldap.controls, and while I do have some experience in
> > several areas, admittedly compared to you I'm probably the most
> > under-qualified programmer to actually do it.
> 
> You're always welcome to send demo code and get it commented here.
> 

Thanks! I feel all warm and fuzzy and welcome ;-)

> > That said, I first wanted to ask whether something like python-ldap
> > becoming a replication client (through server controls) was feasible in
> > your opinion(s).
> 
> No matter which sync protocol you implement it's very likely that you need
> python-LDAP from CVS HEAD (will be python 2.4) since this version contains
> code to extract response controls from intermediate responses. Beware that
> this may still be subject of API changes especially regarding ldap.controls
> and ldap.extop.
> 

Sure, that is fair enough. I was planning on doing any work based on CVS HEAD 
anyways, as I always consider it easier to upgrade (too) late then it is to 
stick with a modified, unsupported version forever ;-)

> Some additional ASN.1 work for encoding/decoding controls is needed too.
> I'm currently using pyasn1.sf.net for that which is outside python-ldap.
> 
> > I think RFC 3928[1] is the corresponding standard.
> > Another standard was proposed in RFC 4533[2] but that one bounced in
> > favor of the former.
> 
> Which sync protocol standard suits your needs depends on the LDAP server
> your application is talking to.
> 
> If you use the OpenLDAP server the OpenLDAP developers strongly recommend
> syncrepl. There were already some people here implementing syncrepl (RFC
> 4533) based on python-ldap.
> 

I'd be very interested in this work. Do you have a reference, perhaps? I can 
find some coversations on the topic, but no code / show-case implementation.

That said, it's interesting people have implemented this based on RFC 4533; 
Would you agree or disagree implementing the superseeded may actually be a bad 
thing to do for python-ldap, or would you wish any such implementation to be 
compatible with 4533 as well, somehow?

> Personally I'm currently using LDAP persistent search retrieving data from
> a Novell eDirectory server since this is the control this server supports.
> 

I'm successfully using the paging search results control[1] (against python-
ldap version 2.3.10) against a (simulated) very large LDAP tree, after which I 
realized this type of iteration does in no way apply any changes to the tree 
as instantly as the daemon being a replication client.

> Other LDAP servers have other sync controls, e.g. MS AD implemented the
> proprietary DirSync control, etc.
> 

I'll stick to the Free and RFC compatible for now, if that's OK with you? ;-)

Kind regards,

Jeroen van Meeuwen

[1] http://git.kolab.org/pykolab/tree/pykolab/auth/ldap/__init__.py#n150

-- 
Senior Engineer, Kolab Systems AG

e: van...@ko...
t: +316 42 801 403
w: http://www.kolabsys.com

pgp: 9342 BF08

On 03/11/2011 05:40 AM, Michael Ströder wrote:
> Jeroen van Meeuwen (Kolab Systems) wrote:
>> I'm looking to implement LDAP_CONTROL_SYNC(*) capabilities to
>> python-ldap's ldap.controls, and while I do have some experience in
>> several areas, admittedly compared to you I'm probably the most
>> under-qualified programmer to actually do it.
> You're always welcome to send demo code and get it commented here.
>
>> That said, I first wanted to ask whether something like python-ldap
>> becoming a replication client (through server controls) was feasible in
>> your opinion(s).
> No matter which sync protocol you implement it's very likely that you need
> python-LDAP from CVS HEAD (will be python 2.4) since this version contains
> code to extract response controls from intermediate responses. Beware that
> this may still be subject of API changes especially regarding ldap.controls
> and ldap.extop.
>
> Some additional ASN.1 work for encoding/decoding controls is needed too. I'm
> currently using pyasn1.sf.net for that which is outside python-ldap.
>
>> I think RFC 3928[1] is the corresponding standard.
>> Another standard was proposed in RFC 4533[2] but that one bounced in
>> favor of the former.
> Which sync protocol standard suits your needs depends on the LDAP server your
> application is talking to.
>
> If you use the OpenLDAP server the OpenLDAP developers strongly recommend
> syncrepl. There were already some people here implementing syncrepl (RFC 4533)
> based on python-ldap.

I'm currently working on a project that requires me to do a syncrepl 
from python and after much, much reading I'm afraid that the python-ldap 
library does not implement 4533 correctly.

Sync cookies are only retrieved by python-ldap if they are returned in a 
server control, however this is only the case in an 
LDAP_RES_SEARCH_RESULT or an LDAP_RES_SEARCH_ENTRY packets.  The 
protocol passes both deletes and presence records in 
LDAP_RES_INTERMEDIATE packets, which don't get returned to the python 
caller as they don't have LDAP entries in them, and cookies are also 
returned in these intermediate result packets, but not in a server 
control, so those are missed.

To see the proper handling of the syncrepl protocol it's instructional 
to read through the do_syncrep2 found in the file 
servers/slapd/syncrepl.c of the openldap source.  I'm working on moving 
that code over to a new function in the python-ldap module, but I'm not 
sure whether my company is going to allow me to release the code back to 
the project.  If I do it, they probably will.  If we pay someone else to 
do it, possibly not.

In any case, a python-LDAP syncrepl client can be made to work, but if 
there are deletes during a period when the client is not connected I 
believe they will be lost during the catchup phase of the sync.

I'm certainly not an OpenLDAP nor a python-LDAP expert, so if I'm 
mistaken about anything I've said above, please feel free to set me 
straight.  I just thought it would be good to share the caveat as I 
understand it.

Sincerely,
e.

> Personally I'm currently using LDAP persistent search retrieving data from a
> Novell eDirectory server since this is the control this server supports.
>
> Other LDAP servers have other sync controls, e.g. MS AD implemented the
> proprietary DirSync control, etc.
>
> Ciao, Michael.
>
> ------------------------------------------------------------------------------
> Colocation vs. Managed Hosting
> A question and answer guide to determining the best fit
> for your organization - today and in the future.
> http://p.sf.net/sfu/internap-sfd2d
> _______________________________________________
> Python-LDAP-dev mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-ldap-dev

Eric Brunson wrote:
> On 03/11/2011 05:40 AM, Michael Ströder wrote:
>> No matter which sync protocol you implement it's very likely that you need
>> python-LDAP from CVS HEAD (will be python 2.4) since this version contains
>> code to extract response controls from intermediate responses.
> 
> I'm currently working on a project that requires me to do a syncrepl 
> from python and after much, much reading I'm afraid that the python-ldap 
> library does not implement 4533 correctly.
> 
> Sync cookies are only retrieved by python-ldap if they are returned in a 
> server control, however this is only the case in an 
> LDAP_RES_SEARCH_RESULT or an LDAP_RES_SEARCH_ENTRY packets.  The 
> protocol passes both deletes and presence records in 
> LDAP_RES_INTERMEDIATE packets, which don't get returned to the python 
> caller as they don't have LDAP entries in them, and cookies are also 
> returned in these intermediate result packets, but not in a server 
> control, so those are missed.

The patches in CVS HEAD were contributed by Rich exactly to make syncrepl
possible with python-ldap. If you think the current implementation in CVS HEAD
still has deficiencies regarding controls in intermediate responses I happily
will review a patch. ;-)

Ciao, Michael.

On 03/11/2011 11:00 AM, Michael Ströder wrote:
> Eric Brunson wrote:
>> On 03/11/2011 05:40 AM, Michael Ströder wrote:
>>> No matter which sync protocol you implement it's very likely that you need
>>> python-LDAP from CVS HEAD (will be python 2.4) since this version contains
>>> code to extract response controls from intermediate responses.
>> I'm currently working on a project that requires me to do a syncrepl
>> from python and after much, much reading I'm afraid that the python-ldap
>> library does not implement 4533 correctly.
>>
>> Sync cookies are only retrieved by python-ldap if they are returned in a
>> server control, however this is only the case in an
>> LDAP_RES_SEARCH_RESULT or an LDAP_RES_SEARCH_ENTRY packets.  The
>> protocol passes both deletes and presence records in
>> LDAP_RES_INTERMEDIATE packets, which don't get returned to the python
>> caller as they don't have LDAP entries in them, and cookies are also
>> returned in these intermediate result packets, but not in a server
>> control, so those are missed.
> The patches in CVS HEAD were contributed by Rich exactly to make syncrepl
> possible with python-ldap. If you think the current implementation in CVS HEAD
> still has deficiencies regarding controls in intermediate responses I happily
> will review a patch. ;-)

Wow, Michael, that is super awesome news.  I'll check the CVS head, try 
it out and get back to you.

Sincerely,
e.

Eric Brunson wrote:
> On 03/11/2011 11:00 AM, Michael Ströder wrote:
>> Eric Brunson wrote:
>>> On 03/11/2011 05:40 AM, Michael Ströder wrote:
>>>> No matter which sync protocol you implement it's very likely that
>>>> you need
>>>> python-LDAP from CVS HEAD (will be python 2.4) since this version
>>>> contains
>>>> code to extract response controls from intermediate responses.
>>> I'm currently working on a project that requires me to do a syncrepl
>>> from python and after much, much reading I'm afraid that the python-ldap
>>> library does not implement 4533 correctly.
>>>
>>> Sync cookies are only retrieved by python-ldap if they are returned in a
>>> server control, however this is only the case in an
>>> LDAP_RES_SEARCH_RESULT or an LDAP_RES_SEARCH_ENTRY packets.  The
>>> protocol passes both deletes and presence records in
>>> LDAP_RES_INTERMEDIATE packets, which don't get returned to the python
>>> caller as they don't have LDAP entries in them, and cookies are also
>>> returned in these intermediate result packets, but not in a server
>>> control, so those are missed.
>> The patches in CVS HEAD were contributed by Rich exactly to make syncrepl
>> possible with python-ldap. If you think the current implementation in
>> CVS HEAD
>> still has deficiencies regarding controls in intermediate responses I
>> happily
>> will review a patch. ;-)
> 
> Wow, Michael, that is super awesome news.  I'll check the CVS head, try
> it out and get back to you.

Make sure to set the right arguments for LDAPObject.result4().

Ciao, Michael.

On 03/11/2011 11:24 AM, Michael Ströder wrote:
> Eric Brunson wrote:
>> On 03/11/2011 11:00 AM, Michael Ströder wrote:
>>> Eric Brunson wrote:
>>>> On 03/11/2011 05:40 AM, Michael Ströder wrote:
>>>>> No matter which sync protocol you implement it's very likely that
>>>>> you need
>>>>> python-LDAP from CVS HEAD (will be python 2.4) since this version
>>>>> contains
>>>>> code to extract response controls from intermediate responses.
>>>> I'm currently working on a project that requires me to do a syncrepl
>>>> from python and after much, much reading I'm afraid that the python-ldap
>>>> library does not implement 4533 correctly.
>>>>
>>>> Sync cookies are only retrieved by python-ldap if they are returned in a
>>>> server control, however this is only the case in an
>>>> LDAP_RES_SEARCH_RESULT or an LDAP_RES_SEARCH_ENTRY packets.  The
>>>> protocol passes both deletes and presence records in
>>>> LDAP_RES_INTERMEDIATE packets, which don't get returned to the python
>>>> caller as they don't have LDAP entries in them, and cookies are also
>>>> returned in these intermediate result packets, but not in a server
>>>> control, so those are missed.
>>> The patches in CVS HEAD were contributed by Rich exactly to make syncrepl
>>> possible with python-ldap. If you think the current implementation in
>>> CVS HEAD
>>> still has deficiencies regarding controls in intermediate responses I
>>> happily
>>> will review a patch. ;-)
>> Wow, Michael, that is super awesome news.  I'll check the CVS head, try
>> it out and get back to you.
> Make sure to set the right arguments for LDAPObject.result4().
>
> Ciao, Michael.

Michael and all,

The new code works great, thanks so much for the new features.

I do have one issue, and maybe I'm just not looking in the correct 
place.  The Sync Info Message returns a syncInfoValue which is a BER 
encoded ASN.1 CHOICE structure:

    syncInfoValue ::= CHOICE {
               newcookie      [0] syncCookie,
               refreshDelete  [1] SEQUENCE {
                   cookie         syncCookie OPTIONAL,
                   refreshDone    BOOLEAN DEFAULT TRUE
               },
               refreshPresent [2] SEQUENCE {
                   cookie         syncCookie OPTIONAL,
                   refreshDone    BOOLEAN DEFAULT TRUE
               },
               syncIdSet      [3] SEQUENCE {
                   cookie         syncCookie OPTIONAL,
                   refreshDeletes BOOLEAN DEFAULT FALSE,
                   syncUUIDs      SET OF syncUUID
               }
           }

The data is returned and I've been able to successfully decode it with 
the PyASN1 BER codec, but I can't find any indication of the choice 
index being returned in the value.   I don't know that the refreshDelete 
and the refreshPresent are distinguishable from each other without 
additional information, but I see that the value being returned from 
result4() is simply what ldap_parse_intermediate() returns, without any 
indication of the choice index.  Looking at the raw BER encoded packet 
in wireshark, it would seem that the two bytes before the data being 
returned have the index embedded in the second byte.

I'm sure this must simply be something I'm overlooking.  Any help?

Thanks,
e.

Eric Brunson wrote:
> The new code works great, thanks so much for the new features.
> 
> I do have one issue, and maybe I'm just not looking in the correct
> place.  The Sync Info Message returns a syncInfoValue which is a BER
> encoded ASN.1 CHOICE structure:
> 
>     syncInfoValue ::= CHOICE {
>               newcookie      [0] syncCookie,
>               refreshDelete  [1] SEQUENCE {
>                   cookie         syncCookie OPTIONAL,
>                   refreshDone    BOOLEAN DEFAULT TRUE
>               },
>               refreshPresent [2] SEQUENCE {
>                   cookie         syncCookie OPTIONAL,
>                   refreshDone    BOOLEAN DEFAULT TRUE
>               },
>               syncIdSet      [3] SEQUENCE {
>                   cookie         syncCookie OPTIONAL,
>                   refreshDeletes BOOLEAN DEFAULT FALSE,
>                   syncUUIDs      SET OF syncUUID
>               }
>           }
> 
> The data is returned and I've been able to successfully decode it with
> the PyASN1 BER codec, but I can't find any indication of the choice
> index being returned in the value.   I don't know that the refreshDelete
> and the refreshPresent are distinguishable from each other without
> additional information, but I see that the value being returned from
> result4() is simply what ldap_parse_intermediate() returns, without any
> indication of the choice index.  Looking at the raw BER encoded packet
> in wireshark, it would seem that the two bytes before the data being
> returned have the index embedded in the second byte.
> 
> I'm sure this must simply be something I'm overlooking.  Any help?

Could you share a short script demonstrating this?

Ciao, Michael.

On 03/23/2011 01:11 AM, Michael Ströder wrote:
> Eric Brunson wrote:
>> The new code works great, thanks so much for the new features.
>>
>> I do have one issue, and maybe I'm just not looking in the correct
>> place.  The Sync Info Message returns a syncInfoValue which is a BER
>> encoded ASN.1 CHOICE structure:
>>
>>      syncInfoValue ::= CHOICE {
>>                newcookie      [0] syncCookie,
>>                refreshDelete  [1] SEQUENCE {
>>                    cookie         syncCookie OPTIONAL,
>>                    refreshDone    BOOLEAN DEFAULT TRUE
>>                },
>>                refreshPresent [2] SEQUENCE {
>>                    cookie         syncCookie OPTIONAL,
>>                    refreshDone    BOOLEAN DEFAULT TRUE
>>                },
>>                syncIdSet      [3] SEQUENCE {
>>                    cookie         syncCookie OPTIONAL,
>>                    refreshDeletes BOOLEAN DEFAULT FALSE,
>>                    syncUUIDs      SET OF syncUUID
>>                }
>>            }
>>
>> The data is returned and I've been able to successfully decode it with
>> the PyASN1 BER codec, but I can't find any indication of the choice
>> index being returned in the value.   I don't know that the refreshDelete
>> and the refreshPresent are distinguishable from each other without
>> additional information, but I see that the value being returned from
>> result4() is simply what ldap_parse_intermediate() returns, without any
>> indication of the choice index.  Looking at the raw BER encoded packet
>> in wireshark, it would seem that the two bytes before the data being
>> returned have the index embedded in the second byte.
>>
>> I'm sure this must simply be something I'm overlooking.  Any help?
> Could you share a short script demonstrating this?

I've done some more reading and I think I'm mistaken about there being 
some sort of index indicating the type of choice that was encoded.  The 
documentation for the PyASN library implies that the decoder has to 
infer the choice based on the structure of the data, which seems odd.

Thanks for the reply, I'll get back to you when I figure something out.

e.