> # ignore conflicts with other databases, as we need to push out to same suffix
> hidden on
> suffix "dc=suretecsystems,dc=com"
> rootdn "cn=slapd-ldap"
> uri ldap://localhost:9012/
>
> lastmod on
>
> # We don't need any access to this DSA
> restrict all
>
> acl-bind bindmethod=simple
> binddn="cn=replicator,dc=suretecsystems,dc=com"
> credentials=testing
>
> syncrepl rid=001
> provider=ldap://localhost:9011/
> binddn="cn=replicator,dc=suretecsystems,dc=com"
> bindmethod=simple
> credentials=testing
> searchbase="dc=suretecsystems,dc=com"
> type=refreshAndPersist
> retry="5 5 300 5"
>
> overlay syncprov
As you can see, you can let your imagination go wild using Syncrepl and
{{slapd-ldap(8)}} tailoring your replication to fit your specific network
topology.
H2: Pull Based
directory. In previous releases, replication was discussed in terms of
a {{master}} server and some number of {{slave}} servers. A master
accepted directory updates from other clients, and a slave only
accepted updates from a (single) master. The replication structure
was rigidly defined and any particular database could only fulfill
a single role, either master or slave.
As OpenLDAP now supports a wide variety of replication topologies, these
terms have been deprecated in favor of {{provider}} and
{{consumer}}: A provider replicates directory updates to consumers;
consumers receive replication updates from providers. Unlike the
rigidly defined master/slave relationships, provider/consumer roles
are quite fluid: replication updates received in a consumer can be
further propagated by that consumer to other servers, so a consumer
can also act simultaneously as a provider. Also, a consumer need not
be an actual LDAP server; it may be just an LDAP client.
The following sections will describe the replication technology and
discuss the various replication options that are available.
H2: Replication Technology
H3: LDAP Sync Replication
The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
short, is a consumer-side replication engine that enables the
consumer {{TERM:LDAP}} server to maintain a shadow copy of a
{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer-side
as one of the {{slapd}}(8) threads. It creates and maintains a
{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer
and executes as one of the {{slapd}}(8) threads. It creates and maintains a
consumer replica by connecting to the replication provider to perform
the initial DIT content load followed either by periodic content
polling or by timely updates upon content changes.
Syncrepl uses the LDAP Content Synchronization (or LDAP Sync for
short) protocol as the replica synchronization protocol. It provides
Syncrepl uses the LDAP Content Synchronization protocol (or LDAP Sync for
short) as the replica synchronization protocol. LDAP Sync provides
a stateful replication which supports both pull-based and push-based
synchronization and does not mandate the use of a history store.
In pull-based replication the consumer periodically
polls the provider for updates. In push-based replication the consumer
listens for updates that are sent by the provider in realtime. Since the
protocol does not require a history store, the provider does not need to
maintain any log of updates it has received. (Note
that the syncrepl engine is extensible and additional replication
protocols may be supported in the future.)
Syncrepl keeps track of the status of the replication content by
maintaining and exchanging synchronization cookies. Because the
...
...
@@ -331,7 +88,7 @@ privileges for the DIT fragment to be replicated. The consumer
server can stop the replication also without the need for provider-side
changes and restart.
Syncrepl supports both partial and sparse replications. The shadow
Syncrepl supports partial, sparse, and fractional replications. The shadow
DIT fragment is defined by a general search criteria consisting of
base, scope, filter, and attribute list. The replica content is
also subject to the access privileges of the bind identity of the
...
...
@@ -346,56 +103,60 @@ of controls and other protocol elements which extend the LDAP search
operation. This section introduces the LDAP Content Sync protocol
only briefly. For more information, refer to {{REF:RFC4533}}.
The LDAP Sync protocol supports both polling and listening for
changes by defining two respective synchronization operations:
The LDAP Sync protocol supports both polling and listening for changes
by defining two respective synchronization operations:
{{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented
by the {{refreshOnly}} operation. The client copy is synchronized
to the server copy at the time of polling. The server finishes the
by the {{refreshOnly}} operation. The consumer
polls the provider using an LDAP Search request with an LDAP Sync
control attached. The consumer copy is synchronized
to the provider copy at the time of polling using the information
returned in the search. The provider finishes the
search operation by returning {{SearchResultDone}} at the end of
the search operation as in the normal search. The listening is
implemented by the {{refreshAndPersist}} operation. Instead of
the search operation as in the normal search. Listening is
implemented by the {{refreshAndPersist}} operation. As the name
implies, it begins with a search, like refreshOnly. Instead of
finishing the search after returning all entries currently matching
the search criteria, the synchronization search remains persistent
in the server. Subsequent updates to the synchronization content
in the server cause additional entry updates to be sent to the
client.
in the provider. Subsequent updates to the synchronization content
in the provider cause additional entry updates to be sent to the
consumer.
The {{refreshOnly}} operation and the refresh stage of the
{{refreshAndPersist}} operation can be performed with a present
phase or a delete phase.
In the present phase, the server sends the client the entries updated
within the search scope since the last synchronization. The server
sends all requested attributes, be it changed or not, of the updated
In the present phase, the provider sends the consumer the entries updated
within the search scope since the last synchronization. The provider
sends all requested attributes, be they changed or not, of the updated
entries. For each unchanged entry which remains in the scope, the
server sends a present message consisting only of the name of the
provider sends a present message consisting only of the name of the
entry and the synchronization control representing state present.
The present message does not contain any attributes of the entry.
After the client receives all update and present entries, it can
reliably determine the new client copy by adding the entries added
to the server, by replacing the entries modified at the server, and
by deleting entries in the client copy which have not been updated
nor specified as being present at the server.
After the consumer receives all update and present entries, it can
reliably determine the new consumer copy by adding the entries added
to the provider, by replacing the entries modified at the provider, and
by deleting entries in the consumer copy which have not been updated
nor specified as being present at the provider.
The transmission of the updated entries in the delete phase is the
same as in the present phase. The server sends all the requested
same as in the present phase. The provider sends all the requested
attributes of the entries updated within the search scope since the
last synchronization to the client. In the delete phase, however,
the server sends a delete message for each entry deleted from the
last synchronization to the consumer. In the delete phase, however,
the provider sends a delete message for each entry deleted from the
search scope, instead of sending present messages. The delete
message consists only of the name of the entry and the synchronization
control representing state delete. The new client copy can be
control representing state delete. The new consumer copy can be
determined by adding, modifying, and removing entries according to
the synchronization control attached to the {{SearchResultEntry}}
message.
In the case that the LDAP Sync server maintains a history store and
can determine which entries are scoped out of the client copy since
the last synchronization time, the server can use the delete phase.
If the server does not maintain any history store, cannot determine
In the case that the LDAP Sync provider maintains a history store and
can determine which entries are scoped out of the consumer copy since
the last synchronization time, the provider can use the delete phase.
If the provider does not maintain any history store, cannot determine
the scoped-out entries from the history store, or the history store
does not cover the outdated synchronization state of the client,
the server should use the present phase. The use of the present
does not cover the outdated synchronization state of the consumer,
the provider should use the present phase. The use of the present
phase is much more efficient than a full content reload in terms
of the synchronization traffic. To reduce the synchronization
traffic further, the LDAP Sync protocol also provides several
...
...
@@ -403,22 +164,20 @@ optimizations such as the transmission of the normalized {{EX:entryUUID}}s
and the transmission of multiple {{EX:entryUUIDs}} in a single
{{syncIdSet}} message.
At the end of the {{refreshOnly}} synchronization, the server sends
a synchronization cookie to the client as a state indicator of the
client copy after the synchronization is completed. The client
At the end of the {{refreshOnly}} synchronization, the provider sends
a synchronization cookie to the consumer as a state indicator of the
consumer copy after the synchronization is completed. The consumer
will present the received cookie when it requests the next incremental
synchronization to the server.
synchronization to the provider.
When {{refreshAndPersist}} synchronization is used, the server sends
When {{refreshAndPersist}} synchronization is used, the provider sends
a synchronization cookie at the end of the refresh stage by sending
a Sync Info message with TRUE refreshDone. It also sends a
a Sync Info message with refreshDone=TRUE. It also sends a
synchronization cookie by attaching it to {{SearchResultEntry}}
generated in the persist stage of the synchronization search. During
the persist stage, the server can also send a Sync Info message
containing the synchronization cookie at any time the server wants
to update the client-side state indicator. The server also updates
a synchronization indicator of the client at the end of the persist
stage.
messages generated in the persist stage of the synchronization search. During
the persist stage, the provider can also send a Sync Info message
containing the synchronization cookie at any time the provider wants
to update the consumer-side state indicator.
In the LDAP Sync protocol, entries are uniquely identified by the
{{EX:entryUUID}} attribute value. It can function as a reliable
...
...
@@ -428,7 +187,6 @@ reliable identifier. The {{EX:entryUUID}} is attached to each
{{SearchResultEntry}} or {{SearchResultReference}} as a part of the
synchronization control.
H4: Syncrepl Details
The syncrepl engine utilizes both the {{refreshOnly}} and the
...
...
@@ -444,10 +202,10 @@ provider.
The syncrepl engine utilizes both the present phase and the delete
phase of the refresh synchronization. It is possible to configure
a per-scope session log in the provider server which stores the
a session log in the provider which stores the
{{EX:entryUUID}}s of a finite number of entries deleted from a
replication content. Multiple replicas of single provider content
share the same per-scope session log. The syncrepl engine uses the
database. Multiple replicas share the same session log. The syncrepl
engine uses the
delete phase if the session log is present and the state of the
consumer server is recent enough that no session log entries are
truncated after the last synchronization of the client. The syncrepl
...
...
@@ -538,20 +296,33 @@ this change without the use of the session log.
For configuration, please see the {{SECT:Syncrepl}} section.
H2: Deployment Alternatives
While the LDAP Sync specification only defines a narrow scope for replication,
the OpenLDAP implementation is extremely flexible and supports a variety of
operating modes to handle other scenarios not explicitly addressed in the spec.
H3: Delta-syncrepl replication
* Disadvantages of Syncrepl replication:
* Disadvantages of LDAP Sync replication:
OpenLDAP's syncrepl replication is an object-based replication mechanism.
LDAP Sync replication is an object-based replication mechanism.
When any attribute value in a replicated object is changed on the provider,
each consumer fetches and processes the complete changed object {{B:both changed and unchanged attribute values}} during replication. This works well, but has drawbacks in some situations.
each consumer fetches and processes the complete changed object
{{B:both changed and unchanged attribute values}} during replication.
One advantage of this approach is that when multiple changes occur to
a single object, the precise sequence of those changes need not be preserved;
only the final state of the entry is significant. But this approach
may have drawbacks when the usage pattern involves single changes to
multiple objects.
For example, suppose you have a database consisting of 100,000 objects of 1 KB
each. Further, suppose you routinely run a batch job to change the value of
a single two-byte attribute value that appears in each of the 100,000 objects
on the master. Not counting LDAP and TCP/IP protocol overhead, each time you
run this job each consumer will transfer and process {{B:1 GB}} of data to process
{{B:200KB of changes!}}
run this job each consumer will transfer and process {{B:1 GB}} of data to
process {{B:200KB of changes!}}
99.98% of the data that is transmitted and processed in a case like this will
be redundant, since it represents values that did not change. This is a waste
...
...
@@ -564,45 +335,55 @@ demonstrate a very real problem that is encountered in some LDAP deployments.
Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address
situations like the one described above. Delta-syncrepl works by maintaining a
changelog of a selectable depth on the provider. The replication consumer on
each consumer checks the changelog for the changes it needs and, as long as
the changelog contains the needed changes, the delta-syncrepl consumer fetches
them from the changelog and applies them to its database. If, however, a replica
changelog of a selectable depth on the provider. The replication consumer
checks the changelog for the changes it needs and, as long as
the changelog contains the needed changes, the consumer fetches the changes
from the changelog and applies them to its database. If, however, a replica
is too far out of sync (or completely empty), conventional syncrepl is used to
bring it up to date and replication then switches to the delta-syncrepl mode.
bring it up to date and replication then switches back to the delta-syncrepl
mode.
For configuration, please see the {{SECT:Delta-syncrepl}} section.
H2: Mixture of both Pull and Push based
H3: N-Way Multi-Master replication
Multi-Master replication is a replication technique using Syncrepl to replicate
data to multiple Master Directory servers.
data to multiple provider ("Master") Directory servers.
* Advantages of Multi-Master replication:
H4: Valid Arguments for Multi-Master replication
- If any master fails, other masters will continue to accept updates
- Avoids a single point of failure
- Masters can be located in several physical sites i.e. distributed across the
network/globe.
- Good for Automatic failover/High Availability
* If any provider fails, other providers will continue to accept updates
* Avoids a single point of failure
* Providers can be located in several physical sites i.e. distributed across
the network/globe.
* Good for Automatic failover/High Availability
* Disadvantages of Multi-Master replication:
H4: Invalid Arguments for Multi-Master replication
