replication.sdf 48.7 KB
Newer Older
Kurt Zeilenga's avatar
Kurt Zeilenga committed
1
# $OpenLDAP$
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
2
# Copyright 1999-2020 The OpenLDAP Foundation, All Rights Reserved.
Kurt Zeilenga's avatar
Kurt Zeilenga committed
3
# COPYING RESTRICTIONS APPLY, see COPYRIGHT.
4

5
H1: Replication
6

7
8
9
Replicated directories are a fundamental requirement for delivering a 
resilient enterprise deployment.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
10
{{PRD:OpenLDAP}} has various configuration options for creating a replicated 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
directory. In previous releases, replication was discussed in terms of
a {{master}} server and some number of {{slave}} servers. A master
accepted directory updates from other clients, and a slave only
accepted updates from a (single) master. The replication structure
was rigidly defined and any particular database could only fulfill
a single role, either master or slave.

As OpenLDAP now supports a wide variety of replication topologies, these
terms have been deprecated in favor of {{provider}} and
{{consumer}}: A provider replicates directory updates to consumers; 
consumers receive replication updates from providers. Unlike the
rigidly defined master/slave relationships, provider/consumer roles
are quite fluid: replication updates received in a consumer can be
further propagated by that consumer to other servers, so a consumer
can also act simultaneously as a provider. Also, a consumer need not
be an actual LDAP server; it may be just an LDAP client.

The following sections will describe the replication technology and
discuss the various replication options that are available.

H2: Replication Technology
32

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
33
H3: LDAP Sync Replication
34
35
36
37

The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for
short, is a consumer-side replication engine that enables the
consumer {{TERM:LDAP}} server to maintain a shadow copy of a
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
38
39
{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer
and executes as one of the {{slapd}}(8) threads. It creates and maintains a
40
replica by connecting to the replication provider to perform
41
42
43
the initial DIT content load followed either by periodic content
polling or by timely updates upon content changes.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
44
Syncrepl uses the LDAP Content Synchronization protocol (or LDAP Sync for
45
short) as the consumer synchronization protocol.  LDAP Sync provides
46
47
a stateful replication which supports both pull-based and push-based
synchronization and does not mandate the use of a history store.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
48
49
50
51
In pull-based replication the consumer periodically
polls the provider for updates. In push-based replication the consumer
listens for updates that are sent by the provider in realtime. Since the
protocol does not require a history store, the provider does not need to
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
52
maintain any log of updates it has received (Note
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
53
that the syncrepl engine is extensible and additional replication
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
54
protocols may be supported in the future.).
55
56
57
58
59
60

Syncrepl keeps track of the status of the replication content by
maintaining and exchanging synchronization cookies. Because the
syncrepl consumer and provider maintain their content status, the
consumer can poll the provider content to perform incremental
synchronization by asking for the entries required to make the
61
62
63
consumer up-to-date with the provider content. Syncrepl
also enables convenient management of consumers by maintaining replication
status.  The consumer database can be constructed from a consumer-side
64
or a provider-side backup at any synchronization status. Syncrepl
65
can automatically resynchronize the consumer database to be up-to-date
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
with the current provider content.

Syncrepl supports both pull-based and push-based synchronization.
In its basic refreshOnly synchronization mode, the provider uses
pull-based synchronization where the consumer servers need not be
tracked and no history information is maintained.  The information
required for the provider to process periodic polling requests is
contained in the synchronization cookie of the request itself.  To
optimize the pull-based synchronization, syncrepl utilizes the
present phase of the LDAP Sync protocol as well as its delete phase,
instead of falling back on frequent full reloads. To further optimize
the pull-based synchronization, the provider can maintain a per-scope
session log as a history store. In its refreshAndPersist mode of
synchronization, the provider uses a push-based synchronization.
The provider keeps track of the consumer servers that have requested
a persistent search and sends them necessary updates as the provider
replication content gets modified.

84
With syncrepl, a consumer can create a replication agreement without
85
86
87
88
89
90
changing the provider's configurations and without restarting the
provider server, if the consumer server has appropriate access
privileges for the DIT fragment to be replicated. The consumer
server can stop the replication also without the need for provider-side
changes and restart.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
91
Syncrepl supports partial, sparse, and fractional replications.  The shadow
92
DIT fragment is defined by a general search criteria consisting of
93
base, scope, filter, and attribute list.  The consumer content is
94
95
96
97
also subject to the access privileges of the bind identity of the
syncrepl replication connection.


Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
98
H4: The LDAP Content Synchronization Protocol
99
100
101
102
103
104
105

The LDAP Sync protocol allows a client to maintain a synchronized
copy of a DIT fragment. The LDAP Sync operation is defined as a set
of controls and other protocol elements which extend the LDAP search
operation. This section introduces the LDAP Content Sync protocol
only briefly.  For more information, refer to {{REF:RFC4533}}.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
106
107
The LDAP Sync protocol supports both polling and listening for changes
by defining two respective synchronization operations:
108
{{refreshOnly}} and {{refreshAndPersist}}.  Polling is implemented
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
109
110
111
112
113
by the {{refreshOnly}} operation. The consumer
polls the provider using an LDAP Search request with an LDAP Sync
control attached. The consumer copy is synchronized
to the provider copy at the time of polling using the information
returned in the search.  The provider finishes the
114
search operation by returning {{SearchResultDone}} at the end of
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
115
116
117
the search operation as in the normal search.  Listening is
implemented by the {{refreshAndPersist}} operation. As the name
implies, it begins with a search, like refreshOnly. Instead of
118
119
finishing the search after returning all entries currently matching
the search criteria, the synchronization search remains persistent
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
120
121
122
in the provider. Subsequent updates to the synchronization content
in the provider cause additional entry updates to be sent to the
consumer.
123
124
125
126
127

The {{refreshOnly}} operation and the refresh stage of the
{{refreshAndPersist}} operation can be performed with a present
phase or a delete phase.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
128
129
130
In the present phase, the provider sends the consumer the entries updated
within the search scope since the last synchronization. The provider
sends all requested attributes, be they changed or not, of the updated
131
entries.  For each unchanged entry which remains in the scope, the
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
132
provider sends a present message consisting only of the name of the
133
134
entry and the synchronization control representing state present.
The present message does not contain any attributes of the entry.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
135
136
137
138
139
After the consumer receives all update and present entries, it can
reliably determine the new consumer copy by adding the entries added
to the provider, by replacing the entries modified at the provider, and
by deleting entries in the consumer copy which have not been updated
nor specified as being present at the provider.
140
141

The transmission of the updated entries in the delete phase is the
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
142
same as in the present phase. The provider sends all the requested
143
attributes of the entries updated within the search scope since the
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
144
145
last synchronization to the consumer. In the delete phase, however,
the provider sends a delete message for each entry deleted from the
146
147
search scope, instead of sending present messages.  The delete
message consists only of the name of the entry and the synchronization
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
148
control representing state delete.  The new consumer copy can be
149
150
151
152
determined by adding, modifying, and removing entries according to
the synchronization control attached to the {{SearchResultEntry}}
message.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
153
154
155
156
In the case that the LDAP Sync provider maintains a history store and
can determine which entries are scoped out of the consumer copy since
the last synchronization time, the provider can use the delete phase.
If the provider does not maintain any history store, cannot determine
157
the scoped-out entries from the history store, or the history store
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
158
159
does not cover the outdated synchronization state of the consumer,
the provider should use the present phase.  The use of the present
160
161
162
163
164
165
166
phase is much more efficient than a full content reload in terms
of the synchronization traffic.  To reduce the synchronization
traffic further, the LDAP Sync protocol also provides several
optimizations such as the transmission of the normalized {{EX:entryUUID}}s
and the transmission of multiple {{EX:entryUUIDs}} in a single
{{syncIdSet}} message.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
167
168
169
At the end of the {{refreshOnly}} synchronization, the provider sends
a synchronization cookie to the consumer as a state indicator of the
consumer copy after the synchronization is completed.  The consumer
170
will present the received cookie when it requests the next incremental
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
171
synchronization to the provider.
172

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
173
When {{refreshAndPersist}} synchronization is used, the provider sends
174
a synchronization cookie at the end of the refresh stage by sending
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
175
a Sync Info message with refreshDone=TRUE.  It also sends a
176
synchronization cookie by attaching it to {{SearchResultEntry}}
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
177
178
179
180
messages generated in the persist stage of the synchronization search. During
the persist stage, the provider can also send a Sync Info message
containing the synchronization cookie at any time the provider wants
to update the consumer-side state indicator.
181
182
183
184
185
186
187
188
189

In the LDAP Sync protocol, entries are uniquely identified by the
{{EX:entryUUID}} attribute value. It can function as a reliable
identifier of the entry. The DN of the entry, on the other hand,
can be changed over time and hence cannot be considered as the
reliable identifier.  The {{EX:entryUUID}} is attached to each
{{SearchResultEntry}} or {{SearchResultReference}} as a part of the
synchronization control.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
190
H4: Syncrepl Details
191
192
193
194
195
196
197
198
199
200
201
202
203
204

The syncrepl engine utilizes both the {{refreshOnly}} and the
{{refreshAndPersist}} operations of the LDAP Sync protocol.  If a
syncrepl specification is included in a database definition,
{{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread
and schedules its execution. If the {{refreshOnly}} operation is
specified, the syncrepl engine will be rescheduled at the interval
time after a synchronization operation is completed.  If the
{{refreshAndPersist}} operation is specified, the engine will remain
active and process the persistent synchronization messages from the
provider.

The syncrepl engine utilizes both the present phase and the delete
phase of the refresh synchronization. It is possible to configure
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
205
a session log in the provider which stores the
206
{{EX:entryUUID}}s of a finite number of entries deleted from a
207
208
209
database. Multiple consumers share the same session log. The syncrepl
engine uses the delete phase if the session log is present and the state
of the consumer server is recent enough that no session log entries are
210
211
truncated after the last synchronization of the client.  The syncrepl
engine uses the present phase if no session log is configured for
212
the replication content or if the consumer is too outdated
213
214
215
216
217
218
219
220
221
222
223
224
225
226
to be covered by the session log.  The current design of the session
log store is memory based, so the information contained in the
session log is not persistent over multiple provider invocations.
It is not currently supported to access the session log store by
using LDAP operations. It is also not currently supported to impose
access control to the session log.

As a further optimization, even in the case the synchronization
search is not associated with any session log, no entries will be
transmitted to the consumer server when there has been no update
in the replication context.

The syncrepl engine, which is a consumer-side replication engine,
can work with any backends. The LDAP Sync provider can be configured
227
as an overlay on any backend, but works best with the {{back-bdb}},
228
{{back-hdb}}, or {{back-mdb}} backends.
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246

The LDAP Sync provider maintains a {{EX:contextCSN}} for each
database as the current synchronization state indicator of the
provider content.  It is the largest {{EX:entryCSN}} in the provider
context such that no transactions for an entry having smaller
{{EX:entryCSN}} value remains outstanding.  The {{EX:contextCSN}}
could not just be set to the largest issued {{EX:entryCSN}} because
{{EX:entryCSN}} is obtained before a transaction starts and
transactions are not committed in the issue order.

The provider stores the {{EX:contextCSN}} of a context in the
{{EX:contextCSN}} attribute of the context suffix entry. The attribute
is not written to the database after every update operation though;
instead it is maintained primarily in memory. At database start
time the provider reads the last saved {{EX:contextCSN}} into memory
and uses the in-memory copy exclusively thereafter. By default,
changes to the {{EX:contextCSN}} as a result of database updates
will not be written to the database until the server is cleanly
247
shut down. A checkpoint facility exists to cause the {{EX:contextCSN}} to
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
be written out more frequently if desired.

Note that at startup time, if the provider is unable to read a
{{EX:contextCSN}} from the suffix entry, it will scan the entire
database to determine the value, and this scan may take quite a
long time on a large database. When a {{EX:contextCSN}} value is
read, the database will still be scanned for any {{EX:entryCSN}}
values greater than it, to make sure the {{EX:contextCSN}} value
truly reflects the greatest committed {{EX:entryCSN}} in the database.
On databases which support inequality indexing, setting an eq index
on the {{EX:entryCSN}} attribute and configuring {{contextCSN}}
checkpoints will greatly speed up this scanning step.

If no {{EX:contextCSN}} can be determined by reading and scanning
the database, a new value will be generated. Also, if scanning the
database yielded a greater {{EX:entryCSN}} than was previously
recorded in the suffix entry's {{EX:contextCSN}} attribute, a
checkpoint will be immediately written with the new value.

267
The consumer also stores its replication state, which is the provider's
268
{{EX:contextCSN}} received as a synchronization cookie, in the
269
{{EX:contextCSN}} attribute of the suffix entry.  The replication state
270
271
272
273
274
275
276
277
278
279
280
281
282
maintained by a consumer server is used as the synchronization state
indicator when it performs subsequent incremental synchronization
with the provider server. It is also used as a provider-side
synchronization state indicator when it functions as a secondary
provider server in a cascading replication configuration.  Since
the consumer and provider state information are maintained in the
same location within their respective databases, any consumer can
be promoted to a provider (and vice versa) without any special
actions.

Because a general search filter can be used in the syncrepl
specification, some entries in the context may be omitted from the
synchronization content.  The syncrepl engine creates a glue entry
283
284
to fill in the holes in the consumer context if any part of the
consumer content is subordinate to the holes. The glue entries will
285
286
287
288
289
290
291
292
not be returned in the search result unless {{ManageDsaIT}} control
is provided.

Also as a consequence of the search filter used in the syncrepl
specification, it is possible for a modification to remove an entry
from the replication scope even though the entry has not been deleted
on the provider. Logically the entry must be deleted on the consumer
but in {{refreshOnly}} mode the provider cannot detect and propagate
293
this change without the use of the session log on the provider.
294

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
295
296
297
For configuration, please see the {{SECT:Syncrepl}} section.


Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
298
299
300
301
302
303
304
H2: Deployment Alternatives

While the LDAP Sync specification only defines a narrow scope for replication,
the OpenLDAP implementation is extremely flexible and supports a variety of
operating modes to handle other scenarios not explicitly addressed in the spec.


Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
305
306
H3: Delta-syncrepl replication

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
307
* Disadvantages of LDAP Sync replication:
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
308

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
309
LDAP Sync replication is an object-based replication mechanism. 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
310
When any attribute value in a replicated object is changed on the provider, 
311
312
each consumer fetches and processes the complete changed object, including
{{B:both the changed and unchanged attribute values}} during replication.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
313
314
315
316
317
One advantage of this approach is that when multiple changes occur to
a single object, the precise sequence of those changes need not be preserved;
only the final state of the entry is significant. But this approach
may have drawbacks when the usage pattern involves single changes to
multiple objects.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
318

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
319
For example, suppose you have a database consisting of 102,400 objects of 1 KB 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
320
each. Further, suppose you routinely run a batch job to change the value of 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
321
a single two-byte attribute value that appears in each of the 102,400 objects 
322
on the provider. Not counting LDAP and TCP/IP protocol overhead, each time you 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
323
run this job each consumer will transfer and process {{B:100 MB}} of data to
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
324
process {{B:200KB of changes!}}
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
325
326
327
328
329
330
331
332
333
334
335
336

99.98% of the data that is transmitted and processed in a case like this will 
be redundant, since it represents values that did not change. This is a waste 
of valuable transmission and processing bandwidth and can cause an unacceptable 
replication backlog to develop. While this situation is extreme, it serves to 
demonstrate a very real problem that is encountered in some LDAP deployments.


* Where Delta-syncrepl comes in:

Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address 
situations like the one described above. Delta-syncrepl works by maintaining a 
337
changelog of a selectable depth in a separate database on the provider. The replication consumer 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
338
339
checks the changelog for the changes it needs and, as long as 
the changelog contains the needed changes, the consumer fetches the changes
340
from the changelog and applies them to its database. If, however, a consumer
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
341
is too far out of sync (or completely empty), conventional syncrepl is used to 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
342
343
bring it up to date and replication then switches back to the delta-syncrepl
mode.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
344

345
346
347
348
349
Note: since the database state is stored in both the changelog DB and the
main DB on the provider, it is important to backup/restore both the changelog
DB and the main DB using slapcat/slapadd when restoring a DB or copying
it to another machine.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
350
351
352
For configuration, please see the {{SECT:Delta-syncrepl}} section.


353
H3: N-Way Multi-Provider Replication
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
354

355
356
Multi-Provider replication is a replication technique using Syncrepl to replicate 
data to multiple provider ("Provider") Directory servers. 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
357

358
H4: Valid Arguments for Multi-Provider replication
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
359

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
360
361
362
363
364
* If any provider fails, other providers will continue to accept updates
* Avoids a single point of failure
* Providers can be located in several physical sites i.e. distributed across
the network/globe.
* Good for Automatic failover/High Availability
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
365

366
H4: Invalid Arguments for Multi-Provider replication
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
367

368
(These are often claimed to be advantages of Multi-Provider replication but
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
369
370
371
372
373
those claims are false):

* It has {{B:NOTHING}} to do with load balancing
* Providers {{B:must}} propagate writes to {{B:all}} the other servers, which 
means the network traffic and write load spreads across all 
374
of the servers the same as for single-provider.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
375
* Server utilization and performance are at best identical for
376
Multi-Provider and Single-Provider replication; at worst Single-Provider is
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
377
378
379
superior because indexing can be tuned differently to optimize for the
different usage patterns between the provider and the consumers.

380
H4: Arguments against Multi-Provider replication
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
381
382
383
384

* Breaks the data consistency guarantees of the directory model
* {{URL:http://www.openldap.org/faq/data/cache/1240.html}}
* If connectivity with a provider is lost because of a network partition, then 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
385
"automatic failover" can just compound the problem
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
386
* Typically, a particular machine cannot distinguish between losing contact
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
387
 with a peer because that peer crashed, or because the network link has failed
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
388
* If a network is partitioned and multiple clients start writing to each of the 
389
"providers" then reconciliation will be a pain; it may be best to simply deny 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
390
writes to the clients that are partitioned from the single provider
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
391
392


393
For configuration, please see the {{SECT:N-Way Multi-Provider}} section below
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
394
395
396
397

H3: MirrorMode replication

MirrorMode is a hybrid configuration that provides all of the consistency
398
399
400
guarantees of single-provider replication, while also providing the high
availability of multi-provider. In MirrorMode two providers are set up to
replicate from each other (as a multi-provider configuration), but an
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
401
external frontend is employed to direct all writes to only one of
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
402
403
404
the two servers. The second provider will only be used for writes if
the first provider crashes, at which point the frontend will switch to
directing all writes to the second provider. When a crashed provider is
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
405
repaired and restarted it will automatically catch up to any changes
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
406
on the running provider and resync.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
407
408
409
410

H4: Arguments for MirrorMode

* Provides a high-availability (HA) solution for directory writes (replicas handle reads)
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
411
412
* As long as one provider is operational, writes can safely be accepted
* Provider nodes replicate from each other, so they are always up to date and
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
413
can be ready to take over (hot standby)
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
414
* Syncrepl also allows the provider nodes to re-synchronize after any downtime
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
415

416

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
417
418
H4: Arguments against MirrorMode

419
* MirrorMode is not what is termed as a Multi-Provider solution. This is because 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
420
421
422
423
writes have to go to just one of the mirror nodes at a time
* MirrorMode can be termed as Active-Active Hot-Standby, therefore an external 
server (slapd in proxy mode) or device (hardware load balancer)
is needed to manage which provider is currently active
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
424
425
426
427
428
429
430
431
* Backups are managed slightly differently
- If backing up the Berkeley database itself and periodically backing up the 
transaction log files, then the same member of the mirror pair needs to be 
used to collect logfiles until the next database backup is taken 

For configuration, please see the {{SECT:MirrorMode}} section below


Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
432
433
434
435
436
437
438
439
440
441
442
H3: Syncrepl Proxy Mode

While the LDAP Sync protocol supports both pull- and push-based replication,
the push mode (refreshAndPersist) must still be initiated from the consumer
before the provider can begin pushing changes. In some network configurations,
particularly where firewalls restrict the direction in which connections
can be made, a provider-initiated push mode may be needed. 

This mode can be configured with the aid of the LDAP Backend
({{SECT: Backends}} and {{slapd-ldap(8)}}). Instead of running the
syncrepl engine on the actual consumer, a slapd-ldap proxy is set up
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
443
near (or collocated with) the provider that points to the consumer,
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
444
445
446
447
448
449
450
451
452
453
454
and the syncrepl engine runs on the proxy.

For configuration, please see the {{SECT:Syncrepl Proxy}} section.

H4: Replacing Slurpd

The old {{slurpd}} mechanism only operated in provider-initiated
push mode.  Slurpd replication was deprecated in favor of Syncrepl
replication and has been completely removed from OpenLDAP 2.4.

The slurpd daemon was the original replication mechanism inherited from 
455
456
UMich's LDAP and operated in push mode: the provider pushed changes to the 
replicas. It was replaced for many reasons, in brief:
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
457
458
459
460

 * It was not reliable
 ** It was extremely sensitive to the ordering of records in the replog
 ** It could easily go out of sync, at which point manual intervention was 
461
462
   required to resync the replica database with the provider directory
 ** It wasn't very tolerant of unavailable servers. If a replica went down 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
463
464
465
   for a long time, the replog could grow to a size that was too large for 
   slurpd to process
 * It only worked in push mode
466
467
 * It required stopping and restarting the provider to add new replicas
 * It only supported single provider replication
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
468
469
470
471
472
473
474
475
476
477
478
479
480
481

Syncrepl has none of those weaknesses:

 * Syncrepl is self-synchronizing; you can start with a consumer database
   in any state from totally empty to fully synced and it will automatically
   do the right thing to achieve and maintain synchronization
 ** It is completely insensitive to the order in which changes occur
 ** It guarantees convergence between the consumer and the provider
    content without manual intervention
 ** It can resynchronize regardless of how long a consumer stays out
    of contact with the provider
 * Syncrepl can operate in either direction
 * Consumers can be added at any time without touching anything on the
   provider
482
 * Multi-provider replication is supported
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
483
484


Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
485
486
487
488
489
H2: Configuring the different replication types

H3: Syncrepl

H4: Syncrepl configuration
490
491
492
493

Because syncrepl is a consumer-side replication engine, the syncrepl
specification is defined in {{slapd.conf}}(5) of the consumer
server, not in the provider server's configuration file.  The initial
494
loading of the consumer content can be performed either by starting
495
the syncrepl engine with no synchronization cookie or by populating
496
the consumer by loading an {{TERM:LDIF}} file dumped as a
497
498
499
500
501
backup at the provider.

When loading from a backup, it is not required to perform the initial
loading from the up-to-date backup of the provider content. The
syncrepl engine will automatically synchronize the initial consumer
502
503
to the current provider content. As a result, it is not
required to stop the provider server in order to avoid the replication
504
505
506
507
inconsistency caused by the updates to the provider content during
the content backup and loading process.

When replicating a large scale directory, especially in a bandwidth
508
constrained environment, it is advised to load the consumer
509
510
511
512
513
514
515
516
from a backup instead of performing a full initial load using
syncrepl.


H4: Set up the provider slapd

The provider is implemented as an overlay, so the overlay itself
must first be configured in {{slapd.conf}}(5) before it can be
517
518
used. The provider has two primary configuration directives and
two secondary directives for when delta-syncrepl is being used.
519
520
Because the LDAP Sync search is subject to access control, proper
access control privileges should be set up for the replicated
521
522
content.

523
524
The two primary options to configure are the checkpoint and
sessionlog behaviors.
525

526
527
528
529
530
The {{EX:contextCSN}} checkpoint is configured by the

>	syncprov-checkpoint <ops> <minutes>

directive. Checkpoints are only tested after successful write
531
operations. If {{<ops>}} operations or more than {{<minutes>}}
532
time has passed since the last checkpoint, a new checkpoint is
533
performed. Checkpointing is disabled by default.
534
535
536

The session log is configured by the

537
>	syncprov-sessionlog <ops>
538

539
540
541
directive, where {{<ops>}} is the maximum number of session log
entries the session log can record. All write operations (except Adds)
are recorded in the log.
542
543
544
545
546

Note that using the session log requires searching on the {{entryUUID}}
attribute. Setting an eq index on this attribute will greatly benefit
the performance of the session log on the provider.

547
548
549
550
The reloadhint option is configured by the

>	syncprov-reloadhint <TRUE|FALSE>

551
552
directive. It must be set TRUE when using the accesslog overlay for
delta-based syncrepl replication support. The default is FALSE.
553
554
555
556
557
558
559
560
561
562
563
564
565

The nonpresent option should only be configured if the overlay is
being placed on top of a log database, such as when used with
delta-syncrepl.

The nonpresent option is configured by the

>	syncprov-nopresent <TRUE|FALSE>

directive. This value should only be set TRUE for a syncprov instance
on top of a log database (such as one managed by the accesslog overlay).
The default is FALSE.

566
567
A more complete example of the {{slapd.conf}}(5) content is thus:

568
>	database mdb
569
570
571
>	maxsize 85899345920
>	suffix dc=example,dc=com
>	rootdn dc=example,dc=com
572
573
574
575
576
577
578
579
580
581
>	directory /var/ldap/db
>	index objectclass,entryCSN,entryUUID eq
>
>	overlay syncprov
>	syncprov-checkpoint 100 10
>	syncprov-sessionlog 100


H4: Set up the consumer slapd

582
583
The syncrepl directive is specified in the database section of
{{slapd.conf}}(5) for the consumer context. The syncrepl engine
584
585
586
is backend independent and the directive can be defined with any
database type.

587
>	database mdb
588
589
590
>	maxsize 85899345920
>	suffix dc=example,dc=com
>	rootdn dc=example,dc=com
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
>	directory /var/ldap/db
>	index objectclass,entryCSN,entryUUID eq
>
>	syncrepl rid=123
>		provider=ldap://provider.example.com:389
>		type=refreshOnly
>		interval=01:00:00:00
>		searchbase="dc=example,dc=com"
>		filter="(objectClass=organizationalPerson)"
>		scope=sub
>		attrs="cn,sn,ou,telephoneNumber,title,l"
>		schemachecking=off
>		bindmethod=simple
>		binddn="cn=syncuser,dc=example,dc=com"
>		credentials=secret

In this example, the consumer will connect to the provider {{slapd}}(8)
at port 389 of {{FILE:ldap://provider.example.com}} to perform a
polling ({{refreshOnly}}) mode of synchronization once a day.  It
will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple
authentication with password "secret".  Note that the access control
privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set
appropriately in the provider to retrieve the desired replication
content. Also the search limits must be high enough on the provider
to allow the syncuser to retrieve a complete copy of the requested
content.  The consumer uses the rootdn to write to its database so
it always has full permissions to write all content.

The synchronization search in the above example will search for the
entries whose objectClass is organizationalPerson in the entire
subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes
are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}},
{{EX:title}}, and {{EX:l}}. The schema checking is turned off, so
that the consumer {{slapd}}(8) will not enforce entry schema
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
625
checking when it processes updates from the provider {{slapd}}(8).
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649

For more detailed information on the syncrepl directive, see the
{{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}}
chapter of this admin guide.


H4: Start the provider and the consumer slapd

The provider {{slapd}}(8) is not required to be restarted.
{{contextCSN}} is automatically generated as needed: it might be
originally contained in the {{TERM:LDIF}} file, generated by
{{slapadd}} (8), generated upon changes in the context, or generated
when the first LDAP Sync search arrives at the provider.  If an
LDIF file is being loaded which did not previously contain the
{{contextCSN}}, the {{-w}} option should be used with {{slapadd}}
(8) to cause it to be generated. This will allow the server to
startup a little quicker the first time it runs.

When starting a consumer {{slapd}}(8), it is possible to provide
a synchronization cookie as the {{-c cookie}} command line option
in order to start the synchronization from a specific state.  The
cookie is a comma separated list of name=value pairs. Currently
supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}.
{{<csn>}} represents the current synchronization state of the
650
consumer. {{<rid>}} identifies a consumer locally
651
652
within the consumer server. It is used to relate the cookie to the
syncrepl definition in {{slapd.conf}}(5) which has the matching
653
{{<rid>}}.  The {{<rid>}} must have no more than 3 decimal
654
digits.  The command line cookie overrides the synchronization
655
cookie stored in the consumer database.
656
657


Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
658
659
H3: Delta-syncrepl

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
660
H4: Delta-syncrepl Provider configuration
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
661

662
Setting up delta-syncrepl requires configuration changes on both the provider and 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
663
664
replica servers:

665
>     # Give the replicator DN unlimited read access.  This ACL needs to be
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
666
667
668
>     # merged with other ACL statements, and/or moved within the scope
>     # of a database.  The "by * break" portion causes evaluation of
>     # subsequent rules.  See slapd.access(5) for details.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
669
>     access to *
670
>        by dn.base="cn=replicator,dc=example,dc=com" read
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
671
672
673
>        by * break
>     
>     # Set the module path location
674
>     modulepath /usr/lib/openldap
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
675
>     
676
677
>     # Load the mdb backend
>     moduleload back_mdb.la
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
678
679
680
681
682
683
684
685
>     
>     # Load the accesslog overlay
>     moduleload accesslog.la
>     
>     #Load the syncprov overlay
>     moduleload syncprov.la
>     
>     # Accesslog database definitions
686
>     database mdb
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
687
688
>     suffix cn=accesslog
>     rootdn cn=accesslog
689
690
>     directory /var/lib/db/accesslog
>     maxsize 85899345920
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
691
>     index default eq
692
>     index entryCSN,objectClass,reqEnd,reqResult,reqStart,reqDN
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
693
694
695
696
697
>     
>     overlay syncprov
>     syncprov-nopresent TRUE
>     syncprov-reloadhint TRUE
>     
698
>     # Let the replicator DN have limitless searches
699
>     limits dn.exact="cn=replicator,dc=example,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
700
701
>     
>     # Primary database definitions
702
703
704
705
>     database mdb
>     suffix "dc=example,dc=com"
>     rootdn "cn=manager,dc=example,dc=com"
>     maxsize 85899345920
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
>     
>     ## Whatever other configuration options are desired
>     
>     # syncprov specific indexing
>     index entryCSN eq
>     index entryUUID eq
>     
>     # syncrepl Provider for primary db
>     overlay syncprov
>     syncprov-checkpoint 1000 60
>     
>     # accesslog overlay definitions for primary db
>     overlay accesslog
>     logdb cn=accesslog
>     logops writes
>     logsuccess TRUE
>     # scan the accesslog DB every day, and purge entries older than 7 days
>     logpurge 07+00:00 01+00:00
>     
725
>     # Let the replicator DN have limitless searches
726
>     limits dn.exact="cn=replicator,dc=example,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
727

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
728
For more information, always consult the relevant man pages ({{slapo-accesslog}}(5) and {{slapd.conf}}(5))
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
729
730


Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
731
H4: Delta-syncrepl Consumer configuration
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
732

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
733
>     # Replica database configuration
734
735
736
737
>     database mdb
>     suffix "dc=example,dc=com"
>     rootdn "cn=manager,dc=example,dc=com"
>     maxsize 85899345920
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
738
739
740
741
742
743
744
745
746
>     
>     ## Whatever other configuration bits for the replica, like indexing
>     ## that you want
>     
>     # syncrepl specific indices
>     index entryUUID eq
>     
>     # syncrepl directives
>     syncrepl  rid=0
747
>               provider=ldap://ldapprovider.example.com:389
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
748
>               bindmethod=simple
749
>               binddn="cn=replicator,dc=example,dc=com"
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
750
>               credentials=secret
751
>               searchbase="dc=example,dc=com"
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
752
753
754
755
756
757
758
>               logbase="cn=accesslog"
>               logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
>               schemachecking=on
>               type=refreshAndPersist
>               retry="60 +"
>               syncdata=accesslog
>     
759
760
>     # Refer updates to the provider
>     updateref               ldap://ldapprovider.example.com
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
761
762
763


The above configuration assumes that you have a replicator identity defined 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
764
765
in your database that can be used to bind to the provider. In addition, 
all of the databases (primary, replica, and the accesslog 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
766
storage database) should also have properly tuned {{DB_CONFIG}} files that meet 
767
your needs if using the bdb or hdb backends.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
768

769
Note: An accesslog database is unique to a given provider. It should
770
never be replicated.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
771

772
H3: N-Way Multi-Provider
773

774
775
For the following example we will be using 3 Provider nodes. Keeping in line with
{{B:test050-syncrepl-multiprovider}} of the OpenLDAP test suite, we will be configuring
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
{{slapd(8)}} via {{B:cn=config}}

This sets up the config database:

>     dn: cn=config
>     objectClass: olcGlobal
>     cn: config
>     olcServerID: 1
>     
>     dn: olcDatabase={0}config,cn=config
>     objectClass: olcDatabaseConfig
>     olcDatabase: {0}config
>     olcRootPW: secret

second and third servers will have a different olcServerID obviously:

>     dn: cn=config
>     objectClass: olcGlobal
>     cn: config
>     olcServerID: 2
>     
>     dn: olcDatabase={0}config,cn=config
>     objectClass: olcDatabaseConfig
>     olcDatabase: {0}config
>     olcRootPW: secret

802
This sets up syncrepl as a provider (since these are all providers):
803
804
805
806
807
808
809

>     dn: cn=module,cn=config
>     objectClass: olcModuleList
>     cn: module
>     olcModulePath: /usr/local/libexec/openldap
>     olcModuleLoad: syncprov.la

810
Now we setup the first Provider Node (replace $URI1, $URI2 and $URI3 etc. with your actual ldap urls):
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840

>     dn: cn=config
>     changetype: modify
>     replace: olcServerID
>     olcServerID: 1 $URI1
>     olcServerID: 2 $URI2
>     olcServerID: 3 $URI3
>     
>     dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config
>     changetype: add
>     objectClass: olcOverlayConfig
>     objectClass: olcSyncProvConfig
>     olcOverlay: syncprov
>     
>     dn: olcDatabase={0}config,cn=config
>     changetype: modify
>     add: olcSyncRepl
>     olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple
>       credentials=secret searchbase="cn=config" type=refreshAndPersist
>       retry="5 5 300 5" timeout=1
>     olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple
>       credentials=secret searchbase="cn=config" type=refreshAndPersist
>       retry="5 5 300 5" timeout=1
>     olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple
>       credentials=secret searchbase="cn=config" type=refreshAndPersist
>       retry="5 5 300 5" timeout=1
>     -
>     add: olcMirrorMode
>     olcMirrorMode: TRUE

841
Now start up the provider and a consumer/s, also add the above LDIF to the first consumer, second consumer etc. It will then replicate {{B:cn=config}}. You now have N-Way Multi-Provider on the config database.
842

843
We still have to replicate the actual data, not just the config, so add to the provider (all active and configured consumers/providers will pull down this config, as they are all syncing). Also, replace all {{${}}} variables with whatever is applicable to your setup:
844
845
846
847
848
849
850
851
852

>     dn: olcDatabase={1}$BACKEND,cn=config
>     objectClass: olcDatabaseConfig
>     objectClass: olc${BACKEND}Config
>     olcDatabase: {1}$BACKEND
>     olcSuffix: $BASEDN
>     olcDbDirectory: ./db
>     olcRootDN: $MANAGERDN
>     olcRootPW: $PASSWD
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
853
>     olcLimits: dn.exact="$MANAGERDN" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
>     olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple
>       credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
>       interval=00:00:00:10 retry="5 5 300 5" timeout=1
>     olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple
>       credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
>       interval=00:00:00:10 retry="5 5 300 5" timeout=1
>     olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple
>       credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly
>       interval=00:00:00:10 retry="5 5 300 5" timeout=1
>     olcMirrorMode: TRUE
>     
>     dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config
>     changetype: add
>     objectClass: olcOverlayConfig
>     objectClass: olcSyncProvConfig
>     olcOverlay: syncprov

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
871
872
873
874
875
876
877
878
Note: All of your servers' clocks must be tightly synchronized using
e.g. NTP {{http://www.ntp.org/}}, atomic clock, or some other reliable
time reference.

Note: As stated in {{slapd-config}}(5), URLs specified in {{olcSyncRepl}}
directives are the URLs of the servers from which to replicate. These
must exactly match the URLs {{slapd}} listens on ({{-h}} in {{SECT:Command-Line Options}}).
Otherwise slapd may attempt to replicate from itself, causing a loop.
879

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
880
H3: MirrorMode
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
881
882

MirrorMode configuration is actually very easy. If you have ever setup a normal
883
slapd syncrepl provider, then the only change is the following two directives:
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
884
885

>       mirrormode  on
886
>       serverID    1
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
887

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
888
Note: You need to make sure that the {{serverID}} of each mirror node is 
889
different and add it as a global configuration option.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
890
891
892

H4: Mirror Node Configuration

893
894
The first step is to configure the syncrepl provider the same as in the 
{{SECT:Set up the provider slapd}} section.
895

896
Here's a specific cut down example using {{SECT:LDAP Sync Replication}} in
897
{{refreshAndPersist}} mode:
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
898
899
900

MirrorMode node 1:

901
902
903
904
>       # Global section
>       serverID    1
>       # database section
>       
905
>       # syncrepl directive    
906
>       syncrepl      rid=001
907
>                     provider=ldap://ldap-sid2.example.com
908
909
910
911
912
913
914
915
>                     bindmethod=simple
>                     binddn="cn=mirrormode,dc=example,dc=com"
>                     credentials=mirrormode
>                     searchbase="dc=example,dc=com"
>                     schemachecking=on
>                     type=refreshAndPersist
>                     retry="60 +"
>
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
916
917
918
919
>       mirrormode on

MirrorMode node 2:

920
921
922
923
>       # Global section
>       serverID    2
>       # database section
>       
924
>       # syncrepl directive
925
>       syncrepl      rid=001
926
>                     provider=ldap://ldap-sid1.example.com
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
927
928
929
930
931
932
933
934
935
936
937
>                     bindmethod=simple
>                     binddn="cn=mirrormode,dc=example,dc=com"
>                     credentials=mirrormode
>                     searchbase="dc=example,dc=com"
>                     schemachecking=on
>                     type=refreshAndPersist
>                     retry="60 +"
>       
>       mirrormode on

It's simple really; each MirrorMode node is setup {{B:exactly}} the same, except
938
939
that the {{serverID}} is unique, and each consumer is pointed to 
the other server.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
940

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
941
H5: Failover Configuration
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
942
943
944
945

There are generally 2 choices for this; 1.  Hardware proxies/load-balancing or 
dedicated proxy software, 2. using a Back-LDAP proxy as a syncrepl provider

946
A typical enterprise example might be:
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
947

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
948
949
!import "dual_dc.png"; align="center"; title="MirrorMode Enterprise Configuration"
FT[align="Center"] Figure X.Y: MirrorMode in a Dual Data Center Configuration
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
950

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
951
H5: Normal Consumer Configuration
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
952
953
954
955
956

This is exactly the same as the {{SECT:Set up the consumer slapd}} section. It
can either setup in normal {{SECT:syncrepl replication}} mode, or in 
{{SECT:delta-syncrepl replication}} mode.

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
957
H4: MirrorMode Summary
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
958

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
959
You will now have a directory architecture that provides all of the 
960
961
consistency guarantees of single-provider replication, while also providing the 
high availability of multi-provider replication.
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
962

963

Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
964
965
966
967
968
969
970
971
H3: Syncrepl Proxy

!import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode"
FT[align="Center"] Figure X.Y: Replacing slurpd

The following example is for a self-contained push-based replication solution:

>	#######################################################################
972
>	# Standard OpenLDAP Provider
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
973
974
975
976
977
978
979
980
981
982
>	#######################################################################
>	
>	include     /usr/local/etc/openldap/schema/core.schema
>	include     /usr/local/etc/openldap/schema/cosine.schema
>	include     /usr/local/etc/openldap/schema/nis.schema
>	include     /usr/local/etc/openldap/schema/inetorgperson.schema
>	
>	include     /usr/local/etc/openldap/slapd.acl
>	
>	modulepath  /usr/local/libexec/openldap
983
>	moduleload  back_mdb.la
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
984
985
986
987
988
989
990
991
992
>	moduleload  syncprov.la
>	moduleload  back_monitor.la
>	moduleload  back_ldap.la
>	
>	pidfile     /usr/local/var/slapd.pid
>	argsfile    /usr/local/var/slapd.args
>	
>	loglevel    sync stats
>	
993
>	database    mdb
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
994
995
>	suffix      "dc=suretecsystems,dc=com"
>	directory   /usr/local/var/openldap-data
996
>	maxsize     85899345920
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
>	
>	checkpoint      1024 5
>	
>	index       objectClass eq
>	# rest of indexes
>	index       default     sub
>	
>	rootdn		"cn=admin,dc=suretecsystems,dc=com"
>	rootpw	  	testing	
>	
>	# syncprov specific indexing
>	index entryCSN eq
>	index entryUUID eq
>	
>	# syncrepl Provider for primary db
>	overlay syncprov
>	syncprov-checkpoint 1000 60
>	
1015
>	# Let the replicator DN have limitless searches
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
>	limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
>	
>	database    monitor
>	
>	database    config
>	rootpw	  	testing
>	
>	##############################################################################
>	# Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
>	##############################################################################
>	
>	database        ldap
>	# ignore conflicts with other databases, as we need to push out to same suffix
>	hidden		    on
>	suffix          "dc=suretecsystems,dc=com"
>	rootdn          "cn=slapd-ldap"
>	uri             ldap://localhost:9012/
>	
>	lastmod         on
>	        
>	# We don't need any access to this DSA
>	restrict        all
>	
>	acl-bind        bindmethod=simple
>	                binddn="cn=replicator,dc=suretecsystems,dc=com"
>	                credentials=testing
>	
>	syncrepl        rid=001
>	                provider=ldap://localhost:9011/
>	                binddn="cn=replicator,dc=suretecsystems,dc=com"
>	                bindmethod=simple
>	                credentials=testing
>	                searchbase="dc=suretecsystems,dc=com"
>	                type=refreshAndPersist
>	                retry="5 5 300 5"
>	
>	overlay         syncprov

A replica configuration for this type of setup could be:

>	#######################################################################
1057
>	# Standard OpenLDAP Replica without Syncrepl
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
>	#######################################################################
>	
>	include     /usr/local/etc/openldap/schema/core.schema
>	include     /usr/local/etc/openldap/schema/cosine.schema
>	include     /usr/local/etc/openldap/schema/nis.schema
>	include     /usr/local/etc/openldap/schema/inetorgperson.schema
>	
>	include     /usr/local/etc/openldap/slapd.acl
>	
>	modulepath  /usr/local/libexec/openldap
1068
>	moduleload  back_mdb.la
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1069
1070
1071
1072
1073
1074
1075
1076
1077
>	moduleload  syncprov.la
>	moduleload  back_monitor.la
>	moduleload  back_ldap.la
>	
>	pidfile     /usr/local/var/slapd.pid
>	argsfile    /usr/local/var/slapd.args
>	
>	loglevel    sync stats
>	
1078
>	database    mdb
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1079
>	suffix      "dc=suretecsystems,dc=com"
1080
>	directory   /usr/local/var/openldap-consumer/data
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1081
>	
1082
>	maxsize         85899345920
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1083
1084
1085
1086
1087
1088
1089
1090
1091
>	checkpoint      1024 5
>	
>	index       objectClass eq
>	# rest of indexes
>	index       default     sub
>	
>	rootdn		"cn=admin,dc=suretecsystems,dc=com"
>	rootpw	  	testing	
>	
1092
>	# Let the replicator DN have limitless searches
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1093
1094
1095
1096
>	limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited
>	
>	updatedn "cn=replicator,dc=suretecsystems,dc=com"
>	
1097
>	# Refer updates to the provider
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1098
1099
1100
1101
1102
1103
1104
1105
1106
>	updateref   ldap://localhost:9011
>	
>	database    monitor
>	
>	database    config
>	rootpw	  	testing

You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be:
	
1107
>	# Give the replicator DN unlimited read access.  This ACL may need to be
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
>	# merged with other ACL statements.
>	
>	access to *
>	     by dn.base="cn=replicator,dc=suretecsystems,dc=com" write
>	     by * break
>	
>	access to dn.base=""
>	        by * read
>	
>	access to dn.base="cn=Subschema"
>	        by * read
>	
>	access to dn.subtree="cn=Monitor"
>	    by dn.exact="uid=admin,dc=suretecsystems,dc=com" write
>	    by users read
>	    by * none
>	
>	access to *
>	        by self write
>	        by * read 

In order to support more replicas, just add more {{database ldap}} sections and
increment the {{syncrepl rid}} number accordingly.

1132
Note: You must populate the Provider and Replica directories with the same data, 
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1133
1134
unlike when using normal Syncrepl

1135
If you do not have access to modify the provider directory configuration you can
Quanah Gibson-Mount's avatar
Quanah Gibson-Mount committed
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
configure a standalone ldap proxy, which might look like:

!import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode"
FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version

The following configuration is an example of a standalone LDAP Proxy:

>	include     /usr/local/etc/openldap/schema/core.schema
>	include     /usr/local/etc/openldap/schema/cosine.schema
>	include     /usr/local/etc/openldap/schema/nis.schema
>	include     /usr/local/etc/openldap/schema/inetorgperson.schema
>	
>	include     /usr/local/etc/openldap/slapd.acl
>	
>	modulepath  /usr/local/libexec/openldap
>	moduleload  syncprov.la
>	moduleload  back_ldap.la
>	
>	##############################################################################
>	# Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap
>	##############################################################################
>	
>	database        ldap
>	# ignore conflicts with other databases, as we need to push out to same suffix
>	hidden		    on
>	suffix          "dc=suretecsystems,dc=com"
>	rootdn          "cn=slapd-ldap"
>	uri             ldap://localhost:9012/
>	
>	lastmod         on
>	        
>	# We don't need any access to this DSA
>	restrict        all
>	
>	acl-bind        bindmethod=simple
>	                binddn="cn=replicator,dc=suretecsystems,dc=com"
>	                credentials=testing
>	
>	syncrepl        rid=001
>	                provider=ldap://localhost:9011/
>	                binddn="cn=replicator,dc=suretecsystems,dc=com"
>	                bindmethod=simple
>	                credentials=testing
>	                searchbase="dc=suretecsystems,dc=com"
>	                type=refreshAndPersist
>	                retry="5 5 300 5"
>	
>	overlay         syncprov

As you can see, you can let your imagination go wild using Syncrepl and 
{{slapd-ldap(8)}} tailoring your replication to fit your specific network 
topology.