1. 03 Sep, 2001 1 commit
  2. 02 Sep, 2001 4 commits
  3. 01 Sep, 2001 4 commits
  4. 31 Aug, 2001 2 commits
  5. 30 Aug, 2001 3 commits
  6. 29 Aug, 2001 7 commits
  7. 28 Aug, 2001 3 commits
  8. 27 Aug, 2001 2 commits
  9. 26 Aug, 2001 1 commit
  10. 25 Aug, 2001 1 commit
  11. 24 Aug, 2001 2 commits
  12. 15 Aug, 2001 1 commit
  13. 04 Aug, 2001 5 commits
  14. 03 Aug, 2001 2 commits
  15. 02 Aug, 2001 2 commits
    • Dmitry Kovalev's avatar
      A big bunch of improvements, contributed by Sam Drake and Raj Damani. · 2f4d324f
      Dmitry Kovalev authored
      Summary of changes is cited below.
      The patch still needs some cosmetic changes to be made, but is ready for testing.
      -----Original Message-----
      From: Sam Drake [mailto:drake@timesten.com]
      Sent: Saturday, April 07, 2001 10:40 PM
      To: 'mitya@seismic.ru'
      Cc: openldap-devel@OpenLDAP.org
      Subject: RE: Slapd frontend performance issues
      FYI, here is a short description of the changes I made.  I'll package up the
      changes asap, but it may take a couple of days.
      The performance numbers quoted in this report were seen at my location with
      a 100,000 object database ... the slower numbers I mentioned earlier were
      reported by a customer with a 1,000,000 object database.
      I also can't explain the very poor performance I saw with OpenLDAP and LDBM
      with a 100,000 object database.
      ...Sam Drake / TimesTen Performance Software
      Work Performed
      OpenLDAP 2.0.9, including back-sql, was built successfully on Solaris
      8 using gcc.  The LDAP server itself, slapd, passed all tests bundled
      with OpenLDAP.  OpenLDAP was built using Sleepycat LDBM release 3.1.17
      as the "native" storage manager.
      The experimental back-sql facility in slapd was also built
      successfully.  It was built using Oracle release 8.1.7 and the Oracle
      ODBC driver and ODBC Driver Manager from Merant.  Rudimentary testing
      was performed with the data and examples provided with back-sql, and
      back-sql was found to be functional.
      Slapd and back-sql were then tested with TimesTen, using TimesTen
      4.1.1.  Back-sql was not immediately functional with TimesTen due to a
      number of SQL limitations in the TimesTen product.
      Functional issues encountered were:
      1. Back-sql issued SELECT statements including the construct,
         "UPPER(?)".  While TimesTen supports UPPER, it does not support the
         use of parameters as input to builtin functions.  Back-sql was
         modified to convert the parameter to upper case prior to giving it
         to the underlying database ... a change that is appropriate for all
      2. Back-sql issued SELECT statements using the SQL CONCAT function.
         TimesTen does not support this function.  Back-sql was modified to
         concatentate the necessary strings itself (in "C" code) prior to
         passing the parameters to SQL.  This change is also appropriate for
         all databases, not just TimesTen.
      Once these two issues were resolved, back-sql could successfully
      process LDAP searches using the sample data and examples provided with
      While performance was not measured at this point, numerous serious
      performance problems were observed with the back-sql code and the
      generated SQL.  In particular:
      1. In the process of implementing an LDAP search, back-sql will
         generate and execute a SQL query for all object classes stored in
         back-sql.  During the source of generating each SQL query, it is
         common for back-sql to determine that a particular object class can
         not possibly have any members satisfying the search.  For example,
         this can occur if the query searches an attribute of the LDAP
         object that does not exist in the SQL schema.  In this case,
         back-sql would generate and issue the SQL query anyway, including a
         clause such as "WHERE 1=0" in the generated SELECT.  The overhead
         of parsing, optimizing and executing the query is non-trivial, and
         the answer (the empty set) is known in advance. Solution: Back-sql
         was modified to stop executing a SQL query when it can be
         predetermined that the query will return no rows.
      2. Searches in LDAP are fundamentally case-insensitive ("abc" is equal
         to "aBc").  However, in SQL this is not normally the case.
         Back-sql thus generated SQL SELECT statements including clauses of
         the form, "WHERE UPPER(attribute) = 'JOE'".  Even if an index is
         defined on the attribute in the relational database, the index can
         not be used to satisfy the query, as the index is case sensitive.
         The relational database then is forced to scan all rows in the
         table in order to satisfy the query ... an expensive and
         non-scalable proposition.  Solution: Back-sql was modified to allow
         the schema designer to add additional "upper cased" columns to the
         SQL schema.  These columns, if present, contain an upper cased
         version of the "standard" field, and will be used preferentially
         for searching.  Such columns can be provided for all searchable
         columns, some columns, or no columns.  An application using
         database "triggers" or similar mechanisms can automatically
         maintain these upper cased columns when the standard column is
      3. In order to implement the hierarchical nature of LDAP object
         hierarchies, OpenLDAP uses suffix searches in SQL.  For example, to
         find all objects in the subtree "o=TimesTen,c=us", a SQL SELECT
         statement of the form, "WHERE UPPER(dn) LIKE '%O=TIMESTEN,C=US'"
         would be employed.  Aside from the UPPER issue discussed above, a
         second performance problem in this query is the use of suffix
         search.  In TimesTen (and most relational databases), indexes can
         be used to optimize exact-match searches and prefix searches.
         However, suffix searches must be performed by scanning every row in
         the table ... an expensive and non-scalable proposition.  Solution:
         Back-sql was modified to optionally add a new "dn_ru" column to the
         ldap_entries table.  This additional column, if present, contains a
         byte-reversed and upper cased version of the DN.  This allows
         back-sql to generate indexable prefix searches.  This column is
         also easily maintained automatically through the use of triggers.
      A simple database schema was generated holding the LDAP objects and
      attributes specified by our customer.  An application was written to
      generate test databases.  Both TimesTen and Oracle 8.1.7 were
      populated with 100,000 entry databases.
      Load Times
      Using "slapadd" followed by "slapindex", loading and indexing 100,000
      entries in an LDBM database ran for 19 minutes 10 seconds.
      Using a C++ application that used ODBC, loading 100,000 entries into
      a disk based RDBMS took 17 minutes 53 seconds.
      Using a C++ application that used ODBC, loading 100,000 entries into
      TimesTen took 1 minute 40 seconds.
      Search Times
      The command, "timex timesearch.sh '(cn=fname210100*)'" was used to
      test search times.  This command issues the same LDAP search 4000
      times over a single LDAP connection.  Both the client and server
      (slapd) were run on the same machine.
      With TimesTen as the database, 4000 queries took 14.93 seconds, for a
      rate of 267.9 per second.
      With a disk based RDBMS as the database, 4000 queries took 77.79 seconds,
      for a
      rate of 51.42 per second.
      With LDBM as the database, 1 query takes 76 seconds, or 0.076 per
      second.  Something is clearly broken.
    • Kurt Zeilenga's avatar
      Fix bug introduced during TLS rework · 16fa8c4a
      Kurt Zeilenga authored