List Archives

This forum is an archive of all posts to our mailing list over the past few years. The forum is set read only therefore to contribute you will need to join our list community. See more info about this here.

List Archives

Unanswered Active Topics

Forums Search

Forums >ActiveDir Mail List Archive >List Archives

Subject: [ActiveDir] Changing MaxPageSize

Prev Next

You are not authorized to post a reply.

Page 2 of 2

Author

Messages

dmitrig

Posts:59

04/18/2008 9:47 AM
The change will be picked up immediately, without reboot. However, this policy is only enforced at connect time. I.e. if you have existing connections that violate the policy, they will not be kicked out. -----Original Message----- From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Shaun Little Sent: Friday, April 18, 2008 5:54 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Changing MaxPageSize I have a quick question, I have just made a change to the maxconnections attribute in the queryPolicy class using LDP. Will the change take effect after a set period of time or is a reboot of the server required? Thanks Shaun -----Original Message----- From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brian Arkills Sent: 17 April 2008 17:48 To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Changing MaxPageSize OK, but that doesn't change my objection to it. The net effect of your proposed solution is to remove any server-imposed size limit. Effectively this is the same as option #2 in your list, except you now have an unlimited maxpagesize. Stepping back a bit, let me say a few things about how other LDAP directories handle this, and maybe this will inspire what I'd think would be better solutions. First, a review of how AD handles size limits, then an example of another LDAP directory. So AD has many limits which together weave together into the overall picture. There's a server-side limit (called MaxPageSize), which by default is 1000. There's a second server-side limit which you might think about as an absolute maximum server-side limit (called MaxResultSetSize), which by default is 262144. I see that this latter limit hasn't been noted yet in the thread. There's also the possibility of a client side size limit which all depends on what the client sets it to be. AD imposes the lower of the client size limit or the server size limit unless the paging control is invoked. If the paging control is invoked, then it imposes the lower of the client size limit or the absolute maximum server-side limit but it pages the results back to the client. In other words, AD uses the paging control as a way for clients to get around a soft limit. But it still has a hard upper limit. As a side note, as far as I know AD is unique in this regard among LDAP directories; no other LDAP directories employ the paging control as a way to "get around" a server size limit. One of the annoying things here is that the term MaxPageSize is misleading--you don't have to be doing any paging to be subject to that size limit. Take openldap as another example. With OpenLDAP, there is a default server size limit. I think it's 500, if I recall correctly. As with all ldap clients there's also a client side size limit. Paging is not used as a way to circumvent the limit. Instead, you can set a different size limit on a per-user basis. So my authentication token might be given a server size limit of 150000, but everyone else is subject to the default of 500. Like AD, the net limit is the lower of the client side limit and the server side limit. If you used paging, it would just break that result set into smaller pieces, but it could not be used to circumvent any of the limits. This approach is extremely similar to how other LDAP directories handle size limits. I'm not sure I understand why Microsoft designed their size limit approach to be so different from every other LDAP directory on the planet. However, I also don't see that your proposed solution to the stated problem makes sense (it also doesn't address the MaxResultSetSize). I'd think a more sensible solution would be to propose that certain users could have a different size limit which overrode the MaxPageSize and possibly the MaxResultSetSize. Such a solution would be a minor change in architecture but I'd think it would be doable. Currently the way the size limits are set is via a queryPolicy object in the config partition. There's a default query policy object, but you can create additional ones. You'd do so, with the limits you'd want, but the missing pieces would be: a) being able to associate that policy object with a user object (this might require a schema change) b) having the AD code honor this new behavior > -----Original Message----- > From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir- > owner@mail.activedir.org] On Behalf Of Dmitri Gavrilov > Sent: Tuesday, April 15, 2008 12:25 PM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] Changing MaxPageSize > > If this proposal is implemented, then no changes are required to client > apps. They don't need to pass any new controls. > > Here's the scenario: > > You have an old ldap app that does not know how to page. > Your domain grows, and the app stops working because it can only get > 1000 users max. > > What do you do? Currently, you only have two options: > 1. Try to fix the app. > 2. Increase MaxPageSize. > > 1 is often impossible. 2 is generally frowned upon (e.g. in this > thread). You are out of options! > > I am trying to propose a new option, that would allow to do paging at > server, while presenting the search results to the client as a regular > ldap search result. > > -----Original Message----- > From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir- > owner@mail.activedir.org] On Behalf Of Brian Arkills > Sent: Tuesday, April 15, 2008 11:58 AM > To: ActiveDir@mail.activedir.org > Subject: RE: [ActiveDir] Changing MaxPageSize > > I'm somewhat new to this list, but very familiar with both AD and LDAP > (see http://www.amazon.com/LDAP-Directories-Explained-Introduction- > Independent/dp/020178792X/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=120828514 > 0&sr=8-1). > > I don't understand the strategy of this proposal. Here's what I'm > hearing: > > -Client has software which doesn't support the LDAP paging control > that's been around for years and which is supported by lots of ldap- > based software. > -Instead of getting the software vendor to implement and support that > well-supported LDAP paging control, there's an idea that the client > would like Microsoft to implement a new LDAP control which > circumvents the maxpagesize that the server enforces on all LDAP search > requests. > > If the software vendor won't support the well-known LDAP paging > control, how are you going to get them to support a brand-new LDAP > paging control? > > Put another way, all LDAP searches of AD are currently subject to the > server-side maxpagesize limit (this is a common thing to do in LDAP > directories). There are many good reasons for such a limit to be in > place, so you have little to no chance of convincing Microsoft (or many > of us) that completely removing that limit is a good idea. So this > means you need an LDAP control extension to allow certain requestors to > circumvent that limit. But if you have such a control, then anyone can > call it which means you've lost the value of having a limit at all. So > then your control needs some authorization logic built into it. > > > -----Original Message----- > > From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir- > > owner@mail.activedir.org] On Behalf Of Joe Kaplan > > Sent: Tuesday, April 15, 2008 10:05 AM > > To: ActiveDir@mail.activedir.org > > Subject: Re: [ActiveDir] Changing MaxPageSize > > > > I'll make sure this gets brought up during one of the chalk talks > > unless I > > happen to not be there, but if I can't, I'll ask someone else to > bring > > it > > up. > > > > We did just discuss AD/LDS on Vista. > > > > Joe K. > > > > ----- Original Message ----- > > From: "Dmitri Gavrilov" <dmitrig@microsoft.com> > > To: <ActiveDir@mail.activedir.org> > > Sent: Tuesday, April 15, 2008 11:28 AM > > Subject: RE: [ActiveDir] Changing MaxPageSize > > > > > > Yes, something like this. > > Yes, it will consume server memory, while the search is being run. > But > > there's nothing new here - you can already do this today, if you > > request > > lots of large objects. We have proper controls (max output packet > size, > > max > > query duration, etc) to protect server. > > > > Yes, if client is paging, then we won't use this mechanism (double- > > paging > > looks scary to me). Client's page sizes would have to be under the > > "server > > maxPageSize" value. > > > > From: ActiveDir-owner@mail.activedir.org > > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer > > Sent: Tuesday, April 15, 2008 9:17 AM > > To: ActiveDir@mail.activedir.org > > Subject: Re: [ActiveDir] Changing MaxPageSize > > > > Dmitri, > > > > So it would be like an intermediary buffer between the client and > > server, > > but on the server itself? > > > > Client -> Server -> Buffer using paging control queues up results -> > > Returns > > results to client in one stream ( ? ) > > > > If paging is requested, then the server side buffer is not used? > > Would > > this consume a lot of memory space for the server with a large amount > > of > > searches to hold the results before returning to the client? > > > > I obviously don't know as much as I would like about how the control > > extension works, but I am curious. > > > > Thanks, > > Jef > > > > > > > > > > > > From: Dmitri Gavrilov<mailto:dmitrig@microsoft.com> > > Sent: Tuesday, April 15, 2008 11:07 AM > > To: ActiveDir@mail.activedir.org<mailto:ActiveDir@mail.activedir.org> > > Subject: RE: [ActiveDir] Changing MaxPageSize > > > > We would still respect paging requests, and continue working as > > previously. > > > > The only difference is that we will now support setting MaxPageSize > > to > > large values. Today, you cannot get more than 1000 entries in a > single > > search query. If we make this change, then you will be. > > > > In a sense, this functionality is supported today in ADSI. ADSI/S.DS > > hides > > paging from you. IIRC, you can set the pageSize parameter, but you > > still get > > the result as one continuous stream of entries. ADSI queries the > server > > for > > you and fetches the next page when you run out of entries. My > proposal > > to > > enable similar functionality on the server. > > > > But we will, of course, continue supporting client-initiated paged > > searches, > > just like we do today. > > > > From: ActiveDir-owner@mail.activedir.org > > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer > > Sent: Tuesday, April 15, 2008 8:58 AM > > To: activedir@mail.activedir.org > > Subject: RE: [ActiveDir] Changing MaxPageSize > > > > Dmitri, > > > > This sounds like it would kill any chance of a client who wants to do > > paging,no? what would happen if a client specified a page size on > it's > > own? > > > > It would still send a large result set to the client which could > > negatively > > affect the client as well. > > > > > > > > Jef Kazimer > > ------- > > http://jeftek.com > > ________________________________ > > From: dmitrig@microsoft.com > > To: ActiveDir@mail.activedir.org > > Date: Tue, 15 Apr 2008 08:47:04 -0700 > > Subject: RE: [ActiveDir] Changing MaxPageSize > > Well, basically the proposal is what I briefly described below... > > > > We would build a paging mechanism in LDAP head on the server. Then, > we > > would > > support setting MaxPageSize policy to ridiculously large (practically > > unlimited) values, and instead do paging at the server side (using > some > > internally configured "server" page size). Each page would be > computed > > in a > > separate transaction, but then all results would be packaged and sent > > to the > > client as a single response. Then the practical limit would be the > max > > packet size (10Mb by default). > > > > What this achieves: > > * Downlevel apps that don't support paging can continue > > working. > > * Server is not getting killed by long-running transactions. > > > > The downside: > > * Paging is now hidden from the app, but we are still paging. > > This > > means all the bad transactional properties of paging are now sort of > > hidden > > from the app: entries might be missed or reported multiple times (in > > case > > they are moved or modified between page requests). There's nothing > new > > here - you can get this today, when paging. But now, we hide it at > > server > > side, which makes it less apparent to apps. > > > > I'll be at PG dinner today, we can chat about it. > > > > Dmitri > > > > > > From: ActiveDir-owner@mail.activedir.org > > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer > > Sent: Tuesday, April 15, 2008 8:29 AM > > To: activedir@mail.activedir.org > > Subject: RE: [ActiveDir] Changing MaxPageSize > > > > Dmitri, > > > > Can you point me in the direction of what this proposal is? Can you > > provide a summary for all those here? > > > > When you say "hidden", how do you mean? > > > > Jef > > > > Jef Kazimer > > ------- > > http://jeftek.com > > ________________________________ > > From: dmitrig@microsoft.com > > To: ActiveDir@mail.activedir.org > > Date: Tue, 15 Apr 2008 08:15:57 -0700 > > Subject: RE: [ActiveDir] Changing MaxPageSize > > This serves as a very appropriate and timely reminder for my earlier > > post: > > > > Those of you that are coming to MVP summit: I implore you to push AD > > team to > > finally fix this one. There was a good proposal on the table... If > they > > do > > it, then the paging can be hidden at the server side (basically, AD > > devs do > > it once, instead of pushing app devs to do it many times in their > > apps). > > Those of you with strong influence on MSFT (meaning you are wielding > > large > > customers), you can do this too. Then this age-long question will be > > finally > > put to rest. > > > > > > From: ActiveDir-owner@mail.activedir.org > > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe > > Sent: Tuesday, April 15, 2008 8:06 AM > > To: ActiveDir@mail.activedir.org > > Subject: RE: [ActiveDir] Changing MaxPageSize > > > > Nothing new here. My client has one application which cannot utilize > > paging, so we dedicated a DC for it and defined a custom query policy > > which > > applies only to that DC. Also it is configured not to register any > SRV > > records [except for GUID ldap name and CNAME (replication with other > > DCs)], > > and runs no other apps/services, so I've tried to limit its client > > facing > > activity as much as possible. They were willing to dedicate the h/w > > and > > understand the implications, which was good enough for me. I don't > > miss a > > chance to tell them how bad it is and how easy would be to overcome > the > > risk - they usually come back with "it won't be for that much > longer". > > It's > > been 7 years now LOL (the good news is it honestly has not caused > a > > problem on that DC to date). > > > > -DaveC > > > > ________________________________ > > From: ActiveDir-owner@mail.activedir.org > > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer > > Sent: Tuesday, April 15, 2008 10:39 AM > > To: ActiveDir@mail.activedir.org > > Subject: [ActiveDir] Changing MaxPageSize > > Hi All, > > > > yesterday I had stumbled upon a blog post suggesting to resolve the > > 1,000 > > result limit when querying AD, one should increase the MaxPageSize > > value. > > I personally think this is a bad idea, but in response, they asked > how > > one > > can achieve the results without changing maxPageSize. > > > > So I wrote up what I thought could be done here: > > http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize- > > ldap-query-policy/ > > > > Essentially the only ways I could think of would be to utilize the > > Paging > > LDAP control, but if the application doesn't support that, I also > > suggested > > creating tighter, more exacting queries. If you have to change > > maxPageSize > > (and all the issues that go with it), do so on an isolated DC > > specifically > > for the application if it was that critical. > > > > Are there any other ways people have used to achieve this when the > > application is incapable of paging, and they won't budge? > > > > Thanks, > > > > Jef > > > > This email was sent to you by Reuters, the global news and > information > > company. > > To find out more about Reuters visit www.about.reuters.com > > Any views expressed in this message are those of the individual > sender, > > except where the sender specifically states them to be the views of > > Reuters > > Limited. > > Reuters Limited is part of the Reuters Group of companies, of which > > Reuters > > Group PLC is the ultimate parent company. Reuters Group PLC - > > Registered > > office address: The Reuters Building, South Colonnade, Canary Wharf, > > London > > E14 5EP, United Kingdom > > Registered No: 3296375 > > Registered in England and Wales > > > > List info : http://www.activedir.org/List.aspx > > List FAQ : http://www.activedir.org/ListFAQ.aspx > > List archive: http://www.activedir.org/ma/default.aspx > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: http://www.activedir.org/ma/default.aspx > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: http://www.activedir.org/ma/default.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ma/default.aspx This message is confidential, so please treat it appropriately and for its intended purpose only. In particular, if it refers to any technical data, terms or prices not generally available or known, such items are "commercially sensitive information" within the terms of the Freedom of Information Act 2000 and related laws. As it would be prejudicial to RM's commercial interests if these were disclosed, please refrain from doing so. As Internet communications are not secure, please be aware that RM cannot accept responsibility for its contents. Any views or opinions presented are those of the author only and not of RM. If you are not the intended recipient of this e-mail, please accept our apologies and arrange for copies of it to be deleted. For your information, RM may intercept incoming and outgoing email communications. RM Education plc Registered Office: New Mill House, 183 Milton Park, Abingdon, Oxfordshire, OX14 4SE, England Registered Number: 1148594 List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ma/default.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ma/default.aspx

dwells

Posts:39

Shadow

Posts:6

efleis

Posts:15

You are not authorized to post a reply.

Page 2 of 2

Forums >ActiveDir Mail List Archive >List Archives > [ActiveDir] Changing MaxPageSize

ActiveForums 3.7

Syndicate

Friends

List Archives

List Archives