| Author | Messages | |
JefTek
Posts:48
 | | 04/15/2008 10:43 AM |
| Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
| | | |
| DavidCliffe
Posts:10
| | 04/15/2008 11:08 AM |
| Nothing new here. My client has one application which cannot utilize
paging, so we dedicated a DC for it and defined a custom query policy
which applies only to that DC. Also it is configured not to register
any SRV records [except for GUID ldap name and CNAME (replication with
other DCs)], and runs no other apps/services, so I've tried to limit its
client facing activity as much as possible. They were willing to
dedicate the h/w and understand the implications, which was good enough
for me. I don't miss a chance to tell them how bad it is and how easy
would be to overcome the risk - they usually come back with "it won't be
for that much longer". It's been 7 years now LOL (the good news is
it honestly has not caused a problem on that DC to date).
-DaveC
________________________________
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 10:39 AM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve
the 1,000 result limit when querying AD, one should increase the
MaxPageSize value. I personally think this is a bad idea, but in
response, they asked how one can achieve the results without changing
maxPageSize.
So I wrote up what I thought could be done here:
http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-lda
p-query-policy/
Essentially the only ways I could think of would be to utilize
the Paging LDAP control, but if the application doesn't support that, I
also suggested creating tighter, more exacting queries. If you have to
change maxPageSize (and all the issues that go with it), do so on an
isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when
the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender,
except where the sender specifically states them to be the views of Reuters Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company.
Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
| | | |
| dmitrig
Posts:59
| | 04/15/2008 11:18 AM |
| This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to finally fix this one. There was a good proposal on the table... If they do it, then the paging can be hidden at the server side (basically, AD devs do it once, instead of pushing app devs to do it many times in their apps). Those of you with strong influence on MSFT (meaning you are wielding large customers), you can do this too. Then this age-long question will be finally put to rest.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
Sent: Tuesday, April 15, 2008 8:06 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize paging, so we dedicated a DC for it and defined a custom query policy which applies only to that DC. Also it is configured not to register any SRV records [except for GUID ldap name and CNAME (replication with other DCs)], and runs no other apps/services, so I've tried to limit its client facing activity as much as possible. They were willing to dedicate the h/w and understand the implications, which was good enough for me. I don't miss a chance to tell them how bad it is and how easy would be to overcome the risk - they usually come back with "it won't be for that much longer". It's been 7 years now LOL (the good news is it honestly has not caused a problem on that DC to date).
-DaveC
________________________________
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 10:39 AM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
| | | |
| JefTek
Posts:48
| | 04/15/2008 11:33 AM |
| Dmitri,
Can you point me in the direction of what this proposal is? Can you provide a summary for all those here?
When you say "hidden", how do you mean?
JefJef Kazimer ------- http://jeftek.com
From: dmitrig@microsoft.comTo: ActiveDir@mail.activedir.orgDate: Tue, 15 Apr 2008 08:15:57 -0700Subject: RE: [ActiveDir] Changing MaxPageSize
This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to finally fix this one. There was a good proposal on the table… If they do it, then the paging can be hidden at the server side (basically, AD devs do it once, instead of pushing app devs to do it many times in their apps). Those of you with strong influence on MSFT (meaning you are wielding large customers), you can do this too. Then this age-long question will be finally put to rest.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David CliffeSent: Tuesday, April 15, 2008 8:06 AMTo: ActiveDir@mail.activedir.orgSubject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize paging, so we dedicated a DC for it and defined a custom query policy which applies only to that DC. Also it is configured not to register any SRV records [except for GUID ldap name and CNAME (replication with other DCs)], and runs no other apps/services, so I've tried to limit its client facing activity as much as possible. They were willing to dedicate the h/w and understand the implications, which was good enough for me. I don't miss a chance to tell them how bad it is and how easy would be to overcome the risk - they usually come back with "it won't be for that much longer". It's been 7 years now LOL (the good news is it honestly has not caused a problem on that DC to date).
-DaveC
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef KazimerSent: Tuesday, April 15, 2008 10:39 AMTo: ActiveDir@mail.activedir.orgSubject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United KingdomRegistered No: 3296375Registered in England and Wales
| | | |
| dmitrig
Posts:59
| | 04/15/2008 11:48 AM |
| Well, basically the proposal is what I briefly described below...
We would build a paging mechanism in LDAP head on the server. Then, we would support setting MaxPageSize policy to ridiculously large (practically unlimited) values, and instead do paging at the server side (using some internally configured "server" page size). Each page would be computed in a separate transaction, but then all results would be packaged and sent to the client as a single response. Then the practical limit would be the max packet size (10Mb by default).
What this achieves:
* Downlevel apps that don't support paging can continue working.
* Server is not getting killed by long-running transactions.
The downside:
* Paging is now hidden from the app, but we are still paging. This means all the bad transactional properties of paging are now sort of hidden from the app: entries might be missed or reported multiple times (in case they are moved or modified between page requests). There's nothing new here - you can get this today, when paging. But now, we hide it at server side, which makes it less apparent to apps.
I'll be at PG dinner today, we can chat about it.
Dmitri
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:29 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
Can you point me in the direction of what this proposal is? Can you provide a summary for all those here?
When you say "hidden", how do you mean?
Jef
Jef Kazimer
-------
http://jeftek.com
________________________________
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:15:57 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to finally fix this one. There was a good proposal on the table... If they do it, then the paging can be hidden at the server side (basically, AD devs do it once, instead of pushing app devs to do it many times in their apps). Those of you with strong influence on MSFT (meaning you are wielding large customers), you can do this too. Then this age-long question will be finally put to rest.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
Sent: Tuesday, April 15, 2008 8:06 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize paging, so we dedicated a DC for it and defined a custom query policy which applies only to that DC. Also it is configured not to register any SRV records [except for GUID ldap name and CNAME (replication with other DCs)], and runs no other apps/services, so I've tried to limit its client facing activity as much as possible. They were willing to dedicate the h/w and understand the implications, which was good enough for me. I don't miss a chance to tell them how bad it is and how easy would be to overcome the risk - they usually come back with "it won't be for that much longer". It's been 7 years now LOL (the good news is it honestly has not caused a problem on that DC to date).
-DaveC
________________________________
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 10:39 AM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
| | | |
| JefTek
Posts:48
| | 04/15/2008 11:59 AM |
| Dmitri,
This sounds like it would kill any chance of a client who wants to do paging,no? what would happen if a client specified a page size on it's own?
It would still send a large result set to the client which could negatively affect the client as well.
Jef Kazimer ------- http://jeftek.com
From: dmitrig@microsoft.comTo: ActiveDir@mail.activedir.orgDate: Tue, 15 Apr 2008 08:47:04 -0700Subject: RE: [ActiveDir] Changing MaxPageSize
Well, basically the proposal is what I briefly described below…
We would build a paging mechanism in LDAP head on the server. Then, we would support setting MaxPageSize policy to ridiculously large (practically unlimited) values, and instead do paging at the server side (using some internally configured “server” page size). Each page would be computed in a separate transaction, but then all results would be packaged and sent to the client as a single response. Then the practical limit would be the max packet size (10Mb by default).
What this achieves:
· Downlevel apps that don’t support paging can continue working.
· Server is not getting killed by long-running transactions.
The downside:
· Paging is now hidden from the app, but we are still paging. This means all the bad transactional properties of paging are now sort of hidden from the app: entries might be missed or reported multiple times (in case they are moved or modified between page requests). There’s nothing new here – you can get this today, when paging. But now, we hide it at server side, which makes it less apparent to apps.
I’ll be at PG dinner today, we can chat about it.
Dmitri
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef KazimerSent: Tuesday, April 15, 2008 8:29 AMTo: activedir@mail.activedir.orgSubject: RE: [ActiveDir] Changing MaxPageSize
Dmitri, Can you point me in the direction of what this proposal is? Can you provide a summary for all those here? When you say "hidden", how do you mean? JefJef Kazimer ------- http://jeftek.com
From: dmitrig@microsoft.comTo: ActiveDir@mail.activedir.orgDate: Tue, 15 Apr 2008 08:15:57 -0700Subject: RE: [ActiveDir] Changing MaxPageSize
This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to finally fix this one. There was a good proposal on the table… If they do it, then the paging can be hidden at the server side (basically, AD devs do it once, instead of pushing app devs to do it many times in their apps). Those of you with strong influence on MSFT (meaning you are wielding large customers), you can do this too. Then this age-long question will be finally put to rest.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David CliffeSent: Tuesday, April 15, 2008 8:06 AMTo: ActiveDir@mail.activedir.orgSubject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize paging, so we dedicated a DC for it and defined a custom query policy which applies only to that DC. Also it is configured not to register any SRV records [except for GUID ldap name and CNAME (replication with other DCs)], and runs no other apps/services, so I've tried to limit its client facing activity as much as possible. They were willing to dedicate the h/w and understand the implications, which was good enough for me. I don't miss a chance to tell them how bad it is and how easy would be to overcome the risk - they usually come back with "it won't be for that much longer". It's been 7 years now LOL (the good news is it honestly has not caused a problem on that DC to date).
-DaveC
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef KazimerSent: Tuesday, April 15, 2008 10:39 AMTo: ActiveDir@mail.activedir.orgSubject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited. Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United KingdomRegistered No: 3296375Registered in England and Wales
| | | |
| dmitrig
Posts:59
| | 04/15/2008 12:09 PM |
| We would still respect paging requests, and continue working as previously.
The only difference is that we will now *support* setting MaxPageSize to large values. Today, you cannot get more than 1000 entries in a single search query. If we make this change, then you will be.
In a sense, this functionality is supported today in ADSI. ADSI/S.DS hides paging from you. IIRC, you can set the pageSize parameter, but you still get the result as one continuous stream of entries. ADSI queries the server for you and fetches the next page when you run out of entries. My proposal to enable similar functionality on the server.
But we will, of course, continue supporting client-initiated paged searches, just like we do today.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:58 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
This sounds like it would kill any chance of a client who wants to do paging,no? what would happen if a client specified a page size on it's own?
It would still send a large result set to the client which could negatively affect the client as well.
Jef Kazimer
-------
http://jeftek.com
________________________________
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:47:04 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
Well, basically the proposal is what I briefly described below...
We would build a paging mechanism in LDAP head on the server. Then, we would support setting MaxPageSize policy to ridiculously large (practically unlimited) values, and instead do paging at the server side (using some internally configured "server" page size). Each page would be computed in a separate transaction, but then all results would be packaged and sent to the client as a single response. Then the practical limit would be the max packet size (10Mb by default).
What this achieves:
* Downlevel apps that don't support paging can continue working.
* Server is not getting killed by long-running transactions.
The downside:
* Paging is now hidden from the app, but we are still paging. This means all the bad transactional properties of paging are now sort of hidden from the app: entries might be missed or reported multiple times (in case they are moved or modified between page requests). There's nothing new here - you can get this today, when paging. But now, we hide it at server side, which makes it less apparent to apps.
I'll be at PG dinner today, we can chat about it.
Dmitri
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:29 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
Can you point me in the direction of what this proposal is? Can you provide a summary for all those here?
When you say "hidden", how do you mean?
Jef
Jef Kazimer
-------
http://jeftek.com
________________________________
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:15:57 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to finally fix this one. There was a good proposal on the table... If they do it, then the paging can be hidden at the server side (basically, AD devs do it once, instead of pushing app devs to do it many times in their apps). Those of you with strong influence on MSFT (meaning you are wielding large customers), you can do this too. Then this age-long question will be finally put to rest.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
Sent: Tuesday, April 15, 2008 8:06 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize paging, so we dedicated a DC for it and defined a custom query policy which applies only to that DC. Also it is configured not to register any SRV records [except for GUID ldap name and CNAME (replication with other DCs)], and runs no other apps/services, so I've tried to limit its client facing activity as much as possible. They were willing to dedicate the h/w and understand the implications, which was good enough for me. I don't miss a chance to tell them how bad it is and how easy would be to overcome the risk - they usually come back with "it won't be for that much longer". It's been 7 years now LOL (the good news is it honestly has not caused a problem on that DC to date).
-DaveC
________________________________
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 10:39 AM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
| | | |
| JefTek
Posts:48
| | 04/15/2008 12:19 PM |
| Dmitri,
So it would be like an intermediary buffer between the client and server, but on the server itself?
Client -> Server -> Buffer using paging control queues up results -> Returns results to client in one stream ( ? )
If paging is requested, then the server side buffer is not used? Would this consume a lot of memory space for the server with a large amount of searches to hold the results before returning to the client?
I obviously don't know as much as I would like about how the control extension works, but I am curious.
Thanks,
Jef
From: Dmitri Gavrilov
Sent: Tuesday, April 15, 2008 11:07 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
We would still respect paging requests, and continue working as previously.
The only difference is that we will now *support* setting MaxPageSize to large values. Today, you cannot get more than 1000 entries in a single search query. If we make this change, then you will be.
In a sense, this functionality is supported today in ADSI. ADSI/S.DS hides paging from you. IIRC, you can set the pageSize parameter, but you still get the result as one continuous stream of entries. ADSI queries the server for you and fetches the next page when you run out of entries. My proposal to enable similar functionality on the server.
But we will, of course, continue supporting client-initiated paged searches, just like we do today.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:58 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
This sounds like it would kill any chance of a client who wants to do paging,no? what would happen if a client specified a page size on it's own?
It would still send a large result set to the client which could negatively affect the client as well.
Jef Kazimer
-------
http://jeftek.com
--------------------------------------------------------------------------------
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:47:04 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
Well, basically the proposal is what I briefly described below.
We would build a paging mechanism in LDAP head on the server. Then, we would support setting MaxPageSize policy to ridiculously large (practically unlimited) values, and instead do paging at the server side (using some internally configured "server" page size). Each page would be computed in a separate transaction, but then all results would be packaged and sent to the client as a single response. Then the practical limit would be the max packet size (10Mb by default).
What this achieves:
· Downlevel apps that don't support paging can continue working.
· Server is not getting killed by long-running transactions.
The downside:
· Paging is now hidden from the app, but we are still paging. This means all the bad transactional properties of paging are now sort of hidden from the app: entries might be missed or reported multiple times (in case they are moved or modified between page requests). There's nothing new here - you can get this today, when paging. But now, we hide it at server side, which makes it less apparent to apps.
I'll be at PG dinner today, we can chat about it.
Dmitri
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:29 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
Can you point me in the direction of what this proposal is? Can you provide a summary for all those here?
When you say "hidden", how do you mean?
Jef
Jef Kazimer
-------
http://jeftek.com
--------------------------------------------------------------------------------
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:15:57 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to finally fix this one. There was a good proposal on the table. If they do it, then the paging can be hidden at the server side (basically, AD devs do it once, instead of pushing app devs to do it many times in their apps). Those of you with strong influence on MSFT (meaning you are wielding large customers), you can do this too. Then this age-long question will be finally put to rest.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
Sent: Tuesday, April 15, 2008 8:06 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize paging, so we dedicated a DC for it and defined a custom query policy which applies only to that DC. Also it is configured not to register any SRV records [except for GUID ldap name and CNAME (replication with other DCs)], and runs no other apps/services, so I've tried to limit its client facing activity as much as possible. They were willing to dedicate the h/w and understand the implications, which was good enough for me. I don't miss a chance to tell them how bad it is and how easy would be to overcome the risk - they usually come back with "it won't be for that much longer". It's been 7 years now LOL (the good news is it honestly has not caused a problem on that DC to date).
-DaveC
------------------------------------------------------------------------------
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 10:39 AM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
| | | |
| dmitrig
Posts:59
| | 04/15/2008 12:29 PM |
| Yes, something like this.
Yes, it will consume server memory, while the search is being run. But there's nothing new here - you can already do this today, if you request lots of large objects. We have proper controls (max output packet size, max query duration, etc) to protect server.
Yes, if client is paging, then we won't use this mechanism (double-paging looks scary to me). Client's page sizes would have to be under the "server maxPageSize" value.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 9:17 AM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] Changing MaxPageSize
Dmitri,
So it would be like an intermediary buffer between the client and server, but on the server itself?
Client -> Server -> Buffer using paging control queues up results -> Returns results to client in one stream ( ? )
If paging is requested, then the server side buffer is not used? Would this consume a lot of memory space for the server with a large amount of searches to hold the results before returning to the client?
I obviously don't know as much as I would like about how the control extension works, but I am curious.
Thanks,
Jef
From: Dmitri Gavrilov<mailto:dmitrig@microsoft.com>
Sent: Tuesday, April 15, 2008 11:07 AM
To: ActiveDir@mail.activedir.org<mailto:ActiveDir@mail.activedir.org>
Subject: RE: [ActiveDir] Changing MaxPageSize
We would still respect paging requests, and continue working as previously.
The only difference is that we will now *support* setting MaxPageSize to large values. Today, you cannot get more than 1000 entries in a single search query. If we make this change, then you will be.
In a sense, this functionality is supported today in ADSI. ADSI/S.DS hides paging from you. IIRC, you can set the pageSize parameter, but you still get the result as one continuous stream of entries. ADSI queries the server for you and fetches the next page when you run out of entries. My proposal to enable similar functionality on the server.
But we will, of course, continue supporting client-initiated paged searches, just like we do today.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:58 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
This sounds like it would kill any chance of a client who wants to do paging,no? what would happen if a client specified a page size on it's own?
It would still send a large result set to the client which could negatively affect the client as well.
Jef Kazimer
-------
http://jeftek.com
________________________________
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:47:04 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
Well, basically the proposal is what I briefly described below...
We would build a paging mechanism in LDAP head on the server. Then, we would support setting MaxPageSize policy to ridiculously large (practically unlimited) values, and instead do paging at the server side (using some internally configured "server" page size). Each page would be computed in a separate transaction, but then all results would be packaged and sent to the client as a single response. Then the practical limit would be the max packet size (10Mb by default).
What this achieves:
* Downlevel apps that don't support paging can continue working.
* Server is not getting killed by long-running transactions.
The downside:
* Paging is now hidden from the app, but we are still paging. This means all the bad transactional properties of paging are now sort of hidden from the app: entries might be missed or reported multiple times (in case they are moved or modified between page requests). There's nothing new here - you can get this today, when paging. But now, we hide it at server side, which makes it less apparent to apps.
I'll be at PG dinner today, we can chat about it.
Dmitri
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:29 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
Can you point me in the direction of what this proposal is? Can you provide a summary for all those here?
When you say "hidden", how do you mean?
Jef
Jef Kazimer
-------
http://jeftek.com
________________________________
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:15:57 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to finally fix this one. There was a good proposal on the table... If they do it, then the paging can be hidden at the server side (basically, AD devs do it once, instead of pushing app devs to do it many times in their apps). Those of you with strong influence on MSFT (meaning you are wielding large customers), you can do this too. Then this age-long question will be finally put to rest.
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
Sent: Tuesday, April 15, 2008 8:06 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize paging, so we dedicated a DC for it and defined a custom query policy which applies only to that DC. Also it is configured not to register any SRV records [except for GUID ldap name and CNAME (replication with other DCs)], and runs no other apps/services, so I've tried to limit its client facing activity as much as possible. They were willing to dedicate the h/w and understand the implications, which was good enough for me. I don't miss a chance to tell them how bad it is and how easy would be to overcome the risk - they usually come back with "it won't be for that much longer". It's been 7 years now LOL (the good news is it honestly has not caused a problem on that DC to date).
-DaveC
________________________________
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 10:39 AM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 result limit when querying AD, one should increase the MaxPageSize value. I personally think this is a bad idea, but in response, they asked how one can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here: http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging LDAP control, but if the application doesn't support that, I also suggested creating tighter, more exacting queries. If you have to change maxPageSize (and all the issues that go with it), do so on an isolated DC specifically for the application if it was that critical.
Are there any other ways people have used to achieve this when the application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters Group PLC is the ultimate parent company. Reuters Group PLC - Registered office address: The Reuters Building, South Colonnade, Canary Wharf, London E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
| | | |
| joe
Posts:84
| | 04/15/2008 1:10 PM |
| I'll make sure this gets brought up during one of the chalk talks unless I
happen to not be there, but if I can't, I'll ask someone else to bring it
up.
We did just discuss AD/LDS on Vista. 
Joe K.
----- Original Message -----
From: "Dmitri Gavrilov" <dmitrig@microsoft.com>
To: <ActiveDir@mail.activedir.org>
Sent: Tuesday, April 15, 2008 11:28 AM
Subject: RE: [ActiveDir] Changing MaxPageSize
Yes, something like this.
Yes, it will consume server memory, while the search is being run. But
there's nothing new here - you can already do this today, if you request
lots of large objects. We have proper controls (max output packet size, max
query duration, etc) to protect server.
Yes, if client is paging, then we won't use this mechanism (double-paging
looks scary to me). Client's page sizes would have to be under the "server
maxPageSize" value.
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 9:17 AM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] Changing MaxPageSize
Dmitri,
So it would be like an intermediary buffer between the client and server,
but on the server itself?
Client -> Server -> Buffer using paging control queues up results -> Returns
results to client in one stream ( ? )
If paging is requested, then the server side buffer is not used? Would
this consume a lot of memory space for the server with a large amount of
searches to hold the results before returning to the client?
I obviously don't know as much as I would like about how the control
extension works, but I am curious.
Thanks,
Jef
From: Dmitri Gavrilov<mailto:dmitrig@microsoft.com>
Sent: Tuesday, April 15, 2008 11:07 AM
To: ActiveDir@mail.activedir.org<mailto:ActiveDir@mail.activedir.org>
Subject: RE: [ActiveDir] Changing MaxPageSize
We would still respect paging requests, and continue working as previously.
The only difference is that we will now *support* setting MaxPageSize to
large values. Today, you cannot get more than 1000 entries in a single
search query. If we make this change, then you will be.
In a sense, this functionality is supported today in ADSI. ADSI/S.DS hides
paging from you. IIRC, you can set the pageSize parameter, but you still get
the result as one continuous stream of entries. ADSI queries the server for
you and fetches the next page when you run out of entries. My proposal to
enable similar functionality on the server.
But we will, of course, continue supporting client-initiated paged searches,
just like we do today.
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:58 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
This sounds like it would kill any chance of a client who wants to do
paging,no? what would happen if a client specified a page size on it's own?
It would still send a large result set to the client which could negatively
affect the client as well.
Jef Kazimer
-------
http://jeftek.com
________________________________
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:47:04 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
Well, basically the proposal is what I briefly described below...
We would build a paging mechanism in LDAP head on the server. Then, we would
support setting MaxPageSize policy to ridiculously large (practically
unlimited) values, and instead do paging at the server side (using some
internally configured "server" page size). Each page would be computed in a
separate transaction, but then all results would be packaged and sent to the
client as a single response. Then the practical limit would be the max
packet size (10Mb by default).
What this achieves:
* Downlevel apps that don't support paging can continue working.
* Server is not getting killed by long-running transactions.
The downside:
* Paging is now hidden from the app, but we are still paging. This
means all the bad transactional properties of paging are now sort of hidden
from the app: entries might be missed or reported multiple times (in case
they are moved or modified between page requests). There's nothing new
here - you can get this today, when paging. But now, we hide it at server
side, which makes it less apparent to apps.
I'll be at PG dinner today, we can chat about it.
Dmitri
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 8:29 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Dmitri,
Can you point me in the direction of what this proposal is? Can you
provide a summary for all those here?
When you say "hidden", how do you mean?
Jef
Jef Kazimer
-------
http://jeftek.com
________________________________
From: dmitrig@microsoft.com
To: ActiveDir@mail.activedir.org
Date: Tue, 15 Apr 2008 08:15:57 -0700
Subject: RE: [ActiveDir] Changing MaxPageSize
This serves as a very appropriate and timely reminder for my earlier post:
Those of you that are coming to MVP summit: I implore you to push AD team to
finally fix this one. There was a good proposal on the table... If they do
it, then the paging can be hidden at the server side (basically, AD devs do
it once, instead of pushing app devs to do it many times in their apps).
Those of you with strong influence on MSFT (meaning you are wielding large
customers), you can do this too. Then this age-long question will be finally
put to rest.
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
Sent: Tuesday, April 15, 2008 8:06 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
Nothing new here. My client has one application which cannot utilize
paging, so we dedicated a DC for it and defined a custom query policy which
applies only to that DC. Also it is configured not to register any SRV
records [except for GUID ldap name and CNAME (replication with other DCs)],
and runs no other apps/services, so I've tried to limit its client facing
activity as much as possible. They were willing to dedicate the h/w and
understand the implications, which was good enough for me. I don't miss a
chance to tell them how bad it is and how easy would be to overcome the
risk - they usually come back with "it won't be for that much longer". It's
been 7 years now LOL (the good news is it honestly has not caused a
problem on that DC to date).
-DaveC
________________________________
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
Sent: Tuesday, April 15, 2008 10:39 AM
To: ActiveDir@mail.activedir.org
Subject: [ActiveDir] Changing MaxPageSize
Hi All,
yesterday I had stumbled upon a blog post suggesting to resolve the 1,000
result limit when querying AD, one should increase the MaxPageSize value.
I personally think this is a bad idea, but in response, they asked how one
can achieve the results without changing maxPageSize.
So I wrote up what I thought could be done here:
http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/
Essentially the only ways I could think of would be to utilize the Paging
LDAP control, but if the application doesn't support that, I also suggested
creating tighter, more exacting queries. If you have to change maxPageSize
(and all the issues that go with it), do so on an isolated DC specifically
for the application if it was that critical.
Are there any other ways people have used to achieve this when the
application is incapable of paging, and they won't budge?
Thanks,
Jef
This email was sent to you by Reuters, the global news and information
company.
To find out more about Reuters visit www.about.reuters.com
Any views expressed in this message are those of the individual sender,
except where the sender specifically states them to be the views of Reuters
Limited.
Reuters Limited is part of the Reuters Group of companies, of which Reuters
Group PLC is the ultimate parent company. Reuters Group PLC - Registered
office address: The Reuters Building, South Colonnade, Canary Wharf, London
E14 5EP, United Kingdom
Registered No: 3296375
Registered in England and Wales
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ma/default.aspx
| | | |
| JefTek
Posts:48
| | 04/15/2008 1:30 PM |
| Joe,
That brings a question to mind...
Does anyone know of any applications which ship with ADLDS to be installed on a client to be used as a data store? I have always assumed that is why they have Windows XP support (outside of development), so that an application could be packaged with it, but I have never heard of any.
JefJef Kazimer ------- http://jeftek.com
> From: joe@joekaplan.net> To: ActiveDir@mail.activedir.org> Subject: Re: [ActiveDir] Changing MaxPageSize> Date: Tue, 15 Apr 2008 12:05:22 -0500> > I'll make sure this gets brought up during one of the chalk talks unless I > happen to not be there, but if I can't, I'll ask someone else to bring it > up.> > We did just discuss AD/LDS on Vista. > > Joe K.> > ----- Original Message ----- > From: "Dmitri Gavrilov" <dmitrig@microsoft.com>> To: <ActiveDir@mail.activedir.org>> Sent: Tuesday, April 15, 2008 11:28 AM> Subject: RE: [ActiveDir] Changing MaxPageSize> > > Yes, something like this.> Yes, it will consume server memory, while the search is being run. But > there's nothing new here - you can already do this today, if you request > lots of large objects. We have proper controls (max output packet size, max > query duration, etc) to protect server.> > Yes, if client is paging, then we won't use this mechanism (double-paging > looks scary to me). Client's page sizes would have to be under the "server > maxPageSize" value.> > From: ActiveDir-owner@mail.activedir.org > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer> Sent: Tuesday, April 15, 2008 9:17 AM> To: ActiveDir@mail.activedir.org> Subject: Re: [ActiveDir] Changing MaxPageSize> > Dmitri,> > So it would be like an intermediary buffer between the client and server, > but on the server itself?> > Client -> Server -> Buffer using paging control queues up results -> Returns > results to client in one stream ( ? )> > If paging is requested, then the server side buffer is not used? Would > this consume a lot of memory space for the server with a large amount of > searches to hold the results before returning to the client?> > I obviously don't know as much as I would like about how the control > extension works, but I am curious.> > Thanks,> Jef> > > > > > From: Dmitri Gavrilov<mailto:dmitrig@microsoft.com>> Sent: Tuesday, April 15, 2008 11:07 AM> To: ActiveDir@mail.activedir.org<mailto:ActiveDir@mail.activedir.org>> Subject: RE: [ActiveDir] Changing MaxPageSize> > We would still respect paging requests, and continue working as previously.> > The only difference is that we will now *support* setting MaxPageSize to > large values. Today, you cannot get more than 1000 entries in a single > search query. If we make this change, then you will be.> > In a sense, this functionality is supported today in ADSI. ADSI/S.DS hides > paging from you. IIRC, you can set the pageSize parameter, but you still get > the result as one continuous stream of entries. ADSI queries the server for > you and fetches the next page when you run out of entries. My proposal to > enable similar functionality on the server.> > But we will, of course, continue supporting client-initiated paged searches, > just like we do today.> > From: ActiveDir-owner@mail.activedir.org > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer> Sent: Tuesday, April 15, 2008 8:58 AM> To: activedir@mail.activedir.org> Subject: RE: [ActiveDir] Changing MaxPageSize> > Dmitri,> > This sounds like it would kill any chance of a client who wants to do > paging,no? what would happen if a client specified a page size on it's own?> > It would still send a large result set to the client which could negatively > affect the client as well.> > > > Jef Kazimer> -------> http://jeftek.com> ________________________________> From: dmitrig@microsoft.com> To: ActiveDir@mail.activedir.org> Date: Tue, 15 Apr 2008 08:47:04 -0700> Subject: RE: [ActiveDir] Changing MaxPageSize> Well, basically the proposal is what I briefly described below...> > We would build a paging mechanism in LDAP head on the server. Then, we would > support setting MaxPageSize policy to ridiculously large (practically > unlimited) values, and instead do paging at the server side (using some > internally configured "server" page size). Each page would be computed in a > separate transaction, but then all results would be packaged and sent to the > client as a single response. Then the practical limit would be the max > packet size (10Mb by default).> > What this achieves:> * Downlevel apps that don't support paging can continue working.> * Server is not getting killed by long-running transactions.> > The downside:> * Paging is now hidden from the app, but we are still paging. This > means all the bad transactional properties of paging are now sort of hidden > from the app: entries might be missed or reported multiple times (in case > they are moved or modified between page requests). There's nothing new > here - you can get this today, when paging. But now, we hide it at server > side, which makes it less apparent to apps.> > I'll be at PG dinner today, we can chat about it.> > Dmitri> > > From: ActiveDir-owner@mail.activedir.org > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer> Sent: Tuesday, April 15, 2008 8:29 AM> To: activedir@mail.activedir.org> Subject: RE: [ActiveDir] Changing MaxPageSize> > Dmitri,> > Can you point me in the direction of what this proposal is? Can you > provide a summary for all those here?> > When you say "hidden", how do you mean?> > Jef> > Jef Kazimer> -------> http://jeftek.com> ________________________________> From: dmitrig@microsoft.com> To: ActiveDir@mail.activedir.org> Date: Tue, 15 Apr 2008 08:15:57 -0700> Subject: RE: [ActiveDir] Changing MaxPageSize> This serves as a very appropriate and timely reminder for my earlier post:> > Those of you that are coming to MVP summit: I implore you to push AD team to > finally fix this one. There was a good proposal on the table... If they do > it, then the paging can be hidden at the server side (basically, AD devs do > it once, instead of pushing app devs to do it many times in their apps). > Those of you with strong influence on MSFT (meaning you are wielding large > customers), you can do this too. Then this age-long question will be finally > put to rest.> > > From: ActiveDir-owner@mail.activedir.org > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe> Sent: Tuesday, April 15, 2008 8:06 AM> To: ActiveDir@mail.activedir.org> Subject: RE: [ActiveDir] Changing MaxPageSize> > Nothing new here. My client has one application which cannot utilize > paging, so we dedicated a DC for it and defined a custom query policy which > applies only to that DC. Also it is configured not to register any SRV > records [except for GUID ldap name and CNAME (replication with other DCs)], > and runs no other apps/services, so I've tried to limit its client facing > activity as much as possible. They were willing to dedicate the h/w and > understand the implications, which was good enough for me. I don't miss a > chance to tell them how bad it is and how easy would be to overcome the > risk - they usually come back with "it won't be for that much longer". It's > been 7 years now LOL (the good news is it honestly has not caused a > problem on that DC to date).> > -DaveC> > ________________________________> From: ActiveDir-owner@mail.activedir.org > [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer> Sent: Tuesday, April 15, 2008 10:39 AM> To: ActiveDir@mail.activedir.org> Subject: [ActiveDir] Changing MaxPageSize> Hi All,> > yesterday I had stumbled upon a blog post suggesting to resolve the 1,000 > result limit when querying AD, one should increase the MaxPageSize value. > I personally think this is a bad idea, but in response, they asked how one > can achieve the results without changing maxPageSize.> > So I wrote up what I thought could be done here: > http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-ldap-query-policy/> > Essentially the only ways I could think of would be to utilize the Paging > LDAP control, but if the application doesn't support that, I also suggested > creating tighter, more exacting queries. If you have to change maxPageSize > (and all the issues that go with it), do so on an isolated DC specifically > for the application if it was that critical.> > Are there any other ways people have used to achieve this when the > application is incapable of paging, and they won't budge?> > Thanks,> > Jef> > This email was sent to you by Reuters, the global news and information > company.> To find out more about Reuters visit www.about.reuters.com> Any views expressed in this message are those of the individual sender, > except where the sender specifically states them to be the views of Reuters > Limited.> Reuters Limited is part of the Reuters Group of companies, of which Reuters > Group PLC is the ultimate parent company. Reuters Group PLC - Registered > office address: The Reuters Building, South Colonnade, Canary Wharf, London > E14 5EP, United Kingdom> Registered No: 3296375> Registered in England and Wales> > List info : http://www.activedir.org/List.aspx> List FAQ : http://www.activedir.org/ListFAQ.aspx> List archive: http://www.activedir.org/ma/default.aspx
| | | |
| barkills
Posts:10
| | 04/15/2008 3:02 PM |
| I'm somewhat new to this list, but very familiar with both AD and LDAP (see http://www.amazon.com/LDAP-Directories-Explained-Introduction-Independent/dp/020178792X/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1208285140&sr=8-1).
I don't understand the strategy of this proposal. Here's what I'm hearing:
-Client has software which doesn't support the LDAP paging control that's been around for years and which is supported by lots of ldap-based software.
-Instead of getting the software vendor to implement and support that well-supported LDAP paging control, there's an idea that the client would like Microsoft to implement a *new* LDAP control which circumvents the maxpagesize that the server enforces on all LDAP search requests.
**If the software vendor won't support the well-known LDAP paging control, how are you going to get them to support a brand-new LDAP paging control?**
Put another way, all LDAP searches of AD are currently subject to the server-side maxpagesize limit (this is a common thing to do in LDAP directories). There are many good reasons for such a limit to be in place, so you have little to no chance of convincing Microsoft (or many of us) that completely removing that limit is a good idea. So this means you need an LDAP control extension to allow certain requestors to circumvent that limit. But if you have such a control, then anyone can call it which means you've lost the value of having a limit at all. So then your control needs some authorization logic built into it.
> -----Original Message-----
> From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-
> owner@mail.activedir.org] On Behalf Of Joe Kaplan
> Sent: Tuesday, April 15, 2008 10:05 AM
> To: ActiveDir@mail.activedir.org
> Subject: Re: [ActiveDir] Changing MaxPageSize
>
> I'll make sure this gets brought up during one of the chalk talks
> unless I
> happen to not be there, but if I can't, I'll ask someone else to bring
> it
> up.
>
> We did just discuss AD/LDS on Vista.
>
> Joe K.
>
> ----- Original Message -----
> From: "Dmitri Gavrilov" <dmitrig@microsoft.com>
> To: <ActiveDir@mail.activedir.org>
> Sent: Tuesday, April 15, 2008 11:28 AM
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
>
> Yes, something like this.
> Yes, it will consume server memory, while the search is being run. But
> there's nothing new here - you can already do this today, if you
> request
> lots of large objects. We have proper controls (max output packet size,
> max
> query duration, etc) to protect server.
>
> Yes, if client is paging, then we won't use this mechanism (double-
> paging
> looks scary to me). Client's page sizes would have to be under the
> "server
> maxPageSize" value.
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 9:17 AM
> To: ActiveDir@mail.activedir.org
> Subject: Re: [ActiveDir] Changing MaxPageSize
>
> Dmitri,
>
> So it would be like an intermediary buffer between the client and
> server,
> but on the server itself?
>
> Client -> Server -> Buffer using paging control queues up results ->
> Returns
> results to client in one stream ( ? )
>
> If paging is requested, then the server side buffer is not used?
> Would
> this consume a lot of memory space for the server with a large amount
> of
> searches to hold the results before returning to the client?
>
> I obviously don't know as much as I would like about how the control
> extension works, but I am curious.
>
> Thanks,
> Jef
>
>
>
>
>
> From: Dmitri Gavrilov<mailto:dmitrig@microsoft.com>
> Sent: Tuesday, April 15, 2008 11:07 AM
> To: ActiveDir@mail.activedir.org<mailto:ActiveDir@mail.activedir.org>
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> We would still respect paging requests, and continue working as
> previously.
>
> The only difference is that we will now *support* setting MaxPageSize
> to
> large values. Today, you cannot get more than 1000 entries in a single
> search query. If we make this change, then you will be.
>
> In a sense, this functionality is supported today in ADSI. ADSI/S.DS
> hides
> paging from you. IIRC, you can set the pageSize parameter, but you
> still get
> the result as one continuous stream of entries. ADSI queries the server
> for
> you and fetches the next page when you run out of entries. My proposal
> to
> enable similar functionality on the server.
>
> But we will, of course, continue supporting client-initiated paged
> searches,
> just like we do today.
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 8:58 AM
> To: activedir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> Dmitri,
>
> This sounds like it would kill any chance of a client who wants to do
> paging,no? what would happen if a client specified a page size on it's
> own?
>
> It would still send a large result set to the client which could
> negatively
> affect the client as well.
>
>
>
> Jef Kazimer
> -------
> http://jeftek.com
> ________________________________
> From: dmitrig@microsoft.com
> To: ActiveDir@mail.activedir.org
> Date: Tue, 15 Apr 2008 08:47:04 -0700
> Subject: RE: [ActiveDir] Changing MaxPageSize
> Well, basically the proposal is what I briefly described below...
>
> We would build a paging mechanism in LDAP head on the server. Then, we
> would
> support setting MaxPageSize policy to ridiculously large (practically
> unlimited) values, and instead do paging at the server side (using some
> internally configured "server" page size). Each page would be computed
> in a
> separate transaction, but then all results would be packaged and sent
> to the
> client as a single response. Then the practical limit would be the max
> packet size (10Mb by default).
>
> What this achieves:
> * Downlevel apps that don't support paging can continue
> working.
> * Server is not getting killed by long-running transactions.
>
> The downside:
> * Paging is now hidden from the app, but we are still paging.
> This
> means all the bad transactional properties of paging are now sort of
> hidden
> from the app: entries might be missed or reported multiple times (in
> case
> they are moved or modified between page requests). There's nothing new
> here - you can get this today, when paging. But now, we hide it at
> server
> side, which makes it less apparent to apps.
>
> I'll be at PG dinner today, we can chat about it.
>
> Dmitri
>
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 8:29 AM
> To: activedir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> Dmitri,
>
> Can you point me in the direction of what this proposal is? Can you
> provide a summary for all those here?
>
> When you say "hidden", how do you mean?
>
> Jef
>
> Jef Kazimer
> -------
> http://jeftek.com
> ________________________________
> From: dmitrig@microsoft.com
> To: ActiveDir@mail.activedir.org
> Date: Tue, 15 Apr 2008 08:15:57 -0700
> Subject: RE: [ActiveDir] Changing MaxPageSize
> This serves as a very appropriate and timely reminder for my earlier
> post:
>
> Those of you that are coming to MVP summit: I implore you to push AD
> team to
> finally fix this one. There was a good proposal on the table... If they
> do
> it, then the paging can be hidden at the server side (basically, AD
> devs do
> it once, instead of pushing app devs to do it many times in their
> apps).
> Those of you with strong influence on MSFT (meaning you are wielding
> large
> customers), you can do this too. Then this age-long question will be
> finally
> put to rest.
>
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
> Sent: Tuesday, April 15, 2008 8:06 AM
> To: ActiveDir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> Nothing new here. My client has one application which cannot utilize
> paging, so we dedicated a DC for it and defined a custom query policy
> which
> applies only to that DC. Also it is configured not to register any SRV
> records [except for GUID ldap name and CNAME (replication with other
> DCs)],
> and runs no other apps/services, so I've tried to limit its client
> facing
> activity as much as possible. They were willing to dedicate the h/w
> and
> understand the implications, which was good enough for me. I don't
> miss a
> chance to tell them how bad it is and how easy would be to overcome the
> risk - they usually come back with "it won't be for that much longer".
> It's
> been 7 years now LOL (the good news is it honestly has not caused a
> problem on that DC to date).
>
> -DaveC
>
> ________________________________
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 10:39 AM
> To: ActiveDir@mail.activedir.org
> Subject: [ActiveDir] Changing MaxPageSize
> Hi All,
>
> yesterday I had stumbled upon a blog post suggesting to resolve the
> 1,000
> result limit when querying AD, one should increase the MaxPageSize
> value.
> I personally think this is a bad idea, but in response, they asked how
> one
> can achieve the results without changing maxPageSize.
>
> So I wrote up what I thought could be done here:
> http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-
> ldap-query-policy/
>
> Essentially the only ways I could think of would be to utilize the
> Paging
> LDAP control, but if the application doesn't support that, I also
> suggested
> creating tighter, more exacting queries. If you have to change
> maxPageSize
> (and all the issues that go with it), do so on an isolated DC
> specifically
> for the application if it was that critical.
>
> Are there any other ways people have used to achieve this when the
> application is incapable of paging, and they won't budge?
>
> Thanks,
>
> Jef
>
> This email was sent to you by Reuters, the global news and information
> company.
> To find out more about Reuters visit www.about.reuters.com
> Any views expressed in this message are those of the individual sender,
> except where the sender specifically states them to be the views of
> Reuters
> Limited.
> Reuters Limited is part of the Reuters Group of companies, of which
> Reuters
> Group PLC is the ultimate parent company. Reuters Group PLC -
> Registered
> office address: The Reuters Building, South Colonnade, Canary Wharf,
> London
> E14 5EP, United Kingdom
> Registered No: 3296375
> Registered in England and Wales
>
> List info : http://www.activedir.org/List.aspx
> List FAQ : http://www.activedir.org/ListFAQ.aspx
> List archive: http://www.activedir.org/ma/default.aspx
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ma/default.aspx
| | | |
| dmitrig
Posts:59
| | 04/15/2008 3:27 PM |
| If this proposal is implemented, then no changes are required to client apps. They don't need to pass any new controls.
Here's the scenario:
You have an old ldap app that does not know how to page.
Your domain grows, and the app stops working because it can only get 1000 users max.
What do you do? Currently, you only have two options:
1. Try to fix the app.
2. Increase MaxPageSize.
1 is often impossible. 2 is generally frowned upon (e.g. in this thread). You are out of options!
I am trying to propose a new option, that would allow to do paging at server, while presenting the search results to the client as a regular ldap search result.
-----Original Message-----
From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brian Arkills
Sent: Tuesday, April 15, 2008 11:58 AM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] Changing MaxPageSize
I'm somewhat new to this list, but very familiar with both AD and LDAP (see http://www.amazon.com/LDAP-Directories-Explained-Introduction-Independent/dp/020178792X/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1208285140&sr=8-1).
I don't understand the strategy of this proposal. Here's what I'm hearing:
-Client has software which doesn't support the LDAP paging control that's been around for years and which is supported by lots of ldap-based software.
-Instead of getting the software vendor to implement and support that well-supported LDAP paging control, there's an idea that the client would like Microsoft to implement a *new* LDAP control which circumvents the maxpagesize that the server enforces on all LDAP search requests.
**If the software vendor won't support the well-known LDAP paging control, how are you going to get them to support a brand-new LDAP paging control?**
Put another way, all LDAP searches of AD are currently subject to the server-side maxpagesize limit (this is a common thing to do in LDAP directories). There are many good reasons for such a limit to be in place, so you have little to no chance of convincing Microsoft (or many of us) that completely removing that limit is a good idea. So this means you need an LDAP control extension to allow certain requestors to circumvent that limit. But if you have such a control, then anyone can call it which means you've lost the value of having a limit at all. So then your control needs some authorization logic built into it.
> -----Original Message-----
> From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-
> owner@mail.activedir.org] On Behalf Of Joe Kaplan
> Sent: Tuesday, April 15, 2008 10:05 AM
> To: ActiveDir@mail.activedir.org
> Subject: Re: [ActiveDir] Changing MaxPageSize
>
> I'll make sure this gets brought up during one of the chalk talks
> unless I
> happen to not be there, but if I can't, I'll ask someone else to bring
> it
> up.
>
> We did just discuss AD/LDS on Vista.
>
> Joe K.
>
> ----- Original Message -----
> From: "Dmitri Gavrilov" <dmitrig@microsoft.com>
> To: <ActiveDir@mail.activedir.org>
> Sent: Tuesday, April 15, 2008 11:28 AM
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
>
> Yes, something like this.
> Yes, it will consume server memory, while the search is being run. But
> there's nothing new here - you can already do this today, if you
> request
> lots of large objects. We have proper controls (max output packet size,
> max
> query duration, etc) to protect server.
>
> Yes, if client is paging, then we won't use this mechanism (double-
> paging
> looks scary to me). Client's page sizes would have to be under the
> "server
> maxPageSize" value.
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 9:17 AM
> To: ActiveDir@mail.activedir.org
> Subject: Re: [ActiveDir] Changing MaxPageSize
>
> Dmitri,
>
> So it would be like an intermediary buffer between the client and
> server,
> but on the server itself?
>
> Client -> Server -> Buffer using paging control queues up results ->
> Returns
> results to client in one stream ( ? )
>
> If paging is requested, then the server side buffer is not used?
> Would
> this consume a lot of memory space for the server with a large amount
> of
> searches to hold the results before returning to the client?
>
> I obviously don't know as much as I would like about how the control
> extension works, but I am curious.
>
> Thanks,
> Jef
>
>
>
>
>
> From: Dmitri Gavrilov<mailto:dmitrig@microsoft.com>
> Sent: Tuesday, April 15, 2008 11:07 AM
> To: ActiveDir@mail.activedir.org<mailto:ActiveDir@mail.activedir.org>
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> We would still respect paging requests, and continue working as
> previously.
>
> The only difference is that we will now *support* setting MaxPageSize
> to
> large values. Today, you cannot get more than 1000 entries in a single
> search query. If we make this change, then you will be.
>
> In a sense, this functionality is supported today in ADSI. ADSI/S.DS
> hides
> paging from you. IIRC, you can set the pageSize parameter, but you
> still get
> the result as one continuous stream of entries. ADSI queries the server
> for
> you and fetches the next page when you run out of entries. My proposal
> to
> enable similar functionality on the server.
>
> But we will, of course, continue supporting client-initiated paged
> searches,
> just like we do today.
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 8:58 AM
> To: activedir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> Dmitri,
>
> This sounds like it would kill any chance of a client who wants to do
> paging,no? what would happen if a client specified a page size on it's
> own?
>
> It would still send a large result set to the client which could
> negatively
> affect the client as well.
>
>
>
> Jef Kazimer
> -------
> http://jeftek.com
> ________________________________
> From: dmitrig@microsoft.com
> To: ActiveDir@mail.activedir.org
> Date: Tue, 15 Apr 2008 08:47:04 -0700
> Subject: RE: [ActiveDir] Changing MaxPageSize
> Well, basically the proposal is what I briefly described below...
>
> We would build a paging mechanism in LDAP head on the server. Then, we
> would
> support setting MaxPageSize policy to ridiculously large (practically
> unlimited) values, and instead do paging at the server side (using some
> internally configured "server" page size). Each page would be computed
> in a
> separate transaction, but then all results would be packaged and sent
> to the
> client as a single response. Then the practical limit would be the max
> packet size (10Mb by default).
>
> What this achieves:
> * Downlevel apps that don't support paging can continue
> working.
> * Server is not getting killed by long-running transactions.
>
> The downside:
> * Paging is now hidden from the app, but we are still paging.
> This
> means all the bad transactional properties of paging are now sort of
> hidden
> from the app: entries might be missed or reported multiple times (in
> case
> they are moved or modified between page requests). There's nothing new
> here - you can get this today, when paging. But now, we hide it at
> server
> side, which makes it less apparent to apps.
>
> I'll be at PG dinner today, we can chat about it.
>
> Dmitri
>
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 8:29 AM
> To: activedir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> Dmitri,
>
> Can you point me in the direction of what this proposal is? Can you
> provide a summary for all those here?
>
> When you say "hidden", how do you mean?
>
> Jef
>
> Jef Kazimer
> -------
> http://jeftek.com
> ________________________________
> From: dmitrig@microsoft.com
> To: ActiveDir@mail.activedir.org
> Date: Tue, 15 Apr 2008 08:15:57 -0700
> Subject: RE: [ActiveDir] Changing MaxPageSize
> This serves as a very appropriate and timely reminder for my earlier
> post:
>
> Those of you that are coming to MVP summit: I implore you to push AD
> team to
> finally fix this one. There was a good proposal on the table... If they
> do
> it, then the paging can be hidden at the server side (basically, AD
> devs do
> it once, instead of pushing app devs to do it many times in their
> apps).
> Those of you with strong influence on MSFT (meaning you are wielding
> large
> customers), you can do this too. Then this age-long question will be
> finally
> put to rest.
>
>
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of David Cliffe
> Sent: Tuesday, April 15, 2008 8:06 AM
> To: ActiveDir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> Nothing new here. My client has one application which cannot utilize
> paging, so we dedicated a DC for it and defined a custom query policy
> which
> applies only to that DC. Also it is configured not to register any SRV
> records [except for GUID ldap name and CNAME (replication with other
> DCs)],
> and runs no other apps/services, so I've tried to limit its client
> facing
> activity as much as possible. They were willing to dedicate the h/w
> and
> understand the implications, which was good enough for me. I don't
> miss a
> chance to tell them how bad it is and how easy would be to overcome the
> risk - they usually come back with "it won't be for that much longer".
> It's
> been 7 years now LOL (the good news is it honestly has not caused a
> problem on that DC to date).
>
> -DaveC
>
> ________________________________
> From: ActiveDir-owner@mail.activedir.org
> [mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Jef Kazimer
> Sent: Tuesday, April 15, 2008 10:39 AM
> To: ActiveDir@mail.activedir.org
> Subject: [ActiveDir] Changing MaxPageSize
> Hi All,
>
> yesterday I had stumbled upon a blog post suggesting to resolve the
> 1,000
> result limit when querying AD, one should increase the MaxPageSize
> value.
> I personally think this is a bad idea, but in response, they asked how
> one
> can achieve the results without changing maxPageSize.
>
> So I wrote up what I thought could be done here:
> http://jeftek.com/iam/activedirectory/avoid-changing-the-maxpagesize-
> ldap-query-policy/
>
> Essentially the only ways I could think of would be to utilize the
> Paging
> LDAP control, but if the application doesn't support that, I also
> suggested
> creating tighter, more exacting queries. If you have to change
> maxPageSize
> (and all the issues that go with it), do so on an isolated DC
> specifically
> for the application if it was that critical.
>
> Are there any other ways people have used to achieve this when the
> application is incapable of paging, and they won't budge?
>
> Thanks,
>
> Jef
>
> This email was sent to you by Reuters, the global news and information
> company.
> To find out more about Reuters visit www.about.reuters.com
> Any views expressed in this message are those of the individual sender,
> except where the sender specifically states them to be the views of
> Reuters
> Limited.
> Reuters Limited is part of the Reuters Group of companies, of which
> Reuters
> Group PLC is the ultimate parent company. Reuters Group PLC -
> Registered
> office address: The Reuters Building, South Colonnade, Canary Wharf,
> London
> E14 5EP, United Kingdom
> Registered No: 3296375
> Registered in England and Wales
>
> List info : http://www.activedir.org/List.aspx
> List FAQ : http://www.activedir.org/ListFAQ.aspx
> List archive: http://www.activedir.org/ma/default.aspx
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ma/default.aspx
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ma/default.aspx
| | | |
| barkills
Posts:10
| | 04/17/2008 12:51 PM |
| OK, but that doesn't change my objection to it. The net effect of your proposed solution is to remove any server-imposed size limit. Effectively this is the same as option #2 in your list, except you now have an unlimited maxpagesize.
Stepping back a bit, let me say a few things about how other LDAP directories handle this, and maybe this will inspire what I'd think would be better solutions. First, a review of how AD handles size limits, then an example of another LDAP directory.
So AD has many limits which together weave together into the overall picture. There's a server-side limit (called MaxPageSize), which by default is 1000. There's a second server-side limit which you might think about as an absolute maximum server-side limit (called MaxResultSetSize), which by default is 262144. I see that this latter limit hasn't been noted yet in the thread. There's also the possibility of a client side size limit which all depends on what the client sets it to be. AD imposes the lower of the client size limit or the server size limit unless the paging control is invoked. If the paging control is invoked, then it imposes the lower of the client size limit or the absolute maximum server-side limit but it pages the results back to the client. In other words, AD uses the paging control as a way for clients to get around a soft limit. But it still has a hard upper limit. As a side note, as far as I know AD is unique in this regard among LDAP directories; no other LDAP directories employ the paging control as a way to "get around" a server size limit. One of the annoying things here is that the term MaxPageSize is misleading--you don't have to be doing any paging to be subject to that size limit.
Take openldap as another example. With OpenLDAP, there is a default server size limit. I think it's 500, if I recall correctly. As with all ldap clients there's also a client side size limit. Paging is not used as a way to circumvent the limit. Instead, you can set a different size limit on a per-user basis. So my authentication token might be given a server size limit of 150000, but everyone else is subject to the default of 500. Like AD, the net limit is the lower of the client side limit and the server side limit. If you used paging, it would just break that result set into smaller pieces, but it could not be used to circumvent any of the limits. This approach is extremely similar to how other LDAP directories handle size limits.
I'm not sure I understand why Microsoft designed their size limit approach to be so different from every other LDAP directory on the planet. However, I also don't see that your proposed solution to the stated problem makes sense (it also doesn't address the MaxResultSetSize). I'd think a more sensible solution would be to propose that certain users could have a different size limit which overrode the MaxPageSize and possibly the MaxResultSetSize. Such a solution would be a minor change in architecture but I'd think it would be doable. Currently the way the size limits are set is via a queryPolicy object in the config partition. There's a default query policy object, but you can create additional ones. You'd do so, with the limits you'd want, but the missing pieces would be:
a) being able to associate that policy object with a user object (this might require a schema change)
b) having the AD code honor this new behavior
> -----Original Message-----
> From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-
> owner@mail.activedir.org] On Behalf Of Dmitri Gavrilov
> Sent: Tuesday, April 15, 2008 12:25 PM
> To: ActiveDir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> If this proposal is implemented, then no changes are required to client
> apps. They don't need to pass any new controls.
>
> Here's the scenario:
>
> You have an old ldap app that does not know how to page.
> Your domain grows, and the app stops working because it can only get
> 1000 users max.
>
> What do you do? Currently, you only have two options:
> 1. Try to fix the app.
> 2. Increase MaxPageSize.
>
> 1 is often impossible. 2 is generally frowned upon (e.g. in this
> thread). You are out of options!
>
> I am trying to propose a new option, that would allow to do paging at
> server, while presenting the search results to the client as a regular
> ldap search result.
>
> -----Original Message-----
> From: ActiveDir-owner@mail.activedir.org [mailto:ActiveDir-
> owner@mail.activedir.org] On Behalf Of Brian Arkills
> Sent: Tuesday, April 15, 2008 11:58 AM
> To: ActiveDir@mail.activedir.org
> Subject: RE: [ActiveDir] Changing MaxPageSize
>
> I'm somewhat new to this list, but very familiar with both AD and LDAP
> (see http://www.amazon.com/LDAP-Directories-Explained-Introduction-
> Independent/dp/020178792X/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=120828514
> 0&sr=8-1).
>
> I don't understand the strategy of this proposal. Here's what I'm
> hearing:
>
> -Client has software which doesn't support the LDAP paging control
> that's been around for years and which is supported by lots of ldap-
> based software.
> -Instead of getting the software vendor to implement and support that
> well-supported LDAP paging control, there's an idea that the client
> would like Microsoft to implement a *new* LDAP control which
> circumvents the maxpagesize that the server enforces on all LDAP search
> requests.
>
> **If the software vendor won't support the well-known LDAP paging
> control, how are you going to get them to support a brand-new LDAP
> paging control?**
>
> Put another way, all LDAP searches of AD are currently subject to the
> server-side maxpagesize limit (this is a common thing to do in LDAP
> directories). There are many good reason |
|
|