| Author | Messages | |
bsonposh
Posts:171
 | | 07/16/2008 7:51 PM |
| I have a blog post up about using S.DS.P to do a user count. I appreciate
anyone with a large AD that is will to verify the performance (my test were
done on 4 different Domains ranging from 200k-750k users.
http://bsonposh.com/archives/325
I have already started extensive testing (up to about 240) comparing my DSP
vs ADFind. I will be posting those soon as well.
Here are some sneak peaks of the testing so far (just under a second
difference between the two.) Not bad for a script vs a Compiled app.
http://www.bsonposh.com/uploads/Prod700k.png
http://www.bsonposh.com/uploads/ProdVM400k.png
http://www.bsonposh.com/uploads/PhysicalBox.png
http://www.bsonposh.com/uploads/VMBox.png
Thanks!
| | | |
| listmail
Posts:463
| | 07/16/2008 7:55 PM |
| http://blog.joeware.net/2008/06/09/1363/
================
PowerShell + S.DS.Protocols Versus AdFind..
So <https://mvp.support.microsoft.com/profile/Brandon> Brandon stepped up
and did something I didn't think anyone would step up and do. Start
profiling and comparing the performance of PowerShell and .NET DS Protocols
against a native App using the wldap32 LDAP libraryΏ]. Brandon's first blog
entry on the topic is here - <http://bsonposh.com/archives/325>
http://bsonposh.com/archives/325
While I am going to question Brandon's testing a little here, especially his
results, I commend him for taking the time and energy to step up and
actually do thisΑ].
<SNIP>
After seeing those numbers I thought wow and then tried it all again but
flipped it around and had PoS run first and then AdFind. In summary the PoS
stuff started at 22:51:49 and finished at 23:30:22, that is 37 minutes and
33 seconds or 2253 seconds or an average of 225.3 seconds for 10 runs
(compared to 21:09:47 to 21:45:22 which is 35 minutes and 35 seconds or 2135
seconds or an average of 213.5 seconds for 10 runs). The AdFind stuff
started at 23:30:22 and finished at 23:37:59, that is 7 minutes 37 seconds
or 457 seconds or an average of 45.7 seconds for 10 runs (compared to
21:02:08 to 21:09:47 which is 7 minutes 39 seconds or 459 seconds or an
average of 45.9 seconds for 10 runs). So we are seeing that AdFind is
between 4.22 and 5.85 times faster than PoS.
<SNIP>
Thoughts?
joe
================
As Brandon said.... "Not bad for a script vs a Compiled app." ;o)
joe
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Saturday, June 07, 2008 10:32 PM
To: activedir@activedir.org
Subject: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
I have a blog post up about using S.DS.P to do a user count. I appreciate
anyone with a large AD that is will to verify the performance (my test were
done on 4 different Domains ranging from 200k-750k users.
http://bsonposh.com/archives/325
I have already started extensive testing (up to about 240) comparing my DSP
vs ADFind. I will be posting those soon as well.
Here are some sneak peaks of the testing so far (just under a second
difference between the two.) Not bad for a script vs a Compiled app.
http://www.bsonposh.com/uploads/Prod700k.png
http://www.bsonposh.com/uploads/ProdVM400k.png
http://www.bsonposh.com/uploads/PhysicalBox.png
http://www.bsonposh.com/uploads/VMBox.png
Thanks!
| | | |
| guyt76
Posts:11
| | 07/16/2008 7:57 PM |
| joe, I've noticed that you've been running the tests from 32bit OS. Not
sure which OS Brandon used for his tests. It definitely could be a cause
for some differences in the speed.
Another potential issue I can see in the tests is the way the commands
were executed. From my experience, object instantiation/initialization
has a much higher performance hit in .NET code. Add to this a possible
environment initialization fat for running a PoS script (please correct
me here as I'm far from being a PoS expert) and you might see quite
different numbers depending how PoS environment is set up.
I think that a fair test would be to compare the following two:
1) Native C++ app that runs all the tests (and reports timing) that
does all the work in the code without any fat of CreateProcess(), etc...
2) PoS script that does the same - the script should not call any
external commands/scripts and should run the tests in a loop, reporting
the timing for each test.
Unfortunately currently I don't even have a compiler and test
environment available to do it myself (don't ask), otherwise I would
already be brushing off the heavy rust from my C++ skills...
Guy
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of joe
Sent: Monday, June 09, 2008 4:18 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
Read the full blog posting, I detail everything I did and went over it
multiple times prior to posting trying to work out where there might be
issues. I took issue in some of the testing methodology used and
detailed (and IMO corrected) that. Running the tests a lot doesn't
automatically make them better.
Also real life isn't going to have you running commands the way you are
running them at all; there is no way to really get a real life testing
that actually means something relative to the apps. So I did what I did
because running the same command over and over again in sequence helps
get everything out of the way except what the app itself is doing and
helps flag out when you have bad tests popping up. You aren't testing
the DS, you are testing the application/script and until you have
consistent numbers, you don't know if your results are what you are
testing or something else getting in the way.
In this test sequence for PoS
1. 21:09:47 - 21:13:17 = 210 seconds (3 minutes 30 seconds)
2. 21:13:17 - 21:16:47 = 210 seconds (3 minutes 30 seconds)
3. 21:16:47 - 21:20:14 = 207 seconds (3 minutes 27 seconds)
4. 21:20:14 - 21:23:49 = 215 seconds (3 minutes 35 seconds)
5. 21:23:49 - 21:27:18 = 209 seconds (3 minutes 29 seconds)
6. 21:27:18 - 21:30:45 = 207 seconds (3 minutes 27 seconds)
7. 21:30:45 - 21:34:14 = 209 seconds (3 minutes 29 seconds)
8. 21:34:14 - 21:37:50 = 216 seconds (3 minutes 36 seconds)
9. 21:37:50 - 21:41:22 = 212 seconds (3 minutes 32 seconds)
10. 21:41:22 - 21:45:22 = 240 seconds (4 minutes 0 seconds)
You could consider discarding result #10 because it is "off" from the
rest, percentage wise it is still pretty closeΏ] but if you are looking
for the tight grouping of statistics, it feels like it is offΐ].
Something else likely cropped up during that test that was happening in
the DS or on the network. If 3,4,5 all did that then you would expect
something was happening at the moment that those tests were being run.
Splitting the commands up in the way you did makes it tougher to look
for and expect consistent results. If you were truly shooting for real
life, you would have different searches entirely, but that isn't going
to tell you anything in relation to the apps, but instead the DS perf
combined with the apps. You don't have consistency to see the DS
variances.
I kind of wonder how much overhead the PoS is introducing in these tests
versus .NET. Need to build that script into a NET compiled app and do
that testing too.
Ώ] In fact the deviation on the AdFind numbers was higher
percentage-wise between largest delta from low value. Something like 13%
for PoS versus 19% for AdFind though the absolute seconds values were
smaller with AdFind but AdFind was faster overall so less seconds it
could deviate to maintain a tight percentage group of course.
ΐ] Though removing that value only impacts the Average for that
sequence by something like 3 seconds which is still within the deviation
within the other test values.
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
________________________________
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Monday, June 09, 2008 9:04 AM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
Hrmm... not sure what your test did differently, but I know mine was
more complete.
I ran the test 60 times on 4 different environments (200k, 300k, 400k,
and 700k) consistently the same way each time with the ~ same results
each time. I used production quality boxes and VM's. I used 2008 and
2003. I used the command locally and remotely. I posted the screen shots
and will be blogging my method tonight.
I believe my methodology of switching up the commands is a better
indication of real life.
I do find it strange that you had such a huge difference in speak
though. Even directorysearcher wasn't that slow. Even thought I tested
this in 4 different environments, I am still wanting to see some other
production environments. I am working on two more atm.
On Mon, Jun 9, 2008 at 3:20 AM, joe <listmail@joeware.net> wrote:
http://blog.joeware.net/2008/06/09/1363/
================
PowerShell + S.DS.Protocols Versus AdFind....
So Brandon <https://mvp.support.microsoft.com/profile/Brandon> stepped
up and did something I didn't think anyone would step up and do... Start
profiling and comparing the performance of PowerShell and .NET DS
Protocols against a native App using the wldap32 LDAP libraryΏ].
Brandon's first blog entry on the topic is here -
http://bsonposh.com/archives/325 <http://bsonposh.com/archives/325>
While I am going to question Brandon's testing a little here, especially
his results, I commend him for taking the time and energy to step up and
actually do thisΑ].
<SNIP>
After seeing those numbers I thought wow and then tried it all again but
flipped it around and had PoS run first and then AdFind. In summary the
PoS stuff started at 22:51:49 and finished at 23:30:22, that is 37
minutes and 33 seconds or 2253 seconds or an average of 225.3 seconds
for 10 runs (compared to 21:09:47 to 21:45:22 which is 35 minutes and 35
seconds or 2135 seconds or an average of 213.5 seconds for 10 runs). The
AdFind stuff started at 23:30:22 and finished at 23:37:59, that is 7
minutes 37 seconds or 457 seconds or an average of 45.7 seconds for 10
runs (compared to 21:02:08 to 21:09:47 which is 7 minutes 39 seconds or
459 seconds or an average of 45.9 seconds for 10 runs). So we are seeing
that AdFind is between 4.22 and 5.85 times faster than PoS.
<SNIP>
Thoughts?
joe
================
As Brandon said.... "Not bad for a script vs a Compiled app." ;o)
joe
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
________________________________
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Saturday, June 07, 2008 10:32 PM
To: activedir@activedir.org
Subject: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
I have a blog post up about using S.DS.P to do a user count. I
appreciate anyone with a large AD that is will to verify the performance
(my test were done on 4 different Domains ranging from 200k-750k users.
http://bsonposh.com/archives/325
I have already started extensive testing (up to about 240) comparing my
DSP vs ADFind. I will be posting those soon as well.
Here are some sneak peaks of the testing so far (just under a second
difference between the two.) Not bad for a script vs a Compiled app.
http://www.bsonposh.com/uploads/Prod700k.png
http://www.bsonposh.com/uploads/ProdVM400k.png
http://www.bsonposh.com/uploads/PhysicalBox.png
http://www.bsonposh.com/uploads/VMBox.png
Thanks!
| | | |
| bsonposh
Posts:171
| | 07/16/2008 7:57 PM |
| I ran from both x64 and x86 to both x 64 and x86.
I am working on a new round of test that change the test to be the same as
joe.
I would really appreciate anyone else that is willing and able to run these
test. I find it odd that I was able to run this on 4 completely different
environments and achieve very similar results (~1 sec difference.) While
joe's DC is quite different.
Here is the actual script I ran for the DSP vs ADFind. This is how I
automated the testing.
#######################################################
Param($count=3)
$dn = ([ADSI]"").distinguishedName
function TestOne {
Write-Host " + Test ${i}.1"
Write-Host " + Running ADFind Test"
$joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
"(objectclass=user)" 2>&1 | out-Null }
Write-Host " - $($joetime.TotalSeconds)"
Write-Host " + Running DSP Test"
$DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
Write-Host " - $($DSPTime11.TotalSeconds)"
$myresults = "" | select @{n="ADFind"
;e={$joeTime.TotalSeconds}},
@{n="DSP Using 1.1"
;e={$DSPTime11.TotalSeconds}}
$myresults
}
function TestTwo {
Write-Host " + Test ${i}.2"
Write-Host " + Running DSP Test"
$DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
Write-Host " - $($DSPTime11.TotalSeconds)"
Write-Host " + Running ADFind Test"
$joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
"(objectclass=user)" 2>&1 | out-Null }
Write-Host " - $($joetime.TotalSeconds)"
$myresults = "" | select @{n="ADFind"
;e={$joeTime.TotalSeconds}},
@{n="DSP Using 1.1"
;e={$DSPTime11.TotalSeconds}}
$myresults
}
Write-Host
for($i = 0 ; $i -le $count ; $i++)
{
Write-Host "+ Test $i"
TestOne
TestTwo
}
Write-Host
###############################################################
On Mon, Jun 9, 2008 at 12:07 PM, Guy Teverovsky <guyt@netguru.co.il> wrote:
> joe, I've noticed that you've been running the tests from 32bit OS. Not
> sure which OS Brandon used for his tests. It definitely could be a cause for
> some differences in the speed.
>
>
>
> Another potential issue I can see in the tests is the way the commands were
> executed. From my experience, object instantiation/initialization has a much
> higher performance hit in .NET code. Add to this a possible environment
> initialization fat for running a PoS script (please correct me here as I'm
> far from being a PoS expert) and you might see quite different numbers
> depending how PoS environment is set up.
>
>
>
> I think that a fair test would be to compare the following two:
>
>
>
> 1) Native C++ app that runs all the tests (and reports timing) that
> does all the work in the code without any fat of CreateProcess(), etc…
>
> 2) PoS script that does the same – the script should not call any
> external commands/scripts and should run the tests in a loop, reporting the
> timing for each test.
>
>
>
> Unfortunately currently I don't even have a compiler and test environment
> available to do it myself (don't ask), otherwise I would already be brushing
> off the heavy rust from my C++ skills…
>
>
>
> Guy
>
>
>
>
>
> *From:* ActiveDir-owner@mail.activedir.org [mailto:
> ActiveDir-owner@mail.activedir.org] *On Behalf Of *joe
> *Sent:* Monday, June 09, 2008 4:18 PM
> *To:* ActiveDir@mail.activedir.org
> *Subject:* RE: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>
>
>
> Read the full blog posting, I detail everything I did and went over it
> multiple times prior to posting trying to work out where there might be
> issues. I took issue in some of the testing methodology used and detailed
> (and IMO corrected) that. Running the tests a lot doesn't automatically make
> them better.
>
>
>
> Also real life isn't going to have you running commands the way you are
> running them at all; there is no way to really get a real life testing that
> actually means something relative to the apps. So I did what I did because
> running the same command over and over again in sequence helps get
> everything out of the way except what the app itself is doing and helps flag
> out when you have bad tests popping up. You aren't testing the DS, you are
> testing the application/script and until you have consistent numbers, you
> don't know if your results are what you are testing or something else
> getting in the way.
>
>
>
> In this test sequence for PoS
>
> 1. 21:09:47 - 21:13:17 = 210 seconds (3 minutes 30 seconds)
> 2. 21:13:17 - 21:16:47 = 210 seconds (3 minutes 30 seconds)
> 3. 21:16:47 - 21:20:14 = 207 seconds (3 minutes 27 seconds)
> 4. 21:20:14 - 21:23:49 = 215 seconds (3 minutes 35 seconds)
> 5. 21:23:49 - 21:27:18 = 209 seconds (3 minutes 29 seconds)
> 6. 21:27:18 - 21:30:45 = 207 seconds (3 minutes 27 seconds)
> 7. 21:30:45 - 21:34:14 = 209 seconds (3 minutes 29 seconds)
> 8. 21:34:14 - 21:37:50 = 216 seconds (3 minutes 36 seconds)
> 9. 21:37:50 - 21:41:22 = 212 seconds (3 minutes 32 seconds)
> 10. 21:41:22 - 21:45:22 = 240 seconds (4 minutes 0 seconds)
>
>
>
> You could consider discarding result #10 because it is "off" from the rest,
> percentage wise it is still pretty closeΏ] but if you are looking for the
> tight grouping of statistics, it feels like it is offΐ]. Something else
> likely cropped up during that test that was happening in the DS or on the
> network. If 3,4,5 all did that then you would expect something was happening
> at the moment that those tests were being run. Splitting the commands up in
> the way you did makes it tougher to look for and expect consistent results.
> If you were truly shooting for real life, you would have different searches
> entirely, but that isn't going to tell you anything in relation to the apps,
> but instead the DS perf combined with the apps. You don't have consistency
> to see the DS variances.
>
>
>
>
>
> I kind of wonder how much overhead the PoS is introducing in these tests
> versus .NET. Need to build that script into a NET compiled app and do that
> testing too.
>
>
>
>
>
>
>
> Ώ] In fact the deviation on the AdFind numbers was higher percentage-wise
> between largest delta from low value. Something like 13% for PoS versus 19%
> for AdFind though the absolute seconds values were smaller with AdFind but
> AdFind was faster overall so less seconds it could deviate to maintain a
> tight percentage group of course.
>
>
>
> ΐ] Though removing that value only impacts the Average for that sequence
> by something like 3 seconds which is still within the deviation within the
> other test values.
>
>
>
>
>
> --
>
> O'Reilly Active Directory Third Edition -
> http://www.joeware.net/win/ad3e.htm
>
>
>
>
>
>
> ------------------------------
>
> *From:* ActiveDir-owner@mail.activedir.org [mailto:
> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
> *Sent:* Monday, June 09, 2008 9:04 AM
> *To:* ActiveDir@mail.activedir.org
> *Subject:* Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>
> Hrmm... not sure what your test did differently, but I know mine was more
> complete.
>
>
>
> I ran the test 60 times on 4 different environments (200k, 300k, 400k, and
> 700k) consistently the same way each time with the ~ same results each time.
> I used production quality boxes and VM's. I used 2008 and 2003. I used the
> command locally and remotely. I posted the screen shots and will be blogging
> my method tonight.
>
>
>
> I believe my methodology of switching up the commands is a better
> indication of real life.
>
>
>
> I do find it strange that you had such a huge difference in speak though.
> Even directorysearcher wasn't that slow. Even thought I tested this in 4
> different environments, I am still wanting to see some other production
> environments. I am working on two more atm.
>
> On Mon, Jun 9, 2008 at 3:20 AM, joe <listmail@joeware.net> wrote:
>
> http://blog.joeware.net/2008/06/09/1363/
>
>
>
> ================
>
>
>
> *PowerShell + S.DS.Protocols Versus AdFind….*
>
>
>
> So Brandon <https://mvp.support.microsoft.com/profile/Brandon> stepped up
> and did something I didn't think anyone would step up and do… Start
> profiling and comparing the performance of PowerShell and .NET DS Protocols
> against a native App using the wldap32 LDAP libraryΏ]. Brandon's first blog
> entry on the topic is here - http://bsonposh.com/archives/325
>
> While I am going to question Brandon's testing a little here, especially
> his results, I commend him for taking the time and energy to step up and
> actually do thisΑ].
>
> <SNIP>
>
>
>
> After seeing those numbers I thought wow and then tried it all again but
> flipped it around and had PoS run first and then AdFind. In summary the PoS
> stuff started at 22:51:49 and finished at 23:30:22, that is 37 minutes and
> 33 seconds or 2253 seconds or an average of 225.3 seconds for 10 runs
> (compared to 21:09:47 to 21:45:22 which is 35 minutes and 35 seconds or 2135
> seconds or an average of 213.5 seconds for 10 runs). The AdFind stuff
> started at 23:30:22 and finished at 23:37:59, that is 7 minutes 37 seconds
> or 457 seconds or an average of 45.7 seconds for 10 runs (compared to
> 21:02:08 to 21:09:47 which is 7 minutes 39 seconds or 459 seconds or an
> average of 45.9 seconds for 10 runs). *So we are seeing that AdFind is
> between 4.22 and 5.85 times faster than PoS.*
>
>
>
> <SNIP>
>
>
>
> Thoughts?
>
> joe
>
> ================
>
>
>
>
>
> As Brandon said.... "Not bad for a script vs a Compiled app." ;o)
>
>
>
>
>
> joe
>
>
>
>
>
> --
>
> O'Reilly Active Directory Third Edition -
> http://www.joeware.net/win/ad3e.htm
>
>
>
>
>
>
> ------------------------------
>
> *From:* ActiveDir-owner@mail.activedir.org [mailto:
> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
> *Sent:* Saturday, June 07, 2008 10:32 PM
> *To:* activedir@activedir.org
> *Subject:* [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>
> I have a blog post up about using S.DS.P to do a user count. I appreciate
> anyone with a large AD that is will to verify the performance (my test were
> done on 4 different Domains ranging from 200k-750k users.
>
>
>
> http://bsonposh.com/archives/325
>
>
>
> I have already started extensive testing (up to about 240) comparing my DSP
> vs ADFind. I will be posting those soon as well.
>
>
>
> Here are some sneak peaks of the testing so far (just under a second
> difference between the two.) Not bad for a script vs a Compiled app.
>
> http://www.bsonposh.com/uploads/Prod700k.png
> http://www.bsonposh.com/uploads/ProdVM400k.png
> http://www.bsonposh.com/uploads/PhysicalBox.png
> http://www.bsonposh.com/uploads/VMBox.png
>
>
>
> Thanks!
>
>
>
| | | |
| listmail
Posts:463
| | 07/16/2008 8:07 PM |
| > I am sure joe will fess up why his findings were off 
Fess up?? That sounds like I did something wrong...
I am still trying to work out what happened before I write up the details,
but apparantly PoS got a bug up its ass and the instance I was initially
running was having issues and going extremely slow whereas another instance
I opened up and ran next to it ran the script closer to what I was expecting
for perf (i.e. within about 25%). How you would detect that in the normal
course I have no idea. Dean/Brandon says it is a glitch, probably rare, but
I don't think there is a way to ascertain how rare it is unless I can find
some way to detect it. It seems to run some scripts just fine but the AD
script it fell flat on its face with. I guess someone could run a count the
AD objects script when they open the instances to validate you have a good
one. Or just hope you can identify a bad one when you get one.
I am trying to duplicate the problem. I have managed to slow a new instance
down but not to the extent that I see in that first instance of PoS. I have
managed to crash PoS quite a few times though and found that when I hit
Control Break once it closed the whole instance out instead of stopping the
script that was running.
One thing that I have found extremely annoying though is that during some of
the testing, running two instances of the test framework script Brandon had
on his Blog site is enough to practically kill my test PC which is a dual
core AMD with 3GB of RAM. The two instances of PoS suck up nearly all the
RAM until they start throwing out of memory exceptions. As I mentioned to
Dean at one point... wow, I could be running 400 command prompts and not
using as much RAM as this one idle PoS instance has tied up....
I'll write more but to be completely honest, I am even less enamored with
PoS and its overall perf now than when I started this test.
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Monday, June 09, 2008 7:35 PM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
FYI: http://bsonposh.com/archives/328
summary: My finding stand. S.DS.P is ~1 sec slower than adfind.exe in large
AD's. as well as some joe rebuttle rebuttles
I am sure joe will fess up why his findings were off
Dean spent most of the day in awe (bit of exeration) that it was actually
was as fast I stated.
Next step is properties... I think adfind.exe is going to start pulling way
ahead here, but I was surprised before so lets see.
On Mon, Jun 9, 2008 at 12:22 PM, Brandon Shell <tshell@gmail.com> wrote:
I ran from both x64 and x86 to both x 64 and x86.
I am working on a new round of test that change the test to be the same as
joe.
I would really appreciate anyone else that is willing and able to run these
test. I find it odd that I was able to run this on 4 completely different
environments and achieve very similar results (~1 sec difference.) While
joe's DC is quite different.
Here is the actual script I ran for the DSP vs ADFind. This is how I
automated the testing.
#######################################################
Param($count=3)
$dn = ([ADSI]"").distinguishedName
function TestOne {
Write-Host " + Test ${i}.1"
Write-Host " + Running ADFind Test"
$joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
"(objectclass=user)" 2>&1 | out-Null }
Write-Host " - $($joetime.TotalSeconds)"
Write-Host " + Running DSP Test"
$DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
Write-Host " - $($DSPTime11.TotalSeconds)"
$myresults = "" | select @{n="ADFind"
;e={$joeTime.TotalSeconds}},
@{n="DSP Using 1.1"
;e={$DSPTime11.TotalSeconds}}
$myresults
}
function TestTwo {
Write-Host " + Test ${i}.2"
Write-Host " + Running DSP Test"
$DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
Write-Host " - $($DSPTime11.TotalSeconds)"
Write-Host " + Running ADFind Test"
$joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
"(objectclass=user)" 2>&1 | out-Null }
Write-Host " - $($joetime.TotalSeconds)"
$myresults = "" | select @{n="ADFind"
;e={$joeTime.TotalSeconds}},
@{n="DSP Using 1.1"
;e={$DSPTime11.TotalSeconds}}
$myresults
}
Write-Host
for($i = 0 ; $i -le $count ; $i++)
{
Write-Host "+ Test $i"
TestOne
TestTwo
}
Write-Host
###############################################################
On Mon, Jun 9, 2008 at 12:07 PM, Guy Teverovsky <guyt@netguru.co.il> wrote:
joe, I've noticed that you've been running the tests from 32bit OS. Not sure
which OS Brandon used for his tests. It definitely could be a cause for some
differences in the speed.
Another potential issue I can see in the tests is the way the commands were
executed. From my experience, object instantiation/initialization has a much
higher performance hit in .NET code. Add to this a possible environment
initialization fat for running a PoS script (please correct me here as I'm
far from being a PoS expert) and you might see quite different numbers
depending how PoS environment is set up.
I think that a fair test would be to compare the following two:
1) Native C++ app that runs all the tests (and reports timing) that does
all the work in the code without any fat of CreateProcess(), etc.
2) PoS script that does the same - the script should not call any
external commands/scripts and should run the tests in a loop, reporting the
timing for each test.
Unfortunately currently I don't even have a compiler and test environment
available to do it myself (don't ask), otherwise I would already be brushing
off the heavy rust from my C++ skills.
Guy
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of joe
Sent: Monday, June 09, 2008 4:18 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
Read the full blog posting, I detail everything I did and went over it
multiple times prior to posting trying to work out where there might be
issues. I took issue in some of the testing methodology used and detailed
(and IMO corrected) that. Running the tests a lot doesn't automatically make
them better.
Also real life isn't going to have you running commands the way you are
running them at all; there is no way to really get a real life testing that
actually means something relative to the apps. So I did what I did because
running the same command over and over again in sequence helps get
everything out of the way except what the app itself is doing and helps flag
out when you have bad tests popping up. You aren't testing the DS, you are
testing the application/script and until you have consistent numbers, you
don't know if your results are what you are testing or something else
getting in the way.
In this test sequence for PoS
1. 21:09:47 - 21:13:17 = 210 seconds (3 minutes 30 seconds)
2. 21:13:17 - 21:16:47 = 210 seconds (3 minutes 30 seconds)
3. 21:16:47 - 21:20:14 = 207 seconds (3 minutes 27 seconds)
4. 21:20:14 - 21:23:49 = 215 seconds (3 minutes 35 seconds)
5. 21:23:49 - 21:27:18 = 209 seconds (3 minutes 29 seconds)
6. 21:27:18 - 21:30:45 = 207 seconds (3 minutes 27 seconds)
7. 21:30:45 - 21:34:14 = 209 seconds (3 minutes 29 seconds)
8. 21:34:14 - 21:37:50 = 216 seconds (3 minutes 36 seconds)
9. 21:37:50 - 21:41:22 = 212 seconds (3 minutes 32 seconds)
10. 21:41:22 - 21:45:22 = 240 seconds (4 minutes 0 seconds)
You could consider discarding result #10 because it is "off" from the rest,
percentage wise it is still pretty closeΏ] but if you are looking for the
tight grouping of statistics, it feels like it is offΐ]. Something else
likely cropped up during that test that was happening in the DS or on the
network. If 3,4,5 all did that then you would expect something was happening
at the moment that those tests were being run. Splitting the commands up in
the way you did makes it tougher to look for and expect consistent results.
If you were truly shooting for real life, you would have different searches
entirely, but that isn't going to tell you anything in relation to the apps,
but instead the DS perf combined with the apps. You don't have consistency
to see the DS variances.
I kind of wonder how much overhead the PoS is introducing in these tests
versus .NET. Need to build that script into a NET compiled app and do that
testing too.
Ώ] In fact the deviation on the AdFind numbers was higher percentage-wise
between largest delta from low value. Something like 13% for PoS versus 19%
for AdFind though the absolute seconds values were smaller with AdFind but
AdFind was faster overall so less seconds it could deviate to maintain a
tight percentage group of course.
ΐ] Though removing that value only impacts the Average for that sequence by
something like 3 seconds which is still within the deviation within the
other test values.
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Monday, June 09, 2008 9:04 AM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
Hrmm... not sure what your test did differently, but I know mine was more
complete.
I ran the test 60 times on 4 different environments (200k, 300k, 400k, and
700k) consistently the same way each time with the ~ same results each time.
I used production quality boxes and VM's. I used 2008 and 2003. I used the
command locally and remotely. I posted the screen shots and will be blogging
my method tonight.
I believe my methodology of switching up the commands is a better indication
of real life.
I do find it strange that you had such a huge difference in speak though.
Even directorysearcher wasn't that slow. Even thought I tested this in 4
different environments, I am still wanting to see some other production
environments. I am working on two more atm.
On Mon, Jun 9, 2008 at 3:20 AM, joe <listmail@joeware.net> wrote:
http://blog.joeware.net/2008/06/09/1363/
================
PowerShell + S.DS.Protocols Versus AdFind..
So <https://mvp.support.microsoft.com/profile/Brandon> Brandon stepped up
and did something I didn't think anyone would step up and do. Start
profiling and comparing the performance of PowerShell and .NET DS Protocols
against a native App using the wldap32 LDAP libraryΏ]. Brandon's first blog
entry on the topic is here - <http://bsonposh.com/archives/325>
http://bsonposh.com/archives/325
While I am going to question Brandon's testing a little here, especially his
results, I commend him for taking the time and energy to step up and
actually do thisΑ].
<SNIP>
After seeing those numbers I thought wow and then tried it all again but
flipped it around and had PoS run first and then AdFind. In summary the PoS
stuff started at 22:51:49 and finished at 23:30:22, that is 37 minutes and
33 seconds or 2253 seconds or an average of 225.3 seconds for 10 runs
(compared to 21:09:47 to 21:45:22 which is 35 minutes and 35 seconds or 2135
seconds or an average of 213.5 seconds for 10 runs). The AdFind stuff
started at 23:30:22 and finished at 23:37:59, that is 7 minutes 37 seconds
or 457 seconds or an average of 45.7 seconds for 10 runs (compared to
21:02:08 to 21:09:47 which is 7 minutes 39 seconds or 459 seconds or an
average of 45.9 seconds for 10 runs). So we are seeing that AdFind is
between 4.22 and 5.85 times faster than PoS.
<SNIP>
Thoughts?
joe
================
As Brandon said.... "Not bad for a script vs a Compiled app." ;o)
joe
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Saturday, June 07, 2008 10:32 PM
To: activedir@activedir.org
Subject: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
I have a blog post up about using S.DS.P to do a user count. I appreciate
anyone with a large AD that is will to verify the performance (my test were
done on 4 different Domains ranging from 200k-750k users.
http://bsonposh.com/archives/325
I have already started extensive testing (up to about 240) comparing my DSP
vs ADFind. I will be posting those soon as well.
Here are some sneak peaks of the testing so far (just under a second
difference between the two.) Not bad for a script vs a Compiled app.
http://www.bsonposh.com/uploads/Prod700k.png
http://www.bsonposh.com/uploads/ProdVM400k.png
http://www.bsonposh.com/uploads/PhysicalBox.png
http://www.bsonposh.com/uploads/VMBox.png
Thanks!
| | | |
| bsonposh
Posts:171
| | 07/16/2008 8:09 PM |
| By "fess up", I meant I'd hoped you'd mention that your later tests were
closer to mine and Deans. That said, you spent a tremendous amount of time
on your blog criticizing my testing so I think was hoping for some kind of
retraction on your conclusions.
On Tue, Jun 10, 2008 at 9:33 AM, joe <listmail@joeware.net> wrote:
> > I am sure joe will fess up why his findings were off
>
> Fess up?? That sounds like I did something wrong...
>
> I am still trying to work out what happened before I write up the details,
> but apparantly PoS got a bug up its ass and the instance I was initially
> running was having issues and going extremely slow whereas another instance
> I opened up and ran next to it ran the script closer to what I was expecting
> for perf (i.e. within about 25%). How you would detect that in the normal
> course I have no idea. Dean/Brandon says it is a glitch, probably rare, but
> I don't think there is a way to ascertain how rare it is unless I can find
> some way to detect it. It seems to run some scripts just fine but the AD
> script it fell flat on its face with. I guess someone could run a count the
> AD objects script when they open the instances to validate you have a good
> one. Or just hope you can identify a bad one when you get one.
>
> I am trying to duplicate the problem. I have managed to slow a new
> instance down but not to the extent that I see in that first instance of
> PoS. I have managed to crash PoS quite a few times though and found that
> when I hit Control Break once it closed the whole instance out instead of
> stopping the script that was running.
>
> One thing that I have found extremely annoying though is that during some
> of the testing, running two instances of the test framework script Brandon
> had on his Blog site is enough to practically kill my test PC which is a
> dual core AMD with 3GB of RAM. The two instances of PoS suck up nearly all
> the RAM until they start throwing out of memory exceptions. As I mentioned
> to Dean at one point... wow, I could be running 400 command prompts and not
> using as much RAM as this one idle PoS instance has tied up....
>
> I'll write more but to be completely honest, I am even less enamored with
> PoS and its overall perf now than when I started this test.
>
> --
> O'Reilly Active Directory Third Edition -
> http://www.joeware.net/win/ad3e.htm
>
>
>
> ------------------------------
> *From:* ActiveDir-owner@mail.activedir.org [mailto:
> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
> *Sent:* Monday, June 09, 2008 7:35 PM
> *To:* ActiveDir@mail.activedir.org
> *Subject:* Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>
> FYI: http://bsonposh.com/archives/328
>
> summary: My finding stand. S.DS.P is ~1 sec slower than adfind.exe in large
> AD's. as well as some joe rebuttle rebuttles
>
> I am sure joe will fess up why his findings were off
>
> Dean spent most of the day in awe (bit of exeration) that it was
> actually was as fast I stated.
>
> Next step is properties... I think adfind.exe is going to start pulling way
> ahead here, but I was surprised before so lets see.
>
> On Mon, Jun 9, 2008 at 12:22 PM, Brandon Shell <tshell@gmail.com> wrote:
>
>> I ran from both x64 and x86 to both x 64 and x86.
>>
>> I am working on a new round of test that change the test to be the same as
>> joe.
>>
>> I would really appreciate anyone else that is willing and able to run
>> these test. I find it odd that I was able to run this on 4 completely
>> different environments and achieve very similar results (~1 sec difference.)
>> While joe's DC is quite different.
>> Here is the actual script I ran for the DSP vs ADFind. This is how I
>> automated the testing.
>>
>> #######################################################
>> Param($count=3)
>> $dn = ([ADSI]"").distinguishedName
>> function TestOne {
>> Write-Host " + Test ${i}.1"
>> Write-Host " + Running ADFind Test"
>> $joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
>> "(objectclass=user)" 2>&1 | out-Null }
>> Write-Host " - $($joetime.TotalSeconds)"
>> Write-Host " + Running DSP Test"
>> $DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
>> Write-Host " - $($DSPTime11.TotalSeconds)"
>> $myresults = "" | select @{n="ADFind"
>> ;e={$joeTime.TotalSeconds}},
>> @{n="DSP Using 1.1"
>> ;e={$DSPTime11.TotalSeconds}}
>> $myresults
>> }
>> function TestTwo {
>> Write-Host " + Test ${i}.2"
>> Write-Host " + Running DSP Test"
>> $DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
>> Write-Host " - $($DSPTime11.TotalSeconds)"
>> Write-Host " + Running ADFind Test"
>> $joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
>> "(objectclass=user)" 2>&1 | out-Null }
>> Write-Host " - $($joetime.TotalSeconds)"
>> $myresults = "" | select @{n="ADFind"
>> ;e={$joeTime.TotalSeconds}},
>> @{n="DSP Using 1.1"
>> ;e={$DSPTime11.TotalSeconds}}
>> $myresults
>> }
>> Write-Host
>> for($i = 0 ; $i -le $count ; $i++)
>> {
>> Write-Host "+ Test $i"
>> TestOne
>> TestTwo
>> }
>> Write-Host
>> ###############################################################
>> On Mon, Jun 9, 2008 at 12:07 PM, Guy Teverovsky <guyt@netguru.co.il>
>> wrote:
>>
>>> joe, I've noticed that you've been running the tests from 32bit OS. Not
>>> sure which OS Brandon used for his tests. It definitely could be a cause for
>>> some differences in the speed.
>>>
>>>
>>>
>>> Another potential issue I can see in the tests is the way the commands
>>> were executed. From my experience, object instantiation/initialization has a
>>> much higher performance hit in .NET code. Add to this a possible environment
>>> initialization fat for running a PoS script (please correct me here as I'm
>>> far from being a PoS expert) and you might see quite different numbers
>>> depending how PoS environment is set up.
>>>
>>>
>>>
>>> I think that a fair test would be to compare the following two:
>>>
>>>
>>>
>>> 1) Native C++ app that runs all the tests (and reports timing) that
>>> does all the work in the code without any fat of CreateProcess(), etc…
>>>
>>> 2) PoS script that does the same – the script should not call any
>>> external commands/scripts and should run the tests in a loop, reporting the
>>> timing for each test.
>>>
>>>
>>>
>>> Unfortunately currently I don't even have a compiler and test environment
>>> available to do it myself (don't ask), otherwise I would already be brushing
>>> off the heavy rust from my C++ skills…
>>>
>>>
>>>
>>> Guy
>>>
>>>
>>>
>>>
>>>
>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *joe
>>> *Sent:* Monday, June 09, 2008 4:18 PM
>>> *To:* ActiveDir@mail.activedir.org
>>> *Subject:* RE: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>
>>>
>>>
>>> Read the full blog posting, I detail everything I did and went over it
>>> multiple times prior to posting trying to work out where there might be
>>> issues. I took issue in some of the testing methodology used and detailed
>>> (and IMO corrected) that. Running the tests a lot doesn't automatically make
>>> them better.
>>>
>>>
>>>
>>> Also real life isn't going to have you running commands the way you are
>>> running them at all; there is no way to really get a real life testing that
>>> actually means something relative to the apps. So I did what I did because
>>> running the same command over and over again in sequence helps get
>>> everything out of the way except what the app itself is doing and helps flag
>>> out when you have bad tests popping up. You aren't testing the DS, you are
>>> testing the application/script and until you have consistent numbers, you
>>> don't know if your results are what you are testing or something else
>>> getting in the way.
>>>
>>>
>>>
>>> In this test sequence for PoS
>>>
>>> 1. 21:09:47 - 21:13:17 = 210 seconds (3 minutes 30 seconds)
>>> 2. 21:13:17 - 21:16:47 = 210 seconds (3 minutes 30 seconds)
>>> 3. 21:16:47 - 21:20:14 = 207 seconds (3 minutes 27 seconds)
>>> 4. 21:20:14 - 21:23:49 = 215 seconds (3 minutes 35 seconds)
>>> 5. 21:23:49 - 21:27:18 = 209 seconds (3 minutes 29 seconds)
>>> 6. 21:27:18 - 21:30:45 = 207 seconds (3 minutes 27 seconds)
>>> 7. 21:30:45 - 21:34:14 = 209 seconds (3 minutes 29 seconds)
>>> 8. 21:34:14 - 21:37:50 = 216 seconds (3 minutes 36 seconds)
>>> 9. 21:37:50 - 21:41:22 = 212 seconds (3 minutes 32 seconds)
>>> 10. 21:41:22 - 21:45:22 = 240 seconds (4 minutes 0 seconds)
>>>
>>>
>>>
>>> You could consider discarding result #10 because it is "off" from the
>>> rest, percentage wise it is still pretty closeΏ] but if you are looking for
>>> the tight grouping of statistics, it feels like it is offΐ]. Something else
>>> likely cropped up during that test that was happening in the DS or on the
>>> network. If 3,4,5 all did that then you would expect something was happening
>>> at the moment that those tests were being run. Splitting the commands up in
>>> the way you did makes it tougher to look for and expect consistent results.
>>> If you were truly shooting for real life, you would have different searches
>>> entirely, but that isn't going to tell you anything in relation to the apps,
>>> but instead the DS perf combined with the apps. You don't have consistency
>>> to see the DS variances.
>>>
>>>
>>>
>>>
>>>
>>> I kind of wonder how much overhead the PoS is introducing in these tests
>>> versus .NET. Need to build that script into a NET compiled app and do that
>>> testing too.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Ώ] In fact the deviation on the AdFind numbers was higher
>>> percentage-wise between largest delta from low value. Something like 13% for
>>> PoS versus 19% for AdFind though the absolute seconds values were smaller
>>> with AdFind but AdFind was faster overall so less seconds it could deviate
>>> to maintain a tight percentage group of course.
>>>
>>>
>>>
>>> ΐ] Though removing that value only impacts the Average for that sequence
>>> by something like 3 seconds which is still within the deviation within the
>>> other test values.
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> O'Reilly Active Directory Third Edition -
>>> http://www.joeware.net/win/ad3e.htm
>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
>>> *Sent:* Monday, June 09, 2008 9:04 AM
>>> *To:* ActiveDir@mail.activedir.org
>>> *Subject:* Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>
>>> Hrmm... not sure what your test did differently, but I know mine was more
>>> complete.
>>>
>>>
>>>
>>> I ran the test 60 times on 4 different environments (200k, 300k, 400k,
>>> and 700k) consistently the same way each time with the ~ same results each
>>> time. I used production quality boxes and VM's. I used 2008 and 2003. I used
>>> the command locally and remotely. I posted the screen shots and will be
>>> blogging my method tonight.
>>>
>>>
>>>
>>> I believe my methodology of switching up the commands is a better
>>> indication of real life.
>>>
>>>
>>>
>>> I do find it strange that you had such a huge difference in speak though.
>>> Even directorysearcher wasn't that slow. Even thought I tested this in 4
>>> different environments, I am still wanting to see some other production
>>> environments. I am working on two more atm.
>>>
>>> On Mon, Jun 9, 2008 at 3:20 AM, joe <listmail@joeware.net> wrote:
>>>
>>> http://blog.joeware.net/2008/06/09/1363/
>>>
>>>
>>>
>>> ================
>>>
>>>
>>>
>>> *PowerShell + S.DS.Protocols Versus AdFind….*
>>>
>>>
>>>
>>> So Brandon <https://mvp.support.microsoft.com/profile/Brandon> stepped
>>> up and did something I didn't think anyone would step up and do… Start
>>> profiling and comparing the performance of PowerShell and .NET DS Protocols
>>> against a native App using the wldap32 LDAP libraryΏ]. Brandon's first blog
>>> entry on the topic is here - http://bsonposh.com/archives/325
>>>
>>> While I am going to question Brandon's testing a little here, especially
>>> his results, I commend him for taking the time and energy to step up and
>>> actually do thisΑ].
>>>
>>> <SNIP>
>>>
>>>
>>>
>>> After seeing those numbers I thought wow and then tried it all again but
>>> flipped it around and had PoS run first and then AdFind. In summary the PoS
>>> stuff started at 22:51:49 and finished at 23:30:22, that is 37 minutes and
>>> 33 seconds or 2253 seconds or an average of 225.3 seconds for 10 runs
>>> (compared to 21:09:47 to 21:45:22 which is 35 minutes and 35 seconds or 2135
>>> seconds or an average of 213.5 seconds for 10 runs). The AdFind stuff
>>> started at 23:30:22 and finished at 23:37:59, that is 7 minutes 37 seconds
>>> or 457 seconds or an average of 45.7 seconds for 10 runs (compared to
>>> 21:02:08 to 21:09:47 which is 7 minutes 39 seconds or 459 seconds or an
>>> average of 45.9 seconds for 10 runs). *So we are seeing that AdFind is
>>> between 4.22 and 5.85 times faster than PoS.*
>>>
>>>
>>>
>>> <SNIP>
>>>
>>>
>>>
>>> Thoughts?
>>>
>>> joe
>>>
>>> ================
>>>
>>>
>>>
>>>
>>>
>>> As Brandon said.... "Not bad for a script vs a Compiled app." ;o)
>>>
>>>
>>>
>>>
>>>
>>> joe
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> O'Reilly Active Directory Third Edition -
>>> http://www.joeware.net/win/ad3e.htm
>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
>>> *Sent:* Saturday, June 07, 2008 10:32 PM
>>> *To:* activedir@activedir.org
>>> *Subject:* [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>
>>> I have a blog post up about using S.DS.P to do a user count. I appreciate
>>> anyone with a large AD that is will to verify the performance (my test were
>>> done on 4 different Domains ranging from 200k-750k users.
>>>
>>>
>>>
>>> http://bsonposh.com/archives/325
>>>
>>>
>>>
>>> I have already started extensive testing (up to about 240) comparing my
>>> DSP vs ADFind. I will be posting those soon as well.
>>>
>>>
>>>
>>> Here are some sneak peaks of the testing so far (just under a second
>>> difference between the two.) Not bad for a script vs a Compiled app.
>>>
>>> http://www.bsonposh.com/uploads/Prod700k.png
>>> http://www.bsonposh.com/uploads/ProdVM400k.png
>>> http://www.bsonposh.com/uploads/PhysicalBox.png
>>> http://www.bsonposh.com/uploads/VMBox.png
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>
>>
>
| | | |
| listmail
Posts:463
| | 07/16/2008 8:13 PM |
| The data spoke for itself. Now the question is why did it do that in one
instance but not in the next instance I opened. I am not just going to
dismiss it because you didn't like it until I personally have some
understanding of it.
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Tuesday, June 10, 2008 10:35 AM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
By "fess up", I meant I'd hoped you'd mention that your later tests were
closer to mine and Deans. That said, you spent a tremendous amount of time
on your blog criticizing my testing so I think was hoping for some kind of
retraction on your conclusions.
On Tue, Jun 10, 2008 at 9:33 AM, joe <listmail@joeware.net> wrote:
> I am sure joe will fess up why his findings were off
Fess up?? That sounds like I did something wrong...
I am still trying to work out what happened before I write up the details,
but apparantly PoS got a bug up its ass and the instance I was initially
running was having issues and going extremely slow whereas another instance
I opened up and ran next to it ran the script closer to what I was expecting
for perf (i.e. within about 25%). How you would detect that in the normal
course I have no idea. Dean/Brandon says it is a glitch, probably rare, but
I don't think there is a way to ascertain how rare it is unless I can find
some way to detect it. It seems to run some scripts just fine but the AD
script it fell flat on its face with. I guess someone could run a count the
AD objects script when they open the instances to validate you have a good
one. Or just hope you can identify a bad one when you get one.
I am trying to duplicate the problem. I have managed to slow a new instance
down but not to the extent that I see in that first instance of PoS. I have
managed to crash PoS quite a few times though and found that when I hit
Control Break once it closed the whole instance out instead of stopping the
script that was running.
One thing that I have found extremely annoying though is that during some of
the testing, running two instances of the test framework script Brandon had
on his Blog site is enough to practically kill my test PC which is a dual
core AMD with 3GB of RAM. The two instances of PoS suck up nearly all the
RAM until they start throwing out of memory exceptions. As I mentioned to
Dean at one point... wow, I could be running 400 command prompts and not
using as much RAM as this one idle PoS instance has tied up....
I'll write more but to be completely honest, I am even less enamored with
PoS and its overall perf now than when I started this test.
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Monday, June 09, 2008 7:35 PM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
FYI: http://bsonposh.com/archives/328
summary: My finding stand. S.DS.P is ~1 sec slower than adfind.exe in large
AD's. as well as some joe rebuttle rebuttles
I am sure joe will fess up why his findings were off
Dean spent most of the day in awe (bit of exeration) that it was actually
was as fast I stated.
Next step is properties... I think adfind.exe is going to start pulling way
ahead here, but I was surprised before so lets see.
On Mon, Jun 9, 2008 at 12:22 PM, Brandon Shell <tshell@gmail.com> wrote:
I ran from both x64 and x86 to both x 64 and x86.
I am working on a new round of test that change the test to be the same as
joe.
I would really appreciate anyone else that is willing and able to run these
test. I find it odd that I was able to run this on 4 completely different
environments and achieve very similar results (~1 sec difference.) While
joe's DC is quite different.
Here is the actual script I ran for the DSP vs ADFind. This is how I
automated the testing.
#######################################################
Param($count=3)
$dn = ([ADSI]"").distinguishedName
function TestOne {
Write-Host " + Test ${i}.1"
Write-Host " + Running ADFind Test"
$joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
"(objectclass=user)" 2>&1 | out-Null }
Write-Host " - $($joetime.TotalSeconds)"
Write-Host " + Running DSP Test"
$DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
Write-Host " - $($DSPTime11.TotalSeconds)"
$myresults = "" | select @{n="ADFind"
;e={$joeTime.TotalSeconds}},
@{n="DSP Using 1.1"
;e={$DSPTime11.TotalSeconds}}
$myresults
}
function TestTwo {
Write-Host " + Test ${i}.2"
Write-Host " + Running DSP Test"
$DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1 }
Write-Host " - $($DSPTime11.TotalSeconds)"
Write-Host " + Running ADFind Test"
$joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
"(objectclass=user)" 2>&1 | out-Null }
Write-Host " - $($joetime.TotalSeconds)"
$myresults = "" | select @{n="ADFind"
;e={$joeTime.TotalSeconds}},
@{n="DSP Using 1.1"
;e={$DSPTime11.TotalSeconds}}
$myresults
}
Write-Host
for($i = 0 ; $i -le $count ; $i++)
{
Write-Host "+ Test $i"
TestOne
TestTwo
}
Write-Host
###############################################################
On Mon, Jun 9, 2008 at 12:07 PM, Guy Teverovsky <guyt@netguru.co.il> wrote:
joe, I've noticed that you've been running the tests from 32bit OS. Not sure
which OS Brandon used for his tests. It definitely could be a cause for some
differences in the speed.
Another potential issue I can see in the tests is the way the commands were
executed. From my experience, object instantiation/initialization has a much
higher performance hit in .NET code. Add to this a possible environment
initialization fat for running a PoS script (please correct me here as I'm
far from being a PoS expert) and you might see quite different numbers
depending how PoS environment is set up.
I think that a fair test would be to compare the following two:
1) Native C++ app that runs all the tests (and reports timing) that does
all the work in the code without any fat of CreateProcess(), etc.
2) PoS script that does the same - the script should not call any
external commands/scripts and should run the tests in a loop, reporting the
timing for each test.
Unfortunately currently I don't even have a compiler and test environment
available to do it myself (don't ask), otherwise I would already be brushing
off the heavy rust from my C++ skills.
Guy
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of joe
Sent: Monday, June 09, 2008 4:18 PM
To: ActiveDir@mail.activedir.org
Subject: RE: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
Read the full blog posting, I detail everything I did and went over it
multiple times prior to posting trying to work out where there might be
issues. I took issue in some of the testing methodology used and detailed
(and IMO corrected) that. Running the tests a lot doesn't automatically make
them better.
Also real life isn't going to have you running commands the way you are
running them at all; there is no way to really get a real life testing that
actually means something relative to the apps. So I did what I did because
running the same command over and over again in sequence helps get
everything out of the way except what the app itself is doing and helps flag
out when you have bad tests popping up. You aren't testing the DS, you are
testing the application/script and until you have consistent numbers, you
don't know if your results are what you are testing or something else
getting in the way.
In this test sequence for PoS
1. 21:09:47 - 21:13:17 = 210 seconds (3 minutes 30 seconds)
2. 21:13:17 - 21:16:47 = 210 seconds (3 minutes 30 seconds)
3. 21:16:47 - 21:20:14 = 207 seconds (3 minutes 27 seconds)
4. 21:20:14 - 21:23:49 = 215 seconds (3 minutes 35 seconds)
5. 21:23:49 - 21:27:18 = 209 seconds (3 minutes 29 seconds)
6. 21:27:18 - 21:30:45 = 207 seconds (3 minutes 27 seconds)
7. 21:30:45 - 21:34:14 = 209 seconds (3 minutes 29 seconds)
8. 21:34:14 - 21:37:50 = 216 seconds (3 minutes 36 seconds)
9. 21:37:50 - 21:41:22 = 212 seconds (3 minutes 32 seconds)
10. 21:41:22 - 21:45:22 = 240 seconds (4 minutes 0 seconds)
You could consider discarding result #10 because it is "off" from the rest,
percentage wise it is still pretty closeΏ] but if you are looking for the
tight grouping of statistics, it feels like it is offΐ]. Something else
likely cropped up during that test that was happening in the DS or on the
network. If 3,4,5 all did that then you would expect something was happening
at the moment that those tests were being run. Splitting the commands up in
the way you did makes it tougher to look for and expect consistent results.
If you were truly shooting for real life, you would have different searches
entirely, but that isn't going to tell you anything in relation to the apps,
but instead the DS perf combined with the apps. You don't have consistency
to see the DS variances.
I kind of wonder how much overhead the PoS is introducing in these tests
versus .NET. Need to build that script into a NET compiled app and do that
testing too.
Ώ] In fact the deviation on the AdFind numbers was higher percentage-wise
between largest delta from low value. Something like 13% for PoS versus 19%
for AdFind though the absolute seconds values were smaller with AdFind but
AdFind was faster overall so less seconds it could deviate to maintain a
tight percentage group of course.
ΐ] Though removing that value only impacts the Average for that sequence by
something like 3 seconds which is still within the deviation within the
other test values.
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Monday, June 09, 2008 9:04 AM
To: ActiveDir@mail.activedir.org
Subject: Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
Hrmm... not sure what your test did differently, but I know mine was more
complete.
I ran the test 60 times on 4 different environments (200k, 300k, 400k, and
700k) consistently the same way each time with the ~ same results each time.
I used production quality boxes and VM's. I used 2008 and 2003. I used the
command locally and remotely. I posted the screen shots and will be blogging
my method tonight.
I believe my methodology of switching up the commands is a better indication
of real life.
I do find it strange that you had such a huge difference in speak though.
Even directorysearcher wasn't that slow. Even thought I tested this in 4
different environments, I am still wanting to see some other production
environments. I am working on two more atm.
On Mon, Jun 9, 2008 at 3:20 AM, joe <listmail@joeware.net> wrote:
http://blog.joeware.net/2008/06/09/1363/
================
PowerShell + S.DS.Protocols Versus AdFind..
So <https://mvp.support.microsoft.com/profile/Brandon> Brandon stepped up
and did something I didn't think anyone would step up and do. Start
profiling and comparing the performance of PowerShell and .NET DS Protocols
against a native App using the wldap32 LDAP libraryΏ]. Brandon's first blog
entry on the topic is here - <http://bsonposh.com/archives/325>
http://bsonposh.com/archives/325
While I am going to question Brandon's testing a little here, especially his
results, I commend him for taking the time and energy to step up and
actually do thisΑ].
<SNIP>
After seeing those numbers I thought wow and then tried it all again but
flipped it around and had PoS run first and then AdFind. In summary the PoS
stuff started at 22:51:49 and finished at 23:30:22, that is 37 minutes and
33 seconds or 2253 seconds or an average of 225.3 seconds for 10 runs
(compared to 21:09:47 to 21:45:22 which is 35 minutes and 35 seconds or 2135
seconds or an average of 213.5 seconds for 10 runs). The AdFind stuff
started at 23:30:22 and finished at 23:37:59, that is 7 minutes 37 seconds
or 457 seconds or an average of 45.7 seconds for 10 runs (compared to
21:02:08 to 21:09:47 which is 7 minutes 39 seconds or 459 seconds or an
average of 45.9 seconds for 10 runs). So we are seeing that AdFind is
between 4.22 and 5.85 times faster than PoS.
<SNIP>
Thoughts?
joe
================
As Brandon said.... "Not bad for a script vs a Compiled app." ;o)
joe
--
O'Reilly Active Directory Third Edition -
http://www.joeware.net/win/ad3e.htm
_____
From: ActiveDir-owner@mail.activedir.org
[mailto:ActiveDir-owner@mail.activedir.org] On Behalf Of Brandon Shell
Sent: Saturday, June 07, 2008 10:32 PM
To: activedir@activedir.org
Subject: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
I have a blog post up about using S.DS.P to do a user count. I appreciate
anyone with a large AD that is will to verify the performance (my test were
done on 4 different Domains ranging from 200k-750k users.
http://bsonposh.com/archives/325
I have already started extensive testing (up to about 240) comparing my DSP
vs ADFind. I will be posting those soon as well.
Here are some sneak peaks of the testing so far (just under a second
difference between the two.) Not bad for a script vs a Compiled app.
http://www.bsonposh.com/uploads/Prod700k.png
http://www.bsonposh.com/uploads/ProdVM400k.png
http://www.bsonposh.com/uploads/PhysicalBox.png
http://www.bsonposh.com/uploads/VMBox.png
Thanks!
| | | |
| bsonposh
Posts:171
| | 07/16/2008 8:28 PM |
| Just added some tweaks to my CSV output as well as outputting headers.
Property : ADFind
Count : 42
Average : *84.4337922357143*
Property : DSP
Count : 42
Average : *68.1905818952381
*
For those that have to look twice... that is DSP 16sec faster than ADFind in
a 400k user environment. I am doing further testing against several other
environments, but looks promising.
On Wed, Jun 11, 2008 at 1:59 PM, Brandon Shell <tshell@gmail.com> wrote:
> ~1 second difference on a 1.1 query outputting the dn.
>
> Funny enough... s.ds.p is actually faster (~5sec in 400k environment ) when
> I start adding properties while outputting csv, but we (joe and I) already
> discussed why that is.
>
> adfind.exe starts gaining ground when I start to output objects instead of
> just text (~10secs slower in 400k environment.)
>
> I am not done with those test (only done like 120 test) which is why I
> havent posted them.
>
> On Wed, Jun 11, 2008 at 1:23 PM, joe <listmail@joeware.net> wrote:
>
>> ~1 second difference for what set of values? When I get a chance to play
>> with it again, maybe this weekend if it is crappy out, I will do the full
>> set of tests but as I mentioned to you and Dean when I realized and
>> mentioned the objectclass=* vs objectclass=user issue to explain the count
>> deviation, I wasn't seeing 1 second differences, I was seeing differences in
>> the 25% range which is multiple seconds. Which is still very good but there
>> is a delta.
>>
>> I don't consider that problem secondary, it directly impacts the results
>> when it happens and doesn't appear to be detectable other than it goes slow
>> as molasses which may not be directly noticable to someone running a
>> specific script the first time.
>>
>>
>> --
>> O'Reilly Active Directory Third Edition -
>> http://www.joeware.net/win/ad3e.htm
>>
>>
>>
>> ------------------------------
>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
>> *Sent:* Wednesday, June 11, 2008 12:56 PM
>>
>> *To:* ActiveDir@mail.activedir.org
>> *Subject:* Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>
>> After about 600+ test on several different environments (different OSs,
>> different shells, different platforms), Deans Test, Several other people,
>> and your test after restarting your shell, we are consistently at ~1 sec
>> difference. I think it is clear S.DS.P is as fast as raw access. I didn't
>> expect it, but I am very happy about it.
>>
>> I am very interested why your shell did that, but that is secondary to the
>> actually orginal test.
>>
>> On Wed, Jun 11, 2008 at 12:25 PM, joe <listmail@joeware.net> wrote:
>>
>>> The data spoke for itself. Now the question is why did it do that in
>>> one instance but not in the next instance I opened. I am not just going to
>>> dismiss it because you didn't like it until I personally have some
>>> understanding of it.
>>>
>>>
>>> --
>>> O'Reilly Active Directory Third Edition -
>>> http://www.joeware.net/win/ad3e.htm
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
>>> *Sent:* Tuesday, June 10, 2008 10:35 AM
>>>
>>> *To:* ActiveDir@mail.activedir.org
>>> *Subject:* Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>
>>> By "fess up", I meant I'd hoped you'd mention that your later tests
>>> were closer to mine and Deans. That said, you spent a tremendous amount of
>>> time on your blog criticizing my testing so I think was hoping for some kind
>>> of retraction on your conclusions.
>>> On Tue, Jun 10, 2008 at 9:33 AM, joe <listmail@joeware.net> wrote:
>>>
>>>> > I am sure joe will fess up why his findings were off
>>>>
>>>> Fess up?? That sounds like I did something wrong...
>>>>
>>>> I am still trying to work out what happened before I write up the
>>>> details, but apparantly PoS got a bug up its ass and the instance I was
>>>> initially running was having issues and going extremely slow whereas another
>>>> instance I opened up and ran next to it ran the script closer to what I was
>>>> expecting for perf (i.e. within about 25%). How you would detect that in the
>>>> normal course I have no idea. Dean/Brandon says it is a glitch, probably
>>>> rare, but I don't think there is a way to ascertain how rare it is unless I
>>>> can find some way to detect it. It seems to run some scripts just fine but
>>>> the AD script it fell flat on its face with. I guess someone could run a
>>>> count the AD objects script when they open the instances to validate you
>>>> have a good one. Or just hope you can identify a bad one when you get one.
>>>>
>>>> I am trying to duplicate the problem. I have managed to slow a new
>>>> instance down but not to the extent that I see in that first instance of
>>>> PoS. I have managed to crash PoS quite a few times though and found that
>>>> when I hit Control Break once it closed the whole instance out instead of
>>>> stopping the script that was running.
>>>>
>>>> One thing that I have found extremely annoying though is that during
>>>> some of the testing, running two instances of the test framework script
>>>> Brandon had on his Blog site is enough to practically kill my test PC which
>>>> is a dual core AMD with 3GB of RAM. The two instances of PoS suck up nearly
>>>> all the RAM until they start throwing out of memory exceptions. As I
>>>> mentioned to Dean at one point... wow, I could be running 400 command
>>>> prompts and not using as much RAM as this one idle PoS instance has tied
>>>> up....
>>>>
>>>> I'll write more but to be completely honest, I am even less enamored
>>>> with PoS and its overall perf now than when I started this test.
>>>>
>>>> --
>>>> O'Reilly Active Directory Third Edition -
>>>> http://www.joeware.net/win/ad3e.htm
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
>>>> *Sent:* Monday, June 09, 2008 7:35 PM
>>>> *To:* ActiveDir@mail.activedir.org
>>>> *Subject:* Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>>
>>>> FYI: http://bsonposh.com/archives/328
>>>>
>>>> summary: My finding stand. S.DS.P is ~1 sec slower than adfind.exe in
>>>> large AD's. as well as some joe rebuttle rebuttles
>>>>
>>>> I am sure joe will fess up why his findings were off
>>>>
>>>> Dean spent most of the day in awe (bit of exeration) that it was
>>>> actually was as fast I stated.
>>>>
>>>> Next step is properties... I think adfind.exe is going to start pulling
>>>> way ahead here, but I was surprised before so lets see.
>>>>
>>>> On Mon, Jun 9, 2008 at 12:22 PM, Brandon Shell <tshell@gmail.com>
>>>> wrote:
>>>>
>>>>> I ran from both x64 and x86 to both x 64 and x86.
>>>>>
>>>>> I am working on a new round of test that change the test to be the same
>>>>> as joe.
>>>>>
>>>>> I would really appreciate anyone else that is willing and able to run
>>>>> these test. I find it odd that I was able to run this on 4 completely
>>>>> different environments and achieve very similar results (~1 sec difference.)
>>>>> While joe's DC is quite different.
>>>>> Here is the actual script I ran for the DSP vs ADFind. This is how I
>>>>> automated the testing.
>>>>>
>>>>> #######################################################
>>>>> Param($count=3)
>>>>> $dn = ([ADSI]"").distinguishedName
>>>>> function TestOne {
>>>>> Write-Host " + Test ${i}.1"
>>>>> Write-Host " + Running ADFind Test"
>>>>> $joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
>>>>> "(objectclass=user)" 2>&1 | out-Null }
>>>>> Write-Host " - $($joetime.TotalSeconds)"
>>>>> Write-Host " + Running DSP Test"
>>>>> $DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1
>>>>> }
>>>>> Write-Host " - $($DSPTime11.TotalSeconds)"
>>>>> $myresults = "" | select @{n="ADFind"
>>>>> ;e={$joeTime.TotalSeconds}},
>>>>> @{n="DSP Using 1.1"
>>>>> ;e={$DSPTime11.TotalSeconds}}
>>>>> $myresults
>>>>> }
>>>>> function TestTwo {
>>>>> Write-Host " + Test ${i}.2"
>>>>> Write-Host " + Running DSP Test"
>>>>> $DSPTime11 = Measure-command { D:\Scripts\Test-DSProtocalsSP.ps1
>>>>> }
>>>>> Write-Host " - $($DSPTime11.TotalSeconds)"
>>>>> Write-Host " + Running ADFind Test"
>>>>> $joeTime = Measure-Command { D:\Scripts\adfind -b "$dn" -c -f
>>>>> "(objectclass=user)" 2>&1 | out-Null }
>>>>> Write-Host " - $($joetime.TotalSeconds)"
>>>>> $myresults = "" | select @{n="ADFind"
>>>>> ;e={$joeTime.TotalSeconds}},
>>>>> @{n="DSP Using 1.1"
>>>>> ;e={$DSPTime11.TotalSeconds}}
>>>>> $myresults
>>>>> }
>>>>> Write-Host
>>>>> for($i = 0 ; $i -le $count ; $i++)
>>>>> {
>>>>> Write-Host "+ Test $i"
>>>>> TestOne
>>>>> TestTwo
>>>>> }
>>>>> Write-Host
>>>>> ###############################################################
>>>>> On Mon, Jun 9, 2008 at 12:07 PM, Guy Teverovsky <guyt@netguru.co.il>
>>>>> wrote:
>>>>>
>>>>>> joe, I've noticed that you've been running the tests from 32bit OS.
>>>>>> Not sure which OS Brandon used for his tests. It definitely could be a cause
>>>>>> for some differences in the speed.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Another potential issue I can see in the tests is the way the commands
>>>>>> were executed. >From my experience, object instantiation/initialization has
>>>>>> a much higher performance hit in .NET code. Add to this a possible
>>>>>> environment initialization fat for running a PoS script (please correct me
>>>>>> here as I'm far from being a PoS expert) and you might see quite different
>>>>>> numbers depending how PoS environment is set up.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I think that a fair test would be to compare the following two:
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1) Native C++ app that runs all the tests (and reports timing)
>>>>>> that does all the work in the code without any fat of CreateProcess(), etc…
>>>>>>
>>>>>> 2) PoS script that does the same – the script should not call any
>>>>>> external commands/scripts and should run the tests in a loop, reporting the
>>>>>> timing for each test.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Unfortunately currently I don't even have a compiler and test
>>>>>> environment available to do it myself (don't ask), otherwise I would already
>>>>>> be brushing off the heavy rust from my C++ skills…
>>>>>>
>>>>>>
>>>>>>
>>>>>> Guy
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>>>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *joe
>>>>>> *Sent:* Monday, June 09, 2008 4:18 PM
>>>>>> *To:* ActiveDir@mail.activedir.org
>>>>>> *Subject:* RE: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>>>>
>>>>>>
>>>>>>
>>>>>> Read the full blog posting, I detail everything I did and went over it
>>>>>> multiple times prior to posting trying to work out where there might be
>>>>>> issues. I took issue in some of the testing methodology used and detailed
>>>>>> (and IMO corrected) that. Running the tests a lot doesn't automatically make
>>>>>> them better.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Also real life isn't going to have you running commands the way you
>>>>>> are running them at all; there is no way to really get a real life testing
>>>>>> that actually means something relative to the apps. So I did what I did
>>>>>> because running the same command over and over again in sequence helps get
>>>>>> everything out of the way except what the app itself is doing and helps flag
>>>>>> out when you have bad tests popping up. You aren't testing the DS, you are
>>>>>> testing the application/script and until you have consistent numbers, you
>>>>>> don't know if your results are what you are testing or something else
>>>>>> getting in the way.
>>>>>>
>>>>>>
>>>>>>
>>>>>> In this test sequence for PoS
>>>>>>
>>>>>> 1. 21:09:47 - 21:13:17 = 210 seconds (3 minutes 30 seconds)
>>>>>> 2. 21:13:17 - 21:16:47 = 210 seconds (3 minutes 30 seconds)
>>>>>> 3. 21:16:47 - 21:20:14 = 207 seconds (3 minutes 27 seconds)
>>>>>> 4. 21:20:14 - 21:23:49 = 215 seconds (3 minutes 35 seconds)
>>>>>> 5. 21:23:49 - 21:27:18 = 209 seconds (3 minutes 29 seconds)
>>>>>> 6. 21:27:18 - 21:30:45 = 207 seconds (3 minutes 27 seconds)
>>>>>> 7. 21:30:45 - 21:34:14 = 209 seconds (3 minutes 29 seconds)
>>>>>> 8. 21:34:14 - 21:37:50 = 216 seconds (3 minutes 36 seconds)
>>>>>> 9. 21:37:50 - 21:41:22 = 212 seconds (3 minutes 32 seconds)
>>>>>> 10. 21:41:22 - 21:45:22 = 240 seconds (4 minutes 0 seconds)
>>>>>>
>>>>>>
>>>>>>
>>>>>> You could consider discarding result #10 because it is "off" from the
>>>>>> rest, percentage wise it is still pretty closeΏ] but if you are looking for
>>>>>> the tight grouping of statistics, it feels like it is offΐ]. Something else
>>>>>> likely cropped up during that test that was happening in the DS or on the
>>>>>> network. If 3,4,5 all did that then you would expect something was happening
>>>>>> at the moment that those tests were being run. Splitting the commands up in
>>>>>> the way you did makes it tougher to look for and expect consistent results.
>>>>>> If you were truly shooting for real life, you would have different searches
>>>>>> entirely, but that isn't going to tell you anything in relation to the apps,
>>>>>> but instead the DS perf combined with the apps. You don't have consistency
>>>>>> to see the DS variances.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I kind of wonder how much overhead the PoS is introducing in these
>>>>>> tests versus .NET. Need to build that script into a NET compiled app and do
>>>>>> that testing too.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ώ] In fact the deviation on the AdFind numbers was higher
>>>>>> percentage-wise between largest delta from low value. Something like 13% for
>>>>>> PoS versus 19% for AdFind though the absolute seconds values were smaller
>>>>>> with AdFind but AdFind was faster overall so less seconds it could deviate
>>>>>> to maintain a tight percentage group of course.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ΐ] Though removing that value only impacts the Average for that
>>>>>> sequence by something like 3 seconds which is still within the deviation
>>>>>> within the other test values.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> O'Reilly Active Directory Third Edition -
>>>>>> http://www.joeware.net/win/ad3e.htm
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>>>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
>>>>>> *Sent:* Monday, June 09, 2008 9:04 AM
>>>>>> *To:* ActiveDir@mail.activedir.org
>>>>>> *Subject:* Re: [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>>>>
>>>>>> Hrmm... not sure what your test did differently, but I know mine was
>>>>>> more complete.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I ran the test 60 times on 4 different environments (200k, 300k, 400k,
>>>>>> and 700k) consistently the same way each time with the ~ same results each
>>>>>> time. I used production quality boxes and VM's. I used 2008 and 2003. I used
>>>>>> the command locally and remotely. I posted the screen shots and will be
>>>>>> blogging my method tonight.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I believe my methodology of switching up the commands is a better
>>>>>> indication of real life.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I do find it strange that you had such a huge difference in speak
>>>>>> though. Even directorysearcher wasn't that slow. Even thought I tested this
>>>>>> in 4 different environments, I am still wanting to see some other production
>>>>>> environments. I am working on two more atm.
>>>>>>
>>>>>> On Mon, Jun 9, 2008 at 3:20 AM, joe <listmail@joeware.net> wrote:
>>>>>>
>>>>>> http://blog.joeware.net/2008/06/09/1363/
>>>>>>
>>>>>>
>>>>>>
>>>>>> ================
>>>>>>
>>>>>>
>>>>>>
>>>>>> *PowerShell + S.DS.Protocols Versus AdFind….*
>>>>>>
>>>>>>
>>>>>>
>>>>>> So Brandon <https://mvp.support.microsoft.com/profile/Brandon>stepped up and did something I didn't think anyone would step up and do…
>>>>>> Start profiling and comparing the performance of PowerShell and .NET DS
>>>>>> Protocols against a native App using the wldap32 LDAP libraryΏ]. Brandon's
>>>>>> first blog entry on the topic is here -
>>>>>> http://bsonposh.com/archives/325
>>>>>>
>>>>>> While I am going to question Brandon's testing a little here,
>>>>>> especially his results, I commend him for taking the time and energy to step
>>>>>> up and actually do thisΑ].
>>>>>>
>>>>>> <SNIP>
>>>>>>
>>>>>>
>>>>>>
>>>>>> After seeing those numbers I thought wow and then tried it all again
>>>>>> but flipped it around and had PoS run first and then AdFind. In summary the
>>>>>> PoS stuff started at 22:51:49 and finished at 23:30:22, that is 37 minutes
>>>>>> and 33 seconds or 2253 seconds or an average of 225.3 seconds for 10 runs
>>>>>> (compared to 21:09:47 to 21:45:22 which is 35 minutes and 35 seconds or 2135
>>>>>> seconds or an average of 213.5 seconds for 10 runs). The AdFind stuff
>>>>>> started at 23:30:22 and finished at 23:37:59, that is 7 minutes 37 seconds
>>>>>> or 457 seconds or an average of 45.7 seconds for 10 runs (compared to
>>>>>> 21:02:08 to 21:09:47 which is 7 minutes 39 seconds or 459 seconds or an
>>>>>> average of 45.9 seconds for 10 runs). *So we are seeing that AdFind
>>>>>> is between 4.22 and 5.85 times faster than PoS.*
>>>>>>
>>>>>>
>>>>>>
>>>>>> <SNIP>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> joe
>>>>>>
>>>>>> ================
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> As Brandon said.... "Not bad for a script vs a Compiled app." ;o)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> joe
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> O'Reilly Active Directory Third Edition -
>>>>>> http://www.joeware.net/win/ad3e.htm
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> *From:* ActiveDir-owner@mail.activedir.org [mailto:
>>>>>> ActiveDir-owner@mail.activedir.org] *On Behalf Of *Brandon Shell
>>>>>> *Sent:* Saturday, June 07, 2008 10:32 PM
>>>>>> *To:* activedir@activedir.org
>>>>>> *Subject:* [ActiveDir] [OT: Slightly] S.DS.P Performance Test
>>>>>>
>>>>>> I have a blog post up about using S.DS.P to do a user count. I
>>>>>> appreciate anyone with a large AD that is will to verify the performance (my
>>>>>> test were done on 4 different Domains ranging from 200k-750k users.
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://bsonposh.com/archives/325
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have already started extensive testing (up to about 240) comparing
>>>>>> my DSP vs ADFind. I will be posting those soon as well.
>>>>>>
>>>>>&g |
|
|