| Author | Messages | |
chaselton
Posts:78
 | | 07/20/2010 3:54 PM |
| Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| skradel
Posts:177
| | 07/20/2010 4:05 PM |
| The 12-hour thing suggests that ldifde is trying to parse your file as one
gigantic entry, although even then, 12 hours is outrageously excessive for
66k entries. Might want to take a look in a bit editor like xvi32 and see
if your line endings mismatch those produced by a sample ldifde export. I
would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes
(admincount, lastlogontimestamp, dscorepropagationdata, whencreated,
whenchanged, objectcategory, instancetype, usncreated, usnchanged,
instancetype) as the import will fail on certain of these, and some others
are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a
small subset of the file, say, 3 or 4 entries. This will help you detect
syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton
<chaselton@uchicago.edu>wrote:
> Hello,
>
> I’m using LDIFDE to export and import about 66,000 users, from a domain in
> one forest to a domain in a separate forest. I used the following command
> to export the users:
>
>
>
> ldifde –n –f C:\filename.ldf –s servername –r “(objectCategory=person)” –o
> “exclusions”
>
>
>
> and did a find and replace on the ldf file to change all of the references
> from the export domain to the import domain. I also did a find and replace
> to add a “-“ after each entry and a blank space after the “-“. Here’s a
> sample entry in the resulting file, with sensitive info obfuscated:
>
>
>
> dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> cn: username
>
> instanceType: 4
>
> whenCreated: 20080129160427.0Z
>
> whenChanged: 20100503184651.0Z
>
> displayName: username
>
> uSNCreated: 5963864
>
> uSNChanged: 45461899
>
> name: username
>
> codePage: 0
>
> countryCode: 0
>
> userWorkstations: WORKSTATION
>
> adminCount: 1
>
> accountExpires: 9223372036854775807
>
> sAMAccountName: username
>
> userPrincipalName: username@domain.local
>
> objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 16010108151056.0Z
>
> lastLogonTimestamp: 129173852491219051
>
> -
>
>
>
> I used the following command to import the file:
>
>
>
> ldifde -i -f filename.ldf -j C:\ -v -u -q 2
>
>
>
> I’m running into two issues. The first is that the import starts, then
> displays the “Loading entries” status message for about 12 hours. This
> wouldn’t be a problem if the import was successful, or even partially
> successful, but it always fails with the following error:
>
> There is a syntax error in the input file
>
> Failed on line 1. The last token starts with '*char*'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> The *char* value changes every time the script is run. I’ve done a
> search on this error and the few things I’ve found I’ve already implemented
> or don’t apply to my situation.
>
>
>
> What I’m wondering is, does ldifde have a size limit for the input file?
> Does it load every entry in the ldf file into memory before writing the
> changes to the directory (which would explain the 12 hour run time for
> 66,000 accounts)? Is there anything obvious I’ve done wrong?
>
>
>
> Thanks in advance for any help given.
>
>
>
> CJH
>
>
>
| | | |
| RobSilver
Posts:0
| | 07/20/2010 4:39 PM |
| Do you have any "non-related" errors in the system logs at the time of this issue.
It could be a memory/page file issue, but that's just a guess based on the symptoms and the different 'char' changing on each attempt. Have you kept an eye on the system performance (memory) while this is running?
Rob Silver<http://robsilver.org/>
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: 20 July 2010 3:54 PM
To: 'ActiveDir@mail.activedir.org'
Subject: [ActiveDir] LDIFDE Issue
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| chaselton
Posts:78
| | 07/20/2010 6:08 PM |
| For some reason I haven't received any mail from the mailing list since Sunday so I'm posting here.
Processor and memory usage is high, but there are no errors in the event log. I'll try omitting the suggested attributes, paring the list down and trying again. | | | |
| hatstand
Posts:1
| | 07/20/2010 7:32 PM |
| I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org [activedir-owner@mail.activedir.org] on behalf of Steve Kradel [skradel@zetetic.net]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I’m using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde –n –f C:\filename.ldf –s servername –r “(objectCategory=person)” –o “exclusions”
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a “-“ after each entry and a blank space after the “-“. Here’s a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I’m running into two issues. The first is that the import starts, then displays the “Loading entries” status message for about 12 hours. This wouldn’t be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I’ve done a search on this error and the few things I’ve found I’ve already implemented or don’t apply to my situation.
What I’m wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I’ve done wrong?
Thanks in advance for any help given.
CJH
| | | |
| kurtbuff
Posts:200
| | 07/21/2010 3:34 AM |
| Stupid question:
Should this:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
instead be this:
dn: "CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local"
because of spaces?
Kurt
On Tue, Jul 20, 2010 at 07:53, Cynthia Haselton <chaselton@uchicago.edu> wrote:
> Hello,
>
> I’m using LDIFDE to export and import about 66,000 users, from a domain in
> one forest to a domain in a separate forest. I used the following command
> to export the users:
>
>
>
> ldifde –n –f C:\filename.ldf –s servername –r “(objectCategory=person)” –o
> “exclusions”
>
>
>
> and did a find and replace on the ldf file to change all of the references
> from the export domain to the import domain. I also did a find and replace
> to add a “-“ after each entry and a blank space after the “-“. Here’s a
> sample entry in the resulting file, with sensitive info obfuscated:
>
>
>
> dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> cn: username
>
> instanceType: 4
>
> whenCreated: 20080129160427.0Z
>
> whenChanged: 20100503184651.0Z
>
> displayName: username
>
> uSNCreated: 5963864
>
> uSNChanged: 45461899
>
> name: username
>
> codePage: 0
>
> countryCode: 0
>
> userWorkstations: WORKSTATION
>
> adminCount: 1
>
> accountExpires: 9223372036854775807
>
> sAMAccountName: username
>
> userPrincipalName: username@domain.local
>
> objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 16010108151056.0Z
>
> lastLogonTimestamp: 129173852491219051
>
> -
>
>
>
> I used the following command to import the file:
>
>
>
> ldifde -i -f filename.ldf -j C:\ -v -u -q 2
>
>
>
> I’m running into two issues. The first is that the import starts, then
> displays the “Loading entries” status message for about 12 hours. This
> wouldn’t be a problem if the import was successful, or even partially
> successful, but it always fails with the following error:
>
> There is a syntax error in the input file
>
> Failed on line 1. The last token starts with 'char'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> The char value changes every time the script is run. I’ve done a search on
> this error and the few things I’ve found I’ve already implemented or don’t
> apply to my situation.
>
>
>
> What I’m wondering is, does ldifde have a size limit for the input file?
> Does it load every entry in the ldf file into memory before writing the
> changes to the directory (which would explain the 12 hour run time for
> 66,000 accounts)? Is there anything obvious I’ve done wrong?
>
>
>
> Thanks in advance for any help given.
>
>
>
> CJH
>
>
| | | |
| TG
Posts:298
| | 07/21/2010 3:50 AM |
| Yes.
----- Original Message -----
From: "Kurt Buff" [kurt.buff@gmail.com]
Sent: 07/20/2010 07:33 PM MST
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
Stupid question:
Should this:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
instead be this:
dn: "CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local"
because of spaces?
Kurt
On Tue, Jul 20, 2010 at 07:53, Cynthia Haselton <chaselton@uchicago.edu> wrote:
> Hello,
>
> I’m using LDIFDE to export and import about 66,000 users, from a domain in
> one forest to a domain in a separate forest. I used the following command
> to export the users:
>
>
>
> ldifde –n –f C:\filename.ldf –s servername –r “(objectCategory=person)” –o
> “exclusions”
>
>
>
> and did a find and replace on the ldf file to change all of the references
> from the export domain to the import domain. I also did a find and replace
> to add a “-“ after each entry and a blank space after the “-“. Here’s a
> sample entry in the resulting file, with sensitive info obfuscated:
>
>
>
> dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> cn: username
>
> instanceType: 4
>
> whenCreated: 20080129160427.0Z
>
> whenChanged: 20100503184651.0Z
>
> displayName: username
>
> uSNCreated: 5963864
>
> uSNChanged: 45461899
>
> name: username
>
> codePage: 0
>
> countryCode: 0
>
> userWorkstations: WORKSTATION
>
> adminCount: 1
>
> accountExpires: 9223372036854775807
>
> sAMAccountName: username
>
> userPrincipalName: username@domain.local
>
> objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 16010108151056.0Z
>
> lastLogonTimestamp: 129173852491219051
>
> -
>
>
>
> I used the following command to import the file:
>
>
>
> ldifde -i -f filename.ldf -j C:\ -v -u -q 2
>
>
>
> I’m running into two issues. The first is that the import starts, then
> displays the “Loading entries” status message for about 12 hours. This
> wouldn’t be a problem if the import was successful, or even partially
> successful, but it always fails with the following error:
>
> There is a syntax error in the input file
>
> Failed on line 1. The last token starts with 'char'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> The char value changes every time the script is run. I’ve done a search on
> this error and the few things I’ve found I’ve already implemented or don’t
> apply to my situation.
>
>
>
> What I’m wondering is, does ldifde have a size limit for the input file?
> Does it load every entry in the ldf file into memory before writing the
> changes to the directory (which would explain the 12 hour run time for
> 66,000 accounts)? Is there anything obvious I’ve done wrong?
>
>
>
> Thanks in advance for any help given.
>
>
>
> CJH
>
>
The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. You are deemed to have accepted these risks if you communicate with us by e-mail.
| | | |
| skradel
Posts:177
| | 07/21/2010 4:06 AM |
| In LDIF? Nope. In CSV, yes.
See: http://www.ietf.org/rfc/rfc2849.txt
On Tue, Jul 20, 2010 at 10:52 PM, Tony Gordon <Tony.Gordon@hewitt.com>wrote:
> Yes.
>
>
> ----- Original Message -----
> From: "Kurt Buff" [kurt.buff@gmail.com]
> Sent: 07/20/2010 07:33 PM MST
> To: activedir@mail.activedir.org
> Subject: Re: [ActiveDir] LDIFDE Issue
>
>
>
> Stupid question:
>
> Should this:
>
> dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> instead be this:
>
> dn: "CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local"
>
> because of spaces?
>
> Kurt
>
>
| | | |
| gazzadownunder
Posts:10
| | 07/21/2010 9:16 AM |
|
Any time I do anything with LDIFDE, I always start small just import a
few objects to make sure you have the correct options set and you get
the desired results.
I also use the -k and -j options so it skips any over any minor errors and
produce a log file with the details of the error and data that it's using to
create the object. I mention this as sometime LDIFDE will change the
format\data during the import and it maybe doing something you didn't expect.
Gary.
________________________________
From: Steve Kradel <skradel@zetetic.net>
To: activedir@mail.activedir.org
Sent: Tue, 20 July, 2010 4:04:24 PM
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one
gigantic entry, although even then, 12 hours is outrageously excessive for 66k
entries. Might want to take a look in a bit editor like xvi32 and see if your
line endings mismatch those produced by a sample ldifde export. I would also
omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount,
lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged,
objectcategory, instancetype, usncreated, usnchanged, instancetype) as the
import will fail on certain of these, and some others are simply undesirable to
import.
Finally, rather than trying to apply 66,000 users all at once, test with a small
subset of the file, say, 3 or 4 entries. This will help you detect syntax
errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu>
wrote:
Hello,
>I’m using LDIFDE to export and import about 66,000 users, from a domain in one
>forest to a domain in a separate forest. I used the following command to export
>the users:
>
>ldifde –n –f C:\filename.ldf –s servername –r “(objectCategory=person)” –o
>“exclusions”
>
>and did a find and replace on the ldf file to change all of the references from
>the export domain to the import domain. I also did a find and replace to add a
>“-“ after each entry and a blank space after the “-“. Here’s a sample entry in
>the resulting file, with sensitive info obfuscated:
>
>dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>changetype: add
>objectClass: top
>objectClass: person
>objectClass: organizationalPerson
>objectClass: user
>cn: username
>instanceType: 4
>whenCreated: 20080129160427.0Z
>whenChanged: 20100503184651.0Z
>displayName: username
>uSNCreated: 5963864
>uSNChanged: 45461899
>name: username
>codePage: 0
>countryCode: 0
>userWorkstations: WORKSTATION
>adminCount: 1
>accountExpires: 9223372036854775807
>sAMAccountName: username
>userPrincipalName: username@domain.local
>objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
>dSCorePropagationData: 20090519185234.0Z
>dSCorePropagationData: 20090519185234.0Z
>dSCorePropagationData: 20090519185234.0Z
>dSCorePropagationData: 16010108151056.0Z
>lastLogonTimestamp: 129173852491219051
>-
>
>I used the following command to import the file:
>
>ldifde -i -f filename.ldf -j C:\ -v -u -q 2
>
>I’m running into two issues. The first is that the import starts, then displays
>the “Loading entries” status message for about 12 hours. This wouldn’t be a
>problem if the import was successful, or even partially successful, but it
>always fails with the following error:
>There is a syntax error in the input file
>Failed on line 1. The last token starts with 'char'.
>0 entries modified successfully.
>An error has occurred in the program
>
>The char value changes every time the script is run. I’ve done a search on
>this error and the few things I’ve found I’ve already implemented or don’t apply
>to my situation.
>
>What I’m wondering is, does ldifde have a size limit for the input file? Does
>it load every entry in the ldf file into memory before writing the changes to
>the directory (which would explain the 12 hour run time for 66,000 accounts)?
>Is there anything obvious I’ve done wrong?
>
>Thanks in advance for any help given.
>
>CJH
>
| | | |
| chaselton
Posts:78
| | 07/21/2010 4:21 PM |
| Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org [activedir-owner@mail.activedir.org] on behalf of Steve Kradel [skradel@zetetic.net]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| chaselton
Posts:78
| | 07/21/2010 6:48 PM |
| This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new file and saved it with a different file name. I ran the following command...
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
...and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running the program. Same error as before. I then opened the file and saved it in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I'm going to try eliminating the -u option (I read in a post that using -u with a Unicode file causes an error) and see if that works. If it doesn't, not sure what else to do, other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org [activedir-owner@mail.activedir.org] on behalf of Steve Kradel [skradel@zetetic.net]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| chaselton
Posts:78
| | 07/21/2010 7:02 PM |
| Here's a sample entry after I deleted as many attributes as I could. This is the first entry in the smaller .ldf file; the rest of the entries are unmodified. I figured I would stick to changing one entry until I figured out the problem.
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: User One
sn: One
givenName: User
distinguishedName: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
displayName: User One
name: User One
sAMAccountName: _root_userone
-
Running the command "ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k -q 2", minus the quotes, produces the following error
There is a syntax error in the input file
Failed on line 14. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
It doesn't matter which attributes I remove, I still run into this error. The only thing that changes is the line number in the "Failed on line..." error and the character in the "The last token starts with..."
At this point I'm thinking of ditching ldifde for another import utility. Anyone have recommendations?
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:47 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new file and saved it with a different file name. I ran the following command...
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
...and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running the program. Same error as before. I then opened the file and saved it in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I'm going to try eliminating the -u option (I read in a post that using -u with a Unicode file causes an error) and see if that works. If it doesn't, not sure what else to do, other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org [activedir-owner@mail.activedir.org] on behalf of Steve Kradel [skradel@zetetic.net]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| jitendrakalyankar
Posts:3
| | 07/21/2010 7:27 PM |
| Instead of sAMAccountName: _root_userone try using sAMAccountName:
root_userone...not an expert but just thought it may help...
Kind regards,
Jitendra Kalyankar
On Wed, Jul 21, 2010 at 11:31 PM, Cynthia Haselton
<chaselton@uchicago.edu>wrote:
> Here’s a sample entry after I deleted as many attributes as I could.
> This is the first entry in the smaller .ldf file; the rest of the entries
> are unmodified. I figured I would stick to changing one entry until I
> figured out the problem.
>
>
>
> dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> cn: User One
>
> sn: One
>
> givenName: User
>
> distinguishedName: CN=User One,OU=Users and
> Groups,OU=DCS,DC=domain,DC=local
>
> displayName: User One
>
> name: User One
>
> sAMAccountName: _root_userone
>
> -
>
>
>
> Running the command “ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k
> -q 2”, minus the quotes, produces the following error
>
>
>
> There is a syntax error in the input file
>
> Failed on line 14. The last token starts with '_'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> It doesn’t matter which attributes I remove, I still run into this error.
> The only thing that changes is the line number in the “Failed on line…”
> error and the character in the “The last token starts with…”
>
>
>
> At this point I’m thinking of ditching ldifde for another import utility.
> Anyone have recommendations?
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Cynthia Haselton
> *Sent:* Wednesday, July 21, 2010 12:47 PM
>
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> This is getting a bit ridiculous
>
> I copied a bunch of the entries from the larger file, pasted them in a new
> file and saved it with a different file name. I ran the following command…
>
>
>
> ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
>
>
>
> …and got the same error as before.
>
>
>
> There is a syntax error in the input file
>
> Failed on line 1. The last token starts with '*char*'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> I then tried removing some attributes from the first entry and re-running
> the program. Same error as before. I then opened the file and saved it in
> Unicode format. This time I got a different version of the same error:
>
>
>
> There is a syntax error in the input file
>
> Failed on line 15. The last token starts with '_'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> This is frustrating me to no end. I’m going to try eliminating the –u
> option (I read in a post that using –u with a Unicode file causes an error)
> and see if that works. If it doesn’t, not sure what else to do, other than
> deleting attributes until the import actually works.
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Cynthia Haselton
> *Sent:* Wednesday, July 21, 2010 10:21 AM
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> Finally got the replies…
>
> The RID master is on the DC from which I’m doing the import. I’m using the
> –j option but will add the –k…though I thought I saw an article that
> discouraged using it. I’d have to dig for it, and I’d rather fix this issue
> than do that.
>
>
>
> I’m still doing the large file import (with all 66,000+ users) as I haven’t
> found an easy way to split the large file into smaller ones, or filter the
> search so that it returns a small amount of users to work with. Will update
> with results.
>
>
>
> Thanks for all the help,
>
> CJH
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Chris Boller
> *Sent:* Tuesday, July 20, 2010 1:24 PM
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> I agree with Steve omitting those attributes and trying to import just a
> few
>
>
>
> Also, if you're creating a huge load of objects it might be worth moving
> the RID master role to that DC as I've seen it where it gets exhausted and
> bombs out.
>
>
>
> Chris
>
>
> ------------------------------
>
> *From:* activedir-owner@mail.activedir.org [
> activedir-owner@mail.activedir.org] on behalf of Steve Kradel [
> skradel@zetetic.net]
> *Sent:* 20 July 2010 16:04
> *To:* activedir@mail.activedir.org
> *Subject:* Re: [ActiveDir] LDIFDE Issue
>
> The 12-hour thing suggests that ldifde is trying to parse your file as one
> gigantic entry, although even then, 12 hours is outrageously excessive for
> 66k entries. Might want to take a look in a bit editor like xvi32 and see
> if your line endings mismatch those produced by a sample ldifde export. I
> would also omit the changetype / '-' syntax as an unnecessary complication.
>
>
>
> Additionally, I would advise you to omit system-managed attributes
> (admincount, lastlogontimestamp, dscorepropagationdata, whencreated,
> whenchanged, objectcategory, instancetype, usncreated, usnchanged,
> instancetype) as the import will fail on certain of these, and some others
> are simply undesirable to import.
>
> Finally, rather than trying to apply 66,000 users all at once, test with a
> small subset of the file, say, 3 or 4 entries. This will help you detect
> syntax errors and schema violations more easily.
>
>
>
> --Steve
>
>
>
> On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu>
> wrote:
>
> Hello,
>
> I’m using LDIFDE to export and import about 66,000 users, from a domain in
> one forest to a domain in a separate forest. I used the following command
> to export the users:
>
>
>
> ldifde –n –f C:\filename.ldf –s servername –r “(objectCategory=person)” –o
> “exclusions”
>
>
>
> and did a find and replace on the ldf file to change all of the references
> from the export domain to the import domain. I also did a find and replace
> to add a “-“ after each entry and a blank space after the “-“. Here’s a
> sample entry in the resulting file, with sensitive info obfuscated:
>
>
>
> dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> cn: username
>
> instanceType: 4
>
> whenCreated: 20080129160427.0Z
>
> whenChanged: 20100503184651.0Z
>
> displayName: username
>
> uSNCreated: 5963864
>
> uSNChanged: 45461899
>
> name: username
>
> codePage: 0
>
> countryCode: 0
>
> userWorkstations: WORKSTATION
>
> adminCount: 1
>
> accountExpires: 9223372036854775807
>
> sAMAccountName: username
>
> userPrincipalName: username@domain.local
>
> objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 16010108151056.0Z
>
> lastLogonTimestamp: 129173852491219051
>
> -
>
>
>
> I used the following command to import the file:
>
>
>
> ldifde -i -f filename.ldf -j C:\ -v -u -q 2
>
>
>
> I’m running into two issues. The first is that the import starts, then
> displays the “Loading entries” status message for about 12 hours. This
> wouldn’t be a problem if the import was successful, or even partially
> successful, but it always fails with the following error:
>
> There is a syntax error in the input file
>
> Failed on line 1. The last token starts with '*char*'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> The *char* value changes every time the script is run. I’ve done a
> search on this error and the few things I’ve found I’ve already implemented
> or don’t apply to my situation.
>
>
>
> What I’m wondering is, does ldifde have a size limit for the input file?
> Does it load every entry in the ldf file into memory before writing the
> changes to the directory (which would explain the 12 hour run time for
> 66,000 accounts)? Is there anything obvious I’ve done wrong?
>
>
>
> Thanks in advance for any help given.
>
>
>
> CJH
>
>
>
>
>
| | | |
| chaselton
Posts:78
| | 07/21/2010 7:29 PM |
| In one last attempt, I deleted all attributes except for the initial dn:. Here's what the entry looked like
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
-
This is the error I got:
There is a syntax error in the input file
Failed on line 7. The last token starts with 'u'.
0 entries modified successfully.
An error has occurred in the program
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 1:02 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Here's a sample entry after I deleted as many attributes as I could. This is the first entry in the smaller .ldf file; the rest of the entries are unmodified. I figured I would stick to changing one entry until I figured out the problem.
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: User One
sn: One
givenName: User
distinguishedName: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
displayName: User One
name: User One
sAMAccountName: _root_userone
-
Running the command "ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k -q 2", minus the quotes, produces the following error
There is a syntax error in the input file
Failed on line 14. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
It doesn't matter which attributes I remove, I still run into this error. The only thing that changes is the line number in the "Failed on line..." error and the character in the "The last token starts with..."
At this point I'm thinking of ditching ldifde for another import utility. Anyone have recommendations?
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:47 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new file and saved it with a different file name. I ran the following command...
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
...and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running the program. Same error as before. I then opened the file and saved it in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I'm going to try eliminating the -u option (I read in a post that using -u with a Unicode file causes an error) and see if that works. If it doesn't, not sure what else to do, other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org [activedir-owner@mail.activedir.org] on behalf of Steve Kradel [skradel@zetetic.net]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| rwilper
Posts:37
| | 07/21/2010 7:37 PM |
| I thought that you only needed the "-" on a line by itself to end an entry if the changetype is "modify"
-Ross
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 11:28 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
In one last attempt, I deleted all attributes except for the initial dn:. Here's what the entry looked like
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
-
This is the error I got:
There is a syntax error in the input file
Failed on line 7. The last token starts with 'u'.
0 entries modified successfully.
An error has occurred in the program
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 1:02 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Here's a sample entry after I deleted as many attributes as I could. This is the first entry in the smaller .ldf file; the rest of the entries are unmodified. I figured I would stick to changing one entry until I figured out the problem.
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: User One
sn: One
givenName: User
distinguishedName: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
displayName: User One
name: User One
sAMAccountName: _root_userone
-
Running the command "ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k -q 2", minus the quotes, produces the following error
There is a syntax error in the input file
Failed on line 14. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
It doesn't matter which attributes I remove, I still run into this error. The only thing that changes is the line number in the "Failed on line..." error and the character in the "The last token starts with..."
At this point I'm thinking of ditching ldifde for another import utility. Anyone have recommendations?
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:47 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new file and saved it with a different file name. I ran the following command...
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
...and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running the program. Same error as before. I then opened the file and saved it in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I'm going to try eliminating the -u option (I read in a post that using -u with a Unicode file causes an error) and see if that works. If it doesn't, not sure what else to do, other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org [activedir-owner@mail.activedir.org] on behalf of Steve Kradel [skradel@zetetic.net]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| skradel
Posts:177
| | 07/21/2010 7:41 PM |
| Agreed, the - token is to indicate the end of a "mod-attribute-value"
section, not for an entry-level add.
--Steve
On Wed, Jul 21, 2010 at 2:35 PM, Wilper, Ross A <rwilper@stanford.edu>wrote:
> I thought that you only needed the “-“ on a line by itself to end an
> entry if the changetype is “modify”
>
>
>
> -Ross
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Cynthia Haselton
> *Sent:* Wednesday, July 21, 2010 11:28 AM
>
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> In one last attempt, I deleted all attributes except for the initial dn:.
> Here’s what the entry looked like
>
>
>
> dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> -
>
>
>
> This is the error I got:
>
>
>
> There is a syntax error in the input file
>
> Failed on line 7. The last token starts with 'u'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Cynthia Haselton
> *Sent:* Wednesday, July 21, 2010 1:02 PM
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> Here’s a sample entry after I deleted as many attributes as I could. This
> is the first entry in the smaller .ldf file; the rest of the entries are
> unmodified. I figured I would stick to changing one entry until I figured
> out the problem.
>
>
>
> dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> cn: User One
>
> sn: One
>
> givenName: User
>
> distinguishedName: CN=User One,OU=Users and
> Groups,OU=DCS,DC=domain,DC=local
>
> displayName: User One
>
> name: User One
>
> sAMAccountName: _root_userone
>
> -
>
>
>
> Running the command “ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k
> -q 2”, minus the quotes, produces the following error
>
>
>
> There is a syntax error in the input file
>
> Failed on line 14. The last token starts with '_'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> It doesn’t matter which attributes I remove, I still run into this error.
> The only thing that changes is the line number in the “Failed on line…”
> error and the character in the “The last token starts with…”
>
>
>
> At this point I’m thinking of ditching ldifde for another import utility.
> Anyone have recommendations?
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Cynthia Haselton
> *Sent:* Wednesday, July 21, 2010 12:47 PM
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> This is getting a bit ridiculous
>
> I copied a bunch of the entries from the larger file, pasted them in a new
> file and saved it with a different file name. I ran the following command…
>
>
>
> ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
>
>
>
> …and got the same error as before.
>
>
>
> There is a syntax error in the input file
>
> Failed on line 1. The last token starts with '*char*'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> I then tried removing some attributes from the first entry and re-running
> the program. Same error as before. I then opened the file and saved it in
> Unicode format. This time I got a different version of the same error:
>
>
>
> There is a syntax error in the input file
>
> Failed on line 15. The last token starts with '_'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> This is frustrating me to no end. I’m going to try eliminating the –u
> option (I read in a post that using –u with a Unicode file causes an error)
> and see if that works. If it doesn’t, not sure what else to do, other than
> deleting attributes until the import actually works.
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Cynthia Haselton
> *Sent:* Wednesday, July 21, 2010 10:21 AM
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> Finally got the replies…
>
> The RID master is on the DC from which I’m doing the import. I’m using the
> –j option but will add the –k…though I thought I saw an article that
> discouraged using it. I’d have to dig for it, and I’d rather fix this issue
> than do that.
>
>
>
> I’m still doing the large file import (with all 66,000+ users) as I haven’t
> found an easy way to split the large file into smaller ones, or filter the
> search so that it returns a small amount of users to work with. Will update
> with results.
>
>
>
> Thanks for all the help,
>
> CJH
>
>
>
> *From:* activedir-owner@mail.activedir.org [mailto:
> activedir-owner@mail.activedir.org] *On Behalf Of *Chris Boller
> *Sent:* Tuesday, July 20, 2010 1:24 PM
> *To:* activedir@mail.activedir.org
> *Subject:* RE: [ActiveDir] LDIFDE Issue
>
>
>
> I agree with Steve omitting those attributes and trying to import just a
> few
>
>
>
> Also, if you're creating a huge load of objects it might be worth moving
> the RID master role to that DC as I've seen it where it gets exhausted and
> bombs out.
>
>
>
> Chris
>
>
> ------------------------------
>
> *From:* activedir-owner@mail.activedir.org [
> activedir-owner@mail.activedir.org] on behalf of Steve Kradel [
> skradel@zetetic.net]
> *Sent:* 20 July 2010 16:04
> *To:* activedir@mail.activedir.org
> *Subject:* Re: [ActiveDir] LDIFDE Issue
>
> The 12-hour thing suggests that ldifde is trying to parse your file as one
> gigantic entry, although even then, 12 hours is outrageously excessive for
> 66k entries. Might want to take a look in a bit editor like xvi32 and see
> if your line endings mismatch those produced by a sample ldifde export. I
> would also omit the changetype / '-' syntax as an unnecessary complication.
>
>
>
> Additionally, I would advise you to omit system-managed attributes
> (admincount, lastlogontimestamp, dscorepropagationdata, whencreated,
> whenchanged, objectcategory, instancetype, usncreated, usnchanged,
> instancetype) as the import will fail on certain of these, and some others
> are simply undesirable to import.
>
> Finally, rather than trying to apply 66,000 users all at once, test with a
> small subset of the file, say, 3 or 4 entries. This will help you detect
> syntax errors and schema violations more easily.
>
>
>
> --Steve
>
>
>
> On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu>
> wrote:
>
> Hello,
>
> I’m using LDIFDE to export and import about 66,000 users, from a domain in
> one forest to a domain in a separate forest. I used the following command
> to export the users:
>
>
>
> ldifde –n –f C:\filename.ldf –s servername –r “(objectCategory=person)” –o
> “exclusions”
>
>
>
> and did a find and replace on the ldf file to change all of the references
> from the export domain to the import domain. I also did a find and replace
> to add a “-“ after each entry and a blank space after the “-“. Here’s a
> sample entry in the resulting file, with sensitive info obfuscated:
>
>
>
> dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
>
> changetype: add
>
> objectClass: top
>
> objectClass: person
>
> objectClass: organizationalPerson
>
> objectClass: user
>
> cn: username
>
> instanceType: 4
>
> whenCreated: 20080129160427.0Z
>
> whenChanged: 20100503184651.0Z
>
> displayName: username
>
> uSNCreated: 5963864
>
> uSNChanged: 45461899
>
> name: username
>
> codePage: 0
>
> countryCode: 0
>
> userWorkstations: WORKSTATION
>
> adminCount: 1
>
> accountExpires: 9223372036854775807
>
> sAMAccountName: username
>
> userPrincipalName: username@domain.local
>
> objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 20090519185234.0Z
>
> dSCorePropagationData: 16010108151056.0Z
>
> lastLogonTimestamp: 129173852491219051
>
> -
>
>
>
> I used the following command to import the file:
>
>
>
> ldifde -i -f filename.ldf -j C:\ -v -u -q 2
>
>
>
> I’m running into two issues. The first is that the import starts, then
> displays the “Loading entries” status message for about 12 hours. This
> wouldn’t be a problem if the import was successful, or even partially
> successful, but it always fails with the following error:
>
> There is a syntax error in the input file
>
> Failed on line 1. The last token starts with '*char*'.
>
> 0 entries modified successfully.
>
> An error has occurred in the program
>
>
>
> The *char* value changes every time the script is run. I’ve done a
> search on this error and the few things I’ve found I’ve already implemented
> or don’t apply to my situation.
>
>
>
> What I’m wondering is, does ldifde have a size limit for the input file?
> Does it load every entry in the ldf file into memory before writing the
> changes to the directory (which would explain the 12 hour run time for
> 66,000 accounts)? Is there anything obvious I’ve done wrong?
>
>
>
> Thanks in advance for any help given.
>
>
>
> CJH
>
>
>
>
>
| | | |
| TG
Posts:298
| | 07/21/2010 7:41 PM |
| Have you try to change the samaccountname so it does not start with _?
Thank you, Tony.
Tony Gordon
Windows 2003 & 2000 MCSE, Windows 2003 MCSA, PMP
ITS Infrastructure Engineering
Tel 847.295.5000 x37892 | Fax 847.883.7892
tony dot gordon at hewitt dot tld | www.hewitt.com
P Please consider the environment before printing this e-mail.
From:
"Cynthia Haselton" <chaselton@uchicago.edu>
To:
"activedir@mail.activedir.org" <activedir@mail.activedir.org>
Date:
07/21/2010 01:01 PM
Subject:
RE: [ActiveDir] LDIFDE Issue
Sent by:
activedir-owner@mail.activedir.org
Here?s a sample entry after I deleted as many attributes as I could. This
is the first entry in the smaller .ldf file; the rest of the entries are
unmodified. I figured I would stick to changing one entry until I figured
out the problem.
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: User One
sn: One
givenName: User
distinguishedName: CN=User One,OU=Users and
Groups,OU=DCS,DC=domain,DC=local
displayName: User One
name: User One
sAMAccountName: _root_userone
-
Running the command ?ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k
-q 2?, minus the quotes, produces the following error
There is a syntax error in the input file
Failed on line 14. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
It doesn?t matter which attributes I remove, I still run into this error.
The only thing that changes is the line number in the ?Failed on line??
error and the character in the ?The last token starts with??
At this point I?m thinking of ditching ldifde for another import utility.
Anyone have recommendations?
From: activedir-owner@mail.activedir.org [
mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:47 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new
file and saved it with a different file name. I ran the following
command?
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
?and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running
the program. Same error as before. I then opened the file and saved it
in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I?m going to try eliminating the ?u
option (I read in a post that using ?u with a Unicode file causes an
error) and see if that works. If it doesn?t, not sure what else to do,
other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org [
mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies?
The RID master is on the DC from which I?m doing the import. I?m using
the ?j option but will add the ?k?though I thought I saw an article that
discouraged using it. I?d have to dig for it, and I?d rather fix this
issue than do that.
I?m still doing the large file import (with all 66,000+ users) as I
haven?t found an easy way to split the large file into smaller ones, or
filter the search so that it returns a small amount of users to work with.
Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org [
mailto:activedir-owner@mail.activedir.org] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a
few
Also, if you're creating a huge load of objects it might be worth moving
the RID master role to that DC as I've seen it where it gets exhausted and
bombs out.
Chris
From: activedir-owner@mail.activedir.org
[activedir-owner@mail.activedir.org] on behalf of Steve Kradel
[skradel@zetetic.net]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one
gigantic entry, although even then, 12 hours is outrageously excessive for
66k entries. Might want to take a look in a bit editor like xvi32 and see
if your line endings mismatch those produced by a sample ldifde export. I
would also omit the changetype / '-' syntax as an unnecessary
complication.
Additionally, I would advise you to omit system-managed attributes
(admincount, lastlogontimestamp, dscorepropagationdata, whencreated,
whenchanged, objectcategory, instancetype, usncreated, usnchanged,
instancetype) as the import will fail on certain of these, and some others
are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a
small subset of the file, say, 3 or 4 entries. This will help you detect
syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu
> wrote:
Hello,
I?m using LDIFDE to export and import about 66,000 users, from a domain in
one forest to a domain in a separate forest. I used the following command
to export the users:
ldifde ?n ?f C:\filename.ldf ?s servername ?r ?(objectCategory=person)? ?o
?exclusions?
and did a find and replace on the ldf file to change all of the references
from the export domain to the import domain. I also did a find and
replace to add a ?-? after each entry and a blank space after the ?-?.
Here?s a sample entry in the resulting file, with sensitive info
obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I?m running into two issues. The first is that the import starts, then
displays the ?Loading entries? status message for about 12 hours. This
wouldn?t be a problem if the import was successful, or even partially
successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I?ve done a search
on this error and the few things I?ve found I?ve already implemented or
don?t apply to my situation.
What I?m wondering is, does ldifde have a size limit for the input file?
Does it load every entry in the ldf file into memory before writing the
changes to the directory (which would explain the 12 hour run time for
66,000 accounts)? Is there anything obvious I?ve done wrong?
Thanks in advance for any help given.
CJH
The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. You are deemed to have accepted these risks if you communicate with us by e-mail.
| | | |
| chaselton
Posts:78
| | 07/21/2010 8:58 PM |
| Okay, will try that. Thanks.
I am getting closer. After changing the "CN" in the distinguished name to "cn", I got the following error
Add error on line 1: Unwilling To Perform
The server side error is "Unable to update the password. The value provided for
the new password does not meet the length, complexity, or history requirement of
the domain."
There is a syntax error in the input file
Failed on line 18. The last token starts with '-'.
0 entries modified successfully.
An error has occurred in the program
I'll remove the "-" and try again
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Steve Kradel
Sent: Wednesday, July 21, 2010 1:40 PM
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
Agreed, the - token is to indicate the end of a "mod-attribute-value" section, not for an entry-level add.
--Steve
On Wed, Jul 21, 2010 at 2:35 PM, Wilper, Ross A <rwilper@stanford.edu<mailto:rwilper@stanford.edu>> wrote:
I thought that you only needed the "-" on a line by itself to end an entry if the changetype is "modify"
-Ross
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 11:28 AM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
In one last attempt, I deleted all attributes except for the initial dn:. Here's what the entry looked like
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
-
This is the error I got:
There is a syntax error in the input file
Failed on line 7. The last token starts with 'u'.
0 entries modified successfully.
An error has occurred in the program
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 1:02 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
Here's a sample entry after I deleted as many attributes as I could. This is the first entry in the smaller .ldf file; the rest of the entries are unmodified. I figured I would stick to changing one entry until I figured out the problem.
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: User One
sn: One
givenName: User
distinguishedName: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
displayName: User One
name: User One
sAMAccountName: _root_userone
-
Running the command "ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k -q 2", minus the quotes, produces the following error
There is a syntax error in the input file
Failed on line 14. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
It doesn't matter which attributes I remove, I still run into this error. The only thing that changes is the line number in the "Failed on line..." error and the character in the "The last token starts with..."
At this point I'm thinking of ditching ldifde for another import utility. Anyone have recommendations?
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:47 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new file and saved it with a different file name. I ran the following command...
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
...and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running the program. Same error as before. I then opened the file and saved it in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I'm going to try eliminating the -u option (I read in a post that using -u with a Unicode file causes an error) and see if that works. If it doesn't, not sure what else to do, other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] on behalf of Steve Kradel [skradel@zetetic.net<mailto:skradel@zetetic.net>]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| chaselton
Posts:78
| | 07/21/2010 9:06 PM |
| That did it. Sort of.
The "cn" entry was not added, the "CN" entry was. I changed the "cn" entry back to "CN" for the distinguished name and re-ran the command, but that particular entry was still not added.
EDIT: The userAccountControl field was the key. When I changed it from 66048 to 66050 the entry was added.
From: Cynthia Haselton
Sent: Wednesday, July 21, 2010 2:57 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Okay, will try that. Thanks.
I am getting closer. After changing the "CN" in the distinguished name to "cn", I got the following error
Add error on line 1: Unwilling To Perform
The server side error is "Unable to update the password. The value provided for
the new password does not meet the length, complexity, or history requirement of
the domain."
There is a syntax error in the input file
Failed on line 18. The last token starts with '-'.
0 entries modified successfully.
An error has occurred in the program
I'll remove the "-" and try again
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Steve Kradel
Sent: Wednesday, July 21, 2010 1:40 PM
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
Agreed, the - token is to indicate the end of a "mod-attribute-value" section, not for an entry-level add.
--Steve
On Wed, Jul 21, 2010 at 2:35 PM, Wilper, Ross A <rwilper@stanford.edu<mailto:rwilper@stanford.edu>> wrote:
I thought that you only needed the "-" on a line by itself to end an entry if the changetype is "modify"
-Ross
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 11:28 AM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
In one last attempt, I deleted all attributes except for the initial dn:. Here's what the entry looked like
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
-
This is the error I got:
There is a syntax error in the input file
Failed on line 7. The last token starts with 'u'.
0 entries modified successfully.
An error has occurred in the program
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 1:02 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
Here's a sample entry after I deleted as many attributes as I could. This is the first entry in the smaller .ldf file; the rest of the entries are unmodified. I figured I would stick to changing one entry until I figured out the problem.
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: User One
sn: One
givenName: User
distinguishedName: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
displayName: User One
name: User One
sAMAccountName: _root_userone
-
Running the command "ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k -q 2", minus the quotes, produces the following error
There is a syntax error in the input file
Failed on line 14. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
It doesn't matter which attributes I remove, I still run into this error. The only thing that changes is the line number in the "Failed on line..." error and the character in the "The last token starts with..."
At this point I'm thinking of ditching ldifde for another import utility. Anyone have recommendations?
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:47 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new file and saved it with a different file name. I ran the following command...
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
...and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running the program. Same error as before. I then opened the file and saved it in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I'm going to try eliminating the -u option (I read in a post that using -u with a Unicode file causes an error) and see if that works. If it doesn't, not sure what else to do, other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] on behalf of Steve Kradel [skradel@zetetic.net<mailto:skradel@zetetic.net>]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
| rwilper
Posts:37
| | 07/21/2010 9:08 PM |
| Ah yes, when you create a user object, you must specify unicodePWD or specify that the account is disabled. There are a number of other attributes that must be supplied (cn,samAccountName)
-Ross
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:57 PM
To: activedir@mail.activedir.org
Subject: RE: [ActiveDir] LDIFDE Issue
Okay, will try that. Thanks.
I am getting closer. After changing the "CN" in the distinguished name to "cn", I got the following error
Add error on line 1: Unwilling To Perform
The server side error is "Unable to update the password. The value provided for
the new password does not meet the length, complexity, or history requirement of
the domain."
There is a syntax error in the input file
Failed on line 18. The last token starts with '-'.
0 entries modified successfully.
An error has occurred in the program
I'll remove the "-" and try again
From: activedir-owner@mail.activedir.org [mailto:activedir-owner@mail.activedir.org] On Behalf Of Steve Kradel
Sent: Wednesday, July 21, 2010 1:40 PM
To: activedir@mail.activedir.org
Subject: Re: [ActiveDir] LDIFDE Issue
Agreed, the - token is to indicate the end of a "mod-attribute-value" section, not for an entry-level add.
--Steve
On Wed, Jul 21, 2010 at 2:35 PM, Wilper, Ross A <rwilper@stanford.edu<mailto:rwilper@stanford.edu>> wrote:
I thought that you only needed the "-" on a line by itself to end an entry if the changetype is "modify"
-Ross
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 11:28 AM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
In one last attempt, I deleted all attributes except for the initial dn:. Here's what the entry looked like
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
-
This is the error I got:
There is a syntax error in the input file
Failed on line 7. The last token starts with 'u'.
0 entries modified successfully.
An error has occurred in the program
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 1:02 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
Here's a sample entry after I deleted as many attributes as I could. This is the first entry in the smaller .ldf file; the rest of the entries are unmodified. I figured I would stick to changing one entry until I figured out the problem.
dn: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: User One
sn: One
givenName: User
distinguishedName: CN=User One,OU=Users and Groups,OU=DCS,DC=domain,DC=local
displayName: User One
name: User One
sAMAccountName: _root_userone
-
Running the command "ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -k -q 2", minus the quotes, produces the following error
There is a syntax error in the input file
Failed on line 14. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
It doesn't matter which attributes I remove, I still run into this error. The only thing that changes is the line number in the "Failed on line..." error and the character in the "The last token starts with..."
At this point I'm thinking of ditching ldifde for another import utility. Anyone have recommendations?
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 12:47 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
This is getting a bit ridiculous
I copied a bunch of the entries from the larger file, pasted them in a new file and saved it with a different file name. I ran the following command...
ldifde -i -f ad.local.user.group.1.ldf -v -j C:\ -u -k -q 2
...and got the same error as before.
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
I then tried removing some attributes from the first entry and re-running the program. Same error as before. I then opened the file and saved it in Unicode format. This time I got a different version of the same error:
There is a syntax error in the input file
Failed on line 15. The last token starts with '_'.
0 entries modified successfully.
An error has occurred in the program
This is frustrating me to no end. I'm going to try eliminating the -u option (I read in a post that using -u with a Unicode file causes an error) and see if that works. If it doesn't, not sure what else to do, other than deleting attributes until the import actually works.
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Cynthia Haselton
Sent: Wednesday, July 21, 2010 10:21 AM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
Finally got the replies...
The RID master is on the DC from which I'm doing the import. I'm using the -j option but will add the -k...though I thought I saw an article that discouraged using it. I'd have to dig for it, and I'd rather fix this issue than do that.
I'm still doing the large file import (with all 66,000+ users) as I haven't found an easy way to split the large file into smaller ones, or filter the search so that it returns a small amount of users to work with. Will update with results.
Thanks for all the help,
CJH
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [mailto:activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] On Behalf Of Chris Boller
Sent: Tuesday, July 20, 2010 1:24 PM
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: RE: [ActiveDir] LDIFDE Issue
I agree with Steve omitting those attributes and trying to import just a few
Also, if you're creating a huge load of objects it might be worth moving the RID master role to that DC as I've seen it where it gets exhausted and bombs out.
Chris
________________________________
From: activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org> [activedir-owner@mail.activedir.org<mailto:activedir-owner@mail.activedir.org>] on behalf of Steve Kradel [skradel@zetetic.net<mailto:skradel@zetetic.net>]
Sent: 20 July 2010 16:04
To: activedir@mail.activedir.org<mailto:activedir@mail.activedir.org>
Subject: Re: [ActiveDir] LDIFDE Issue
The 12-hour thing suggests that ldifde is trying to parse your file as one gigantic entry, although even then, 12 hours is outrageously excessive for 66k entries. Might want to take a look in a bit editor like xvi32 and see if your line endings mismatch those produced by a sample ldifde export. I would also omit the changetype / '-' syntax as an unnecessary complication.
Additionally, I would advise you to omit system-managed attributes (admincount, lastlogontimestamp, dscorepropagationdata, whencreated, whenchanged, objectcategory, instancetype, usncreated, usnchanged, instancetype) as the import will fail on certain of these, and some others are simply undesirable to import.
Finally, rather than trying to apply 66,000 users all at once, test with a small subset of the file, say, 3 or 4 entries. This will help you detect syntax errors and schema violations more easily.
--Steve
On Tue, Jul 20, 2010 at 10:53 AM, Cynthia Haselton <chaselton@uchicago.edu<mailto:chaselton@uchicago.edu>> wrote:
Hello,
I'm using LDIFDE to export and import about 66,000 users, from a domain in one forest to a domain in a separate forest. I used the following command to export the users:
ldifde -n -f C:\filename.ldf -s servername -r "(objectCategory=person)" -o "exclusions"
and did a find and replace on the ldf file to change all of the references from the export domain to the import domain. I also did a find and replace to add a "-" after each entry and a blank space after the "-". Here's a sample entry in the resulting file, with sensitive info obfuscated:
dn: CN=username,OU=Users and Groups,OU=DCS,DC=domain,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: username
instanceType: 4
whenCreated: 20080129160427.0Z
whenChanged: 20100503184651.0Z
displayName: username
uSNCreated: 5963864
uSNChanged: 45461899
name: username
codePage: 0
countryCode: 0
userWorkstations: WORKSTATION
adminCount: 1
accountExpires: 9223372036854775807
sAMAccountName: username
userPrincipalName: username@domain.local<mailto:username@domain.local>
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=domain,DC=local
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 20090519185234.0Z
dSCorePropagationData: 16010108151056.0Z
lastLogonTimestamp: 129173852491219051
-
I used the following command to import the file:
ldifde -i -f filename.ldf -j C:\ -v -u -q 2
I'm running into two issues. The first is that the import starts, then displays the "Loading entries" status message for about 12 hours. This wouldn't be a problem if the import was successful, or even partially successful, but it always fails with the following error:
There is a syntax error in the input file
Failed on line 1. The last token starts with 'char'.
0 entries modified successfully.
An error has occurred in the program
The char value changes every time the script is run. I've done a search on this error and the few things I've found I've already implemented or don't apply to my situation.
What I'm wondering is, does ldifde have a size limit for the input file? Does it load every entry in the ldf file into memory before writing the changes to the directory (which would explain the 12 hour run time for 66,000 accounts)? Is there anything obvious I've done wrong?
Thanks in advance for any help given.
CJH
| | | |
|
|