System.OutOfMemoryException

Topics: Bugs, General Questions
Jul 5, 2007 at 4:55 PM
I receive the following error message whenever I try to build a DocProject.

"Step 8 System.OutOfMemoryException: Build Assembler {sandcastle.help1x.config}:
Exception of type 'System.OutOfMemoryException' was thrown."

Any thoughts on what is wrong? I have the newest SandCastle installed (June '07 refresh) and the 1.6.2RC DocProject installed.

Thanks
Coordinator
Jul 5, 2007 at 5:26 PM
Hi,

Did you check to see if you were actually out of memory?

Sandcastle's Build Assembler loads a lot of data in memory while it processes the topics. It could just be that you don't have enough memory on your system for this utility to function properly.

I suggest asking the Sandcastle forums about Build Assembler and OutOfMemoryException and maybe you'll get a response from Microsoft about whether there is anything that can be done about this.

- Dave
Jul 5, 2007 at 6:57 PM
I have 2GB of RAM and 3GB of swap space in my development machine. I hope that Build Assembler doesn't require more memory than that for my small project. I will post on the SandCastle forum as well and see what those guys have to say.

Thanks
Coordinator
Jul 5, 2007 at 7:33 PM
Hi,

Well it sounds like you certainly have enough for a small project, at least :)

I have 2GB as well but I've never run into this problem attempting to build help for a medium-sized library, even with a few copies of VS open, some office programs such as Outlook, various other software and services, and a VM running XP.

I'd really like to see the responses that you get from the forums!

BTW, if you can reproduce this issue have you tried looking at your memory usage in Task Manager at the time that the build fails? I'm curious to know what the difference between memory consumption is before the build starts compared to when it fails.

- Dave
Jul 5, 2007 at 8:49 PM
I am able to generate my documentation if I manually do each step from the command-line. BuildAssembler doesn't give me any grief. Earlier after your first reply I watched the Task Manager and it looked like my memory usage only went up by about 10% while performing a DocProject build. I haven't posted on the Sandcastle forum yet because I've gotten it to work through the command-line and batch files. If you'd like me to post there I can but I'm guessing that I'll just get a response saying that it must be a DocProject issue. If there is any other information that you'd like regarding my setup, output, etc. let me know and I'll get that to you.

Thanks,
David
Coordinator
Jul 5, 2007 at 9:14 PM
Edited Jul 5, 2007 at 9:30 PM
Hi David,

Thanks for the response.

I can see why people might say it's a DocProject issue, but the bottom-line is that Sandcastle's Build Assembler program is throwing OutOfMemoryException. I doubt there's anything I can do about that in DocProject's code.

I'm glad that you were able to build it on the command-line, but that doesn't necessarily mean that DocProject is causing the problem either. For one thing, Visual Studio's memory usage, which in my experience can be quite high, doesn't have any effect on a command-line build. I think that .NET places constraints on the amount of memory a single process can use as well.

It may still be worth the effort to notify Microsoft that their component may throw an exception in certain scenarios, but it's entirely up to you if you want to start a thread on the Sandcastle forum. I'd do it but if the Sandcastle team requires more information I can't help them since I can't reproduce the error in DocProject.

- Dave
Coordinator
Jul 5, 2007 at 9:29 PM
Hi David,

So I thought about this some more and then I realized that I completely ignored your offer to provide me with information about your setup so that I can try to reproduce the problem. Sorry :) I just assumed that I couldn't reproduce it since I've done so many tests with hardly any memory left on my machine and I've never run into this problem.

I'd like to try if you're willing to describe the steps that I need to take.

Thanks,
Dave
Jul 5, 2007 at 10:12 PM
One thing that I forgot to mention is that I'm running Windows Vista 32-bit Business with Visual Studio 2005 Professional SP1. Could it be some sort of strange Vista issue?

There really aren't too many steps that I actually have to do to recreate the exception. I added a DocProject to my solution, followed the Wizard then hit build and waited for bad things to happen. Here are the Wizard settings I used

Screen 1: Sandcastle Engine
Screen 2: Visual Studio 2005 presentation
Screen 3: Checked both Compiled Help 1.x and Compiled Help 2.x
Screen 4: Entered "Landis+Gyr Bin File Handler API Documentation" for the header and nothing for the footer
Screen 5: Selected my project which was the only thing listed

Here is a snippet from the Build Output that shows where the failure occurred. If you'd like to see the rest of the output let me know. Oh and this happened with more than one project. The project that generated this error only had about two dozen lines of actual code which included a single class with a couple of constructors, properties, and methods.

Info: Loaded 477 shared content items.
Info: Instantiating component.
Info: Creating MSDN URL resolver.
Info: Searching directory 'F:\Sandcastle\Data\Reflection' for targets files of the form '*.xml'.

Step 8 System.OutOfMemoryException: Build Assembler {sandcastle.help1x.config}:
Exception of type 'System.OutOfMemoryException' was thrown.

Total Time Elapsed: 00:01:07.3727042

Successful Steps: 7 of 21
Failed Steps: 1

David
Coordinator
Jul 5, 2007 at 10:22 PM
Hi David,

Thanks for the information.

First, I doubt this is a Vista issue since I develop and use DocProject on my Vista box without having this problem :)

I'm curious to know, does the build always fails at the same point during Step 8 in both projects? i.e., Info: Searching directory ... for targets files of the form '*.xml'

Thanks,
Dave
Jul 6, 2007 at 1:35 PM
Ok so my other project, which is considerably larger, got farther. The output near the failure is below. After the build below I tried building my other project to confirm that it fails consistently at the same spot in my previous post and it did. Then I built my second project (now 2 instances of VS2005 are open) and it failed at the same point as the first one (Info: Searching directory ... for targets files of the form '*.xml'). That was strange I thought so I closed the smaller project and rebuilt the other and it failed again with the output at the end of this post. Maybe VS2005 is doing something strange with memory management? Is it possible that VS2005, .NET 2.0, or both somehow limits the amount of memory available to a process running within VS2005?

Info: Searching directory 'F:\Sandcastle\Data\Reflection' for targets files of the form '*.xml'.
Info: Searching directory '.' for targets files of the form 'reflection.xml'.
Info: Loaded 189092 reference targets.
Info: Instantiating component.
Info: Building topic N:ANSITables
Info: Building topic T:ANSITables.CharFormatType
Info: Building topic T:ANSITables.DataOrder
Info: Building topic T:ANSITables.IANSIDataType
Info: Building topic AllMembers.T:ANSITables.IANSIDataType
Warn: Invalid referenceLink element.
Info: Building topic Methods.T:ANSITables.IANSIDataType
Warn: Invalid referenceLink element.
Info: Building topic Properties.T:ANSITables.IANSIDataType
Warn: Invalid referenceLink element.
Info: Building topic P:ANSITables.IANSIDataType.ByteSize
Info: Building topic P:ANSITables.IANSIDataType.Name
Info: Building topic P:ANSITables.IANSIDataType.Parent
Info: Building topic M:ANSITables.IANSIDataType.ToByteArray
Info: Building topic M:ANSITables.IANSIDataType.ToString
Info: Building topic P:ANSITables.IANSIDataType.TypeName
Info: Building topic P:ANSITables.IANSIDataType.Value
Info: Building topic T:ANSITables.IANSITable
Info: Building topic AllMembers.T:ANSITables.IANSITable
Warn: Invalid referenceLink element.
Info: Building topic Methods.T:ANSITables.IANSITable
Warn: Invalid referenceLink element.
Info: Building topic Properties.T:ANSITables.IANSITable
Warn: Invalid referenceLink element.
Info: Building topic Overload:ANSITables.IANSITable.SetField
Info: Building topic P:ANSITables.IANSITable.Access
Info: Building topic M:ANSITables.IANSITable.GetComSafeEnumerator
Info: Building topic M:ANSITables.IANSITable.GetField(System.String)

Step 8 System.OutOfMemoryException: Build Assembler {sandcastle.help1x.config}:
Exception of type 'System.OutOfMemoryException' was thrown.

David
Coordinator
Jul 6, 2007 at 2:49 PM
Hi David,

It's interesting to me that you can reproduce the issue at the same point in the build process each time. This indicates that you may be hitting the limit of per-process memory usage, if there is one. I do remember reading that there is a maximum limit imposed by the CLR but I can't find a reference to that right now. I could be wrong. Although it really doesn't matter anyway if you feel that you still have more than enough RAM free during the build. How much memory is allocated to the process when you start Visual Studio? You said before that another 10% is used by DocProject and Sandcastle during a build, so how much does that amount to in bytes?

From poking around the newsgroups my guess is that anywhere between 900MB and 1.2GB of RAM can be allocated in a managed process, normally. This may depend upon several factors that include the OS type/version and runtime/system configuration. Is Visual Studio's memory usage anywhere near that figure for you?

I'll consider adding a configuration option that allows you to run Build Assembler in an external process, forfeiting synchronization with the Error List though. Hopefully that will help to fix the problem in a future version of DocProject for anyone that's pressing the memory limit.

One other thing about OutOfMemoryException is that it commonly occurs when the program is executing an infinitely recursive loop, which was actually my first guess when you posted the build output since it seemed to be failing on a recursive operation: searching a directory structure for all *.xml files. I don't want to rule this one out yet as a possibility for the failure.

Thanks,
Dave
Jul 6, 2007 at 3:02 PM
Yeah, my first reaction was that an infinitely recursive loop was occurring. Although I would have thought a StackOverflowException would have occurred. Perhaps a tail recursive loop could cause an OutOfMemoryException.

I monitored the actual memory usage of devenv.exe as the build process ran. My previous 10% number was a bit off (try 500% change), I was originally just watching the overall system memory usage. The statistics are below.

Start: 43,388 KB
End: 269,704 KB

It's a big change but it's still no where near the 900mb - 1.2gb that you mentioned, and it definately isn't anywhere near my system's available memory.

David
Coordinator
Jul 6, 2007 at 3:28 PM
Hi David,


Yeah, my first reaction was that an infinitely recursive loop was occurring. Although I would have thought a StackOverflowException would have occurred. Perhaps a tail recursive loop could cause an OutOfMemoryException.

It could be a recursive while loop, which wouldn't have an effect on the call stack. But regardless, I doubt that's the problem anyway.


I monitored the actual memory usage of devenv.exe as the build process ran. My previous 10% number was a bit off (try 500% change), I was originally just watching the overall system memory usage. The statistics are below.

Start: 43,388 KB
End: 269,704 KB

It's a big change but it's still no where near the 900mb - 1.2gb that you mentioned, and it definately isn't anywhere near my system's available memory.


Does your Start figure include Visual Studio's initial memory usage before you began to build the project?

- Dave
Coordinator
Jul 6, 2007 at 3:31 PM
BTW, I just checked out the ECMA C# and Common Language Infrastructure Standards specifications and I didn't find any mention of a maximum per-process memory size, so if there is one it's being imposed by Windows.

- Dave
Jul 6, 2007 at 3:48 PM
The "Start" figure is the value that Task Manager reported for devenv.exe before I clicked Build and the "End" figure is the value reported when the exception was thrown. Both values were stable when I read them. The peak memory usage was approximately 271,000 KB just before the exception occurred.

David

p.s. How do you make a recursive while loop? (Just curious)
Coordinator
Jul 6, 2007 at 3:59 PM
Hi David,


The "Start" figure is the value that Task Manager reported for devenv.exe before I clicked Build and the "End" figure is the value reported when the exception was thrown. Both values were stable when I read them. The peak memory usage was approximately 271,000 KB just before the exception occurred.

Ok, so then I'm stumped. If you're not out of memory, what else could cause an OutOfMemoryException? I'll do some research and get back to you. If you find anything yourself please post it here.


p.s. How do you make a recursive while loop? (Just curious)

Great question! If I knew I'd tell you ;)

I kind of remembered doing something like that in the past - recursion without a method call, using a while loop with a goto statement IIRC. Ridiculous, I known, which is why I said that I doubt it's the problem. Please just ignore me :)

- Dave
Coordinator
Jul 6, 2007 at 5:32 PM
Hi David,

I did some research and all that I've come up with is that the error might be occurring due to memory fragmentation. i.e., There may not be a block of contiguous memory that is big enough to store some of Build Assembler's data. For instance, if code such as this is used: byte[] bytes = new byte[x];, where x is some large number.

Posting this issue to the Sandcastle forums may help since this issue is apparently specific to Sandcastle's code-base. Microsoft may be able to reduce the amount of contiguous memory that is required by the Build Assembler. But even if they can't/don't, I'm still going to see if I can implement that feature I mentioned previously, to be able to configure DocProject to run Build Assembler in an external process.

If you decide to post this to the Sandcastle forums then you may want to run perfmon first and get some accurate memory usage data, or just wait until someone asks :)

I'll monitor the thread too and if someone asks a question related to DocProject I'll be sure to answer it.

Thanks,
Dave
Aug 30, 2007 at 5:55 PM
Hey Everyone,

I think I might have solved this problem.

The OutOfMemoryException is an eroneous error, I think. What I have found to be the problem is the number of compiler warnings. In Visual Studio, when you compile, you have a maximum number of error messages & warnings that will be displayed by the interface. When you reach the limit, an "Too many errors and warnings".

Sandcastle produces a warning for every public property/class/etc. that is not documented with the XML documentation attributes (your triple-slash comments).

If you have many properties/classes/etc. that are not documented, you get LOTS of errors. Thus, Visual Studio throws an error, and it is caught by the sandcastle engine, and displayed as an OutOfMemoryException error.

So, to fix this problem, do one of the following:

Either ensure that all your properties/classes/etc are documented
or
Increase the number of errors that Visual Studio will allow in a compile (http://blogs.msdn.com/fxcop/archive/2006/05/04/faq-how-do-i-increase-the-number-of-code-analysis-warnings-displayed-in-the-error-list-david-kean.aspx)
Coordinator
Aug 30, 2007 at 7:01 PM
Thanks for the info and the links :)

I don't think this solves the OP's problem though. The original example was too simple for this to be the answer (if you look at the build output that was posted it's extremely small).

- Dave
Apr 9, 2008 at 11:22 AM
Edited Apr 9, 2008 at 1:30 PM
Same problem here:

Win XP
Visual Studio 2008
Sandcastle January 2008
DocProject2008-Beta2

Visual Studio compiles manually fine, but the IDE crashes at Step 7 of 7: Build Assembler {sandcastle.help1x.config}

When compiling, it generates 205 warnings like following:

C:\Programme\Dave Sexton\DocProject\2008\bin\DaveSexton.DocProject.targets : BuildAssembler warning : Entries for the key 'N:System.Data.SqlClient' occur in both 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\System.Data.SqlClient.xml' and 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\System.Data.xml'. The last entry will be used.
Coordinator
Apr 9, 2008 at 3:57 PM
Hi,

Has it ever built successfully or does it always fail on step 7?

The warnings are generated by Sandcastle because the Framework reflection files (that Sandcastle installs) have duplicate entries. Although I doubt that 205 warning messages will use up all of your memory, DocProject 1.10.0 RC introduced a project setting that you can use to disable various build assembler trace output, such as those warnings. The setting is not available in DocProject 2008 Beta 2 but it will be in the next release.

Unfortunately, this issue was never actually resolved by the OP, as far as I know. Sandcastle's BuildAssembler.exe program, and the help build process in general, requires a lot of system resources and I don't think that there's a way for DocProject to explicitly manage memory to prevent this issue in a managed framework. Note however that 1.10.0 RC and the next version of DocProject 2008 do forcefully clean up memory after each of the build assembler and Help 2.x compiler steps are executed, but it's memory that the GC can free anyway so I doubt that it will have any effect in your case.

I suggest that you try to build your DocProject or DocSite in a smaller solution (i.e., less projects) and also try using the DocProject External UI and the command-line: msbuild.exe "your_docproject.csproj". You should find that at least one approach works for you.

- Dave
Apr 10, 2008 at 9:16 AM
Edited Apr 10, 2008 at 9:53 AM
Hi Dave,

With the IDE, it always failes on Build Assembler {sandcastle.help1x.config} when a project is attached. The step number differs according to which options are selected.

Here is what I found so far:
  • msbuild.exe compiles successfully
  • DocProject compiles using IDE when no project is attached
  • DocProject crashes with a System.IO.FileLoadException at Step 7 of 19: Build Assembler {sandcastle.help1x.config} when one or more projects are attached, using the IDE. In my case, I used a tiny project with only one class.
  • Eric Woodruff's Sandcastle Presentation File Patches didn't help.

I have about 1GB of RAM, but I doubt that that is the problem. While running the compiler, memory usage didn't seem to increase much and killing most running processes didn't make a difference.

When I click on the "Help 1.x build component stack", I get the following:

Could not load file or assembly 'BuildComponents, Version=2.3.8000.26, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Das System kann die angegebene Datei nicht finden.

System.IO.FileNotFoundException: Could not load file or assembly 'BuildComponents, Version=2.3.8000.26, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Das System kann die angegebene Datei nicht finden.
File name: 'BuildComponents, Version=2.3.8000.26, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
at DaveSexton.DocProject.Sandcastle.BuildComponents.SandcastleBuildComponent..ctor(String configurationFile, Int32 index, Int32 lineNumber, String commentDescription, String configuration)
at DaveSexton.DocProject.Sandcastle.BuildComponents.SandcastleBuildComponentManager.LoadFromFile()


I don't see that assembly in the Global Assembly Cache or application folder.
Coordinator
Apr 10, 2008 at 4:03 PM
Edited Apr 10, 2008 at 4:04 PM
Hi,

The error that you're reporting, FileNotFoundException, is not the same as the error discussed in this thread: OutOfMemoryException.

You are see this exception because DocProject 2008 Beta 2 was built against the Sandcastle October 2007 CTP, not the January 2008 Release. Check the release notes again and there is a link to download the appropriate version of Sandcastle.

EDIT: Actually, the link is on the release page for DocProject 2008 Beta 2. The release notes wiki links to the January 2008 Release.

- Dave
Apr 10, 2008 at 10:23 PM
Edited Apr 10, 2008 at 10:27 PM
Ok, that works now. Thanks :) I was probably following the wiki release notes.
Coordinator
Apr 11, 2008 at 1:36 AM
Cool, thanks for letting me know.

BTW, I've updated the release notes to make it clearer.

- Dave
Sep 1, 2008 at 12:49 PM
Hi everyone.

I'd like to thank the_skipster for his advice. It really helped me out. I was getting the same Out-of-Memory Exception when building CHM for 4 projects with ~250 types total (DocProject 1.11.0RC + Sandcastle 2.4.10520). I'd raised the registry value mentioned above to 500 and got the build successfully completed. It took up to 900Mb of RAM (devenv.exe) and 15 mins to accomplish the construction of this CHM with 3216 topics.
Dec 4, 2008 at 3:53 PM
Has anyone found a resolution to the System.OutOfMemoryException yet?  Looks as though I'm having the same issue as ProudGecko.  The step number that it fails on changes depending on the options I have set... but it always generates the exception on the "Build Assembler {sandcastee.help1x.config}" step.

I'm not having an issue with the number of errors/warnings (total of 19 warnings) being reported or anything as mentioned by the_skipster.

I just downloaded & installed DocProject & Sandcastle yesterday..

 

Coordinator
Dec 5, 2008 at 1:16 PM
Hi bgirl,

Try disabling other packages and add-ins that you have installed.  It's possible that other tools may be using up all of VS's memory.  Of course, if you prefer to have everything running then I suggest building your DocProjects and DocSites using the External UI or the command-line instead.

- Dave
Dec 5, 2008 at 2:45 PM
Alright, I'll give that a shot later today, thanks  :)
Dec 5, 2008 at 8:39 PM
I tried it out, with no other add-ins running it still does the same thing.  I did it from the External UI and still no luck.  So I actually created an empty project/solution and ran it against that and it did work... so maybe it's the size of the project I'm attempting to document.  I'll try again later when I have a bit more time to play around with it...
Dec 5, 2008 at 9:41 PM
I found a way around it - I created a totally separate project just for DocProject, and I pointed it to the assembly/xml I wanted to document and had to run it from the External UI.

The CPU was still running at 100% for a majority of the time, and it took a super long time, but it finally finished without any errors!!  What a happy way to start my weekend  :)