Content download problems with large packages in ConfigMgr 2012

The customer was trying to install an AutoCAD type application via Configuration Manager 2012 (SP1 CU2) and the installation kept hanging in Software Center with the message “downloading data”.

The background is that this application had previously worked ok, but since then the source files have been altered and updated on the DPs. The source data is itself around a whopping 9Gb.

So let’s go through the logs.

AppDiscovery, AppEnforce and AppIntentEval all look normal. Application is not detected and will be installed.

Execmgr makes a call to CAS requesting content. So far so good.

CAS requests content locations and receives 3 valid DPs.

CAS submits a CTM (Content Transfer Manager) job, and receives confirmation of the locations and submits a DTS (Data Transfer Service) job.

Now the DTS starts the download and immediately hits a problem.

GetDirectoryList_HTTP(‘http://dp:80/SMS_DP_SMSPKG$/Content_45106a31-c15b-4d29-ba68-97e1b97a5e9e.1’) failed with code 0x80004005.

Error retrieving manifest (0x80004005).  Will attempt retry 1 in 30 seconds.

Sure enough it retries only to get:

Non-recoverable error retrieving manifest (0x80004005).

And this happens in turn for each available DP. Unfortunately Software Center will just sit there doing nothing. This last point may be because the installation is running in a task sequence but let’s not dwell on that for now.

So obviously a problem with the content on the DPs. Agreed?

Redistribute application to DPs – same error

Remove application from all Dps and redistribute – same error

Zip the source from 9gb down to 4Gb, remove from DPs and redistribute – same error, but wait, when i redistributed to the DPs it still took a long time, not half the time as i expected.

I had to get the Content ID from the DTS log and then look at the ContentLibrary on the DP. Found the content but it didn’t match the source. In fact it didn’t match any updated source, seemed to be the original content from before the first update. Very strange.

So to look at how distmgr and ContentLibrary work in more detail:

Source is copied to the “primary site server” and stored in a ContentLibrary there, even though there is no DP on this server. This is a small throwback to SCCM 2007 and is unavoidable. Anyway, from there the content is copied to the DPs and stored in ContentLibrary. Removing an application from the DP goes quite quickly in the console but looking at the  distmgr log and smsdpprov log on the DP you can see that the data itself is not fully removed for some time after, depending on the size of the content. If you redistribute the content before it is fully removed from ContentLibrary then distmgr will skip copying the files that already exist. Also, distmgr will copy the content from the ContentLibrary on the primary server, if it exists there, rather than directly from the source. So in actual fact, with a 9Gb application the source was never correctly updated on the DPs – it’s very difficult to say exactly which files didn’t match the hash values in the manifest but obviously we need to completely this application’s content from the DPs, from the primary server ContentLibrary and wait until everything is completely gone before redistributing.

Easier said than done. To remove from the DPs is not so difficult, just as with any normal application. But make sure that distmgr and smsdpprov confirm it is gone, AND check manually in ContentLibrary in DataLib under the ContentID to make sure it is physically gone. For 9Gb this can take about 30 minutes.

Then have a look at the ContentLib on the primary server. It hasn’t detected that it needs to update from the source so you need to trigger that. Here is what i did although maybe there is a better way, i had run out of patience by this point so i didn’t wait to see if it would also remove itself.

I removed all previous revisions of the application from revision history. Then changed the source path on the deployment type to point to an empty folder. Now monitoring distmgr i can see he has spotted this and creates a new content instance in the library with a new ContentID. The old ContentID however remains and is flagged as orphaned. Check ContentLibrary for the physical presence and this query against the DB:

select * from OrphanedContents

After a maximum of 60 minutes, the content cleanup cycle will run on the server and remove these orphaned contents. That’s about how long it takes to find out that there is very little documentation on the content cleanup task on the internet, and very little help to be found searching for “delete orphaned content sccm 2012” in Bing or Google or whatever… It helps pass the time though.

So now we have a large application with an empty source path, and not distributed to DPs. And checking that we can see that there are absolutely no more traces on DPs, in ContentLibraries, or in the DB.

Now set the source back to the correct source path on the deployment type. And wait until this is completely updated on the primary site in ContentLibrary. I didn’t wait this time for the cleanup task but i checked back later and references to the empty source where gone.

Check the ContentLibrary on the primary server, you need the new ContentID from distmgr.log, to see if the new content is physically there. Once it is distribute to DPs. Again check distmgr, also PkgXferMgr.log, on the primary server and smsdpprov.log on a DP until they are finished processing the content. Check the content is physically in the ContentLibrary in DataLib under the ContentID.

Now try installing the application again on the client. This time it installs no problem.

A couple of things to point out here:

This deployment is an application delivered via task sequence but it applies to pure AppModel deployments as well.

We have a primary server which serves as SMSProvider and the DPs are all only on separate boxes.

The problem is here when you have a very large amount of data in the source and you update through the console quicker than it actually takes for the processes to finish. On smaller content this may be ok but the larger the packet the more chance that something will get skipped in the file copy process.

Advertisement