Monday, July 31, 2017

Step by Step SCCM 1706 Upgrade Guide

Two days ago Microsoft has released new branch update (1706) for System Center Configuration Manager.

1706 brings changes around deploy and managing Windows 10 and Office 365 along with reload boot images, boundary groups for software update points, Windows update reset tool, Improvements for SQL Server Always On Availability Groups and many more.
Full list of new feature read Microsoft release notes

Before proceeding with SCCM branch upgrade, it is very important to review SCCM Current Branch servicing (upgrade) checklist

Installing SCCM CB 1706 update:
If SCCM CB 1706 update is not available (if you are not in first wave of customers)in SCCM console and you want to install without waiting then you need to run FastRingScript_1706.exe

Downloading  SCCM CB 1706 update using FastRingScript_1706.exe:
1. Click on "Check for Updates" from \Administration\Overview\Updates and Servicing

2. If update is not available, then get FastRingScript_1706.exe from TechNet gallery
3. Extract the downloaded  EnableFastRing1706.exe
4. Launch PowerShell as Administrator
5. Run the script from elevated PowerShell window (ex: EnableFastRing1706_all.ps1 SCCB )
     Note: Just use server name without FQDN

6. You will get the command(s) completed successfully message;

7. Go to servicing node in SCCM console then click Check for updates;

8. Wait for 1-2 min then review the dmpdownloader.log file;

The log should have - Found a new available update;

Then downloading large file with bits

9. Refresh the Updates and servicing node in SCCM console, you can see the 1706 update in
     downloading state.

10. Wait until the 1706 update status changes from downloading to Available. It may take some time to complete the download. Once the download is completed, the status will change to 

Installing SCCM CB 1706 update:
1. Like any other previous updates, first run the Run Prerequisite check or run the Install update Pack

2. The installer will start the Configuration Manager Updates wizard. Click Next on the General tab;

3.  Select required features to be installed then click Next;

4.  Select required client update options then click Next;

 5. Review and confirm the selected options then click Next;

6. Close the completion window.

7. Now the 1706 update state will change from Ready to Install to Installing;

8. We can view the detailed progress of the update installation from \Monitoring\Overview\Updates 
      and Servicing Status\Configuration Manager 1706 node From the ribbon click on Show status
   We can also monitor the upgrade process hman.log file.

 9. Update Pack installation Status window open with task name and status;

10. It will take 20-30 min (based on the server performance) to complete the upgrade.
      Once the update is installed, Configuration Manager 1706 update status will be changed from    
       Available to Installed.

Console Upgrade:
After upgrading the site server to SCCM Current Branch 1706, If we re-launch or check the console version, we will get a popup message saying A new version of the console is available (5.00.8540.1300).
Click OK to upgrade the console and follow the on screen prompts to complete the upgrade process.

Once the update is installed the version number of SCCM will be;
         System Center Configuration Manager Version: 1706
         Console Version: 5.00.8540.1300
         Site Version: 5.0.8540.1000

Click here for complete SCCM 1511 Current Branch setup step by step guide.
Click here for complete SCCM 1511 Current Branch step by step guide, step by step migration guide, step by step monitoring and health check guide and step by step SCCM Current Branch servicing guide.

Saturday, July 8, 2017

MP has discarded a report when processing Relay

You may see below error message for SMS_MP_CONTROL_MANAGER  in component status node under monitoring after upgrading SCCM Current Branch to 1702

MP has discarded a report when processing Relay.
Possible cause: Corruption or invalid user definition.
Solution: Check the logs to identify the cause. If the problem is persistent raise the issue with Microsoft support

Background history:
SMS_MP_CONTROL_MANAGER reporting MP has discarded a report when processing Relay error after upgrading SCCM CB to 1702.
All the MP's are reporting similar error since the SCCM branch upgrade.

<InstallDIr>\SMS\MP\OUTBOXES\\bad folder contains thousands of badrelay XML files.

Reviewing MP_Retry.log file (Can be found  C:\Windows\ccm\logs on the server where MP is installed) shows;
instance of MpEvent_DiscardingInventoryReport
Mp Task: saved bad report as D:\SMS\mp\outboxes\\bad\badRelayZGERVQ7W.XML

The reason for this error is, this is the On Demand request as per MP_Location which causes the above error. These are generated for the Packages where On Demand distribution is enable.

To find out which packages configured with On Demand distribution is enabled, run the below query in SQL Management Studio, then look for  PKG_DISTRIBUTE_ON_DEMAND value is 1.

   (PkgFlags&0x40000000)/0x40000000 AS PKG_DISTRIBUTE_ON_DEMAND 
  dbo.v_Package pkg

Once you have identified and removed the on demand distribution, the error will stop and you can see the number of logs in bad folder gradually go away. It will take few days before the folder gets empty.

Friday, July 7, 2017

SCCM report for packages where on demand distribution is enabled

This query will provide package name and on demand distribution status.
If on demand distribution status is set to 1, then on demand is enabled for that package.
SELECT  Name, 
   (PkgFlags&0x40000000)/0x40000000 AS PKG_DISTRIBUTE_ON_DEMAND 
FROM  dbo.v_Package pkg 
More SCCM custom reports can be found here

Saturday, June 24, 2017

Remote configuration failed on WSUS

On one of the recent patch Tuesday, our Software Update Point (SUP) had issues and failed to sync with below error messages;
Remote configuration failed on WSUS
wsyncmgr.log  shows;
Sync failed: WSUS server not configured. Please refer to WCM.log for configuration error details.. Source: CWSyncMgr::DoSync wsyncmgr.log

WCM.log shows;
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host~~   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~   --- End of inner exception stack trace ---~~   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)~~   --- End of inner exception stack trace ---~~   at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args)~~   at Microsoft.SystemsManagementServer.WSUS.WSUSServer.ConnectToWSUSServer(String ServerName, Boolean UseSSL, Int32 PortNumber)

The same SUP has been working fine until today. No configuration changes been made.
When I checked the software update point the CPU usage was so high that I cannot even launch the WSUS console.
It could be a coincident that the high CPU usage is because of the clients scanning for new updates.
Due to the high CPU usage, most of the clients were failed to complete the scan and as well as the upstream server failed to complete the update sync.

So I have noticed three main issues;
1. Upstream sync failed
2. Client failing to scan for the updates
3. Unable to launch the WSUS console

So to get the upstream sync successful, it is required to bring the CPU utilisation down by blocking the client devices to scan the updates.

Note: WSUS sync will stop working because of so many reasons. However, if it was working in the past and stopped working suddenly without any configuration changes to the infrastructure then most likely it would be an issue with the load on the server and server resources availability. First to eliminate the server load, block the clients to reach out to the SUP. Then test launching the WSUS console then try upstream server sync.Without blocking the client access to the SUP, even if we re-install WSUS and SUP we will have the same issue again.

To block client machines to scan for the updates, On Software Update Point;
go to C:\program Files\update services\webservices then re-name ClientWebService folder to something else.
This will make the Software Update Point unavailable to the clients. Once the sync is completed, then re-name the folder back to ClientWebService then restart the IIS services.
Now the upstream sync successfully completed and the clients also completed the scan successfully.

On the side note, make sure all the expired and superseded updates are removed from the WSUS database periodically. This is an important process in keeping the WSUS performance up. The more expired and superseded updates in the database, the slower the performance will be.

Saturday, April 29, 2017

Error: VSS_E_WRITERERROR_TIMEOUT. Error Code = 0x80,042,3f2

SCCM site database maintenance task failed with one of the below error;

After GatherWriterMetadata SMS Writer status = FAILED_AT_PREPARE_BACKUP.
Error: VSS_E_WRITERERROR_TIMEOUT. Error Code = 0x80,042,3f2.

ERROR: SQL Backup task failed. Error message - Asynchronous QueryStatus failed or did not finish properly.
STATMSG: ID=5052 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_SITE_BACKUP" SYS=sccb.w2k8.lab SITE=SCB PID=35284 TID=34312 GMTDATE=Wed Apr 26 23:25:37.402 2017 ISTR0="\\sccb.w2k8.lab" ISTR1="CM_CP1;" ISTR2="Asynchronous QueryStatus failed or did not finish properly." ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0
Error: Sql Server could not prepare for the Backup.

Current state of SQL backup: [failed]. Aborting it.
SMS_SITE_BACKUP 4/26/2017 11:25:37 PM 34312 (0x8608)

SMS_SITE_BACKUP failed. Please see previous errors.

Above errors can be seen in smsbkup.log under <CM installDir>\Program Files\Microsoft Configuration Manager\Logs

Below event viewer msgs can be seen on the site database server;

Event ID: 12346
Volume Shadow Copy Error: An error 0x8007000e, Not enough storage is available to complete this operation.
 was encountered while trying to initialize the Registry Writer.  This may cause future shadow-copy creations to fail.

Event ID: 12347
Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers.  The Registry Writer failed to respond to a query from VSS. Check to see that the Event Service and Volume Shadow Copy Service are operating properly, and please check the Application event log for any other events.

   Gathering Writer Data
   Executing Asynchronous Operation

   Execution Context: Requestor
   Current State: GatherWriterMetadata

If we investigate, further;
- COM+ Event System service is running
- Volume Shadow Copy service is running
- SMS_SITE_VSS_WRITER is running and can restart the service without any error
- On the command line run vssadmin list writers  and should list all the writers

When I ran vssadmin list writers on the site server, I have received various vss writers names and with last error. All of the writers showed stable except SMS Writer.
For the SMS Writer, state: [7] Last error: Timed out.

Then ran vssadmin list writers on the Site database server and received empty list.

A quick google showed me various articles to re-register the dll’s to initialise the writers.
However decided to restart the server as it has been up and running for good few months.
After the restart, I re-ran the vssadmin list writers  and this time it showed various writers.

Now initiated backup by starting the SMS_SITE_BACKUP service.
Even though I have received "After GatherWriterMetadata SMS Writer status = FAILED_AT_PREPARE_BACKUP. Error: VSS_E_WRITERERROR_TIMEOUT. Error Code = 0x80,042,3f2." Error in the beginning of the backup process, it has completed the site backup successfully.

The reason for the site database backup failure was, on the site database server the Registry Writer was not present. For some reason all the VSS writers had disappeared from the database server.
After restarting the server, the writers were back and everything worked as normal.