UR5 for SCOM 2012 R2

Kevin Holman Blog:

KB Article for OpsMgr:  http://support.microsoft.com/kb/3023138

KB Article for all System Center components:  http://support2.microsoft.com/kb/3021802

Download catalog site:  http://catalog.update.microsoft.com/v7/site/Search.aspx?q=3023138

Key fixes:

• Monitoringhost process crashes because of bind failures against Active Directory
  A fix was made to prevent Monitoringhost.exe from crashing when it cannot connect to Active Directory Domain Services (AD DS). The stack trace from the crash displays a "System.DirectoryServices.ActiveDirectory.ActiveDirectoryServerDownException" error.
• RunAs accounts cannot be edited because of "Specified cast is invalid" exception
  RunAs accounts can be distributed only to a selected computer through the distribution tab of Run As Account properties. When a computer that is in the distribution list is decommissioned from Operations Manager and the Run As account is opened, you receive the following exception, and no computers are shown on the list:

  System.InvalidCastException: Specified cast is not valid. at Microsoft.EnterpriseManagement.Monitoring.Internal.MonitoringObjectGenerated.get_Id()

• Application crashes when a search is finished without filter criteria on Distributed application designer
  A fix was made to make sure that the console does not crash when a search is finished without filter criteria on Distributed application designer.
• MonitoringHost crashes with the exception - System.OverflowException: Value was either too large or too small for an Int32
  Changes to entities are tracked by using the EntityChangeLogId in the EntityChangelog table and the EntityChangeLogId is mastered in the table as an Int64 value. ManagedType tables that begin with a MT_* refer to the EntityChangeLogID. The code that was used to insert into the ManagedType table had an Int32 data type instead of an Int64 data type. When the EntityChangeLogID reached greater than the Int32 limit, the exception crashed the MonitoringHost. This is fixed and the MonitoringHost no longer crashes.
• Support to troubleshoot PowerShell scripts
  When traces are collected to troubleshoot PowerShell scripts, the logs do not display the parameters that are passed to PowerShell scripts. A code change was made to log this information to make troubleshooting easy.
• Operational Insights through Operations Manager
  System Center Advisor is improved with many exciting features and is rebranded as "Operational Insights." Learn more about the new Operational Insightsfeature. The product name is updated in the Operations Manager plug-in for System Center Advisor.
• "Reset the Baseline," "Pause the baseline," and "Resume the baseline" tasks fail when they are run against an Optimized performance collection rule
  You receive the following exception when this problem occurs:

  Microsoft.EnterpriseManagement.ContainerException: The container could not find a component with name 'TaskRuntime' compatible with type 'Microsoft.EnterpriseManagement.ServiceDataLayer.ITaskRuntimeServiceInternal, Microsoft.EnterpriseManagement.DataAccessService.Core

• An exception occurs when you edit subscriptions that contain deleted monitors or objects
  A subscription is created by using the "created by specific rules or monitors" criterion. When such a subscription is edited and the monitors or rules associated with them are deleted, you receive the following exception:

  System.NullReferenceException: Object reference not set to an instance of an object.
  at Microsoft.EnterpriseManagement.Mom.Internal.UI.Controls.SourceChooserCriteriaItem.GetSources()
  at Microsoft.EnterpriseManagement.Mom.Internal.UI.Controls.SourceChooserCriteriaItem.Search(CancelFlagWrapper cancelFlag)

  This is fixed not to throw an exception and display the other valid rules or monitors.
• You can't set Widget column width
  When an explicit column width is set for a widget, the width is not honored in the grid that is displayed. This was caused because of an incorrect Regex validation expression. This has now been fixed and the explicit column width should start reflecting in the UI.
• Event 4506: Data was dropped because of too much outstanding data
  When too much data is pushed into a module, Operations Manager logs 4506 messages and then drops data. The hardcoded limit is 5120 items globally (in the agent). This global limit is parameterized and exposed through the registry. You can set the value by creating one of the two following registry keys:

  Registry location:

  HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\System Center\Health Service

  DWORD Name: MaximumGlobalPendingDataCount
  DWORD Value:nnnn
  (Where nnnn is the number of items globally. Set this value to more than 5120, as needed.)
  Registry location:

  HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HealthService\Parameters

  DWORD Name: MaximumGlobalPendingDataCount
  DWORD Value:nnnn
  (Where nnnn is the number of items globally. Set this value to more than 5120, as needed.)

• Support for handling Datawarehouse time-outs
  Operations Manager datawarehouse throws events such as 31551, 31552, 31553 when there are problems inserting data to the datawarehouse. One reason these events can occur are because of SQL time-out exceptions. When such events related to SQL time-out exceptions are logged, the logs now also contain information about how to extend these timeouts. You can read more about these exceptions here
• Can't continue scan with NOLOCK because of data movement
  When a component such as Orchestrator tries to call the Operations Manager database function fn_AlertViewChanges, it could throw the "Could not continue scan with NOLOCK because of data movement" exception. This is triggered by a redundant NOLOCK statement in the SQL SP. This is fixed, as are any failures that are caused by this exception.
• $ScriptContext.Context does not persist the value in PowerShell widgets
  PowerShell widgets that use $ScriptContext.Context display a blank. With this fix, the context now persists.
• Evaluation version alert
  Operations Manager previously displayed a Warning 60 days, 30 days, 1 day, and last-day hourly before the expiration of the evaluation period. This has been changed to an error so that you know Operations Manager will stop working as soon as the evaluation period is over. It also contains a link to a KB article that helps you convert the evaluation version to a registered version.

 

Xplat updates:

• Updating the UNIX/Linux agent for Operations Manager resets deep monitoring status for JEE application servers.
  This issue applies to Java Application Server instances that have “Deep Monitoring” enabled. When you upgrade the UNIX/Linux agent for Operations Manager, the instances are no longer “Deep Monitored,” and they require another execution of the Enable Deep Monitoring management pack task.
• By default, the rpcimap monitor for Red Hat Enterprise Linux 5 is now disabled.
  Red Hat Enterprise Linux 5.10 and later versions removes the default rpcimap daemon that's present in earlier Red Hat Enterprise Linux 5 releases. Critical alerts are generated for the rpcimap service that's not running on Red Hat Enterprise Linux 5.10. By default, the rpcimap monitor for Red Hat Enterprise Linux 5 is now disabled. However, it can be re-enabled by an override in the management pack.
• Monitoring in time zone UTC +13 (for example, Auckland, New Zealand with daylight saving time) causes the “Unix Process Monitoring template” to fail.
  This issue applies to all UNIX/Linux servers that are located in the UTC +13 time zone. When you try to use the Unix Process Monitoring template for a server that's located in UTC +13 time zone, the following warning may be logged in the agent’s log (/var/opt/microsoft/scx/log/scx.log):Warning [scx.core.providers.provess_provider:266:2250:140361722332992] SCX_UnixProcess_Class_Provider: EnumerateInstances - Formal argument 'offsetFromUTC' is invalid: SCXRelativeTime: years=0 months=0 days=0 hours=0 minutes=780 microseconds=0 - [/home/serviceb/ScxCore_R2URNext_Debian50_x64/pal/source/code/scxcorelib/pal/scxtime/absolute.cpp:856]
• When you use the "scxadmin -log-rotate" utility, this stops logging to the agent’s log (/var/opt/microsoft/scx/log/scx.log) after log rotation is completed.
  When you use the scxadmin program together with the -log-rotate option, logging in /var/opt/microsoft/scx/log/scx.log may not resume unless a restart is issued (scxadmin -restart).

Lets get started.

From reading the KB article – the order of operations is:

1. Install the update rollup package on the following server infrastructure:
   • Management servers
   • Gateway servers
   • Web console server role computers
   • Operations console role computers
2. Apply SQL scripts.
3. Manually import the management packs.
4. Update Agents

Now, we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents.

       5.  Update Unix/Linux MP’s and Agents.

1.  Management Servers

Since there is no RMS anymore, it doesn’t matter which management server I start with.  There is no need to begin with whomever holds the RMSe role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 3 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

Then extract the contents:

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update does not provide any feedback that it had success or failure.  You can check the application log for the MsiInstaller events for that:

Log Name:      Application
Source:        MsiInstaller
Date:          2/22/2015 1:40:06 PM
Event ID:      1036
Task Category: None
Level:         Information
Keywords:      Classic
User:          OPSMGR\foo
Computer:      SCOM01.opsmgr.net
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2012 Server. Product Version: 7.1.10226.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2012 R2 Operations Manager UR5 Update Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute. 

Next up – run the Web Console update:

This runs much faster.   A quick file spot check:

Lastly – install the console update (make sure your console is closed):

A quick file spot check:

Secondary Management Servers:

I now move on to my secondary management servers, applying the server update, then the console update. 

On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files.  I check online, and make sure that I have configured Windows Update to give me updates for additional products:

This shows me two applicable updates for this server:

I apply these updates (along with some additional Windows Server Updates I was missing, and reboot each management server, until all management servers are updated.

Updating Gateways:

I can use Windows Update or manual installation.

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

2. Apply the SQL Scripts

In the path on your management servers, where you installed/extracted the update, there are two SQL script files: 

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups
(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

First – let’s run the script to update the OperationsManager database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file.  Make sure it is pointing to your OperationsManager database, then execute the script.

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

or

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a large environment, you might have to run this several times, or even potentially shut down the services on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:  This script has been updated in UR5.  Even if you previously ran this script in UR1, UR2, UR3, or UR4, you must run this again.

Next, we have a script in UR5 to run against the warehouse DB.  Do not skip this step under any circumstances.    From:

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups
(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql.  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

If you see a warning about line endings, choose Yes to continue.

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

3. Manually import the management packs?

There are 25 management packs in this update!

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor, and language specific.  Only import the ones you need, and that are correct for your language.  I will remove all the Advisor MP’s for other languages, and I am left with the following:

The TFS MP bundles are only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc.  If you are not currently using these MP’s, there is no need to import or update them.  I’d skip this MP import unless you already have these MP’s present in your environment.

The Advisor MP’s are only needed if you are using System Center Operational Insights (Previously known as Advisor) services.

However, the Image and Visualization libraries deal with Dashboard updates, and these always need to be updated.

I import all of these without issue.

4.  Update Agents

Agents should be placed into pending actions by this update (mine worked great) for any agent that was not manually installed (remotely manageable = yes):

 

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending.

In this case – my agents that were reporting to a management server that was updated using Windows Update – did NOT place agents into pending.  Only the agents reporting to the management server for which I manually executed the patch worked.

You can approve these – which will result in a success message once complete:

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

5.  Update Unix/Linux MPs and Agents

Next up – I download and extract the updated Linux MP’s for SCOM 2012

http://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1042.0 is current at this time for SCOM 2012 R2 UR5.
****Note – take GREAT care when downloading – that you select the correct download for R2.  You must scroll down in the list and select the MSI for 2012 R2:

Download the MSI and run it.  It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP’s you are already using.   These are mine for RHEL, SUSE, and the Universal Linux libraries.  Looks like we have added RHEL7 and SUSE12 since I last updated:

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports.  Give it plenty of time to complete the process of the import and MPB deployments.

Next up – you would upgrade your agents on the Unix/Linux monitored agents.  You can now do this straight from the console:

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

Mine FAILED, with an SSH exception about copying the new agent.  It turns out my files were not updated on the management server – see pic:

I had to restart the Healthservice on the management server, and within a few minutes all the new files were there.

Finally:

6.  Update the remaining deployed consoles

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the UR5 update.

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

Known issues:

See the existing list of known issues documented in the KB article.

1.  Many people are reporting that the SQL script is failing to complete when executed.  You should attempt to run this multiple times until it completes without error.  You might need to stop the Exchange correlation engine, stop all the SCOM services on the management servers, and/or bounce the SQL server services in order to get a successful completion in a busy management group.  The errors reported appear as below:

------------------------------------------------------
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
--------------------------------------------------------