Summary: Guest blogger, Rudolf Vesely, shows how to evaluate time synchronization.
Microsoft Scripting Guy, Ed Wilson, is here. Today is day 1 of a 3-part series by Rudolf Vesely. Here, Rudolf tells us about himself:
I work as a lead cloud architect in Tieto Productivity Cloud (TPC) at Tieto Corporation. I am a blogger on the Technology Stronghold blog, and I am the author of large series of articles called How to Build Microsoft Private Cloud. I started my career in IT as a developer (assembler, object-oriented programming, C++, C#, .NET, and ASP.NET). However, I moved to operations many years ago. Programming and scripting still remain my hobbies, and this is probably the reason why I am a big propagator of DevOps in my company. I believe Windows PowerShell is a fantastic tool that facilitates the process of joining dev and ops worlds together, and I am very glad that I started learning with Windows PowerShell 1.0.
Contact information:
- Blog: Technology Stronghold
- Twitter: @rudolfvesely
Introducing the problem
In enterprise environments, you should never ever use standard scripts for monitoring because such scripts are not monitored or highly available. Also you should avoid situations where scripts that are developed by multiple persons run on multiple servers in an uncontrolled way.
In enterprise environments, you need to have a unified platform and use enterprise-grade services for your automation scripts—and Microsoft has an answer for that. The answer is Service Management Automation (SMA) and System Center Operations Manager (SCOM) to monitor SMA.
SMA ensures high availability and SCOM ensures that you are notified about failures. In some cases, you do not have to be notified about individual failures that could be caused, for example, by network transient failures. But you definitely should be notified about repetitive failures, and this is perfect place for SCOM monitoring. SMA requires workflows, so this is another reason why you should learn about Windows PowerShell workflows.
For testing and learning purposes, I have Hyper-V clusters and a few independent Hyper-V hosts at home. Several months ago I found that I had an issue with time synchronization. The issue was not technical because the time was synchronized across my whole domain. But it was very confusing to have the wrong time on all my devices.
I decided that I need a monitoring system that is able to autocorrect, so I wrote a Windows PowerShell module with five cmdlets. This module is able to do these tasks and much more. I choose to use a workflow over an advanced function because I wanted to be able to monitor all servers in parallel and to run monitoring tasks as an SMA runbook.
There are two ways to get my free code:
- To download the module from the Microsoft TechNet Gallery, see Workflows to get status, invoke, test, monitor and repair Time Synchronization.
- To get the full source code from my blog, see PowerShell Module for Time Synchronization - PowerShell Workflows to get status, start, invoke, test, monitor and repair Time Sync.
There are following cmdlets (workflows) in the module:
- Get-VDateTimeInternetUtc
- Get-VSystemTimeSynchronization
- Start-VSystemTimeSynchronization
- Wait-VSystemTimeSynchronization
- Test-VSystemTimeSynchronization
Wait-VSystemTimeSynchronization depends on Get-VSystemTimeSynchronization and Start-VSystemTimeSynchronization, and Test-VSystemTimeSynchronization depends on all the other workflows.
Getting started
Let’s begin…
Most of the logic is done via Get-VSystemTimeSynchronization, which gathers information about the current state of time synchronization from local or multiple servers in parallel. I always initialize my functions with:
$ErrorActionPreference = 'Stop'
if ($PSBoundParameters['Debug']) { $DebugPreference = 'Continue' }
Set-PSDebug -Strict
Set-StrictMode -Version Latest
But in the case of workflows, I do not have so many possibilities, so I use:
$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'
Now I continue with an inline script. You should avoid using inline scripts if possible because code in an inline script is not a workflow. In my workflow, I decided to use inline script for two reasons:
- I do a lot of actions that cannot be done in the workflow.
- I wanted to use InlineScript remoting: InlineScript {} -PSComputerName.
Unfortunately, there is an issue in how inline scripts reuse remote sessions when the PSComputerName parameter is used. I observed that if I run an inline script remotely multiple times in a short period and with different parameters, the old set of parameters is used instead of the new one. In my case, this is not an issue because it is unlikely that someone would change the parameters often.
The inline script starts with PSCustomObject, and during execution, I fill its nulled properties:
$outputItem = [PsCustomObject]@{
DateTimeUtc = $null
ComputerNameNetBIOS = $env:COMPUTERNAME
ComputerNameFQDN = $null
# For example: @('0.pool.ntp.org', '1.pool.ntp.org', '2.pool.ntp.org', '3.pool.ntp.org')
ConfiguredNTPServerName = $null
# For example: '0.pool.ntp.org,0x1 1.pool.ntp.org,0x1 2.pool.ntp.org,0x1 3.pool.ntp.org,0x1'
ConfiguredNTPServerNameRaw = $null
# True if defined by policy
ConfiguredNTPServerByPolicy = $null
SourceName = $null
SourceNameRaw = $null
LastTimeSynchronizationDateTime = $null
LastTimeSynchronizationElapsedSeconds = $null
ComparisonNTPServerName = $(if ($Using:CompareWithNTPServerName) { $Using:CompareWithNTPServerName } else { $null })
ComparisonNTPServerTimeDifferenceSeconds = $null
# Null when no source is required, True / False when it is required
StatusRequiredSourceName = $null
# Null when no type is required, True / False when it is required
StatusRequiredSourceType = $null
# True when date is not unknown, False when it is unknown
StatusDateTime = $null
# Null when maximum of seconds was not specified, True / False when it was specified
StatusLastTimeSynchronization = $null
# Null when no comparison or when not connection and error should be ignored, True / False when number of seconds was obtained
StatusComparisonNTPServer = $null
Status = $null
StatusEvents = @()
Error = $null
ErrorEvents = @()
}
Win32Time service
Then I start the W32Time (Windows Time) service because the w32tm command requires it. As you can see, all parts of the code that can possibly generate an exception are enclosed in Try/Catch block because I do not want to stop the execution of the script, and I want to have information about any exception in the ErrorEvents property of the output object.
try
{
if ((Get-Service -Name W32Time).Status -ne 'Running')
{
Write-Verbose -Message '[Get] [Start service] [Verbose] Start service: W32Time (Windows Time)'
Start-Service -Name W32Time
}
}
catch
{
$outputItem.ErrorEvents += ('[Get] [Start service] [Exception] {0}' -f $_.Exception.Message)
}
Now I continue with gathering data. I need the FQDN (if the server has a suffix), and I need output from w32tm /query /status, which will be processed further.
try
{
# W32tm
$w32tmOutput = & 'w32tm' '/query', '/status'
# FQDN
$ipGlobalProperties = [System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties()
if ($ipGlobalProperties.DomainName)
{
$outputItem.ComputerNameFQDN = '{0}.{1}' -f
$ipGlobalProperties.HostName, $ipGlobalProperties.DomainName
}
else
{
$outputItem.ComputerNameFQDN = $null
}
}
catch
{
$outputItem.ErrorEvents += ('[Get] [Gather data] [Exception] {0}' -f $_.Exception.Message)
}
The NTP servers
Then I want a list of configured NTP servers. This configuration could be set by Group Policy (would have precedence):
HKLM\SOFTWARE\Policies\Microsoft\W32Time\Parameters
Or I can get it directly in the registry by using the w32tm command:
HKLM\SYSTEM\CurrentControlSet\Services\W32Time\Parameters
It is possible to split the list of servers into array and remove the configuration flag:
try
{
if (Test-Path -Path HKLM:\SOFTWARE\Policies\Microsoft\W32Time\Parameters -PathType Container)
{
$configuredNtpServerNameRegistryPolicy = Get-ItemProperty `
-Path HKLM:\SOFTWARE\Policies\Microsoft\W32Time\Parameters `
-Name 'NtpServer' -ErrorAction SilentlyContinue |
Select-Object -ExpandProperty NtpServer
}
else
{
$configuredNtpServerNameRegistryPolicy = $null
}
if ($configuredNtpServerNameRegistryPolicy)
{
$outputItem.ConfiguredNTPServerByPolicy = $true
# Policy override
$outputItem.ConfiguredNTPServerNameRaw = $configuredNtpServerNameRegistryPolicy.Trim()
}
else
{
$outputItem.ConfiguredNTPServerByPolicy = $false
# Exception if not exists
$outputItem.ConfiguredNTPServerNameRaw = ((Get-ItemProperty `
-Path HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Parameters -Name 'NtpServer').NtpServer).Trim()
}
if ($outputItem.ConfiguredNTPServerNameRaw)
{
$outputItem.ConfiguredNTPServerName = $outputItem.ConfiguredNTPServerNameRaw.Split(' ') -replace ',0x.*'
}
}
catch
{
$outputItem.ErrorEvents += ('[Get] [Configured NTP Server] [Exception] {0}' -f $_.Exception.Message)
}
Let‘s continue and get the current source that the system uses. The source is specified in the output of w32tm /query /status command that was executed at the beginning.
$sourceNameRaw =
$sourceNameRaw.ToString().Replace('Source:', '').Trim()
$outputItem.SourceNameRaw = $sourceNameRaw
$outputItem.SourceName = $sourceNameRaw -replace ',0x.*'
I implemented a lot of switches that you can enable and specify as the desired state. Then you do not have to spend time going through output from hundreds of servers—you can check the output from those that do not have a Status property with an output object $true.
Check that current source is equal to one of the NTP servers specified in the –RequiredSourceName parameter:
if ($Using:RequiredSourceName -contains $outputItem.SourceName)
{
$outputItem.StatusRequiredSourceName = $true
}
else
{
$outputItem.StatusRequiredSourceName = $false
$outputItem.ErrorEvents += ('[Get] [Source name] [Error] Current: {0}; Required: {1}' -f
$outputItem.SourceName, ($Using:RequiredSourceName -join ', '))
}
Checking the source
First check that the source is not the internal clock:
if (($Using:RequiredSourceTypeConfiguredInRegistry -or $Using:RequiredSourceTypeNotLocal) -and
($outputItem.SourceNameRaw -eq 'Local CMOS Clock' -or
$outputItem.SourceNameRaw -eq 'Free-running System Clock'))
{
$outputItem.StatusRequiredSourceType = $false
$outputItem.ErrorEvents += ('[Get] [Source type] [Error] Time synchronization source: Local')
}
Check that the current source is not the Hyper-V service:
if (($Using:RequiredSourceTypeConfiguredInRegistry -or $Using:RequiredSourceTypeNotByHost) -and
$outputItem.SourceNameRaw -eq 'VM IC Time Synchronization Provider')
{
$outputItem.StatusRequiredSourceType = $false
$outputItem.ErrorEvents += ('[Get] [Source type] [Error] Time synchronization source: Hyper-V')
}
Check that the current source is equal to one of the configured NTP servers. For example, when the source should be an NTP server, but time synchronization does not work, the current source is “Local CMOS Clock.”
if ($Using:RequiredSourceTypeConfiguredInRegistry -and
$outputItem.ConfiguredNTPServerName -notcontains $outputItem.SourceName)
{
$outputItem.StatusRequiredSourceType = $false
$outputItem.ErrorEvents += ('[Get] [Source type] [Error] Not equal to one of the NTP servers that are define in Windows Registry')
}
Now it is time to check when the last synchronization happened. Get the data from w32tm /query /status that was executed at the beginning:
$lastTimeSynchronizationDateTimeRaw = $w32tmOutput |
Select-String -Pattern '^Last Successful Sync Time:'
$outputItem.StatusDateTime = $false
if ($lastTimeSynchronizationDateTimeRaw)
{
$lastTimeSynchronizationDateTimeRaw =
$lastTimeSynchronizationDateTimeRaw.ToString().Replace('Last Successful Sync Time:', '').Trim()
Calculate time since last sync
Now calculate and evaluate the time duration since the last synchronization. If the last time sync is unknown or if it is processed longer ago than is specified, save a record about it. The record will be included in the output object.
if ($lastTimeSynchronizationDateTimeRaw -eq 'unspecified')
{
$outputItem.ErrorEvents += '[Last time synchronization] [Error] Date and time: Unknown'
}
else
{
$outputItem.LastTimeSynchronizationDateTime = [DateTime]$lastTimeSynchronizationDateTimeRaw
$outputItem.LastTimeSynchronizationElapsedSeconds = [int]((Get-Date) - $outputItem.LastTimeSynchronizationDateTime).TotalSeconds
$outputItem.StatusDateTime = $true
<#
Last time synchronization: Test: Maximum number of seconds
#>
if ($Using:LastTimeSynchronizationMaximumNumberOfSeconds -gt 0)
{
if ($outputItem.LastTimeSynchronizationElapsedSeconds -eq $null -or
$outputItem.LastTimeSynchronizationElapsedSeconds -lt 0 -or
$outputItem.LastTimeSynchronizationElapsedSeconds -gt $Using:LastTimeSynchronizationMaximumNumberOfSeconds)
{
$outputItem.StatusLastTimeSynchronization = $false
$outputItem.ErrorEvents += ('[Get] [Last time synchronization] [Error] Elapsed: {0} seconds; Defined maximum: {1} seconds' -f
$outputItem.LastTimeSynchronizationElapsedSeconds, $Using:LastTimeSynchronizationMaximumNumberOfSeconds)
}
else
{
$outputItem.StatusLastTimeSynchronization = $true
}
}
}
One of the additional features is to compare time between a target computer (local or a remote server) and the specified NTP server. There is no reason to compare time between a computer and an NTP server that is used for regular time synchronization, but it is possible to use this feature to compare time, for example, against a public NTP server (a firewall opening for UDP 123 is required).
At first, I use w32tm to get the exact time difference between the target server and the specified NTP server:
$w32tmOutput = & 'w32tm' '/stripchart',
('/computer:{0}' -f $Using:CompareWithNTPServerName),
'/dataonly', '/samples:1' |
Select-Object -Last 1
Now calculate and evaluate the time duration like in the previous case:
if ($outputItem.ComparisonNTPServerTimeDifferenceSeconds -eq $null -or
$outputItem.ComparisonNTPServerTimeDifferenceSeconds -lt ($Using:CompareWithNTPServerMaximumTimeDifferenceSeconds * -1) -or
$outputItem.ComparisonNTPServerTimeDifferenceSeconds -gt $Using:CompareWithNTPServerMaximumTimeDifferenceSeconds)
{
$outputItem.ErrorEvents += ('[Get] [Compare with NTP] [Error] Elapsed: {0} seconds; Defined maximum: {1} seconds' -f
$outputItem.ComparisonNTPServerTimeDifferenceSeconds, $Using:CompareWithNTPServerMaximumTimeDifferenceSeconds)
}
else
{
$outputItem.StatusComparisonNTPServer = $true
}
Evaluate overall status
And that is all from the inline script. Now it is time to evaluate overall status and return the object. Overall status is $false when a single error or exception occurred, when a single value was not in the specified range, or when the current state is not the specified desired state:
if ($outputItem.ErrorEvents)
{
Write-Warning -Message ('[Get] Results: False: {0}' -f ($outputItem.ErrorEvents -join "; "))
$outputItem.Status = $false
$outputItem.Error = $true
}
else
{
$outputItem.Status = $true
$outputItem.Error = $false
}
$outputItem.DateTimeUtc = (Get-Date).ToUniversalTime()
$outputItem
The following image shows the output from the evaluation:
That is all for today. Tomorrow, I will focus more on workflows, and I will explain how to do error handling in parallel operations.
~Rudolf
Thanks, Rudolf. This is great stuff. I am really looking forward to Part 2.
I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.
Ed Wilson, Microsoft Scripting Guy