Summary: Guest blogger, Laerte Junior, discusses using Windows PowerShell to troubleshoot SQL Server by parsing the SQL error log.
Microsoft Scripting Guy, Ed Wilson, is here. Today we have another guest blogger, Laerte Junior. Here is a little bit about Laerte.
Laerte Junior is a SQL Server specialist and an active member of WW SQL Server and the Windows PowerShell community. He also is a huge Star Wars fan (yes, he has the Darth Vader´s Helmet with the voice changer). He has a passion for DC comics and living the simple life. May The Force be with all of us. If you want to see more about what we can do with Windows PowerShell and SQL Server, don´t miss my LiveMeeting on July 18. For more information, see LiveMeeting VC PowerShell PASS–Troubleshooting SQL Server With PowerShell–English.
Take it away, Laerte…
One of the most basic and efficient troubleshooting steps that a SQL Server DBA does is to look for error messages in the SQL Server error log. It is a relatively simple task when it comes to working with a single server or even a few servers running SQL Server. However, this starts to be hard job when we think about a corporate-model environment consisting of several hundred SQL Server instances. In today’s blog, we will see how to do this troubleshooting—even when the SQL Server instance is offline. The technique will scale to N servers, and produce various types of output. I hope you enjoy the blog because it has saved me a lot of time in my daily job.
If The SQL Server instance is online and you want to check the SQL error log, it is easy to use the SMO Server. An example of doing this is presented in the excellent Hey, Scripting Guy! Blog post with information from Aaron Nelson, Use PowerShell to Get the SQL Server Error Log.
Use WMI to query an offline event log
OK, something happens on my server, the SQL Server instance is offline, and I need to check the SQL error log. How can I do this? Well, two ways of doing this are to use WMI or use SQL Server Management Studio (SSMS). A feature to read the SQL Server error log offline was introduced in SQL Server 2012. This version added two new WMI classes to the Management WMI Provider. The classes are in the ComputerManagement11 namespace: root\Microsoft\SqlServer\ComputerManagement11. The two classes are SqlErrorLogFile and SqlErrorLogEvent.
By using SSMS or the Graphical SQL Server Client tool, you can easily access the SQL error log, but it starts to be painful when you need to do it for more than one instance and if you need to filter the message to look for a specific error. Because of the difficulty in working with multiple instances and searching multiple logs, I want to use Windows PowerShell.
On a test SQL Server instance, stop the SQL Server service. First, let´s take a look at the Windows Management Instrumentation classes that are contained in the ComputerManagement11 namespace. The following code accomplishes this task.
Get-WmiObject -Namespace "Root\Microsoft\SqlServer\ComputerManagement11" –List
The output from the previous command is shown here.
Yes, you can play around with ALL these classes. Is that cool or what?
The following code illustrates querying the local computer and the default SQL Server instance.
Get-WmiObject -Class SqlErrorLogEvent -computername MylocalComputer -Namespace "Root\Microsoft\SqlServer\ComputerManagement11"
In this example, I illustrate querying the local computer and a named instance of SQL Server, inst1.
Get-WmiObject -Query "Select * from SqlErrorLogEvent where InstanceName = 'Inst1'"-Namespace "Root\Microsoft\SqlServer\ComputerManagement11"
Notice that unlike WMI for server events, each SQL Server instance has your path to the root. The path to the WMI Management Provider is always one, even with several SQL Server instances. What you need to do to access the right instance is filter the InstanceName property in your WQL.
How do I know which properties I can use? Simply pipe the Get-WmiObject to a Get-Member cmdlet. This technique is shown here.
Get-WmiObject -Namespace "Root\Microsoft\SqlServer\ComputerManagement11" -Class SqlErrorLogEvent | Get-Member
In addition, you might want to use some of the tools that are mentioned in Hey, Scripting Guy! How Do I Find the Names of WMI Classes?
To work on a remote computer, use the ComputerName parameter as shown here.
Get-WmiObject -Class "SqlErrorLogEvent" `
-ComputerName MyRemoteComputer `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11"
When working with multiple servers and no SQL Server named instance, you can do the following:
1. Create a .txt file with the name of the servers, such as:
Server1
Server2
2. Use the following query:
Get-WmiObject -Class "SqlErrorLogEvent" `
-ComputerName (Get-Content c:\Temp\Servers.txt) `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" |
select @{Expression={($_.__Server) };Label = "Server"},`
InstanceName,@{Expression={([Management.ManagementDateTimeConverter]::ToDateTime($_.logdate)) };Label= "Logdate"},Message,ProcessInfo
The ComputerName parameter in WmiObject is a string[], so its accepts an array of string objects, and you can use it with Get-Content.
We can export the error log to a CSV file, but of course, you don’t want to export ALL of the messages. Let’s create a filter that looks for the words “Error” and “Fail” and excludes the phrase “Found 0 Errors.”
By using Where-Object with the Get-WmiObject cmdlet, I come up with the command that is shown here.
Get-WmiObject -Class "SqlErrorLogEvent" `
-ComputerName (Get-Content c:\Temp\Servers.txt) `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" |
Where-Object { ($_.Message -like "*Error*" `
-or $_.Message -like "*Fail*") `
-and ($_.Message -notlike "*Found 0 Errors*")} |
Select-Object @{Expression={($_.__Server) };Label = "Server"},`
InstanceName,@{Expression={([Management.ManagementDateTimeConverter]::ToDateTime($_.logdate)) };Label= "Logdate"},Message,ProcessInfo
You can create the filter by using WMI Query Language (WQL). This results in the command that is shown here.
$Query = "Select * from SqlErrorLogEvent where (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
Get-WmiObject -Query $query `
-ComputerName (Get-Content c:\Temp\Servers.txt) `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" |
Select-Object @{Expression={($_.__Server) };Label = "Server"},`
InstanceName,@{Expression={([Management.ManagementDateTimeConverter]::ToDateTime($_.logdate)) };Label= "Logdate"},Message,ProcessInfo
The fatal answer that you probably are asking is, “Which is faster?” Using WQL, of course. I performed a benchmark by using Measure-Command, and the result follows:
It is about milliseconds, I know. I did this on three servers, but imagine this on a several hundred servers. In this case, the change can make a BIG difference.
Now we can export the results to a CSV file. The following code accomplishes this task.
$Query = "Select * from SqlErrorLogEvent where (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
Get-WmiObject -Query $query `
-ComputerName (Get-Content c:\Temp\Servers.txt) `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" `
| select @{Expression={($_.__Server) };Label = "Server"},`
InstanceName,@{Expression={([Management.ManagementDateTimeConverter]::ToDateTime($_.logdate)) };Label= "Logdate"},Message,ProcessInfo `
| Export-Csv "c:\temp\SQLErrorLog\SQLErrorLog.csv" `
-NoTypeInformation -Force
Note that the search is performed in ALL of the SQL error logs. If you want to specify the last log or a specific log, just filter by using the FileName property. You can use the LogDate property to also filter by date and time. In the following technique, FileName is the physical SQL error log name, for example, ErrorLog, ErrorLog.0, ErrorLog.1…
$Query = "Select * from SqlErrorLogEvent where FileName = 'Errorlog' and LogDate >= 'First Date' and <= 'Second Date'"
When you work with multiple servers (default and named SQL Server instances), we need to make a simple change in the code because as we saw previously, we need to filter for the specific instance name in the WQL statement. Now in my text file, I have the instance names as shown here:
R2D2
R2D2\Inst1
Yoda
Obiwan
The code can be something like the code that follows:
Get-Content c:\Temp\Servers.txt | % {
#split the computer and SQL Server Instance Name
$SplitNames = $_.split('/')
#is Default instance ?
if ($SplitNames[1] -ne $null -and $SplitNames[1] -ne 'MSSQLSERVER') {
$Query = "Select * from SqlErrorLogEvent where (InstanceName = '$($SplitNames[1])') and (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
} else {
$Query = "Select * from SqlErrorLogEvent where (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
}
#Get the computerName
$ComputerName = $SplitNames[0]
Get-WmiObject -query $query `
-ComputerName $ComputerName `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" `
| select @{Expression={($_.__Server) };Label = "Server"},`
InstanceName,@{Expression={([Management.ManagementDateTimeConverter]::ToDateTime($_.logdate)) };Label= "Logdate"}Message,ProcessInfo
}
Writing the data to a SQL database
Now suppose that I have several servers SQL Server and several named instances. Does the query need to be serialized? No, it can be done in asynchronous mode by using background Windows PowerShell jobs and storing the results it in a SQL Server table. First, you need to download the Chad Miller´s Out-DataTable and Write-DataTable, and put them in your functions module for your Windows PowerShell profile.
Next, we need to create a SQL Server table in the Database Repository. In this case, I use my R2D2 named SQL Instance, with the SQLServerRepository database and a table named tbl_SQLSErrorLog. Here is the TSQL command I use:
CREATE TABLE [dbo].[tbl_SQLErrorLog](
[CurrentDate] [datetime] NULL,
[ServerName] [varchar](20) NULL,
[InstanceName] [varchar](20) NULL,
[LogDate] [datetime] NULL,
[Message] [varchar](max) NULL,
[ProcessInfo] [varchar](50) NULL
)
Now let’s run the code without Windows PowerShell jobs.
Get-Content c:\Temp\Servers.txt | Foreach-Object {
#split the computer and Instance Name
$SplitNames = $_.split('/')
#is Default instance ?
if ($SplitNames[1] -ne $null -and $SplitNames[1] -ne 'MSSQLSERVER') {
$Query = "Select * from SqlErrorLogEvent where (InstanceName = '$($SplitNames[1])') and (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
} else {
$Query = "Select * from SqlErrorLogEvent where (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
}
#Get the computerName
$ComputerName = $SplitNames[0]
$Data = (Get-WmiObject -query $query `
-ComputerName $ComputerName `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" |
Select-Object @{Expression={(Get-Date) };Label = "CurrentDate"},`
@{Expression={($_.__Server) };Label = "ServerName"},`
InstanceName,@{Expression={([Management.ManagementDateTimeConverter]::ToDateTime($_.logdate)) };Label= "Logdate"},`
Message,
ProcessInfo )
$DataTable = Out-DataTable -InputObject $Data
Write-DataTable -ServerInstance R2D2 -Database SQLServerRepository -TableName tbl_SQLErrorLog -Data $DataTable
}
Now place the code into asynchronous mode by using background Windows PowerShell jobs. This command is shown here:
Get-Content c:\Temp\Servers.txt | % {
#split the computer and Instance Name
$SplitNames = $_.split('/')
#is Default instance ?
if ($SplitNames[1] -ne $null -and $SplitNames[1] -ne 'MSSQLSERVER') {
$Query = "Select * from SqlErrorLogEvent where (InstanceName = '$($SplitNames[1])') and (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
} else {
$Query = "Select * from SqlErrorLogEvent where (Message like '%Error%' or Message like '%Fail%' ) and (not message like '%Found 0 Errors%')"
}
#Get the computerName
$ComputerName = $SplitNames[0]
Start-job -Name "$($ComputerName)$($SplitNames[1])" -InitializationScript {Ipmo Functions -Force -DisableNameChecking} `
-ScriptBlock { $Data = (Get-WmiObject -query $args[0] `
-ComputerName $args[1] `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" |
Select-Object @{Expression={(Get-Date) };Label = "CurrentDate"},`
@{Expression={($_.__Server) };Label = "ServerName"},`
InstanceName,@{Expression={([Management.ManagementDateTimeConverter]::ToDateTime($_.logdate)) };Label= "Logdate"},`
Message,
ProcessInfo )
$DataTable = Out-DataTable -InputObject $Data
Write-DataTable -ServerInstance R2D2 -Database SQLServerRepository -TableName tbl_SQLErrorLog -Data $DataTable
} -ArgumentList $Query, $ComputerName
}
The output from the command is shown in the image that follows.
And the Oscar goes to Windows PowerShell again. Here is the CREATE TABLE command from the SQL Server Management Studio.
Can you imagine doing this in Windows PowerShell 3.0 by using workflows and the ForEach statement in parallel? I can...and I DO need a good Brazilian coffee to digest the idea.
Tip You can schedule and run it in a SQL Server Agent job, but you need to add one line to the code.
For more information, see my post about it, Dooh PowerShell Trick–Running Scripts that Has Posh Jobs on a SQL Agent Job.
Creating alerts to specific problems
I need an alert to a specific problem that is logged in the SQL Server error log. Sometimes we need an alert that is so specific and temporary that is hard for a third-party tool to have it. Then we can use the Register-WmiEvent cmdlet.
Note I will not discuss WMI, WQL, and Temporary events. If you want to study them, I recommend the eBook written by Windows PowerShell MVP, and a great friend, Ravikanth Chaganti, WMI Query Language via PowerShell. You can also investigate my articles on Simple-Talk and on my blog. You should also review An Insider’s Guide to Using WMI Events and PowerShell, which references and more than 300 pages of Hey, Scripting Guy! Blogs on the topic.
In the WMI for server events, we can configure an alert to audit a table statement by using the class Alter_Table and the WQL “Select * from ALTER_TABLE”. This is simple because the class is an event class. Unfortunately, in the Computer Management WMI, we don’t have an event class, so we need to use a generic type of the WQL event query. Let’s configure an event that is to be fired when a logon fails on a computer named Yoda from my workstation, R2D2. Remote WMI.
$query = " Select * FROM __InstanceCreationEvent WITHIN 1
WHERE TargetInstance ISA 'SqlErrorLogEvent'
and TargetInstance.Message like '%Login failed for user%'"
Register-WMIEvent -ComputerName Yoda `
-Namespace "Root\Microsoft\SqlServer\ComputerManagement11" `
-Query $query `
-SourceIdentifier "SQLErrorLOG" `
-Action { Write-Host -ForegroundColor Yellow "It WOOOORKS";$global:MyEvent = $event
}
Now, try to log on by using SSMS on Yoda and using a logon that does not exist to see if the event is triggered. But first, I need to obtain the information about the event because I added $global:MyEvent = $event. All information about the event is stored in a variable called $event. Because Register-WmiEvent starts a Windows PowerShell job, and we know that it runs in another space, and we don’t have access to it outside its space, the MyEvent global variable will store the $event.
After the event triggers, you can type $MyEvent to see all the information. For more information, see my blog, Create a Monitoring Server for SQL Server with PowerShell.
That is it, guys. I hope you enjoy it. Thanks to my friend, Ed Wilson, who kindly give me the honor to have a guest post in the major source of information for Windows PowerShell, the Hey, Scripting Guy, Blog.
~ Laerte
Thank you, Laerte, for a great blog post. I love the way you tailored your scenario, and came back around to monitoring. Great job. Thanks for sharing.
I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.
Ed Wilson, Microsoft Scripting Guy