Summary: Guest blogger and Windows PowerShell Guru Klaus Schulte talks about using Oracle ODP.NET and Windows PowerShell to simplify data access to Oracle databases.
Microsoft Scripting Guy, Ed Wilson, is here. It is early in the morning in Frankfurt, Germany, and the Scripting Wife and I just returned from Prague where we had a wonderful time visiting with one of the new Windows PowerShell MVPs David Moravec. David is a super nice person, and he took the time to show us around his lovely city. I must have taken nearly 500 pictures—won’t know until I upload them to my portable USB hard drive. Anyway, we feel really blessed to have had the opportunity to spend the day with such an interesting person.
And this brings our three-week tour to a close (in just a few hours). We have some time in the Frankfurt airport before our flights. This is a good thing because the Frankfurt airport has lots of things to do and to see. It is like a shopping mall with airplanes attached to it. Because I knew that things would be a bit confused today, I asked my good friend and PowerShell GURU Klaus Schulte to supply a guest post for today. With no further ado, here is Klaus.
How to use Windows PowerShell to query an Oracle database
By living in a networked world, with Internet access everywhere, information retrieval is often a snap. You don’t need to go to the living room and search through encyclopedias or dictionaries any more to find information—clicks on the touchscreen of your smartphone may do the job in seconds! Right, everything has changed, but behind the nice-colored icons, most of the information retrieval is still relying on some old-fashioned techniques—called databases. At the company I work for we definitely know that at least this will not change in the next couple of years.
In fact, it is crucial to get at some database information, even in scripts, to retrieve information regarding the current workflow, the status of individual tasks, and error messages indicating a flaw in the workflow.
By using Windows Powershell and ADO.NET, we are in a comfortable situation that let us query a database on the fly to get the required piece of information necessary for further processing.
Unlike Cmd.exe but like VBScript, we could use ADODB for that job, or we could rely on ADO.NET and the System.Data namespace to query databases. Even for our Oracle databases, the .NET Framework did support us by the Microsoft System.Data.OracleClient, which has been part of the .NET Framework ever since but now it has been marked as deprecated and will no longer be supported in future Framework versions starting with .NET Framework 4.
Non-Microsoft products are available, of course, and a free Oracle version ODP.NET has been around for some years, too. But we always used the Microsoft provider because it has been part of the Framework anyway and it worked fine out of the box.
But, nowadays, as most companies are migrating to 64-bit systems, another level of complexity has been added to the provider choice: You have to pay attention to the combination of installed Oracle clients and data providers on the machines your company is running because the Microsoft and the Oracle provider rely and cooperate with an installed Oracle client version on each workstation and this can be a 32-bit or a 64-bit client. You can’t easily provide an application that works in both worlds … you have to know which operating system you are running!
Some non-Microsoft products don’t need the Oracle client installed and can communicate directly with Oracle.
Note You can walk through this article using the unmanaged Oracle.DataAccess.dll without any changes (except for the name of the dll). And if you use the Microsoft System.Data.OracleClient you will also be able to follow me with very little effort! Even if you use another RDBMS, chances are that you can easily build an analogue scenario for your database.
Download and install the Oracle ODP.NET
You can download a complete package called ODAC ( Oracle Data Access Components ) from Oracle, which includes developer tools for Visual Studio, Data Provider for (ASP) .NET 4 and (ASP) .NET 2, Database Extensions for .NET 2, and Services for MTS, Provider for OLE DB, Objects for OLE, ODBC driver, SQL Plus, and the instant client.
Everything can be installed via the Oracle Universal installer, which is part of the pack, too. I won’t recommend using the ODBC driver unless you have to because it is the slowest connection method. OLE DB might deliver an acceptable performance for your applications, but if you can choose freely, the Oracle Data Provider for .NET should be your first choice.
As I mentioned above, the Oracle Instant Client is part of the ODAC and there is one thing I want to point out here: You have to install an Oracle Client for each of data access method.
Also, be aware that the standard Oracle Client may be a large piece of software! Depending on the functionality needed, you may have to dedicate over 500 MB on each workstation. The Instant Client is an improvement, which reduces the size of the client to about 100 MB as there are only a couple of binaries (DLL) delivered in this package, which can be copied to each workstation without the need of any further Setup. However, it only contains the most basic functionality where additional add-ons might be needed.
A major problem is that depending on your software mix, it might be necessary to install the 32-bit and the 64-bit versions of the client, which is a challenge on its own, if you have to make both versions available on one machine.
Another option that offers .NET access to the database is: Just use the Oracle Data Provider for .NET, which can be obtained as a separate download from Oracle called ODP.NET and is available in a 32- and 64-bit versions. This was the way I preferred to query our database up to now.
A couple of weeks ago, Oracle released the public beta of an ODP.NET version that makes life much easier now.
The brand new alpha version of the Oracle Managed DataAccess provider OPD.NET is available for download at the Oracle .Net Developer center. (You have to create an Oracle account for free, if you haven’t already got one.)
When the download is complete, just unzip the file ODP.NET_Managed_1120350_Beta.zip to a location like C:\Oracle and start using the Oracle.ManagedDataAccess.dll.
The good news is:
- ODP.NET runs on x32 and x64 Windows.
- You don’t have to install an Oracle client.
- The size of ODP.NET is less than 7 MB.
The bad news is:
- By now, there is no production version of ODP.NET available.
- The current beta has some limitations compared to unmanaged ODP.NET.
- You still need the unmanaged Oracle.ManagedDataAccessDTC.dll if you need to work with the Distributed Transaction Coordinator.
These issues should be gone with the production version that should be available next year.
We can use this DLLon Windows 7 and Windows Server 2008 or later with Microsoft .NET Framework 4 installed (the same requirements as Windows Powershell 3.0). Of course, it will be reasonable to have access to an Oracle database server (10g R2 or higher), too.
Now we want to execute some SQL Server commands against database.
Note If you don’t have an Oracle database handy, you can download the free XE version (similar to SQL Server Express) and use the example schema hr, which is installed as part of the Setup procedure of the XE database. Download the Oracle 11g R2 Database Express Edition. (You have to create an Oracle account for free, if you haven’t already got one.)
Import the Oracle.ManageDataAccess.dll
First, import this library. If you unzipped the dll to “C:\Oracle” you can start with:
Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
The next step is to create a connection to the database. Here you need to supply credentials to access the database and schema as well as a host and service name.
Unlock the hr account and supply the password hr by using this technique: Open a console window, type sqlplus / as sysdba to connect to the locally running XE database, and if you are connected, type: alter user hr identified by hr account unlock;
You should be able to connect by the connection string: "User Id=hr;Password=hr;Data Source=localhost/XE"
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("User Id=hr;Password=hr;Data Source=localhost/XE")
Next, you check if the connection can be opened!
But, first, you should examine the $con object, as shown here:
PS C:\Users\Schulte> $con
ConnectionString : User Id=hr;Password=hr;Data Source=localhost/XE
State : Closed
ModuleName :
ActionName :
ClientId :
ClientInfo :
ConnectionTimeout : 15
Database :
DatabaseName :
DatabaseDomainName :
HostName :
InstanceName :
ServiceName :
DataSource : localhost/XE
ServerVersion :
StatementCacheSize : 0
Site :
Container :
Interestingly, you can see that the properties except for the connection string are already preset: The standard connection timeout is 15 (seconds), the data source localhost/XE is taken from the connection string, a statement cache size is preset to 0, and the connection state is still closed.
Let’s open the connection now:
$con.open()
If the database and listener are running on the local machine and the service XE is available to user hr with password hr, that command should work and inspecting the $con object again will reveal some changes, as shown here:
PS C:\Users\Schulte> $con
ConnectionString : User Id=hr;Data Source=localhost/XE
State : Open
ModuleName :
ActionName :
ClientId :
ClientInfo :
ConnectionTimeout : 15
Database :
DatabaseName : XE
DatabaseDomainName :
HostName : wdo-2907
InstanceName : xe
ServiceName : XE
DataSource : localhost/XE
ServerVersion : 11.2.0.2.0
StatementCacheSize : 40
Site :
Container :
The properties DatabaseName, HostName, InstanceName, ServiceName, and ServerVersion have values now. And the StatementCacheSize has been updated to a value reported by the server, too.
It is mandatory to catch errors here as it is somewhat likely that you won’t get connected on the first try because you may have the credentials or other parts of the connection string wrong, or the listener or the database may not be running, or network problems might exist. We can never rely on a successful connect.
Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
try {
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("User Id=hr;Password=hr;Data Source=localhost/XE")
$con.open()
"Connected to database: {0} running on host: {1} - Servicename: {2} - Serverversion: {3}" -f `
$con.DatabaseName, $con.HostName, $con.ServiceName, $con.ServerVersion
}
catch
{
Write-Error ("Can't open connection: {0}`n{1}" -f `
$con.ConnectionString, $_.Exception.ToString())
}
finally
{
if ($con.State -eq 'Open') { $con.close() }
}
Let’s assume that we could successfully connect the database by using the connection string. Connecting is fine, but you usually want to do something else—like querying the database for some results! Whether the connection is open or not, you can ask the $con object to create a command object, as shown here:
$cmd=$con.CreateCommand()
Or, you can create a new cmd object directly but with more writing effort:
$cmd=new-object Oracle.ManagedDataAccess.Client.OracleCommand
Examining the $cmd object returns some interesting properties:
PS C:\Users\Schulte> $cmd
AddRowid : False
AddToStatementCache : True
ArrayBindCount : 0
BindByName : False
CommandText :
CommandTimeout : 0
CommandType : Text
Connection : Oracle.ManagedDataAccess.Client.OracleConnection
Transaction :
DesignTimeVisible : True
FetchSize : 131072
RowSize : 0
InitialLOBFetchSize : 0
InitialLONGFetchSize : 0
UpdatedRowSource : Both
Parameters : {}
Notification :
NotificationAutoEnlist : True
Site :
Container :
The CommandType is Text by default meaning that you will be able to directly supply a select command.
Note Other enumeration values are StoredProcedure and TableDirect.
CommandText will be the property where the text of the select command will be stored:
$cmd.CommandText="Select 'Hello world!' Greeting from dual"
Here we just produce a greeting message, which can run on any schema because it uses the build in table “dual”. There is just one piece missing: How do you execute the command?
Execute the command
You need a reader object to read the result set returned from the database. But it’s crucial to have the connection open before you can execute the reader!
$con.Open()
$rdr=$cmd.ExecuteReader()
Once you have the $rdr object, you can call its read method to fetch the next data row, if any data is available, and if it returns a result, we can use, for example, the GetString member function to retrieve the n-th field of the returned resultset where n is an integer starting with 0:
if ($rdr.Read()) {$rdr.GetString(0)}
If this is all you need, we have to close the connection in the end!
$con.Close()
So, all in all, here are the easy steps ready for use now:
Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("User Id=hr;Password=hr;Data Source=localhost/XE")
$cmd=$con.CreateCommand()
$cmd.CommandText="Select 'Hello world!' Greeting from dual"
$con.Open()
$rdr=$cmd.ExecuteReader()
if ($rdr.Read()) {
$rdr.GetString(0)
}
$con.Close()
Hello world!
WOW! We received an impressing sign of life from our database! Let’s try something more difficult: Add two attributes to the output of our select statement, as shown here:
$cmd.CommandText= @"
select
1 intCount,
'Number_' || to_char(1) strCount,
sysdate datCount
from dual
"@
I love to enclose the whole query string in a so-called Windows Powershell here-string, which lets me keep the original formatting of the SQL Server statement even if it spans several rows! Just exchanging the previous select statement with the new statement results in the following:
Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("User Id=hr;Password=hr;Data Source=localhost/XE")
$cmd=$con.CreateCommand()
$cmd.CommandText= @"
select
1 intCount,
'Number_' || to_char(1) strCount,
sysdate datCount
from dual
"@
$con.Open()
$rdr=$cmd.ExecuteReader()
if ($rdr.Read()) {
$rdr.GetString(0)
}
$con.Close()
Ausnahme beim Aufrufen von "GetString" mit 1 Argument(en): "Die angegebene Umwandlung ist ungültig."
In Zeile:15 Zeichen:5
+ $rdr.GetString(0)
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : InvalidCastException
My free translation for this error message is: Exception calling “GetString” with 1 argument(s): “The conversion is invalid.”
What happened here? As our database has a well-elaborated type system, it refuses to convert the intCount value 1 implicitly to a string that is the type of data required to execute the statement:
$rdr.GetString(0)
It says: GetString, not GetInteger, or something else more appropriate!
What is “more appropriate”? By using Windows Powershell 3.0, you can have a look at the member function by using the new IntelliSense feature:
This is a great help if you are exploring non-Microsoft objects, and it accelerates typing and helps avoid typos, too. J OK, you may scroll down to find more candidates, or just try functions GetByte or GetDecimal at first sight. Let’s try GetDecimal, shown here:
$rdr=$cmd.ExecuteReader()
if ($rdr.Read()) {
$rdr.GetDecimal(0)
}
$con.Close()
1
It worked! If you prefer to scroll down, you may find some more Oracle-specific GetOracle -functions in the box:
As you won’t find a GetOracleInteger function, I would definitely go for GetOracleDecimal, in this case. Let’s see what we’ll get now:
$rdr=$cmd.ExecuteReader()
if ($rdr.Read()) {
$rdr.GetOracleDecimal(0)
}
$con.Close()
IsNull : False
BinData : {2, 193, 2, 0...}
IsInt : True
IsPositive : True
IsZero : False
Format :
Value : 1
It worked somehow, but we get more information than expected and needed. The expected result is represented by the value property. If you want to know more about the object returned by GetOracleDecimal, I would suggest to store it in a variable $result and expect it later on to find out that it is of type OracleDecimal, as shown here:
PS C:\Users\Schulte> $result.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True OracleDecimal System.ValueType
And it has some properties, including the interesting value property:
PS C:\Users\Schulte> $result
IsNull : False
BinData : {2, 193, 2, 0...}
IsInt : True
IsPositive : True
IsZero : False
Format :
Value : 1
And it has a couple of member functions associated to it:
TypeName: Oracle.ManagedDataAccess.Types.OracleDecimal
Name MemberType Definition
---- ---------- ----------
CompareTo Method int CompareTo(System.Object obj), int IComparable.CompareTo(System.Object obj)
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetSchema Method System.Xml.Schema.XmlSchema IXmlSerializable.GetSchema()
GetType Method type GetType()
ReadXml Method void IXmlSerializable.ReadXml(System.Xml.XmlReader reader)
ToByte Method byte ToByte()
ToDouble Method double ToDouble()
ToInt16 Method int16 ToInt16()
ToInt32 Method int ToInt32()
ToInt64 Method long ToInt64()
ToSingle Method float ToSingle()
ToString Method string ToString()
WriteXml Method void IXmlSerializable.WriteXml(System.Xml.XmlWriter writer)
BinData Property byte[] BinData {get;}
Format Property string Format {get;set;}
IsInt Property bool IsInt {get;}
IsNull Property bool IsNull {get;}
IsPositive Property bool IsPositive {get;}
IsZero Property bool IsZero {get;}
Value Property decimal Value {get;}
Well, it is obvious that we could use the To… functions to convert the resulting decimal value to some other sub datatypes.
I won’t discuss further details here, but you may be asking yourself this: Why do we have to deal with this proprietary OracleDecimal type instead of just using the simple and straight forward Decimal type? Well, there is one reason in the world of databases that is incompatible to the type system of most other programming or script languages and always requires special treatment: Null values!
Using special Oracle data types
In most traditional programming languages special functions have been introduced to test for nullable values. That’s where it makes perfectly sense to use the special Oracle data types even if it is a bit more effort to extract the value!
That said, our new query does work for us, but hey, we only received one value where three values should be available: intCount, strCount, and datCount!
The solution is rather obvious so we can simply add two statements after the GetOracleDecimal call:
PS C:\Users\Schulte> Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("User Id=hr;Password=hr;Data Source=localhost/XE")
$cmd=$con.CreateCommand()
$cmd.CommandText= @"
select
1 intCount,
'Number_' || to_char(1) strCount,
sysdate datCount
from dual
"@
$con.Open()
$rdr=$cmd.ExecuteReader()
if ($rdr.Read()) {
$intCount=$rdr.GetOracleDecimal(0)
$strCount=$rdr.GetOracleString(1)
$datCount=$rdr.GetOracleDate(2)
}
$con.Close()
"Resultset: {0}, {1}, {2}" -f $intCount.Value, $strCount.Value, $datCount.Value
Resultset: 1, Number_1, 15.11.2012 13:31:51
Wonderful! We can query rather complex Oracle result sets now! But there’s only one row that we have retrieved from the database by now. What about receiving several rows as result of a query? Let’s change the select command to something more useful, as shown here:
$cmd.CommandText= @"
with resultset (intCount, strCount, datCount) as (
select 1 intCount, 'Number_' || to_char(1) strCount, sysdate datCount
from dual
union all (
select r.intCount+1, 'Number_' || to_char(intCount+1), sysdate+intCount+1
from resultset r
where r.intCount < 10
)
)
select * from resultset
"@
You don’t have to completely understand this recursive select statement by using a recursive with clause according to modern ANSI SQL, but it does produces the same output as the previous select if you exchange the constant 10 by 1. If you don’t, it produces 10 lines of output increasing the values by one for each row:
INTCOUNT | STRCOUNT | DATCOUNT |
1 | Number_1 | 15.11.2012 13:40 |
2 | Number_2 | 17.11.2012 13:40 |
3 | Number_3 | 18.11.2012 13:40 |
4 | Number_4 | 19.11.2012 13:40 |
5 | Number_5 | 20.11.2012 13:40 |
6 | Number_6 | 21.11.2012 13:40 |
7 | Number_7 | 22.11.2012 13:40 |
8 | Number_8 | 23.11.2012 13:40 |
9 | Number_9 | 24.11.2012 13:40 |
10 | Number_10 | 25.11.2012 13:40 |
The easiest way to reproduce this kind of output with our little Windows Powershell script is: Output the content of the three count variables inside the $rdr.read -script block, and exchange the if statement with a while statement, as shown here:
PS C:\Users\Schulte> Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("User Id=hr;Password=hr;Data Source=localhost/XE")
$cmd=$con.CreateCommand()
$cmd.CommandText= @"
with resultset (intCount, strCount, datCount) as (
select 1 intCount, 'Number_' || to_char(1) strCount, sysdate datCount
from dual
union all (
select r.intCount+1, 'Number_' || to_char(intCount+1), sysdate+intCount+1
from resultset r
where r.intCount < 10
)
)
select * from resultset
"@
$con.Open()
$rdr=$cmd.ExecuteReader()
while ($rdr.Read()) {
$intCount=$rdr.GetOracleDecimal(0)
$strCount=$rdr.GetOracleString(1)
$datCount=$rdr.GetOracleDate(2)
"{0,3}, {1,10}, {2}" -f $intCount.Value, $strCount.Value, $datCount.Value
}
$con.Close()
1, Number_1, 15.11.2012 13:49:08
2, Number_2, 17.11.2012 13:49:08
3, Number_3, 18.11.2012 13:49:08
4, Number_4, 19.11.2012 13:49:08
5, Number_5, 20.11.2012 13:49:08
6, Number_6, 21.11.2012 13:49:08
7, Number_7, 22.11.2012 13:49:08
8, Number_8, 23.11.2012 13:49:08
9, Number_9, 24.11.2012 13:49:08
10, Number_10, 25.11.2012 13:49:08
That was a snap!
We are just missing the header line:
INTCOUNT | STRCOUNT | DATCOUNT |
Of course, we could add this in a one liner by using a constant string, which is not the way I would like to do that! I would prefer to get this column header information from the database and in fact the data reader $rdr can query this information with a call to the GetSchemaTable() method, as shown here:
PS C:\Users\Schulte> $rdr.GetSchemaTable()
ColumnName : INTCOUNT
ColumnOrdinal : 0
ColumnSize : 22
NumericPrecision : 38
NumericScale : 127
IsUnique :
IsKey :
IsRowID : False
BaseColumnName :
BaseSchemaName :
BaseTableName :
DataType : System.Decimal
ProviderType : 107
AllowDBNull : True
IsAliased : True
IsByteSemantic :
IsExpression : False
IsHidden : False
IsReadOnly : True
IsLong : False
ColumnName : STRCOUNT
ColumnOrdinal : 1
ColumnSize : 47
… output abbreviated…
Along with a whole bunch of other valuable information about our data columns, we can access the ColumnName property:
PS C:\Users\Schulte> $rdr.GetSchemaTable() | select ColumnName
ColumnName
----------
INTCOUNT
STRCOUNT
DATCOUNT
If we can retrieve the column names, we may come up with a nice header line after some formatting:
"{0,10}, {1,10}, {2,8}" -f ($rdr.GetSchemaTable() | select ColumnName -ExpandProperty ColumnName)
$con.Open()
$rdr=$cmd.ExecuteReader()
"{0,10}, {1,10}, {2,8}" -f ($rdr.GetSchemaTable() | select ColumnName -ExpandProperty ColumnName)
while ($rdr.Read()) {
$intCount=$rdr.GetOracleDecimal(0)
$strCount=$rdr.GetOracleString(1)
$datCount=$rdr.GetOracleDate(2)
"{0,10}, {1,10}, {2}" -f $intCount.Value, $strCount.Value, $datCount.Value
}
$con.Close()
INTCOUNT, STRCOUNT, DATCOUNT
1, Number_1, 15.11.2012 14:09:32
2, Number_2, 17.11.2012 14:09:32
3, Number_3, 18.11.2012 14:09:32
4, Number_4, 19.11.2012 14:09:32
5, Number_5, 20.11.2012 14:09:32
6, Number_6, 21.11.2012 14:09:32
7, Number_7, 22.11.2012 14:09:32
8, Number_8, 23.11.2012 14:09:32
9, Number_9, 24.11.2012 14:09:32
10, Number_10, 25.11.2012 14:09:32
This is quite OK, but it’s not the way we want to do this in Windows Powershell. We would prefer to have objects here and let Windows Powershell do all the formatting for us. Let’s change the read block again:
$con.Open()
$rdr=$cmd.ExecuteReader()
$columnNames=$rdr.GetSchemaTable() | Select-Object -ExpandProperty ColumnName
$resultSet=@()
while ($rdr.Read()) {
$result=New-Object object
$result | Add-Member -NotePropertyName $columnNames[0] -NotePropertyValue $rdr.GetOracleDecimal(0)
$result | Add-Member -NotePropertyName $columnNames[1] -NotePropertyValue $rdr.GetOracleString(1)
$result | Add-Member -NotePropertyName $columnNames[2] -NotePropertyValue $rdr.GetOracleDate(2)
$resultSet += $result
}
$con.Close()
$resultSet | Format-Table -AutoSize
Executing this will result in a nicely formatted object output:
PS C:\Users\Schulte> D:\Script\Query_Oracle5.ps1
INTCOUNT STRCOUNT DATCOUNT
-------- -------- --------
1 Number_1 11/15/2012 14:31:08
2 Number_2 11/17/2012 14:31:08
3 Number_3 11/18/2012 14:31:08
4 Number_4 11/19/2012 14:31:08
5 Number_5 11/20/2012 14:31:08
6 Number_6 11/21/2012 14:31:08
7 Number_7 11/22/2012 14:31:08
8 Number_8 11/23/2012 14:31:08
9 Number_9 11/24/2012 14:31:08
10 Number_10 11/25/2012 14:31:08
Best of all: You can even use Out –GridView instead of Format-Table –AutoSize, if you want to display the data even more nicely. Additionally, the filtering and sorting capabilities of a data grid view are now for free.
To get this result in a much easier way (less to write), use can also use the following script:
Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("User Id=hr;Password=hr;Data Source=localhost/XE")
$cmd=$con.CreateCommand()
$cmd.CommandText= @"
with resultset (intCount, strCount, datCount) as (
select 1 intCount, 'Number_' || to_char(1) strCount, sysdate datCount
from dual
union all (
select r.intCount+1, 'Number_' || to_char(intCount+1), sysdate+intCount+1
from resultset r
where r.intCount < 10
)
)
select * from resultset
"@
$da=New-Object Oracle.ManagedDataAccess.Client.OracleDataAdapter($cmd);
$resultSet=New-Object System.Data.DataTable
[void]$da.fill($resultSet)
$resultSet | Format-Table -AutoSize
The result is identical, except the formatting of the INTCOUNT column:
INTCOUNT STRCOUNT DATCOUNT
-------- -------- --------
1 Number_1 15.11.2012 14:41:32
2 Number_2 17.11.2012 14:41:32
3 Number_3 18.11.2012 14:41:32
4 Number_4 19.11.2012 14:41:32
5 Number_5 20.11.2012 14:41:32
6 Number_6 21.11.2012 14:41:32
7 Number_7 22.11.2012 14:41:32
8 Number_8 23.11.2012 14:41:32
9 Number_9 24.11.2012 14:41:32
10 Number_10 25.11.2012 14:41:32
I did exchange the data reader with a data adapter, which has a Fill method that can automatically populate a dataset or data table with the results of the select statement. This is at least a timesaver writing the script code. A rather interesting question came to mind now: Is it a timesaver regarding performance, too? Well, we probably should do some testing now. But measuring the execution time of the select command is most likely faulty if we just execute one select. Maybe 10 selects would provide a better measurement basis and retrieving more than 10 rows, maybe 100, might be better, too. To do so, we are getting a bit more Windows Powershell-stylish now and encapsulate the two scripts in functions. We omit the Format-Table statement in the end and return the whole $resultSet as result of the new functions. But we should definitely consider some more changes.
The new versions of the script might look even better if we parameterize the select statement and probably the connection string, too!
Before we do this, I have to admit that even then, we have still some things to consider that would be nice to have: Querying a database using a fixed select statement might be more or less “sub optimal”. Additionally, the use of a constant connection string is not the best solution. There are some further topics that could have been addressed like using parameters in select statements, executing queries that don’t return result sets or issuing other statements that don’t select data at all. Nevertheless, we’ll concentrate on the connection string and the select command here, and ignore other potential improvements.
I added rudimentary error checking and a very simple checking on both input parameters, too. We just set the whole function inside a try-catch-finally block to catch and report any errors that are likely to happen during database operations.
The “ValidateScript” instruction is used to check that the connection string includes at least the “data source” property and that the select string contains the word “select.” Additionally, I made both parameters mandatory without supplying defaults, which wouldn’t make too much sense in my opinion.
Finally, we have the following script featuring two functions Get-OracleResultRdr and Get-OracleResultDa:
Add-Type -Path “C:\Oracle\Oracle.ManagedDataAccess.dll"
function Get-OracleResultRdr
{
param (
[Parameter(Mandatory=$true)]
[ValidateScript({$_ -match '\bdata source\b'})]
[string]$conString,
[ValidateScript({$_ -match '\bselect\b'})]
[Parameter(Mandatory=$true)]
[string]$sqlString
)
$resultSet=@()
try {
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($conString)
$cmd=$con.CreateCommand()
$cmd.CommandText= $sqlString
$con.Open()
$rdr=$cmd.ExecuteReader()
$columnNames=$rdr.GetSchemaTable() | Select-Object -ExpandProperty ColumnName
while ($rdr.Read()) {
$result = New-Object PsObject
$result | Add-Member -NotePropertyName $columnNames[0] -NotePropertyValue $rdr.GetOracleDecimal(0)
$result | Add-Member -NotePropertyName $columnNames[1] -NotePropertyValue $rdr.GetOracleString(1)
$result | Add-Member -NotePropertyName $columnNames[2] -NotePropertyValue $rdr.GetOracleDate(2)
$resultSet += $result
}
} catch {
Write-Error ($_.Exception.ToString())
} finally {
if ($con.State -eq 'Open') { $con.close() }
}
$resultSet
}
function Get-OracleResultDa
{
param (
[Parameter(Mandatory=$true)]
[ValidateScript({$_ -match '\bdata source\b'})]
[string]$conString,
[ValidateScript({$_ -match '\bselect\b'})]
[Parameter(Mandatory=$true)]
[string]$sqlString
)
$resultSet=@()
try {
$con = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($conString)
$cmd=$con.CreateCommand()
$cmd.CommandText= $sqlString
$da=New-Object Oracle.ManagedDataAccess.Client.OracleDataAdapter($cmd);
$resultSet=New-Object System.Data.DataTable
[void]$da.fill($resultSet)
} catch {
Write-Error ($_.Exception.ToString())
} finally {
if ($con.State -eq 'Open') { $con.close() }
}
$resultSet
}
# Test data for functions
$conString="User Id=hr;Password=hr;Data Source=localhost/XE"
$sqlString=@"
with resultset (intCount, strCount, datCount) as (
select 1 intCount, 'Number_' || to_char(1) strCount, sysdate datCount
from dual
union all (
select r.intCount+1, 'Number_' || to_char(intCount+1), sysdate+intCount+1
from resultset r
where r.intCount < 1000
)
)
select * from resultset
"@
Get-OracleResultRdr $conString $sqlString
Get-OracleResultDa $conString $sqlString
I mentioned above that it might be necessary to do some repetitions if you want to measure the execution time of each command to get a feeling for the performance of both procedures, but, in fact, it is rather obvious that the first command lasts longer than the second does. However, measure the time for 10 loops of each command now. Instead of the last two lines, do the following:
$timeRdr = Measure-Command {1..10 | %{Get-OracleResultRdr $conString $sqlString}}
$timeDa = Measure-Command {1..10 | %{Get-OracleResultDa $conString $sqlString}}
"Time used by the DataReader : " + $timeRdr.TotalSeconds/10
"Time used by the dataAdapter: " + $timeDa.TotalSeconds /10
The results are:
PS C:\Users\Schulte> D:\Script\Query_Oracle8.ps1
Time used by the DataReader : 1.46688318
Time used by the dataAdapter: 0.0084921
Unbelievable! And definitely wrong, as I can tell you from my experience! 8.4921[ms] is not a realistic time.
But I can explain to you that each database relies on heavy, well elaborated, and highly tuned caching algorithms that prevent a reasonable timing if you loop through the same statement. The statement is preparsed and cached, the previously calculated execution plan is used again and if result set caching is available, the execution may be skipped at all, and the old result set will just be returned to the client.
So, we will have to clear the buffer cache and shared pool before we can execute each statement once, if we really would like to have reasonable timing data for at least one execution of each statement. I’ll open a sqlplus command window and execute the two commands now before running each of the functions:
SQL> alter system flush buffer_cache;
System altered.
SQL> alter system flush shared_pool;
System altered.
PS C:\Users\Schulte> Measure-Command {Get-OracleResultRdr $conString $sqlString}
Days : 0
Hours : 0
Minutes : 0
Seconds : 2
Milliseconds : 269
Ticks : 22693179
TotalDays : 2,62652534722222E-05
TotalHours : 0,000630366083333333
TotalMinutes : 0,037821965
TotalSeconds : 2,2693179
TotalMilliseconds : 2269,3179
SQL> alter system flush buffer_cache;
System altered.
SQL> alter system flush shared_pool;
System altered.
PS C:\Users\Schulte> Measure-Command {Get-OracleResultDa $conString $sqlString}
Days : 0
Hours : 0
Minutes : 0
Seconds : 0
Milliseconds : 292
Ticks : 2922746
TotalDays : 3,38280787037037E-06
TotalHours : 8,11873888888889E-05
TotalMinutes : 0,00487124333333333
TotalSeconds : 0,2922746
TotalMilliseconds : 292,2746
These data are more realistic! But still the data adapter is ready in 0.3 seconds whereas the reader operation lasts 2.27 seconds. Is it real that the data adapter received the results 7 to 8 times faster than the reader? Very unlikely, I would say.
So, let’s try to tune the reader, or at least find out if it is really that slow. I will flush the database cache each time before I test the statement. We can definitely expect that most of the time if used inside the loop:
while ($rdr.Read()) {
$result = New-Object PsObject
$result | Add-Member -NotePropertyName $columnNames[0] -NotePropertyValue $rdr.GetOracleDecimal(0)
$result | Add-Member -NotePropertyName $columnNames[1] -NotePropertyValue $rdr.GetOracleString(1)
$result | Add-Member -NotePropertyName $columnNames[2] -NotePropertyValue $rdr.GetOracleDate(2)
$resultSet += $result
}
So, let’s read the data without processing them at all.
while ($rdr.Read()) {}
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,2412746
That’s far better now. Let’s see what happens if we include the GetOracle<type> methods inside the loop. It may be that they are slowing the operation down:
while ($rdr.Read()) {
$intCount = $rdr.GetOracleDecimal(0)
$strCount = $rdr.GetOracleString(1)
$datCount = $rdr.GetOracleDate(2)
}
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,2379125
Well, this is not too bad at all! It seems to be even faster than the preceding measurement, which is, of course, hardly possible. It is more or less a result of inaccurate timing for such fast operations. But, in general, it indicated that the behavior didn’t change significantly.
Even if we consume the data, we are still pretty fast getting at the results! So what else could be the reason why?
Object creation
Let’s add the object creation to the scenario:
while ($rdr.Read()) {
$result = New-Object PsObject
$intCount = $rdr.GetOracleDecimal(0)
$strCount = $rdr.GetOracleString(1)
$datCount = $rdr.GetOracleDate(2)
}
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,2718845
No remarkable changes, too! Let’s add up the newly created (empty) object to form the result set:
while ($rdr.Read()) {
$result = New-Object PsObject
$intCount = $rdr.GetOracleDecimal(0)
$strCount = $rdr.GetOracleString(1)
$datCount = $rdr.GetOracleDate(2)
$resultSet += $result
}
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,407326
No significant change, but the objects were empty! Falling back to the previous scenario: Let’s add properties to the $result object and don’t add them up in $resultset:
while ($rdr.Read()) {
$result = New-Object PsObject
$result | Add-Member -NotePropertyName $columnNames[0] -NotePropertyValue $rdr.GetOracleDecimal(0)
$result | Add-Member -NotePropertyName $columnNames[1] -NotePropertyValue $rdr.GetOracleString(1)
$result | Add-Member -NotePropertyName $columnNames[2] -NotePropertyValue $rdr.GetOracleDate(2)
}
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
1,9345301
That’s it! At least partially. We found the slow operation: Adding members to the PsCustomObject seems to be very time consuming.
Comparing it to the original 2.27[s] we can imagine that the time to add up the $results to the $resultset adds another 0.3 seconds penalty to the execution time and you might have already observed it: We can get rid of this extra time.
In fact, it is not necessary to build the $resultset ourselves, we can emit each $result inside the loop and let Windows Powershell organize the stream of results! This would be an improvement that Windows Powershell offers for free! It is less coding and less failure and as we have seen here less execution time! So, the next thing we will do is: Eliminate the variable $resultset und return $result in each iteration:
while ($rdr.Read()) {
$result = New-Object PsObject
$result | Add-Member -NotePropertyName $columnNames[0] -NotePropertyValue $rdr.GetOracleDecimal(0)
$result | Add-Member -NotePropertyName $columnNames[1] -NotePropertyValue $rdr.GetOracleString(1)
$result | Add-Member -NotePropertyName $columnNames[2] -NotePropertyValue $rdr.GetOracleDate(2)
$result
}
Back to the main insight now: Adding members to our object is slowing the function down! The final question is: Do we have alternative ways to build object and are they faster? Well, we can build objects like this in Windows Powershell:
while ($rdr.Read()) {
$result = "" | Select-Object $columnNames
$result.$($columnNames[0]) = $rdr.GetOracleDecimal(0)
$result.$($columnNames[1]) = $rdr.GetOracleString(1)
$result.$($columnNames[2]) = $rdr.GetOracleDate(2)
$result
}
Surprisingly, this makes a big difference:
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,3962451
With Windows Powershell 3.0, you can also do it that way:
while ($rdr.Read()) {
$result = New-Object psobject -Property @{
$columnNames[0] = $rdr.GetOracleDecimal(0);
$columnNames[1] = $rdr.GetOracleString(1);
$columnNames[2] = $rdr.GetOracleDate(2);
}
$result
}
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,7647542
This seems to make a difference, too. It is still faster than Add-Member, but slower than the second solution … at least in this case. But wait! There is still another new solution available in Windows Powershell 3.0. We have the new [pscustomobject] type accelerator available now:
while ($rdr.Read()) {
$result = [pscustomobject] @{
$columnNames[0] = $rdr.GetOracleDecimal(0);
$columnNames[1] = $rdr.GetOracleString(1);
$columnNames[2] = $rdr.GetOracleDate(2);
}
$result
}
PS C:\Users\Schulte> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,373167
WOW! The fastest solution so far—it may be a tie between this and the second solution because we can’t know exactly if the timing is good enough.
Note I could also eliminate the assignment to $result and return the object directly in the last two cases, which won’t change the execution much. I just left it there for a better comparison to the first two solutions where we have to use the $result variable.
The last thing I want to do now is to generalize the solution a little further! We still have used a special query up to now that returns three values in each row with fixed data types: OracleDecimal, OracleString, and OracleDate.
This is very special and the question arises if we can modify the solution further to accept other types of data and more or less than three columns per row. Of course, we can but as a constructor of a [pscustomobject] with variable initial values is not available, can we still profit the fastest solution or will we have to go back to the Add-Member solution, which is very slow?
Another problem is the usage of highly specialized type dependent GetOracle<type> functions that require us to know the type of object that is returned from the database. It would be possible to use these functions if we evaluate the data returned from the call to $rdr.GetSchemaTable(), where this information is part of the row description:
[DBG]: PS C:\Users\Schulte>> $rdr.GetSchemaTable().rows[0].Datatype.name
Decimal
[DBG]: PS C:\Users\Schulte>> $rdr.GetSchemaTable().rows[1].Datatype.name
String
[DBG]: PS C:\Users\Schulte>> $rdr.GetSchemaTable().rows[2].Datatype.name
DateTime
But even if we had the type information, we would further have to use this information in a switch statement to retrieve the function call that is appropriate for the current field type. This is not fun! But wait, there is an easier way out! We can use the type neutral function:
[DBG]: PS C:\Users\Schulte>> $rdr.GetOracleValue
OverloadDefinitions
-------------------
System.Object GetOracleValue(int i)
This function doesn’t depend on the type and returns an object.
Putting it all together again, we can build a hashtable as our $result object that is composed of the column names and the corresponding values that we will retrieve in looping through the fields returned in the result set. Nicely enough, the field count is a property of the data reader:
while ($rdr.Read()) {
$result=@{}
0..($rdr.FieldCount-1) | %{
$result.Add($columnNames[$_], $rdr.GetOracleValue($_))
}
[pscustomobject]$result
}
Returning a pscustomobject based on the hashtable $result works. This way we can use the fast constructor but have a variable initialization. Sounds fine, but hey, the result isn’t that cute. In fact, we are back to where we started from: We have a time of over 2 seconds again.
PS C:\Users\Schulte> Measure-Command {Get-OracleResultRdr $conString $sqlString}
TotalSeconds : 2,1450785
A little additional overhead would be OK, if we can generalize queries. But is it really true that we are back to where we started from? I really thought so at first but investigating things further I discovered that the loop construct followed by the pipe is quite slow.
Exchanging it with o for-loop returns better results:
while ($rdr.Read()) {
$result=@{}
for ($i=0; $i -lt $rdr.FieldCount; $i++) {
# 0..($rdr.FieldCount-1) | %{
$result.Add($columnNames[$i], $rdr.GetOracleValue($i))
}
[pscustomobject]$result
}
PS C:\Users\Schulte> Measure-Command {Get-OracleResultRdr $conString $sqlString}
TotalSeconds : 0,5363667
This is quite acceptable for a generalized solution! Again, it wasn’t a fault of the database that returned the results too slow, it was a problem with my script that slowed things down. Something to remember!
If you are not convinced that it really does what it is supposed to do, we can supply some alternative queries just to present the results. Here is result of a query that returns the Fibonacci numbers and the depth level of the recursion (or just a counter if you prefer that ).
The Fibonacci numbers are defined by Fib(0) = 0; fib(1) = 1; fib(n) = fib(n-1)+fib(n-2) for n > 1
In easy words: Start with two numbers 0 and 1. Calculate the next number by adding the last two numbers: 0, 1, (0+1)=1, (1+1)=2, (1+2)=3, (2+3)=5, (3+5)=8, …
We could do this very easily in Windows Powershell with a recursive function:
function fibonacci($n)
{
if ($n -eq 0) {0}
elseif ($n -eq 1) {1}
else {$(fibonacci($n-1))+$(fibonacci($n-2))}
}
# produce some test values
0..20 | %{ fibonacci $_}
The results are:
0
1
1
2
3
5
8
13
21
34
55
89
144
233
377
610
987
1597
2584
4181
You may notice that calculating the last values takes a noticeable amount of time. Let’s do a quick check to unveil the number of recursive calls used to calculate the results and do an additional call to the Fibonacci function to see how much time we did spent to get at the results:
function fibonacci($n)
{
$global:calls++
if ($n -eq 0) {0}
elseif ($n -eq 1) {1}
else {$(fibonacci($n-1))+$(fibonacci($n-2))}
}
# produce some test values
0..19 | %{ $calls=0; "Fibonacci ({0,2}) = {1,5} --- {2,5} recursive calls in {3,-9} seconds" `
-f $_, (fibonacci $_), $calls, (Measure-Command {fibonacci $_}).TotalSeconds}
Fibonacci ( 0) = 0 --- 1 recursive calls in 0,0001971 seconds
Fibonacci ( 1) = 1 --- 1 recursive calls in 7,33E-05 seconds
Fibonacci ( 2) = 1 --- 3 recursive calls in 0,0001956 seconds
Fibonacci ( 3) = 2 --- 5 recursive calls in 0,0003013 seconds
Fibonacci ( 4) = 3 --- 9 recursive calls in 0,000552 seconds
Fibonacci ( 5) = 5 --- 15 recursive calls in 0,0009108 seconds
Fibonacci ( 6) = 8 --- 25 recursive calls in 0,0015319 seconds
Fibonacci ( 7) = 13 --- 41 recursive calls in 0,0021847 seconds
Fibonacci ( 8) = 21 --- 67 recursive calls in 0,0031991 seconds
Fibonacci ( 9) = 34 --- 109 recursive calls in 0,0053804 seconds
Fibonacci (10) = 55 --- 177 recursive calls in 0,0096687 seconds
Fibonacci (11) = 89 --- 287 recursive calls in 0,0143352 seconds
Fibonacci (12) = 144 --- 465 recursive calls in 0,0238466 seconds
Fibonacci (13) = 233 --- 753 recursive calls in 0,0401255 seconds
Fibonacci (14) = 377 --- 1219 recursive calls in 0,0643954 seconds
Fibonacci (15) = 610 --- 1973 recursive calls in 0,1061213 seconds
Fibonacci (16) = 987 --- 3193 recursive calls in 0,1682571 seconds
Fibonacci (17) = 1597 --- 5167 recursive calls in 0,2774713 seconds
Fibonacci (18) = 2584 --- 8361 recursive calls in 0,4582997 seconds
Fibonacci (19) = 4181 --- 13529 recursive calls in 0,7425936 seconds
Well, nearly 20 thousand recursions per second isn’t too bad at all. Just one last remark. There is also an iterative solution available that is by far faster, of course, as shown here:
function fibonacci($n)
{
for ($i=0; $i -le $n; $i++){
$global:calls++
if ($i -eq 0) {$n1=$n2=0}
elseif ($i -eq 1) {$n1=1}
else {$n1, $n2 = ($n1+$n2), $n1}
}
$n1
}
# produce some test values
0..19 | %{ $calls=0; "Fibonacci ({0,2}) = {1,5} --- {2,5} recursive calls in {3,-9} seconds" `
-f $_, (fibonacci $_), $calls, (Measure-Command {fibonacci $_}).TotalSeconds}
Fibonacci ( 0) = 0 --- 1 recursive calls in 0,0002074 seconds
Fibonacci ( 1) = 1 --- 2 recursive calls in 6,65E-05 seconds
Fibonacci ( 2) = 1 --- 3 recursive calls in 8,11E-05 seconds
Fibonacci ( 3) = 2 --- 4 recursive calls in 7,55E-05 seconds
Fibonacci ( 4) = 3 --- 5 recursive calls in 8,21E-05 seconds
Fibonacci ( 5) = 5 --- 6 recursive calls in 8,64E-05 seconds
Fibonacci ( 6) = 8 --- 7 recursive calls in 9,23E-05 seconds
Fibonacci ( 7) = 13 --- 8 recursive calls in 9,95E-05 seconds
Fibonacci ( 8) = 21 --- 9 recursive calls in 0,0001038 seconds
Fibonacci ( 9) = 34 --- 10 recursive calls in 0,0001094 seconds
Fibonacci (10) = 55 --- 11 recursive calls in 0,0001156 seconds
Fibonacci (11) = 89 --- 12 recursive calls in 0,0001215 seconds
Fibonacci (12) = 144 --- 13 recursive calls in 0,0001303 seconds
Fibonacci (13) = 233 --- 14 recursive calls in 0,0001365 seconds
Fibonacci (14) = 377 --- 15 recursive calls in 0,0001377 seconds
Fibonacci (15) = 610 --- 16 recursive calls in 0,0001439 seconds
Fibonacci (16) = 987 --- 17 recursive calls in 0,0001498 seconds
Fibonacci (17) = 1597 --- 18 recursive calls in 0,0001554 seconds
Fibonacci (18) = 2584 --- 19 recursive calls in 0,0001433 seconds
Fibonacci (19) = 4181 --- 20 recursive calls in 9,92E-05 seconds
A recursive SQL Server statement looks like this:
with fibonacci (n1, n2, lvl) as (
select 0 n1, 0 n2, 1 lvl
from dual
union all (
select case n1 when 0 then 1 else n1+n2 end, n1, lvl+1
from fibonacci f
where lvl < 100
)
)
select n1, lvl from Fibonacci
And I can tell you that this statement runs in nearly no time, too! I don’t know how the database evaluates this recursively defined statement but it is really fast! So we’ll test our function Get-OracleResultDa using the fibonacci statement:
$conString="User Id=hr;Password=hr;Data Source=localhost/xe"
$sqlString=@"
with fibonacci (n1, n2, lvl) as (
select 0 n1, 0 n2, 1 lvl
from dual
union all (
select case n1 when 0 then 1 else n1+n2 end, n1, lvl+1
from fibonacci f
where lvl < 20
)
)
select n1, lvl from fibonacci
"@
$timeRdr = Measure-Command {$dr=Get-OracleResultRdr $conString $sqlString}
$timeDa = Measure-Command {$da=Get-OracleResultDa $conString $sqlString}
PS C:\Users\Schulte> D:\Script\Query_Oracle14.ps1
Time used by the DataReader : 0.0026543
Time used by the dataAdapter: 0.0016834
And piping it to Out-GridView displays the correct values:
We have two columns of type OracleDecimal, which we can verify by assigning the result to a variable $dr and examining the type of the value:
PS C:\Users\Schulte> $dr=Get-OracleResultRdr $conString $sqlString
PS C:\Users\Schulte> $dr[0].lvl.gettype()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True OracleDecimal System.ValueType
PS C:\Users\Schulte> $dr[1].lvl.gettype()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True OracleDecimal System.ValueType
Another example can easily be obtained by querying the view “sys.user_tables,” which is part of the Oracle data dictionary. It is usually available to each user. Let’s query some of the over 50 columns:
$sqlString=@"
select table_name, num_rows, blocks, avg_row_len, last_analyzed, compression
from user_tables
"@
PS C:\Users\Schulte> D:\Script\Query_Oracle15.ps1
Time used by the DataReader : 0.0223927
Time used by the dataAdapter: 0.0129836
The results are looking good but maybe you expected that the order of the displayed columns should be different according to the select statement. I didn’t mention it before, but, in fact, the columns of the previous example have been swapped too. If you need to keep the order, you can pipe the result to Select-Object and enumerate the fields in the right order:
Get-OracleResultRdr $conString $sqlString | Select-Object table_name, num_rows, blocks, avg_row_len, last_analyzed, compression | Out-GridView
This will rearrange the order of columns in the GridView:
But wait! Why exactly could the order of the columns not be preserved? The loop that returns the columns read by the data reader did return them in the correct order because we did define the order by the parameter $i in the call to $rdr.GetOracleValue($i), so it has to be right!
But there is one flaw in the script code: Adding elements to the hashtable $result doesn’t preserve the order in which they were added.
And here is something new in Windows Powershell 3.0: We can get around this problem by defining an ordered hashtable:
while ($rdr.Read()) {
$result=[ordered]@{}
Adding the [ordered] tag as part of the creation of the hashtable does do the job. So reordering the result by Select-Object is no longer needed.
Examining the types returned reveals that we have the following results:
PS C:\Users\Schulte> $dr[0] | Get-Member -MemberType NoteProperty | Select-Object definition
Definition
----------
Oracle.ManagedDataAccess.Types.OracleDecimal AVG_ROW_LEN=14
Oracle.ManagedDataAccess.Types.OracleDecimal BLOCKS=5
Oracle.ManagedDataAccess.Types.OracleString COMPRESSION=DISABLED
Oracle.ManagedDataAccess.Types.OracleDate LAST_ANALYZED=08/27/2011 08:55:25
Oracle.ManagedDataAccess.Types.OracleDecimal NUM_ROWS=4
Oracle.ManagedDataAccess.Types.OracleString TABLE_NAME=REGIONS
So, we have retrieved three OracleDecimals, two OracleStrings, and one OracleDate here, which is quite close to what you might have guessed. We haven’t done that before, but now it is time to compare the result types returned by the data adapter version and the final data reader version, too:
$da=Get-OracleResultDa $conString $sqlString
Both $da and $dr are arrays of objects, but $dr[0] is a PsCustomObject, whereas $da[0] is a DataRow object.
PS C:\Users\Schulte> $dr[0].GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False PSCustomObject System.Object
PS C:\Users\Schulte> $da[0].GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False DataRow System.Object
This is not too surprising, but there is an additional difference: Our columns that have been “noteproperties” using the data reader are “properties” if we use the datadapter. And even the types are not identical, though similar:
PS C:\Users\Schulte> $da[0] | Get-Member -MemberType Property | Select-Object -ExpandProperty definition
decimal AVG_ROW_LEN {get;set;}
decimal BLOCKS {get;set;}
string COMPRESSION {get;set;}
datetime LAST_ANALYZED {get;set;}
decimal NUM_ROWS {get;set;}
string TABLE_NAME {get;set;}
If we look at the script code …
$da=New-Object Oracle.ManagedDataAccess.Client.OracleDataAdapter($cmd);
$resultSet=New-Object System.Data.DataTable
[void]$da.fill($resultSet)
… and think about the DataTable used to retrieve the results from the database, it might become quite clear that we are not dealing with an Oracle specific object. System.Data.dataTable is a .NET object that doesn’t know of OracleDecimals, OracleStrings, or an OracleDate. It just returns the corresponding .NET types.
Back to our original question: Is the data adapter faster than the data reader? Yes, it looks like that. And that is an observation that is even opposite to some articles I have read before.
We had 0,292 seconds for the data adapter, and now we still have 0,373 seconds for the data reader, which is small, but in recurring queries and maybe if larger result sets have to returned, significant difference!
If you think: Why should I bother? I will always use the data adapter that returns a ready to use dataset or a data table with less coding in a smaller amount of time? First of all: I am not sure if it may not be possible to tune the data reader further with special database parameters like the fetchsize to make it even faster.
Secondly, and more importantly, if you have to return a large amount of rows things may change—especially if you can’t keep the dataset in memory anymore. The data reader can be used to serially progress each data row, the data adapter has to fetch all the data before you can progress any row.
One other thing to remember is that you have a permanent connection to the database while you use the data reader to fetch each row. You have to manually open and close the connection before you start reading the data and after you finished to do so.
The data adapter does this behind the scenes in its fill-method for you. It only connects to the database to populate the data set or table used to keep the results of the query.
In fact I did check with the still available, though deprecated, version of the Microsoft implementation of the Oracle client. If you still want to use it, you may have to load the assembly if you are using the .NET 4 client profile.
Add-type -AssemblyName System.Data.OracleClient
Other necessary changes include replacing Oracle.ManagedDataAccess.Client with System.Data.OracleClient and using a different connection string. Instead of User Id=hr;Password=hr;Data Source=localhost/xe, we have to build a slightly different connection string: $conString="Uid=hr;Pwd=hr;Data Source=localhost/xe"
The whole script looks like that now:
Add-type -AssemblyName System.Data.OracleClient
function Get-OracleResultRdr
{
param (
[Parameter(Mandatory=$true)]
[ValidateScript({$_ -match '\bdata source\b'})]
[string]$conString,
[ValidateScript({$_ -match '\bselect\b'})]
[Parameter(Mandatory=$true)]
[string]$sqlString
)
$resultSet=@()
try {
$con = New-Object System.Data.OracleClient.OracleConnection($conString)
$cmd=$con.CreateCommand()
$cmd.CommandText= $sqlString
$con.Open()
$rdr=$cmd.ExecuteReader()
$columnNames=$rdr.GetSchemaTable() | Select-Object -ExpandProperty ColumnName
while ($rdr.Read()) {
$result=[ordered]@{}
for ($i=0; $i -lt $rdr.FieldCount; $i++) {
# 0..($rdr.FieldCount-1) | %{
$result.Add($columnNames[$i], $rdr.GetOracleValue($i))
}
[pscustomobject]$result
}
} catch {
Write-Error ($_.Exception.ToString())
} finally {
if ($con.State -eq 'Open') { $con.close() }
}
}
function Get-OracleResultDa
{
param (
[Parameter(Mandatory=$true)]
[ValidateScript({$_ -match '\bData Source\b'})]
[string]$conString,
[ValidateScript({$_ -match '\bselect\b'})]
[Parameter(Mandatory=$true)]
[string]$sqlString
)
$resultSet=@()
try {
$con = New-Object System.Data.OracleClient.OracleConnection($conString)
$cmd=$con.CreateCommand()
$cmd.CommandText= $sqlString
$da=New-Object System.Data.OracleClient.OracleDataAdapter($cmd);
$resultSet=New-Object System.Data.DataTable
[void]$da.fill($resultSet)
} catch {
Write-Error ($_.Exception.ToString())
} finally {
if ($con.State -eq 'Open') { $con.close() }
}
$resultSet
}
# Test data for functions
$conString="Uid=hr;Pwd=hr;Data Source=localhost/XE"
$sqlString=@"
with resultset (intCount, strCount, datCount) as (
select 1 intCount, 'Number_' || to_char(1) strCount, sysdate datCount
from dual
union all (
select r.intCount+1, 'Number_' || to_char(intCount+1), sysdate+intCount+1
from resultset r
where r.intCount < 10000
)
)
select * from resultset
"@
$timeRdr = Measure-Command {$dr=Get-OracleResultRdr $conString $sqlString}
$timeDa = Measure-Command {$da=Get-OracleResultDa $conString $sqlString}
"Time used by the DataReader : " + $timeRdr.TotalSeconds
"Time used by the dataAdapter: " + $timeDa.TotalSeconds
After clearing the database buffer cache and the shared pool, up came the following results:
PS C:\Windows\system32> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,5704413
PS C:\Windows\system32> (Measure-Command {Get-OracleResultDa $conString $sqlString}).TotalSeconds
0,2791416
The results have been pretty much the same as those that we have seen before using the managed Oracle data provider!
Note Exchanging the Oracle Managed data provider with the unmanaged version doesn’t change much either.
Lastly, we could still revert to the something “completely different”: We can still use ODBC or OLE DB to query our database!
Using the Oracle OleDb provider requires that it is registered on your local machine. Usually it should have been registered during the installation process of the ODAC if you don’t uncheck that component, but I already have noticed that this is not always true. Oracle offers a 64-bit version of the OleDb provider, which is something that Microsoft doesn’t offer.
If you want the 32-bit version, you have to download the ODAC package from Oracle.
If you have to register any of the two versions, pay attention of the regsvr32.exe version you are using! Let’s say the 32-bit version of ODAC has been unzipped to d:\oracle\ODAC32. In this case, we’ll open Windows Powershell (or a command prompt) with administrative privileges and set the current directory to d:\Oracle\Odac32\oledb\bin\ and call the correct 32 bit version of regsvr32, which is C:\Windows\SysWOW64\regsvr32.exe OraOLEDB11.dll
(if your operating system has been installed to drive C).
The 64-bit version would have to be registered with C:\Windows\System32\regsvr32.exe OraOLEDB11.dll, and you might have set the working directory to d:\Oracle\Odac64\oledb\bin\ before, if you had unzipped it there.
Now we can work with the 32- and/or 64-bit versions of Windows Powershell without problems! Otherwise, we may have encountered error messages reporting that the Oracle.OleDb provider is not registered on the local machine.
The changes to the script required to use the OleDb provider are very little: We have to replace each occurrence of the string Oracle.ManagedDataAccess.Client.Oracle with System.Data.OleDb.OleDb and use a different connection string, as shown here:
$conString="Provider=OraOLEDB.Oracle;Data Source=XE;User Id=hr;Password=hr;"
That’s all! If we clear the Oracle cache and run the usual timings, we can see that using OleDb is a little bit slower that our previous solution.
PS D:\Oracle\Odac32\oledb\bin> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
0,9656623
PS D:\Oracle\Odac32\oledb\bin> (Measure-Command {Get-OracleResultDa $conString $sqlString}).TotalSeconds
0,3766087
Being COM-based might be one reason for that because we have another level of indirection between the database and the client, where a lot of processing, especially type conversion may have to take place.
If you use the Microsoft OleDb provider for Oracle, there is only one change to the connection string required: We have to use another provider name Provider=MSDAORA instead of Provider=OraOLEDB.Oracle
The timing results are quite the same. But remember there is no 64-bit version of this provider available.
The last standard solution we could fall back to is using the old ODBC provider. Windows has some standard ODBC drivers available that include the “Microsoft ODBC for ORACLE” driver … if … if you are running a 32-bit OS or at least are planning to use the 32-bit ODBC driver version. The standard procedure: Clicking on ODBC data sources in administrative tools won’t show the Oracle driver if you are running a 64-bit OS! You might see only a few drivers like these:
The highlighted driver is the Oracle 64-bit ODBC driver that I did install on my system before. You can download it as part of the 64-bit ODAC. The 32-bit version is, of course, part of the 32-bit ODAC download.
If you want to see the 32-bit drivers you have to close this window and execute:
C:\Windows\SysWOW64\odbcad32.exe
This shows a lot of more available drivers:
Additionally, we can find the Microsoft ODBC driver for Oracle here:
Again, there are some same small changes that have to be made to the script if you want to use the ODBC driver. By using the 32-bit driver, these changes include: We have to replace each occurrence of the string Oracle.ManagedDataAccess.Client.Oracle withSystem.Data.Odbc.Odbc and use a different connection string, as shown here:
$conString="Driver={Oracle in ORAHOME11G_Client32};Dbq=xe;UID=hr;PWD=hr"
Additionally, we have to change the validation of the parameter $conString because we did expect a “Data Source” component as part of the connection string.
Here we could probably use the “driver” part:
[Parameter(Mandatory=$true)]
[ValidateScript({$_ -match '\bdriver\b'})]
[string]$conString,
Another subject to change is the call to $rdr.GetOracleValue($i). Because there is no such function! We have to use $rdr.GetValue($i) instead.
The resulting script is listed, as shown:
function Get-OracleResultRdr
{
param (
[Parameter(Mandatory=$true)]
[ValidateScript({$_ -match '\bdriver\b'})]
[string]$conString,
[ValidateScript({$_ -match '\bselect\b'})]
[Parameter(Mandatory=$true)]
[string]$sqlString
)
$resultSet=@()
try {
$con = New-Object System.Data.Odbc.OdbcConnection($conString)
$cmd=$con.CreateCommand()
$cmd.CommandText= $sqlString
$con.Open()
$rdr=$cmd.ExecuteReader()
$columnNames=$rdr.GetSchemaTable() | Select-Object -ExpandProperty ColumnName
while ($rdr.Read()) {
$result=@{}
for ($i=0; $i -lt $rdr.FieldCount; $i++) {
$result.Add($columnNames[$i], $rdr.GetValue($i))
}
[pscustomobject]$result
}
} catch {
Write-Error ($_.Exception.ToString())
} finally {
if ($con.State -eq 'Open') { $con.close() }
}
}
function Get-OracleResultDa
{
param (
[Parameter(Mandatory=$true)]
[ValidateScript({$_ -match '\bdriver\b'})]
[string]$conString,
[ValidateScript({$_ -match '\bselect\b'})]
[Parameter(Mandatory=$true)]
[string]$sqlString
)
$resultSet=@()
try {
$con = New-Object System.Data.Odbc.OdbcConnection($conString)
$cmd=$con.CreateCommand()
$cmd.CommandText= $sqlString
$da=New-Object System.Data.Odbc.OdbcDataAdapter($cmd);
$resultSet=New-Object System.Data.DataTable
[void]$da.fill($resultSet)
} catch {
Write-Error ($_.Exception.ToString())
} finally {
if ($con.State -eq 'Open') { $con.close() }
}
$resultSet
}
# Test data for functions
# $conString="Driver={Microsoft ODBC for Oracle};Server=XE;UID=hr;PWD=hr"
$conString="Driver={Oracle in ORAHOME11G_Client32};Dbq=xe;UID=hr;PWD=hr"
$sqlString=@"
with resultset (intCount, strCount, datCount) as (
select 1 intCount, 'Number_' || to_char(1) strCount, sysdate datCount
from dual
union all (
select r.intCount+1, 'Number_' || to_char(intCount+1), sysdate+intCount+1
from resultset r
where r.intCount < 10000
)
)
select * from resultset
"@
$timeRdr = Measure-Command {Get-OracleResultRdr $conString $sqlString}
$timeDa = Measure-Command {Get-OracleResultDa $conString $sqlString}
"Time used by the DataReader : " + $timeRdr.TotalSeconds
"Time used by the dataAdapter: " + $timeDa.TotalSeconds
Timing the data reader and data adapter reveal the following results:
PS D:\Oracle\Odac32\oledb\bin> (Measure-Command {Get-OracleResultRdr $conString $sqlString}).TotalSeconds
1,099397
PS D:\Oracle\Odac32\oledb\bin> (Measure-Command {Get-OracleResultDa $conString $sqlString}).TotalSeconds
0,7356118
The results are pointing into the right direction: The ODBC driver solution is the slowest way to retrieve the data.
But, if you just retrieve 10 thousand rows with three columns you can probably use any of these technologies without having to wait significantly longer! A real life test with more data will have more significance, of course.
We just scratched the surface as we did query some data from our Oracle database and we are done! We didn’t update, delete, or insert any data. And we didn’t issue DDL commands, used stored procedures, ref cursors, parallel queries, and so on.
This might have to be explored in another article or even more.
But, we have learned a little bit about fetching data from Oracle now by using the new managed data provider and we have seen that performance problems may not be buried in database execution time only, but in the way we try to retrieve the data and return it to our application.
So be warned.
Appendix:
“Access to an Oracle Database Server” is something we may have to discuss further before we can go on. Oracle supports many “Naming Methods” like Easy Connect, Local Naming, Directory Naming and External Naming. If you don’t use Easy Connect, which can only be configured for TCP/IP environments, you may have to provide a “Tnsnames.ora” file which specifies the required connection data. This file contains entries like that:
XE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = MyPC.MyDomain.DE)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = XE)
)
)
The topmost XE is the network alias defined to connect to the database.
If you need to use this tnsnames.ora file, it has to be found by the Oracle client.
A well-known algorithm is used to search for this file (“Sqlnet.ora”, “Listener.Ora” and other configuration files may be there too). Without all the details, it is important to know that the lookup procedure includes the path of the executable, other locations in the order specified by the PATH variable, and a path pointed to by the TNS_ADMIN environment variable.
~Klaus
Klaus, thank you for a comprehensive and educational blog post. I really appreciate you taking the time to research and to write this article. Join me tomorrow as I talk about using Windows PowerShell to discover to hidden WMI classes so I can inventory my HID devices. It is fun, I promise.
I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.
Ed Wilson, Microsoft Scripting Guy