Showing posts with label MicroSoft SQL Server. Show all posts
Showing posts with label MicroSoft SQL Server. Show all posts

Sunday 21 November 2021

Missing SQL Server Configuration Manager (SSCM) from the Start Menu

 Have you encountered an issue where the SQL Server Configuration Manager console is not available among the other SQL Server-related start menu applications?

This happened to me after upgrading from Windows 10 to Windows 11.

I have two instances installed on my PC (2016 & 2019). But SSCM is missing on both of them in the start menu.











But it's available and you can access it either way shown below:

1. Directly from the directory.

The relevant management consol service (.msc) file can be found in C:\Windows\System32 folder. You can directly browse the file and open the application.








2. Accessing via Computer Management Console.

SSCM will be available in the Computer Management Console, under "Services and Applications"











3. Using Microsoft Management Console.

Prior to using SSCM, you need to add it to MMC first. You can open Microsoft Management Console, by running "mmc" in the Run dialogue.








Once the console is opened, add the SQL Server Configuration Manager using Add/Remove Snap-In.



























Hope this will be helpful for you.

Thursday 4 November 2021

Full Backup Vs. Differential Backup Vs. Log Backup

When it comes to backing up SQL Server databases, it's critical to know what backup types are there and what you should choose. In order to do this, you must first understand what SQL Server will include in each of these backup files.

If your database is in the "Full Recovery" model, you have the option to choose whether you want to back up
the data file, log file or both.

You will be given these options:







To illustrate we will consider a hypothetical database which we will be backing up hourly.

Full Backup


As the name implies full backup will include all the details (data), every time you back up the database.

































Consider the above case, where we will take full backup hourly.

Full Backup #01 will contain all the data which was there since the beginning.
Full Backup #02 will contain all the data including the changes that happened between 8:00am - 9:00am
Full Backup #03 will contain all the data including the changes that happened between 9:00am - 10:00am

The backup file size will grow since it will contain everything all the time. You need only one backup in order to restore the database and it will take the database to the status (or time) when the backup was initiated.

Differential Backup

Unlike the full backup, the differential backup will only contain the changes that happened since the last full backup



Full Backup #01 will contain all the data which was there since the beginning.
Differential Backup #01 will contain only the data changes that happened between 8:00am - 9:00am
Differential Backup #02 will contain only the data changes that happened between 8:00am - 10:00am

The differential backup size is comparatively less than the full backup since it only contains changes since the last full backup. You need to have both full backup and the differential backup (last one) in order to restore the database.

Log Backup


The log backup will only backup the transaction log file. It will not contain any data pages from the data file. A log backup will always contain changes in the log file since the last log backup.
































Consider the above case. Even though the database is fully backed up at 8:00AM the first transaction log backup taken at 9:00am will contain details since 7:00am.

Full Backup #01 will contain details from both data and log file as of 8:00am
Log Backup #01 will only contain details from the transaction log between 7:00am - 9:00am 
Log Backup #02 will only contain details from the transaction log between 9:00am - 10:00am 

With the log backups, you can take the database to any time of the day (in this case up to hourly blocks). But you need to retain all the log backups in order to do that.





Monday 1 November 2021

Removing Invalid Logins (Both Server and Database Principals) from SQL Server

Have you ever come across a situation where you need to remove SQL Logins of users which have already left the organisation and not valid anymore ?

This could be very tedious task since you need to remove the permission from all the databases in the server which users has access to prior removing the SQL login. It could be more painful when the user is owning a schema in the database. Then you need to assign the ownership of the schema to [dbo] prior dropping the permission from the database.

We will see how we can achieve this step by step and finally we will make a script which we can run and ultimately produce a script which we can use to clean up the SQL Server.

1. Frist we need to find out the logins which SQL Server is considered as invalid.  We can use a system stored procedure "sys.sp_validatelogins" for this.

  
CREATE TABLE #InvalidUsers(
	[SID]		VARBINARY(85)
	,LOGIN_NAME	SYSNAME
)

INSERT INTO #InvalidUsers (
	[SID]
	,LOGIN_NAME
)
EXEC sys.sp_validatelogins






















2. Next we need to iterate each user in all the databases on the server and remove them from the databases. We will generate a script and use it to drop users. We need to consider the following during the script generation.

- Whether the user owns a schema in the database (if so we need to change the schema ownership to dbo)

-  Whether the user owns a the database. If so we need to change db ownership to a different user. In this example I will be changing the ownership to "sa"

We will insert the details into another temporary table. I will explain part by part, but once the script is completed you can execute it as a whole. I have provided the completed script at the end.

We will use another system stored procedure to execute a query in all the databases in the SQL Server instance (sys.sp_MSforeachdb)

	
DECLARE @LoginName AS sysname
	,@Sql AS NVARCHAR(2000)

DECLARE cur CURSOR LOCAL  READ_ONLY FORWARD_ONLY FOR 
SELECT LOGIN_NAME FROM #InvalidUsers

OPEN cur

FETCH NEXT FROM cur INTO @LoginName
WHILE @@FETCH_STATUS = 0
BEGIN
    
	SET @Sql = CONCAT(
		'USE ?;INSERT INTO #InvalidUserDbInfo SELECT DB_NAME(),DP.sid,DP.name, S.[name] as OwnedSchema 
        FROM sys.database_principals AS DP 
        LEFT JOIN sys.schemas as S on S.principal_id = DP.principal_id 
        WHERE [type] = ''U'' and DP.name = ''',@LoginName,''''
	)
	EXEC sys.sp_MSforeachdb @Sql
	FETCH NEXT FROM cur INTO @LoginName
END


CLOSE cur
DEALLOCATE cur


We will use the details populated into the table #InvalidUserDbInfo to generate a script to drop users from databases, change database ownerships and change schema ownerships.

   
DECLARE 
	@LoginName AS sysname
	,@Sql AS NVARCHAR(2000)
	,@DBName AS NVARCHAR(128)
	,@DbNameCurrent AS NVARCHAR(128) = ''
	,@SqlSyntax AS NVARCHAR(MAX) = ''
	,@OwnedSchema AS VARCHAR(128)

DECLARE cur CURSOR LOCAL  READ_ONLY FORWARD_ONLY FOR 
SELECT 
	IUDI.DBNAME
		,IUDI.LOGIN_NAME 
		,IUDI.OWNED_SCH
FROM 
	#InvalidUserDbInfo AS IUDI WHERE IUDI.DBNAME <> 'master' ORDER BY IUDI.DBNAME

OPEN cur

FETCH NEXT FROM cur INTO @DBName,@LoginName,@OwnedSchema

WHILE @@FETCH_STATUS = 0
BEGIN
	
	IF @DbNameCurrent <> @DBName
	BEGIN
		IF @DbNameCurrent <> ''
			SET @SqlSyntax += CONCAT(CHAR(13),'---------------------------------',CHAR(13),CHAR(13))

		SET @SqlSyntax += CONCAT('USE [',@DBName,'];',CHAR(13),CHAR(13))
		
		SET @DbNameCurrent = @DBName
		
	END

	IF (ISNULL(@OwnedSchema,'') <> '' AND ISNULL(@OwnedSchema,'') <> @LoginName)
	BEGIN
		SET @SqlSyntax += CONCAT('ALTER AUTHORIZATION ON SCHEMA::[',@OwnedSchema,'] TO [dbo]', CHAR(13))
	END

	SET @SqlSyntax += CONCAT('DROP USER IF EXISTS [',@LoginName,']',CHAR(13))

	FETCH NEXT FROM cur INTO @DBName,@LoginName,@OwnedSchema

END

CLOSE cur
DEALLOCATE cur

SELECT CAST('<![CDATA[' + CHAR(13) + @SqlSyntax + CHAR(13) + ']]>' AS XML) AS CleanupScript_Step_2

This will return a clickable result set. Upon clicking you will get the script to remove users from databases.




3. Finally we consider the database ownerships and drop the login from the SQL Server Instance

   
DECLARE 
	@SqlSyntax AS NVARCHAR(MAX) = ''


SET @SqlSyntax += CONCAT(' --== Changing Database Ownership ==--',CHAR(13),CHAR(13),'USE master;',CHAR(13))

SELECT 
	@SqlSyntax = @SqlSyntax + CHAR(13) + CONCAT('ALTER AUTHORIZATION ON DATABASE::[',S.[name],'] TO [sa]') 
FROM 
	sys.databases AS S
	JOIN #InvalidUsers AS U
		ON SUSER_SNAME(S.owner_sid) = U.LOGIN_NAME

SET @SqlSyntax+= CONCAT(CHAR(13),CHAR(13))

------------------------------------------------------------------------------------------


SELECT 
	@SqlSyntax = @SqlSyntax + CHAR(13) + 
    	CONCAT('IF EXISTS (SELECT 0 FROM [sys].[server_principals] AS [SP] WHERE [SP].[name] = N'''
        	,U.LOGIN_NAME
            ,''')'
            ,CHAR(13),CHAR(9)
            ,'DROP LOGIN [',U.LOGIN_NAME,']',CHAR(13),CHAR(13)
        )
FROM
	[sys].[server_principals] AS [SP]
	JOIN #InvalidUsers AS U
		ON U.LOGIN_NAME = SP.[name]


SELECT CAST('<![CDATA[' + CHAR(13) + @SqlSyntax + CHAR(13) + ']]>' AS XML) AS CleanupScript_Step_3  
  

This will generate a script similar to the one shown below.














Following is the full script which I have combine all the steps and which can be executed in a one go. You can download the script from the following link:

  

CREATE TABLE #InvalidUsers(
	[SID]		VARBINARY(85)
	,LOGIN_NAME	SYSNAME
)

INSERT INTO #InvalidUsers (
	[SID]
	,LOGIN_NAME
)
EXEC sys.sp_validatelogins

------------------------------------------------------------------------------------------

CREATE TABLE  #InvalidUserDbInfo(
	DBNAME		NVARCHAR(128)		
	,[SID]			VARBINARY(85)
	,LOGIN_NAME	SYSNAME
	,OWNED_SCH		VARCHAR(128)
)

DECLARE 
	@LoginName AS sysname
	,@Sql AS NVARCHAR(2000)
	,@DBName AS NVARCHAR(128)
	,@DbNameCurrent AS NVARCHAR(128) = ''
	,@SqlSyntax AS NVARCHAR(MAX) = ''
	,@OwnedSchema AS VARCHAR(128)

DECLARE cur CURSOR LOCAL  READ_ONLY FORWARD_ONLY FOR 
SELECT LOGIN_NAME FROM #InvalidUsers

		

OPEN cur

FETCH NEXT FROM cur INTO @LoginName
WHILE @@FETCH_STATUS = 0
BEGIN
    
	SET @Sql = CONCAT(
		'USE ?;INSERT INTO #InvalidUserDbInfo 
		SELECT DB_NAME(),DP.sid,DP.name, S.[name] as OwnedSchema 
		FROM sys.database_principals AS DP 
		LEFT JOIN sys.schemas as S on S.principal_id = DP.principal_id 
		WHERE [type] = ''U'' and DP.name = ''',@LoginName,''''
	)
	EXEC sys.sp_MSforeachdb @Sql
	FETCH NEXT FROM cur INTO @LoginName
END


CLOSE cur
DEALLOCATE cur

------------------------------------------------------------------------------------------

DECLARE cur CURSOR LOCAL  READ_ONLY FORWARD_ONLY FOR 
SELECT 
	IUDI.DBNAME
		,IUDI.LOGIN_NAME 
		,IUDI.OWNED_SCH
FROM 
	#InvalidUserDbInfo AS IUDI WHERE IUDI.DBNAME <> 'master' ORDER BY IUDI.DBNAME

OPEN cur

FETCH NEXT FROM cur INTO @DBName,@LoginName,@OwnedSchema

WHILE @@FETCH_STATUS = 0
BEGIN
	
	IF @DbNameCurrent <> @DBName
	BEGIN
		IF @DbNameCurrent <> ''
			SET @SqlSyntax += CONCAT(CHAR(13),'---------------------------------',CHAR(13),CHAR(13))

		SET @SqlSyntax += CONCAT('USE [',@DBName,'];',CHAR(13),CHAR(13))
		
		SET @DbNameCurrent = @DBName
		
	END

	IF (ISNULL(@OwnedSchema,'') <> '' AND ISNULL(@OwnedSchema,'') <> @LoginName)
	BEGIN
		SET @SqlSyntax += CONCAT('ALTER AUTHORIZATION ON SCHEMA::[',@OwnedSchema,'] TO [dbo]', CHAR(13))
	END

	SET @SqlSyntax += CONCAT('DROP USER IF EXISTS [',@LoginName,']',CHAR(13))

	FETCH NEXT FROM cur INTO @DBName,@LoginName,@OwnedSchema

END

CLOSE cur
DEALLOCATE cur

------------------------------------------------------------------------------------------

SET @SqlSyntax += 
	CONCAT(CHAR(13),CHAR(13)
    ,' --== Changing Database Ownership ==--',CHAR(13),CHAR(13)
    ,'USE master;',CHAR(13))

SELECT 
	@SqlSyntax = @SqlSyntax + CHAR(13) + CONCAT('ALTER AUTHORIZATION ON DATABASE::[',S.[name],'] TO [sa]') 
FROM 
	sys.databases AS S
	JOIN #InvalidUsers AS U
		ON SUSER_SNAME(S.owner_sid) = U.LOGIN_NAME

SET @SqlSyntax+= CONCAT(CHAR(13),CHAR(13))

------------------------------------------------------------------------------------------


SELECT 
	@SqlSyntax = @SqlSyntax + CHAR(13) + 
		CONCAT('IF EXISTS (SELECT 0 FROM [sys].[server_principals] AS [SP] WHERE [SP].[name] = N'''
			,U.LOGIN_NAME,''')'
			,CHAR(13),CHAR(9)
			,'DROP LOGIN [',U.LOGIN_NAME,']',CHAR(13),CHAR(13))
FROM
	[sys].[server_principals] AS [SP]
	JOIN #InvalidUsers AS U
		ON U.LOGIN_NAME = SP.[name]


SELECT CAST('<![CDATA[' + CHAR(13) + @SqlSyntax + CHAR(13) + ']]>' AS XML) AS CleanupScript_Step_3
    

Hope this will be very useful to you!



Sunday 31 October 2021

Attaching SQL Database with FILESTREAM data

 Have you ever come across a situation where you need to attach a database using existing files (mdf, ldf & file stream data), but you don't have an option to mention the file stream folder when you use SSMS.

This won't be an issue if the filestream folder resides on its original path.

Let's look at an example. I have SQL data, log and filestream data which I will first attach using SSMS.

  1. Right-click on the database folder in SSMS and choose "Attach"
  2. Click the "Add" button and choose the data file (.mdf). Upon selecting the mdf file it will list the associated ldf file as well. But make sure to change the path accordingly from the second pane.
  3. Click on Ok.


In the above example, it will succeed without an issue since I have not changed the paths.

Now we will detach the database and change the file stream folder name to a different one. I will change the folder name to "SampleSqlDatabase_FSData_New" (This is the most common case where you will get the relevant files from the production environment which you need to attach to a test or development environment.

Now if you try to attach the database using SSMS in the same way you will get the following error message.











Even if you try to generate the scripts using SSMS it doesn't provide you with the option to include the filestream folder.

















   
USE [master]
GO
CREATE DATABASE [SampleSqlDatabase] ON 
( FILENAME = N'D:\_SQL_DATA\SampleSqlDatabase_Data.mdf' ),
( FILENAME = N'D:\_SQL_LOG\SampleSqlDatabase_Log.ldf' )
 FOR ATTACH
GO

We can use the above code snippet and include the file stream path.

    
USE [master]
GO
CREATE DATABASE [SampleSqlDatabase] ON 
(FILENAME = N'D:\_SQL_DATA\SampleSqlDatabase_Data.mdf'),
(FILENAME = N'D:\_SQL_LOG\SampleSqlDatabase_Log.ldf'),
FILEGROUP NewFSGroup CONTAINS FILESTREAM (
	NAME = 'NewDatabaseFileStream'
	,FILENAME = 'D:\_SQL_DATA\SampleSqlDatabase_FSData_New'
)
FOR ATTACH
GO

Then you will be able to attach the database without an issue. You can use any name for the FILEGROUP and NAME (under the FILEGROUP). It doesn't need to be the same as your source database.

Hope this will be a helpful tip to you.

SQL Server Transaction Log file behaviour in Full Recovery Mode

In a previous blog post, I have explained different types of recovery models in SQL Server databases. During the explanations, I have stated that the transaction log data will be truncated during certain backup types. We will look into this more closely and see how it works.

Note: Even the details are removed from the transaction log file, the physical size may not change unless you issue a DBCC SHRINKFILE command. Only the file space will be re-used once the details are truncated.

We will start with creating a sample database named 'SqlTransactionLogExample'

      
USE master;
GO 

CREATE DATABASE [SqlTransactionLogExample];

--== Makesure the database is in "Full" recovery model ==--

SELECT 
	[name]
	,DATABASEPROPERTYEX([name],'Recovery') AS recovery_model
FROM
	sys.databases
WHERE
	[name] = 'SqlTransactionLogExample'
    

We will create one table and insert records (1,000,000 rows to see a significant change in the file size).

   
USE SqlTransactionLogExample;
GO

CREATE TABLE dbo.UidInformation(
	Id				UNIQUEIDENTIFIER NOT NULL
	,PayLoad		CHAR(1000) NOT NULL CONSTRAINT [DF_SampleTable] DEFAULT (SPACE(1000))
)

INSERT INTO dbo.UidInformation (
	Id
	,PayLoad
)
SELECT	
	NEWID() AS Id,
	([100000].Num 
	+ [10000].Num  
	+ [1000].Num 
	+ [100].Num 
	+ [10].Num 
	+ [1].Num) AS PayLoad
FROM (
	SELECT 1 AS Num UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL 
	SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 
	UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9 UNION ALL 
	SELECT 0
) AS [1]
CROSS JOIN (
	SELECT 10 AS Num UNION ALL SELECT 20 UNION ALL SELECT 30 UNION ALL 
	SELECT 40 UNION ALL SELECT 50 UNION ALL SELECT 60 UNION ALL 
	SELECT 70 UNION ALL SELECT 80 UNION ALL SELECT 90 UNION ALL 
	SELECT 0
) AS [10]
CROSS JOIN (
	SELECT 100 AS Num UNION ALL SELECT 200 UNION ALL SELECT 300 
	UNION ALL SELECT 400 UNION ALL SELECT 500 UNION ALL SELECT 600 
	UNION ALL SELECT 700 UNION ALL SELECT 800 UNION ALL SELECT 900 
	UNION ALL SELECT 0
) AS [100]
CROSS JOIN (
	SELECT 1000 AS Num UNION ALL SELECT 2000 UNION ALL SELECT 3000 
	UNION ALL SELECT 4000 UNION ALL SELECT 5000 UNION ALL SELECT 6000 
	UNION ALL SELECT 7000 UNION ALL SELECT 8000 UNION ALL SELECT 9000 
	UNION ALL SELECT 0
) AS [1000]
CROSS JOIN (
	SELECT 10000 AS Num UNION ALL SELECT 20000 UNION ALL SELECT 30000 
	UNION ALL SELECT 40000 UNION ALL SELECT 50000 UNION ALL SELECT 60000 
	UNION ALL SELECT 70000 UNION ALL SELECT 80000 UNION ALL SELECT 90000 
	UNION ALL SELECT 0
) AS [10000]
CROSS JOIN (
	SELECT 100000 AS Num UNION ALL SELECT 200000 UNION ALL SELECT 300000 
	UNION ALL SELECT 400000 UNION ALL SELECT 500000 UNION ALL SELECT 600000 
	UNION ALL SELECT 700000 UNION ALL SELECT 800000 UNION ALL SELECT 900000 
	UNION ALL SELECT 0
) AS [100000]
WHERE(
	[100000].Num 
	+ [10000].Num  
	+ [1000].Num 
	+ [100].Num 
	+ [10].Num 
	+ [1].Num
) BETWEEN 1 AND 1000000
ORDER BY ([100000].Num + [10000].Num  + [1000].Num + [100].Num + [10].Num + [1].Num)


Now we will inspect the number of entries in the transaction log.

   
  SELECT COUNT(*) FROM sys.fn_dblog (NULL,NULL)
  --Returns 2511475


Note: Record count may be varied in the transaction log of your database.


We will check the size of the transaction log file 



Now we will take a full database backup

USE master;
GO

BACKUP DATABASE [SqlTransactionLogExample] TO DISK = 'D:\_SQL_BACKUP\SqlTransactionLogExample_FullDB_Backup.bak'

Now we will inspect number of entries in the transaction log file again.

USE SqlTransactionLogExample
GO

SELECT COUNT(*) FROM sys.fn_dblog (NULL,NULL)

--Returns 2511475


Note: There is a slight increment of the record count since there are new entries written to the log file during the database backup.

We will now inspect the transaction log entries

  
SELECT COUNT(*) FROM sys.fn_dblog (NULL,NULL)
-- Returns 2511545


We will now take a backup of the transaction log 

  
BACKUP LOG [SqlTransactionLogExample] TO 
DISK = 'D:\_SQL_BACKUP\SqlTransactionLogExample_LogBackup_AfterInsert.bak'


And if we inspect the number of entries in the log file it has drastically reduced.

  
SELECT COUNT(*) FROM sys.fn_dblog (NULL,NULL)
--Returns 10


However, the space consumed by the log file is still the same. This is the reason I have mentioned it previously. In order to release the space, we have to issue the DBCC SHRINKFILE command.


--This is to find the name of the logical log file name
SELECT * FROM sys.sysfiles
--SqlTransactionLogExample_log
DBCC SHRINKFILE('SqlTransactionLogExample_log',1)


Hope this will be helpful to you.



Saturday 30 October 2021

Recovery Models in SQL Server Databases

 "Recovery Model" determines how long the data is to be kept in the transaction logs. It also allows what sort of backups and restores you can perform on the database.


Types of recovery models

There are three types of recovery models in SQL Server

  1. Simple
  2. Full 
  3. Bulk-logged

How can I change the recovery model of a database?

The recovery model of a database can be changed either using the GUI (SSMS) or using a T-SQL statement.

To change using SSMS follow these steps:
1. Right-click the database and choose options
2. Select "Options" from the pages
3. From the Recovery model drop-down list choose the appropriate one.
4. Click OK






To change using T-SQL statement use the following syntax:

      
USE master;
ALTER DATABASE YourDBName SET RECOVERY SIMPLE
You can use options SIMPLE, FULL or BULK_LOGGED

You can find out the recovery model of the databases which resides on your server using the following query.

SELECT 
	[name]
	,DATABASEPROPERTYEX([name],'Recovery') AS recovery_model
FROM
	sys.databases
WHERE
	database_id > 4   

Note: database_id > 4 will ensure that system databases information is excluded.

Now we will look into closely what options it may enable during backup/restore for each type of recovery model.

Simple Recovery Model


  • You cannot take backups of the transaction log explicitly
  • A database backup will always contain both data and transaction log file











  • Transaction log will not grow beyond what's been defined. The space in the transaction log file will be re-used (overwritten) once the transactions are written to the data file.
  • In case of a disaster transaction occurred between the last full backup and the disaster timeline cannot be recovered. (E.g: Assume we have a database in which the backups are taken every hour. (8am, 9am etc.) In case a disaster happens at 10:45am, transactions between 10am and 10:45am will be lost)


















Bulk-logged Recovery Model



Full Recovery Model

 
  • Supports transaction log backup
  • Chances of a data loss are minimal (subject to the backup strategy implemented)
  • Log truncation must be done manually (Upon taking the transaction log backup, it will mark unused space which will be overwritten later by future transactions)
  • Transaction log could grow large compared to the "Simple" recovery model

The scope of this post is to give a brief idea of the recovery models in SQL Server. In a future post, I will explain how to bring a database to a certain date and time by having a Full Recovery Model and a good backup strategy.


Tuesday 20 March 2018

Data Encryption in SQL Server using T-SQL Functions (ENCRYPTBYPASSPHRASE, DECRYPTBYPASSPHRASE & HASHBYTES)

Decade ago data was just an entity which helped business to operate smoothly. By then data was considered as some sort of business related information just stored in a database, which can be retrieved based on the demand/requirement as per the demand. E.g: a bunch of products, transactions such as invoices, receipts etc. or customer details.

But today data has become an important entity, which drives business towards success. In today’s fast-moving world, companies who owned data and does analytics has become the most successful companies.

However one of the major concerns we have today is how to protect these data. Especially the sensitive ones. Since more data is being exposed to the cloud, it’s essential to protect it from going to the wrong hands and it has become a major problem since hackers nowadays are well equipped and are always on the look for stealing this valuable information whenever possible, since it’ll be a valuable asset in the open market.

But protecting the data from unauthorized access is a must. Failing to do so can have unexpected consequences. Entire business could get wiped out of the business due to this. Hence enterprises should seriously consider protecting their data and we will discuss how we can achieve this in SQL Server through data encryption.

Ways of Data Encryption in SQL Server

There are few ways of encrypting data in SQL Server. We will discuss the advantages and disadvantages of each method.

SQL Server provides following methods to encrypt data:

  • T-SQL Functions
  • Using Symmetric Keys**
  • Using Asymmetric Keys**
  • Using Certificates**
  • Transparent Data Encryption**

**Note : In this article I only plan to explain encryption/decryption functionality using T-SQL. I will talk about other methods which is mentioned about in future articles.

Using T-SQL Functions

Encrypting data using ENCRYPTBYPASSPHRASE 

Encryption is done using T-SQL function ENCRYPTBYPASSPHRASE.

ENCRYPTBYPASSPHRASE(passphrase,text_to_encrypt)

The first parameter is the passphrase which can be any text or a variable of type NVARCHAR, CHAR, VARCHAR, BINARY, VARBINARY, or NCHAR. The function uses this passphrase to generate a symmetric key.

For the illustration purpose we will create a table which to hold employee details

CREATE TABLE dbo.Employee(
    Id   			 INT
    ,EmpName   	 VARCHAR(100)
    ,EmpDOB   		 SMALLDATETIME
    ,SSN   		 VARBINARY(128)
)

This example is to demonstrate the data encryption during INSERT DML Statement

INSERT INTO dbo.Employee(
	Id
	,EmpName
	,EmpDOB
	,SSN
)
VALUES(
	1
	,'Luke'
	,'01-June-1980'
	,ENCRYPTBYPASSPHRASE('Pa$$W0rd4EnCRyPt10n','111-22-3333')
) 

image

Further details can be found in the Microsoft Documentation:
https://docs.microsoft.com/en-us/sql/t-sql/functions/encryptbypassphrase-transact-sql


Decrypting data using T-SQL function DECRYPTBYPASSPHRASE 

Will take the same details which we inserted during the first case. The encrypted data can be decrypted using SQL function DECRYPTBYPASSPHRASE. If any attempt has been made to decrypt the data without using DECRYPTBYPASSPHRASE nor providing the proper details, it will fail the operation.

Without Decryption

SELECT 
	Id,EmpName,EmpDOB,CONVERT(VARCHAR(128),SSN) AS SSN
FROM 
	dbo.Employee
WHERE 
	Id = 1

image


With Decryption (Incorrect Pass-phrase)

SELECT 
	Id
	,EmpName
	,EmpDOB
	,DECRYPTBYPASSPHRASE('IncorrectPassword',SSN ) AS SSN 
FROM 
	dbo.Employee
WHERE 
	Id = 1

image

But providing the correct pass-phrase will return the correct details

SELECT 
	Id
	,EmpName
	,EmpDOB
	,CONVERT(VARCHAR(128),DECRYPTBYPASSPHRASE('Pa$$W0rd4EnCRyPt10n',SSN )) AS SSN
FROM 
	dbo.Employee
WHERE 
	Id = 1

image

However there could be a requirement which you need to protect your data, not from stealing, but from getting updated with someone else’s.

One classic example is a login table. Suppose we have a table which stores login credentials, which is having the following structure.

*Note: In real world cases, usually it’s more secure if you hash passwords rather than encrypting them. But I am using encryption for illustration purpose.

So if a person has access to update the details on the password column, he/she can easily replace the contents with their own and log on using that. This can be stopped by providing two additional values when details are inserted to the table using ENCRYPTPASSPHRASE.

CREATE TABLE dbo.LoginCredentails(
	UserId		INT
	,UserName	VARCHAR(20)
	,Pwd		VARBINARY(128)
)

We will insert two records to the above created table.

INSERT INTO dbo.LoginCredentails(
	UserId
	,UserName
	,Pwd
)
VALUES
	(1001,'luke.skywalker',ENCRYPTBYPASSPHRASE('Pa$$W0rd4EnCRyPt10n','force be with you',1,CAST(1001 AS sysname)))
	,(1002,'darth.vader',ENCRYPTBYPASSPHRASE('Pa$$W0rd4EnCRyPt10n','i am your father',1,CAST(1002 AS sysname)))

Please note that unlike the previous example, we are now providing two additional values to the ENCRYPTBYPASSPHRASE function. The first values is 1, which indicates whether whether an authenticator will be encrypted together with the password. If the value is 1 and authenticator will be added. The second value is the data which from which to derive an authenticator. In this example we will use a value similar to a user id, so that when the value is decrypted, we could use the same value.

Following is a function to fetch the decrypted password based on the UserId. Assume we will be using this when validating the credential prior login.

CREATE FUNCTION  Fn_GetUserPwdById(@UserId AS INT)
RETURNS VARCHAR(50)
AS
BEGIN	

	DECLARE @Pwd AS VARCHAR(50)
	SELECT
		@Pwd = CONVERT(VARCHAR(50),DECRYPTBYPASSPHRASE('Pa$$W0rd4EnCRyPt10n',LC.Pwd,1,CAST(LC.UserId AS sysname))) 
	FROM
		dbo.LoginCredentails AS LC
	WHERE
		LC.UserId = @UserId

	RETURN @Pwd
END

Using the aforementioned function we will retrieve the details.

SELECT 
	UserId
	,UserName
	,dbo.Fn_GetUserPwdById(UserId) AS Pwd 
FROM
	dbo.LoginCredentails




image

But querying the data simply will get you the binary string of the encrypted value.

SELECT 
	UserId
	,UserName
	,Pwd 
FROM
	dbo.LoginCredentails


Suppose if a person has enough privileges to do an update the password with a known one (from an existing user) it’ll allow him/her to login to the system impersonating any user.

UPDATE LC SET LC.Pwd = (
	SELECT LC2.Pwd FROM dbo.LoginCredentails AS LC2 
	WHERE LC2.UserName = 'luke.skywalker'
) 
FROM dbo.LoginCredentails AS LC
WHERE
LC.UserName = 'darth.vader'




image

But if when the same function is used for decryption, it will return NULL for the updated record, preventing the login to be invalid if it’s replaced using an existing one.

image


Hashing data using HASBYTES

Apart from the above mentioned function, there’s another function which can be used to hash data. Unlike encrypting, there’s no way you can reverse the hashed data and see the raw details.

Syntax:

HASHBYTES ( 'algorithm', { @input | 'input' } )  
/*
algorithm::= MD2 | MD4 | MD5 | SHA | SHA1 | SHA2_256 | SHA2_512   
*/

There are two parameters which you require to provide. The first parameter is the algorithm which should be used for hashing. The hashing algorithm can be any of the following:

      • MD2
  • MD4
  • MD5
  • SHA
  • SHA1
  • SHA2_256
  • SHA2_512

The second parameter is the input, which needs to be hashed. This can be either a character or binary string.

The return value is VARBINARY(n). n = maximum 8000 bytes.

Example:

DECLARE 
	@TextToHash AS NVARCHAR(1000) = N'My Secret Message'

SELECT HASHBYTES('SHA1',@TextToHash) AS HashedData


image


Further details can be found in the Microsoft Documentation:
https://docs.microsoft.com/en-us/sql/t-sql/functions/hashbytes-transact-sql


Hope this might be useful to you and please feel free to comment your ideas.

Sunday 3 December 2017

Behaviour of IDENTITY Columns and SEQUENCES with TRANSACTIONS

Few days back, I was caught in a discussion with couple of my colleagues, regarding a problem they are facing with an IDENTITY column.

The issue was that when a transaction is rolled back the identity seed isn’t rolling back as expected. This was causing the business application to loose the id sequence.

There is no fix or a workaround for this. All that I could provide was an explanation.

I will illustrate the issue and an explanation why it’s happening.

Behaviour of IDENTITY Columns

We will create the following table to hold employee details.

CREATE TABLE dbo.EmployeeInfo(
	Id			INT IDENTITY(1,1) NOT NULL,
	EmpName		VARCHAR(100) NOT NULL
)


Now we will insert few records to the table in the following manner.

  1. Without a transaction
  2. With a transaction. But we will rollback the transaction.
  3. With a transaction. But we will commit it.


INSERT INTO dbo.EmployeeInfo (EmpName)
VALUES('John')

BEGIN TRAN
	INSERT INTO dbo.EmployeeInfo (EmpName)
	VALUES('Jane')
ROLLBACK

INSERT INTO dbo.EmployeeInfo (EmpName)
VALUES('James')

SELECT 
	EI.Id
	,EI.EmpName 
FROM
	dbo.EmployeeInfo AS EI


And when checked, you could see the following results.

image

Usually the expectation is to see the employee “James” with an Id of 2.

What you should understand here is that this isn’t a flaw or a bug. This is the exact intended behaviour and it has been explained in the following MSDN article.

https://docs.microsoft.com/en-us/sql/t-sql/statements/create-table-transact-sql-identity-property

image


Behaviour of SEQUENCES

SEQUENCEs were introduced in SQL Server 2012. The purpose of the SEQUENCE objects were to aid in handling the auto increment numbers, in case you prefer to handle the sequence without using an IDENTITY column.

First we will create a sequence object. The minimum syntax required to create a sequence object is a name and the data type. Additionally you can mention many other attributes like starting index, increment seed etc.

CREATE SEQUENCE dbo.TempNumberSequence AS INT

Further details regarding other options can be found on the following URL:

https://docs.microsoft.com/en-us/sql/t-sql/statements/create-sequence-transact-sql


Now we will create a similar table like we created in the previous example, but without an IDENTITY column.

CREATE TABLE dbo.EmployeeInfoSeq(
	Id			INT 
	,EmpName	VARCHAR(100) NOT NULL
)

We will insert 3 records in the same way like we did in the previous example.

DECLARE @NextSeq AS INT
SELECT @NextSeq = NEXT VALUE FOR dbo.TempNumberSequence
INSERT INTO dbo.EmployeeInfoSeq (
	Id
	,EmpName
)
VALUES (
	@NextSeq
	,'John'
)
GO

DECLARE @NextSeq AS INT
SELECT @NextSeq = NEXT VALUE FOR dbo.TempNumberSequence
BEGIN TRAN
	INSERT INTO dbo.EmployeeInfoSeq (
		Id
		,EmpName
	)
	VALUES (
		@NextSeq
		,'Jane'
	)
ROLLBACK
GO


DECLARE @NextSeq AS INT
SELECT @NextSeq = NEXT VALUE FOR dbo.TempNumberSequence
INSERT INTO dbo.EmployeeInfoSeq (
	Id
	,EmpName
)
VALUES (
	@NextSeq
	,'James'
)
GO

Afterwards if you check, you will see the following results.

image

Hope this will help you in you day to day development work.

Wednesday 20 January 2016

Error accessing Oracle Database Objects via Linked Server in SQL Server (The OLE DB provider "OracleOLEDB.Oracle" for linked server reported an error. Access denied.)


In one of the project we are working these days, there was a requirement to fetch some details, directly from the Oracle Database via VIEWS. Initially everything was setup correctly on the Oracle Database & Server side so that we can access the relevant schemas and fetch data without any issue. And once the oracle client is setup and the configurations are correctly setup (“tnsnames.ora”), we were able to fetch the details using .Net Code. And when we checked using the Oracle SQL Developer UI, it was evident that the details were easily fetched.
How ever we faced an issue when we were asked to access and fetch the same set of details from SQL objects using OPENQUERY. Even when we try a simple query such as retrieving “sysdate”, we got an ‘Access Denied’ error.


SELECT FROM OPENQUERY ([LINKED_SERVER], 'SELECT sysdate FROM DUAL')

The OLE DB provider "OracleOLEDB.Oracle" for linked server reported an error. Access denied. Cannot get the column information from OLE DB provider "OraOLEDB.Oracle" for linked server "<Linked_Server>"

image_thumb[2]

After spending some time with the configurations on both SQL and Oracle side, we were able to rectify this issue by allowing “Allow inprocess” option in linked server providers in SQL side.

image_thumb[4]

I am sharing this hoping that it would help someone to resolve the similar kind of issue without any hassle.





Wednesday 25 November 2015

Analyzing SQL Server Error Logs / Agent Logs using T-SQL

Even though you design our SQL Scripts with the best methods using best practices, or configure the SQL Server to perform correctly and in the optimized manner, you cannot prevent things going wrong. Luckily SQL Server does a great job on logging all the issues which we will be encountering during the course. Things could have been worse if you need to go through the error log file using only a text editor like the ‘Note Pad’ application (Favorite text editor of majority people). But fortunately SQL Server had provided us some help when you need to dig deep into Error Log.

image

But things could get more complicated if the Error Log contains lots of records and in those records if you require to swim for the issue which you are looking for.

image

Even though it provides you some searching and filtering capabilities, it could still be very challenging and time consuming.

image

image

However we do have another workaround which might come in handy. That’s to query the Error Logs using T-SQL. This can be done using the system procedure ‘sys.sp_readerrorlog’. This consists with few parameters.
USE [master]
GO
/****** Object:  StoredProcedure [sys].[sp_readerrorlog]    Script Date: 24/11/2015 7:11:36 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER OFF
GO
ALTER proc [sys].[sp_readerrorlog](
    @p1        int = 0,
    @p2        int = NULL,
    @p3        nvarchar(4000) = NULL,
    @p4        nvarchar(4000) = NULL)
as
begin

    if (not is_srvrolemember(N'securityadmin') = 1)
    begin
       raiserror(15003,-1,-1, N'securityadmin')
       return (1)
    end
    
    if (@p2 is NULL)
        exec sys.xp_readerrorlog @p1
    else 
        exec sys.xp_readerrorlog @p1,@p2,@p3,@p4   
end



  1. @p1 –> This represents the error log which you need to inspect (0 ~ Current | 1 ~ Archive #1 etc..)
  2. @p2 –> Type of the error log which you want to inspect (NULL or 1 ~ Error Log | 2 ~ SQL Agent Log)
  3. @p3 –> 1st Search Parameter (A value which you want to search the contents for)
  4. @p4 –> 2nd Search Parameter (A value which you want to search to further refine the result set)

**Please note: Aforementioned parameters are optional. Therefore if you don’t provide any parameters, it will return the whole contents of the current/active Error Log.

Few Examples

1. This will return all entries in the current Error Log
EXEC sys.xp_readerrorlog @p1 = 0


2. This will return all the entries in the current SQL Agent Log
EXEC sys.xp_readerrorlog @p1 = 0, @p2 = 2


3. This will return all the entries in the current SQL Error log where ever the value ‘CLR’ consist.
EXEC sys.sp_readerrorlog @p1=0, @p2=1, @p3='CLR'


image


4. This will return the entries in the current SQL Error log when the value ‘CLR’ and ‘Framework’ exist.
EXEC sys.sp_readerrorlog @p1=0, @p2=1, @p3='CLR', @p4='Framework'


image

When we execute the stored procedure ‘sys.sp_readerrorlog’, inside it will call an extended stored procedure which will accept 7 parameters, which is  ‘sys.xp_readerrorlog’. The parameter details are as follows:

Param #ParameterDetails
1Log Number0 – Current / 1 – Archive #1 / 2 – Archive #2 etc…
2Log Type1 – SQL Error Log / 2 – SQL Agent Log
3Search Text 1Search term which will be searched on the Text column
4Search Text 2Search term which will be searched on the Text column. **If both search texts are supplied it will return rows containing both texts.
5Start DateLog entries which the ‘Log Date’ is newer than the date provided. (including the date provided)
6End DateLog entries which is between the Start Date and End Date
7Sort OrderASC – Ascending / DESC - Descending

Eg:

EXEC sys.xp_readerrorlog 0,1,N'',N'', '20151124','20151125','DESC'    


I hope this information will help you when you need to query the Error Log in order to troubleshoot an issue.

Sunday 22 November 2015

Enabling Instant File Initialization in SQL Server

Technorati Tags:

Every time the SQL data file or log file expands, it fills the newly allocated (expanded) space with zero. There are few good and bad having this feature (Zeroing the allocated space). One downside of this is, this process will block all the sessions which are writing to these files (data and log files), during this initialization period. One might debate this time period will be very small, but it could be an extremely critical one for some process.

However enabling the Instant File Initialization behavior will make sure that the aforementioned issue will not have any effects when SQL Data file is expanded. But there will be a security risk enabling this feature. When this is enabled, it could be a possibility that unallocated part of the SQL data file could contain information related to previously deleted files (OS related information). There are tools which can examine this data and people who will be having access to the data file (most probably DB Administrators) can easily see the underlying data of these unallocated areas.

Before enabling this we will check how SQL will behave with this option disabled:

DBCC TRACEON(3004,3605,-1)
GO

CREATE DATABASE Sample_Database
GO

EXEC sys.sp_readerrorlog
GO

DROP DATABASE Sample_Database
GO

DBCC TRACEOFF(3004,3605,-1)
GO


 


image


You can clearly see that SQL had an operation to zero out both data and the log file. (I have only highlighted the data file [.mdf])


Enabling Instant File Initialization can be done by adding a ‘SA_MANAGE_VOLUME_NAME’ permission (also know as ‘Perform Volume Maintenance Task’) to the SQL Server Startup account. This can be done as follows:


Open the Local Security Policy Management Application. (execute secpol.msc from the run command or from command line). Double click on the ‘Perform Volume Maintenance Tasks’ which can be found In Security Settings –> Local Policies –> User Rights Assignment.


image 


And add the SQL Server start up account to the list.


image


 


SQL Server startup account can be found in ‘Log On’ Tab in ‘SQL Server <Instance>’ in the Services.


image


 


*** Please note: SQL will check whether this feature is enabled or disable during start up. Therefore we need to restart the service once it’s enabled or disabled. Also we can only enable this for SQL data files only. We cannot enable this for log files.


After restarting the SQL Service we will execute the code snippet which we executed earlier. You will be able to see that SQL is only zeroing the log file now.


image