Category Archives: SQL Server 2011 (Denali)

SQL Server – Backup Verification Script – Based on conditions

Problem: There are scenarios where you have to verify the multiple SQL Server instances backup within limited time frame whether it meet your organization backup criteria standards or not.

Solution:
You can execute below T-SQL script to check the backup status as per your organization standard by assigning backup conditions value to backup criteria variable. You can execute below script either by connecting each SQL Server Instance or execute it by registering all the SQL Server Instances in Central Management Server (CMS).

Below T-SQL Script work with all the versions of MSSQL Server and verify FULL, Differential and Transaction Log Backup. It will check the max backup date and evaluate result based on condition as backup is Passed or Not.

Backup Verification Script will return below values. Column names are self-explanatory.
• SQLInstanceName
• DatabaseName
• db_create_ddate
• DB_Recovery_Model
• dbstatus
• windowsservername
• productversion
• productlevel
• edition
• current_datetime
• last_full_backup_date
• last_diff_backup_date
• last_tran_backup_date
• days_since_last_full_backup
• days_since_last_diff_backup
• hours_since_last_tranlog_backup
• Full_Backup_Stauts
• Diff_Backup_Stauts
• Log_Backup_Stauts
• full_backup_location
• diff_backup_location
• tlog_backup_location

You have to specify the backup condition based on your organization standard by assigning values to below variables.

declare @full_backup_criteria_in_days int
declare @diff_backup_criteria_in_days int
declare @log_backup_criteria_in_hours int

-- specify the backup criteria
set @full_backup_criteria_in_days = 7 -- 7 day older Full Backup is Pass 
set @diff_backup_criteria_in_days = 1 -- 1 day older Diff Backup is Pass 
set @log_backup_criteria_in_hours = 2 -- 2 hours older Log Backup is Pass

Complete Backup Verification Script

declare @full_backup_criteria_in_days int
declare @diff_backup_criteria_in_days int
declare @log_backup_criteria_in_hours int

-- specify the backup criteria
set @full_backup_criteria_in_days = 7 -- 7 day older Full Backup is Pass 
set @diff_backup_criteria_in_days = 1 -- 1 day older Diff Backup is Pass 
set @log_backup_criteria_in_hours = 2 -- 2 hours older Log Backup is Pass

select     
           serverproperty('servername') as SQLInstanceName,
           quotename(bkup_full.[database_name]) as DatabaseName,
            ( select    sdb.crdate
              from      [master]..[sysdatabases] sdb
              where     sdb.name = bkup_full.[database_name]
            ) as [db_create_ddate],           
           databasepropertyex(bkup_full.[database_name],'recovery')  as DB_Recovery_Model,
           databasepropertyex(bkup_full.[database_name],'status')  as dbstatus,       
           case serverproperty('isclustered') 
                        when 1 then cast(serverproperty('computernamephysicalnetbios') as varchar)
                        when 0 then cast(serverproperty('machinename') as varchar)
           end as windowsservername,
           serverproperty('productversion') as productversion,
           serverproperty('productlevel') as productlevel,
           serverproperty('edition') as edition,

           current_timestamp as current_datetime,
		   
		   bkup_full.[backup_finish_date] as [last_full_backup_date],           
		   bkup_diff.[backup_finish_date] as [last_diff_backup_date] ,
		   bkup_log.[backup_finish_date] as [last_tran_backup_date] ,

           datediff(dd, bkup_full.[backup_finish_date], current_timestamp) as [days_since_last_full_backup] ,
           datediff(dd, bkup_diff.[backup_finish_date], current_timestamp) as [days_since_last_diff_backup] ,
           datediff(hh, bkup_log.[backup_finish_date], current_timestamp) as [hours_since_last_tranlog_backup] ,
            
           case 
                        when datediff(dd, bkup_full.[backup_finish_date], current_timestamp) <= @full_backup_criteria_in_days  
                        then 'Pass'
                        else 'Fail'
                        end as Full_Backup_Stauts,
            
            case 
                        when datediff(dd, bkup_diff.[backup_finish_date], current_timestamp) <= @diff_backup_criteria_in_days  then 'Pass'
                        else case when quotename(bkup_full.[database_name]) IN ('[master]') then 'N/A' else 'Fail' end                        
                        end as Diff_Backup_Stauts,

            case 
                        when datediff(hh, bkup_log.[backup_finish_date], current_timestamp) <= @log_backup_criteria_in_hours  then 'Pass'
                        else case when databasepropertyex(bkup_full.[database_name],'recovery') = 'SIMPLE' then 'N/A' else 'Fail' end                        
            end as Log_Backup_Stauts,            

            ( select top 1 [physical_device_name]
              from      [msdb]..[backupmediafamily] bkup_media_family
              where     bkup_media_family.[media_set_id] = bkup_full.[media_set_id]
            ) as [full_backup_location] ,
            ( select top 1  [physical_device_name]
              from      [msdb]..[backupmediafamily] bkup_media_family
              where     bkup_media_family.[media_set_id] = bkup_diff.[media_set_id]
            ) as [diff_backup_location] ,
            ( select top 1  [physical_device_name]
              from      [msdb]..[backupmediafamily] bkup_media_family
              where     bkup_media_family.[media_set_id] = bkup_log.[media_set_id]
            ) as [tlog_backup_location]
         
                             
    from    [msdb]..[backupset] as bkup_full
            left join [msdb]..[backupset] as bkup_log on bkup_log.[database_name] = bkup_full.[database_name]
                                                       and bkup_log.[server_name] = bkup_full.[server_name]
                                                       and bkup_log.[type] = N'L'
                                                       and bkup_log.[backup_finish_date] = ( (select  max([backup_finish_date])
                                                                                            from    [msdb]..[backupset] b2
                                                                                            where   b2.[database_name] = bkup_full.[database_name]
                                                                                                    and b2.[server_name] = bkup_full.[server_name]
                                                                                                    and b2.[type] = N'L') )
            left join [msdb]..[backupset] as bkup_diff on bkup_diff.[database_name] = bkup_full.[database_name]
                                                       and bkup_diff.[server_name] = bkup_full.[server_name]
                                                       and bkup_diff.[type] = N'I'
                                                       and bkup_diff.[backup_finish_date] = ( (select max([backup_finish_date])
                                                                                            from    [msdb]..[backupset] b2
                                                                                            where   b2.[database_name] = bkup_full.[database_name]
                                                                                                    and b2.[server_name] = bkup_full.[server_name]
                                                                                                     and b2.[type] = N'I') )
    where   bkup_full.[type] = N'D'
            and bkup_full.[backup_finish_date] = ( (select  max([backup_finish_date])
                                                 from   [msdb]..[backupset] b2
                                                 where  b2.[database_name] = bkup_full.[database_name]
                                                        and b2.[server_name] = bkup_full.[server_name]
                                                        and b2.[type] = N'D') )
            and exists ( select [name]
                         from   [master]..[sysdatabases]
                         where  [name] = bkup_full.[database_name] )
            and bkup_full.[database_name] <> N'tempdb'
Advertisements

RESOURCE_SEMAPHORE WAIT – Performance Troubleshooting

Recently I was involved in troubleshooting the performance issue. All the queries were running slow on SQL instance. Most of the queries usually complete in seconds and that day it was running more than 15 minutes and not return any output.

I have followed below approach to fix this issue.
Step 1: Checked for the blocking if there is any on the SQL instace but didn’t find any.

select * from sys.sysprocesses where blocked <> 0

Step 2: Checked for the active sessions on the server. To find out if there is any session which in KILLED/Rollback state but didn’t find out any. Sessions in KILLED/Rollback states are resource incentive sessions based on different conditions.

sp_who2 active

Step 3: Checked for the overall wait types on the SQL Server and found RESOURCE_SEMAPHORE and Page IO related waits.

SELECT TOP 10
        wait_type ,
        max_wait_time_ms wait_time_ms ,
        signal_wait_time_ms ,
        wait_time_ms - signal_wait_time_ms AS resource_wait_time_ms ,
        100.0 * wait_time_ms / SUM(wait_time_ms) OVER ( )
                                    AS percent_total_waits ,
        100.0 * signal_wait_time_ms / SUM(signal_wait_time_ms) OVER ( )
                                    AS percent_total_signal_waits ,
        100.0 * ( wait_time_ms - signal_wait_time_ms )
        / SUM(wait_time_ms) OVER ( ) AS percent_total_resource_waits
FROM    sys.dm_os_wait_stats
WHERE   wait_time_ms > 0
        AND wait_type NOT IN 
( 'SLEEP_TASK', 'BROKER_TASK_STOP', 'BROKER_TO_FLUSH',
  'SQLTRACE_BUFFER_FLUSH','CLR_AUTO_EVENT', 'CLR_MANUAL_EVENT', 
  'LAZYWRITER_SLEEP', 'SLEEP_SYSTEMTASK', 'SLEEP_BPOOL_FLUSH',
  'BROKER_EVENTHANDLER', 'XE_DISPATCHER_WAIT', 'FT_IFTSHC_MUTEX',
  'CHECKPOINT_QUEUE', 'FT_IFTS_SCHEDULER_IDLE_WAIT', 
  'BROKER_TRANSMITTER', 'FT_IFTSHC_MUTEX', 'KSOURCE_WAKEUP',
  'LAZYWRITER_SLEEP', 'LOGMGR_QUEUE', 'ONDEMAND_TASK_QUEUE',
  'REQUEST_FOR_DEADLOCK_SEARCH', 'XE_TIMER_EVENT', 'BAD_PAGE_PROCESS',
  'DBMIRROR_EVENTS_QUEUE', 'BROKER_RECEIVE_WAITFOR',
  'PREEMPTIVE_OS_GETPROCADDRESS', 'PREEMPTIVE_OS_AUTHENTICATIONOPS',
  'WAITFOR', 'DISPATCHER_QUEUE_SEMAPHORE', 'XE_DISPATCHER_JOIN',
  'RESOURCE_QUEUE' )
ORDER BY wait_time_ms DESC

Step 4: Analyzed the wait types, while analyzing the wait types I found the RESOURCE_SEMAPHORE wait type. Let’s see what is RESOURCE_SEMAPHORE wait type and what does it indicate.

RESOURCE SEMAPHORE SQL wait type “Occurs when a memory request for query cannot be granted immediately due to other concurrent queries or memory pressure, Resource Semaphore high waits and wait times may indicate excessive number of concurrent queries, or excessive memory request amounts”

Let’s query DMV to get more idea on Resource Semaphore wait. In the output of the below query output you have to check grantee_count and waiter_count column values, grantee_count is the number of queries which have their memory granted and the waiter_count is the number of queries which are waiting in queue to get memory

SELECT * FROM sys.dm_exec_query_resource_semaphores

Now let’s check queries waiting for memory using “sys.dm_exec_query_memory_grants” DMV. Columns grant_time and granted_memory_kb will be NULL for those queries which are waiting to get their requested memory

SELECT * FROM sys.dm_exec_query_memory_grants  where grant_time is null

I have also checked the “Memory Grants Pending” performance counter to get more idea. If Memory Grants Pending values is great than 0, than there is a problem.

SELECT OBJECT_NAME,cntr_value AS [Memory Grants Pending]
FROM sys.dm_os_performance_counters
WHERE OBJECT_NAME = 'MSSQL$MyInstance:Memory Manager'
AND counter_name = 'Memory Grants Pending'

Step 5 To resolve the issue immediately, as it was active-active cluster and there was memory available to allocate to SQL Server, I have increased the memory.

Long term fix
Find out the memory consuming queries and tune them
Schedule BULK Processing Jobs during off business hours
Make sure stats and indexes are up to date
If the server load increase, plan to add more memory

Spinlocks lightweight synchronization primitives

Spinlocks are lightweight synchronization primitives which are used to protect access to data structures. Spinlocks are used to access data structures for a very short period of time. When a thread attempting to acquire a spinlock and if it is unable to obtain access it executes in a loop periodically checking to determine if the resource is available instead of immediately yielding.

After some period of time a thread waiting on a spinlock will yield before it is able to acquire the resource in order to allow other threads running on the same CPU to execute. This is known as a backoff.

When a large number of threads are contending for access to a single spinlock and it can lead to performance problems

You can use below DMV to get the SPIN LOCK information

select * from sys.dm_os_spinlock_stats

Examples
FGCP_PRP_FILL
OPT_IDX_STATS
BUF_FREE_LIST

Best Practices SQL Server Transaction Log

Background
In SQL Server Database Recovery model will decide how the transaction log will be logged in transaction log file. Transaction log file extension is .LDF

Full – Transaction log is fully logged (Can take log backup)
Bulk Logged – Bulk transaction is minimally logged (Can take log backup)
Simple – Transaction log will be truncated on checkpoint

In transaction log file transactions are sequentially logged, every record in transaction log file is uniquely identified by log sequence number (LSN). LSN data type is Numeric (25,0)

You can follow below best practices for the transaction log file

1. Don’t create multiple log files : As transactions will be logged into log file sequential manner it would not help for data stripping across multiple files
2. Keep the transaction log file on the separate drive
3. Identify the RPO and RTO for the database and according to that choose the recovery model and correct log backup strategy
4. RAID 1 + 0 is high recommended for transaction log
5. AUTO SHRINK should be always off on the database
6. Pre-allocate the space to transaction log file, it will improve the performance. Don’t depend on the auto growth option.
7. Always set the values of Initial size, max size and growth property of the transaction log file
8. Always set auto growth value, don’t set in percentage
9. Transaction Log file internal fragmentation can also lead the performance and database recovery issue. Database should not have an excessive number of Virtual Log Files (VLFs) inside the Transaction Log. Having a large number of small VLFs can slow down the recovery process that a database goes through on startup or after restoring a backup. Make sure transaction log initial size and log growth defined well to avoid internal fragmentation
10. External fragmentation can be removed by using disk defragmentation utility
11. In case of Transaction log full, please use below query to check the cause of the log full and take the decision accordingly.

SELECT name ,
recovery_model_desc ,
log_reuse_wait_desc
FROM sys.databases
WHERE name = @DatabaseName

Stripping SQL Server Database Backup to Multiple Files

Stripping Database backup to multiple files and on different drives will make the backup speed faster and will reduce the backup duration.

Check the below sample script for the backup and restore. You can perform the same task using SSMS GUI as well.
Backup Script

BACKUP DATABASE [SQLDBPool] TO  
DISK = N'C:\JSpace\Backup\SQLDBPool1.bak',  
DISK = N'C:\JSpace\Backup\SQLDBPool2.bak',  
DISK = N'C:\JSpace\Backup\SQLDBPool3.bak'
WITH NOFORMAT, 
NOINIT,  
NAME = N'SQLDBPool-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10
GO

Restore Script

RESTORE DATABASE [SQLDBPool] FROM  
DISK = N'C:\JSpace\Backup\SQLDBPool1.bak',  
DISK = N'C:\JSpace\Backup\SQLDBPool2.bak',  
DISK = N'C:\JSpace\Backup\SQLDBPool3.bak'
WITH  FILE = 1,  NOUNLOAD,  STATS = 10,replace
GO

Script to find out the traces running on SQL Server instance

You can execute the below script on SQL Server instance to find out the traces, trace type, trace file path and trace status.

select
      TraceType =
       case trace.is_default
            when 1 THEN 'Default/System Trace'
            when 0 THEN 'User Trace'
       end,
      Trace_Status =
      case trace.status
            when 1 THEN 'Running'
            when 0 THEN 'Stopped'
      end,
       ssion.session_id as SessionID,
       [loginName] = coalesce(ssion.login_name,ssion.login_name,'Reader SPID Not mentioned'),
       [Trace_File_Path] = coalesce(trace.[Path],trace.[Path],'OLEDB Client Trace')
      from sys.traces trace
            left join sys.dm_exec_sessions ssion on trace.reader_spid = ssion.session_id