Quantcast
Channel: SQL Server Customer Advisory Team
Viewing all 96 articles
Browse latest View live

SQLCAT.com consolidated index of Whitepapers

$
0
0

In an effort to migrate as much content as possible from SQLCAT.com to the MSDN SQL CAT site we have consolidated all of the Whitepapers published over the last few years.   While all of the content actually exists on Technet and MSDN download sites, SQLCAT was a great resource to start looking.  Hopefully this will make finding those papers a little easier.

The organization is not quite perfect as many of these papers can cover multiple features and topics, but this is pretty close.  If you happen to find any links that are broken or wrong, please post a comment and we will get it addressed. 

More content will be migrated by the end of month, so stay tuned!

 

SQL Server High Availability Backup Recovery

SQL Server 2012 AlwaysOn: Multisite Failover Cluster Instance

Microsoft SQL Server AlwaysOn Solutions Guide for High Availability and Disaster Recovery

Migration Guide: Migrating to SQL Server 2012 Failover Clustering and Availability Groups from Prior Clustering and Mirroring Deployments

AlwaysOn Architecture Guide: Building a High Availability and Disaster Recovery Solution by Using AlwaysOn Availability Groups

AlwaysOn Architecture Guide: Building a High Availability and Disaster Recovery Solution by Using Failover Cluster Instances and Availability Groups

A Technical Case Study: Fast and Reliable Backup and Restore of Multi-Terabytes Database over the Network

Database Mirroring and Log Shipping Working Together

Database Mirroring Best Practices and Performance Considerations

High Availability and Disaster Recovery at ServiceU: A SQL Server 2008 Technical Case Study

High Availability and Disaster Recovery for Microsoft’s SAP Data Tier: A SQL Server 2008 Technical Case Study

Failure Is Not an Option: Zero Data Loss and High Availability

Implementing Application Failover with Database Mirroring

Proven SQL Server Architectures for High Availability and Disaster Recovery

SQL Server 2008 Failover Clustering

SQL Server High Availability and Disaster Recovery for SAP Deployment at QR: A Technical Case Study"

SQL Server Replication: Providing High Availability using Database Mirroring

Partial Database Availability

Using Replication for High Availability and Disaster Recovery

 

 SQL Server Performance

Analyzing I/O Characteristics and Sizing Storage Systems for SQL Server Database Applications

Predeployment I/O Best Practices

SQL Server 2005 Waits and Queues

Database Snapshot Performance Considerations under I/O-Intensive Workloads

High Performance SQL Server Workloads on Hyper-V

Performance Guidance for SQL Server in Windows Azure Virtual Machines

Loading Bulk Data into a Partitioned Table

Running SQL Server 2008 in a Hyper-V Environment - Best Practices and Performance Recommendations

Running SQL Server with Hyper-V Dynamic Memory - Best Practices and Considerations

TEMPDB Capacity Planning and Concurrency Considerations for Index Create and Rebuild

The Data Loading Performance Guide 2008

Troubleshooting Performance Problems in SQL Server 2005

Troubleshooting Performance Problems in SQL Server 2008

Tuning the Performance of Change Data Capture in SQL Server 2008

Diagnosing and Resolving Latch Contention on SQL Server

Diagnosing and Resolving Spinlock Contention on SQL Server

 

Service Broker

Service Broker: Performance and Scalability Techniques

 

Full Text

SQL Server 2005 Full-Text Queries on Large Catalogs: Lessons Learned

 

Analysis Services

Accelerating Microsoft adCenter with Microsoft SQL Server 2008 Analysis Services

REAL PRACTICES: Performance Scaling Microsoft SQL Server 2008 Analysis Services at Microsoft adCenter

Resolving Common Connectivity Issues in SQL Server 2005 Analysis Services Connectivity Scenarios

Scale-Out Querying with Analysis Services

Scale-Out Querying with Analysis Services Using SAN Snapshots

The Analysis Services 2008 Performance Guide

Analysis Services 2005 Performance Guide

Analysis Services Distinct Count Optimization

Analysis Services Processing Best Practices

Analysis Services ROLAP for SQL Server Data Warehouses

Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services

OLAP Design Best Practices for Analysis Services 2005

Precision Considerations for Analysis Services Users

Analysis Services Many-to-Many Dimensions: Query Performance Optimization Techniques

Scale-Out Querying for Analysis Services with Read-Only Databases

SQL Server 2008R2 Analysis Services Operations Guide

Analysis Services 2008 R2 Performance Guide

 

SQL Server Integration Services

Using SQL Server Integration Services and StreamInsight Together 

SSIS Tips Tricks and Best Practices: SSIS for Azure and Hybrid Data Movement

SSIS Tips Tricks and Best Practices: SSIS Operational and Tuning Guide

 

SQL General Administration

Database Upgrade to SQL Server 2008 Tools and Approaches

DBCC SHOWCONTIG Improvements in SQL Server 2005 and comparisons to SQL Server 2000

Disk Partition Alignment Best Practices for SQL Server

Enterprise Policy Management Framework with SQL Server 2008

 

 SQL Server Security and Audit

Reaching Compliance: SQL Server 2008 Compliance Guide

Manage Kerberos Authentication Issues in a Reporting Services Environment

 

SQL Server Application, Planning and Features

Best Practices for Migrating Non-Unicode Data Types to Unicode

Improving Microsoft CRM Performance and Securing Data with SQL Server 2008

New Spatial Features in SQL Server Code-Named “Denali” Community Technology Preview 3

Performance Optimizations for the XML Data Type in SQL Server 2005

SAP with Microsoft SQL Server 2005: Best Practices for High Availability, Performance, and Scalability

SQL Server 2005 Deployment Guidance for Web Hosting Environments

SQL Server 2008 with Siebel CRM Applications"

SQL Server 2008 with Siebel, PeopleSoft and JD Edwards Applications

SQL Server Consolidation Guidance

SQL Server Best Practices – Implementation of Database Object Schemas

ComparingTables Organized with Clustered Indexes versus Heaps

DataCompression: Strategy, Capacity Planning and Best Practices

FILESTREAMDesign and Implementation Considerations

Implementinga SQL Server Fast Track Data Warehouse

MicrosoftSQL Server 2005 Tuning Tips for PeopleSoft8.x 

The Impact of Changing Collations and of Changing Data Types from Non-Unicode to Unicode

Using SQL Server PowerPivot for Excel to Analyze SAP Data

XML Support in Microsoft SQL Server 2005

BizTalk Server 2009 Hyper-V Guide

SQL Server 2008 with Siebel CRM Applications

Working with the New SQL Server 2008 Collations and Earlier Versions of the Native Data Provider

Microsoft SQL Server 2008 R2 PowerPivot Planning and Deployment

New Spatial Features in SQL Server Code-Named “Denali” Community Technology Preview 1

Onboarding SQL Server Private Cloud Environment

 

SharePoint

Configure a Sharepoint Server Farm for minimal downtime during software updates

Configure disaster recovery across SharePoint farms by using SQL Server log shipping

Configuring Kerberos Authentication for Microsoft SharePoint 2010 Products

PowerPivot for SharePoint - Existing Farm Installation

PowerPivot for SharePoint - Transitioning from a Single Server to SharePoint Farm

PowerPivot for SharePoint: Single Server Installation

PowerPivot Technical Diagram: PowerPivot Client/Server Architecture

PowerPivot Technical Diagram: PowerPivot Security Architecture

Edgenet Realizes the Power of Dynamic IT and Self-Service BI

  

Reporting Services

Develop Reporting Services reports using Analysis Services data; a SQL Server 2008 technical case study

 


SQLCAT.Com - eBook Downloads

$
0
0

As of September 1, 2013 we decided to remove SQLCAT.COM site and use MSDN as the primary vehicle to post new SQL Server content. This was done to minimize reader confusion and to streamline content publication.  MSDN SQLCAT blogs already includes most SQLCAT.COM Content and will continue to be updated with more SQLCAT learnings.  You can also find a collection of our work in SQLCAT Guidance eBooks.  

To make the transition to MSDN smoother, we are in the process of reposting a few of SQLCAT.COM’s blogs that are still being very actively viewed .  We have also generated a collection of ebooks including all the technical notes oringally posted on SQLCAT.Com.

You can find these books here:

SQLCAT's Guide to BI and Analytics

SQLCAT's Guide to High Availability and Disaster Recovery

SQLCAT's Guide to Relational Engine

Below you can find a list of the articles included in each of these ebooks.

Also, Follow us on Twitter as we normally use our Twitter handles @SQLCAT and @WinAzureCAT to announce news and new content.

 

SQLCAT's Guide to BI and Analytics

Section 1: Administration 

 

Running Microsoft SQL Server 2008 Analysis Services on Windows Server 2008 vs. Windows Server 2003 and Memory Preallocation: Lessons Learned 4
Scripts for Stress Testing Analysis Services using ASCMD 11
A Solution for Collecting Analysis Services Performance Data From Many Sources For Performance Analysis 12
Compress Many-to-Many C# Utility 13
Create a Processing Log Script Project 14
Powershell Scripts for SQL Server 2008 Analysis Services DMVs 15
Sample Robocopy Script to customer synchronize Analysis Services databases 16


Section 2: Analysis Services Performance 20

 

Analysis Services Synchronization Best Practices 21
Microsoft SQL Server 2008 Analysis Services Consolidation Best Practices 26
Analysis Services Distinct Count Optimization Using Solid State Devices 43
Excel, Cube Formulas, Analysis Services, Performance, Network Latency, and Connection Strings 55
Reintroducing Usage-Based Optimization in SQL Server 2008 Analysis Services 63


Sections 3: Analysis Services Scaleout 70

 

Analysis Services Load Balancing Solution 71
SSAS Monitoring Scripts For Management Data Warehouse 72
ASLB Toolkit 73


Section 4: Real World Scenarios 75

 

Edgenet Realizes the Power of Dynamic IT and Self-Service BI 76


Section 5: Reporting Services 83

 

Reporting Services Scale-Out Deployment Best Practices 85
Report Server Catalog Best Practices 94
Reporting Services Performance Optimizations 100
Reporting Services Scale-Out Architecture 108
SQL Server Reporting Services Disaster Recovery Case Study 114
Scaling Up Reporting Services 2008 vs. Reporting Services 2005: Lessons Learned 123
Reporting Services Performance in SharePoint Integrated Mode in SQL Server 2008 R2 137
Deploying a Business Intelligence Solution Using SharePoint 2007, SQL Server 2008 Reporting Services, and PerformancePoint Monitoring Server 2007 with Kerberos 147


Section 6: SQL Server Integration Services 157

 

Increasing Throughput of Pipelines by Splitting Synchronous Transformations into Multiple Tasks 158
Moving Large Amounts of Data Between Oracle and SQL Server: Findings and Observations 164
SSIS Package For Collecting SSAS DMV Data 168
The “Balanced Data Distributor” for SSIS 169


Section 7: SQL Top 10 172

 

Top 10 Performance and Productivity Reasons to Use SQL Server 2008 for Your Business Intelligence Solutions 173
Top 10 SQL Server Integration Services Best Practices 185
Analysis Services Query Performance Top 10 Best Practices 195

 

 

SQLCAT's Guide to High Availability and Disaster Recovery

 

Section 1: SQL Server Failover Cluster 2

Impact of Adding a Node to a Windows Cluster on the Possible Owners property of existing SQL Server Failover Cluster Instances 3
Six Failover Clustering Benefits Realized from Migrating to SQL Server 2008 9

 

Section 2: SQL Server Always On 12

DO NOT use Windows Failover Cluster Manager to perform Availability Group Failover 13
Comparing I/O characteristics in AlwaysOn Availability Groups and Database Mirroring 19

 

Section 3: SQL Server Mirroring 23

Mirroring a Large Number of Databases in a Single SQL Server Instance 24
Asynchronous Database Mirroring with Log Compression in SQL Server 2008 29

 

 

SQLCAT's Guide to Relational Engine

Section 1: Administration 3

DBCC Checks and Terabyte-Scale Databases 4
Scheduling Sub-Minute Log Shipping in SQL Server 2008 10
Tuning Backup Compression Part 2 13
Restart SQL Audit Policy and Job 23
SQL DMVStats Toolkit 24

Section 2: Database Design 25

SQL Server Partition Management Tool 26
Character data types versus number data types: are there any performance benefits? 27
The Many Benefits of Money…Data Type! 36
How many files should a database have? - Part 1: OLAP workloads 41

Section 3: Fast-track 45

Lessons Learned and Findings from a Large Fast-Track POC 46

Section 4: Performance 67

Top Tips for Maximizing the Performance & Scalability of Dynamics AX 2009 systems on SQL Server 2008 68
Top SQL Server 2005 Performance Issues for OLTP Applications 78
Table-Valued Functions and tempdb Contention 80
Resolving PAGELATCH Contention on Highly Concurrent INSERT Workloads 95
SQL Server Indexing: Using a Low-Selectivity BIT Column First Can Be the Best Strategy 99
Tuning the Performance of Backup Compression in SQL Server 2008 115
Maximizing Throughput with TVPs 129
Bulk Loading Data into a Table with Concurrent Queries 135

Section 5: Real World Scenarios 140

Lessons Learned from Benchmarking a Tier 1 Core Banking ISV Solution - Temenos T24 141

Section 6: Replication 149

Initializing a Transactional Replication Subscriber from an Array-Based Snapshot 150
Upgrading Replication from SQL Server 2000 32-Bit to SQL Server 2008 64-Bit without re-initialization 161

Section 7: Service Broker 162

SQL Server Service Broker: Maintaining Identity Uniqueness Across Database Copies 163

Section 8: Troubleshooting 166

Diagnosing Transaction Log Performance Issues and Limits of the Log Manager 167
Eliminating Deadlocks Caused By Foreign Keys with Large Transactions 174
Resolving scheduler contention for concurrent BULK INSERT 181
Response Time Analysis using Extended Events 185
Memory Error Recovery in SQL Server 2012 186

Section 9: SQL Top 10 189

Top 10 Hidden Gems in SQL 2008 R2 190
Top 10 SQL Server 2008 Features for the Database Administrator (DBA) 192
Top 10 SQL Server 2008 Features for ISV Applications 204
Top 10 Hidden Gems in SQL Server 2005 213
Top 10 Best Practices for Building a Large Scale Relational Data Warehouse 221
Storage Top 10 Best Practices 225
Top 10 Best Practices for SQL Server Maintenance for SAP 228

 

 

SQL 2014 Clustered Columnstore index rebuild and maintenance considerations

$
0
0

This article describes the index rebuild process as well as index maintenance for clustered Columnstore indexes and is directed towards SQL 2014. In SQL 2016, there have been several index maintenance enhancements that will be covered in a separate post.

 

Overview of Columnstore index build or rebuild process

Building (or rebuilding) a Columnstore index can be a resource intensive operation. Index creation time can be 1.5 times longer than a regular b-tree and resource consumption as well as duration depends on a few factors other than physical resources that include

  • Number and data types of columns which determine the memory grant estimation
  • Degree of Parallelism (DOP)
  • Resource Governor settings

 

Plan for Index Rebuild of a Non-Partitioned Table with 6 billion rows:

image

Plan for Index Rebuild of a partitioned table with 6 billion rows spread across 22 partitions

image

The Columnstore index is built in 2 phases indicated by the plans above:

  • Primary (Global) Dictionary: This is built first and is a serial plan irrespective of MAXDOP settings and completed first and persisted. In order to build the primary dictionary we sample 1% of the rows if the table is over a million rows. Given this is a serial plan the duration taken for this step is incurred in all cases. The memory grant here is limited to 10%.
  • Segments and Local Dictionaries: Segments are now built in parallel as are local dictionaries. First an estimate is made on a per-thread basis of memory consumption based on a segment size of 1 million rows. The memory grant is requested and all the threads are started but only the first one builds a segment and once it is done the actual memory grant is known per thread. While this first segment is being built, all other threads wait on it to finish and have the wait type COLUMNSTORE_BUILD_THROTTLE. Given the COLUMNSTORE_INDEX_BUILD_THROTTLE is cumulative as all remainder of the threads wait for the first segment to be built, the higher the DOP on the system, the higher you will see this value. AS you can see in the XE capture below, you see the first segment build completing and then the waits for all the remainder of the threads completing.

 

clip_image006[7]

 

From the first segment build, we know how much memory was needed. Based on this knowledge we activate only N number of threads whose total memory grant will fit into the memory that was granted earlier. This number of threads activated is represented in the plan as “Effective Degree of Parallelism” and can be checked against the actual degree of parallelism as seen in the diagram.

image

 

The Memory Grant information is also available from the Query Plan.

image

 

During the life of the index build, additional memory can be granted within resource governor limits and low memory conditions are checked as well. If a Low memory condition is detected, the segment will be trimmed before it reaches the 1 million row mark.

A segment can be trimmed or closed before the 1 million mark from an index build perspective if

  • Low Memory condition is hit
  • Dictionary is full ( Dictionary size is 16MB)
  • DOP is greater than 1 and the last “N” row groups created don’t have 1 million rows

 

Columnstore Index Build Extended Events

There are a couple Extended Events that can help in diagnosing index build and segment quality related issues.

a. Column_store_index_build_throttle – indicates if the DOP has been throttled.

image

b. Column_store_index_build_low_memory – tells us if a segment is trimmed due to low memory condition

image

c. Column_store_index_build_process_segment – gives us the segment that was trimmed and the trim_reason. This list isn’t exhaustive, 1 = Low Memory, 2 = Dictionary full

image

Columnstore Index build test cases:

The following table depicts Index build results on a non-partitioned table with 6 billion rows. While observing the numbers we can see that the higher the DOP the higher the CPU. Also the higher the DOP, given more threads are spawned, the memory grant goes up.

Note: There wasn’t a noticeable difference when the same tests were performed on a partitioned table. The benefits of having Columnstore index on a partitioned table include being able to rebuild an index at the partition level, partition elimination in conjunction with segment elimination and ability to archive data efficiently.

This is a 6 billion row table that was tested, SQL Server has 60 cores.

StartTime

EndTime

Duration

MAXDOP

Actual DOP

max_grant_kb

CPU

4/10/15 8:02 AM

4/10/15 8:27 AM

0:24:50

64

60

28989608

85%

4/10/15 8:28 AM

4/10/15 9:07 AM

0:39:40

32

32

15461128

50%

4/10/15 9:08 AM

4/10/15 10:08 AM

1:00:13

16

16

7730568

28%

4/10/15 10:09 AM

4/10/15 11:53 AM

1:43:59

8

8

3865288

13%

       

image

If you look at the perfmon charts below and the timings, Rebuild index for a Columnstore index does not scale linearly. This is because the Global dictionary is built in serial as discussed, and is a constant irrespective of MAXDOP for the index build. As you see in the perfmon images below, the dictionary build time takes a larger percentage of the overall time the higher the degree of parallelism.

MAXDOP 32: Building the Global dictionary takes around 20% of the total index build time with MAXDOP 32.

image

MAXDOP 64: Building the Global dictionary takes around 29% of the total index build time when maxdop is 64.

image

 

Columnstore Index Maintenance

There are 2 operations from a Columnstore Index maintenance perspective

INDEX REORGANIZE: This manually moves closed ROWGROUPS into compressed columnar storage and this is done online. You do not have to do this, the tuple mover will ultimately move a closed rowgroup into columnar storage but the tuple mover is single threaded so issuing an index reorganize is a way to manually invoke compression on a closed row group.

INDEX REBUILD: This can be done at a partition Level for a partitioned table, and is an offline index build and at this point there isn’t an ONLINE equivalent. The Rebuild reads and recompresses all the data in the specified partition or the entire index. In an updatable Columnstore, deletes of data that reside in a compressed rowgroup are logical deletes. You can encounter situations where majority of the data in a rowgroup could be deleted. In such cases in order to reclaim that space, you have to REBUILD the index for the partition in question. Another reason one would want to rebuild an index on the partition is to improve rowgroup quality if you have a large number of rowgroups each having a small number of rows. In this case rebuilding an index can consolidate smaller row groups into larger ones which could help both from the compression perspective as well as from the query performance perspective.

 

Here is a sample script which can help identify partitions that are good candidates to be rebuilt based on some thresholds defined in the script.

 

/*---------------------------------------------------------------------------------
The sample scripts are not supported under any Microsoft standard support program or service
and are intented as a supplement to online documentation.The sample scripts are provided AS IS without warranty
of any kind either expressed or implied. Microsoft further disclaims all implied warranties including,
without limitation, any implied warranties of merchantability or of fitness for a particular purpose.
#--------------------------------------------------------------------------------- */

 /*
 Rebuild index statement is printed at partition level if
  a. RGQualityMeasure is not met for @PercentageRGQualityPassed Rowgroups
    -- this is an arbitrary number, what we are saying is that if the average is above this number, don't bother rebuilding as we consider this number to be good quality rowgroups
 b. Second constraint is the Deleted rows, currently the default that is set am setting is 10% of the partition itself. If the partition is very large or small consider adjusting this
 c. In SQL 2014, post index rebuild,the dmv doesn't show why the RG is trimmed to < 1 million in this case in SQL 2014.
   - If the Dictionary is full ( 16MB) then no use in rebuilding this rowgroup as even after rebuild it may get trimmed
   - If dictionary is full only rebuild if deleted rows falls above the threshold
  */

 if object_id('tempdb..#temp') IS NOT NULL
 drop table #temp
 go

 Declare @DeletedRowsPercent Decimal(5,2)
 -- Debug = 1 if you need all rowgroup information regardless
 Declare @Debug int =0
 -- Percent of deleted rows for the partition
 Set @DeletedRowsPercent = 10
 -- RGQuality means we are saying anything over 500K compressed is good row group quality, anything less need to re-evaluate.
 Declare @RGQuality int = 500000
 -- means 50% of rowgroups are < @RGQUality from the rows/rowgroup perspective 
 Declare @PercentageRGQualityPassed smallint = 20
 ;WITH CSAnalysis
 ( object_id,TableName,index_id,partition_number,CountRGs,TotalRows,
  AvgRowsPerRG,CountRGLessThanQualityMeasure,RGQualityMeasure,PercentageRGLessThanQualityMeasure
  ,DeletedRowsPercent,NumRowgroupsWithDeletedRows)
 AS
 (SELECT object_id,object_name(object_id) as TableName, index_id,
  rg.partition_number,count(*) as CountRGs, sum(total_rows) as TotalRows, Avg(total_rows) as AvgRowsPerRG,
  SUM(CASE WHEN rg.Total_Rows <@RGQuality THEN 1 ELSE 0 END) as CountRGLessThanQualityMeasure, @RGQuality as RGQualityMeasure,
  cast((SUM(CASE WHEN rg.Total_Rows <@RGQuality THEN 1.0 ELSE 0 END)/count(*) *100)  as Decimal(5,2))  as PercentageRGLessThanQualityMeasure,
  Sum(rg.deleted_rows * 1.0)/sum(rg.total_rows *1.0) *100 as 'DeletedRowsPercent',
  sum (case when rg.deleted_rows >0 then 1 else 0 end ) as 'NumRowgroupsWithDeletedRows'
  FROM  sys.column_store_row_groups rg
  where rg.state = 3
  group by rg.object_id, rg.partition_number,index_id
),
CSDictionaries  --(maxdictionarysize int,maxdictentrycount int,[object_id] int, partition_number int)
 AS
 (   select max(dict.on_disk_size) as maxdictionarysize, max(dict.entry_count) as maxdictionaryentrycount
  ,max(partition_number) as maxpartition_number,part.object_id, part.partition_number
  from sys.column_store_dictionaries dict
  join sys.partitions part on dict.hobt_id = part.hobt_id
  group by part.object_id, part.partition_number
)
 select a.*,b.maxdictionarysize,b.maxdictionaryentrycount,maxpartition_number
 into #temp from CSAnalysis a
 inner join CSDictionaries b
 on a.object_id = b.object_id and a.partition_number = b.partition_number


-- Maxdop Hint optionally added to ensure we don't spread small amount of rows accross many threads
-- IF we do that, we may end up with smaller rowgroups anyways.
 declare @maxdophint smallint, @effectivedop smallint
 -- True if running from the same context that will run the rebuild index.
 select @effectivedop=effective_max_dop from sys.dm_resource_governor_workload_groups
 where group_id in (select group_id from sys.dm_exec_requests where session_id = @@spid)

 -- Get the Alter Index Statements.
  select 'Alter INDEX ' + QuoteName(IndexName) + ' ON ' + QuoteName(TableName) + '  REBUILD ' +
 Case
 when maxpartition_number = 1 THEN ' '
 else  ' PARTITION = ' + cast(partition_number as varchar(10))
 End
  + ' WITH (MAXDOP ='  + cast((Case  WHEN (TotalRows*1.0/1048576) < 1.0 THEN 1 WHEN (TotalRows*1.0/1048576) < @effectivedop THEN  FLOOR(TotalRows*1.0/1048576) ELSE 0 END) as varchar(10)) + ')'
 as Command
 from #temp a
 inner join
 ( select object_id,index_id,Name as IndexName from sys.indexes
    where type in ( 5,6) -- non clustered columnstore and clustered columnstore
 ) as b
on b.object_id = a.object_id and a.index_id = b.index_id
where ( DeletedRowsPercent >= @DeletedRowsPercent)
-- Rowgroup Quality trigger, percentage less than rowgroup quality as long as dictionary is not full
 OR ( ( ( AvgRowsPerRG < @RGQuality and TotalRows > @RGQuality) AND PercentageRGLessThanQualityMeasure>= @PercentageRGQualityPassed)
  AND maxdictionarysize < ( 16*1000*1000)) -- DictionaryNotFull, lower threshold than 16MB.
 order by TableName,a.index_id,a.partition_number

-- Debug print if needed
if @Debug=1
  Select getdate() as DiagnosticsRunTime,* from #temp
  order by TableName,index_id,partition_number
else
  Select getdate() as DiagnosticsRunTime,* from #temp
  -- Deleted rows trigger
  where ( DeletedRowsPercent >= @DeletedRowsPercent)
  -- Rowgroup Quality trigger, percentage less than rowgroup quality as long as dictionary is not full
  OR ( ( ( AvgRowsPerRG < @RGQuality and TotalRows > @RGQuality) AND PercentageRGLessThanQualityMeasure>= @PercentageRGQualityPassed)
  AND maxdictionarysize < ( 16*1000*1000)) -- DictionaryNotFull, lower threshold than 16MB.
  order by TableName,index_id,partition_number
-- Add logic to actually run those statements


 

Summarizing some of the best practices:

  • Partitioning enables rebuild of an index at the partition level as well as dictionaries created for each partition besides the other manageability benefits.
  • MAXDOP influences the memory grant size. If segments are getting trimmed due to low memory, reducing MAXDOP can help.
  • Resource Govenor settings for the respective pool can be increased from the default of 25% when rebuilding indexes if low memory causes rowgroups to be trimmed.
  • Massive deletions of rows that are already compressed in columnar format require a REBUILD of the index to reclaim that space.

 

 

Denzil Ribeiro

Program Manager SQL/Azure CAT

Choosing hash distributed table vs. round-robin distributed table in Azure SQL DW Service

$
0
0

This topic explains the various Azure SQL Data Warehouse distributed table types, and offers guidance for choosing the type of distributed table to use and when. There are two types of distributed tables in Azure SQL DW at the writing of this article, hash distributed table and round-robin distributed table.

Designing databases to use these distributed tables effectively will help you to achieve the storage and query processing benefits of the Azure SQL DW Service (SQL DW).

In SQL DW a distribution is an Azure SQL database, in which one or more distributed tables are stored. Each instance of SQL DW has many distributions. Many distributions can reside in a single instance of Azure SQL instance.

The amount of distributions are subject to change and not important for understanding this particular topic.

Hash Distributed Table Basics

A hash distributed table is a table whose rows are dispersed across multiple distributions based on a hash function applied to a column. Each SQL instance contains a group of one or more rows. The following diagram depicts how table within SQL DW gets stored as a hash distributed table.

clip_image002

When processing queries involving distributed tables, SQL DW instances execute multiple internal queries, in parallel, within each SQL instance, one per distribution.  These separate processes (independent internal SQL queries) are executed to handle different distributions during query and load processing.

A distribution column is a single column (specified at table creation time) that SQL DW uses to assign each row to a distribution. A deterministic hash function uses the value in the distribution column to assign each row to belong to one and only one distribution. Two identical column values with the same data type will be hashed the same and thus will end up in the same distribution.

In the diagram, each row in the original file is stored on one distribution. The number of rows in each distribution can vary and is usually not identical from distribution to distribution.

There are performance considerations for the selection of a distribution column, such as minimizing data skew, minimizing data movement, and the types of queries executed on the system. For example, query performance improves when two distributed tables are joined on a column that is of the same data type and size. This is called a distribution compatible join or a co-located join.

Round-Robin Distributed Table Basics

A round-robin distributed table is a table where the data is evenly (or as evenly as possible) distributed among all the distributions without the use of a hash function. A row in a round-robin distributed table is non-deterministic and can end up in different distributions each time they are inserted.

Each JOIN to a round-robin distributed table is a data movement operation. The data movement needed to perform join operations is a separate topic and will be published as a separate blog soon.

Usually common dimension tables or tables that doesn’t distribute evenly are good candidates for round-robin distributed table.

The following diagram shows a round-robin distributed table that is stored on different distribution.

clip_image002[7]

Best Practices

In SQL DW, a user query is a logical query that gets divided into many physical queries one for each distribution. The Engine Service on the control node acts as a coordinator and waits for each of these individual queries to finish before returning results or the next part of the multi-step query is executed.

When creating a table in SQL DW, you need to decide if the table will be hash distributed or round-robin distributed. This decision has implications for query performance. Each of these distributed tables may require data movement during query processing when joined together. Data movement in MPP RDBMS system is an expensive but sometimes unavoidable step. In my 8 plus years of working with MPP data warehouse I haven’t seen a real customer workload that can completely eliminate data movement. The objective of a good data warehouse design in SQL DW is to minimize data movement so let’s keep that in mind while choosing table design.

Here are considerations for choosing to use a round-robin distributed table or a hash distributed table:

1. To choose a good distribution design with SQL DW, one should know their data, DDL and queries. This is not unique to SQL DW but for most MPP RDBMS system. You need to minimize data movement queries but also watch out for data that can heavily skew a certain distribution. If one of the distribution has more data than others, it will be the slowest performing distribution. Since SQL DW queries are as fast as its slowest distribution, we need to take notes of any data-heavy (skewed or hot) distribution for the same table.

2. A nullable column is a bad candidate for any hash distributed table. All null columns are hashed the same and thus the rows will end up on the same distribution creating a skewed (hot) distribution. If most of the columns are null able and no good hash distribution can be achieved, that table is a good candidate for round-robin distribution. Choose ‘not null’ columns when creating table that will be hash distributed.

3. Any fact tables that has a default value in a column is also not a good candidate to create a hash distributed table. DW Developers will sometime assign -1 value to an otherwise unknown value or early arriving values for a fact table. These values will create data skew on a particular distribution. Avoid these kind of default value column unless you know for sure that the -1 values are negligible in your data.

4. Large fact tables or historical transaction tables are usually stored as hash distributed tables. These tables usually have a surrogate key that is monotonically increasing and are used in JOIN conditions with other fact and dimension tables. These surrogate keys are a good candidate for distributing the data as there are many unique values in that column. This allows the query operations to be performed across all distributions. Each distribution can work independently on separate subsets of data. This takes advantage of the processing resources across the MPP system. Queries on distributed tables may require data movement between distributions during query execution and that is okay.

5. Dimension tables or other lookup tables in a schema can usually be stored as round-robin tables. Usually these tables connects to more than one fact tables and optimizing for one join may not be the best idea. Also usually dimension tables are smaller which can leave some distributions empty when hash distributed. Round-robin by definition guarantees a uniform data distribution.

6. If you are unsure of query patterns and data, you can start with all tables in round-robin distribution. And as you learn the patterns the data can be easily redistributed on a hash key.

7. When using ‘group by’ SQL DW will shuffle the data on the group by key. When multiple keys are present and statistics is up-to-date SQL DW’s cost based optimizer will pick the right key to shuffle the data. If this group by key is heavily non-unique then the query will be slower. A worst case example would be grouping by gender of a large customer table. If your query is running slower, look into explain plan (add the word ‘explain’ before your query and execute) to find out what key is being used as the shuffle key. There may or may not be anything you can do to change this based on the query.

SQL 2016: Columnstore row group Merge policy and index maintenance improvements

$
0
0

            

   A Columnstore index contains row groups that can that have a maximum of 1,048,576 rows. A row group can be closed and compressed due to multiple reasons before that maximum of 1,048,576 rows is reached. Ideally we want that row count in each row group to be as close to the maximum as possible in order to get both better compression and in turn better segment elimination as fewer segments would have to be scanned. In SQL Server 2014, if a row group was compressed before it reached the maximum of 1,048,576 rows, there wasn’t a way to see the reason why it was compressed early before it even hit the maximum.

For a more in-depth discussion of why row groups could be compressed before reaching the maximum row limit (“trim reason”), see the post Data Loading performance considerations with Clustered Columnstore indexes

In SQL Server 2016 we expose the row group trim reason via the Dynamic Management View (DMV) sys.dm_db_column_store_row_group_physical_stats as shown below.

image

The reasons for trim for a clustered Columnstore index include:

 

Trim_reason_desc

Trim Reason

UNKNOWN_UPGRADED_FROM_PREVIOUS_  VERSION

The reason wasn’t provided prior to SQL Server 2016, so it is also not captured for upgraded data.

NO_TRIM

Not trimmed.  The row group has 1,048,576 rows.

BULKLOAD

BATCHSIZE specified for bulk insert, or end of bulk insert.

REORG_FORCED

REORG with COMPRESS_ALL_ROWGROUPS = ON which closes every open row group and compresses it into columnar format

DICTIONARY_SIZE   

If Dictionary is full, the row group will be trimmed ( 16MB dictionary)

MEMORY_LIMITATION

Memory pressure during index build caused row group to be trimmed

RESIDUAL_ROW_GROUP_INDEXBUILD

Last row group(s) have less than 1 million rows when index rebuilt.

 

Prior to SQL Server 2016, if we wanted to coalesce smaller compressed row groups into larger row groups or if we wanted to reclaim space due to deleted rows, we would have to rebuild the index. This offline index rebuild could be done at the table or the partition level.

For a more in-depth discussion of index rebuild and maintenance in SQL Server 2014 see SQL 2014 Clustered Columnstore index rebuild and maintenance considerations

Consider the two trimmed row group scenarios below:

A. Multiple row groups exist that are compressed and have less than the 1,048,576 row maximum. Here you see the trim reason as BULKLOAD, which means that there was an insert that was greater than 102,400 rows that directly moves that data into compressed row groups.

Looking at the sys.dm_db_column_store_row_group_physical_stats output below, the current state of rows-per row group could be due to multiple sessions (threads) each inserting 125,000 rows or alternatively it could be due to a single INSERT…SELECT which used a parallel plan, with each thread having inserted 125,000 rows.

image

B. The Columnstore index sys.dm_db_column_store_row_group_physical_stats output below shows several row groups with a significant number of rows deleted. In this scenario, the deleted rows in a compressed segment are logical deletes and not physical deletes, so in order to reclaim space, you have to rebuild the index in versions prior to SQL 2016

image

 

SQL Server 2016 Merge Policy

SQL Server 2016 introduces the ability to “merge” smaller, eligible row groups into larger row groups. This is achieved by running INDEX REORGANIZE against a Columnstore index in SQL Server 2016

INDEX REORGANIZE

  • Moves closed row groups into compressed columnar format
  • Merges multiple row groups into larger row groups that fit within the maximum row group size
  • Is an online operation unlike a rebuild index which is offline

A row group is “eligible” to be merged if it means any of the following conditions:

  • The row group is compressed
  • The row group has 10% or more rows deleted
  • The row group was NOT trimmed due to a full dictionary

Here are a couple examples of merge policy at work

a. Single Rowgroup (Self) -merge for deleted rows (reclaiming of space): In the most simplistic case, you don’t even need multiple row groups to be involved in the merge. For example, if a single row group has more than 10% of its rows logically deleted, then it qualifies for a “self-merge” (space is reclaimed by removing the logically deleted rows from the single row group). The below table shows two examples of single-row groups – one that is eligible for self-merge and one that is not.

Row group Size

Deleted Rows

Self-Merge

400,000

120,000

Yes, deleted rows > 10%

150,000

5,000

No

 

b. Merge across multiple row groups: In the example below, only two row groups are being depicted, but more than 2 row groups can be candidates to be merged as will be shown later in this article.

 

Row group 1 row count

Row group 2 row count

Eligible for Merge?

950,000

920,000

No, because row group 1 combined with row group 2 exceed the row maximum per row group

400,000

500,000

Yes, because row group 1 combined with row group 2 result in 900,000 rows (under the 1,048,576 row maximum)

1,000,000

( 200,000 deleted)

500,000

( 300,000 deleted)

Yes, subtracting out the deleted rows, the combined row count across row groups is under the row maximum

 

Tracking Merge Activity:

You can track the merge activity and row group qualification using the Extended Events columnstore_rowgroup_merge_start and columnstore_rowgroup_merge_end events as below:

CREATE EVENT SESSION [TupleMover] ON SERVER

ADD EVENT sqlserver.columnstore_no_rowgroup_qualified_for_merge,
ADD EVENT sqlserver.columnstore_rowgroup_compressed,
ADD EVENT sqlserver.columnstore_rowgroup_merge_complete,
ADD EVENT sqlserver.columnstore_rowgroup_merge_start
ADD TARGET package0.event_file(SET filename=N'XeMerge',max_file_size=(10))
GO
Alter EVENT SESSION [XeMerge] ON SERVER  State = START

 


Now consider the following INDEX REORGANIZE scenario against an existing Columnstore index:

-- Invoke the REORG command to fire Merge and remove deleted rows
-- and coalesce the smaller rowgroups into a larger rowgroup.
alter index cci_temp on FactResellerSalesXL_CCI_temp reorganize

Looking at the output of the extended events, notice that across all the merged row groups there were 1,000,000 rows and 501,786 deleted rows.

image

Once the merge operation is done, the output of sys.dm_db_column_store_row_group_physical_stats shows one compressed rowgroup that contains the rows in all the prior merged row groups. The merged row groups will have a state description of TOMBSTONE momentarily until they are cleaned up.

image

In addition to the new sys.dm_db_column_store_row_group_physical_stats DMV, an operational stats DMV sys.dm_db_column_store_row_group_operational_stats has also been added to SQL Server 2016 which gives us visibility into the frequency of scans and lock contention for partitions and row groups (similar to sys.dm_db_index_operational_stats DMV that exists for regular indexes):

image

In short, the new Merge functionality in INDEX REORGANIZE simplifies Columnstore index maintenance significantly and other DMV’s add visibility into the internals of columnar indexes. There are several other supportability improvements on the Columnstore front not mentioned in this post several other Extended Events and Perfmon counters added to enable better troubleshooting both clustered Columnstore and updatable non-clustered Columnstore indexes

Denzil Ribeiro ( @denzilribeiro )

 

 

SQLCAT @PASS Summit 2015

$
0
0

Are you coming to the PASS Summit 2015 in Seattle? SQLCAT will be in full force at the PASS Summit 2015. We will also bring along our colleagues from the broader AzureCAT.

SQLCAT Sessions

SQLCAT sessions are unique. We bring in real customer stories and present their deployments, architectures, challenges and lessoned learned. This year at the PASS Summit, we will have 6 SQLCAT sessions – each session will have one or more customers co-presenting with SQLCAT members. If you look for real world experience, you don’t want to miss any of these sessions.

SQLCAT: Azure SQL Data Warehouse Lessons Learned and Customer Stories

SQLCAT: Azure SQL Database Customer Experiences and Lessons Learned

SQLCAT: Azure SQL VM Design Patterns and Customer Stories

SQLCAT: SQL Server 2016 Columnstore Customer Scenarios and Best Practices

SQLCAT: SQL Server 2016 Early Adopter Panel

SQLCAT: SQL Server HA DR Customer Panel — 7 Habits of people who build highly available systems

 

SQL Server Clinic

Have a technical question, a troubleshooting challenge, want to have an architecture discussion, or want to find best ways to upgrade your SQL Server – SQL Server Clinic is the place you want to be at. The Clinic is the hub of technical experts from SQLCAT, SQL product development team, CSS and Tiger team. Whether you want a facelift of your SQL Server system or an open heart surgery, the experts at the Clinic will have the right advice for you. Want to learn about great new capabilities in SQL Server 2016, Azure SQL Database, SQL Server in Azure VM or Azure SQL DW – find all your answers in one place.

 

And More …

That’s not all. SQLCAT will be involved in many more events and customer conversations during the Summit. If you have a suggestion on how we can make your experience at the PASS Summit more effective and more productive, don’t hesitate to leave a note.

Thanks, and see you all at the PASS Summit 2015 in Seattle. You are coming, right?

 

Azure SQL Data Warehouse loading patterns and strategies

$
0
0

Authors: John Hoang, Joe Sack and Martin Lee

Abstract

This article provides an overview of the Microsoft Azure SQL Data Warehouse architecture. This new platform-as-a service (PaaS) offering provides independent compute and storage scaling on demand and is currently in public preview. This document provides data loading guidelines for SQL Data Warehouse. Several common loading options are described, such as SSIS, BCP, Azure Data Factory (ADF), and SQLBulkCopy, but the main focus is the PolyBase technology, the preferred and fastest loading method for ingesting data into SQL Data Warehouse. See also What is Azure SQL Data Warehouse?

Introduction

Whether you are building a data mart or a data warehouse, the three fundamentals you must implement are an extraction process, a transformation process, and a loading process—also known as extract, transform, and load (ETL). When working with smaller workloads, the general rule from the perspective of performance and scalability is to perform transformations before loading the data. In the era of big data, however, as data sizes and volumes continue to increase, processes may encounter bottlenecks from difficult-to-scale integration and transformation layers.

As workloads grow, the design paradigm is shifting. Transformations are moving to the compute resource, and workloads are distributed across multiple compute resources. In the distributed world, we call this massively parallel processing (MPP), and the order of these processes differs. You may hear it described as ELT—you extract, load, and then transform as opposed to the traditional ETL order. The reason for this change is today’s highly scalable parallel computing powers, which put multiple compute resources at your disposal such as CPU (cores), RAM, networking, and storage, and you can distribute a workload across them.

With SQL Data Warehouse, you can scale out your compute resources as you need them on demand to maximize power and performance of your heavier workload processes.

However, we still need to load the data before we can transform. In this article, we’ll explore several loading techniques that help you reach maximum data-loading throughput and identify the scenarios that best suit each of these techniques.

Architecture

SQL Data Warehouse uses the same logical component architecture for the MPP system as the Microsoft Analytics Platform System (APS). APS is the on-premises MPP appliance previously known as the Parallel Data Warehouse (PDW).

As you can see in the diagram below, SQL Data Warehouse has two types of components, a Control node and a Compute node:

Figure 1. Control node and Compute nodes in the SQL Data Warehouse logical architecture

image

The Control node is the brain and orchestrator of the MPP engine. We connect to this area when using SQL Data Warehouse to manage and query data. When you send a SQL query to SQL Data Warehouse, the Control node processes that query and converts the code to what we call a DSQL plan, or Distributed SQL plan, based on the cost-based optimization engine. After the DSQL plan has been generated, for each subsequent step, the Control node sends the command to run in each of the compute resources.

The Compute nodes are the worker nodes. They run the commands given to them from the Control node. Compute usage is measured using SQL Data Warehouse Units (DWUs). A DWU, similar to the Azure SQL Database DTU, represents the power of the database engine as a blended measure of CPU, memory, and read and write rates. The smallest compute resource (DWU 100) consists of the Control node and one Compute node. As you scale out your compute resources (by adding DWUs), you increase the number of Compute nodes.

Within the Control node and in each of the Compute resources, the Data Movement Service (DMS) component handles the movement of data between nodes—whether between the Compute nodes themselves or from Compute nodes to the Control node.

DMS also includes the PolyBase technology. An HDFS bridge is implemented within the DMS to communicate with the HDFS file system. PolyBase for SQL Data Warehouse currently supports Microsoft Azure Storage Blob.

Network and data locality

The first considerations for loading data are source-data locality and network bandwidth, utilization, and predictability of the path to the SQL Data Warehouse destination. Depending on where the data originates, network bandwidth will play a major part in your loading performance. For source data residing on your premises, network throughput performance and predictability can be enhanced with a service such as Azure Express Route. Otherwise, you must consider the current average bandwidth, utilization, predictability, and maximum capabilities of your current public Internet-facing, source-to-destination route.

Note Express Route routes your data through a dedicated connection to Azure without passing through the public Internet. ExpressRoute connections offer more reliability, faster speeds, lower latencies, and higher security than typical Internet connections. For more information, see Express Route.

Using PolyBase for SQL Data Warehouse loads

SQL Data Warehouse supports many loading methods, including SSIS, BCP, the SQLBulkCopy API, and Azure Data Factory (ADF). These methods all share a common pattern for data ingestion. By comparison, the PolyBase technology uses a different approach that provides better performance.

PolyBase is by far the fastest and most scalable SQL Data Warehouse loading method to date, so we recommend it as your default loading mechanism. PolyBase is a scalable, query processing framework compatible with Transact-SQL that can be used to combine and bridge data across relational database management systems, Azure Blob Storage, and Hadoop database platform ecosystems (APS only).

Note As a general rule, we recommend making PolyBase your first choice for loading data into SQL Data Warehouse unless you can’t accommodate PolyBase-supported file formats. Currently PolyBase can load data from UTF-8 encoded delimited text files as well as the popular Hadoop file formats RC File, ORC, and Parquet. PolyBase can load data from gzip, zlib and Snappy compressed files. PolyBase does not support other character encodings, including UTF-16, extended ASCII, fixed-file format, and compression formats such as WinZip, JSON, and XML.

As the following architecture diagrams show, each HDFS bridge of the DMS service from every Compute node can connect to an external resource such as Azure Blob Storage, and then bidirectionally transfer data between SQL Data Warehouse and the external resource.

Note As of this writing, SQL Data Warehouse supports only Azure Blob Storage as the external data source.

 

Figure 2. Data transfers between SQL Data Warehouse and an external resource

image[13]

PolyBase data loading is not limited by the Control node, and so as you scale out your DWU, your data transfer throughput also increases. By mapping the external files as external tables in SQL Data Warehouse, the data files can be accessed using standard Transact-SQL commands—that is, the external tables can be referenced as standard tables in your Transact-SQL queries.

Copying data into storage

The general load process begins with migrating your data into Azure Blob Storage. Depending on your network’s capabilities, reliability, and utilization, you can use AZCOPY to upload your source data files to Azure Storage Blobs with an upload rate from 80 MB/second to 120 MB/second.

Then, in SQL Data Warehouse, you configure your credentials that will be used to access Azure Blob Storage:

CREATE DATABASE SCOPED CREDENTIAL myid_credential WITH IDENTITY = ‘myid’, Secret=’mysecretkey';

 

Next you define the external Azure Blob Storage data source with the previously created credential:

CREATE EXTERNAL DATA SOURCE data_1tb WITH (TYPE = HADOOP, LOCATION = ‘wasbs://data_1tb@myid.blob.core.windows.net’, CREDENTIAL= myid_credential);

 

And for the source data, define the file format and external table definition:

CREATE EXTERNAL FILE FORMAT pipedelimited

WITH (FORMAT_TYPE = DELIMITEDTEXT,

      FORMAT_OPTIONS(

          FIELD_TERMINATOR = ‘|’,

          STRING_DELIMITER = ”,

          DATE_FORMAT = ”,

          USE_TYPE_DEFAULT = False)

);

CREATE EXTERNAL TABLE orders_ext (

    o_orderkey bigint NULL,

    o_custkey bigint NULL,

    o_orderstatus char(1),

    o_totalprice decimal(15, 2) NULL,

    o_orderdate date NULL,

    o_orderpriority char(15),

    o_clerk char(15),

    o_shippriority int NULL,

    o_comment varchar(79)

)

WITH (LOCATION=’/orders’,

      DATA_SOURCE = data_1tb,

      FILE_FORMAT = pipedelimited,

      REJECT_TYPE = VALUE,

      REJECT_VALUE = 0

);

 

For more information about PolyBase, see SQL Data Warehouse documentation.

Using CTAS to load initial data

Then you can use a CTAS (CREATE TABLE AS SELECT) operation within SQL Data Warehouse to load the data from Azure Blob Storage to SQL Data Warehouse:

       CREATE TABLE orders_load

       WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = HASH(o_orderkey),

            PARTITION (o_orderdate RANGE RIGHT FOR VALUES (‘1992-01-01′,’1993-01-01′,’1994-01-01′,’1995-01-01′)))

        as select * from orders_ext;

 

CTAS creates a new table. We recommend using CTAS for the initial data load. This is an all-or-nothing operation with minimal logging.

 

Using INSERT INTO to load incremental data

For an incremental load, use INSERT INTO operation. This is a full logging operation but has minimal effect on the load performance. However, roll-back operation on a large transaction can be expensive. Consider breaking your transaction into smaller batches.

       INSERT INTO TABLE orders_load

       select * from orders_current_ext;

Note The source is using different external table, orders_current_ext.  This is the external table defining the path for the incremental data on ASB.

 

Data Reader, Writers consideration

SQL Data Warehouse adjusts the number of external move readers and writers as you scale. As illustrated in Table 1 below, each DWU 100 or node has maximum of 8 readers.  As you scale out, each node gets additional 8 readers.  The maximum number of readers is equal to the total number of distributions, which is currently 60.  For external writers, the number is always equal to the number of distributions regardless of your DWU scale. The number of readers is an important factor in determining your load performance.

Table 1. Number of readers and writers per DWU 100

Number of:

DWU

100

200

300

400

500

600

1,000

1,200

1,500

2,000

Readers

8

16

24

32

40

48

60

60

60

60

Writers

60

60

60

60

60

60

60

60

60

60

 

Best practices and considerations when using PolyBase

Here are a few more things to consider when using PolyBase for SQL Data Warehouse loads:

  • A single PolyBase load operation provides best performance.
  • The load performance scales as you increase DWUs.
  • PolyBase automatically parallelizes the data load process, so you don’t need to explicitly break the input data into multiple sources and issue concurrent loads, unlike some traditional loading practices.
  • Multiple readers will not work against compressed text files (e.g. gzip). Only a single reader is used per compressed file since uncompressing the file in the buffer is single threaded. Alternatively, generate multiple compressed files.  The number of files should be greater than or equal to the total number of readers. 
  • Multiple readers will also work against compressed columnar/block format files (e.g. ORC, RC) since individual blocks are compressed independently.

Known issues when working with different file formats

In addition to the UTF-8 encoding considerations, other known file format issues can arise when using PolyBase.

Mixed intra-file date formats

In a CREATE EXTERNAL FILE FORMAT command, the DATE_FORMAT argument specifies a single format to use for all date and time data in a delimited text file. If the DATE_FORMAT argument isn’t designated, the following default formats are used:

DateTime: ‘yyyy-MM-dd HH:mm:ss’

  • SmallDateTime: ‘yyyy-MM-dd HH:mm’
  • Date: ‘yyyy-MM-dd’
  • DateTime2: ‘yyyy-MM-dd HH:mm:ss’
  • DateTimeOffset: ‘yyyy-MM-dd HH:mm:ss’
  • Time: ‘HH:mm:ss’

For source formats that don’t reflect the defaults, you must explicitly specify a custom date format. However, if different formats are used within one file, there is currently no method for specifying multiple date formats within the PolyBase command.

Fixed-length file format not supported

Fixed-length character file formats—for example, where each column has a fixed width of 10 characters—are not supported today.

If you encounter the restrictions from using PolyBase, considers changing the data extract process to address those limitations.  This could be formatting the dates to PolyBase supported format, transforming the UTF-16 to UTF-8, etc.  If the option is not possible, then your option is to use any one of the methods in the next section.

Using Control-node and single-client gated load methods

In the Architecture section we mentioned that all incoming connections go through the Control node. Although you can increase and decrease the number of compute resources, there is only a single Control node. And as mentioned earlier, one reason why PolyBase provides a superior load rate is that PolyBase data transfer is not limited by the Control node. But if using PolyBase is not currently an option, the following technologies and methods can be used for loading into SQL Data Warehouse:

  • BCP
  • Bulk Insert
  • SSIS
  • SQLBulkCopy
  • Azure Data Factory (ADF)

Note ADF currently uses the same engine as SQLBulkCopy. A version of ADF using PolyBase engine is currently in Preview and will be available publicly by GA.

For these load methods, the bottleneck is on the client machine and the single Control node. Each load uses a single core on the client machine and only accesses the single Control node. Therefore, the load does not scale if you increase DWUs for an SQL Data Warehouse instance.

Note You can, however, increase load throughput if you add parallel loads into either the same table or different tables.

When connecting via a Control-node load method such as SSIS, the single point of entry constrains the maximum throughput you can achieve with a single connection.

Figure 3. Using SSIS, a Control-node load method, for SQL Data Warehouse loading

image

To further maximize throughput, you can run multiple loads in parallel as the following diagram shows:

Figure 4. Using SSIS (parallel loading) for SQL Data Warehouse loading

image

Using multiple client concurrent executions should improve your load throughput – to a point. The number of parallel loads no longer improves your throughput when the maximum capacity of the Control node is reached.

Best practices and considerations for single-client gated load methods

Consider the following when using SSIS, BCP, or other Control-node and client-gated loading methods:

  • Include retry logic—very important for slower methods such as BCP, SSIS, and SQLBulkCopy.
  • For SSIS, consider increasing the client/connection timeout from the default 30 seconds to 300 seconds. For more information about moving data to Azure, see SSIS for Azure and Hybrid Data Movement.
  • Don’t specify the batch size with Control-node gated methods. The goal is to load all or nothing so that the retry logic will restart the load. If you designate a batch size and the load encounters failure (for example, network or database not available), you may need to add more logic to restart from the last successful commit.

Comparing load method performance characteristics

The following table details the results of four separate Azure SQL Data Warehouse load tests using PolyBase, BCP, SQLBulkCopy/ADF, and SSIS:

Table 2. SQL Data Warehouse performance testing results

 

PolyBase

BCP

SQLBulkCopy/ADF

SSIS

Load Rate

FASTEST=================>>>>>>>>>>>>>>>>>>SLOWEST

Rate increase as you increase DWU

Yes

No

No

No

Rate increase as you add concurrent load

No

Yes

Yes

Yes

As you can see, the PolyBase method shows a significantly higher throughput rate compared to BCP, SQLBulkCopy, and SSIS Control-node client gated load methods. If PolyBase is not an option, however, BCP provides the next best load rate.

Regarding loads that improved based on concurrent load (the third row in the chart), keep in mind that SQL Data Warehouse supports up to 32 concurrent queries (loads). For more information about concurrency, see Concurrency and workload management in SQL Data Warehouse.

Conclusion

SQL DW provides many options to load data as we discussed in this article. Each method has its own advantages and disadvantages. It’s easy to “lift and shift” your existing SSIS packages, BCP scripts and other Control-node client gated methods to mitigate migration effort. However, if you require higher speeds for data ingestion, consider rewriting your processes to take advantage of PolyBase with its high throughput, highly scalable loading methodology.

Soften the RBAR impact with Native Compiled UDFs in SQL Server 2016

$
0
0

Reviewers: Joe Sack, Denzil Ribeiro, Jos de Bruijn

Many of us are very familiar with the negative performance implications of using scalar UDFs on columns in queries: my colleagues have posted about issues here and here. Using UDFs in this manner is an anti-pattern most of us frown upon, because of the row-by-agonizing-row (RBAR) processing that this implies. In addition, scalar UDF usage also limits the optimizer to use serial plans. Overall, evil personified!

Native Compiled UDFs introduced

Though the problem with scalar UDFs is well-known, we still come across workloads where this problem is a serious detriment to the performance of the query. In some cases, it may be easy to refactor the UDF as an inline Table Valued Function, but in other cases, it may simply not be possible to refactor the UDF.

SQL Server 2016 offers natively compiled UDFs, which can be of interest where refactoring the UDF to a TVF is not possible, or where the number of referring T-SQL objects  are simply too many. Natively compiled UDFs will NOT eliminate the RBAR agony, but they can make each iteration incrementally faster, thereby reducing the overall query execution time. The big question is how much?

Real-life results

We recently worked with an actual customer workload in the lab. In this workload, we had a query which invoked a scalar UDF in the output list. That means that the UDF was actually executing once per row – in this case a total of 75 million rows! The UDF has a simple CASE expression inside it. However, we wanted to improve query performance so we decided to try rewriting the UDF.

We found the following results with the trivial UDF being refactored as a TVF versus the same UDF being natively compiled (all timings are in milliseconds):

Interpreted (classic)

Native compiled (new in SQL Server 2016)

TVF

CPU Time

12734

8906

3735

Elapsed time

13986

8906

3742

As can be expected, the TVF approach is the fastest, but it is encouraging that the native compiled UDF reduced execution time by solid 36% even though the logic in the UDF was very trivial!

Test Scenario

In order to take this further, I decided to do some testing with a neutral workload. For this neutral test I used the DB1BTicket table which is 230+ million rows and in my test database had a Clustered Columnstore Index created on it.

In this test, I used two different types of UDFs: a trivial one and another one which has more conditional statements in it. The scripts for these are at the end of the post. The results with these different iterations are summarized in the table below:

Interpreted (classic)

Native compiled (new in SQL Server 2016)

TVF

Simple UDF

1672.239 seconds

796.427 seconds

10.473 seconds

Multi-condition UDF

3763.584 seconds

848.106 seconds

Not attempted

Side Note: Parallelism

It is well known that UDFs (even those which do not access data and just do computation) cause a serial plan to be used. Here is the plan with interpreted UDF – as you can see it is serial:

clip_image002[4]

Here is the plan with native compiled UDF – it is still serial:

clip_image004[4]

Lastly, here is the execution plan with TVF – as you can see it is a parallel plan:

clip_image006[4]

Here’s the million-dollar question to you: how badly do you want SQL Server to support parallel plans when UDFs are used anywhere in the plan? Do send us your feedback as comments.

Conclusion

While refactoring the scalar UDF as a TVF ultimately provided the best results, in cases where it is not possible to do this, using native compiled UDFs provides a very useful reduction in query execution time. Therefore, native compiled UDFs can be used as a mitigation or even considered a solution to those thorny cases where RBAR is absolutely necessary.

Appendix: UDF Definitions

Here is the simple UDF, in the classic T-SQL interpreted form:

CREATE FUNCTION dbo.FarePerMile ( @Fare MONEY, @Miles INT )

RETURNS MONEY

    WITH SCHEMABINDING

AS

    BEGIN

        DECLARE @retVal MONEY = ( @Fare / @Miles );

 

        RETURN @retVal;

    END;

GO

 

Here is the simple UDF written as a native compiled version:

CREATE FUNCTION dbo.FarePerMile_native (@Fare money, @Miles int)

RETURNS MONEY

WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER

AS

BEGIN ATOMIC

WITH (TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N’us_english’)

 

DECLARE @retVal money = ( @Fare / @Miles)

RETURN @retVal

END

Here is the simple UDF refactored as a TVF:

                CREATE FUNCTION dbo.FarePerMile_TVF ( @Fare MONEY, @Miles INT )

RETURNS TABLE

AS

RETURN

    SELECT  ( @Fare / @Miles ) AS RetVal;

Now, the multiple condition UDF, in the classic T-SQL interpreted form:

CREATE FUNCTION dbo.FictionalPricingLogic

    (

      @RPCarrier VARCHAR(2) ,

      @Origin VARCHAR(3) ,

      @Fare MONEY ,

      @Miles INT

    )

RETURNS MONEY

    WITH SCHEMABINDING

AS

    BEGIN

        DECLARE @retVal MONEY;

        DECLARE @discount MONEY = 0; – discount percentage

 

        IF ( @RPCarrier = ‘DL’ )

            SELECT  @discount += 0.05;

        IF ( @RPCarrier = ‘AA’ )

            SELECT  @discount += 0.05;

        IF ( @Origin = ‘DFW’ )

            SELECT  @discount += 0.01;

        IF ( @Origin = ‘SEA’ )

            SELECT  @discount += 0.009;

        IF ( @Miles > 500 )

            SELECT  @discount += 0.01;

 

        SELECT  @retVal = @Fare * ( 1.0 - @discount );

 

        RETURN @retVal;

    END;

Here is the multiple condition UDF written as a native compiled version:

CREATE FUNCTION dbo.FictionalPricingLogic_Native

    (

      @RPCarrier VARCHAR(2) ,

      @Origin VARCHAR(3) ,

      @Fare MONEY ,

      @Miles INT

    )

RETURNS MONEY
WITH native_compilation
,schemabinding
,EXECUTE AS OWNER
AS
BEGIN
atomic
WITH (
TRANSACTION ISOLATION LEVEL = snapshot
,LANGUAGE = N’us_english’
)

        DECLARE @retVal MONEY;

        DECLARE @discount MONEY = 0; – discount percentage

 

        IF ( @RPCarrier = ‘DL’ )

            SELECT  @discount += 0.05;

        IF ( @RPCarrier = ‘AA’ )

            SELECT  @discount += 0.05;

        IF ( @Origin = ‘DFW’ )

            SELECT  @discount += 0.01;

        IF ( @Origin = ‘SEA’ )

            SELECT  @discount += 0.009;

        IF ( @Miles > 500 )

            SELECT  @discount += 0.01;

 

        SELECT  @retVal = @Fare * ( 1.0 - @discount );

 

        RETURN @retVal;

    END;

In this test, assume that it was not worth refactoring the multiple condition UDF as a TVF.

Appendix: Test Queries

Here are the sample queries used to test the performance of each of the above UDF variations:

SET STATISTICS TIME ON;

SET STATISTICS IO ON;

 

SELECT  AVG(dbo.FarePerMile(ItinFare, MilesFlown))

FROM    DB1BTicket;

GO

 

SELECT  AVG(dbo.FarePerMile_Native(ItinFare, MilesFlown))

FROM    DB1BTicket;

GO

The below query is the version with the TVF. Note the usage of CROSS APPLY:

SELECT  AVG(myTVF.RetVal)

FROM    DB1BTicket

        CROSS APPLY dbo.FarePerMile_TVF(ItinFare, MilesFlown) AS myTVF;

GO

 

SELECT  AVG(dbo.FictionalPricingLogic(RPCarrier, Origin, ItinFare, MilesFlown))

FROM    DB1BTicket;

GO

 

SELECT  AVG(dbo.FictionalPricingLogic_Native(RPCarrier, Origin, ItinFare,

                                             MilesFlown))

FROM    DB1BTicket;

GO


Assigning surrogate key to dimension tables in SQL DW and APS

$
0
0

Reviewed by: James Rowland-Jones, John Hoang, Denzil Ribeiro, Sankar Subramanian

This article explains how to assign monotonically increasing surrogate/synthetic keys to dimension tables in SQL DW (or APS) using T-SQL. We are going to highlight two possible methods of doing this:

  1. Assign Surrogate keys to the dimension tables where the dimensions are generated in a source system and loaded into SQL DW (or APS).
  2. Extract dimension values from the fact table and assign surrogate keys when the loaded data is such that a single record set has both fact and dimension values.

Background

At the time of the writing of this blog, SQL DW (or APS) does not have the ability to generate IDENTITY values for tables like SQL Server.  Customers however, need a way to generate surrogate values to preserve uniqueness of data entities. Surrogate keys tend to be compact (int, bigint) data types that also to be known to facilitate faster joins and non-redundant distribution in b-tree structure.

NOTE: In SQL DW or (APS), the row_number function generally invokes broadcast data movement operations for dimension tables. This data movement cost is very high in SQL DW. For smaller increment of data, assigning surrogate key this way may work fine but for historical and large data loads this process may take a very long time. In some cases, it may not work due to tempdb size limitations. Our advice is to run the row_number function in smaller chunks of data.

Solution

Method 1:

Following is the structure of customer table in SQL DW. Let’s assume for this table we have already generated surrogate keys for the data in SQL DW. In table c_customer_sk is the surrogate key column.

CREATE TABLE [dbo].[customer]
(
[c_customer_sk] INT NOT NULL, –Surrogate Key
[c_customer_id] CHAR(16) NOT NULL, — Business Key
[c_current_cdemo_sk] INT NULL,
[c_current_hdemo_sk] INT NULL,
[c_current_addr_sk] INT NULL,
[c_first_shipto_date_sk] INT NULL,
[c_first_sales_date_sk] INT NULL,
[c_salutation] CHAR(10) NULL,
[c_first_name] CHAR(20) NULL,
[c_last_name] CHAR(30) NULL,
[c_preferred_cust_flag] CHAR(1) NULL,
[c_birth_day] INT NULL,
[c_birth_month] INT NULL,
[c_birth_year] INT NULL,
[c_birth_country] VARCHAR(20) NULL,
[c_login] CHAR(13) NULL,
[c_email_address] CHAR(50) NULL,
[c_last_review_date] CHAR(10) NULL
);

We have some new data that needs to be loaded into customer table with surrogate key assigned. Load the new customer data into a transient table called customer_staging that does not contain the c_customer_sk column.

CREATE TABLE [dbo].[customer_staging]
(
[c_customer_id] CHAR(16) NOT NULL, — Business Key
[c_current_cdemo_sk] INT NULL,
[c_current_hdemo_sk] INT NULL,
[c_current_addr_sk] INT NULL,
[c_first_shipto_date_sk] INT NULL,
[c_first_sales_date_sk] INT NULL,
[c_salutation] CHAR(10) NULL,
[c_first_name] CHAR(20) NULL,
[c_last_name] CHAR(30) NULL,
[c_preferred_cust_flag] CHAR(1) NULL,
[c_birth_day] INT NULL,
[c_birth_month] INT NULL,
[c_birth_year] INT NULL,
[c_birth_country] VARCHAR(20) NULL,
[c_login] CHAR(13) NULL,
[c_email_address] CHAR(50) NULL,
[c_last_review_date] CHAR(10) NULL
);

Now issue the following insert command that will insert the rows from customer_staging to customer table while assigning unique increasing value to the column c_customer_sk column.

INSERT INTO dbo.customer
SELECT maxid.maxid + ROW_NUMBER() OVER (ORDER BY maxid) AS [c_customer_sk]
,c_customer_id
,c_current_cdemo_sk
,c_current_hdemo_sk
,c_current_addr_sk
,c_first_shipto_date_sk
,c_first_sales_date_sk
,c_salutation
,c_first_name
,c_last_name
,c_preferred_cust_flag
,c_birth_day,c_birth_month
,c_birth_year,c_birth_country
,c_login,c_email_address
,c_last_review_date
FROM dbo.customer_staging
CROSS JOIN
(SELECT ISNULL(MAX(c_customer_sk),0) AS maxid FROM dbo.customer)maxid

If source customer table is small enough such that the ETL logic uploads the whole customer table every time into customer_staging inclusive of new data, a left outer join technique could also be used to exclude the duplicates before inserting the data into customer table.

Note: It is our recommendation to avoid loading the full table for very large dimensions every time.

INSERT INTO dbo.customer
SELECT maxid.maxid + ROW_NUMBER() OVER (ORDER BY maxid) AS [c_customer_sk]
, CStaging.*
FROM
(
select cs.* from dbo.customer_staging cs
left outer join dbo.customer c
on cs.c_customer_id = c.c_customer_id
where c.c_customer_id is null
) CStaging
CROSS JOIN
(SELECT ISNULL(MAX(c_customer_sk),0) AS maxid FROM dbo.customer)maxid;

Method 2:

In some cases, the values of a dimension table need to be extracted from the fact table. The example below is similar to the last one with few simple changes.

Let’s assume dim_customer is our existing dimension table that is populated with attributes from the fact table stage_store_sales.

CREATE TABLE [dbo].[dim_customer]
(
[ss_customer_sk] INT NULL,
[ss_customer_id] CHAR(16) NULL,
[ss_customer_name] VARCHAR(50) NULL
);
CREATE TABLE [dbo].[stage_store_sales]
(
[ss_sold_date_sk] INT NULL,
[ss_sold_time_sk] INT NULL,
[ss_item_sk] INT NOT NULL,
[ss_customer_id] CHAR(16) NULL,  — dim value
[ss_customer_name] VARCHAR(50) NULL, — dim value
[ss_cdemo_sk] INT NULL,
[ss_hdemo_sk] INT NULL,
[ss_addr_sk] INT NULL,
[ss_store_sk] INT NULL,
[ss_promo_sk] INT NULL,
[ss_ticket_number] INT NOT NULL,
[ss_quantity] INT NULL,
[ss_wholesale_cost] DECIMAL(7,2) NULL,
[ss_list_price] DECIMAL(7,2) NULL,
[ss_sales_price] DECIMAL(7,2) NULL,
[ss_ext_discount_amt] DECIMAL(7,2) NULL,
[ss_ext_sales_price] DECIMAL(7,2) NULL,
[ss_ext_wholesale_cost] DECIMAL(7,2) NULL,
[ss_ext_list_price] DECIMAL(7,2) NULL,
[ss_ext_tax] DECIMAL(7,2) NULL,
[ss_coupon_amt] DECIMAL(7,2) NULL,
[ss_net_paid] DECIMAL(7,2) NULL,
[ss_net_paid_inc_tax] DECIMAL(7,2) NULL,
[ss_net_profit] DECIMAL(7,2) NULL
)
WITH (DISTRIBUTION = HASH ([ss_item_sk]));

The following view contains the logic to find and exclude the duplicates from the fact and the dimension table. Only new values will be inserted into the dimension table.

CREATE VIEW [dbo].[V_stage_dim_customer] AS SELECT DISTINCT [ss_customer_sk] = maxid.maxid + ROW_NUMBER() OVER (ORDER BY maxid),[ss_customer_id], ss_customer_name
FROM
(
SELECT DISTINCT ss.[ss_customer_id]
, ss.ss_customer_name
FROM [dbo].[stage_store_sales] [ss] –staging fact table
LEFT OUTER JOIN [dbo].[dim_customer] [dc] –dimension table
ON [ss].[ss_customer_id] = [dc].[ss_customer_id]
WHERE [dc].[ss_customer_id] IS NULL AND [ss].[ss_customer_id] IS NOT NULL
) Cstaging
CROSS JOIN (SELECT ISNULL(MAX([ss_customer_sk]),0) AS maxid FROM [dbo].[dim_customer]) maxid;

Run the following insert-select statement to insert new values into the dimension table from the view.

INSERT INTO dim_customer
SELECT * FROM V_stage_dim_customer;

Note: In some cases, a row number is present in the data file or can easily be added. In that case this row number added with the max value from the existing dimension can be used to create the surrogate key.

SQL Server 2016 SSIS Data Flow Buffer Auto Sizing capability benefits data loading on Clustered Columnstore tables

$
0
0

 

Reviewers: Joe Sack, Arvind Shyamsundar, Murshed Zaman, Sanjay Mishra

 The focus of this blog is to introduce SSIS changes in the Data Flow task in SQL Server 2016 that help data loading into Columnstore tables.

On SQL Server relational data warehouses, Columnstore indexes are being widely used due to the immense value realized by customers from both a performance aspect as well as cost savings due to compressed size on storage.  SSIS is a popular ETL tool that is often used to load data into the warehouse. We covered some of the Data loading best practices with Data Loading Performance Considerations with Clustered Columnstore indexes  on SQL Server 2014 and had mentioned a few caveats with choosing appropriate Maximum insert commit size and its implications on rows potentially landing in a delta store uncompressed due to the Maximum buffer size in SSIS. For details see “SSIS Data Flow Task Max Insert Commit Size and implications on Row group sizes section” here.

On SQL Server 2016, looking at the default SSIS Data Flow Properties you see that the default value for “DefaultBufferSize” is 10MB and the DefaultBufferMaxRows is 10,000.

image

Given the rowgroup max size of 1,048,576 if you try to force your Maximum insert commit size as shown below, you could end up getting very few rows committed per insert and more importantly they can all end up in the delta store which is a rowstore. How many rows are inserted in one transaction totally depends on the row size as to how many rows would fit in a 10MB buffer. Prior to SQL Server 2016, you could only adjust the DefaultBufferSize to a maximum of 100MB. For additional details see the article: SSIS Buffer Sizing

 image

An extended event capture below shows us that each bulk insert ended up with a batch size of 5825 rows.

image

From the DMV output below, you can see that all the rows end up in the Delta store. You could adjust the BufferSize to 100MB and that row count could increase to a higher number but it all depended on the size of the row. Ultimately the rowgroup will fill up and close and will be compressed by Tuple mover but that is not the most efficient way to load data as soon after loading the data is not compressed.

select * from sys.dm_db_column_store_row_group_physical_stats
where object_id = object_id('FctSales')

clip_image008[8]

AutoAdjustBufferSize Data Flow Property

SQL Server 2016  introduces a new Data Flow property called AutoAdjustBufferSize which when set to “true” ignores the DefaultBufferSize setting and the actual buffer size is calculated based on the DefaultBufferMaxRows. The formula used for calculating the Buffersize is specified below

BufferSize = DefaultBufferMaxRows (set at Design time) * Rowsize calculated at runtime.

Given this, you can set my Maximum Insert commit size to an appropriate batch size (a good start is 1,048,576) which is very useful in case of a large data transfer so you are not committing the entire large transfer in one transaction. Looking at the prior example, if I set AutoAdjustBufferSize property to true and set the DefaultBufferMaxRows to 1,048,576 which is the max row group size, you see a difference in behavior where that Max Insert commit size is
honored as the Buffer size is adjusted automatically.

clip_image010[8]image

 Extended Events showing the inserts from the SSIS package show us honoring the Max Insert commit size.

image

And the goal is to get the data into compressed rowgroups directly as you can see below. As an aside, the state INVISIBLE is an in-memory state of a compressed rowgroup before made available to users.

image

Performance test results

From a performance perspective these settings do have ramifications on the load times. This is a sample table used in load tests depicted by the figure below, the row size of this sample table was 1,674 bytes. These test results are on a specific VM with a specific table definition, results of performance in other scenarios will depend on the configuration of the machine, and the table definition. 

Test SSIS Data flow configuration Effective Batch size Rows Elapsed Time
1 AutoAdjustBuffer = FalseDefaultBufferMaxRows = 10000 (default)DefaultBufferSize = 10485760 (default) 10,000 30 million 15:54
2 AutoAdjustBuffer = FalseDefaultBufferMaxRows = 1048576DefaultBufferSize = 104857600 62,638 30 million 12:03
3 AutoAdjustBuffer = TrueDefaultBufferMaxRows = 1048576 1,048,576 30 million 5:56

In the test #2 in the table above, given the max default buffer size is 100MB (which was the maximum on SQL Server 2014), given the individual row was 1674 bytes, at max you would get 62638 rows to fill that buffer before SSIS issued a commit which means it all ended up in the Delta store. Only Test 3 in this scenario ended up in compressed row groups directly.

Note: Increasing the buffer size can result in paging which can be monitored with the “Buffers Spooled” performance counter if the machine running the SSIS package cannot sustain that buffer size in memory

AutoAdjustBufferSize  property of the data flow task is a great addition in SQL Server 2016 to aid in data loading scenarios.

 

Considerations around validation errors 41305 and 41325 on memory optimized tables with foreign keys

$
0
0

Reviewed by: Jos de Bruijn; Joe Sack, Mike Weiner, Mike Ruthruff, Kun Cheng

Transactions on memory optimized tables in SQL Server 2014, SQL Server 2016 and Azure SQL Database are implemented with an optimistic concurrency model with multi-version concurrency control. Each transaction has its own transactionally consistent version of rows and the inherent assumption is that there aren’t any conflicts. Unlike on-disk tables, there is no locking and write-write conflicts are detected (error 41302) if concurrent transactions update the same rows. Even to maintain higher isolation levels, locks aren’t taken and hence validation has to occur at transaction commit time.

To recap a transaction lifetime there are 3 phases as described below, in this blog we will focus more on the validation phase

Capture

For more details see the books Online article : https://msdn.microsoft.com/en-us/library/dn133169.aspx

On transaction commit, you have to validate that no other transaction has updated or changed the rows you have read if using the Repeatable Read isolation level, or that no phantom rows were inserted into the range that you have read if the isolation level is Serializable. If all validation succeeds, then the transaction commits.

The summary of the validation errors is in the article Guidelines for Retry Logic for Transactions on Memory-Optimized Tables .

  • 41305. The current transaction failed to commit due to a repeatable read validation failure.
  • 41325. The current transaction failed to commit due to a serializable validation failure.

While testing a customer workload we encountered a delete transaction that was under the Snapshot Isolation level that was still failing with one of the validation errors 413235 listed above. This was puzzling, because under the Snapshot isolation level, you don’t expect to fail with repeatable read validation errors.

SQL Server 2016 did introduce support for foreign keys on memory optimized tables along with many other surface area improvements for In-memory OLTP as detailed here. Foreign keys do introduce a difference in the behavior with regards to validation that you may not expect. If you update or delete data from a table which has foreign key relationships, validation for adherence of those relationships has to happen under a higher isolation level in order to ensure that no rows are changed such that the constraint is violated. This means that if a constraint is violated due to DML performed in concurrent transactions, then the commit will fail with a validation failure. On-disk tables on the other hand acquire row locks or key locks on the tables involved in foreign constraints and concurrent transactions trying to delete or update those rows will be blocked. Unlike on-disk tables, memory optimized tables are lock free and hence validations at the right isolation level are required to ensure correctness.

Let’s take a look at a simple master/detail table relationship. Scripts to reproduce are at the end of the article in the appendix.

  • tblMaster is the master table
  • tblDetails is the child table has a foreign key that is defined on the tblMaster table’s EventID column

Exhibit 1: Validation errors due to lack of supporting indexes

In the sequence defined in the diagram below, you see that the statements themselves were executed at the Snapshot Isolation Level and yet the error we get is a “repeatable read” validation error, something we do not necessarily expect. Also Transaction 1 below is actually updating a row with EventID =4, which isn’t one of the rows that Transaction 2 is deleting and hence we did not expect this error. This is an example of a repeatable read validation error that was avoidable with proper indexing. When it comes to foreign keys always add an index on the Foreign key to support lookups as we will see shortly.

Sequence Transaction 1 Transaction 2
1  BEGIN TRAN
 
2  DELETE FROM tblMaster WITH (SNAPSHOT) WHERE ID = 5
 
3 UPDATE tbldetails WITH (SNAPSHOT)
SET EventName = ‘Event22′
WHERE EventID = 4 and languageID = 1
 
4  COMMIT
 
5  Msg 41305, Level 16, State 0, Line 18
The current transaction failed to commit due to a repeatable read validation failure.
 

Given the errors, we then looked at the execution plan of the delete statement, which shed some light on the root cause. Looking at the plan below, you can see a scan on the tblDetails table that reads the entire table, and then applies a filter to get the few rows affected. Since the entire table is read, that qualifies as the “read set” of the delete. Updating a row in that read set results in a repeatable read validation though it technically does not violate the constraint.


The output of dm_db_xtp_transactions shows the same read set being shown if you capture this before the commit for the transaction.

SELECT xtp_transaction_id,transaction_id,session_id
	,state_desc,result_desc,read_set_row_count,write_set_row_count
	,scan_set_count,commit_dependency_count
FROM sys.dm_db_xtp_transactions

 


Seeing that the Plan was suboptimal and was acquiring a larger read set than was needed due to lack of proper indexing, creating an index on the appropriate foreign key columns resulted in a different plan and the read set was then much smaller as we aren’t scanning the entire table.

ALTER TABLE tblDetails
ADD INDEX idxtblDetails_EventID NONCLUSTERED(EventID)



Adding this index removed these repeatable read validation errors resulting from rows being updated that were not in the range being deleted. This was a case of an avoidable validation error.

Exhibit 2: Serializable validation errors due to DML pattern

In the sequence below, the delete is done with SNAPSHOT isolation level, yet the error message that we get indicates a serializable validation error. This validation error occurs as we are inserting a value into the child table with the same key that we are deleting off the parent which would end up being a violation of the constraint. This is an expected validation error and is fundamentally due to application data entry order.

Sequence Transaction 1 Transaction 2
1  BEGIN TRAN
 
2  DELETE FROM tblMaster WITH (SNAPSHOT) WHERE ID = 5
3 INSERT INTO tbldetails VALUES (5,900001,’Event1′,’US’) 
4  COMMIT
 
5  Msg 41325, Level 16, State 0, Line 39
The current transaction failed to commit due to a serializable validation failure. 
 

If you did the same sequence of operations on memory optimized tables serially from one transaction, you would get the following behavior where the insert would fail as it conflicted with the key in the master table.

Sequence Transaction 1
1 BEGIN TRAN
 
2 DELETE FROM tblMaster WITH (SNAPSHOT) WHERE ID = 5
 
3  INSERT INTO tbldetails VALUES (5,900001,’Event1′,’US’) 
4 COMMIT
 
5 Msg 547, Level 16, State 0, Line 56
The INSERT statement conflicted with the FOREIGN KEY constraint “FK_tblDetails_tblMaster”. The conflict occurred in database “HKCCI”, table “dbo.tblMaster”, column ‘Id’.
The statement has been terminated.

If we did this same set of operations on disk based tables, at the default isolation level which is READCOMMITTED, the behavior would be below. Transaction 1 is the one that fails in this case and while Transaction 2 is open, transaction 1 is blocked.

Sequence Transaction 1 ( disk based ) Transaction 2 ( disk based)
1  BEGIN TRAN
 
2  DELETE FROM tblMaster_ondisk WHERE ID = 5
 
3  INSERT INTO tbldetails_ondisk VALUES (5,900001,’Event1′,’US’) 
à Blocked
4  COMMIT
 
5
  Msg 547, Level 16, State 0, Line 11
The INSERT statement conflicted with the FOREIGN KEY constraint “FK_tblDetails_tblMaster_ondisk”. The conflict occurred in database “HKCCI”, table “dbo.tblMaster_ondisk”, column ‘Id’.

Exhibit 3: Repeatable read validation errors due to DML pattern

Below is an example of an expected occurrence of a repeatable read validation error. In this case a value is inserted into the read-set of the rows affected by the delete and specifically into that range being deleted. This is due to how data is being inserted by the application and has to be handled and retried with the guidelines specified in Guidelines for Retry Logic for Transactions on Memory-Optimized Tables

Sequence Transaction 1 (in-memory) Transaction 2 (in-memory)
1  BEGIN TRAN
2  INSERT tbldetails (EventID) VALUES (5)  
3 DELETE FROM tblMaster WITH (SNAPSHOT) WHERE ID = 5 
4  COMMIT
 
5  Msg 41305, Level 16, State 0, Line 18
The current transaction failed to commit due to a repeatable read validation failure. 

Doing this same sequence on an on-disk table results in the delete being blocked and finally failing with a constraint violation error.

Sequence Transaction 1 (on disk) Transaction 2 ( on disk)
1  BEGIN TRAN
 
2  INSERT tbldetails_ondisk (EventID) VALUES (5)
 
3 DELETE FROM tblMaster_ondiskWHERE ID = 5
–>  This  statement is blocked
4  COMMIT
 
5  Msg 547, Level 16, State 0, Line 20
The DELETE statement conflicted with the REFERENCE constraint “FK_tblDetails_tblMaster_ondisk”. The conflict occurred in database “HKCCI”, table “dbo.tblDetails_ondisk”, column ‘EventId’

In either of the cases Extended Events with the error_number event can help in figuring out where in your application code validation errors are being raised to take the appropriate actions.

CREATE EVENT SESSION [TraceUserErrors_Validation] ON SERVER
ADD EVENT sqlserver.error_reported(
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.client_hostname
		,sqlserver.query_hash_signed,sqlserver.query_plan_hash_signed,sqlserver.session_id
		,sqlserver.sql_text,sqlserver.tsql_frame)
    WHERE ([package0].[equal_int64]([error_number],(41325))
			OR [package0].[equal_int64]([error_number],(41305))
			OR [error_number]=(41300)
			OR [error_number]=(41301)))
ADD TARGET package0.event_file 	(SET filename=N'c:\Temp\TraceUserErrors_Validation.xel'
				,max_file_size=(250),max_rollover_files=(2)),
ADD TARGET package0.histogram	(SET filtering_event_name=N'sqlserver.error_reported'
				,source=N'error_number',source_type=(0))
WITH (EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS)
GO


Output of the histogram target gives you a quick look at number of validation errors occurring grouped by the error number.

Summarizing, with the introduction of foreign key support for In-Memory OLTP, it is imperative to index the keys appropriately so that joins are efficient not only for explicitly defined joins, but also for validations that happen under the covers when you insert, update or delete a row so that we avoid unexpected validation errors.

For other validation errors, given the optimistic concurrency , applications using memory optimized tables should also include retry logic for valid cases of validation errors as specified in the article Guidelines for Retry Logic for Transactions on Memory-Optimized Tables

Appendix: TSQL Scripts to reproduce the behavior

 Setup Script:

USE [master]
GO
/****** Object:  Database [HKCCI]    Script Date: 3/24/2016 9:37:53 AM ******/
CREATE DATABASE InMemoryOLTP
ON  PRIMARY ( NAME = N'HKCCI', FILENAME = N'c:\temp\InMemoryOLTP.mdf'),
FILEGROUP [HKCCI_InMemory] CONTAINS MEMORY_OPTIMIZED_DATA  DEFAULT
(NAME = N'HKCCI_InMemory', FILENAME = N'c:\temp\InMemoryOLTP' , MAXSIZE = UNLIMITED)
 LOG ON ( NAME = N'HKCCI_log', FILENAME = N'c:\temp\InMemoryOLTP_log.ldf')
GO

USE InMemoryOLTP
go
-- Master
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[tblMaster]
(
	[Id] [int]  NOT NULL,
	[ExternalId] [int] NOT NULL,
	[IsActive] [bit] NOT NULL,
	[UTCStartTime] [datetime2](7) NOT NULL,
INDEX [NC_tblMaster_ExternalId] NONCLUSTERED ([ExternalId] ASC),
 CONSTRAINT [PK_tblMaster] PRIMARY KEY  NONCLUSTERED HASH ([Id]) WITH ( BUCKET_COUNT = 32000),
) WITH ( MEMORY_OPTIMIZED = ON , DURABILITY = SCHEMA_AND_DATA )
GO

-- Details table
CREATE TABLE [dbo].[tblDetails]
(
	[DetailID] int identity,
	[EventId] [int] NOT NULL,
	[LanguageId] [int] NOT NULL,
	[EventName] [nvarchar](512) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
	[CountryName] [nvarchar](512) COLLATE SQL_Latin1_General_CP1_CI_AS NULL

INDEX [NC_tblDetails_LanguageId] NONCLUSTERED ([LanguageId] ASC),
 CONSTRAINT [PK_tblDetails]  PRIMARY KEY NONCLUSTERED HASH ([DetailID]) WITH ( BUCKET_COUNT = 32768),
) WITH ( MEMORY_OPTIMIZED = ON , DURABILITY = SCHEMA_AND_DATA )
GO
ALTER TABLE [dbo].[tblDetails]  WITH CHECK ADD  CONSTRAINT [FK_tblDetails_tblMaster] FOREIGN KEY([EventId])
REFERENCES [dbo].[tblMaster] ([Id])
GO

--insert some data
INSERT INTO tblmaster VALUES ( 1,1,1,getutcdate())
INSERT INTO tblmaster VALUES ( 2,2,1,getutcdate())
INSERT INTO tblmaster VALUES ( 3,3,1,getutcdate())
INSERT INTO tblmaster VALUES ( 4,4,1,getutcdate())
INSERT INTO tblmaster VALUES ( 5,5,1,getutcdate())
INSERT INTO tblmaster VALUES ( 6,6,1,getutcdate())
GO

-- Translations
WITH Nbrs_3( n ) AS ( SELECT 1 UNION SELECT 0 ),
 Nbrs_2( n ) AS ( SELECT 1 FROM Nbrs_3 n1 CROSS JOIN Nbrs_3 n2 ),
 Nbrs_1( n ) AS ( SELECT 1 FROM Nbrs_2 n1 CROSS JOIN Nbrs_2 n2 ),
 Nbrs_0( n ) AS ( SELECT 1 FROM Nbrs_1 n1 CROSS JOIN Nbrs_1 n2 ),
 Nbrs ( n ) AS ( SELECT 1 FROM Nbrs_0 n1 CROSS JOIN Nbrs_0 n2 )
INSERT INTO tblDetails SELECT top 5000 1,ROW_NUMBER()  OVER (ORDER BY NEWID()) as LanguageID, 'Event1','AU' FROM Nbrs
GO

WITH Nbrs_3( n ) AS ( SELECT 1 UNION SELECT 0 ),
 Nbrs_2( n ) AS ( SELECT 1 FROM Nbrs_3 n1 CROSS JOIN Nbrs_3 n2 ),
 Nbrs_1( n ) AS ( SELECT 1 FROM Nbrs_2 n1 CROSS JOIN Nbrs_2 n2 ),
 Nbrs_0( n ) AS ( SELECT 1 FROM Nbrs_1 n1 CROSS JOIN Nbrs_1 n2 ),
 Nbrs ( n ) AS ( SELECT 1 FROM Nbrs_0 n1 CROSS JOIN Nbrs_0 n2 )
INSERT INTO tblDetails SELECT top 5000 2,ROW_NUMBER()  OVER (ORDER BY NEWID()) as LanguageID, 'Event2','AU'FROM Nbrs
GO

--insert into EventTranslations values(2,3,'Event1','FR')
WITH Nbrs_3( n ) AS ( SELECT 1 UNION SELECT 0 ),
 Nbrs_2( n ) AS ( SELECT 1 FROM Nbrs_3 n1 CROSS JOIN Nbrs_3 n2 ),
 Nbrs_1( n ) AS ( SELECT 1 FROM Nbrs_2 n1 CROSS JOIN Nbrs_2 n2 ),
 Nbrs_0( n ) AS ( SELECT 1 FROM Nbrs_1 n1 CROSS JOIN Nbrs_1 n2 ),
 Nbrs ( n ) AS ( SELECT 1 FROM Nbrs_0 n1 CROSS JOIN Nbrs_0 n2 )
INSERT INTO tblDetails SELECT top 5000 3,ROW_NUMBER()  OVER (ORDER BY NEWID()) as LanguageID, 'Event3','CA' FROM Nbrs
GO
WITH Nbrs_3( n ) AS ( SELECT 1 UNION SELECT 0 ),
 Nbrs_2( n ) AS ( SELECT 1 FROM Nbrs_3 n1 CROSS JOIN Nbrs_3 n2 ),
 Nbrs_1( n ) AS ( SELECT 1 FROM Nbrs_2 n1 CROSS JOIN Nbrs_2 n2 ),
 Nbrs_0( n ) AS ( SELECT 1 FROM Nbrs_1 n1 CROSS JOIN Nbrs_1 n2 ),
 Nbrs ( n ) AS ( SELECT 1 FROM Nbrs_0 n1 CROSS JOIN Nbrs_0 n2 )
INSERT INTO tblDetails
SELECT TOP 10000 4,ROW_NUMBER()  OVER (ORDER BY NEWID()) as LanguageID, 'Event4','UK' FROM Nbrs
GO
SELECT COUNT(*) from tblDetails
GO

Validation Transaction 1 script:

--**********Step 1 ( Validation_transaction1.sql)
-- Open the Tran and Delete Parent Table
-- You can enable Execution Plans for this, does a Scan on EventTranslations
BEGIN TRAN
DELETE FROM tblMaster WITH (SNAPSHOT) WHERE ID = 5
GO

-- Now Execute Step2 from Validation_Transaction2.sql file

---************** Step 3
-- you should hit the error
/*
Msg 41305, Level 16, State 0, Line 6
The current transaction failed to commit due to a repeatable read validation failure. */
COMMIT
GO

---************** Step 4 ( Validation_transaction1.sql)
--- FIX and repeat Step 1 , Step 2, and Step 3.
ALTER TABLE tblDetails
ADD INDEX idxtblDetails_EventID NONCLUSTERED(EventID)
GO

-- ************** Step 5 ( Validation_transaction1.sql)
-- Lets try to repro the Serializable error
DELETE FROM tblDetails WITH (SNAPSHOT) WHERE EventID = 1
BEGIN TRAN
DELETE FROM tblMaster WITH (SNAPSHOT) WHERE ID = 1
GO

-- Now Execute ***Step6 **** from Validation_transaction2.sql

--******Step 7 ( Validation_transaction1.sql)
/*
Msg 41325, Level 16, State 0, Line 31
The current transaction failed to commit due to a serializable validation failure. */

COMMIT
GO

---************** Step 8 ( Validation_transaction1.sql)
delete from tblDetails where eventid = 5
delete from tblmaster where id = 5
go
insert into tblMaster WITH (SNAPSHOT) values(5,2,1,getdate())
GO
BEGIN TRAN
INSERT tbldetails  WITH (SNAPSHOT) VALUES(5,-2,'Event111','US')
GO

--********** Step 10 ( Validation_transaction1.sql
/*
Msg 41305, Level 16, State 0, Line 58
The current transaction failed to commit due to a repeatable read validation failure.
*/
COMMIT
GO

Validation Transaction 2 script:

--***** Step 2 ( Validation transaction 2.sql)
UPDATE tbldetails WITH (SNAPSHOT)
SET EventName = 'Event221'
WHERE EventID = 4 and languageID = 1

--- Now Execute Step 3 from the Validation_transaction1.sql file

--********* Step 6 ( Validation_transaction2.sql)
INSERT INTO tbldetails VALUES (1,900001,'Event1','US')

--***** Step 9 ( Validation_transaction2.sql )
DELETE FROM tblMaster WITH (SNAPSHOT) WHERE ID = 5

SQLCAT @ SQL Nexus conference

$
0
0

SQLCAT is coming to SQL Nexus in Copenhagen on May 2nd-4th and running a full day pre-conference focusing on Azure SQL Database, SQLCAT: A Day in the life of an Azure SQL DB Developer/DBA/DevOps

In this demo rich session, we will cover various aspects of Azure SQL Database Service based on experience working with large customers and ISVs on Azure including:

  • End-End monitoring
  • Network Connectivity and transient fault handling
  • Performance Troubleshooting and Application aspects.
  • In-Memory Enhancements and how they related to Azure SQL Database
  • Security aspects and ways to protect data in the cloud
  • Elastic Database Pools, Jobs, Transactions, Elastic Scale.
  • High Availability and Disaster Recovery
  • Migration strategies and best practices from on-prem.

If you are planning on attending a pre-con, we hope to see you there!

In the main conference also have a session sharing our learnings from taking multiple customers into production on SQL Server 2016 pre-release. In this session we will share challenges faced and lessons learned during these customer engagements spread across multiple features in SQL Server 2016.

“Session Title: Top 10 Learnings from early customer adoption with SQL Server 2016”

We are looking forward to seeing you at either of the sessions, and hope you enjoy the conference!

Checkpoint process for memory-optimized tables in SQL 2016 and implications on the log

$
0
0

Reviewed by: Alejandro Saenz, Jos de Bruijn; Joe Sack, Mike Weiner,Kun Cheng, Raghavendra TK

A durable memory-optimized table (created as SCHEMA_AND_DATA) will have it’s transactions recorded into the transaction log so that on failure, recovery is possible. This logging ensures the durability of the transactions. Unlike disk-based tables, memory-optimized tables do not use WAL ( write-ahead logging) protocol, transactions are written to the log only on a commit and dirty data is never written to disk.

The checkpoint process is a background process used by SQL Server to ensure that in the event of a crash, the time it takes to recover isn’t very long.

Checkpoints for disk-based tables

A checkpoint for disk-based tables will result in flushing dirty pages to disk and in the simple recovery model the log will also be truncated. For disk-based tables, the database engine supports automatic, indirect and manual checkpoints. Automatic checkpoints for disk-based tables are based on the recovery interval configuration option. In addition, with SQL Server 2016, the default on newly created databases is to have “indirect” checkpoint enabled which shortens that recovery time period.

Checkpoints for memory optimized tables

For memory-optimized tables, the checkpoint process flushes both the data streams that contain all new versions of records and the delta stream that contains deleted versions of records to data and delta files. Each pair of data/delta files constitutes a checkpoint file pair or CFP and these files have the extensions .hkckp. All the data/delta files are written to sequentially or as streaming IO rather than random IO. To prevent the files from growing indefinitely , the checkpoint process will also do a “merge” if it is possible to merge subsequent checkpoint file pairs that have been closed.

Data and Delta files

Checkpoint behavior differences with memory-optimized tables

Checkpoint for memory optimized tables is done by a background worker thread in the In-Memory OLTP engine which is separate than the checkpoint thread for disk-based tables. Automatic checkpoint for memory-optimized tables will execute based on the following condition:

  • If the log has produced 1.5 GB since the last checkpoint, this will trigger the Engine to checkpoint memory optimized tables.

This difference in behavior in checkpoint can cause confusion with regards when the log is purged when you have memory optimized tables. For disk based tables, the log is truncated on a log backup for full recovery model or an automatic checkpoint for a database in simple recovery mode. This changes when we have a memory optimized table, a transaction log backup doesn’t necessarily truncate the log until the thresholds specified above are met.

Of course, at any time though, a manual checkpoint command can be issued.

Also, with SQL Server 2016 there is a special case, based on the conditions below, which will vary the checkpoint characteristics for memory-optimized tables. This is referred to as the “Large checkpoint” process, which is enabled on larger machines if all of the following conditions are true

  • The server has 16 or more logical processors
  • The server has 128GB or greater memory.
  • The server is capable of greater than 200MB/sec I/O measured for the IO subsystem of that database.

Details on this behavior are provided below in the section titled Large checkpoint for memory-optimized tables

Memory-optimized table checkpoint behavior example

Here is an example that shows what happens with checkpoints on memory-optimized tables. In summary we will:

  • Create a database
  • Establish the log chain by taking a backup
  • Populate some data and commit, thereby growing the log
  • Take a log backup

The expectation is that after the log backup, the log space used will be minimal unless there is some open transaction/ un-replicated transaction, or some other factors that prevents the log from being purged.

Example:

CREATE DATABASE [InMemoryOLTP]
 CONTAINMENT = NONE
 ON  PRIMARY
( NAME = N'IMOLTP', FILENAME = N'c:\temp\InMemoryOLTP.mdf'),
 FILEGROUP [IMOLTP_InMemory] CONTAINS MEMORY_OPTIMIZED_DATA  DEFAULT
( NAME = N'IMOLTP_InMemory', FILENAME = N'c:\temp\InMemoryOLTP')
 LOG ON
( NAME = N'IMOLTP_log', FILENAME = N'c:\temp\InMemoryOLTP_log.ldf')
 GO

USE InMemoryOLTP
GO
CREATE TABLE dbo.SalesOrder_MemOpt
(
 order_id int identity not null,
 order_date datetime not null,
 order_status tinyint not null,
 amount float not null,
 Constraint PK_SalesOrderID PRIMARY KEY NONCLUSTERED HASH (order_id) WITH (BUCKET_COUNT = 10000)
) WITH ( MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA)
GO
-- Create Natively compiled procedure to speed up inserts.
CREATE PROCEDURE [dbo].[InsertSalesOrder_Native_Batch]
@order_status tinyint = 1,
@amount float = 100,
@OrderNum int = 100
  WITH
    NATIVE_COMPILATION,
    SCHEMABINDING,
    EXECUTE AS OWNER
AS
BEGIN ATOMIC
  WITH (TRANSACTION  ISOLATION LEVEL = SNAPSHOT,LANGUAGE = 'us_english')
	declare @counter int = 1
	WHILE @counter <= @OrderNum
	BEGIN
	   INSERT INTO dbo.SalesOrder_MemOpt values(getdate(),@order_status,@amount)
	   SET @counter= @counter+1
	END
END
GO

-- Create backup to start chain, don't need the backups
 CHECKPOINT
 go
 BACKUP DATABASE InMemoryOLTP to disk = 'c:\temp\test.bak' WITH INIT,COMPRESSION
 BACKUP LOG InMemoryOLTP to disk = 'c:\temp\test.bak' WITH COMPRESSION
 GO
-- Check Log Space used column for DB
 DBCC SQLPERF(logspace)
 GO

-- Check Log Space, file checkpoint stats, file states
	-- Do we have any reason why the log can't be purged?
	SELECT log_reuse_wait_desc,*
	FROM sys.databases
	WHERE name = 'InMemoryOLTP'
	GO
	-- What percentage of the log is used for this DB?
	DBCC sqlperf(logspace)
	GO
	-- How much log is generated since last checkpoint?
	SELECT log_bytes_since_last_close /(1024*1024) as Log_bytes_since_last_close_mb,time_since_last_close_in_ms, current_checkpoint_id,*
	FROM sys.dm_db_xtp_checkpoint_stats
	GO
	-- Have checkpoint files been flushed?
	SELECT  state_desc,
       file_type_desc,
       count(state_desc) count,
       sum(file_size_in_bytes)/(1024*1024) file_size_in_mb_bytes,
	   Avg(file_size_in_bytes)/(1024*1024) Avgfile_size_in_mb_bytes
	FROM sys.dm_db_xtp_checkpoint_files
	GROUP BY state_desc, file_type_desc
	ORDER BY file_size_in_mb_bytes desc

DMV outputs

-- This proc inserts 900K rows 20 times
-- Attempt to grow the log to ~ 1GB
BEGIN TRAN
EXEC dbo.[InsertSalesOrder_Native_Batch]  1,100,900000
DELETE FROM dbo.SalesOrder_MemOpt WITH(SNAPSHOT)
COMMIT
GO 20
-- Backup the log
-- Do you expect the log to be purged?
BACKUP LOG InMemoryOLTP TO DISK = 'c:\temp\test.bak' WITH INIT,COMPRESSION
GO

As shown below, we have ~ 1GB of Log generated since last checkpoint, and after the backup log, the log spaced used % is still at 51%, and there is 1 active data file only which indicates the data/delta file stream has not been flushed to disk given this was a new database.

DMV output
Now when we issue a manual checkpoint, the active files go up to 5 as data/delta files are flushed, and log bytes since last checkpoint go to 0. However, the log spaced used is still at 51% as we are in full recovery model and a backup log will free that up.

-- Manual checkpoint
CHECKPOINT
GO

DMV output

After the backup the Log space used drops to minimal amount as expected.

-- Backup to Free up the log
BACKUP LOG InMemoryOLTP to DISK = 'c:\temp\test.bak' WITH INIT, COMPRESSION
GO

DMV output

Had we waited until 1.5 GB of Log was filled up, no manual checkpoint would have to be issued and the log would be purged based on the size based threshold.

Large checkpoint for memory-optimized tables

Large checkpoints were designed for high throughput systems with very high log generation rates of up-to 1GB/sec. The purpose is to ensure that the checkpoint process would not be continually executing and would scale efficiently.

In the SQL error log, you will see messages indicating that we have detected that is a larger machine and as such by default “Large checkpoints” are used.

 2016-04-29 21:03:58.080               Server   SQL Server detected 4 sockets with 8 cores per socket and 16 logical processors per socket, 64 total logical processors; using 64 logical processors based on SQL Server licensing. This is an informational message; no user action is required.
…
2016-04-29 21:03:58.080               Server   Detected 1048541 MB of RAM. This is an informational message; no user action is required.
…
2016-04-29 21:04:00.110               Server   In-Memory OLTP initialized on highend machine.

 

With Large checkpoints enabled, there are the following behavior changes:

  • Data files created are now 1GB and delta files are 128MB instead of the smaller 128MB/16MB sizes.
  • Automatic checkpoints are executed when 12 GB of the log is filled up since the last checkpoint.

Given this new behavior with SQL Server 2016, there are a few scenarios to consider where large checkpoints do have ramifications:

  • In particular, on test systems where a workload may be executed and then paused at some point (not continuous workload running) you may notice that on small databases you can have 5GB of FREE files pre-created on disk and 12GB of log not cleared even after a log backup.
  • There is the potential big increase in the recovery time if a crash occurs with lots of log between checkpoints. In the case of Large checkpoints, there could be up to 12GB of produced log between checkpoints that may need to be recovered on a failure.

The primary design point of large checkpoints was for the very high throughput systems. If your average Log generation rate is less than 300MB/sec outside of short spikes, you can disable Large checkpoints with the trace flag 9912 by adding it to the startup parameters ( -T9912 ) and the behavior will revert to the prior behavior of checkpoint described without large checkpoints.

To measure log generation rate you can use the perfmon counters below:

  • SQL Server 2016 XTP Transaction Log – Log bytes written\sec
  • MSSQLSERVER: Databases – Log bytes Flushed\sec

If sustained log generation rate is > 300MB/sec do not disable large checkpoints as it can affect performance, if  log generation rate is lower then -T9912 is a great alternative in particular to reduce the risk of increased recovery time and size of the active log.

 

Should you upgrade to SQL Server 2016? Yes. Today. Now!

$
0
0

SQL Server 2016 became generally available starting June 1st, 2016. And you should be upgrading to SQL Server 2016 now, if you haven’t already. SQL Server 2016, obviously provides tons of value to your applications and IT solutions, but if you are uncertain if you should wait a bit longer, then read on.

There are many customers already running SQL Server 2016 in production. Can you guess when the first customer deployed SQL Server 2016 in production? June 2015. Yes, that is not a typo and you read it correctly. The first customer adopted SQL Server 2016 in production a couple days after we released CTP2 (the first production-ready build for SQL Server 2016). Exactly a year back. One year before GA. And, since then, many customers have taken advantage of the great value in SQL Server 2016 and deployed in production. Many of these applications serve core mission-critical business operations for the customers.

Cloud-born and battle-hardened. Azure SQL DB, the SQL Server database service in the cloud, is also an early adopter of SQL Server 2016. 1.7 million Azure SQL DB databases running thousands of varieties of applications of various shapes and sizes have been running SQL Server 2016 for months. The experience of managing millions of databases has helped harden SQL Server 2016.

If you are not running on SQL Server 2016, you are missing out on some really cool features that can not only help improve your business operations, but can help improve the life as database professionals as well. I will discuss some of my favorite features in the “Sweet 16” blog series starting next week. You can also read the SQL Server technical documentation here.

When are you upgrading to SQL Server 2016? What are you waiting for?

Early Customer Experiences with SQL Server R Services

$
0
0

Reviewers: Dotan Elharrar, Sanjay Mishra, Mike Ruthruff, Umachandar Jayachandran, Jeannine Nelson-Takaki

SQLCAT and the SQL Server product team deeply engaged with many early adopter customers over the last year who were interested in actively testing SQL Server 2016 R Services across a variety of scenarios.  These customers saw significant benefits from moving their legacy code to SQL Server R Services.  This blog post provides a mosaic-view of the early adopter customer experiences along with common issues they faced and recommendations on how to avoid common pitfalls.

But first…

This blog post makes some assumptions around your familiarity with SQL Server R Services.  If the topic is entirely new to you, we recommend the following resources in order to get familiar with the SQL Server R Services feature landscape:

In-database Advanced Analytics with R in SQL Server 2016: Basic overview of SQL Server R Services

Getting Started with SQL Server R Services: Books Online, core starting reference

Advanced Analytics with R and SQL: Video from Build conference

SQL Server R Services Tutorials: Books Online tutorials for both the data scientist and database developer audiences

Architecture Overview (SQL Server R Services)

For the architecture overview, be sure to review all three sub-topics – including “R Interoperability in SQL Server R Services”, “New Components in SQL Server to Support R Services”, and the “Security Overview”.

What scenarios did customers want to address?

Early adopter customer industries varied significantly and included energy, investments, credit, health care, real estate, accounting and audit software.  Specific scenarios included:

  • Assessing customer and business risk factors.
  • Predictive maintenance of company machinery.
  • Predictive marketing and campaign optimization.
  • Increasing the efficiency of inventory scheduling, ensuring that inventory doesn’t go to waste and that staffing levels are appropriate.
  • Fraud detection.
  • Sales forecasting.
  • Product valuation based on comparative products and recent sales history.
  • Product pricing recommendations.

Several customers had existing R solutions already in-use, although there was one customer who had their existing linear regression solution implemented via SSAS \ Data Mining Models and SQL Server, and wanted to consolidate down to a single solution using SQL Server R Services.

Why are customers interested?

The motivations behind early adopters exploring SQL Server R Services clustered around the following themes:

  • Legacy performance issues. For this theme, the customer uses R today – but the existing process was too slow and was not scaling sufficiently to meet business demands. The reasons for “slow” varied across customers – but often included:
    • The overhead of pulling large data sets from a remote SQL Server instance to a data scientist workstation (network latency for large result sets). For example, one data scientist team had exploration activities taking days due to data set size and locality.
    • Throughput limits due to single-threaded open source R function usage.
    • Memory constraints of open source R functions (Open-R functions requiring all data in memory in order to process, and no streaming capabilities). For one customer, they were unable to perform a linear model operation against 100 million rows and 25 attributes due to out-of-memory errors on the analytics server.
  • Difficulty in moving an R solution to production. Several of our early adopter data scientists and statisticians used R as their primary tool, however once they finalized a solution, in order to move to production some of our early adopter customers had to rewrite their R code to another language in order to run in production. This re-coding was often performed by a different team (for one customer this was a full re-write to Java).  This re-write takes time and is error-prone. Related to this – we had some customers that had to cobble together multiple technologies in order to provide a single end-to-end solution, so complexity and supportability was a related pain-point.
  • Inefficiencies and communication gaps between the data scientist and database engineer teams. Some of our customers described how they used R in isolation from the SQL Server database engineer teams. With the two teams not always communicating, the data scientist would write R code where procedural T-SQL would have been a better choice and vice-versa. With minimal communication happening between the two roles – data may also get scrubbed independently and redundantly by various data scientists instead of leveraging a centralized data source that is already pre-processed by the database engineering team.  We also found that most database engineers were not familiar with R, classic statistics and machine learning concepts – and conversely, most data scientists and statisticians were not deep T-SQL and SQL Server experts.

Of the motivations to explore SQL Server R Services – improving performance of critical R-related operations was by far the biggest motivating factor, whereas the team-communication-gaps tended to be a second-order theme that we uncovered as we dug deeper into each customer’s scenario and requirements.

What results have customers seen so far?

A few customer experiences seen so far:

  • Performance improvements with model generation and scoring against millions of rows. One customer was able to reproduce their modeling workflow functionality-results within SQL Server R Services and have it execute an attribution selection, segmentation and batch scoring process against a 20 million row table in under 1 hour of duration. Their original legacy methodology took more than two days for the equivalent process.
  • Performance improvements for the generation of many small models. Another customer had a legacy product valuation process which was limited to ~ 200 valuations per hour. This scenario involved the generation of several models per product inventory item based on a small set of comparable inventory items across millions of products. After porting their ~ 700 lines of legacy R code to SQL Server R Services, they were able to scale to ~ 12,000 valuations per minute.
  • Performance improvements using RevoScaleR functions. One customer saw performance improvements for linear regression operations using the RevoScaleR rxLinMod function against 10 million rows with 25 attributes, showing 3,500 seconds to generate the model in the legacy method via the open-source lm function vs. only 12 seconds using RevoScaleR’s rxLinMod and SQL Server R Services.
  • Solution simplification. Performance wasn’t the only positive result. Customers have benefited from the ability to minimize code changes between the data scientist and database developer roles and also simplify and minimize the overall solution’s moving parts.  We also saw benefits from having the two roles collaborate more closely together.

Common issues facing early adopters of SQL Server R Services

This next section will walk through some of the early lessons learned from early adopter customers.

Remembering the Simple Reality Check

One common scenario we saw with early adopter customers was as follows:

  1. The DBA team was in charge of installing SQL Server R Services. So they go through the instructions from Books Online (and sometimes may skip a step as people sometimes do) and then hand off the server to the data scientist and/or statistician.
  2. The data scientist tries using sp_execute_external_script or using pushed-compute context from their IDE-of-choice to the server and then immediate encounter an error message.

The recommendation in this case was to be sure to run a “reality check” test against the SQL Server instance in order to ensure SQL Server R Services is working properly.  For example:

EXECUTE   sp_execute_external_script
@language = N'R'
, @script = N'OutputDataSet <- InputDataSet'
, @input_data_1 = N'SELECT 1 as Col'
WITH RESULT SETS ((col int not null));
GO

This should return a result set with the value “1”.  If instead an error is raised, one valuable troubleshooting resource is as follows:

Known Issues for SQL Server R Services (Books Online)

For more difficult issues, we also asked customers to provide the files from the Log\ExtensibilityLog error log directory for SQL Server R Services under the SQL Server instance error log directory (you can identify this path by executing SELECT SERVERPROPERTY(‘ErrorLogFileName’)).

What packages are installed?

While SQL Server R Services comes out-of-the-box with R base packages and the RevoScaleR package, we found that most customers needed other packages installed in order to fully port their legacy code.  So another common lesson learned was to determine what packages were installed for SQL Server R Services.

The following query shows a list of installed packages on the SQL Server instance:

EXECUTE [sys].[sp_execute_external_script] @language = N'R',
@script = N'
packagematrix <- installed.packages();
NameOnly <- packagematrix[,1];
OutputDataSet <- as.data.frame(NameOnly);'
WITH RESULT SETS((PackageName NVARCHAR (250)));

Alternatively, a data scientist can use an R IDE to check which packages are installed on the SQL Server instance (using that instance for the compute context):

con <- "Driver={SQL Server};Server= SQL1;Database=ContosoRetailDW;Trusted_Connection=true;"

sqlCompute <-
RxInSqlServer(
connectionString = con,
autoCleanup = TRUE,
numTasks = 8,
consoleOutput = TRUE
)
rxInstalledPackages(computeContext = sqlCompute)

Recommended reading on this topic:

Installing and Managing R Packages

Note: We found that some packages could be removed and de-referenced later in the overall testing process by replacing the originally-referenced functions with T-SQL or RevoScaleR equivalents. 

Installing packages without Internet access

Most enterprise SQL Server instances will ideally not be exposed directly to the internet, so for package installation this involves downloading the package in the zipped format ahead of time and then copying the zipped files to a folder accessible on the SQL Server instance machine.

Additionally – packages often have dependencies on other packages, and so you must ensure you have the various packages and associated dependencies installed on the SQL Server instance machine.  This can turn into a very time consuming and manual process – however there is a better solution documented here:

Create a Local Package Repository Using miniCRAN

This solution was developed by Microsoft Program Manager Andre de Vries, who developed the miniCRAN package. If you have more than a few packages to be installed, this solution can greatly speed up the process of preparing the SQL Server R Services environment.

Which package type to download?

A few early adopter customers encountered issues when they downloaded the “Package source” version of a package from https://cran.r-project.org/ instead of the Windows binary version.  For example –they would install the tar.gz file instead of the .zip file.  For some packages, this still worked and installation of the package was permitted, but for other packages, a cryptic error message would be raised.

The takeaway was as follows… To use a package with SQL Server R Services, download the zipped file in the Windows binary format.

R code verification; did it actually work before?

After early adopter customers installed SQL Server R Services and the required packages, the next step was to ensure functional parity between the legacy code and the code run within the context of SQL Server R Services.  This was how we started the in-person SQL/R labs, by first ensuring that, given the same data set and R code, we output the same valid results.

This functionality testing led to early issues with some customers around syntax errors in the R code after migration to an sp_execute_external_script call.  In some cases, the syntax errors were due to copy/paste issues or case-sensitivity problems.  Other times however, we would end up troubleshooting code that had newly-introduced issues prior to encapsulation in sp_execute_external_script.  So one of the common lessons and practices we asked customers to adopt was to have them test the code first in their IDE of choice (for example RStudio or R Tools for Visual Studio (RTVS)) prior to migration – before troubleshooting in the context of sp_execute_external_script.

Selecting an IDE

Most of our early adopter customers were already using RStudio, but some customers expressed interest in Microsoft options as well, and so we pointed them to R Tools for Visual Studio once it became available (RTVS). The overall recommendation was to do most R-code editing in their preferred IDE and then when ready, transfer the R script code to sp_execute_external_script.  We received customer feedback along the way that it was tough to debug the code in its static form in sp_execute_external_script, and hence the recommendation to use an IDE first prior to migration.

Visualization options

While our early adopter customers were greatly interested in various advanced analytic algorithms in service of their business and application needs, they also were also still interested in using the powerful visualizations that R packages natively provide.  That led to the following common questions around how best to render images from R within the context of SQL Server R Services.

For visualizations, there are a few options:

  • Storing the image blob in SQL Server.
  • BCP’ing out the blob to a file.
  • Writing the file to a local file directory.
  • Not storing the image, but instead just calling the stored procedure and rendering the image in real-time via SQL Server Reporting Services and Report Builder.
  • Rendering via Power BI.

Furthermore, you can output both a data set and a scalar varbinary(max) value in a single call to sp_execute_external_script by using OUTPUT parameters.

See the following reference for examples on creating plots using R in T-SQL, returning and storing plots as a varbinary data type and BCP’ing out the file:

Explore and Visualize the Data (In-Database Advanced Analytics Tutorial)

The examples are part of an overall 6-step tutorial that covers an end-to-end In-Database SQL Server R Services scenario.

For another example, the following code creates a correlation plot across columns based on data from the FactInventory table, outputting the file to a temp directory:

EXECUTE [sys].[sp_execute_external_script]
@language = N'R',
@script =
N'
library(corrplot)
corInput <- cor(InputDataSet)

mainDir <- ''C:\\Temp''
dir.create(mainDir, recursive = TRUE, showWarnings = FALSE)

print("Creating correlation plot file:", quote=FALSE)
dest_filename = tempfile(pattern = ''corrplot_'', tmpdir = mainDir)
dest_filename = paste(dest_filename, ''.png'',sep="")
print(dest_filename, quote=FALSE);

png(height=1000, width=1000, file=dest_filename)
corrplot(corInput, method = "pie")
dev.off();
',
@input_data_1 =
N'SELECT OnHandQuantity, StoreKey, ProductKey,
OnOrderQuantity, SafetyStockQuantity, DaysInStock,
MinDayInStock, MaxDayInStock, Aging, InventoryKey
FROM dbo.FactInventory TABLESAMPLE (10 PERCENT) REPEATABLE (2707)'
GO

This produces a graphical display of a correlation matrix across the columns of this table. Figure 1 shows a positive correlation between SafetyStockQuantity and OnHandQuantity and mild negative correlations between DaysInStock and the Aging table columns.  The color and shading help indicate the correlation strength and direction – with deeper blues indicating a positive correlation and deeper reds showing a negative correlation:

corrplot_1f502ded1d52

Figure 1

While data scientists are already accustomed to the rich graphical capabilities that R natively provides, we found that the database engineers will also benefit as well (for example – visualizing query execution characteristics across different compatibility levels for a fixed number of queries, as demonstrated via the ggplot package as follows):

AvgDurationTimeByQueryId

Figure 2

Figure 2 shows average duration by query, along with min/max ranges, across compatibility levels 110, 120, 130 and also 120 with trace flag 4199 enabled.  In this particular scenario, we had Query Store data stored for a regression testing lab engagement and used qplot from the ggplot2 R package to generate the visualization, including using facet_grid(~ compatibility_level) to lay out panels in a grid.

There is a significant amount of untapped potential for DBAs here – and not just with visualizations but also the machine learning algorithms now surfaced via T-SQL (outlier detection in error logs, capacity planning for disk storage, clustering of workload characteristics and more).

We did receive requests to show R visualizations within SQL Server Management Studio natively, similar to geospatial data type output, although that wasn’t a feature that landed in RTM.

Performance optimizations

After working with customers to ensure the functionality seen in SQL Server R Services matches legacy methods, we then tested various methods for maximizing the performance and scalability of the new process.

Most early adopter customers were interested in answering the following questions:

  • How do we scale our advanced analytic processes?
  • How do we performance tune SQL Server R Services?
  • Where are there optimization opportunities for T-SQL, R and RevoScaleR?

A few common themes arose from customer engagements:

  • Separation of the model-and-scoring phases based on application requirements. For example – some customers only needed the model to be generated on a daily basis.  Others even less frequently.
  • Optimization of queries used for R-script ingestion.
  • Modification of the Resource Governor external resource pool settings.
  • Configuration of the batch size.
  • Configuration of parallelism and the number of threads.
  • Algorithm-based configurations – for example – varying the numbers of trees used for rxDForest and finding the right tradeoff between execution duration and model accuracy. For these cases, having a scientist available to determine what was “good enough” was critical.
  • Porting to RevoScaleR equivalents (mentioned earlier in this blog post).

These subjects and more are covered in the following reference:

SQL Server R Services Performance Tuning (Books Online)

Be sure to review all three section topics (“SQL Server Configuration”, “R and Data Optimization”, and “Performance Case Study”).

Beyond the aforementioned performance tuning options, we saw a few string factor-related performance and validity tips based on work in the lab:

  • For one customer scenario, the preference was to explicitly specify the levels for factors using the colInfo argument. When generating column info for the “colinfo” argument via rxCreateColInfo – we found that narrowing down the ingested SELECT list to only the attributes needed significantly reduced the duration (13 hours vs. 5 minutes in one test example).
  • In terms of validity, we initially ran into scoring discrepancies between the legacy model generated and the model generated in SQL Server R Services when generating colInfo definition for RxSqlServerData. The solution was to ensure that sorting of levels was set to TRUE (colInfoDef <- rxCreateColInfo(data = attributeDataColInfo, sortLevels = TRUE)). Otherwise, the default for sortLevels was FALSE.

Leveraging RevoScaleR

Early adopter customers saw significant performance gains for large data-set model generation and scoring activities when leveraging RevoScaleR implementations of some of common Open-R functions. When performing code reviews, we called out scenarios where existing Open-R functions had RevoScaleR equivalents.

The following summary shows rx functions and their nearest Open-R equivalents:

Summary of rx Functions (Books Online)

For example – the lm linear regression function and associated arguments map easily to the rxLinMod RevoScaleR function equivalent.

While there is a significant amount of coverage, not all functions have RevoScaleR equivalents.  One customer wanted to use the rlm “Robust Fitting of Linear Models” function – which did not have a RevoScaleR equivalent.  This didn’t block overall efforts however, since we still were able to use the rlm function in SQL Server R Services and saw significant performance gains due to localizing the data (removing the remote network pull of the data sets from the SQL Server instance to the data scientist application server).

Working with native strengths of both R and SQL Server

One consistent theme we pushed with customer engagements was ensuring that we optimize both from a R and a SQL Server perspective.  There are some areas where SQL Server T-SQL will naturally excel and likewise, there will be areas where R makes most sense.

For example, the following query uses the R command “aggregate” to return total OnHandQuantity by ProductKey:

EXEC sp_execute_external_script
@language = N'R'
, @script = N'
OutputDataSet <-
aggregate(InputDataSet$OnHandQuantity, by=list(ProductKey=InputDataSet$ProductKey), FUN=sum)'
, @input_data_1 = N'SELECT ProductKey, OnHandQuantity FROM dbo.FactInventory'
WITH RESULT SETS ((
[ProductKey] [INT] NOT NULL,
[SumOnHandQuantity] [INT] NOT NULL));
GO

On a laptop with 8 logical schedulers and 16GB of RAM, this query takes 13 seconds to execute.

Contrast this to the following T-SQL semantic equivalent which takes less than one second to execute:

SELECT ProductKey, SUM(OnHandQuantity) AS SumOnHandQuantity
FROM dbo.FactInventory
GROUP BY ProductKey
ORDER BY ProductKey;

Conversely, coding a linear regression operation using T-SQL is unwieldy, verbose and inflexible compared to the R-equivalent.  The following example uses rxLinMod to generate a model based on FactInventory sampled data and then returns the R-Squared value from the mode with just a few lines of code:

EXECUTE [sys].[sp_execute_external_script]
@language = N'R',
@script =
N'
# model the relationship between a dependent variable
# and one or more explanatory variables
InventoryModel <- rxLinMod(OnHandQuantity ~ SafetyStockQuantity, InputDataSet)

# proportion of the variance in the dependent variable that is predictable from the independent variable
InventoryModelDetails <- InventoryModel$r.squared

OutputDataSet <- data.frame(InventoryModelDetails)',
@input_data_1 =
N'SELECT OnHandQuantity, SafetyStockQuantity
FROM dbo.FactInventory TABLESAMPLE (10 PERCENT) REPEATABLE (2707)'
WITH RESULT SETS ((
[r.squared] [FLOAT] NOT NULL));
GO

When porting to SQL Server R Services, we asked customers to take this as an opportunity to refactor code based on the strengths of both T-SQL and R.  This doesn’t just include T-SQL re-writes, but also includes considerations around the queries used for R-script ingestion (for these queries, think about index opportunities, data compression, Columnstore indexes, and the universe of potential SQL Server optimization techniques).

Resource Governance and Monitoring

Hand-in-hand with performance and scalability, customers also had a natural interest in resource governance and how best to manage the collocation of SQL Server R Services workloads with other concurrent activities.  For an overview, see the following Books Online reference:

Resource Governance for R Services (Books Online)

Customers also had questions regarding monitoring both the SQL Server and R-related operations.  Details on this subject can be found here:

Monitoring R Services

Be sure to review the sub-sections covering DMVs, Extended Events and using R Code Profiling functions. For our most recent SQLCAT lab, we used the RPROF function to help understand where the most time was spent on internal function calls (helping us optimize a lengthy R script where it was not immediately evident where the bottlenecks resided).

Customers also used standard perfmon counters for many of the performance tuning exercises, looking at CPU, memory, and I/O patterns both by SQL Server and also R related processes.

Security Considerations

Early adopter customers had several questions around the SQL Server R Services security model and some of these questions helped shape the following Books Online content:

Modify the User Account Pool for SQL Server R Services

Security Considerations for the R Runtime in SQL Server

One specific customer experience to call out… The least-privilege local accounts came as a surprise to one customer’s security team when they saw the twenty new local user accounts, named MSSQLSERVER01 through MSSQLSERVER20, created on the SQL Server machine.  From a database engineer perspective, it is very important to review this topic and have a solid understanding of how the security model works so you can communicate what to expect with technical business partners.

Multiple inputs and outputs from sp_execute_external_script

The sp_execute_external_script procedure allows for a single input data set via the @input_data_1 argument, however it was common for customers to need more than one input data set for various purposes.  For these scenarios, the following options are available:

  • Use rxSQLServerData within the R-Script definition to define one or more data sources (see Create SQL Server Data Objects using RxSqlServerData for examples).
  • Use RODBC within the R-Script definition to define one or more data sources.
  • Define input parameters to sp_execute_external_script that contain JSON data. Then you can use a package like jsonlite to shred the data to individual data frames within the R-Script.

You can mix and match the aforementioned methods and even keep using @input_data_1 in conjunction with these inlined methods – although the general rule of thumb as recommended by the product team is to save @input_data_1 for your largest result set.

Note: For one customer lab, product engineering noticed that the RODBC package’s “sqlSave” function to save the results actually performs a one-row-at-a-time operation.  To optimize for this particular scenario, we removed this code and instead returned the results of sp_execute_external_script as a result set within in a procedure.  The customer then populated a table using an INSERT…EXEC operation instead.

What about the output of multiple result sets?  This was another common customer request, and as of SQL Server 2016, you can choose to:

  • Output nothing (just use the R script to perform various operations and then store results within SQL Server inline via the R script)
  • Output a single result set
  • Return a single result set along with one or more scalar output parameters.

You can also choose to store the result sets directly to SQL Server programmatically via the R-script definition and then retrieve them via separate procedure calls.

Tip of the Iceberg

This blog post just scratches the surface of the capabilities and tips for getting successfully started with SQL Server R Services.  Now that the RTM version of SQL Server 2016 is generally available, we expect more feedback to start flowing in and also expect more themes and patterns to emerge.


SQLSweet16!, Episode 1: Backup Compression for TDE-enabled Databases

$
0
0

 Sanjay Mishra, with contributions from Aasav Prakash

Reviewed by: Denzil Ribeiro, Murshed Zaman, Arvind Shyamsundar, Mike Ruthruff

Celebrating the release of SQL Server 2016, we are starting a new blog series on exciting new scenarios and features. Aptly named “Sweet 16” series, we will discuss sixteen new capabilities in SQL Server 2016 (one on each post) that help improve your business, and enhance your life as database professionals. Among the ocean of new capabilities, we have chosen 16, so will we will miss out on some very good ones. If we miss out your favorite ones, and you would like to hear from us, let us know and we will try our best to cover.

Starting with a hidden-gem: Backup compression for TDE-enabled databases.

Backup Compression and Transparent Data Encryption (TDE) have been two immensely valuable and popular features in SQL Server. Both were released as part of SQL Server 2008. However, had very little co-existence till now. While backup compression worked great for databases that were not enabled for TDE, its effectiveness for TDE-enabled databases was very limited. We described this behavior in our SQL Server 2008 blog post (please refer to pages 22-23 in our old blog collection document), and recommended not to use backup compression for TDE-enabled databases.

These two wonderful features could not stay strangers to each other for long. Starting with SQL Server 2016, you can now get the benefits of backup compression for TDE-enabled databases. Sweet!

This is one of the least publicized features in SQL Server 2016, and has managed to remain under the radar.

Here are some numbers to get you interested. I performed some tests with a database of about 115 GB size (real world data, not synthetic test data) first on SQL Server 2014, and then on SQL Server 2016 on the same server. The results are very impressive on SQL Server 2016.

Figure 1: Backup Compression with TDE (SQL Server 2014)

Figure 1: Backup Compression with TDE (SQL Server 2014)

As illustrated in Figure 1, in SQL Server 2014 (and prior versions) backup compression doesn’t help reduce the backup size of a TDE-enabled database. However, due to the time spent in attempting the compression, the backup takes much longer to complete.

Things change completely in SQL Server 2016. You not only get great compression of the backup file size, but the backup time is reduced significantly as well, illustrated in Figure 2.

Figure 2: Backup Compression with TDE (SQL Server 2016)

Figure 2: Backup Compression with TDE (SQL Server 2016)

The performance numbers presented here are examples. Your mileage will vary based on your data, workload and hardware.

Important to Know

It is important to know that while backing up a TDE-enable database, the compression will kick in ONLY if MAXTRANSFERSIZE is specified in the BACKUP command. Moreover, the value of MAXTRANSFERSIZE must be greater than 65536 (64 KB). The minimum value of the MAXTRANSFERSIZE parameter is 65536, and if you specify MAXTRANSFERSIZE = 65536 in the BACKUP command, then compression will not kick in. It must be “greater than” 65536. In fact, 65537 will do just good. It is recommended that you determine your optimum MAXTRANSFERSIZE through testing, based on your workload and storage subsystem. The default value of MAXTRANSFERSIZE for most devices is 1 MB, however, if you rely on the default, and skip specifying MAXTRANSFERSIZE explicitly in your BACKUP command, compression will be skipped.

My Wish List

My wish list is that backup compression would work for TDE-enabled databases, irrespective of MAXTRANSFERSIZE setting. Specifying MAXTRANSFERSIZE in the BACKUP command may be a simple thing for many, but may be a bit cumbersome for some. Many DBAs rely on scripts generated from SSMS, which doesn’t specify MAXTRANSFERSIZE, and some DBAs who craft their own BACKUP scripts rely on the default value of MAXTRANSFERSIZE. Moreover, the log shipping UI doesn’t provide an option to specify MAXTRANSFERSIZE. It would be nice, if backup compression worked for all, without having to rely on explicitly specifying MAXTRANSFERSIZE.

What is on your wish list?

Call to Action

Many of you have asked for this capability for years. And, now that it is in the product, I urge you to go and test with your databases (especially the larger ones), and share some of your findings – compression ratio, backup time, backup size, etc.

 

 

PolyBase Setup Errors and Possible Solutions

$
0
0

Blog Authors: Murshed Zaman and Sumin Mohanan

Reviewer(s): Barbara Kess

Prologue

PolyBase is a new feature in SQL Server 2016. It was popularized by APS (Microsoft Analytics Platform System) and Azure SQL DW. PolyBase allows access to relational and non-relational data from SQL Server using familiar T-SQL language. It allows you to run queries on external data that resides in Hadoop or Azure blob storage. Optionally, it can push query operations to Hadoop.  If you are interested in learning more about PolyBase, you can look at the PolyBase Guide on MSDN.

PolyBase setup is well documented on MSDN. But since the feature has connectivity to data that resides outside of SQL Server, like Hadoop, some of our customers have had situations for which it was necessary to involve engineering to understand and guide the situation to a successful installation and/or data access.

The purpose of this document is to highlight some of these issues we have seen with various customers and the possible solutions. Keep in mind these are some of the errors that we have seen and mitigated. The error message you see, although it may look the same, may have a different solution.  We are assuming that if you are using PolyBase with Hadoop, you know the basics of Hadoop and know how to look at various Hadoop logs.

Customer Scenario:

SQL Server 2016 or SQL DW connected to Azure blob storage. The CREATE EXTERNAL TABLE DDL points to a directory (and not a specific file) and the directory contains files with different schemas.

SSMS Error:

Select query on the external table gives the following error:

Msg 7320, Level 16, State 110, Line 14
Cannot execute the query “Remote Query” against OLE DB provider “SQLNCLI11” for linked server “(null)”. Query aborted– the maximum reject threshold (0 rows) was reached while reading from an external source: 1 rows rejected out of total 1 rows processed.
(/nation/sensors.ldjson.txt)Column ordinal: 0, Expected data type: INT, Offending value: {“id”:”S2740036465E2B”,”time”:”2016-02-26T16:59:02.9300000Z”,”temp”:23.3,”hum”:0.77,”wind”:17,”press”:1032,”loc”:[-76.90914996169623,38.8929314364726]} (Column Conversion Error), Error: Error converting data type NVARCHAR to INT.

Keep in mind there may be derivations of this error. The name of the first rejected file shows in SSMS with offending data types or values.

Possible Reason:

The reason this error happens is because each file has different schema. The PolyBase external table DDL when pointed to a directory recursively reads all the files in that directory. When a column or data type mismatch happens, this error could be seen in SSMS.

Possible Solution:

If the data for each table consists of one file, then use the filename in the LOCATION section prepended by the directory of the external files. If there are multiple files per table, put each set of files into different directories in Azure Blob Storage and then you can point LOCATION to the directory instead of a particular file. The latter suggestion is the best practices recommended by SQLCAT even if you have one file per table.

Example:

Create External Table foo
(col1 int)WITH (LOCATION=‘/bar/foobar.txt’,DATA_SOURCE);
OR
Create External Table foo
(col1 int) WITH (LOCATION = ‘/bar/’, DATA_SOURCE);

Customer Scenario:

SQL Server 2016 is setup to access a supported Hadoop Cluster. Kerberos security is not enforced in Hadoop Cluster.

SSMS Error:

Select on the external table gives the following error:

Msg 105019, Level 16, State 1, Line 55
EXTERNAL TABLE access failed due to internal error: ‘Java exception raised on call to HdfsBridge_Connect: Error [Unable to instantiate LoginClass] occurred while accessing external file.’
Msg 7320, Level 16, State 110, Line 55
Cannot execute the query “Remote Query” against OLE DB provider “SQLNCLI11” for linked server “(null)”. EXTERNAL TABLE access failed due to internal error: ‘Java exception raised on call to HdfsBridge_Connect: Error [Unable to instantiate LoginClass] occurred while accessing external file.’

DW Engine Server Log:

Interrogation of DWEngine Server log shows the following error:
Note: DWEngine_server.log is usually located under Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\Polybase\

3/15/2016 5:29:01 PM [Thread:16432] [EngineInstrumentation:EngineQueryErrorEvent] (Error, High):
EXTERNAL TABLE access failed due to internal error: ‘Java exception raised on call to HdfsBridge_Connect: Error [com.microsoft.polybase.client.KerberosSecureLogin] occurred while accessing external file.’
Microsoft.SqlServer.DataWarehouse.Common.ErrorHandling.MppSqlException: EXTERNAL TABLE access failed due to internal error: ‘Java exception raised on call to HdfsBridge_Connect: Error [com.microsoft.polybase.client.KerberosSecureLogin] occurred while accessing external file.’ —> Microsoft.SqlServer.DataWarehouse.DataMovement.Common.ExternalAccess.HdfsAccessException: Java exception raised on call to HdfsBridge_Connect: Error [com.microsoft.polybase.client.KerberosSecureLogin] occurred while accessing external file.
at Microsoft.SqlServer.DataWarehouse.DataMovement.Common.ExternalAccess.ExternalHadoopBridge.OpenBridge()
at Microsoft.SqlServer.DataWarehouse.DataMovement.Common.ExternalAccess.HdfsBridgeFileAccess.GetFileMetadata(String filePath)
at Microsoft.SqlServer.DataWarehouse.Sql.Statements.HadoopFile.ValidateFile(ExternalFileState fileState)

Possible Reason:

Kerberos is not enabled in Hadoop Cluster, but Kerberos security is enabled in core-site.xml, yarn-site.xml, or the hdfs-site.xml that resides in SQL Server 2016 (usually located under Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf).

Possible Solution:

Comment out the Kerberos security information from the above mentioned files in the SQL Server 2016 instance.

Customer Scenario:

SQL Server 2016 is setup to access a supported Hadoop Cluster or Azure Blob Storage.

SSMS Error:

Select on an external table shows the following error:

Msg 8680, Level 17, State 5, Line 118
Internal Query Processor Error: The query processor encountered an unexpected error during the processing of a remote query phase.

DWEngine Server Log:

[Thread:5216] [ControlNodeMessenger:ErrorEvent] (Error, High): ***** DMS System has disconnected nodes :
[Thread:5216] [ControlNodeMessenger:ErrorEvent] (Error, High): ***** DMS System has disconnected nodes :
[Thread:5216] [ControlNodeMessenger:ErrorEvent] (Error, High): ***** DMS System has disconnected nodes :

Possible Reason:

The reason for this error could be that SQL Server was not restarted after configuring PolyBase.

Possible Solution:

Restart SQL Server. Check DWEngine Server Log to make sure you don’t see DMS disconnections after the restart.

Customer Scenario:

SQL Server 2016 is connected to an unsecured Hadoop cluster (Kerberos is not enabled). PolyBase is configured to push computation to Hadoop cluster.

Query: select count(*) from foo WITH (FORCE EXTERNALPUSHDOWN);

SSMS Error:

Msg 105019, Level 16, State 1, Line 1
EXTERNAL TABLE access failed due to internal error: ‘Java exception raised on call to JobSubmitter_PollJobStatus: Error [java.net.ConnectException: Call From big1506sql2016/172.16.1.4 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused] occurred while accessing external file.’
OLE DB provider “SQLNCLI11” for linked server “(null)” returned message “Unspecified error”.
Msg 7421, Level 16, State 2, Line 1
Cannot fetch the rowset from OLE DB provider “SQLNCLI11” for linked server “(null)”. .

Hadoop Yarn Log Error:

Job setup failed : org.apache.hadoop.security.AccessControlException: Permission denied: user=pdw_user, access=WRITE, inode=”/user”:hdfs:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5525)

Possible Reason:

With Kerberos disabled, PolyBase will use pdw_user as the user for accessing HDFS and submitting MapReduce jobs.

Possible Solution:

Create pdw_user on Hadoop and give it sufficient permissions to the directories used during mapreduce processing. Also make sure that pdw_user is the owner of the /user/pdw_user HDFS directory.

Below is an example of how to create home directory and assign permissions for pdw_user:

sudo -u hdfs hadoop fs -mkdir /user/pdw_user

sudo -u hdfs hadoop fs -chown pdw_user /user/pdw_user

After this make sure that pdw_user has read, write, and execute permissions on /user/pdw_user directory and /tmp directory has 777 permissions.

Customer Scenario:

SQL Server 2016 PolyBase is setup with Hadoop Cluster or Azure Blob Storage.

SSMS Error:

Any Select query fails with the following error.
Msg 106000, Level 16, State 1, Line 1
Java heap space

Possible Reason:

Illegal input may cause the java out of memory error.  In this particular case the file was not in UTF8 format. DMS tries to read the whole file as one row since it cannot decode the row delimiter and runs into Java heap space error.

Possible Solution:

Convert the file to UTF8 format since PolyBase currently requires UTF8 format for text delimited files.

Customer Scenario:

Customer’s intention is to setup SQL Server 2016 PolyBase to connect to Azure Blob Storage.

SSMS Error:

Msg 105019, Level 16, State 1, Line 74
EXTERNAL TABLE access failed due to internal error: ‘Java exception raised on call to HdfsBridge_Connect: Error [No FileSystem for scheme: wasbs] occurred while accessing external file.’

Possible Reason:

Hadoop connectivity is not set to the configuration value for accessing Azure Blob storage.

Possible Solution:

Set the Hadoop connectivity to a value (preferably 7) which supports Azure Blob Storage and restart SQL Server. List of connectivity values and supported types can be found at https://msdn.microsoft.com/en-us/library/mt143174.aspx.

Customer Scenario:

Trying to export data to Azure blob storage or Hadoop file system using PolyBase with CREATE EXTERNAL TABLE AS SELECT (CETAS) syntax from SQL Server 2016.

SSMS Error:

Msg 156, Level 15, State 1, Line 177
Incorrect syntax near the keyword ‘WITH’.
Msg 319, Level 15, State 1, Line 177
Incorrect syntax near the keyword ‘with’. If this statement is a common table expression, an xmlnamespaces clause or a change tracking context clause, the previous statement must be terminated with a semicolon.

Reason:

CETAS is not a supported statement in SQL Server 2016 for PolyBase.

Solution:

Create the external table first and then use INSERT INTO SELECT to export to the external location. More details can be found at https://msdn.microsoft.com/en-us/library/mt652313.aspx

Epilogue

This blog does not cover all the issues you can encounter when accessing data by using PolyBase. We plan to revise this document regularly with additional scenarios for the benefit of the masses.

SQLSweet16!, Episode 2: Availability Groups Automatic Seeding

$
0
0

Reviewed by: Denzil Ribeiro, Murshed Zaman, Mike Weiner, Kun Cheng, Luis Vargas, Girish Mittur V, Arvind Shyamsundar, Mike Ruthruff

If you have been using Availability Groups (or Database Mirroring prior to that), you are probably used to backing up the primary and restoring (with norecovery) the secondary (first a full database backup, followed by transaction log backups) for creating an Availability Group (AG). Automatic Seeding will come as a breath of fresh air to you.

SQL Server 2016 introduces automatic seeding for Availability Groups, which can significantly ease the experience of creating an AG. Whether you used the AG wizard or scripts to create AGs, you likely created a file share first (where the backup files will be written to, and from where the secondary pick up the backup files to restore). With automatic seeding, gone are the file share and the numerous IOs to the file share. Instead, automatic seeding uses the database mirroring endpoints to stream the bytes (after reading from database files) to the secondary and apply them. Getting rid of the file share and the IO involved with it, automatic seeding dramatically reduces the seeding time for AGs. Moreover, if your AG involves multiple databases, you no longer need to perform backup + restore for each database individually; automatic seeding is a replica level setting and applies to all the databases in the AG.

You can choose mixed seeding techniques in an AG, i.e., use automatic seeding for some replicas and use backup + restore for other replicas.

Creating an AG with Automatic Seeding

As of this writing, the only way to use automatic seeding is using T-SQL. If you use the AG wizard to create AGs, you can generate the T-SQL script from the wizard and then edit it to include the automatic seeding setting. While stepping through the wizard, on the “Select Initial Data Synchronization” page, you need to select the option “Skip initial data synchronization” as shown in Figure 1.

Figure 1: AG wizard selection to generate script for automatic seeding

Figure 1: AG wizard selection to generate script for automatic seeding

This selection is not intuitive, given your ultimate goal, but is a workaround till the future version of the wizard includes an option for automatic seeding. Step through the rest of the wizard, and on the last page of the wizard, DON’T click on “Finish”, rather click on “Script” to generate the T-SQL script to a new query window.

You need to make two changes to the generated script:

  1. Edit the CREATE AVAILABILITY GROUP command to specify automatic seeding (SEEDING_MODE = AUTOMATIC, is the only change you need to do on this statement), as shown in the following code sample (shows only the section of the Create AG script that needs changes):
:Connect <primary instance>

USE [master]
GO

CREATE AVAILABILITY GROUP <AG Name>
WITH (AUTOMATED_BACKUP_PREFERENCE = SECONDARY,
DB_FAILOVER = OFF,
DTC_SUPPORT = NONE)
FOR DATABASE <DB Name>
REPLICA ON
  '<primary instance>' WITH
    (ENDPOINT_URL = 'primary_TCP_endpoint:port',
    FAILOVER_MODE = AUTOMATIC, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT,
    SEEDING_MODE = AUTOMATIC,
    BACKUP_PRIORITY = 50, SECONDARY_ROLE(ALLOW_CONNECTIONS = NO)),
  '<secondary instance>' WITH
    (ENDPOINT_URL = 'secondary_TCP_endpoint:port',
    FAILOVER_MODE = AUTOMATIC, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT,
    SEEDING_MODE = AUTOMATIC,
    BACKUP_PRIORITY = 50, SECONDARY_ROLE(ALLOW_CONNECTIONS = NO));
GO

2. On the secondary replica, grant CREATE ANY DATABASE to the AG, as shown in the code sample below:

:Connect <secondary instance>

ALTER AVAILABILITY GROUP <AG Name> JOIN;
GO

ALTER AVAILABILITY GROUP <AG Name> GRANT CREATE ANY DATABASE;
GO

Execute the full script after these edits to create an AG with automatic seeding.

Adding a replica to an existing AG

To use automatic seeding while adding a replica to an existing AG, add the replica as you would normally do, and after the replica is added, do the following two steps to start the seeding:

  1. On the newly added secondary replica, grant CREATE ANY DATABASE to the AG, as shown in the code sample below:
-- on the newly added secondary

ALTER AVAILABILITY GROUP <AG Name> JOIN;
GO

ALTER AVAILABILITY GROUP <AG Name> GRANT CREATE ANY DATABASE;
GO
  1. On the primary replica, modify the seeding mode for the newly added secondary replica, as shown in the code sample below:
-- on the primary

ALTER AVAILABILITY GROUP <AG Name>
MODIFY REPLICA ON <secondary instance> WITH (SEEDING_MODE = AUTOMATIC)
GO

Automatic Seeding is Faster than Backup-Restore

Automatic seeding eliminates the need for an intermediate file share for writing backup file and then use those backup files for restore. This helps reduce the end-to-end seeding time for an AG. Sweet!

Figure 2 compares the time taken to seed a database of about 115 GB through backup-restore (with backup compression) and automatic seeding.

Figure 2: Automatic seeding faster than backup+restore

Figure 2: Automatic seeding faster than backup+restore

You can reduce the network traffic while seeding by compressing the stream. Use trace flag 9567 to compress the seeding stream. As illustrated in Figure 3, you can see significant reduction in network traffic.

Figure 3: Reduced network traffic using seeding compression

Figure 3: Reduced network traffic using seeding compression

Important to Know

There are a couple of things to be aware of while using automatic seeding.

Transaction Log cannot be Truncated while Seeding

Automatic seeding blocks log truncation. If there is no or little workload on the primary while seeding the secondary, then there may be no impact. But if there is a workload on the primary generating significant transaction log, then you may run the risk of filing up the transaction log file. Be aware of the workload volume, log growth rate, and time to seed the replica(s) while using automatic seeding.

When using automatic seeding be aware that there is a known issue with diagnosing the reason for blocked log truncation. When log truncation is blocked for other reasons, the LOG_REUSE_WAIT_DESC column in the sys.databases view reflects the reason for the hold up. However, when log truncation is blocked due to automatic seeding, the LOG_REUSE_WAIT_DESC column in the sys.databases view shows value of “NOTHING”, indicating that nothing prevents log truncation and the log can actually be truncated (when a log backup is taken), which is misleading. This is a known issue, and is expected to be addressed in the product soon.

Incorrect value in the is_compression_enabled column in sys.dm_hadr_physical_seeding_stats while using compression

The DMV sys.dm_hadr_physical_seeding_stats shows the status of current ongoing seeding activities. The is_compression_enabled column in this DMV reflects whether the seeding is being done with compression (using trace flag 9567) or without compression. However, there is a known issue on this. While seeding with compression, this column shows a value of 0 (meaning no compression), which is incorrect. This known issue is expected to be addressed in the product soon.

My Wish List

Given that automatic seeding reduces the seeding time significantly, it is a very useful feature. I have three items on my wish list to make it practical for more scenarios.

  • The Create AG wizard to have an option to specify automatic seeding. Generating the script through the wizard and then editing it cumbersome and error-prone. An option in the wizard to specify automatic seeding will significantly improve user experience.
  • Should be able to truncate the log while seeding in progress. Filling up the transaction log file or the log drive is nightmare for any database. All activities on the database will come to a halt till log space is cleared. To eliminate that risk, it will be great if automatic seeding can read from transaction log backups, and allow the transaction log to be truncated while the seeding is in progress.
  • The Create AG wizard and the DDL should provide an option to enable compression when using automatic seeding, instead of a trace flag (trace flags are instance level settings, whereas automatic seeding is scoped to an AG).

What is on your wish list?

Call to Action

Try out automatic seeding with your big and highly active databases, and let us know of your learnings, and share some data points, such as the time to seed, amount of compression achieved for the seeding stream (using trace flag 9567), log growth while seeding, etc. For further reading, the blog post from our Support team is a very good read, especially for troubleshooting scenarios.

 

SQLSweet16!, Episode 3: Parallel INSERT … SELECT

$
0
0

Sanjay Mishra, with Arvind Shyamsundar

Reviewed By: Sunil Agarwal, Denzil Ribeiro, Mike Ruthruff, Mike Weiner

Loading large amounts of data from one table to another is a common task in many applications. Over the years, there have been several techniques to improve the performance of the data loading operations. SQL Server 2014 allowed parallelism for SELECT … INTO operations. However, the users needed more flexibility, in terms of placement of the target table, existing data in the target table, etc., which are not possible with the SELECT … INTO statement. Loading data into an existing table (with or without existing data) through an INSERT … SELECT statement has been a serial operation. Until SQL Server 2016.

SQL Server 2016, under certain conditions, allows an INSERT … SELECT statement to operate in parallel, thereby significantly reducing the data loading time for these applications. A hidden gem!

Figure 1 illustrates loading time with and without parallelism. The test was performed on an 8-core machine (Figure 3 shows the degree of parallelism achieved), on a table with 50 million rows. Your mileage will vary.

Figure 1: INSERT ... SELECT with and without parallelism (SQL Server 2016)

Figure 1: INSERT … SELECT with and without parallelism (SQL Server 2016)

Important to Know

Two important criteria must be met to allow parallel execution of an INSERT … SELECT statement.

  1. The database compatibility level must be 130. Execute “SELECT name, compatibility_level FROM sys.databases” to determine the compability level of your database, and if it is not 130, execute “ALTER DATABASE <MyDB> SET COMPATIBILITY_LEVEL = 130” to set it to 130. Changing the compatibility level of a database influences some behavior changes. You should test and ensure that your overall application works well with the new compatibility level.
  2. Must use the TABLOCK hint with the INSERT … SELECT statement. For example: INSERT INTO table_1 WITH (TABLOCK) SELECT * FROM table_2.

There are a few restrictions under which parallel insert is disabled, even when the above requirements are met. We will cover the restrictions, and work arounds, if applicable, in a separate blog post.

How to know you are getting parallelism

The simplest way to check for parallelism is the execution plan. Figure 2 shows an execution plan for an INSERT … SELECT statement without parallelism.

Figure 2: INSERT ... SELECT execution plan without parallelism

Figure 2: INSERT … SELECT execution plan without parallelism

Under appropriate conditions, the same statement can use parallelism, as shown in Figure 3.

Figure 3: INSERT ... SELECT execution plan with parallelism

Figure 3: INSERT … SELECT execution plan with parallelism

Call to Action

Tell us about your application scenarios where you plan to use this functionality. Also if you are already taking advantage of this hidden gem, share some data points, such as data volume, load time, the benefits you are seeing from parallelism, etc.

Transaction dependency limits with memory optimized tables – Error 41839

$
0
0

Reviewed by: Jos de Bruijn,Mike Weiner, Mike Ruthruff , Kun Cheng

As we discussed in a prior blog on considerations around validation errors , part of transaction processing on memory optimized tables include transactions taking dependencies on other transactions during validation and commit processing. With SQL Server 2014 and SQL Server 2016, there is a limit of 8 incoming or outgoing transaction dependencies that can be taken. In other words, up-to 8 transactions can take a dependency on a single transaction TX1 (incoming dependency) or alternatively a transaction can depend on 8 other transactions. If any more than 8 incoming or outgoing dependencies are taken, they fail with the error below:

Error 41839 – “Transaction exceeded the maximum number of commit dependencies and the last statement was aborted. Retry the statement.”

Note: Error 41839 was introduced in SQL Server 2016.
In SQL Server 2014 the error number 41301 covered both normal dependency failures and the scenario discussed here on dependency limits being exceeded, which made it hard to distinguish between the two distinct cases.

The window of time in which the dependency is taken is very short and only during validation and commit processing and most workloads will never run into this. There are valid application scenarios though that can hit this error. One of the most common scenarios where we have seen this error is when a large number of read transactions take a dependency on the same write transaction. This can happen for example if the same record (or range of records) in a table may be frequently updated and at that commit time those same records are accessed by a large number of read threads all trying to query that range.

We are more likely to see this when

  • The write transaction commit takes a longer period of time. Long here is relative, it could be due to Log IO bottleneck when doing the commit or potentially a synchronous Availability group replica which has a commit delay or the statement itself is not indexed appropriately.
  • The read transactions could have a missing index and as a result process more rows or take a longer time.
  • Repeatable read or serializable isolation levels are being used.
  • Artificial workloads (testing) can be more susceptible to it if querying the same record which is not the normal pattern.
  • A more complex case involving many different types of transactions and incoming and outgoing dependencies.

Let us walk through reproducing and troubleshooting the scenario with the script below:

-- Setup script to Create a database and necessary objects
CREATE DATABASE IMOLTP
GO
-- create database with a memory-optimized filegroup and a container.
ALTER DATABASE IMOLTP ADD FILEGROUP IMOLTP_MOD CONTAINS MEMORY_OPTIMIZED_DATA
ALTER DATABASE IMOLTP ADD FILE (NAME='imoltp_mod1', FILENAME='D:\temp\imoltp_mod1')
TO FILEGROUP IMOLTP_MOD
ALTER DATABASE IMOLTP SET MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT=ON
GO
USE IMOLTP
go
--Create Events table and insert a few rows
CREATE TABLE tblEvent(	EventID int not null PRIMARY KEY NONCLUSTERED,
				EventName nvarchar(100),
				EventDescription nvarchar(max)
			) WITH(MEMORY_OPTIMIZED=on)
GO
INSERT INTO tblEvent VALUES (1,'OldEvent',REPLICATE(N'o',30000))
INSERT INTO tblEvent VALUES (2,'NewestEvent',REPLICATE(N'n',30000))
GO

Read Transaction: We will simulate a scenario where there are a large number of readers going after the same record which is the latest event record and executing the statement below

DECLARE @EventName nvarchar(5);
SELECT @EventName= EventName from tblEvent where EventID=2

In order to do that, we will use RML utilities and specifically ostress.exe from the RML utilities to simulate 100 connections doing that same reader type activity

C:\RML Utilities> ostress.exe -SMySQLServerName -E -dIMOLTP -n100 -r100000 -Q"DECLARE @EventName nvarchar(25); SELECT @EventName=EventName from tblEvent where EventID=2" -q

Parameters: -n = 100 threads, -r = 100000 iterations, -S = ServerName , -q = quiet mode , -Q = Query to run.

Write Transaction: We have a transaction that needs to update the description of the event, the update below is not realistic but it can be any transaction that changes that record forcing other transactions to take a dependency on it. The longer the execution of this transaction, the more the dependencies could pile up.

BEGIN TRAN
	update tblEvent set EventDescription = REPLICATE(N'a', 30000) where EventID=2
COMMIT
GO

Once you issue the commit statement above, on the DOS window that was running the select statements, you will see errors such as:

07/04/16 19:33:15.508 [0x0000AA0C] [spid 105] SQLState: 42000, Native Error: 41839, Severity: 16, State: 153, Line: 1 [SQL Server] Transaction exceeded the maximum number of commit dependencies and the last statement was aborted. Retry the statement.

From the individual error itself, all you know is that it has exceeded number of commit dependencies. You can view commit dependencies either via Extended events or for longer running transactions perhaps with sys.dm_db_xtp_transactions DMV.

To help troubleshoot this issue one can create an Extended Event session, detailed below, to track how many dependency errors we are hitting as well as view the statements that take dependencies.

-- Dependency Diagnostics Extended Events
-- Tracks error 41839 as well as TX dependency acquires
CREATE EVENT SESSION [TraceUserDependency] ON SERVER
ADD EVENT sqlserver.error_reported(
    ACTION(sqlserver.session_id,sqlserver.sql_text)
    WHERE ([error_number]=(41839))),
ADD EVENT XtpEngine.dependency_acquiredtx_event(
    ACTION(sqlserver.session_id,sqlserver.sql_text,sqlserver.transaction_id,sqlserver.tsql_frame,sqlserver.tsql_stack)),
ADD EVENT XtpEngine.waiting_for_dependenciestx_event(
    ACTION(sqlserver.session_id,sqlserver.sql_text,sqlserver.transaction_id,sqlserver.tsql_frame,sqlserver.tsql_stack))
ADD TARGET package0.event_file(SET filename=N'd:\temp\TraceUserErrors_Dependency.xel',max_file_size=(250),max_rollover_files=(2)),
ADD TARGET package0.histogram(SET filtering_event_name=N'sqlserver.error_reported',source=N'error_number',source_type=(0))
WITH (STARTUP_STATE=OFF)
GO

-- START
ALTER EVENT SESSION [TraceUserDependency] on SERVER STATE = START

Now Rerun the Read transactions with RML and the write Transactions and you will be able to look at the output of the Extended Events which shows 8 dependencies all on the same transaction ID ( which is our write transaction – see TxIdDependentOn column) from different sessions and different transactions ( see transaction_id column).

Any further dependencies that exceed the 8 limit will fail with the Error 41839. The Extended Event trace output also allows you to see if the transactions failing are performing the same work or are different transactions so that you can tune accordingly.

From the Extended Event histogram you can also see 92 statements hit the error 41839 which indicates there were more than 8 dependencies on this transaction.

Considerations around resolving the dependency errors:

As a first step to help avoid this error, one should evaluate to ensure there are no performance problems such as non-ideal or missing indexes or an IO bottleneck on the transaction log. In order to rule out log related bottleneck, consider setting delayed durability to the database as a test only as that should eliminate latency of writing to the log before the commit processing is complete.

If neither of these are factors in consideration, you can use Trace flag 9926
documented here . This can be enabled as a startup option or enabled on the fly with DBCC TRACEON as shown below.

If you re-run the same steps with the Trace flag 9926 enabled, you will not get that error anymore.

DBCC TRACEON(9926,-1)

GO

If you look at the errorlog, you will see a few messages that are only enabled with that trace flag. These indicate that we have crossed the dependency limit, and a message is written to the log at every factor of 2, when 8,16,32,64… dependencies are exceeded. The message below is specifically for “incoming” dependencies but it can be for “outgoing” dependencies as well.

For more details about transaction handling with memory optimized tables see the MSDN article: https://msdn.microsoft.com/library/mt668435.aspx

Denzil Ribeiro ( @denzilribeiro )

Viewing all 96 articles
Browse latest View live