Recently I was doing some perfomance tuning for a client, who kept getting timeouts. The timeouts were being caused by some locking queries, with a lock type of "NETWORKIO". This is odd, as everything was running on the same server, so it shouldn't really have much network IO when connecting to 127.0.0.1.

My solution

After much frustraction and googling, I saw someone recommend updating statistics using sp_updatestats to speed up queries, which might resolve NETWORKIO locks faster. That made all the difference. Now the site is zippy as hell. But why did that work?

After some cursory research, I found that sp_updatestats is a convienence function for running UPDATE STATISTICS on every table in the current database. The statistics they talk about are about the keys in each index on the table. The stats are then analyzed by the query optimizer to get the best usage of indexes. If you do a "display estimated execution plan" on a query, the stats are whats used to make those estimates. Then it made sense.

In an earlier attempt to speed everything up, I had taken a trace of the activity and ran the index tuning wizard on about an hours worth of actual usage. Applying the recommended indexes sped up the site nicely, but the real benefit wasn't realized because the statistics on the new indices hadn't been calculated, so the query optimizer couldn't do its job very well. After getting my stats lined up, the optimizer kicked in and everything was very responsive.

Conclusion

If you change your indexes about, run EXEC sp_updatestats. There is an option on a database maintanence plan to do this, and I believe I'm going to start enabling it.

I received a helpful comment on my last entry, and sat down to try out this seemingly ideal solution:

Setup and basic test

I started by creating a table variable, and populating it with some data:

Subquery test

Next test is my goal, the randomish sample. I make the 10 row random sample in a subquery, and then delete everything not in that sample. Yes, this is kinda backwards, but the actual code is more complicated than my simplification here, so its easier to generate all the possibilities then remove all that don't fall into our sample.

TOP N	Rows left in @limiter
TOP 20000	2000
TOP 2000	2000
TOP 1999	1999
TOP 1998	1997
Huh? The top 1998 should have returned 1998 rows; the 2 rows not in that sample should have been deleted. Lemme run that again to make sure I didn't mess up.
TOP 1998	2000
What the hell? Is this some weird query caching thing? Let me run it a few more times...
TOP 1998	1998
TOP 1998	1999
Seriously, WTF. The execution plan just creates more questions: It's somehow processing 4M rows from a table variable with 2K rows!? Then the TOP 1998 gets performed in the Filter operation, which reduces it to 1999 rows? Ok, so 2K * 2K = 4M, so the WHERE Id NOT IN () is effectively performing a cross join, which makes sense, but why the 1999 rows?
TOP 10	1990

TOP in a subquery is broken?

Consider this repro:

Out with table variables

So, back to the beginning:

Fine, you win, SQL Server

Fine. If you want it that badly MSSQL, you can have it. I'll keep my slimy solution from the last entry (for now), as you obviously don't want to let me do anything in a straighforward manner. I'll try to forget the inconsistencies I found today, but it'll take time to heal. Be patient with me. And don't get all jealous if you see me installing postgresql to see if it can do this better than you. You know I'm locked in to you.

I was recently presented with a seemingly simple task, write a query to return a random* sample of rows from a table, with the number of rows to pull determined at runtime. The surrounding problem is a bit more complex, but this was the specific task I was trying to accomplish, and SQL Server 2000 didn't make it very easy. The rest of my task involved deleting rows where the id wasn't in my random sample.

First try, my almost-dream syntax

My first attempt was a bit naive:

That snippet will of course fail, giving this error message:

Use ROWCOUNT

By using SET ROWCOUNT @limit, I could instruct SQL Server to only return @limit rows when performing a SELECT. So, this was the query to delete rows not in my random sample:

Use a table variable

Ok, next thought is to use a table variable, insert my rows in a random order, numbering each row, and then pull the rows with a number below my limit:

Use a temporary table and INSERT INTO

I was trying to avoid temporary tables, since they have some performance issues, but figured maybe there wasn't any other way. So, I broke down and used a temp table:

Use a temporary table, INSERT INTO, and a table variable to mask the identity column

I actually wrote a variant of this:

Use a temporary table, INSERT INTO, and a dummy expression

So, I dug through some more documentation, finding this gem:

When an existing identity column is selected into a new table, the new column inherits the IDENTITY property, unless one of the following conditions is true:

• The SELECT statement contains a join, GROUP BY clause, or aggregate function.
• Multiple SELECT statements are joined by using UNION.
• The identity column is listed more than one time in the select list.
• The identity column is part of an expression.

If any one of these conditions is true, the column is created NOT NULL instead of inheriting the IDENTITY property. All rules and restrictions for the identity columns apply to the new table.

All in all, very, very frustrating, and I'm not sure if the TOP 2147483647 option above is worse than the hack at the end. Votes are welcome.

I hate how much I jump through hoops and reinvent to do what should be very simple tasks. I mean, c'mon, we've been programming for over 50 years now, and I'm pretty sure pulling a random sample from a dataset is not a new task. We should be better than this.

So, I was seeing some weirdness with NOT IN, and reduced it to this minimal case:

SELECT 'foo'
WHERE 1 NOT IN (NULL, 2)
---- 
(0 row(s) affected)

SELECT 'foo'
WHERE 1 IN (NULL, 2)
---- 
(0 row(s) affected)

SELECT 'foo'
WHERE NOT(1 IN (NULL, 2))
---- 
(0 row(s) affected)

SELECT 'foo'
WHERE 1 IN (NULL, 2, 1)
---- 
foo

(1 row(s) affected)

SELECT 'foo'
WHERE 1 NOT IN (2, 3)
---- 
foo

(1 row(s) affected)

From the MSDN docs (paraphrased):

a list of expressions to test for a match. All expressions must be of the same type as the test_expression.

SELECT 'foo'
WHERE 1 NOT IN (CAST(NULL as int), 2)
---- 
(0 row(s) affected)

I guess this means anywhere I'm using NOT IN with a subquery, I need to be sure to wrap the selected column in an ISNULL. Can anyone tell me I'm wrong on this? Pretty please?

A lot of my job involves importing data, and over the years I've found a few nice ways to do this. As Monday was a holiday (and I didn't want to do any actual work) I figured I'd spend some time to put more of my brain on the web. As always, this is only my personal experiences about what seems to work for me, and if you know of better ways to do it, please leave some comments.

There are couple of major cases I've come across the need to import data, and they should be treated a little differently:

• Initially importing a customer's old data into a new database schema.
• Habitual imports of data from other systems.

These tend to be a hairy. The data you get from the customer is often is an inconvienent format, and in sorry shape. The whole reason they need a new database is because their old one is god-awful, and doesn't meet their needs. Here we have to deal with schema changes, invalid data, inconsistent relations, and usually a bunch of data the customer isn't really going to use, but needs in there anyway.

For these large imports you want to automate everything you can. You're obviously going to have some script or program to select from one and insert into the other, and you want to make the import a one step operation. You want to push the button then go get a beer and pray it worked.
There are two ways I've approached this, and they have their ups and downs.

Get it all in the DB, then munge.

Copy the source data, in it's original schema, into your database, and then run queries to convert the old schema to the new schema.
On SQL Server, DTS packages are a good way to automate this. An easy way to make one is to use Enterprise Manager:

1. Browse to the table list for your database.
2. Right click, choose All Tasks->Import data
3. Follow the wizard, choosing data sources and data destinations. When you choose destination tables, make sure you don't have any naming conflicts. To make it easy for me to tell what is imported, I often prefix the imported table names with "imp_" or "import_".
4. At the end of the wizard, you have the option to save the process. Do so however you choose, DTS packages are pretty easy to deal with, so that's my preference.

• Be generous with PRINT statments and comments. I frequently use PRINT statements as comments, so when its actually running I can easily track the progress and have a decent idea of where it breaks. I use Query Analyzer mostly, and that is notoriously bad about what line the script failed on.
• Use set operations as much as you can. The major advantage of importing the old data/schema intact is to allow set operations, which is absurdly faster than looping over a dataset.
• Make a clean up script that wipes your destination database. This can be useful after a failed import, or when the customer gives you a newer copy of the source data.
• Wrap your import in a transaction. This allows you to recover automatically from failures. If you don't, then you'll need to manually run a clean up script after each failure.
• Keep an eye on your transaction logs. They can get out of control if they aren't restricted, and an afternoon of debugging an import script can balloon the thing to preposterous sizes. Clear the log regularly, possibly as part of the cleaning script.

• Changing column types. The old database may have been storing numeric data as text. Before getting the old data in, you'll need to correct all the type conversion errors first. Manually search the old data for problems, using WHERE ISNUMERIC(column) = 0. Once you find the problem rows, add some UPDATE statements into your import script. You'll end up with a lot of statements like this:

  UPDATE import_munchkin_leagues SET

  members = '4'

  WHERE members = '4r'

  Now that your data can be converted, I like using DDL statements to evolve the old schema, making it closer to the destination schema.

  ALTER TABLE import_munchkin_leagues ALTER COLUMN members int;

  ALTER TABLE import_munchkin_leagues ALTER COLUMN dues money

  Now the actual data import is trivial:

  INSERT INTO MunchkinLeagues (Members, Name, Dues)

  (SELECT members, league_name, dues

  FROM import_munchkin_leagues)

  You could do the conversions in the SELECT statement, but I find it a lot easier to think about and debug as seperate steps.
• Making lookup tables. A simple

  INSERT INTO SlipperTypes (Name)

  (SELECT DISTINCT slipper_type FROM import_witches)

  works in a lot of cases, but when types are entered by users, you'll end up with a lot of near-duplicates ("Ruby" versus "ruby" versus "rubyt") that will need to just be hashed out with the client. Making a simple tool to let the customer replace all uses of a bad type ("rubyt") with a proper type ("Ruby") is a good way to easily fix these, and that task can often be put back on the client.
• Maintaining relationships. If you have more than a flat table to import, then you'll need to maintain the relationships. I like adding a column to an old table that links to it's counterpart in the new schema. After copying the rows from the old table to the new table, update that column the old table to point at it's corresponding row in the new table. I usually need one of those bridges for each major table in the old schema.
• Normalizing. A lot of times the old schema will have columns like: "Color1", "Color2", "ColorN" to get a M-N relationship. Usually the new schema will have a "Colors" table, with a helper table to model the many-to-many relation. Use UNIONs to convert those columns to rows:

  INSERT INTO Colors (Name)

  (SELECT Color1 FROM import_horses

  UNION

  SELECT Color2 FROM import_horses

  UNION

  SELECT Color3 FROM import_horses)

  In SQL Server, the UNION operator eliminates duplicates, but I'm not sure about other RDBMS, you might need a DISTINCT in there. Now, assuming you've added a column linking to your destination table, you can seed your many-to-many table to import the relationship:

  INSERT INTO HorseColors (ColorId, HorseId)

  (

  SELECT Colors.Id, HorseId FROM Colors

  JOIN (

  SELECT Color1, HorseId FROM import_horses

  UNION

  SELECT Color2, HorseId FROM import_horses

  UNION

  SELECT Color3, HorseId FROM import_horses

  ) ih ON ih.Color1 = Colors.Name

  )

Munge in object space

O/R mappers are pretty common now, and sometimes its quicker to write a console application, and run the imports in object space. Odds are you already have a data access layer of some kind for your new database. There are a few pros and cons.
Pros

• If the DALs are generated, this can be faster to write.
• Sometimes its easier to think in object space.

• Speed. All the data is having to go into your app, then to the database. You also can't easily do set operations in most OO languages, so everything gets iterated over at some point. I had one import application run for about 4 days.

I'll put down my thoughts on habitual imports some other time. For now, I need to do some of that actual work.

tells the driver to always read "intermixed" data columns as text

1. Open up Query analyzer, connected as a user with admin priveleges.
2. Run this query, substituting "db_name" with the proper database name:

   BACKUP LOG db_name WITH NO_LOG
   GO
   DBCC SHRINKDATABASE( db_name, 0)
   GO

3. Check the size of the log file