Randomness

Getting a randomish sample with SQL Server 2000

Ryan — Thu, 09 Feb 2006 19:10:00 GMT

I was recently presented with a seemingly simple task, write a query to return a random* sample of rows from a table, with the number of rows to pull determined at runtime. The surrounding problem is a bit more complex, but this was the specific task I was trying to accomplish, and SQL Server 2000 didn't make it very easy. The rest of my task involved deleting rows where the id wasn't in my random sample.

First try, my almost-dream syntax

My first attempt was a bit naive:

SELECT TOP @limit Id

FROM Items

ORDER BY NEWID()

@limit is a variable which has how many rows I should be returning. I order by NEWID(), which makes a GUID for every row, as this is the best way I know of to get it ordered randomly. I don't think its a terribly good way, but its the best I know. If you have a better way, please comment or email.

That snippet will of course fail, giving this error message:

Incorrect syntax near '@limit'.

Ok, so I can't dynamically set the TOP. Fine.

Use ROWCOUNT

By using SET ROWCOUNT @limit, I could instruct SQL Server to only return @limit rows when performing a SELECT. So, this was the query to delete rows not in my random sample:

--limit results to @limit rows

SET ROWCOUNT @limit

DELETE FROM #temp WHERE Id NOT IN(

--due to rowcount, this should be my random sample of @limit Ids

SELECT Id FROM Items

ORDER BY NEWID()

)

--clear the rowcount, so results aren't limited

SET ROWCOUNT 0

That snippet also fails, giving this error:

The ORDER BY clause is invalid in views, inline functions, derived tables, and subqueries, unless TOP is also specified.

Ok, so I can't use ORDER BY in the subquery. Well damn, that would have been really clean. I could add a TOP 2147483647 in there, but that seemed like a bad idea.

Use a table variable

Ok, next thought is to use a table variable, insert my rows in a random order, numbering each row, and then pull the rows with a number below my limit:

--make the table variable, the num column will count upwards as we insert, numbering the rows

DECLARE @temp TABLE (id int NOT NULL, num int NOT NULL IDENTITY(1,1))

--insert into the table in random order

INSERT INTO @temp (id)

(SELECT Id

FROM Items

ORDER BY NEWID())

--get out our sample

SELECT Id FROM @temp WHERE num <= @limit

That snippet also fails, giving this error:

Incorrect syntax near the keyword 'ORDER'.

Oh right, I can't use ORDER BY in the subquery. Fine.

Use a temporary table and INSERT INTO

I was trying to avoid temporary tables, since they have some performance issues, but figured maybe there wasn't any other way. So, I broke down and used a temp table:

--select into the temp table, using IDENTITY to get my row counter

SELECT Id, IDENTITY(int, 1,1) [num]

INTO #temp

FROM Items

ORDER BY NEWID()

--get out our sample

SELECT Id FROM #temp WHERE num < @limit

--clean up

DROP TABLE #temp

That snippet also fails, giving this error:

Cannot add identity column, using the SELECT INTO statement, to table '#temp', which already has column 'Id' that inherits the identity property.

Ok... So the 'identity' property gets inherited into the temp table, and so I can't specify another identity column. I did some searching in the docs, and there didn't seem to be a way to get an autoincrement field without it being an identity field. Ok, fine, I know a retarded way to get around that.

Use a temporary table, INSERT INTO, and a table variable to mask the identity column

I actually wrote a variant of this:

--Make a table to copy my ids into

DECLARE @retarded TABLE (id int)

--copy all my ids in, but leave the 'identity' property behind

INSERT INTO @retarded (id)

(SELECT Id FROM Items)

--select into the temp table, using IDENTITY to get my row counter

SELECT Id, IDENTITY(int, 1,1) [num]

INTO #temp

FROM @retarded

ORDER BY NEWID()

--get out our sample

SELECT Id FROM #temp WHERE num < @limit

--clean up

DROP TABLE #temp

This worked, but made me sick to my stomach. There had to be a better way.

Use a temporary table, INSERT INTO, and a dummy expression

So, I dug through some more documentation, finding this gem:

When an existing identity column is selected into a new table, the new column inherits the IDENTITY property, unless one of the following conditions is true:

The SELECT statement contains a join, GROUP BY clause, or aggregate function.

Multiple SELECT statements are joined by using UNION.

The identity column is listed more than one time in the select list.

The identity column is part of an expression.

If any one of these conditions is true, the column is created NOT NULL instead of inheriting the IDENTITY property. All rules and restrictions for the identity columns apply to the new table.

Ok, so I decided the least despicable thing to do would be to make a dummy expression to drop the identity property:

--select into the temp table, using IDENTITY to get my row counter

SELECT Id*1 [Id], IDENTITY(int, 1,1) [num]

INTO #temp

FROM Items

ORDER BY NEWID()

--get out our sample

SELECT Id FROM #temp WHERE num < @limit

--clean up

DROP TABLE #temp

I did some profiling, and doing id*1 was the same as doing id+0, so I left it as multiplication.

All in all, very, very frustrating, and I'm not sure if the TOP 2147483647 option above is worse than the hack at the end. Votes are welcome.

I hate how much I jump through hoops and reinvent to do what should be very simple tasks. I mean, c'mon, we've been programming for over 50 years now, and I'm pretty sure pulling a random sample from a dataset is not a new task. We should be better than this.

Crystal Reports 9 Database login prompt

Ryan — Thu, 15 Dec 2005 13:20:00 GMT

One client of mine uses Crystal Reports for reporting, and our experience has been pretty bad overall. Part of that might be due to the guy my client hired to create all of them. He seemed like a nice enough guy, but suffice to say when I was asked to start replacing the Crystal Reports in our existing system there was talk of lawsuits. Every report was made using the design view, and they had a funky web interface with really bad UI displaying all the reports.

Anyway, we had SQL Server replication set up, and then the Crystal Reports were looking at a seperate database server, as they were designed in such a way to be enormously inefficient. So, when replication decided to crap itself, we switched over to log shipping, thinking everything would just keep working. One of the many problems that cropped up was the appearance of a database login prompt:

We tried changing the connection information in the Crystal designer, and the Crystal designer would always pull up a report just fine. The damn web interface would always prompt for information. We'd put in the information we entered in the designer, and then it would work. Since we didn't want to have users entering a user/pass every time they wanted to see a report, we kept changing things, trying to find something that would work. Finally we found that it would work if we pointed the report at our live database server, and ignore the backup we made for Crystal to use. As a temporary solution, we started changing all the reports to look at the other server, and googled like mad to try to find a solution.

After finding many, many people having this problem unresolved, and many, many solutions that are applicable if you're coding these up by hand, but nothing for our problem. We could tell that it was using the wrong config information, but had no idea how to change what it was using.

Solution

The solution came after Chris did a text search on the web server, and found some random ASP file where the user info was stored:

%webroot%\ASP\rPortfolio\HTMLViewers\interactiveViewer.asp

The username and password for the web interface was hardcoded there, and we couldn't find anything in Crystal that could change it. Notepad worked just fine. So, I guess for the web viewer, Crystal silently ignores the settings in the individual reports, and just uses that one. Great design.

new favorite acronym

Ryan — Wed, 05 Oct 2005 15:13:00 GMT

WTFAYMTAAAT:

What The Fuck, Are You Mental, There Already Are A Thousand

I saw this applied to RSS Aggregators.

Theoden picking up the slack.

Ryan — Fri, 04 Mar 2005 13:51:00 GMT

Now my brother Theoden has decided to pick up the torch and dispense his own tips. His first tip describes the brilliant LEN function in SQL Server.

Try SELECT LEN('1234 ') to see what he means.

Supers Bowl XXXIV

Ryan — Wed, 02 Feb 2005 15:15:00 GMT

The party is officially on.

Super Bowl XXXIV

Ryan — Sun, 23 Jan 2005 19:17:00 GMT

It's about time for another party. I'm moving into a new place soon, and we are going to have a traditional beer/food/football party. We're going to have the following extravagent items:

Beer
Hoff is borrowing a meat smoker, through which there will be much smoking of meat.
I'm making penalty flags, and those will be thrown about in drunken rage.
Cox has no fucking HD Fox station, so we can't see it HD, but there will still be the ungodly 52" television.

So, I'm not exactly sure where this party will be, but I'll update once I know where I'm living.

arms race

Ryan — Fri, 14 Jan 2005 12:16:00 GMT

As we get more traffic, we've also been getting some comment spam, and apparently Paul Lockwood is as well. We've had a few ideas between Nathan, Russ, and I of how to stop it, but haven't gotten around to implementing any of them.

Assuming the spammers make comments by just submitting HTTP requests from a perl script or something, we thought of a relatively easy fix. Add some new hidden fields, and have the server read from those hidden fields, and then have some javascript to copy content from the existing text fields to the hidden fields onsumbit. The only hassle there is is the spammers learn what the hidden field names are, and adjust their perl scripts. To combat that we'd have to dynamically generate hidden field names, which could be tossed in viewstate without too much fuss. Then the spammers would have to do web crawling, instantiating and controlling a browser so they could get the names right.

One bad thought we had was to integrate SpamBayes, the excellent, excellent spam filter that you should really be running if you use Outlook. Seed SpamBayes with good comments, then have it show/hide comments according to its statistics. Then there would need to be a section in the admin for you to train the thing, but its certainly doable.

My kingdom for a spec! Arrgh!

Ryan — Tue, 28 Dec 2004 19:05:00 GMT

Bitching about a lack of specification isn't really anything new, but I find myself with little else to do at the moment. To tease the requirements from the client, I've really outdone myself this time. I'm working with a contract management system, and depending on different input, it should do various things automatically. So, I've enumerated the input factors, and have for loops nested 4 deep to hit all the possibilites, printing them to the screen with handy links. My hope is that the client can then click through and say "that one is behaving badly, it should do X". The problem arises when the loops generate 36 test cases, running 216 queries involving nasty things like triggers and nested cursors, and the whole process takes 10 minutes to complete. Hence, the blog entry.

The really depressing part is I'm pretty sure the client isn't going to go through these cases "OK"ing them or rejecting them, and its just another in a long line of tools I've made to get around the lack of a proper spec. Luckily, they're paying by the hour, so the only price being paid is in my patience and sanity. Oh, and this is another impossible deadline project, so while I'm trying to figure out if its what they want, they're shitting bricks because its not done.

Ok, end rant.

Sheesh

Ryan — Mon, 25 Oct 2004 18:58:00 GMT

I've noticed with the recent cold weather, people are fucking up their power clock more:

10/25- 1 fix
10/22- 2 fixes
10/20- 1 fix
10/18- 1 fix
10/15- 1 fix

Pull it together, folks, at some point I should do actual work!
</rant>

Seven Simple Rules for Dealing with your Software Developer Friend

Ryan — Thu, 21 Oct 2004 14:12:00 GMT

Seven Simple Rules for Dealing with your Software Developer Friend. Nice, short read.