Programming

RE: Lisp Library Management

Ryan — Fri, 26 May 2006 11:37:00 GMT

Gary King has been working on getting a good library system for lisp, and recently posted his Notes on a Lisp Library Management. I was going to email him back, but figured here was as good a place as any to give my comments. Read that first.

Firstly, I applaud Gary for taking up this mantle. I'm really still starting using lisp, and the lack of authoritative libraries is a hinderer for people like me. Getting a comprehensive library management system is going to help folks like me get up to speed with lisp faster. ASDF is a good start, but I think it needs to be built upon. Secondly, sorry if everything I'm saying here has already been considered.

The problem of getting libraries installed and updated with minimal user effort is a very studied problem with several decent implementations that could be used as starting points.

I think a lot of lessons can be learned by studying perl's CPAN. CPAN has mirrors that automatically sync, and each user picks a list of mirrors near them. Then you just ask for the package name, and it will download, make, run tests, then put the code in a proper system or user path such that programs can import the library. Perhaps a port of CPAN would solve most of these problems.

Another good example could be Debian's apt-get. I think it's command line interface almost perfect, and since many lisp developers are using debian based sytems, the interface would be very familiar. As a client of a lisp library management service, here's what I think would be ideal, presented as a mock shell session (with user input in bold):

~$ asdf install clsql
Looking up definition for package clsql... package found.
GPG key not present, downloading from https://www.b9.com/kevin.gpg.asc... done.
Downloading clsql from http://files.b9.com/clsql/clsql-3.6.1.tar.gz... done.
Checking signature for clsql-3.6.1.tar.gz... file verified.
Checking dependencies...
  clsql depends upon uninstalled package uffi, install uffi? [ynaq] y
    Looking up definition for package uffi... package found.
    Downloading uffi from http://files.b9.com/uffi/uffi-1.5.13.tar.gz... done.
    Checking signature for uffi-1.5.13.tar.gz... file verified.
    Checking dependencies... done.
    Dependencies met.
    Building uffi... done.
    uffi (1.5.13) installed.
Dependencies met.
Building clsql... done.
clsql (3.6.1) installed.
~$ asdf upgrade
Scanning installed packages.
Checking clsql (3.6.1)... up to date.
Checking uffi (1.5.13)... up to date.
Checking my-unversioned-lib (20060503)... outdated!
  Looking up definition for package my-unversioned-lib... package found.
  Downloading my-unversioned-lib from http://example.org/my-unversioned-lib.tar.gz... done.
  Checking signature for my-unversioned-lib.tar.gz... file verified.
  Checking dependencies... done.
  Dependencies met.
  Building my-unversioned-lib... done.
  my-unversioned-lib (20060526) installed.
Upgrade complete, 3 packages processed.
  1 package upgraded.
  2 packages up to date.

Anyway, enough pipe-dreaming for me.

NETWORKIO locks on SQL Server 2000

Ryan — Fri, 12 May 2006 13:52:00 GMT

Recently I was doing some perfomance tuning for a client, who kept getting timeouts. The timeouts were being caused by some locking queries, with a lock type of "NETWORKIO". This is odd, as everything was running on the same server, so it shouldn't really have much network IO when connecting to 127.0.0.1.

My solution

After much frustraction and googling, I saw someone recommend updating statistics using sp_updatestats to speed up queries, which might resolve NETWORKIO locks faster. That made all the difference. Now the site is zippy as hell. But why did that work?

After some cursory research, I found that sp_updatestats is a convienence function for running UPDATE STATISTICS on every table in the current database. The statistics they talk about are about the keys in each index on the table. The stats are then analyzed by the query optimizer to get the best usage of indexes. If you do a "display estimated execution plan" on a query, the stats are whats used to make those estimates. Then it made sense.

In an earlier attempt to speed everything up, I had taken a trace of the activity and ran the index tuning wizard on about an hours worth of actual usage. Applying the recommended indexes sped up the site nicely, but the real benefit wasn't realized because the statistics on the new indices hadn't been calculated, so the query optimizer couldn't do its job very well. After getting my stats lined up, the optimizer kicked in and everything was very responsive.

Conclusion

If you change your indexes about, run EXEC sp_updatestats. There is an option on a database maintanence plan to do this, and I believe I'm going to start enabling it.

Ball of hate, now with Flash™

Ryan — Wed, 05 Apr 2006 17:28:00 GMT

So today I had a client call and ask why their site had errors. They had a secure certificate, and IE was giving a popup:

"This page contains both secure and non-secure items. Do you want to download the non-secure items?"

So, I pull up the page, and check the source, figuring there's some <img src="http://..." somewhere in there. A ctrl-f later, I see there are no non-secure images. The only things not using https are links to other sites, and some crap in the code ghetto that is the markup needed for a flash movie. Specifically, I saw:

<OBJECT classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"

codebase="http://download.macromedia.com/pub/[blah]"

WIDTH="88" HEIGHT="31" id="PoweredBy_88_31_A"

ALIGN="" VIEWASTEXT>

[40 lines of garbage]

<EMBED SRC="[flash file]"

swLiveConnect=FALSE WIDTH=228 HEIGHT=287

QUALITY=AUTOLOW MENU=false PLAY=true

BGCOLOR=#FFFFFF TYPE="application/x-shockwave-flash"

PLUGINSPAGE="http://www.macromedia.com/[blah]">

</EMBED>

So, on Nathan's suggestion, I tried changing this boilerplate mess to make the pluginspace and codebase attributes point to https. That made IE stop throwing the error. I'm not sure why it was downloading things from those URLs (as the error message clearly stated), but now its happy, and my client's clients are no longer scared to use the site.

But the fun doesn't end there. I took a stress break and ran across a blog entry by Jason about his ActiveX update woes. Apparently to avoid paying licensing fees, Microsoft is changing how IE loads plugins (including flash), and unless developers (developers, developers, developers!) change how they render Flash, the users will be presented with a "Click here to activate this" dialog instead of just showing the flash.

Apparently you have to "activate" a plugin before it will run. More information can be found here: Internet Explorer Eolas changes and the Flash plugin.

Like any good security measure, this can be instantly circumvented from the server (attacker) side. Apparently if you set it up with javascript, then it doesn't need to be activated. I guess there's no such thing as malicious javascript. Microsoft has their workaround: activating ActiveX controls, and the author of the helpful article above has a widely used Flash rendering javascript library that works too, called FlashObject. He's written a few books on flash, and his library looks pretty nice, both hiding the <OBJECT> mess and gracefully degrading to plain HTML if the flash plugin is absent.

Thanks Microsoft, thanks to your change (and Jason's post) I've discovered some nice open source library to remove a slurry of meaningless markup, but I would really prefer not to have to do your legal work.

Getting a random sample with SQL Server 2000 revisted

Ryan — Fri, 24 Feb 2006 11:38:00 GMT

I received a helpful comment on my last entry, and sat down to try out this seemingly ideal solution:

--limit results to @limit rows

SET ROWCOUNT @limit

DELETE FROM #temp WHERE Id NOT IN(

--due to rowcount, this should be my random sample of @limit Ids

SELECT TOP 100 PERCENT Id FROM Items

ORDER BY NEWID()

)

--clear the rowcount, so results aren't limited

SET ROWCOUNT 0

It didn't work, and I got really confused when I tried to figure out why, so I started making some simple tests.

Setup and basic test

I started by creating a table variable, and populating it with some data:

DECLARE @limiter TABLE (id int not null)

--get our set into storage

INSERT INTO @limiter (id) (

SELECT TOP 2000 Id

FROM Items

)

Next step, lets see if TOP 100 PERCENT works with ROWCOUNT the way I'd like it to:

SET ROWCOUNT 10

SELECT TOP 100 PERCENT id FROM @limiter

ORDER BY NEWID()

Running this gives me 10 rows, and running it again give me 10 different rows, so my randomish requirement is met.

Subquery test

Next test is my goal, the randomish sample. I make the 10 row random sample in a subquery, and then delete everything not in that sample. Yes, this is kinda backwards, but the actual code is more complicated than my simplification here, so its easier to generate all the possibilities then remove all that don't fall into our sample.

--limit SELECT results

SET ROWCOUNT 10

DELETE FROM @limiter WHERE Id NOT IN (

SELECT TOP 100 PERCENT id FROM @limiter

ORDER BY NEWID()

)

--clear the rowcount, so results aren't limited

SET ROWCOUNT 0

SELECT * FROM @limiter

I expect this to return 10 rows. It returns 2000, the full contents of @limiter. That implies the TOP 100 PERCENT in the subquery returned the full 2000 rows, so nothing was deleted. Checking the execution plan verifies this. SET ROWCOUNT doesn't work on subqueries? Or maybe the TOP 100 PERCENT is confusing SQL Server, so I'll try using a TOP N:

SET ROWCOUNT 10

DELETE FROM @limiter WHERE Id NOT IN (

SELECT TOP 20000 id FROM @limiter

ORDER BY NEWID()

)

--clear the rowcount, so results aren't limited

SET ROWCOUNT 0

SELECT COUNT(id) FROM @limiter

Nope, still get 2000 rows in @limiter after the delete. For grins, I start playing with the TOP statement, and get some confusing results:

TOP N	Rows left in @limiter
TOP 20000	2000
TOP 2000	2000
TOP 1999	1999
TOP 1998	1997
Huh? The top 1998 should have returned 1998 rows; the 2 rows not in that sample should have been deleted. Lemme run that again to make sure I didn't mess up.
TOP 1998	2000
What the hell? Is this some weird query caching thing? Let me run it a few more times...
TOP 1998	1998
TOP 1998	1999
Seriously, WTF. The execution plan just creates more questions: It's somehow processing 4M rows from a table variable with 2K rows!? Then the TOP 1998 gets performed in the Filter operation, which reduces it to 1999 rows? Ok, so 2K * 2K = 4M, so the WHERE Id NOT IN () is effectively performing a cross join, which makes sense, but why the 1999 rows?
TOP 10	1990

Ok, so something is seriously inconsistent with ROWCOUNT and TOP in subqueries. When I take the SET ROWCOUNT out, it still is inconsistent. When I take out the ORDER BY NEWID(), it is still inconsistent.

TOP in a subquery is broken?

Consider this repro:

DECLARE @t TABLE (id int not null)

INSERT INTO @t (id) (

SELECT TOP 20 Id FROM [someTable]

)

SELECT COUNT(Id) FROM @t

WHERE Id IN (

SELECT TOP 10 Id FROM @t

ORDER BY NEWID()

)

What do you think the count should be? Hint: there's more than one answer. Run that query repeatedly, and bask in inconsistency. But... try this one:

SELECT COUNT(Id) FROM [someTable]

WHERE Id IN (

SELECT TOP 10 Id FROM [someTable]

ORDER BY NEWID()

)

And bam, we get 10 every time. So its not TOP and ORDER BY in subqueries, maybe its something to do with table variables? So, with that in mind, lets go back to the original test, but using a temp table instead of a table variable.

Out with table variables

So, back to the beginning:

SELECT TOP 2000 Id INTO #limiter

FROM Items

--limit SELECT results

SET ROWCOUNT 10

DELETE FROM #limiter WHERE Id NOT IN (

SELECT TOP 100 PERCENT id FROM #limiter

ORDER BY NEWID()

)

--clear the rowcount, so results aren't limited

SET ROWCOUNT 0

SELECT COUNT(id) FROM #limiter

Result: 1990 rows. It should have 10. So, table variables aren't the problem.

Fine, you win, SQL Server

Fine. If you want it that badly MSSQL, you can have it. I'll keep my slimy solution from the last entry (for now), as you obviously don't want to let me do anything in a straighforward manner. I'll try to forget the inconsistencies I found today, but it'll take time to heal. Be patient with me. And don't get all jealous if you see me installing postgresql to see if it can do this better than you. You know I'm locked in to you.

Getting a randomish sample with SQL Server 2000

Ryan — Thu, 09 Feb 2006 19:10:00 GMT

I was recently presented with a seemingly simple task, write a query to return a random* sample of rows from a table, with the number of rows to pull determined at runtime. The surrounding problem is a bit more complex, but this was the specific task I was trying to accomplish, and SQL Server 2000 didn't make it very easy. The rest of my task involved deleting rows where the id wasn't in my random sample.

First try, my almost-dream syntax

My first attempt was a bit naive:

SELECT TOP @limit Id

FROM Items

ORDER BY NEWID()

@limit is a variable which has how many rows I should be returning. I order by NEWID(), which makes a GUID for every row, as this is the best way I know of to get it ordered randomly. I don't think its a terribly good way, but its the best I know. If you have a better way, please comment or email.

That snippet will of course fail, giving this error message:

Incorrect syntax near '@limit'.

Ok, so I can't dynamically set the TOP. Fine.

Use ROWCOUNT

By using SET ROWCOUNT @limit, I could instruct SQL Server to only return @limit rows when performing a SELECT. So, this was the query to delete rows not in my random sample:

--limit results to @limit rows

SET ROWCOUNT @limit

DELETE FROM #temp WHERE Id NOT IN(

--due to rowcount, this should be my random sample of @limit Ids

SELECT Id FROM Items

ORDER BY NEWID()

)

--clear the rowcount, so results aren't limited

SET ROWCOUNT 0

That snippet also fails, giving this error:

The ORDER BY clause is invalid in views, inline functions, derived tables, and subqueries, unless TOP is also specified.

Ok, so I can't use ORDER BY in the subquery. Well damn, that would have been really clean. I could add a TOP 2147483647 in there, but that seemed like a bad idea.

Use a table variable

Ok, next thought is to use a table variable, insert my rows in a random order, numbering each row, and then pull the rows with a number below my limit:

--make the table variable, the num column will count upwards as we insert, numbering the rows

DECLARE @temp TABLE (id int NOT NULL, num int NOT NULL IDENTITY(1,1))

--insert into the table in random order

INSERT INTO @temp (id)

(SELECT Id

FROM Items

ORDER BY NEWID())

--get out our sample

SELECT Id FROM @temp WHERE num <= @limit

That snippet also fails, giving this error:

Incorrect syntax near the keyword 'ORDER'.

Oh right, I can't use ORDER BY in the subquery. Fine.

Use a temporary table and INSERT INTO

I was trying to avoid temporary tables, since they have some performance issues, but figured maybe there wasn't any other way. So, I broke down and used a temp table:

--select into the temp table, using IDENTITY to get my row counter

SELECT Id, IDENTITY(int, 1,1) [num]

INTO #temp

FROM Items

ORDER BY NEWID()

--get out our sample

SELECT Id FROM #temp WHERE num < @limit

--clean up

DROP TABLE #temp

That snippet also fails, giving this error:

Cannot add identity column, using the SELECT INTO statement, to table '#temp', which already has column 'Id' that inherits the identity property.

Ok... So the 'identity' property gets inherited into the temp table, and so I can't specify another identity column. I did some searching in the docs, and there didn't seem to be a way to get an autoincrement field without it being an identity field. Ok, fine, I know a retarded way to get around that.

Use a temporary table, INSERT INTO, and a table variable to mask the identity column

I actually wrote a variant of this:

--Make a table to copy my ids into

DECLARE @retarded TABLE (id int)

--copy all my ids in, but leave the 'identity' property behind

INSERT INTO @retarded (id)

(SELECT Id FROM Items)

--select into the temp table, using IDENTITY to get my row counter

SELECT Id, IDENTITY(int, 1,1) [num]

INTO #temp

FROM @retarded

ORDER BY NEWID()

--get out our sample

SELECT Id FROM #temp WHERE num < @limit

--clean up

DROP TABLE #temp

This worked, but made me sick to my stomach. There had to be a better way.

Use a temporary table, INSERT INTO, and a dummy expression

So, I dug through some more documentation, finding this gem:

When an existing identity column is selected into a new table, the new column inherits the IDENTITY property, unless one of the following conditions is true:

The SELECT statement contains a join, GROUP BY clause, or aggregate function.

Multiple SELECT statements are joined by using UNION.

The identity column is listed more than one time in the select list.

The identity column is part of an expression.

If any one of these conditions is true, the column is created NOT NULL instead of inheriting the IDENTITY property. All rules and restrictions for the identity columns apply to the new table.

Ok, so I decided the least despicable thing to do would be to make a dummy expression to drop the identity property:

--select into the temp table, using IDENTITY to get my row counter

SELECT Id*1 [Id], IDENTITY(int, 1,1) [num]

INTO #temp

FROM Items

ORDER BY NEWID()

--get out our sample

SELECT Id FROM #temp WHERE num < @limit

--clean up

DROP TABLE #temp

I did some profiling, and doing id*1 was the same as doing id+0, so I left it as multiplication.

All in all, very, very frustrating, and I'm not sure if the TOP 2147483647 option above is worse than the hack at the end. Votes are welcome.

I hate how much I jump through hoops and reinvent to do what should be very simple tasks. I mean, c'mon, we've been programming for over 50 years now, and I'm pretty sure pulling a random sample from a dataset is not a new task. We should be better than this.

Crystal Reports 9 Database login prompt

Ryan — Thu, 15 Dec 2005 13:20:00 GMT

One client of mine uses Crystal Reports for reporting, and our experience has been pretty bad overall. Part of that might be due to the guy my client hired to create all of them. He seemed like a nice enough guy, but suffice to say when I was asked to start replacing the Crystal Reports in our existing system there was talk of lawsuits. Every report was made using the design view, and they had a funky web interface with really bad UI displaying all the reports.

Anyway, we had SQL Server replication set up, and then the Crystal Reports were looking at a seperate database server, as they were designed in such a way to be enormously inefficient. So, when replication decided to crap itself, we switched over to log shipping, thinking everything would just keep working. One of the many problems that cropped up was the appearance of a database login prompt:

We tried changing the connection information in the Crystal designer, and the Crystal designer would always pull up a report just fine. The damn web interface would always prompt for information. We'd put in the information we entered in the designer, and then it would work. Since we didn't want to have users entering a user/pass every time they wanted to see a report, we kept changing things, trying to find something that would work. Finally we found that it would work if we pointed the report at our live database server, and ignore the backup we made for Crystal to use. As a temporary solution, we started changing all the reports to look at the other server, and googled like mad to try to find a solution.

After finding many, many people having this problem unresolved, and many, many solutions that are applicable if you're coding these up by hand, but nothing for our problem. We could tell that it was using the wrong config information, but had no idea how to change what it was using.

Solution

The solution came after Chris did a text search on the web server, and found some random ASP file where the user info was stored:

%webroot%\ASP\rPortfolio\HTMLViewers\interactiveViewer.asp

The username and password for the web interface was hardcoded there, and we couldn't find anything in Crystal that could change it. Notepad worked just fine. So, I guess for the web viewer, Crystal silently ignores the settings in the individual reports, and just uses that one. Great design.

Contributing to Firefox cont.

Ryan — Thu, 15 Dec 2005 09:29:00 GMT

A few weeks ago I wrote about Contributing to Firefox on WinXP, and last night, my patch got committed to the trunk! Whenever the next Firefox point release comes out, millions of people will be using the simple for loop that I wrote. Here's how the process worked:

11-30-2005: submitted patch and requested review
12-04-2005: passed review
12-14-2005: I politely asked around on #xul and #developers if there was anything else I was expected to do before the patch was committed. I was told to email the reviewer or make a comment on the bug. I opted to make a comment on the bug.
12-14-2005: Gavin Sharp replied to my comment apologizing for the delay, and committed my patch.

So, except for the 10-day delay and slight confusion over my role, I'd say it went pretty smoothly. This experience certainly encourages me to make more contributions. I've been using Sunbird for calendar stuff, and have seen a few minor glitches in that, maybe I'll toss in a patch or two there next.

Contributing to Firefox on WinXP

Ryan — Wed, 30 Nov 2005 12:54:00 GMT

In the process of converting my remote XUL apps to use Firefox 1.5, I found a bug in <tree>. The column picker breaks when you've added things to Array.prototype. That took many hours to find, after removing and re-adding countless lines of XUL and javascript. It came down to some functions we were adding to Array.prototype in one of our libraries. When I commented those out, it worked just fine. I installed the Console² extension, which gives some better error reporting, and found the a javascript error occurring in a file called tree.xml. It was a mess of XBL, but the problem was apparent.

The orignal code:

				  
var hidden = !tree.enableColumnDrag; 
const anonids = ["menuseparator", "menuitem"]; 
for (var anonid in anonids) { 
  var element = document.getAnonymousElementByAttribute(this, "anonid", anonids[anonid]); 
  element.hidden = hidden; 
} 

				

The problem is the usage of in. in is rather introspective, and will iterate over all the properties on the object, including user-added functions. My guess is the author of this code is used to the Python in keyword, which just iterates over the values in the object. I figure changing that to not use the in keyword should solve the problem:

				  
var hidden = !tree.enableColumnDrag; 
const anonids = ["menuseparator", "menuitem"]; 
for (var i = 0; i < anonids.length; i++) { 
  var element = document.getAnonymousElementByAttribute(this, "anonid", anonids[i]); 
  element.hidden = hidden; 
} 

				

To test this out, I went into my Firefox chrome directory (C:\Program Files\Mozilla Firefox\chrome for me), unpacked toolkit.jar, and made that change in chrome\toolkit\content\global\bindings\tree.xml. Console² let me know where to look. In order to get Firefox to read my files, I had to edit chrome\toolkit.manifest to look in the unpacked directory instead of the jar.

Original toolkit reference:

content mozapps jar:toolkit.jar!/content/mozapps/ xpcnativewrappers=yes

New toolkit reference:

content mozapps toolkit/content/mozapps/ xpcnativewrappers=yes

Now that Firefox is looking at the unpacked source with the updated tree.xml, I restarted Firefox and saw that it worked. The next step was to file a bug, and submit a patch.

As this was the first time I filed a bug at Bugzilla, I did it wrong. Asking around on #js, Hannibal kindly told me to make a diff from CVS, and request a review. I had to install cygwin, and partially followed Jeff's instructions for setting up an MSVC toolkit Firefox/Thunderbird build environment. I wasn't concerned with needing to actually build it, so I only followed the bits about cygwin. After getting a bash shell working, I followed the Mozilla Source Code via CVS instructions, and got the source on my machine. The build failed, but I didn't need it to build me anything, I just needed all the source.

Once I had the source, it was all very easy, following the Mozilla Source Code via CVS verbatim. After some browsing, I found a policy page for Toolkit patches, the Tookit Review Requirements. From there I uploaded my patch to my bug, and requested review. Within an hour or so, my request had been bumped to another toolkit dev, (the original guy is on vacation), and my bug's status had gone from UNCONFIRMED to NEW, and its looking to be put in the mozilla1.9alpha1 milestone.

So, it looks like I might get a very trivial piece of code into Firefox, and have learned a lot more about how Mozilla's internals work, and how open source development works. If I get ambitious, I might get the source for other Mozilla projects and see if I can contribute to those.

Server found request content type to be '', but expected 'text/xml'.

Ryan — Thu, 10 Nov 2005 17:43:00 GMT

I got this error when calling a C# web service from javascript. With a message like that, you'd think there was a problem with the Content-Type on my HTTP request.

Using the fine LiveHTTPHeaders Firefox extension, I saw in the headers:

Content-Type: text/xml; charset=UTF-8

Ok, so the content-type seemed ok, and it was time to start looking at my most recent changes to see where the bug was.

After a bit of hunting, I found the problem. My web service had an int parameter, and I was passing in 't1245'. That caused a casting error, which .NET cleverly reported back as a problem with the content-type.

Moral of the story: make your error messages meaningful.

XmlSerializer eats default attributes

Ryan — Mon, 17 Oct 2005 16:06:00 GMT

So, I'm still working with XML generation, and I discovered an interesting design choice made by the writers of XmlSerializer. My efforts from yesterday were so I could use the XmlSerializer to generate flawless XML from C# objects, without going through the bother of string manipulation. The XML it generates is well-formed, but it was leaving off an attribute ("Version"), and I wasn't sure why. I googled around and found another dev with the same problem. Craig Andera's post "DefaultValue and XmlSerializer Don't Mix?" has a quick reproduction.

If you have an attribute with a default value defined in the DTD/XSD, and the C# variable representing that attribute has that value, then the XmlSerializer chooses not to render the attribute. Anyone reading the XML must be using the same DTD/XSD, and they already know the default value, so why waste bandwidth/space by rendering that attribute?

Well, when the reader of my XML is using that missing "Version" attribute as a key for how it should process my XML, it makes a big difference. Based on the error message from the vendor, I think they are switching on that "Version" to pick a DTD, then validating against that and processing the request. It would be nice to have some kind of "RenderDefaultAttributesAnyway" flag I could set before serializing. In the end I just changed the DTD to remove the default value, and all was well.