Latest Posts
I have a new blog intended to replace this one. For whatever reason, I cannot get dotText to export my posts correctly. I plan on keeping this site up for a while, but all new content is at:
http://russ.unwashedmeme.com/blog.
Gary King has been working on getting a good library system for lisp, and recently posted his Notes on a Lisp Library Management. I was going to email him back, but figured here was as good a place as any to give my comments. Read that first.
Firstly, I applaud Gary for taking up this mantle. I'm really still starting using lisp, and the lack of authoritative libraries is a hinderer for people like me. Getting a comprehensive library management system is going to help folks like me get up to speed with lisp faster. ASDF is a good start, but I think it needs to be built upon. Secondly, sorry if everything I'm saying here has already been considered.
The problem of getting libraries installed and updated with minimal user effort is a very studied problem with several decent implementations that could be used as starting points.
I think a lot of lessons can be learned by studying perl's CPAN. CPAN has mirrors that automatically sync, and each user picks a list of mirrors near them. Then you just ask for the package name, and it will download, make, run tests, then put the code in a proper system or user path such that programs can import the library. Perhaps a port of CPAN would solve most of these problems.
Another good example could be Debian's apt-get. I think it's command line interface almost perfect, and since many lisp developers are using debian based sytems, the interface would be very familiar. As a client of a lisp library management service, here's what I think would be ideal, presented as a mock shell session (with user input in bold):
~$ asdf install clsql
Looking up definition for package clsql... package found.
GPG key not present, downloading from https://www.b9.com/kevin.gpg.asc... done.
Downloading clsql from http://files.b9.com/clsql/clsql-3.6.1.tar.gz... done.
Checking signature for clsql-3.6.1.tar.gz... file verified.
Checking dependencies...
clsql depends upon uninstalled package uffi, install uffi? [ynaq] y
Looking up definition for package uffi... package found.
Downloading uffi from http://files.b9.com/uffi/uffi-1.5.13.tar.gz... done.
Checking signature for uffi-1.5.13.tar.gz... file verified.
Checking dependencies... done.
Dependencies met.
Building uffi... done.
uffi (1.5.13) installed.
Dependencies met.
Building clsql... done.
clsql (3.6.1) installed.
~$ asdf upgrade
Scanning installed packages.
Checking clsql (3.6.1)... up to date.
Checking uffi (1.5.13)... up to date.
Checking my-unversioned-lib (20060503)... outdated!
Looking up definition for package my-unversioned-lib... package found.
Downloading my-unversioned-lib from http://example.org/my-unversioned-lib.tar.gz... done.
Checking signature for my-unversioned-lib.tar.gz... file verified.
Checking dependencies... done.
Dependencies met.
Building my-unversioned-lib... done.
my-unversioned-lib (20060526) installed.
Upgrade complete, 3 packages processed.
1 package upgraded.
2 packages up to date.
Anyway, enough pipe-dreaming for me.
Recently I was doing some perfomance tuning for a client, who kept getting timeouts. The timeouts were being caused by some locking queries, with a lock type of "NETWORKIO". This is odd, as everything was running on the same server, so it shouldn't really have much network IO when connecting to 127.0.0.1.
My solution
After much frustraction and googling, I saw someone recommend updating statistics using sp_updatestats to speed up queries, which might resolve NETWORKIO locks faster. That made all the difference. Now the site is zippy as hell. But why did that work?
After some cursory research, I found that sp_updatestats is a convienence function for running UPDATE STATISTICS on every table in the current database. The statistics they talk about are about the keys in each index on the table. The stats are then analyzed by the query optimizer to get the best usage of indexes. If you do a "display estimated execution plan" on a query, the stats are whats used to make those estimates. Then it made sense.
In an earlier attempt to speed everything up, I had taken a trace of the activity and ran the index tuning wizard on about an hours worth of actual usage. Applying the recommended indexes sped up the site nicely, but the real benefit wasn't realized because the statistics on the new indices hadn't been calculated, so the query optimizer couldn't do its job very well. After getting my stats lined up, the optimizer kicked in and everything was very responsive.
Conclusion
If you change your indexes about, run EXEC sp_updatestats. There is an option on a database maintanence plan to do this, and I believe I'm going to start enabling it.
slme-profile-package had shown that practically all of the running time was spent in one function that did a lot of list building, and that this function was consing a LOT of memory. I tried to change the appends to nconcs, and after a bit of fiddling the profiling reported that the function was only consing about half as much memory as before. The program also runs a nice bit faster.
The problem I had to fix was that I had some list literals in code, that nconc--true to its nature-- was destructively modifying. When recreating a minimal case here SBCL is nice enough to warn me about it, but I don't recall seeing this when originally solving the problem. There was a lot of other code inbetween that may have been breaking its inferencing though. Another thing to notice is sbcl decided that the length of a constant list was constant=3, except in this case the list isn't as constant as it appears to be. :-)
CL-USER> (defun foo (a)
(let ((b '(1 2 3)))
(values (length b)
(nconc b a))))
; in: LAMBDA NIL
; (NCONC B A)
;
; caught WARNING:
; Destructive function NCONC called on constant data.
; See also:
; The ANSI Standard, Special Operator QUOTE
; The ANSI Standard, Section 3.2.2.3
;
; compilation unit finished
; caught 1 WARNING condition
STYLE-WARNING: redefining FOO in DEFUN
FOO
CL-USER> (foo '(bam baz))
3
(1 2 3 BAM BAZ)
CL-USER> (foo '(bam baz))
3
(1 2 3 BAM BAZ BAM BAZ)
CL-USER> (foo '(bam baz bad))
3
(1 2 3 BAM BAZ BAM BAZ BAM BAZ BAD)
CL-USER> (defun foo (a)
(let ((b (list 1 2 3)))
(values (length b)
(nconc b a))))
STYLE-WARNING: redefining FOO in DEFUN
FOO
CL-USER> (foo '(bam baz bad))
3
(1 2 3 BAM BAZ BAD)
CL-USER> (foo '(bam baz bad))
3
(1 2 3 BAM BAZ BAD)
So today I had a client call and ask why their site had errors. They had a secure certificate, and IE was giving a popup:
"This page contains both secure and non-secure items. Do you want to download the non-secure items?"
So, I pull up the page, and check the source, figuring there's some
<img src="http://..." somewhere in there. A ctrl-f later, I see there are no non-secure images. The only things not using https are links to other sites, and some crap in the
code ghetto that is the markup needed for a flash movie. Specifically, I saw:
|
| <OBJECT classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" |
| codebase="http://download.macromedia.com/pub/[blah]" |
| WIDTH="88" HEIGHT="31" id="PoweredBy_88_31_A" |
| ALIGN="" VIEWASTEXT> |
| |
| [40 lines of garbage] |
| |
| <EMBED SRC="[flash file]" |
| swLiveConnect=FALSE WIDTH=228 HEIGHT=287 |
| QUALITY=AUTOLOW MENU=false PLAY=true |
| BGCOLOR=#FFFFFF TYPE="application/x-shockwave-flash" |
| PLUGINSPAGE="http://www.macromedia.com/[blah]"> |
| </EMBED> |
So, on
Nathan's suggestion, I tried changing this boilerplate mess to make the
pluginspace and
codebase attributes point to https. That made IE stop throwing the error. I'm not sure why it was downloading things from those URLs (as the error message clearly stated), but now its happy, and my client's clients are no longer scared to use the site.
But the fun doesn't end there. I took a stress break and ran across a blog entry by Jason about his ActiveX update woes. Apparently to avoid paying licensing fees, Microsoft is changing how IE loads plugins (including flash), and unless developers (developers, developers, developers!) change how they render Flash, the users will be presented with a "Click here to activate this" dialog instead of just showing the flash.
Apparently you have to "activate" a plugin before it will run. More information can be found here: Internet Explorer Eolas changes and the Flash plugin.
Like any good security measure, this can be instantly circumvented from the server (attacker) side. Apparently if you set it up with javascript, then it doesn't need to be activated. I guess there's no such thing as malicious javascript. Microsoft has their workaround: activating ActiveX controls, and the author of the helpful article above has a widely used Flash rendering javascript library that works too, called FlashObject. He's written a few books on flash, and his library looks pretty nice, both hiding the <OBJECT> mess and gracefully degrading to plain HTML if the flash plugin is absent.
Thanks Microsoft, thanks to your change (and Jason's post) I've discovered some nice open source library to remove a slurry of meaningless markup, but I would really prefer not to have to do your legal work.
I wrote my first trac plugin! You can read a bit about it here:
Trac Tab. It just allows me to easily add new tabs to the mainmenu in trac that point to trac pages with an Iframe inside pointing somewhere else. We are using it to link to our billing software from inside of trac, so now our trac project for a client, can have our billing software more closely at hand. Yay!
I really wanted to be excited about .Net 2.0 and C#'s 3.0 LINQ preview. However, between .Net1.1 and .Net2 coming out, I started to play with LISP.
First, I was digging through some C# 2.0 the other day (now with Generics and "Anonymous Delegates Outer variable capture" (craptastic, hacked, statically typed closures)), and while reading a blogpost (that I lost, sorry) the blogger maid the excellent point that in removing static typing from some things (generics) they had to add reams of code. Something there should smell bad already. If you find yourself adding reams of code (and more, new syntax) to get away with leaving out half of what the language originally made you specify, maybe the language just shouldn't force you to specify that info. I dunno maybe I'm crazy and/or stupid and/or a type-ist?(someone who's against typing?).
On to LINQ: When I saw
this forthcoming tech, LINQ, I thought, "Hey relational algebra-ish support inside of the language, nice". Then I looked at the demo and the only thing I can think of is all of the lisp constructs to do this sort of high end querying. All of the LISP ways that were built into the language long before its standardization (ANSI standardization: December 8, 1994) a decade ago. I am fairly sure that Ruby blocks can be put to use to accomplish this same sort of thing. Also Python list comprehensions. It really makes me question why people are willing to wait another 2-3 years for LINQ when they are a download away from better, more dynamic frameworks. Oh... right... its next to impossible to talk to sql server outside of an MS environment (vendor lockin). And as always learning anything new (that isnt ms )
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; C# Linq Demo from
this .net blog
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
XDocument doc = XDocument.Load("blog.xml");
var d = new Dictionary
();
d.Add("2005/09","September 2005");
d.Add("2005/08","August 2005");
var a = new string[] { "greasemonkey", "ajax" };
var query =
from
item in doc.Descendants("item"),
key in d.Keys,
tag in a
where item.Element("date").Value.Contains(key) &&
item.Element("tags").Value.Contains(tag)
orderby
(string) item.Element("date") descending
select new XElement("item",
new XElement("month",d[key]),
item.Element("date"),
item.Element("title"),
item.Element("tags"));
foreach (var result in query)
Console.WriteLine(result);
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Lisp with Loop which as far as I can tell is about as close as I can get LINQ
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(mapc
(lambda (elem) (print elem))
(loop for item in doc.Decendants("items")
for key in d.Keys
for tag in a
when (and
item
(find key (value (item.Element "date")))
(find tag (value (item.Element "tags"))))
collect (make-instance 'XElement
:tag-name "month"
:children (list (make-instance 'XElement :tag-name "month")
(item.Element "date")
(item.Element "title")
(item.Element "tags")))
into new-elems
unless item do (return (sort new-rows #'xml-item-date-< )))
There might be more that I am missing (for example an XML library (use ClosureXML), and a predicate that will work for the sorting he wants). About the nicest part of the LINQ stuff seems to be the OrderBy clause (also possibly GroupBy which isnt covered in the demo, but I could see being cool). However, sorting is more or less a done job given an appropriate predicate. Also there might be slightly more involved iterating going on in the query, but nothing that couldnt be handled with a modicum more code, wrapped in a macro to give it that linq-ish look, and packaged up to be useful in a week.
I received a helpful comment on my last entry, and sat down to try out this seemingly ideal
solution:
|
| |
| SET ROWCOUNT @limit |
| |
| DELETE FROM #temp WHERE Id NOT IN( |
| |
| SELECT TOP 100 PERCENT Id FROM Items |
| ORDER BY NEWID() |
| ) |
| |
| SET ROWCOUNT 0 |
It didn't work, and I got really confused when I tried to figure out why, so I started making some
simple tests.
Setup and basic test
I started by creating a table variable, and populating it with some data:
|
| DECLARE @limiter TABLE (id int not null) |
| |
| INSERT INTO @limiter (id) ( |
| SELECT TOP 2000 Id |
| FROM Items |
| ) |
Next step, lets see if TOP 100 PERCENT works with ROWCOUNT the way I'd like it to:
|
| SET ROWCOUNT 10 |
| |
| SELECT TOP 100 PERCENT id FROM @limiter |
| ORDER BY NEWID() |
Running this gives me 10 rows, and running it again give me 10 different rows, so my randomish requirement is met.
Subquery test
Next test is my goal, the randomish sample. I make the 10 row random sample in a subquery, and then delete everything not in that sample. Yes, this is kinda backwards, but the actual code is more complicated than my simplification here, so its easier to generate all the possibilities then remove all that don't fall into our sample.
|
| |
| SET ROWCOUNT 10 |
| |
| DELETE FROM @limiter WHERE Id NOT IN ( |
| SELECT TOP 100 PERCENT id FROM @limiter |
| ORDER BY NEWID() |
| ) |
| |
| |
| SET ROWCOUNT 0 |
| SELECT * FROM @limiter |
I expect this to return 10 rows. It returns 2000, the full contents of @limiter. That implies the TOP 100 PERCENT in the subquery returned the full 2000 rows, so nothing was deleted. Checking the execution plan verifies this. SET ROWCOUNT doesn't work on subqueries? Or maybe the TOP 100 PERCENT is confusing SQL Server, so I'll try using a TOP N:
|
| SET ROWCOUNT 10 |
| |
| DELETE FROM @limiter WHERE Id NOT IN ( |
| SELECT TOP 20000 id FROM @limiter |
| ORDER BY NEWID() |
| ) |
| |
| |
| SET ROWCOUNT 0 |
| SELECT COUNT(id) FROM @limiter |
Nope, still get 2000 rows in @limiter after the delete. For grins, I start playing with the TOP statement, and get some confusing results:
| TOP N | Rows left in @limiter |
| TOP 20000 | 2000 |
| TOP 2000 | 2000 |
| TOP 1999 | 1999 |
| TOP 1998 | 1997 |
| Huh? The top 1998 should have returned 1998 rows; the 2 rows not in that sample should have been deleted. Lemme run that again to make sure I didn't mess up. |
| TOP 1998 | 2000 |
| What the hell? Is this some weird query caching thing? Let me run it a few more times... |
| TOP 1998 | 1998 |
| TOP 1998 | 1999 |
Seriously, WTF. The execution plan just creates more questions:
It's somehow processing 4M rows from a table variable with 2K rows!? Then the TOP 1998 gets performed in the Filter operation, which reduces it to 1999 rows? Ok, so 2K * 2K = 4M, so the WHERE Id NOT IN () is effectively performing a cross join, which makes sense, but why the 1999 rows?
|
| TOP 10 | 1990 |
Ok, so something is seriously inconsistent with ROWCOUNT and TOP in subqueries. When I take the SET ROWCOUNT out, it still is inconsistent. When I take out the ORDER BY NEWID(), it is still inconsistent.
TOP in a subquery is broken?
Consider this repro:
|
| DECLARE @t TABLE (id int not null) |
| |
| INSERT INTO @t (id) ( |
| SELECT TOP 20 Id FROM [someTable] |
| ) |
| |
| SELECT COUNT(Id) FROM @t |
| WHERE Id IN ( |
| SELECT TOP 10 Id FROM @t |
| ORDER BY NEWID() |
| ) |
What do you think the count should be? Hint: there's more than one answer. Run that query repeatedly, and bask in inconsistency. But... try this one:
|
| SELECT COUNT(Id) FROM [someTable] |
| WHERE Id IN ( |
| SELECT TOP 10 Id FROM [someTable] |
| ORDER BY NEWID() |
| ) |
And bam, we get 10 every time. So its not TOP and ORDER BY in subqueries, maybe its something to do with table variables? So, with that in mind, lets go back to the original test, but using a temp table instead of a table variable.
Out with table variables
So, back to the beginning:
|
| SELECT TOP 2000 Id INTO #limiter |
| FROM Items |
| |
| |
| SET ROWCOUNT 10 |
| |
| DELETE FROM #limiter WHERE Id NOT IN ( |
| SELECT TOP 100 PERCENT id FROM #limiter |
| ORDER BY NEWID() |
| ) |
| |
| |
| SET ROWCOUNT 0 |
| SELECT COUNT(id) FROM #limiter |
Result: 1990 rows. It should have 10. So, table variables aren't the problem.
Fine, you win, SQL Server
Fine. If you want it that badly MSSQL, you can have it. I'll keep my slimy solution from the last entry (for now), as you obviously don't want to let me do anything in a straighforward manner. I'll try to forget the inconsistencies I found today, but it'll take time to heal. Be patient with me. And don't get all jealous if you see me installing postgresql to see if it can do this better than you. You know I'm locked in to you.
So when trying to read from excel files we got this message repeatedly, and could not figure out what the hell it meant. We tried reinstalling some dlls that were suggested by other blog posts, but nothing would make the excel file open. Turns out, our connection string was wrong.
This is a basic correct connection string.
"Provider=Microsoft.Jet.OleDb.4.0;data source=[FILE_PATH];Extended Properties=Excel 8.0;"
UPDATE:
This guy knew the answer if I had read my opened tabs.
I was recently presented with a seemingly simple task, write a query to return a random* sample of rows
from a table, with the number of rows to pull determined at runtime.
The surrounding problem is a bit more complex, but this was the specific task I was trying to accomplish,
and SQL Server 2000 didn't make it very easy. The rest of my task involved deleting rows where the id wasn't in my random sample.
First try, my almost-dream syntax
My first attempt was a bit naive:
|
| SELECT TOP @limit Id |
| FROM Items |
| ORDER BY NEWID() |
@limit is a variable which has how many rows I should be returning.
I order by NEWID(), which makes a GUID for every row, as this is the best way I know of to get it ordered randomly. I don't
think its a terribly good way, but its the best I know. If you have a better way, please comment or email.
That snippet will of course fail, giving this error message:
Incorrect syntax near '@limit'.
Ok, so I can't dynamically set the TOP. Fine.
Use ROWCOUNT
By using SET ROWCOUNT @limit, I could instruct SQL Server to only return @limit rows when performing a SELECT. So, this was the query to
delete rows not in my random sample:
|
| |
| SET ROWCOUNT @limit |
| |
| DELETE FROM #temp WHERE Id NOT IN( |
| |
| SELECT Id FROM Items |
| ORDER BY NEWID() |
| ) |
| |
| SET ROWCOUNT 0 |
That snippet also fails, giving this error:
The ORDER BY clause is invalid in views, inline functions, derived tables, and subqueries, unless TOP is also specified.
Ok, so I can't use ORDER BY in the subquery. Well damn, that would have been really clean. I could add a
TOP 2147483647 in there, but that seemed like a bad idea.
Use a table variable
Ok, next thought is to use a table variable, insert my rows in a random order, numbering each row, and then pull the rows with a number
below my limit:
|
| |
| DECLARE @temp TABLE (id int NOT NULL, num int NOT NULL IDENTITY(1,1)) |
| |
| |
| INSERT INTO @temp (id) |
| (SELECT Id |
| FROM Items |
| ORDER BY NEWID()) |
| |
| |
| SELECT Id FROM @temp WHERE num <= @limit |
That snippet also fails, giving this error:
Incorrect syntax near the keyword 'ORDER'.
Oh right, I can't use ORDER BY in the subquery. Fine.
Use a temporary table and INSERT INTO
I was trying to avoid temporary tables, since they have some performance issues, but figured maybe there wasn't any other way. So,
I broke down and used a temp table:
|
| |
| SELECT Id, IDENTITY(int, 1,1) [num] |
| INTO #temp |
| FROM Items |
| ORDER BY NEWID() |
| |
| |
| SELECT Id FROM #temp WHERE num < @limit |
| |
| |
| DROP TABLE #temp |
That snippet also fails, giving this error:
Cannot add identity column, using the SELECT INTO statement, to table '#temp', which already has column 'Id' that inherits the identity property.
Ok... So the 'identity' property gets inherited into the temp table, and so I can't specify another identity column.
I did some searching in the docs, and there didn't seem to be a way to get an autoincrement field without it
being an identity field. Ok, fine, I know a retarded way to get around that.
Use a temporary table, INSERT INTO, and a table variable to mask the identity column
I actually wrote a variant of this:
|
| |
| DECLARE @retarded TABLE (id int) |
| |
| |
| INSERT INTO @retarded (id) |
| (SELECT Id FROM Items) |
| |
| |
| SELECT Id, IDENTITY(int, 1,1) [num] |
| INTO #temp |
| FROM @retarded |
| ORDER BY NEWID() |
| |
| |
| SELECT Id FROM #temp WHERE num < @limit |
| |
| |
| DROP TABLE #temp |
This worked, but made me sick to my stomach. There
had to be a better way.
Use a temporary table, INSERT INTO, and a dummy expression
So, I dug through some more documentation, finding this gem:
When an existing identity column is selected into a new table, the new column inherits the IDENTITY property, unless one of the following conditions is true:
- The SELECT statement contains a join, GROUP BY clause, or aggregate function.
- Multiple SELECT statements are joined by using UNION.
- The identity column is listed more than one time in the select list.
- The identity column is part of an expression.
If any one of these conditions is true, the column is created NOT NULL instead of inheriting the IDENTITY property. All rules and restrictions for the identity columns apply to the new table.
Ok, so I decided the least despicable thing to do would be to make a dummy expression to drop the identity property:
|
| |
| SELECT Id*1 [Id], IDENTITY(int, 1,1) [num] |
| INTO #temp |
| FROM Items |
| ORDER BY NEWID() |
| |
| |
| SELECT Id FROM #temp WHERE num < @limit |
| |
| |
| DROP TABLE #temp |
I did some profiling, and doing
id*1 was the same as doing
id+0, so I left it as multiplication.
All in all, very, very frustrating, and I'm not sure if the TOP 2147483647 option above is worse than the hack at the end. Votes are welcome.
I hate how much I jump through hoops and reinvent to do what should be very simple tasks.
I mean, c'mon, we've been programming for over 50 years now, and I'm pretty sure pulling a random sample from a dataset is
not a new task. We should be better than this.
As a collectable card:
The Free Textbook Req. A webserver
Provides:
When played on a Professor increases the
professor's charisma. If the professor is one
of the authors then the effect is +3. This
Free Textbooks still count towards the course's
"Too many textbook" penalty.
"I Don't have to pay an extra 100 bucks?!"
|
One client of mine uses Crystal Reports for reporting, and our experience has been pretty bad overall. Part of
that might be due to the guy my client hired to create all of them. He seemed like a nice enough guy, but suffice
to say when I was asked to start replacing the Crystal Reports in our existing system there was talk of lawsuits. Every
report was made using the design view, and they had a funky web interface with really bad UI displaying all the reports.
Anyway, we had SQL Server replication set up, and then the Crystal Reports were looking at a seperate database server, as
they were designed in such a way to be enormously inefficient. So, when replication decided to crap itself, we switched over
to log shipping, thinking everything would just keep working. One of the many problems that cropped up was the appearance
of a database login prompt:
We tried changing the connection information in the Crystal designer, and the Crystal designer would always pull up a report just fine.
The damn web interface would always prompt for information. We'd put in the information we entered in the designer, and then
it would work. Since we didn't want to have users entering a user/pass every time they wanted to see a report, we kept changing
things, trying to find something that would work. Finally we found that it would work if we pointed the report at our live database
server, and ignore the backup we made for Crystal to use. As a temporary solution, we started changing all the reports to look
at the other server, and googled like mad to try to find a solution.
After finding many, many people having this problem unresolved, and many, many solutions that are applicable if you're coding these
up by hand, but nothing for our problem. We could tell that it was using the wrong config information, but had no idea how to
change what it was using.
Solution
The solution came after Chris did a text search on the web server, and found some random ASP file where the user info was stored:
%webroot%\ASP\rPortfolio\HTMLViewers\interactiveViewer.asp
The username and password for the web interface was hardcoded there, and we couldn't find anything in Crystal that could change it.
Notepad worked just fine. So, I guess for the web viewer, Crystal silently ignores the settings in the individual reports, and just
uses that one. Great design.
A few weeks ago I wrote about Contributing to Firefox on WinXP,
and last night, my patch got committed to the trunk! Whenever the next Firefox point release comes out, millions of people
will be using the simple for loop that I wrote. Here's how the process worked:
- 11-30-2005: submitted patch and requested review
- 12-04-2005: passed review
- 12-14-2005: I politely asked around on #xul and
#developers if there was anything else I was expected to do
before the patch was committed. I was told to email the reviewer or make a comment on the bug. I opted to make
a comment on the bug.
- 12-14-2005: Gavin Sharp replied to my comment apologizing for the delay, and committed my patch.
So, except for the 10-day delay and slight confusion over my role, I'd say it went pretty smoothly. This experience
certainly encourages me to make more contributions. I've been using
Sunbird for calendar stuff, and have seen a few minor
glitches in that, maybe I'll toss in a patch or two there next.
In the process of converting my remote XUL apps to use Firefox 1.5, I found a bug in <tree>.
The column picker breaks when you've added things to Array.prototype. That took many hours to find, after removing and re-adding
countless lines of XUL and javascript. It came down to some functions we were adding to Array.prototype in one of
our libraries. When I commented those out, it worked just fine. I installed the
Console2 extension, which gives some better
error reporting, and found the a javascript error occurring in a file called tree.xml. It was a mess of XBL,
but the problem was apparent.
The orignal code:
|
| var hidden = !tree.enableColumnDrag; |
| const anonids = ["menuseparator", "menuitem"]; |
| for (var anonid in anonids) { |
| var element = document.getAnonymousElementByAttribute(this, "anonid", anonids[anonid]); |
| element.hidden = hidden; |
| } |
The problem is the usage of in. in is rather introspective, and will iterate over all
the properties on the object, including user-added functions.
My guess is the author of this code is used to the Python in keyword, which
just iterates over the values in the object. I figure changing that to not use the in keyword should solve the problem:
|
| var hidden = !tree.enableColumnDrag; |
| const anonids = ["menuseparator", "menuitem"]; |
| for (var i = 0; i < anonids.length; i++) { |
| var element = document.getAnonymousElementByAttribute(this, "anonid", anonids[i]); |
| element.hidden = hidden; |
| } |
To test this out, I went into my Firefox chrome directory (C:\Program Files\Mozilla Firefox\chrome for me),
unpacked toolkit.jar, and made that change in chrome\toolkit\content\global\bindings\tree.xml.
Console2 let me know where to look. In
order to get Firefox to read my files, I had to edit chrome\toolkit.manifest to look in the unpacked
directory instead of the jar.
Original toolkit reference:
content mozapps jar:toolkit.jar!/content/mozapps/ xpcnativewrappers=yes
New toolkit reference:
content mozapps toolkit/content/mozapps/ xpcnativewrappers=yes
Now that Firefox is looking at the unpacked source with the updated tree.xml, I restarted Firefox and
saw that it worked. The next step was to file a bug, and submit a patch.
As this was the first time I filed a bug at Bugzilla, I did it wrong.
Asking
around on #js, Hannibal kindly told me to make a diff from CVS, and request a review. I had to install
cygwin, and
partially followed Jeff's instructions for
setting up an MSVC toolkit Firefox/Thunderbird build environment.
I wasn't concerned with needing to actually build it, so I only followed the bits about cygwin.
After getting a bash shell working, I followed the
Mozilla Source Code via CVS
instructions, and got the source on my machine. The build failed, but I didn't need it to build me anything, I just needed all
the source.
Once I had the source, it was all very easy, following the
Mozilla Source Code via CVS verbatim.
After some browsing, I found a policy page for Toolkit patches, the
Tookit Review Requirements.
From there I uploaded my patch to my bug, and
requested review. Within an hour or so, my request had been bumped to another toolkit dev, (the original guy is on vacation),
and my bug's status had gone from UNCONFIRMED to NEW, and its looking to be put in the mozilla1.9alpha1 milestone.
So, it looks like I might get a very trivial piece of code into Firefox, and have learned a lot more about how Mozilla's internals
work, and how open source development works. If I get ambitious, I might get the source for other Mozilla projects and see
if I can contribute to those.
I got this error when calling a C# web service from javascript. With a message like that, you'd think
there was a problem with the Content-Type on my HTTP request.
Using the fine LiveHTTPHeaders
Firefox extension, I saw in the headers:
Content-Type: text/xml; charset=UTF-8
Ok, so the content-type seemed ok, and it was time to start looking at my most recent changes to see where the bug was.
After a bit of hunting, I found the problem. My web service had an int parameter, and I was passing in 't1245'. That
caused a casting error, which .NET cleverly reported back as a problem with the content-type.
Moral of the story: make your error messages meaningful.
I like Lisp, to the point of being accused of "having drunk the koolaid." I read a good number of lisp weblogs (I'm lazy about linking right now). I lurk on #lisp all day long, although a good deal goes by unread I like following the links.
At work I've been trying to get a lisp and UCW setup going. However, one of the key components to making htis happen is to be able to reliably talk to a SQL database. Right now we are a predominantly Microsoft shop so charting a path from our current environment to a lisp one is somewhat tricky.
My experience has been less than good. I have tried a number of different setups and found very few combinations that work. I haven't gotten into what I would consider weird database stuff. Just basic tables, selects and inserts. My unicode test string is: "Iñtërnâtiônàlizætiøn"
The software:
| Attempted setup |
Does it work? |
Error |
Unicode string |
| SBCL -> Postgresql-socket -> postgres |
No - Doesn't connect |
Illegal :ASCII character starting at byte position 0.
[Condition of type SB-IMPL::MALFORMED-ASCII]
the specific byte positions varies, normally in the first 4 but i've seen it accross the first 7 or 8. |
| SBCL -> UFFI -> PostgreSQL library -> postgres |
YES! |
|
Iñtërnâtiônàlizætiøn |
| SBCL -> UFFI -> UnixODBC -> FreeTDS -> MS SQL Server |
'Bit' datatype doesn't work. |
The value
#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X082A6A50 :TYPE (* (SB-ALIEN:SIGNED 8))>
is not of type (SB-ALIEN:ALIEN (* (SB-ALIEN:UNSIGNED 8))).
[Condition of type TYPE-ERROR]
I didn't know true could be negative
|
"I?t?rn?ti?n?liz?ti?n" |
| ACL -> :aodbc -> ODBC32 -> MS SQL Server |
YES! |
|
Iñtërnâtiônàlizætiøn |
| ACL -> Postgresql[-socket] |
For some values of true |
|
Iñtërnâtiônà lizætiøn |
Some of the problems might be easy to solve for someone who knows more about the FFIs than I do. But programming in C is not why I am trying to get CL running. There are combinations that work, but this is hardly reassuring.
So, I'm still working with XML generation, and I discovered an interesting design choice made by the writers of XmlSerializer.
My efforts from yesterday were so I could use the XmlSerializer to generate
flawless XML from C# objects, without going through the bother of string manipulation. The XML it generates is well-formed, but it was leaving off an attribute ("Version"), and
I wasn't sure why. I googled around and found another dev with the same problem. Craig Andera's post
"DefaultValue and XmlSerializer Don't Mix?" has a quick reproduction.
If you have an attribute with a default value defined in the DTD/XSD, and the C# variable representing that attribute has that value, then the
XmlSerializer chooses not to render the attribute. Anyone reading the XML must be using the same DTD/XSD, and they already know the default value,
so why waste bandwidth/space by rendering that attribute?
Well, when the reader of my XML is using that missing "Version" attribute as a key for how it should process my XML, it makes a big difference. Based on the
error message from the vendor, I think they are switching on that "Version" to pick a DTD, then validating against that and processing the request.
It would be nice to have some kind of "RenderDefaultAttributesAnyway" flag I could set before serializing. In the end I just changed the DTD to
remove the default value, and all was well.
I've started to implement a web service, sending and receiving XML snippets to a vendor. I was browsing their
documentation, and was about to start throwing together some code using string functions to build/parse the XML, and then
I remembered all the hubub I've heard about generating code using XML files, and figured there might be some tools out
there to make this go a lot smoother. The vendor was kind enough to provide a sample XML file and a Document Type Definition (DTD) file, so
now all I need to do is convert that to C# classes.
First try: xsd.exe
The first tool I looked at was
xsd.exe
, a command line tool bundled with Visual Studio. When I dug in, I see that it can:
- convert a XML Schema Definition (XSD) file into C# classes
- convert a sample XML file into a XSD file, inferring and guessing as to the details
- convert an External Data Representation (XDR, what the hell is that one?) file into a XSD file, inferring and
guessing as to the details
- generate XSD files for .NET classes in a file or assembly
So... no option for converting a DTD. Ok, time to google for DTD XSD converters.
Second try: dtd2xs and xsd.exe
The first hit was dtd2xs. This is a Java based converter that
can be run from the commandline, used as a library in java programs, and as an applet in webpage. It convienently
comes with some HTML that already has the applet set up, so I loaded up that page and tried to convert my
DTD. After a few seconds, I was presented with a window listing some debug info, and then a line with:
******************** Copy & Paste ********************
and nothing underneath it, where I would expect to see content to Copy and Paste. I ran it again with the same result, and
then viewed source on the frame. The source had the XSD document in there, but wasn't being rendered because all the XML was
meaningless to Firefox. I copied the content into an .xsd file, and then ran
xsd.exe on it. The result? A bunch of
errors about invalid references in the XSD. Maybe I can fix the XSD file using Visual Studio's XSD tools.
Third try: Visual Studio
Next, I tried to use the XSD editor in Visual Studio to see if I could correct the problem. I messed around with the raw XML
of the XSD file, guessing what attributes like "ref" and "name" would mean in the context of defining data. After messing around
with the file, I was able to generate even less sensical errors from xsd.exe. Had I put in the time to learn the XSD
XML dialect, then I'm sure these errors would make sense, but I've no interest in learning more XML acronyms. There's enough of those
in my head already. I decided to try to build it from scratch using Visual Studio's GUI XSD editor.
I dragged an "element" block from the toolbox, added some attributes, and then promptly gave up.
The UI wasn't that great (few tooltips, ambiguous context menu options), and it wasn't clear how to do what should be simple things.
I figure if I'm going to spend time learning how to use a tool, I'd rather use one that will be more efficient, so I'll put my effort
into the command line tools.
Fourth try: xsd.exe again
I decided to take a different approach:
- use
xsd.exe to generate an XSD from my sample XML file.
- use
xsd.exe can generate C# from that XSD.
This actually worked. The result C# was very much generated code, but it had the plumbing to save/load from an XML file or string, and
that is the plumbing I was wanting the most. I took a look at the XSD to glean a bit of knowledge, and started to see the gaps left
by inferring schema from a sample. Default values for attributes were lost, all enumerations were lost (when an attribute's value is
restricted to certain values), all the optional stuff was missing, and overall I wasn't too pleased with the results. I could probably
make it work, but I'd get a much higher quality result if I could retain as much of the vendor's definition as possible. Back to google
for another dtd xsd converter.
Fifth try: Xsdvalid and xsd.exe
The next converter I found was Xsdvalid. This is a sub-component
of a larger XML editor, XMLmind XML Editor.
Xsdvalid is another Java program (I guess Java folks love XML), able to be used as a library or from the command line. I ran
my DTD through it, and it came out with a much larger XSD than dtd2xs. xsd.exe ran without errors on the
first try, and the resulting C# had enums for the enumerations defined in the DTD, and seemed a lot better. My C# is now directly based on
the vendor's definition, with no opportunity for me to make mistakes in translating.
Conclusion
I haven't actually used the generated code yet, but will be writing some unit tests momentarily for parsing/building. I do have high hopes,
so bless you people with XML fetishes for doing this work for me. Also, curse you people with XML fetishes for not picking *one*
XML dialect to describe XML documents and causing this task in the first place.
What the hell is wrong with these people?
This MS page descibes the error. One thing they dont mention is a solution. They do say that:
CAUSE
This issue occurs if the server sends a "Cache-control:no-store" header or sends a "Cache-control:no-cache" header.
What they dont mention is that it isnt only the "Cache-contol" header; the "Pragma:no-cache" header also causes this problem.
A SOLUTION
Remove all of these headers, and make sure that you have the following headers set:
Response.AddHeader("Pragma","private");
Response.AddHeader("Cache-control","private, must-revalidate");
I made some small changes to
TaskPool, in preparation for some folks at
ISI to start trying it out. (those of you who know me know how that one came about :)
WTFAYMTAAAT:
What The Fuck, Are You Mental, There Already Are A Thousand
I saw this applied to RSS Aggregators.
I've now gotten permission to use XUL for another client, and this time around,
I'm trying it a little differently.
Rather than serving up a .xul file, and doing everything with
overlays, DOM manipulation, and XBL, I'm serving up a .aspx file,
setting its content-type to application/vnd.mozilla.xul+xml, and
then using ASP.NET to render XUL instead of HTML. Jeb (do you have a link,
Jeb?) said he was creating XUL apps this way using JSP, so I figured I'd give
it a try. So far I like it. This method lets me lean on my ASP.NET skills while
I build up the XUL skills.
How I have it set up:
-
One
.aspx page per XUL layout, using <html:a> tags
to navigate between them.
-
One ASP.NET user control is on each page, ensuring all XUL layouts have the
same template and include the same javascript and css files. I had tried do
this with overlays,
but didn't have much luck.
-
A few
IHttpHandlers to send RDF to my XUL, and
JSON to my javascript.
-
A few webservices that get called by my javascript, and the work is done in C#.
This seems to work pretty well. The UI and the business logic is completely
seperate, and by using RDF I can take advantage of XUL templates to do lots of
nifty things on the client.
For now, I'm only using this on applications that require windows
authentication to access the site, so to access the web services you'll need to
be logged in. Security is still a concern, though, and eventually I'll have to
expose some of these web services to wild. I've been considering some token
system where the server passes a token to the client when it renders the xul
aspx page, and then the javascript passes that token to the web services, but I
haven't fleshed anything out yet.
Made a bug fix, the latest is available at the
Taskpool article.
This article gives me a lot of hope. I was kind of worried when I read about cutting down the length of gameplay. I was worried that they were going to remove interesting complexity. However it seems that they are trying to make the game more fluid and easier to control; eg. give the same build orders to several different cities at once.
I have been playing with LISP over the last few days to try and hack together a with-gensym macro that removes the need to put all my symbols as gensyms in a macro.
This is what Nathan and I came up with. It allows you to use a (with-gensym (a b c d) ...body...) block. You don't need to unquote the symbols in the body, you can just use the symbols like normal, however in the expansion, there is a let making the symbol gensyms, and all of the symbols are replaced with ,symbol-name. This way I dont ever have to think about gensym again.
The best way to make frozen "girley drinks" : popsicles. Just blend and add booze. Genius.
I came across what appears to be a minor bug in Opera's handling of javascript and character encoding. If I send it a stream of text that is accurately described by the header "Content-Type: text/javascript; charset=utf-8" Opera (at least 8.02) is at a loss for what to do.
When loadng the address directly it just tries to find an external handler for that mime type (and fails for me). If I try to load this bit of javascript through the XMLHttpRequest I get something, however I am not familiar enough with the different east-asian languages to determine what character of beast this is. Suffice it to say that it is not what I intended to get back.
If I change the mime-type to "Content-Type: application/x-javascript; charset=utf-8" or 'text/plain' or 'text/html' it works just dandily. Not to difficult of a workaround, but is this really a workaround or "The Proper Way"?
Safari, Firefox, and Internet explorer did not have any problems digesting the 'text/javascript'. So for trying to follow a de facto standard, which from what I know of Opera they generally try to do, not very good. Most webservers serve 'application/x-javascript' so they do have that one in the bag.
This really comes down to: There is no standard javascript mime-type.
A script block with type="text/javascript" writes: ""
A script block with type="application/x-javascript" writes: ""
My test showed that both of those work in Mozilla and Opera, and I only get "foo" when in IE.
YACaNTS (Yet Another Carbon Nanotube Story). Constantly reading more stories of how carbon nanotubes are awesome.
Still looking for the story about carbon nanotube stem cells.
When I finished
playing with defmemoized a few days ago I realized that I was very close to having a
general caching scheme. At that point I decided that when I got a chance I would write a
new macro based on defmemoized:
defcached.
To make this a caching scheme, I need to have the ability to tell the cache to expire. To
accomplish this I added a new vaiable in to the defining environment (timeout) and a new
function named [symbol]-set-timeout. The set timeout function tells it how long it should
cache the results for each set of parameters passed to it. Its probably about time to break
out CLOS and learn it since an object is probably better than just inserting more functions
into the package name space.
I've been working for the past couple of days to improve our framework for doing partial post-back.
It's more or less standard AJAX fare, I'm just trying to make it easy to do partial page rendering and replacement that uses our existing ASP.Net control structure.
Works great and was easy to do in Mozilla, but of course Inappropriate Expletive has to make life difficult.
I've been trying to make this as resilient to errors as possible. If the AJAX request gets a 400 series error, bail out and try a standard postback. If the server did not send back javascript or gives a 500 series error then print whatever it did get back to the screen.
Replacing the current page with new html in javascript turned out to be more difficult than I expected. According to MSDN you can open the current document, write to it and close it again.
Indeed their sample page does work (in Firefox as well), but when I tried to do the same function through the shell or from the partial postback code it leaves the browser in an inconsistent state at best, and sometimes just crashes it. I ended up clearing the body's innerHTML, adding an IFrame to the body, and writing everything to that. Not exactly a 'pretty' solution.
The communication with the server really needs to be done asynchronously so the browser isn't blocked while waiting for the server.
With XMLHttpRequest this isn't to bad.
I'm not as much of a fan of MSXML though. The problem comes from the readyState variable that reports what stage the request is currently on.
From the Mozilla docs:
readonly PRInt32 readyState
The state of the request.
Possible values:
- 0 UNINITIALIZED open() has not been called yet.
- 1 LOADING send() has not been called yet.
- 2 LOADED send() has been called, headers and status are available.
- 3 INTERACTIVE Downloading, responseText holds the partial data.
- 4 COMPLETED Finished with all operations.
That makes sense to me. Once 2 roles around I can start querying the headers to make sure I am getting what I expect. If it was set up to deal with a stream, that could start processing at step 3, and then everything is completely available in step 4.
The only change I could see is that it might be useful to someone to have a stage immediately after the request is sent but no part of the response has been received yet. Not entirely sure where this would be useful though.
It seems that the IE developers felt that this was the most important step though. "I've sent a request but don't have any data back yet."
From MSDN (parentheses theirs):
- 0 (UNINITIALIZED) The object has been created, but not initialized (the open method has not been called).
- (1) LOADING The object has been created, but the send method has not been called.
- (2) LOADED The send method has been called, but the status and headers are not yet available.
- (3) INTERACTIVE Some data has been received. Calling the responseBody and responseText properties at this state to obtain partial results will return an error, because status and response headers are not fully available.
- (4) COMPLETED All the data has been received, and the complete data is available in the responseBody and responseText properties.
Not only does state 2 implement that, but state 3 is functionally equivalent as well. Since you can't query any of the headers or even start processing a partial stream in state 3, from my perspective it isn't really providing anything extra. The only thing worth hooking here is state 4.
Well, there goes my attempt to speed up the processing by doing as much work as possible in the early states while it is still receiving data. Thanks IE.
Made some more changes, and moved the details of TaskPool to it's own page:
behold!
I've been doing XUL work in XEmacs, and it has been pissing me off that xml-mode wasn't doing indenting or syntax highlighting automatically. I finally asked Nathan about it, and we figured out this addition to my init.el to make working with XML and XUL files indent nicer:
;;make xul files start in xml mode
(setq auto-mode-alist
(append '(("\.xul$" . xml-mode)) auto-mode-alist))
;;make xml-mode indent
(add-hook 'xml-mode-hook
(lambda ()
(setq sgml-indent-data 'T)))
Much, much, much better.
So a while back Nathan made a post about playing with macros.
Since I have wanted to start playing with lisp for a while and I finally got slime going at home, I thought this would be as good a jumping in point as any.
My first endevors where to simply recreate what Nathan had written with out cheating too much. I ended up getting it pretty much right, except that I had missed a #' on the memoize call.
Since this had taken relativly little time, I converted the macro to use gensyms, and tried to think of something else to do. One of the things that always intrigued me a little was the way
that symbols are strings that can be manipulated like most anything else. Memoizing is a pretty basic caching scheme, but it was missing something: the ability to clear the cache.
I decided that I would make defmemoize create two functions; one that would call the original function memoized, and one that would clear that functions cache.
(defmemoized foo (x y z) (sleep x) (+ y z))
;;this should put the functions (foo ,) memoized and (foo-clear-cache) into package scope
On my first attempt, it became pretty obvious that I didnt have a clue about the timing of this.
So I sat down to reconsider what I was doing and realized that, if I was going to declare memoize (the function) in local scope, I needed to move that into the backquoted area.
The reason to declare the function in local scope, is that we can pull the cache hashtable out into a slightly larger local scope so that my clear-cache function has access to the same hashtable.
This lead to version 2..
Now that I had everything scoped correctly, I needed to figure out why my symbol munging wasnt working.
Nathan was working with me and he suggested that I use intern instead of make-symbol. This sort of worked, except that rather than the symbol I wanted,
I got the symbol I wanted inside of pipes (ex: foo-clear-cache -> |foo-clear-cache|). After some mucking about and trying different things,
Nathan noticed that the example code was using capital letters in the keyword arguments. This combined with playing with the function for a few minutes, revealed to him, that if the string we are trying to symbolize is in all caps when we intern it, it comes without pipes. Glad LISP is case insensitive, except when it isnt.
After, we figured out the symbol munging, the last thing I had to do was figure out which symbols should be #' when passed to funcall and which shouldnt. While a little arduous do to CL's 2 cell system, I finally managed to work it out in my final version. Along the way I came up with a little test script that allowed me to verify that both functions were working.
Over all, I am liking LISP a lot, but there is definatly a learning curve. I am hoping that now that I have gotten my feet wet a bit, I will be able to get into it a little more.