Slashdot

I'd completely forgotten I wrote this three years ago:

Joshua Kinsberg has been released. But his bike and invention are impounded, at least until his court date on Friday (after the RNC is over).

http://www.msnbc.msn.com/id/5850151/

NYC's anti-graffiti law is very strict:

http://www.nyc.gov/html/nograffiti/html/legislation.html

"No person shall write, paint or draw any inscription, figure or mark of any type on any public or private building or other structure or any other real or personal property owned, operated or maintained by a public benefit corporation, the city of New York or any agency or instrumentality thereof or by any person, firm, or corporation, or any personal property maintained on a city street or other city-owned property pursuant to a franchise, concession or revocable consent granted by the city, unless the express permission of the owner or operator of the property has been obtained."

I wonder if the framers of that law realized they were banning kids from chalking hopscotch onto their schools' playground or onto the sidewalks in front of their houses. I wonder how many children have been arrested for chalking up a 4-square game.

One important point: the police did not see the chalk-spraying invention being _used_. So the inventor probably could not have been charged with the above law. But the only other anti-graffiti laws describe "aerosol spray paint cans," and the video of the arrest clearly shows the inventor explaining to the police that it uses chalk, not paint. Predictably, the New York Post gets that wrong, describing the invention as "a convoluted spray-paint mechanism": http://www.nypost.com/news/regionalnews/29532.htm

Earlier this month, a family was threatened with a $300 fine for their 6-year-old girl's drawing with sidewalk chalk.

On her own front step.

It's legal of course. The police screwed up. Notice the final clause from the law as of 2004 ("unless... permission of the owner... has been obtained") and the similar clause from the 2005 law Natalie Shea was threatened with (only if "not consented to by the owner").

But a street artist was later arrested for drawing on the sidewalk with chalk (while being filmed by PBS about his artwork!). And I won't be surprised if sooner or later some kid literally chalking hopscotch onto the sidewalk or a school playground gets arrested.

That's the law, after all. We had to make sure nobody chalked anti-Bush slogans while the RNC was in town. And the law's the law.

This is exactly what we do on Slashdot, of course. Every hit, whether to a dynamically-generated perl script page, or to a static .shtml or .rss page, triggers an Apache PerlCleanupHandler which inserts a row into our 'accesslog' table on our MySQL database.

(By putting it in the cleanup phase, we ensure it doesn't affect page delivery times at all; it just means a few more milliseconds that the httpd child is occupied instead of being available to deliver pages, but the only resource it's taking up is RAM.)

Dan writes:

I'm uncomfortable with this solution because it's hard to make it scale. First, you have to hit a database (of some kind) to cross-reference the client IP address with its last fetch time. Maybe that's not a big deal; after all, you're hitting the database to read your website data too. But then you have to write to the database in order to record the new fetch time (if the RSS feed has changed), and database writes are slow.

I'll grant that our accesslog traffic is pretty I/O intensive. But if you were only talking about logging RSS hits and nothing else, it'd be a piece of cake. The table just needs three columns (timestamp, IP address, numeric autoincrement primary key). You expire old entries by deleting off one end of the table while you insert into the other. That way inserts never block, even under MyISAM (though I'd recommend InnoDB).

You only need to keep about an hour of the table around anyway, so it's going to be really slow. How many RSS hits can you get in an hour? A hundred thousand? That's peanuts, especially since each row is fixed size. Crunch that IP address down to a 32-bit int before writing it and each row is 12 bytes, give or take. Throw in the indexes and the whole table is a few megabytes. Even a slow disk should be able to keep up -- but if you're concerned about performance, heck, throw it in RAM.

To catch bandwidth hogs, you create a secondary table that doesn't have so much churn. It has an extra column for the count of RSS hits, so if some miscreant nails your webserver 1,000 times in a minute, the secondary table only gets 1 row. You periodically (every minute or two) check the max id on that table, then

INSERT INTO secondary_table SELECT ip, MAX(ts), COUNT(*) FROM table WHERE id BETWEEN last_checked+1 AND current_max GROUP BY ip

By limiting the id to a range, again, there is no blocking issue with the ongoing inserts. After doing that, you trim off rows from secondary_table older than an exact time amount, and then you're ready to do the only query that even approaches being expensive:

SELECT ip, SUM(hitcount) AS s FROM secondary_table HAVING s > your_limit GROUP BY ip

and you have your list of IP addresses that have exceeded your limit.

What we do is use that data to update a table that keeps track of IP addresses that need to be banned from RSS, and have a PerlAccessHandler function that checks a (heavily cached) copy of that table to see whether the incoming IP gets to proceed to the response phase or not.

Slashdot's resource requirements are actually a lot higher than this, since we log every hit instead of just RSS, we log the query string, user-agent, and so on -- and also because we've voluntarily taken on the privacy burden of MD5'ing incoming IP addresses so we don't know where users are coming from. That makes our IP address field 28 bytes longer than it has to be. But even so, we don't have performance issues. Slashdot's secondary table processing takes about 10-15 seconds every 2 minutes.

As for Dan's concern about IP addresses hidden behind address translation -- yep, that's a concern. (We don't bother checking user-agent because idiots writing RSS-bombing scripts would just spam us with random agents.) The good news is that you can set your limits pretty high and still function, since a large chunk of your incoming bandwidth is that top fraction of a percent of hits that are poorly-written scripts. Even a large number of RSS feeds behind a proxy shouldn't be that magnitude of traffic. We do get reader complaints, though, and for a sample of them, anyone thinking about doing this might want to read this thread first.

The question is how much. The consensus seems to be that if we hit the $60-70 level and stay there for a few months, we're definitely looking at another recession. But what damage could a $50-60 price do? Stephen Roach of Morgan Stanley was concerned, saying in August that

With oil prices now in the high $40s (WTI basis), there is good reason to treat this development as yet another in a long string of energy shocks. The impact of such disruptions depends very much on context -- namely, the vulnerability, or lack thereof, in the underlying economy. When a weak economy is hit by any type of a shock, recession normally results. Conversely, a strong economy is better insulated to withstand such a blow. Most of the oil shocks of the past fall into the former category -- hitting economies when they are vulnerable. Unfortunately, the Oil Shock of 2004 fits that script to a tee. [...]

At the current level of around $47, oil prices are 62% above the $29 average that has prevailed since early 2000. That takes the "real" oil price (i.e., WTI quotes deflated by the headline CPI) back to levels last seen in the late 1980s; in fact, other than the brief spike in late 1990, the current increase represents the sharpest run-up in the real oil price since the late 1970s. I have maintained for some time that the "true" shock probably comes with $50 oil (see my May 10 dispatch, "Global Wildcards"). That would represent in excess of a 70% surge above the post-2000 average -- enough of a spike, in my view, to put it in the ballpark with full-blown oil shocks of the past.

and

"The economy is near its tipping point," Stephen S. Roach, chief economist for Morgan Stanley, said yesterday. He said the nation would likely fall back into recession if oil prices hover near $50 a barrel for three to six months.

"This is an oil shock, absolutely," Roach said, noting that yesterday's closing price was 68 percent higher than the roughly $29 per barrel average that had prevailed since early 2000. "The oil price is high enough to make a real difference to a vulnerable U.S and global economy."

I wonder if W's economic legacy will be a W-shaped recovery.

Update, Nov. 1, 2004: Oil Down $2, Speculators Bet on Kerry Win - LONDON (Reuters) - Oil prices fell heavily on Monday, taking U.S. crude below $50 on speculation that a U.S. election win for Senator John Kerry could ease the geopolitical friction that helped fuel this year's record-breaking rally.

Update, April 8, 2005: "The economists have changed their minds," says the WSJ. The economists who were saying last August that $50-60 oil would cause a recession have bumped their estimate up to $80-90. It's quite possible they were too pessimistic last year (and that I was foolish to believe them). But this writer wonders if they shouldn't have stuck to their guns. Oil peaked recently in the high-$50s, but has declined all this week, to $53.

So any anti-virus software that detects a virus, and then bounces a reply back to the alleged "sender," with a warning about how their product stopped the virus, serves no purpose except to advertise their product.

Such emails are (1) unsolicited and (2) commercial, and are therefore spam.

Example of spam I received from a Sophos product:

Dear Sender,

The Hays Personnel Services Internet Gateway has detected a virus in an email message that you sent. The email has been quarantined and has not been delivered to its intended recipient(s) .

Please scan and clean all your files and attachments to ensure they are free of viruses and then re-send your message.

For your reference, the details of the message you sent are:
Subject: hello
Date: Thu, 12 Feb 2004 11:20:25 +0800
Recipients:
[redacted]

The Virus Detected: Scenarios/Incoming/Incoming Sophos Virus Scan: A virus has been detected: 'W32/MyDoom-A'.

[...]

A number of current viruses spoof the senders email address. If this email has been sent to you in error please accept our apologies.

For further information on the virus specified above, please refer to http://www.sophos.com/ virusinfo/

Whoever wrote that software either knew or should have known that MyDoom spoofs the From line. Therefore, the only reason for sending that mail to me was to say "look how great Sophos is at protecting this company from viruses -- maybe it can protect your company too!" Ironically, that company offers anti-spam solutions as well!

I offer a warning to any company thinking about installing an anti-virus email filter -- if you pick a product that responds to viruses by sending spam, your company's mail server may well be blocked by other mail servers around the world. It's not fair, but that's the way the world works now.

To anyone who writes a review of anti-virus email software: warn your readers off any package which spams!

And to anti-virus companies who engage in this sleazy scam: screw you.

I think it'd be fun, but... maybe next month. Sorry.

I'd post this note on meetup.com, but it doesn't seem like there's any way for me to do so, without giving them money. And I think I'll attend at least one real-world meeting of some kind before I'll be doing that.