FinelyCultured

Time, Space, and our Lonely Island

2011-07-09T00:00:00-07:00

Let’s take a moment to look at our place in the universe.

Now, we’re nowhere near what would be called the center of the universe, but since time and space are relative, we’ll just go ahead and place ourselves at the center of space - after all, from our perspective, we are. This makes things a bit more convenient. As to time, we’re about 13.75Bn years along that axis - for now, we’ll make that relative to the universe. So, if we graph our position in spacetime, we can roughly get this:

Since we can’t see into the future, effectively our perception stops at our current position on the time axis - we can’t see anything past 13.75Bn years on the time axis. The thing is, that’s not our true constraint - the speed of light is our true constraint. Now, the name “Speed of Light” is a tad misleading - What we call the Speed of Light isn’t the speed at which light moves, it’s the maximum speed at which anything in the universe moves. It is the absolute speed limit of our universe: Light happens to move at the top speed available in the universe, which is why we call it the speed of light, but there’s nothing else that moves faster - we can’t get information from one place to another faster than C, which happens to be about 300,000,000 m/s.

This effectively puts blinders on our ability to see the universe - because we can’t transfer information instantaneously, we can’t observe our universe synchronously - everything we see is delayed by the speed limit. Our perceptual universe, then, looks like this:

Anything in the universe that occurs outside that triangle doesn’t exist to us: That is our island in the universe. Our consciousness grows at 300,000,000 m/s, or about 10 trillion Km per year - a pretty good clip, by most measures, but by the scale of the universe, not much. What’s more, the further out we see, the older we see the universe. If we’re gazing at a system that’s five thousand light years away, we’re seeing it as it was five thousand years ago - we have no ability to percieve what’s happened in the five thousand years following, and no ability to see that system as it exists now: for all intents, the 5000 years since we saw that galaxy haven’t happened.

Note that this goes both ways - as far as that galaxy is concerned, the last 5000 years of our history haven’t happened either. For looking at star systems, this isn’t too bad - galaxies operate on a very long time scale. Think of the human race, though, 5000 years ago: we had agriculture and had finished domesticating the animals. Were we to train our instruments on ourselves 5000 years ago, we would have seen nothing. In fact, the earliest reasonably powerful radio transmissions occurred maybe a hundred years ago - let’s turn our ‘perception triangle’ upside down for a minute:

For another species to be aware of us, they need to be within this triangle. For us to hear back from them takes twice as long as for them to hear from us, so they’ve got half the triangle. In short, for a species to know we exist, they must be within 100 light years, and for us to hear back now, they’d need to be within 50. An intelligent species which exists outside that triangle is invisible to us, and us to them - it could take thousands of years before our perceptions overlapped, and the longer it takes, the less ability we have to engage in a two-way dialog.

Beyond a certain distance, effective communication becomes impossible: sentient timescales simply don’t allow it. A hypothetical species which happened upon our first transmissions today would be responding to a species for whom powered flight was still a dream. Think of the last hundred years of human development and project forward another hundred - and remember this is the best possible scenario and the absolute tip of our perceptual island! In what capacity could we maintain a dialogue over these timescales? To whom would a species be speaking in a thousand years?

The size of this triangle will grow as we move forward, as will our perceptual island in space, but for all the wonders and size of the universe, we’re stuck between two constraints: The infathomable size of the universe and our inability to percieve it.

My appologies for what’s turned into a very depressing post. I’m a child of Sci-Fi: I believe in mankind’s future among the stars, and I can hope for a way to violate what seem to be the iron-clad rules of the universe, but, at least for today, humanity are the only inhabitants of our island: alone in our own universe, waiting for something else to wash up on our shores.

(For a moderately less depressing view of man in the universe, I suggest David Deutsch’s brilliant TED talk)

Some Notes on Money

2011-06-14T00:00:00-07:00

With a massive deficit and a crippling recession afoot, I think it’s helpful to consider the nature of money. If you’ve a passing familiarity with the field of economics, much of this will likely be remedial, but I find it useful to refresh every now and again, as money can become a concept rather divorced from itself in the face of heated debate. If you’re new to the subject, I hope this will be illuminating.

What is Money?

The simple answer is, money is what we make of it. There are some fundamental precepts money must follow, but these are largely driven by perception, even when dealing with ‘hard money’ such as silver and gold. More broadly, Money has three basic purposes:

A medium of exchange: Money facilitates trades between different parties
A unit of account: Money allows different goods to be valued on a common scale
A store of value: Money allows individuals to enact different halves of a trade at different times.

The two most important properties are the first and third: a medium of exchange and a store of value. They also conflict with one another: As a medium of exchange, an ample money supply is a virtue, whereas a store of value relies fundamentally on scarcity. In the face of the current political climate, I’m not sure the latter value needs much defense: the dangers of inflation seem well represented in the ideological forum these days. I’ll therefore focus on the former, whose prerequisites seem sadly bereft of popular support at the moment.

Money as a Medium of Exchange

In a barter economy, trade is neccessarily symmetric, which means if someone who has an item you want doesn’t want any of the items you have, you cannot affect an exchange. Likewise, if someone else wants an item of yours, but has nothing you want, the exchange cannot be executed. What’s needed is a third, intermediate good, which can facilitate these trades. Historically, the first solution to this problem was debt: Bob is owed three ears of corn by Sally, an obligation he can pass to another in exchange for other goods. This system of debts, however, requires either good relationships or good bookkeeping (the first use of written language), and a closed system in either case. Money, originally a common commodity such as shells, allows traders to carry their debts and credits between systems: as long as the currency of choice is recognized in two different markets, trust relationships are not required to carry over between the two.

However, by electing a commodity as our medium of exchange, we now need enough of that commodity to facilitate whatever trades we want, and herein lies the rub: if I want to buy three ears of corn from a man who wants to buy a loaf of bread, and six shells is the generally accepted value of both three ears of corn and a loaf of bread, and only three shells are available in the market, I can’t obtain the corn and the other man can’t obtain the bread, regardless of the virtue of the trade.

In short, a shortage of money restricts all trade, regardless of willingness, mutual benefit, societal advantage, or relative availability of resources.

Since money and trade are fundamentally expressions of value rendered (I value the man who gives me corn as much as he values the man who gives him bread), the amount of trade that can potentially be transacted in a market is a direct function of the overall value which can be rendered by all members of the market. This, in turn, is a function of population and skill level of these people and the resources available to them. For money not to restrict trade, there must be enough of it available to fully represent all the value which can be rendered by all the people in an economy. Money supply, then, must trend with population (typically up), productivity (hopefully up), and resource availability (a mixed bag). In general, we need a positive growth rate of money. This is why true hard or fixed currencies can be a significant drag on an economy: The population may grow, the skill level may increase, but if there’s not enough money to go around, we can’t reap the full benefits of societies’ members.

But You Don’t Trade in Shells

And neither do I - I trade in dollars, and we need more of them in the market. Currently, we have a 9% unemployment level, a 3% rate on 10y bonds, and short-term inflation that functionally doesn’t exist. By every impression, we have a real economy that’s short on currency: 9% of the population wants to work (and 20% wants to work more), there’s practically zero concern about default risk on treasury bills from the actual market, and no sign whatsoever of anything that looks like an overheated economy - in fact, we’ve got a raft of skilled people sitting around twiddling their thumbs. Our GDP growth rate might be positive again, but we’re 3% below the long-term trend, which means there are 3% more transaction that could be realized in the economy than currently are. We have massive amounts of value that could be released in the economy but that sits around for want of money, a commodity whose quantity we control outright.

But we’re not arguing about how to jump-start the economy or how to get money into the hands of people who need and will spend it - we’re arguing about the deficit and how to remove even more money from the economy. Money is the lifeblood of the economy: Our patient took a bullet and right now we’re arguing how much more we should bleed them before they start to feel better.

Digital Manors and Economic Experiments

2011-04-25T00:00:00-07:00

The IFTF’s Marina Gorbis published a post today about the HuffPost v. Bloggers lawsuits discussing growing discontent over what she dubs the “Digital Manor” economy - the crux of the argument is the growing discontent over large companies reaping financial benefit from the freely (or very cheaply) contributed works of their users.

Though I spend an awful lot of time writing code, I’m an economist by training. One of the things I love about economics is that, while you can’t run a controlled experiment, any time you get a few people together long enough you’re basically running an experiment (I suspect the body of modern economic theory owes more to the methods of zoology than physics).

As the internet became popularized in the mid-90’s, its promise was of a vastly democratic medium, an attribute maintained to a large degree today. Where the “Manors” began to form is essentially in marketing - the bloggers who wrote for HuffPost didn’t do so because they Really Liked Arianna Huffington, they did so for a pittance and a lot of publicity. Add some buzzwords about Network Effects and Virtuous Cycles and you wind up with Facebook, Google, and HuffPost being multi-billion dollar companies.

So now we’ve learned something new about people and economics - you can control for production costs (in this case, the cost of producing a blog is close to zero), barriers to entry (again, practically zero), and geography (the internet is “everywhere”) and still wind up with industrial consolidation - potentially moreso than with those factors, since whereas in a pre-digital economy HuffPost might have been a large regional publishing house or magazine, now they’re one of the biggest political commentary sites in the world. Scale economies dominate in a world with no variable costs.

Before they became the powerhouse they are now, though, the bloggers happily worked for next to nothing for the HuffPost. I suspect many felt they were part of a community - and in a real sense, they were. The HuffPost model hasn’t changed - they’ve been content aggregators since they started allowing guest bloggers - but now they’re showing very real revenue, and that’s the seed of this discontent. The issue isn’t necessarily the model, it’s that people are seeing what they see as the full economic benefit of their contribution going primarily to the coordinating group, as opposed to the workers - that old Marxist chestnut.

But that’s the trick of network effects - they’re nonlinear. The value of an individual’s contribution to the network is variable on the size of the network, and in very short order the overall value that the network adds to a users’ content outstrips the value of that content itself. In the case of the HuffPost, the users were happy to contribute for next to nothing to participate in the community and get name recognition - the value they placed on their contribution was relatively low. The overall value of HuffPost, though, grew far faster than just the aggregate values of the blogger’s contributions.

Now we’ve got our second experiment, and we’ve repeated it across many of the popular sites on the internet: When you strip away communication and coordination costs, what’s the value of the individual versus the network, and to what degree do network effects scale? Like all good economic experiments, we can only draw rough analogs here, but in a matter of less than a decade, we’ve seen HuffPost go from a blog to a $300M digital manor, Facebook from a small social network to a $50Bn enterprise touching 10% of the planet, and the Apple App Store to 10Bn downloads.

Manor Lords lived from aggregated taxes on their lands, and Marina’s correct in drawing the parallels to these modern companies - the basic model is the same. I’m not versed in economic history, but I’d wager the early history of the feudal model would look awful similar in terms of tradeoffs.?

As an aside, It’s worth noting that I’ve had the pleasure of attending several events sponsored or hosted by the IFTF - Among their inspirations in their projections and projects, and I don’t think the organization would disagree with this, are the conversations and ideas presented at these events. The IFTF’s role, as I’ve seen it, is both as an original researcher and an aggregator of ideas, and in this fashion they operate a similar ‘digital manor’ model. That’s certainly not to criticize the organization (I rather enjoy the events and conversations), but to say I suspect the true issue is in the scale of aggregated benefits, not necessarily the model itself.

Yahoo, Social Gaming, and Failure

2010-12-16T00:00:00-08:00

If you’re in San Francisco, you may have begun seeing Yahoo’s new bus stop additions: They’ve done what Japan did a decade ago and added touch screens. We’re not talking just a little touch screen, either - Yahoo’s placed 60+” LCD Screens where an entire full-size ad used to go. This is awesome - the potential here is amazing. We can have huge, zoomable, scrollable route maps, get instant directions and transfer info, see where the next busses are, find out if it’s going to rain where we’re going, get flash news updates, find restaurants close by, and even add information for tourists. With an easy interface, an internet connection, and some smarts, Yahoo might have done the coolest thing to bus stops in the states since we added busses.

Unfortunately, Yahoo didn’t. Yahoo took a 60” touch screen, an Internet connection, and a group of customers who all want to be somewhere else, and put games in front of them. No, that’s not a typo. These awesome pieces of interconnected technology are host to just four crappy “social” games.

I’m not even going to touch what games Yahoo decided to put on the bus stop, because it doesn’t matter. All you need to know is that the games are absolutely terrible - they’re unplayably bad. But that really doesn’t matter. You could put World of Duty Modern Warcraft: Black Halo 2 on there and it wouldn’t matter: This Is Not A Place For Social Games, period. This isn’t just a lapse in judgement - a lapse in judgement would be putting these god-awful games on their homepage. Yahoo bought and paid for the installation of many thousands of dollars worth of equipment around the city to pull off this fiasco. Whoever gave this the green light needs to be placed as far away from shareholder money as possible, preferably with a warning sign nearby.

Now, Yahoo announced they’re laying off 700 people and closing down a bunch of their properties. Among them is Delicious, which started as del.icio.us before Yahoo decided the periods were gauche, and so was good interface design, a reasonable API, a responsive website, and just about everything else in the world that made del.icio.us awesome. The reason Delicious failed is Yahoo, pure and simple: the site was great, Yahoo bought it, made it suck, and now they’re killing it. It’s like buying a prize racing horse, feeding it nothing but cheeseburgers for a year, and then shooting it because it’s not winning races anymore.

If the SF bus stop fiasco is any sign of how things are running over in Yahoo-land, I think Delicious might be a mercy killing - At least now the engineers don’t have to keep pretending putting a word scramble in a bus stop is a good idea.

On Social Security, Data, and Bullshit

2010-11-29T00:00:00-08:00

It’s budget deficit season again, which is a polite way of saying it’s time to tie Social Security to a tree and beat it like a piñata until ~~candy~~ fiscal solvency comes out.

I’d like to start by clarifying what Social Security is, and what it’s not. Social Security is not a retirement fund. It’s not an investment vehicle. It’s not designed to turn a profit, nor is it supposed to seek the highest returns. Social Security is a program we created in the 1930s because we were tired of watching the elderly die in the streets. It’s a humanitarian program designed to make sure that the less-fortunate among us can spend their last few years in dignity. It’s a hedge against both our own human fallibility and the vicissitudes of fortune, which were on sharp display in 2008-2010. Most of all, it’s an assertion by American society that we can take care of all of our people. If you’re rich enough or callous enough that none of that matters to you, consider it an anti-guillotine fund - like most of our social programs, it exists because if you don’t give the people a fair shake, they’ll take it, often by force. It’s an insurance policy for us and for our society because Shit Happens.

Now, as far as solvency is concerned: As of October 2010, the CBO estimated that the Social Security trust fund would be depleted by around 2050. At that point, benefits would have to drop to around 75% of their current value. Maintaining program solvency for the next 75 years would require a 1.5% increase in Social Security taxes. Putting aside both the difficulty of projecting finances out for 40+ years and the fact that the current projections are being drawn in a particularly adverse climate, the numbers are still not overly pessimistic, and certainly not the harbinger of fiscal doom they’re often made out to be.

How do we restore solvency past 2085? Let’s talk about a policy that won’t work, a policy that will work, and a policy nobody will like.

Increasing the age at which a recipient can start receiving social security is an extremely popular fix. The program was first implemented in the late 1930’s with a retirement age of 65. Retirement age is now 67. But life expectancy as of 1950 was only 68, whereas it’s around 78 now - that’s an increase of 10 years with no corresponding increase in retirement age. The current proposal is to raise retirement age to 69, which sounds reasonable. The problem is, the increase in life expectancy has largely come from lower mortality rates among children under 5 - remember, life expectancy is usually a mean, not a median value.

Pulling numbers from the CDC, we can see life expectancy from birth and from age 65:

Year	Birth	From 65
1920	54.1	76.9
1950	68.2	79.1
1980	73.7	81.5
2007	77.9	83.6

Between 1950 and 2007, overall life expectancy rose by 9.7 years. Life expectancy after 65 rose by 4.5 years. The numbers are even more dramatic if you look at the data from 1920. The overwhelming increase in life expectancy has been a decrease in child mortality. We’re not living a whole lot longer, we’re just dying a whole lot less. Add to that a report from the GAO that suggests increasing the retirement age may increase disability claims, and this policy looks even more like a lemon.

What would work, though, is relaxing the upper limit for taxable income. Social Security is capped at the first $106,000 of income (in 2011). According to the US Census, there are around 23.5M households making over $100k. They comprise the upper 20% of American households, but their share of total income in the US is - and I have to ballpark here a bit, since the IRS only gives 10% & 25% - roughly 50%.

Let’s do some mathemagic: (Note: Some of the income numbers are from 2008 -they’re IRS numbers)

Total Income (AGI): $8.4 Trillion.
Total Income of top 25%: $8.4Tn * ~50% = $4.2Tn
Total Income of top 25% subject to Social Security Tax: $100k * 23.5M households = $2.35Tn
Total Income of top 25% Not subject to Social Security Tax: $4.2Tn - $2.35Tn = $1.85Tn.

Because social security doesn’t tax above $100k, $1.85Tn of income is not subject to Social Security taxes. We can either enact a 1.8% tax increase on the current wage base or we can increase the base - one of these is an extremely regressive tax, the other is a progressive tax.

Finally, let’s look at means-testing for Social Security disbursements - which means if your retirement savings are already “sufficient”, you’re ineligible to collect Social Security. This is attractive because it shifts the benefits of the program onto the group that needs them and away from those who don’t. It’s unattractive and probably untenable for few reasons.

First, it’s not entirely apparent what “sufficient” would mean. “Sufficient” in Nebraska is destitute on the street in California, and “Sufficient” in Riverside, CA is destitute on the street in Orange County, CA. It’s also subject to adverse selection effects, as I’m sure any good economist would point out - if I know that I can’t collect if my retirement savings exceed (say) $5M, I won’t save beyond $5M unless I’m sure I can make up for it. From a technical standpoint, this is an extremely difficult policy to craft properly.

Second, there’s a moral aspect. I’ve stated my beliefs on Social Security, but they’re just that - my beliefs. If we’re asking everyone to pay in, but we’re not paying out to everyone, we’re engaging in blatant income redistribution, instead of a universal safety net. I’ve a hard time taking another person’s money on the sole dint of my beliefs (a hard time, not an inability), and in this case I’d be doing so with little more than intangibles in return.

I’m quite strongly of the opinion that Social Security is valuable and fundamentally solvent. I think some small changes may need to be made for the overall long-term solvency, but they’re not dramatic, and overall the program is well-designed. I know people may disagree, and that’s fine - this is a democracy, after all. What I’m not OK with is bullshit numbers being bandied about by people with a chip on their shoulder. Public policy has a lot of room for dissent, but no place for ignorance or maliciousness.

Numbers presented here are primarily from the CDC, IRS, US Census, and Social Security administration

Building Documentation With Mediawiki

2010-08-09T00:00:00-07:00

I’ve been talking with a lot of people lately about what I do, and I’ve gotten a lot of questions about our use of Mediawiki - largely in the “Why are you using that?” vein. The first few times I didn’t have a great answer, for reasons anyone who’s really worked with Mediawiki knows - it’s just not that pleasant a system, and it feels crufty, especially when I’m talking to Agile Ruby Devs. It’s worked great for our systems, though, and I wanted to share some of the benefits we’ve gotten, as well as some of the drawbacks of working with Mediawiki.

(Incidentally, if you’re checking the Wiki link above before August 30th or so, you’re not seeing the latest version of the Wiki, which includes around 9 months of improvements.)

Who Am I to Talk About This?

For the last 18 months I’ve been working at Embarcadero Technologies converting the whole of the RADStudio/Delphi/C++Builder documentation over to a MediaWiki based system. As of right now, I have a 7-server continuous-build system designed to build and support both product and language documentation in four languages. I’ve got 18 wikis running off a single codebase with custom skins, custom extensions, and scripts writing and retrieving around half a million pages on a pretty regular basis. It’s fair to say at this point I’m familiar with deploying, maintaining, and working with Mediawiki, both the good and the bad.

Why Mediawiki?

While it brings its share of problems, Mediawiki does a phenomenal amount of what we needed right out of the box. Given that we’ve got an extremely limited budget and an even more limited development team to support a professional commercial product, Mediawiki’s capabilities have been invaluable to us.

Here’s what we’re getting right away:

The Good

Easy Syntax

This was the #1 reason we went with Mediawiki - We wanted something where anyone could write new documentation easily, and eventually we wanted to open up the documentation to contributions from customers. Mediawiki has an easy-to-use syntax, and it’s a syntax that many people are already familiar with. Anyone trying to get user interaction on a site knows that even if you write the content for the user you’re still asking too much of them - we wanted as low a barrier to entry as possible, and it’s a fair guess everyone who hits our site has at least Seen Wiki syntax at some point. So far it’s working: one of my best moments at work was when our localization manager complained in jest that our writers were too productive for his budget. We’ve seen a 40% increase in output over the old XML-based system, which is great, and it’s even easier to bring new people up to speed, which is crucial.

Localization

I’m not sure how strong a consideration this was at the beginning, but boy has it saved us time and stress. Our documentation goes into four different languages, so having a system that handled Japanese out of the box has been crucial for us. Mediawiki has a robust localization framework built in, which is another part of the system we didn’t have to write.

Accessibility

I’m honestly not sure where we sit on the regulations on this one - I’m not sure if our company is required to have a 509 solution. Fortunately, I don’t have to worry about it, because it’s baked in right out of the box.

Scale

Our documentation is around 70,000 pages per language, times four languages, times two product versions. Half of the pages are duplicated because of how our scripts work, so we’re at ballpark 750,000 pages on the DocWiki system. We needed a system we knew could handle anything we threw at it, and Mediawiki’s one of the few that have been field tested to be able to stand up to just about anything. Other CMS’s may be able to handle this sort of load, but we absolutely knew Mediawiki could - we don’t have the resources to plow a few months of dev into a system and have it collapse once we put all our documentation in.

Revision Management

Built in, out of the box, including diffs and comments. We transitioned away from SVN to Mediawiki, so the revision tracking made it very easy for us to track our changes. It’s got easy-to-use diffs and an RSS feed, so we can keep tabs both on the Wiki and on the writers. It’s also got a detailed log, so we can tell who was responsible for anything that happens on the system.

Robust User Management

Including Roles. There’s some quirks here, but overall, it’s another thing we didn’t have to worry about. I’ve been able to expand on this quite a bit to allow mass user imports from our other systems, user searches, and a few other neat tricks that have made our lives easier.

API

We auto-generate a lot of our content, so we needed a clean way to edit certain pages without affecting others. Mediawiki has an API baked in, and the mwclient python scripts work wonders. Another part of the system we didn’t have to build.

Extensibility & Community

Because we’re using a packaged solution for a fairly specialized use case, extensibility was part of the spec. Mediawiki has a shockingly large number of hooks for extensions, and really allows us to do just about anything we want with the right combination of calls. It’s also got an enormous developer community, including the WikiMedia foundation, which contributes its code back to the MW community. As a bonus, you can see anything that’s in use on the WikiMedia servers, which guarantees the code has been put through its paces.

Skinability

In our case, we were able to change the look of the default Mediawiki skin enough to get a distinct look, but at the same time, it’s still clearly Mediawiki. This is a bonus for us, since it tells users what system they’re using and saves us a whole lot of user training.

Lamp & Caching

Say what you will about the LAMP stack, but we’re on an extremely restricted development budget - we needed something that worked. LAMP gets us up and running in 20 minutes on a stack that’s in use practically everywhere - there’s not a problem we’re going to hit that someone else hasn’t seen already. Mediawiki also supports APC, Squid & Memcached right out of the box - with everything set up, we’re catching almost everything somewhere in the caches.

What’s Bad?

It’s Unstructured

Mediawiki is not meant for structured content - it’s basically built for a flat hierarchy. This is a bit of a problem when you’re dealing with any serious product documentation, and was especially a problem for us, as we had highly structured content. We had to invent our way around the table of contents, the index, and the tagging we needed to make our shippable help files.

Search Is a Joke

The built-in Mediawiki search is just absolutely atrocious - even Wikimedia’s using Lucene instead. If you have any reasonable expectation of users finding content on your site, you need a different solution.

Architectural Limits

Because of how Mediawiki stores inter-page links and category information, pages that are extremely link-heavy tank the wiki when you try to save them. Looks like a database lock, but I’m not totally sure. It took me a long time to track this issue down, but that’s my leading contender for the mysterious “The wiki’s dead!” bug.

Spaghetti Code

Especially in the themes. Or at least, a codebase that’s so large and daunting it might as well be spaghetti code. Either way, it really looks like code which has been developed by a large number of different people, none of whom knew each other. Which it basically is, and it’s a triumph for that, but it’s not terribly readable.

PHP

Yeah, we all knew it was coming - PHP seriously sucks as a language. It’s the most universally-deployable web dev language around, but then again, McDonalds is the most universally-deployed hamburger-ingestion venue around - doesn’t make it good.

In a future post, I’ll talk a bit more about the specific challenges we faced and how we dealt with them. I think we’ve got some pretty cool stuff going into our wiki setup, and maybe someone will even find it useful.

If you have any questions, by all means, leave them in the comments below, and I’ll address them as best I can.

Thanks!

It's All About the AI

2010-06-14T00:00:00-07:00

I got to attend ARE2010 recently, which was an amazing lineup of speakers and a great show of the current state of Augmented Reality. The big takeaway for me, though, had nothing to do with AR and everything to do with AI.

Augmented Reality is, at its essence, just a new interface. In some cases, it’s a better interface, in some cases it’s not, but that’s basically what AR brings to the table: a new means of displaying information, albiet now in a way more integrated with the user’s physical surroundings. The promise is in that integration: visions of sci-fi heads-up displays, the sort of stuff the anime artists have been playing with for years, the ability to leverage our ridiculously well-developed online abilities in the real world.

The problem, though, is that we’ve got almost zero spare cognitive bandwidth to play with. The unfortunate fact is parsing what an alert from a mobile system is takes effort, and effort takes focus, and focus is a finite quantity limited to One: What am I focusing on Right Now? This is easily made apparent just by observing people trying to walk while using smart phones, and it’s also why the headset requirement in California is basically window-dressing: the problem isn’t where our hands are, it’s where our heads are.

So AR poses an interesting conundrum: On the one hand, with some work on machine vision and context filters (and goggles - god knows we don’t need “smartphone elbow” to become the carpal tunnel of the 21st century), it could be a radically new and useful way to get highly readable information about the world around you. On the other hand, current systems display entirely, totally, utterly too much information, even assuming someone’s actually focused on the information being displayed. It’s rare I need to know where Every coffee shop in the city is - normally I’m just looking for the best one. A smart, useful AR system needs to be able to filter the infinite feed of information down to the absolute most useful things you could need to know right now and display only those things.

The question of usefulness of the information itself is only half the battle, though. There’s also an issue of context - What am I doing Right Now? What sort of mood am I in? Am I looking for a coffee shop to meet a friend for a chat or a coffee shop to work for the day? Did I already have lunch, or should the place have good food, too? Am I a creature of habit, or do I crave novelty? What’s the weather like? Information Utility varies by person, by day, by time, and by circumstance.

This applies to notifications, too. If I’m engaged in a (physical) conversation, I probably don’t need to get that email about a coupon for a bookstore. The email about the fire in the server room, though, might warrant an interruption. Likewise, if I’m wandering around looking for something to do, the fact that a friend of mine just checked in to the bar a block away is relevant, whereas if I’m in the office, it’s probably not. What’s even trickier is that mood plays a big role, too. Maybe I’m just not feeling like going out today, or maybe I’m on a super-productive streak, in which case anything shy of the second coming of jesus should probably wait until I run out of steam.

What this boils down to is a system that knows you, knows where you are, knows what you’re doing, and knows how to filter all of that down to just what you need. It remembers what you’ve done before, recognizes by the amount of time you’ve spent somewhere and your overall mood there what you thought of the place, can recognize by your behavior what your likely mood is, can tell by your surroundings what you’re up to, and from all that can tailor its recommendations to what’s best for you right now. Basically, it’s a personal assistant in software form - an AI.

We’ve reached a point in the evolution of computers, the internet, and mobility where we no longer have to try to find information or to connect with people. We’ve wired everyone and everything to everywhere (or at least to Google) such that it’s not a question anymore about whether the information exists, it’s whether it floats to the surface of whatever interface we’ve slapped on top of the fire hose of data aimed at our eyeballs and our brains. We don’t have an Access problem anymore, we have an Excess problem, and there’s just no way that dumb filters (or even just slightly smart filters) are going to be able to carry us forward. We need learning filters. We need systems to sit between us and the fire hose, to act as the gatekeeper to the ludicrously limited cognitive time we’ve got. We need something to take the terabytes terabytes of information we’re constantly exposed to and strip it down to the most relevant 40 bits and deliver it to us so that we Know it, we don’t just see it or read it.

I’ve become increasingly aware of the value of cognitive downtime lately. Our brains need time to parse the materials they’re exposed to, and without taking time to pull out of the feed and parse the information we’ve obtained, we lack context and the ability to assimilate the knowledge we’ve gained. Minimizing non-essential or extraneous pulls on our attention allows us to focus more fully on the world around us and even allows us to relax and reflect. With mobility as it exists today, we’re fast-tracking ourselves towards a sort of mass social ADD, and AR without context is only going to worsen the matter.

(Credit to S.Applin, M.Ito, & others working on this problem. Apologies to any other unattributed shoulders on which I’m standing.)

Apple & Matsci

2010-06-08T00:00:00-07:00

Like every other geek in the bay, I eagerly followed the SteveNote on Monday, and I’ve gotta say, the new iPhone is quite a device.

As of right now I’m not terribly impressed with the software. I think android’s still winning that race, but I don’t think that’s where Apple’s really competing.

What caught me was the hardware. The phone itself is REALLY impressive. I’m not talking necessarily about the specs, which Apple’s always a little cagey about and which, frankly, matter a whole lot less than a number of other factors. I’m talking about the physical hardware they built.

Apple invented a new form of laminated glass, a new steel alloy, new manufacturing techniques, a new LCD screen, a new processor, and I’d put good money that neither those cameras nor that gyroscope(!) are off the shelf. The battery design is their own, and I suspect given the relative performance of the Apple touchscreen that Apple had a good hand in its design and manufacture as well. Apple has rapidly moved from a company that makes exceptional software to a company that’s absolutely obsessive about their hardware, and it shows in a big way with the latest iPhone.

Apple’s known for its relentless perfectionism, and it seems like the company realized they reached the limit of what off-the-shelf components could deliver. They’re investing heavily in manufacturing, materials science, and design, and they’re getting a lot of mileage out of it – the race to dethrone the iPhone isn’t about software anymore, which is good for Apple, because software’s cheap. Apple’s no longer a software company, or at least that’s not their competitive advantage – they’ve morphed into a class-leading hardware company, and I have to say, I’m impressed.

And more than a bit envious.

Quick Bits - Verify the Hash of a Download

2010-05-21T00:00:00-07:00

Just whipped up a script to check the SHA1 & MD5 hashes of a file and compare it to the clipboard. I rolled it into an Automator workflow so it can be attached as a folder action.

Basically, the script takes a parameter, which is the full file name, runs openssl sha1 and openssl dgst on it, and compares each value to what’s in the clipboard. If either signature match the clipboard, it pops up a growl notification confirming that the signature matches. If it doesn’t get a match, it growls the Sha1 and MD5 tags.

Get it here: hashworkflow.zip

You’ll need Growl, if you don’t have it, and you’ll need to install growlnotify, which comes with Growl.

Instructions:

Unzip.
Open the .workflow
Go to File -> Save As: (may not be necessary…)
Ctrl+click on your Downloads folder and select Services -> Folder Actions Setup…
Select “Hash Downloads.workflow”

Issues:

The script uses pbpaste to read the clipboard. Since there’s no way to guarantee pbpaste is giving back text, this may cause a problem when there’s large amounts of data on the clipboard. Clipboard contents are assigned to a variable using the syntax PB=pbpaste, so there shouldn’t be a risk of accidentally running a command that isn’t intended, but I haven’t extensively tested for this.

Disclaimer

I’ve tested this for approx. 6 minutes. I make no claims about the safety of this software, and take no responsibility for the results of running it. I took shortcuts and the code sucks. Caveat Downloador.

On Privacy, or Why Mark Zuckerberg Is a Social Pariah

2010-05-08T00:00:00-07:00

I went to PrivacyCampSF yesterday and got to engage in spirited debate on privacy with passionate, intelligent people. The takeaway for me was that this is a problem of relationships, not rules.

The first problem we have with privacy is that we simply don’t have a good definition for what we want kept private. We’re not used to having to define exactly what we consider our identity, nor what we consider a breach of privacy. These are conventions that we all see differently, and we do so on an ad-hoc basis. In each interpersonal relationship, we’re defining on the fly roughly what we consider our identity and roughly what we consider private information, and these definitions are mostly relative to the current circumstances. It’s not just that we’re not defining privacy, it’s that we just don’t have a concrete definition at all.

The second problem is the definition of Identity. We present different personas to different people - this is human nature. We behave differently around our friends than our co-workers, we act differently in different groups, and typically people who don’t do this come off as abrasive. Certain parts of our personalities are relevant to certain situations, and these are typically the parts we present in different scenarios.

The third problem that arises is that frequently what gets presented as our ‘identity’ online is, to say the least, an incomplete picture. A huge, huge part of human communication is non-verbal, and this is something that’s simply lacking online. Our words taken by themselves are far more susceptible to misinterpretation than the whole of our communication, and pictures and videos do not fully fill in the blanks. The problem, then, comes because our online persona is an amalgamation of small parts of our personality, and not a true representation of the whole.¹

I submit that there are three main situations in which we will feel our privacy or identity violated:

When the commitment level in an impersonal relationship is escalated without our consent: When a store starts addressing us by name or we become aware they’ve kept more data on us than we were aware, we feel like we’ve been stalked - and rightfully so. The nature of stalking is a non-consensual escalation of familiarity borne without trust. The behavior of businesses trying to establish “customer relationships” is extremely similar, and off-putting for the same reason.
When parts of our different identities become intermixed: We assume different identities in different groups largely as a means of convenience - certain information is not relevant to a relationship, nor something we want to have influence a relationship. When this information gets added (say, your coworkers find out the size of your genitals), this can be profoundly upsetting.
When we feel we’re being misrepresented: We cultivate our identities carefully, especially in our dealings with others, and we’ve a certain idea of what that identity is. When partial information is revealed or when we feel like we’re being unfairly or inaccurately represented, we feel a powerful need to correct this, and we get very upset. This includes attaching our name as an endorsement of a product or service - we consider this tantamount to misrepresenting us wholesale.

When we engage in a relationship (very broadly) with a person or an entity, we reveal parts of our identity to that entity. The degree to which we do so is commensurate with the trust we have in that person. Each of the three breaches above cause us to lose trust in that entity, as it’s a breach of an unspoken agreement easily as strong as whatever other agreement we’ve formed with that entity. In effect, any given transaction has two components: the transaction itself and the trust transaction.

Facebook has performed all three of these breaches, repeatedly and remorselessly. For those of us who started in college, we gave Facebook a certain part of our identity - typically one with more red cups and horticultural appreciation than what we’d present to those not privy to that persona. Facebook proceeded to open this network up to the rest of the world, instantly revealing a swath of information (and frequently, indiscretions) that we’d not have shared otherwise. Facebook has repeatedly revealed, without our consent (‘opt-out’ might as well say ‘f_ck you’), large amounts of personal information to people we haven’t decided to engage in relationships with, and revealed relationships in ways that were tantamount to slapping our endorsement - putting our identity up as collateral - on a given third party. Most recently, Facebook has, again without soliciting our consent, revealed our information by connection with our friends.

In every instance, Facebook has taken what was a trust relationship and treated it as a purely commercial transaction. We feel betrayed, because we were betrayed, even if Facebook wasn’t aware they were betraying anything.

Thanks to Sally, Doug, and the other people whose names elude me but whose ideas informed me. (Contact information withheld because my sense of irony is still intact.)

Note 1: It’s worth noting here that the ability to hide one’s true personality has helped many a wallflower build a more confident representation online. This ‘pseudonymity’ (gratz, privacycamp), therefore, is not universally negative, but it does obfuscate our true identity.

Using Apache2's Digest Authentication

2009-09-01T00:00:00-07:00

I finally got Apache’s Digest authentication, and since there’s a serious dearth of information online about getting Digest working, I decided I’d write up a bit. It’s not difficult, but I’ve had far more trouble than I should have, and most of that’s due to nonexistant documentation.

Why Digest?

Standard HTTP auth sends the password in plaintext, which is generally bad. On sites with an SSL cert, this is less of a problem, since the traffic is encrypted to begin with, but on smaller sites, it’s a lot of overhead to set up SSL. Digest authentication hashes the password before sending it - it’s an MD5, which isn’t great, but it’s also not your password flapping in the breeze, so to speak. It’s a good intermediate step to make sure you’re not transmitting plaintext passwords anywhere.

What’s the Drawback?

There’s a very small performance hit, and it’s not as easy to set up, but the biggest drawback is that you have to rebuild your apache users file.

So How Do I Do It?

Digest is very similar to standard Apache access configuration, with just a couple changes.

Make sure you have digest enabled - if you have shell access, type: ”a2enmod auth_digest”. Otherwise, in your httpd.conf or apache2.conf file, include the line:
```
 $ LoadModule auth_digest_module modules/mod_auth_digest.so
```
Create your new Digest authentication file. At the shell, type:
```
 $ cd /var/www/
 $ htdigest -c .digest Internal Admin
```
Note: This code creates a user named Admin in the realm “Internal” - we’ll get to realms later.

You’ll be asked for a password, and then asked to confirm the password.
Create your .htaccess file in the directory you want to protect - in this case, we’ll assume it’s http://(yourdomain)/Private:
```
 BrowserMatch "MSIE" AuthDigestEnableQueryStringHack=On
 AuthType Digest
 AuthName "Internal"
 AuthDigestDomain /Private
 AuthUserFile /var/www/.digest
 Require user valid-user
```
The first line is necessary to make Digest work for IE6. IE7 seems fine.

AuthName is the realm you chose earlier - Digest allows you to make multiple “realms” to protect different directories.

Also, note the AuthUserFile line - Apache docs swear that line should read AuthDigestFile. It shouldn’t.

That’s it - you’re using Digest!

The code is interchangeable anywhere you’d normally use standard Apache password auth, and you can use all the other normal .htaccess commands as well. It’s just a mildly more secure, and horrendously documented, alternative to standard HTTP authentication.

References

It's the Enforcement, Stupid

2008-12-25T00:00:00-08:00

Three months into the financial collapse, much of the debate is centering around the role of regulation in the crisis. Democrats are claiming the deregulatory atmosphere in Washington over the last 8 years caused much of the fraud and other criminal behavior that’s come to light. Republicans are arguing that the push for low-income home ownership and the expansion of Fannie Mae and Freddie Mac caused the subprime housing bubble. In the balance lies the layout of whatever eventual fiscal stimulus should pass: Will it be a market-centric solution, focusing on tax breaks and consumer spending, or an infrastructure-centric solution, focused on government spending and programs?

Ultimately, though, regulation was never the cause of the crisis. The question of “too much” or “too little” is moot - Nobody was enforcing the regulations left on the books. The three biggest culprits in the fiasco are the SEC, the Federal Reserve, and the Ratings Agencies.

Certainly, there are others guilty of malfeasance. Subprime lenders lent to anyone dumb or greedy enough to sign their name to a contract, creating loans specifically designed not to blow up soon enough to wind up back with the originators. Investment Banks bought, packaged, and sold these loans with no questions asked, and frequently, with questions specifically avoided. Larger banks issued Credit Default Swaps against these instruments without worrying whether or not they had the capital to cover. Pension plans, hedge funds, and insurance pools invested in CDOs and instruments they didn’t understand, putting their investor’s money at stake. Investors took outsize returns without asking any questions. The money was too good for too long, and nobody was willing to look behind the curtain.

Egregious though this may seem, all of this is par for the course. Wall Street runs on greed - it always has. It’s the job of investors, banks, and funds to seek outsize returns, and the speculators taking out Option ARMs on McMansions were following the same instinct. So why was it different this time?

The worst possible case for Wall St. is unenforced regulation.

In a case of heavy regulation, innovation may be stifled to the detriment of the economy at large. Subprime, for all the villany the phrase now embodies, has helped millions of dedicated, honest individuals into houses who otherwise might not have been able to get a mortgage. Stated income loans are invaluable for individuals with highly variable loans. CDOs, CDSs, and the rest of the alphabet soup of financial innovations are not, by themselves, bad. When investors fully understand the structure of these arrangements and the risks involved and are able to charge for that risk accordingly, these instruments can help spread capital more efficiently. To paraphrase another social movement, there’s no good reason to stop two consenting adults from knowingly engaging in any relationship they deem fit.

For this reason, a fully unregulated market is not by definition a bad thing. In an unregulated market, investors are required to do their own research, assess their own risk, and price accordingly. No doubt, many an investor will lose big, and the overall volume of investment will be much lower. The safety net of regulation cuts a good deal of risk out of the markets, making wider participation more attractive to more investors. Still, savvy investors can make money in an unregulated market, or simply hold their cash, and it’s hard to shed too many tears for greedy suckers. Left solely to their own devices, investors have nobody but themselves to blame for losses.

The worst case, however, is when investors Think they have a safety net that they do not, and that’s exactly what happened with the subprime crisis. The three critical regulatory bodies failed in their duties, and the economy is showing the consequences.

The first failure was that of the ratings agencies. These companies were chartered and endorsed as the standard bearers of risk assessment. It was their job to give unbiased ratings to the various securities, including bonds and even CDOs. At some point (around 1996 for Moody’s), these companies stopped focusing on the investors and started focusing on culling relationships with the issuers of these securities. The ratings started gravitating upwards, and somewhere along the way, piles of subprime mortgages lent with nearly no oversight started earning investment-grade ratings. The AAA ratings were crucial to the explosion of subprime - without the clean bill of health, pension plans, insurance companies, and many other large investment pools would have been unable to buy these investments, and many other investors would have shied away from these instruments. The high ratings gave investors an undue sense of security, one purchased with decades of brand equity from the big three ratings agencies.

The Federal Reserve under Alan Greenspan committed itself to a conscious policy of lax regulation, espousing Greenspan’s own libertarian ideals. The list of mistakes is significant, but the problems started with historically low interest rates over the last five years. The flood of cash flowed straight into the housing market. Greenspan refused to regulate, or even investigate, the new CDO vehicles, nor would the fed investigate numerous reports of fraudulent lending among mortgage originators. Because the Fed stayed so far behind the curve, they were blindsided by the extent of the damage from the subprime collapse. When Bear Stearns collapsed, federal regulators were so shocked by the extent and scale of the shadow banking system they were forced to give a $20Bn company to JP Morgan for $2Bn to prevent the kind of carnage that Lehman Brothers eventually caused. Bernanke has since acted swiftly to stop a complete meltdown, and if he succeeds, I’ll forgive his earlier errors, but when a regulatory body is caught as far behind the curve as the Fed was, their negligence borders on malicious.

Finally, the SEC under Christopher Cox have been the finest group of keystone cops this century. Bernie Madoff’s $50Bn Christmas present shined one hell of a bright light on the laissez-faire attitude the agency chartered to watch after the markets has taken to its job. Madoff was egregious, but the SEC was complicit in the rating agencies’ slide into malaise as well. Numerous regulators and enforcers have commented on the SEC’s reluctance to pursue clear and obvious cases of fraud. But the biggest black mark against the agency is the 2004 decision to allow the Big 5 to lever up to 40:1, well more than double their previous limit and approaching hedge-fund territory. Finally, the SEC’s ban on short-selling was a painfully transparent bit of handout to the banks, investment firms, and whoever else managed to get themselves on the list (which included such noteable financial firms as GE and IBM).

In all three cases, there was either existing regulation, or the organization had the charter to create the regulation, to prevent the buildup that led to the financial collapse. In all three cases, the regulations were unenforced and the groups stayed willfully unaware of the changes in the markets they were charged to protect.

The depth of their complicity, however, was unknown to most investors. Market players were still acting as though they were in a regulated market, assuming the information and the ratings they were receiving were legitimate and the cops were still on duty. This led every participant to engage in behavior far riskier than warranted by the facts, and that nobody got bagged for violating the rules just led to more morally dubious behavior. It’s dangerous to have no regulation, but it’s even more dangerous when nobody knows the regulation isn’t there. Investors have been assuming a safety net that wasn’t in place, and now they’re seeing the level of risk they’ve actually been exposed to.

Unfortunately, the damage is going to take a long time to undo. We can’t just implement new regulation - we’ve had plenty of regulation all along. Even banks, already heavily regulated and forced into onerous disclosures, don’t have enough faith in the legitimacy of their peers’ disclosures to lend appropriately. We need to show the market that we’re committed to transparency, equity, and full disclosure, and that we’re willing to enforce the rules. Only then can investors and lenders re-enter the market with any confidence, and only then can we hope to fully restart our economy.