Jon Udell

Derik Stenerson on the past, present, and future of the iCalendar specification

JonUdell — Mon, 13 Oct 2008 13:30:00 GMT

Derik Stenerson first came to Microsoft on an internship as a Test Engineer on Microsoft Mail. After graduating, he joined Microsoft full time in the email group and worked in various roles on email and scheduling products, including Schedule+ and Exchange. His passion for calendaring and scheduling led to work on the iCalendar standard (IETF RFC 2445) and later on a hosted self-service scheduling solution for small businesses. For the past few years Stenerson has been exercising his other passion for user centered design while building features for Microsoft Dynamics CRM.

Derik Stenerson

Next month marks the tenth anniversary of RFC 2445. To celebrate the occasion, Derik joins Jon Udell on Interviews with Innovators to discuss the past, present, and future of the venerable iCalendar specification.

Scott Prevost explains Powerset's hybrid approach to semantic search

JonUdell — Fri, 26 Sep 2008 09:12:00 GMT

Scott Prevost is General Manager and Director of Product for Powerset, the company whose semantic search engine was recently acquired by Microsoft. In this interview he describes the history of Powerset's natural language engine, and explains how it works as part of a hybrid approach to indexing, retrieval, and ranking.

Scott will expand on these topics in his keynote address at Web 3.0 in October.

Scott Prevost

JU: The notion of search enhanced by natural language understanding has a long history. I was just reading Danny Sullivan's rant about how he's been hearing about this for years, but it's never amounted to anything.

Of course, people are all over the map on this topic, but nonetheless you guys are doing certain demonstrable things, and working on other things. So I'd like to find out more about how the technology -- which was acquired from Xerox, where it had been worked on for a long time -- actually works. What you mean by natural language understanding, how you're applying the technology, and where this is going.

SP: Well, there are a lot of questions tucked in there, but maybe we can start with what we licensed from PARC, what was formerly Xerox PARC. They had been working for 30 years in a linguistic framework called LFG -- lexical functional grammar -- and they built a very robust parser. It's probably parsed more sentences than any other parser in the world.

What it allows us to do is take apart every document that we index, sentence by sentence, uncover its linguistic structure, and then translate that into a semantic representation we can encode in our index.

JU: Can you confirm or deny something that Danny Sullivan reported, which is that it takes on the order of two months to index Wikipedia one time using this method?

SP: [laughs] That's a very, very old number. It all depends on the number of machines, but we do it now on the order of a couple of days.

JU: And it scales linearly?

SP: Yes. And in fact we're working really hard to bring those numbers down. We have a very small data center right now. We're looking at what it takes to stand up a 2 billion document index, and it's absolutely attainable.

I think Danny Sullivan realized, when he wrote another article on the day we launched, that we're doing something different. He called us an understanding engine. It's not the case that we're just applying linguistic technology at runtime, by parsing the query and then trying to use the same old kind of keyword index for retrieval. We're actually doing the heavy lifting at index time.

We're actually reading each sentence in the corpus, pulling out semantic representations, indexing those semantic representations, and then at query time we try to match the meaning of the query to the meaning of what's in the document. That allows us to both increase precision and improve recall.

JU: When you say semantic representation, what it means -- or anyway what's evident in the current version -- is subject/verb/object triples, basically. That seems to be how things are organized.

SP: That's one small part of what the engine does. It's the part we've exposed in the user interface in a very direct way. But actually those are only three of several dozen semantic roles that we uncover at index time, and all of those roles go into selecting documents, and snippets of documents, when we present the organic results.

JU: Really? So even though the patterns aren't exposed in the advanced query interface, they're still used?

SP: That's right, they're still used.

JU: What would be an example of one of those other patterns, and how it's applied?

SP: So, you ask a question like: "When did Hurricane Katrina strike?" The 'when' is a certain kind of semantic role that we've indexed, separately from the subject, verb, and object. There are a number of other roles like that: location, time, other types of relationships.

JU: I saw a private demo, about a year ago, in which one of the most striking examples was something like: "Companies acquired by IBM between 1996 and 2003". At that point, I think the light bulb goes on in people's heads about what this could really be.

That class of query isn't exposed yet, but it's an example of what's possible, right?

SP: Absolutely. That's exactly the direction we're moving in. Initially most of the work we've done has been on the index side. Now we're starting to catch up on the query side, which allows us to complete the loop and do queries like that.

JU: The other piece that's visible on the website, in addition to the Wikipedia stuff, is the Freebase material that you've recently integated. That's an interesting case because there you can pull semantics directly from Freebase. So this becomes more of a query-time interface to something which is already structured and understandable.

SP: Yeah, that's right. Freebase is kind of like Wikipedia, except it's all structured data. Unlike with our core technology, which turns unstructured data into structured data, with Freebase we just go directly to the structured data. But it uses the same linguistic technology on the front end to parse the query, which then gets mapped into a Freebase database call.

But by using linguistic technology to parse the query, we're able to match very flexible ways of saying things. We don't have to imagine every possible way someone might ask for a particular piece of information. The linguistic engine takes care of a lot of that for us.

JU: That's why I can type in something like "Barack Obama's book" and get back the answer Dreams From My Father directly from Freebase.

So, what was the intent of including Freebase along with Wikipedia. What are you trying to show there?

SP: That the linguistic technology can be used with both structured and unstructured data. Freebase just has a lot of really great information.

One of the things about a natural language front end is that it encourages people to ask questions and expect answers. With the Freebase database, it's pretty easy to provide direct answers right at the top of the search results page, which users find to be a nice experience.

Of course you have to be very high-precision, so we've tuned the Freebase stuff for precision rather than recall.

JU: Tell me about the natural language landscape: the variety of approaches that exist, the style that you're using, how that compares to others, how all this fits into the history of the technology.

SP: The technology goes back a long way, three decades or so.

JU: Longer, actually.

SP: Yeah, really since the beginnings of AI people have been trying to use computers to understand and generate language. There have been a number of different approaches: purely symbolic approaches, statistical approaches. We really use a hybrid.

The Xerox technology uses a particular grammatical formalism, and we do use symbolic approaches to our semantic rules. But we also then put these semantic features into our index, and use machine learning and statistical approaches to retrieve and rank results.

It really is a combination. We try not to be religious about these things, but just use best of breed, and choose the right tools for the jobs we're doing.

JU: One of the things that Peter Norvig at Google is always saying is that the real secret to their success is vast quantities of data, and that in the end you don't really need AI, you just need lots and lots of data and the ability to crunch through it.

I assume you would argue that the natural language techniques are also helpful, and that as the quantity of data in your possession grows, the power that it brings to the table will also grow.

SP: Yeah. One thing we try to do with the natural language technology is give a leg up to the statistical and machine learning approaches. If you look at a search engine that just uses keywords, the information you have about the page is pretty slim.

We're trying to capture more information about each page that we index, which enhances our ability to retrieve and rank. For example, it allows us to retrieve documents where there are no keyword matches, but there's a good meaning match.

JU: For example?

SP: So, consider a query like: "What politicians were killed by disease?" Powerset will retrieve documents that don't include the words 'disease' or 'politician' or 'kill', but that are about particular politicians who died from particular diseases.

JU: Is the process of mapping generic terms to specific instances a hybrid of human editorial effort and statistical techniques?

SP: Yeah. We use things like WordNet, which is a giant dictionary or thesaurus of the English language that shows how various word senses relate to each other. We use that with some editing on our own. We also use machine learning techniques to figure out some word relationships, and which are most helpful in retrieval and ranking. So it really is a combination.

JU: When did you start this work?

SP: The company was founded three years ago, and I joined two years ago. But of course the work at PARC goes back 30 years.

JU: You obviously have an academic background in this field.

SP: Yeah, I have a Ph.D. in computation linguistics, as do probably about twenty other people at Powerset.

JU: What's your take on how this engine will start to surface through the various Microsoft online properties?

SP: The two areas where we can make a big impact are, first of all, improving core relevance, which is an absolute must for every search engine. And then also the user experience. Some of the technology -- and you start to see it in the Wikipedia search engine that we put out -- some of it really allows us to do different things in the presentation of these results. Thing that can save the user time, by getting the answer right on the search results page.

Our goal is to continue to work on improving relevance, and we've shown that by using these semantic features we can drive large relevance improvements, but there's still a lot of work to be done there.

JU: In that case, the improvements would be under the covers, the person using Live Search wouldn't know that you were contributing to the relevance of the result.

SP: That's correct. Another way it can happen is by creating a different quality of snippet or caption, things that highlight the parts that match the query instead of just bolding the keywords. Actually highlight the answer right there on the search results page, so you don't have to click through to determine if it's the right page.

JU: There's a related area called entity extraction, and there's been a lot of action there. For example there's a company called ClearForest, recently acquired by Reuters, which has put a lot of work into entity extraction. What's the story on that front?

SP: A lot of companies are working on this, we have our own in-house effort for name recognition and entity recognition, and this is of course really helpful as a kind of light semantic layer. But for us, it becomes deeper because we can start to relate all kinds of entities to one another, based on where we've seen them, and also with the help of things like Freebase.

To follow up on how you'll see the impact in things like Live Search, beyond the improvement in relevance and in the quality of snippet, I think you'll see features like related searches, other ways of presenting information similar to the Factz that are shown in our Wikipedia product, I think you'll see a lot more work on the instant answers, with a database that extends beyond Freebase.

Without committing to particular deliverables, these are the kinds of things I think you can expect to see. And you'll also continue to see growth on powerset.com, where we can be a bit more daring in terms of ways of presenting search results.

JU: Well thanks for your time. This has been interesting, and I'll be fascinated to see how things unfold over the next few years. I've got a feeling you'll have access to a pile of resources to work with...

SP: Yeah, we're really excited about it. As a startup, it's hard to build a full-scale web search engine. Having the resources available, and the really smart people at Live Search, is just a tremendous boost to us.

Kristin Tolle on biomedical initiatives at Microsoft Research

JonUdell — Thu, 18 Sep 2008 15:37:00 GMT

Kristin Tolle is the Senior Research Program Manager for Biomedical Computing for External Research in Microsoft Research. Projects run the gamut, she says, from "bench to bedside". In this interview she discusses two major biomedical initiatives: Cell Phone as a Platform for Health Care, and Computational Challenges of Genome Wide Association Studies.

Kristin Tolle

JU: Give us a sense of the kinds of biomedical projects you're working on internally, as well as those you're working on with external partners. I spoke with George Hripcsak, one of the researchers awarded a grant under the Computational Challenges of Genome Wide Association Studies (GWAS) program, and I know there are others involved there and in other programs as well. I'm interested in what Microsoft brings to the table in terms of helping these folks out with their computational and data management challenges, and also what kinds of things Microsoft learns from these engagements.

KT: The different programs inside of External Research run the gamut from the devices and mobility space, for home health care and elder care, all the way to genome wide association studies. So, we fund projects all the way from bench to bedside.

Because we're a software company, we'll focus on the IT parts, and there's a reason for that. These are often the parts that don't get funded elsewhere, or only get funded sparsely. Our purpose for going into medical funding was to fill those gaps.

JU: And why do you think those gaps exist?

KT: I think it's a misperception, by a lot of the funding agencies, that either something doesn't fall into their area, or that it's not as important as the actual research being done.

The problem is -- and this is why we're funding this area -- you cannot do medical research without computing. You just can't.

JU: Of course not.

Areas that we've funded...well, the biggest RFP we ran this year was Cell Phone as a Platform for Healthcare, and that was 1.4 million dollars toward trying to reach rural and underserved communities with retro technologies like cellphones and televisions, because those are ubiquitous.

JU: Oh, absolutely. I've spoken to Joel Selanikio, who was recently awarded a MacArthur Grant to use handheld devices for field data collection in the third world. It's a huge opportunity, though as you say it's the sort of retro technology that doesn't make people's eyes light up in Silicon Valley, they just don't see the opportunity the same way.

KT: It's true that they don't. But interestingly we've got a lot of researchers in-house, whether we're talking about that situation or about genomics, who have a keen interest in working in these areas. So for example, we gave Fone+ devices to a couple of the people who were winners of that award. The Fone+, which was developed by Microsoft Research Asia, is a phone that sits in a cradle, it's got RGB out to a television set, and USB input ports for mouse, keyboard, etc. So basically it enables your phone to work like a PC.

Now the beauty of this is, if you hook that up to a microscopy device that can do instant visualization of blood cells, determine whether or not somebody has malaria, and display that on a television screen, you've now just set up a lab for doing microscopy anywhere in the world there's a TV and a cellphone.

Another example is something we did with Washington University in St. Louis. They're developing low-cost ultrasound probes. Same thing. They're USB out, and designed to work with laptops, but now with the Fone+ you can plug it into this little cradle and now you've got an ultrasound anywhere in the world where there's power, a TV set, and a cellphone. You can even control the ultrasound device from the phone itself, it's just an amazing technology.

So that's an example where Microsoft Research has developed a technology that facilitates providing health care to rural communities. Although it wasn't initially designed for that, it was initially designed for education. But I took it and sort of twisted it..

[laughs]

...and said, hey, that'd be really good for the cellphone as a platform for healthcare project. I got them to give me a bunch of phones and cradles, and started sending them out to the researchers who had won awards for the RFP I ran this year.

JU: What kinds of things have you heard back?

KT: We've only funded them six months ago, so we won't see results probably until sometime next year. But I've actually seen a demo with the Fone+ and the ultrasound unit already working, so that was impressive. Washington U. is ahead of the game, I'd say.

JU: Are the folks you funded to do these things expected to bring technical chops to the table, in order to extend these devices? Are you working with them to provide support?

KT: Yes, we expect them to bring something to the table. And the ones who win the awards have superior technology. We had 145 people submit to the cellphone as a platform for healthcare. We'd originally planned to fund a million dollars, so that's about 10 projects, but we had to extend it to 1.4 million because we wanted to get to 10% acceptance rate. But even that, for us, is generally fairly low. Usually our acceptance rates are much higher. But we were just bombarded by people trying to come up with solutions for this space.

It was disappointing to only be able to do 14 proposals because when I looked back through them, I'd say 85 were fundable and on the bubble. Isn't that terrible? You wish you could do more.

So, I know you've talked to George about the genome wide association study, but I'd like to head in that direction in terms of some other things we bring to the table.

When I went looking inside MSR for collaborators, what I learned was that there's a plethora of them. It's kind of surprising we hadn't been funding this area before, and it's no surprise to me now that it's become a strong pillar of funding for our organization. In fact we've trimmed a lot of other programs and will be focusing a lot on the healthcare space this time around.

When I went hunting for collaborators I had no trouble finding them, even though I was new to the team, and that was because people consider healthcare the killer application for what they're working on.

But we also had a rich group -- you know, we have a couple of MD/PhDs working here, Eric Horvitz and David Heckerman -- and David does a lot of work in the development of vaccines for HIV and malaria. But he's branching out now into this GWAS area. So he's been looking at Lou Gehrig's disease...

JU: We should define the term GWAS, for people who aren't familiar.

KT: Sure. Genome wide association studies look across the genome to find if there are particular genes implicated in disease. That's one side of it. Another side is looking across the genome to check for reactions to different pharmaceutical agents.

In simplest terms, these studies are what will deliver on being able to provide personalized medicine for all of us in the future.

JU: Exactly, because it's a scan of an individual's complete genome, looking for markers and correlations.

KT: Absolutely right, that's correct. And I believe it will really deliver on personalized medicine for the masses.

And the thing is, it's happening already. We've got 23andMe popping up, Navigenics, people are going to start using their genomic information to make informed decisions about the type of healthcare they receive. They'll be taking that to their doctors and assuming they'll be able to work with it.

So we need to push the IT component of this down so that doctors have access to the information and know how to utilize it. Right now, that's the clinical gap between the research that's taking place in this area and the doctors who are performing the services needed.

JU: So in this case you funded about a half dozen individuals to look into different aspects of this GWAS research...

KT: ... yeah, very different...

JU: Right. So what do you hope will result from it?

KT: This was a new area for us, for Microsoft Research. We'd been dabbling in genomics for a while, but here we wanted to cast a wide net, find out what was going on out there, and find out if there were potential collaborations we could take from there.

When you find people whose work you can help facilitate, you form strategic collaborations with them to take that research to the next level.

Of course we bring a lot of resources to bear on this space. For example, the Microsoft Computational Biology Tools that we've published out on CodePlex, open source.

The other thing we bring to bear is a deep knowledge of machine learning and knowledge representation. And a number of researchers who've been working in general fields, but are now turning their attention to genomics.

A couple of new examples: John Winn, and also Christopher Bishop who literally wrote the book on machine learning and pattern recognition.

JU: This is a pattern I'm seeing often in these external partnerships. In all areas of science, as you say, scientists are necessarily becoming computational in the work they do, it's just the nature of the beast. But they don't necessarily have deep domain expertise in either algorithms or data manipulation and analysis. There are lots of folks at MSR who are deep in those areas, and who can effectively partner with these folks to move things forward.

KT: And it's not just that we have these underlying analysis and infrastructure technologies, we also have the human-computer interaction technologies to make that stuff usable for clinicians, or even the public themselves. So we've got people doing interesting work in how do you make something more understandable? How do you do machine translation across sex, age, status, education?

It's the same type of machine learning problem that you have with regard to going across language. You have to translate between languages, but you even have to translate within a language between different cultures, different demographics.

JU: What does Microsoft learn as a result of these collaborations, and what is Microsoft able to do with that?

KT: Well, the overall goal for External Research is to facilitate time to discovery, and to do so in a way that extends the arm of Microsoft Research. What we learn are which directions to move in. You know, we have publishing and tenure track promotion in Microsoft Research just as in academia. So if we can make our researchers more effective in reaching their goals to publish papers in Science and Nature, that's a fabulous thing. We've facilitated them and extended their reach.

There's also corporate responsibility here as well -- Microsoft, as a company, investing in areas that are important for the future. It's also important for us to keep abreast of the times, and the things taking place now. And finally, we learn things that we may incorporate into our products through tech transfer.

JU: It may be early days to talk about tech transfer from your biomedical projects, but I'd imagine one obvious outcome will be related to the kinds of devices that will be part of the HealthVault program, as sensors start to exist in people's homes, monitoring their vital signs, and transmitting them to the cloud.

KT: Absolutely, there's no doubt about it. The more that we invest in applications, and in sensors that can feed HealthVault, the richer their offering becomes.

The other thing is that we feel we're helping the public become more knowledgeable about their own healthcare. I think that's a common goal we share with the Health Solutions Group, Peter Neupert's organization.

They have other goals as well. So for instance, they have Amalga on the clinical side, and also a project targeted at researchers, trying to take people through the literature search for drug discovery. We're working in conjunction with that. One of the projects we funded under GWAS was a system to predict possible adverse drug reactions based on genome wide association studies.

Then the Columbia project -- George Hripcsak, whom you spoke with -- he's creating tools for researchers to integrate clinical information into the genetic analysis. Well, George's project is being built on top of Amalga. So there's a lot of synergy with the Health Solutions Group. And that's not unplanned. When I was starting out I met with Peter Neupert, back when he had eight people in his organization, and I interviewed him to find out what areas we should be investing in for healthcare. I'd also visited various schools and talked with people in their biomedical programs to find out what they were investing in as well. Then I tried to identify areas that would be relevant both to Microsoft Research and the Health Solutions Group. So, it's not just serendipity.

JU: So your own background is in bioinformatics?

KT: Biomedical computing. I had to form a multidisciplinary PhD committee because my school didn't have a program for this, though they do now, at the University of Arizona. So I had to form a multidisciplinary committee to get a PhD focused on machine learning for healthcare, with computational linguistics thrown in. It was tough, but it was worth it.

JU: Natural language processing was part of your focus as a student?

KT: Absolutely. Although the systems I developed were much more broadly utilized by different organizations. In fact, homeland security has some of the code I developed, which they use to scan for terrorist activities. It was initially developed for the National Library of Medicine to scan through unstructured text and identify keywords for indexers, and also to create small indices so that you could search faster and more accurately for publications in PubMed and CancerLit and other digital libraries. But you could see there were other implications. In fact it's also been used by the Department of Justice to make correlations among police reports.

So it's a generic technology, but my piece of it was targeted toward healthcare, and that's where my background and interests have always been.

JU: Are there other areas you'd like to discuss?

KT: I think we've covered the two major ones. I see us really investing long term in the area of devices, sensors, body sensor networks, ubiquitous and pervasive computing. That'll be a fundamental theme going forward, because it's been one of the more successful areas that we've made investments in.

But I also see us keeping a strong eye on the "omics" -- proteomics, genomics, metabolomics, you name it.

A third important area, and I don't know if it will be short term or long term, which is to address the other thing we talked about, and I don't have an RFP in this, but machine translation for people to be able to understand health care documents.

The average person cannot go out on Medline and read the literature on their disorder.

JU: It is amazing, though, how much context people can assemble for themselves under pressure of intense need.

KT: No doubt about it. But it would be better if we could create facilitating interfaces that would enable people to more readily understand and interpret that information. There's a lot of it out there, it's information overload really, and if we could make it a little easier for them, that would be very valuable.

JU: I wonder how much of that will be done by machine translation, and how much by crowdsourcing various experts at various levels. I think probably both will happen.

KT: Yeah.

You know, another important area -- and I was a bit disappointed when we ran our GWAS RFP that we didn't get anything concrete in this area -- was data visualization for genome wide association studies.

I think that's because it's such a hard problem. These studies are computationally challenging as it is, there's a lot of data that gets generated. Then to visualize it, now you're adding another level of computational complexity such that it's already not realtime just looking at the data, then how do you take it to that next level of visualization? That's going to be an important emerging area going forward.

So for instance, we've been talking with the folks at Oxford about getting a Surface there for collaborative visualization of cancer pathology.

JU: Not just in this area, but in general, we are so underserved by our ability to make sense of large complex data.

KT: Yeah, and we have these cool technologies. I think the WorldWide Telescope could be redeployed in many environments, and I think healthcare is one of those killer applications. We were talking with the National Cancer Institute, and one of the things they'd like to do is take a slice out of the liver while the patient is still on the table and be able to zoom in and zoom out -- it's the same technology.

JU: It's a similar kind of thing. To the extent that we can, in different fields, standardize on data formats and define multidimensional data spaces, we can indeed have browsers and viewers for those spaces. What the Telescope does in its domain is create a browser for a web of astronomy data. So yes, we need to have browsers for webs of genome data, and all kinds of scientific data.

KT: We had a recent paper by Bongshin Lee, she's done a distance encoding tree -- she calls it Detective -- and it's a scalable visualization tool for mapping multiple traits onto evolutionary trees. So we're trying to tackle it inside Microsoft Research, but I was hoping to see more people outside MSR show interest so we could start forming interesting collaborations in that area.

JU: Well, this has been a lot of fun. I hope to follow up on some of those Fone+ applications, that sounds really inspiring.

KT: Yeah, that's the reason I've gone into this area. It is inspiring. There's not only corporate responsibility, there's personal responsibility for me as well, and that's why I like working in this particular space. It's genuinely gratifying to be able to make a difference in an area that, no question about it, is beneficial to society.

JU: Well you've landed in the perfect spot to do that, and it sounds like you're having a blast.

KT: Yes, I am. Well, thanks very much.

JU: Thanks Kristin.

Roger Barga on Trident, a workbench for scientific workflow

JonUdell — Thu, 28 Aug 2008 17:41:00 GMT

Roger Barga, a principal architect with Microsoft's Technical Computing Initiative, is leading the development of Trident, a "workflow workbench" for science. In its first incarnation, the tool will enable oceanographers to automate the management and analysis of vast quantities of data produced by the Neptune sensor array. But as Roger explains in this interview, it's not just about oceanography. Every science is becoming data-intensive. Trident's graphical workflow authoring, reusable data transforms, and support for provenance -- the ability to reliably track and reproduce all the analytic steps leading to a scientific result -- is being used by astronomers too, and is expected to find its way into many other disciplines as well.

Roger Barga

JU: We're here to talk about the Trident, the scientific workflow workbench for oceanography. Give us the 50,000-foot overview, then we'll zoom in.

RB: Scientists are increasingly dealing with large volumes of data coming from disparate sources. The process used to be manageable. You'd get post-docs to convert the raw data from the instruments into readable formats, there was a manual workflow to process the data into useful data products.

JU: Those were the good old days. Or maybe not so good.

RB: Right. Because the time to get from raw data to those useful products was often measured in weeks or months. But now our ability to capture data has outpaced our ability to process and visualize it. And its rising exponentially with the rapid deployment of cheap sensors.

The oceanographic project we're working on, Neptune, is just one example of this. Astronomy, and all other sciences, are experiencing the same trend.

JU: Neptune is a University of Washington oceanographic project ...

RB: ... it's actually an NSF project. The proper name is Ocean Observatories Initiative, and it's being funded for several hundred million dollars. The University of Washington is one of the partners. Monterey Bay Aquarium Research Institute and a number of coastal observatories as well are involved.

JU: So fiberoptic cables are being laid, and lots of oceanographic data will be pouring in.

RB: Exactly. It's transformed oceanography from a data-poor discipline to a data-rich one. They're going to be able to monitor the oceans 24x7 over long periods of time. So the kinds of processes they can study were never within reach before. They could collect data when there was an episodic event, or when they could get funding. Now they'll be collecting permanently.

JU: What's the scope of the sensor network?

RB: They're laying the trench in Monterey to test and deploy the sensors. NSF is reviewing the larger program, and getting ready to fund the Neptune array which will be off the coast of Washington and Oregon. The Canadian version of the Neptune array is up and running and collecting data, but the software infrastructure is still being built as we speak.

JU: What quantities of data is the Canadian array producing?

RB: Gigabytes per day. It can easily handle a couple of high-def video streams coming from the ocean floor.

JU: Really?

RB: Yes. And also in-situ devices that can sequence organisms. It really is like not only taking Internet and power out to the ocean, but also a USB bus that instruments can be plugged into.

JU: What are some of the experiments that become possible with this setup?

RB: For example, being able to understand sediment flows across the ocean floor, how temperature and salinity change, how fresh water flows in from rivers, what kind of life exists at those margins. And understanding that interesting narrow band where life thrives in the ocean. Too high up and the tides affect it, too low and there's not enough light. But really, there are a myriad of things like that.

JU: So an experiment, in this data-intensive new world, involves formulating a hypothesis, looking for patterns in previously-collected data, and then seeing whether data collected in the future supports the hypothesis.

That means you not only need to run an analysis on data, but that you have to be able to repeat that analysis on an evolving body of data. Hence the need for the workflow automation that you're providing in the workbench.

RB: Yes. Another aspect is the need to calibrate and tune the models. If they can do that based on long-term monitoring, it'll remove a lot of the uncertainty in our understanding of the oceans. Versus now, where the data are so sparse that it's hard to validate the model.

JU: I guess also that as your understanding of the data and the models evolves, you might want to rethink what data you're capturing and how you're interpreting it. So, what is it that you've built with Trident, and how does it help you do those things?

RB: Jim Gray was the first person who had the vision of an oceanographer's workbench. His insight was that scientists really want to interact with visualizations of the ocean, but there was a huge gap between the raw data and those visualizations.

Managing information and managing data is one of Microsoft's core strengths. In External Research, we look for partnership opportunities where can bring our technology, learn from applying it to data-intensive stress tests that involve even more data than our commercial products currently handle, and figure out how to use or extend our technology to provide a solution.

Jim pointed out that workflow was one of the key missing ingredients. We looked at the in-house tools, and Windows Workflow was the engine of choice...

JU: ...although it didn't exist at the time Jim floated this idea, right?

RB: Well, yes, it was around in alpha and beta form internally. Jim knew I was doing some of my research using Windows Workflow. Of course he left the solution up to us, but he accurately identified workflow as being a way that the scientist could not only manage the data transformations that were needed, but also create a library of solutions that could be shared and reused.

If you look at how Microsoft works as a company, we build platforms and then we expect ISVs to come in and bridge the gap between the platforms and the user communities. That's the role our group has played. We're looking at the requirements of the scientists, we're looking at the platform Microsoft provides, and we're building on that platform to provide a custom solution to the scientists that will not only accelerate their work, but change how they do science -- enable them to ask and answer questions they couldn't before.

We partnered initially with the University of Washington and Monterey Bay Aquarium Research Institute, or MBARI. They're already gathering data from sensors, so they could describe the spectrum of data we'd have to ingest into our workflows. The University of Washington has a visualization tool called COVE, which scientists are adopting as the preferred way to look at the ocean floor. You can think of it as Virtual Earth for the ocean. If there's bathymetry data, you can pull it in and se the ocean floor.

JU: What kinds of data transformations are needed to get from the sensor outputs to COVE's inputs?

RB: There are probably about two dozen kinds of data sources we need to be able to ingest, based on the instruments and the types of data they put out. Typically it's streaming data in NetCDF format, or some other common format. So the first step is to recognize what kind of data format an instrument or model is kicking out, and transform it into an internal structure that our tool can use.

JU: But the workflow engine is abstracted from the instrumentation data formats and from the visualization tools, right? It's a mechanism for reproducibly running transformations, and managing that pipeline.

RB: Right. But let's start with how we interacted with the scientists. Jim Gray would ask scientists: "What are the top 20 questions you want to ask, and queries you want to run?" From that, he'd get an understanding of how they viewed the data, and what kind of processing was required.

We took the same approach, and asked the scientists which top 20 workflows they perform and which top 20 visualizations they like to see. Then we went through them from top to bottom, talking about the transforms and data integration that were required. We wound up with a set of two dozen transformations that were common across all of these workflows. That became the library of activities -- reusable chunks of code -- that the scientists could call upon to author not only these 20 workflows, but the next 20.

JU: Can you give a couple of examples?

RB: Sure. Regridding. You have two data sets, one's from a model and the other's from a set of deployed sensors out in the ocean. They're on different grid coordinate systems and you need to be able to bring those two together. That may require some interpolation, you might need to drop or add data points, transform coordinates, join data sets.

JU: There might be a temporal variant of the spatial gridding as well, to align different time scales?

RB: Right. Some instruments are getting things every second, some are getting them every 15 minutes. You can ask the user: "Do you want interpolation to take place? Do you want the system to match up the points?" Based on these inputs, the correct workflow gets configured and they see the resulting visualization for the region of ocean they're interested in.

JU: It sounds like some of these primitives will wind up being fairly general, not just specific to oceanography.

RB: Indeed they are. We're producing a version of Trident for oceanography, but many of these activities could be useful for other sciences as well. People in earth sciences, for example, are also using NetCDF and many of the same operations.

We expect that by building a tool which is extensible, and agnostic in terms of the science it supports, you can imagine it being used, for example, to understand the interaction between oceans and warm air currents.

JU: What does the Trident user see and do?

RB: We realized that the authoring experience for scientific workflow is very different from, say, business workflow. In business, you'd have your accountant write your expense report workflow. They'd lock it down, they'd deploy it, everybody would use it from then on, and nobody would touch it until it came back for bug fixes or enhancements.

What we found with scientists is that they want to borrow somebody's workflow that does what they want, or close to it, load that workflow, and then start authoring from that point on.

So we implemented that in Trident. You can search for workflows by purpose, or by the inputs they process. You click on one, and load it into a visual browser because while the oceanographers understand the workflows, they don't want to see C# or Java, they want to see something visual -- boxes that represent the transformations they want to apply.

JU: We've mentioned the Windows Workflow Foundation. For folks who aren't familiar with that system, how would you characterize it? How is it like and unlike a script execution engine?

RB: What's unique about workflow, versus scripting, is that with workflow you tease apart the notion of a schedule, which is the sequence of actions you'd like to have performed. If you were to look inside of each of those steps, you'd see code similar to what you'd find in a script. But on top of the sequence of steps you have an orchestration engine. When you pass this workflow -- this sequence of steps -- over to the orchestration engine, it runs the code inside each of the boxes, but as each one completes, control passes to the orchestration engine.

So we have an abstraction layer, we've opened up the opportunity for reuse, the steps or activities become building blocks. In addition, the orchestration engine can monitor the execution of the workflow, or change the way it executes -- for example, by running blocks in parallel on a multicore machine.

JU: What struck me about the Workflow Foundation was the way in which workflows can be very big or very small. As small as the sequence of interactions with a form on a web page, in which case the orchestration engine can be embedded entirely in the code that's behind that web page.

Or it can be a very big thing. But in any case, since it's part of the .NET Framework, it can exist in a variety of places. It can run locally on a laptop, it can run on a server in the cloud. There's an interesting amount of flexibility in terms of how workflows can be deployed. An application could embed Trident, or Trident could be used as a service.

RB: That's right. That's the magic of it. Yes, it could be hosted in an environment that the scientist is already familiar with. Or for a big institution, you could post it up as a service. Anybody could access it from a browser. And that's part of our mantra here. If we provide this to the scientists, we have to make sure it works with the tools they're comfortable using. You should be able to point your Linux box running Firefox at this tool.

But to your other point, we're experimenting here with workflows that are resource-seeking. You could launch one, perhaps even on your cellphone, and that scheduling engine's going to look for systems that have resources for that workflow, tap into them, and give the user on the cellphone the impression it's running locally.

JU: You've mentioned that the workflow style encourages a level of modularity that you might not otherwise get. It also provides a level of monitoring, control, and auditing. The reason that's important goes back to the idea of reproducibility.

A friend of mine is an HPC expert, and one of his pet peeves is that when people look at HPC they tend to focus on how much raw horsepower can be thrown at a problem. His question is: "Who's worrying about reproducibility and correctness?" It's a really important question.

In your environment, as I understand it, one of the things that you get is the ability to capture and replay and analyze what happened in a workflow, and the ability to faithfully reproduce a sequence of steps. You talked about enabling things that scientists couldn't do before. It's not only that they couldn't analyze large quantities of data, but also that they couldn't automate their own methods, and be able to reflect on them in an automated way.

RB: Right. Even if we couldn't run a workflow faster, and even if we weren't processing a lot more data, one of our key features is support for provenance.

JU: Explain what you mean by provenance.

RB: Think about it in terms of art. For a given piece of art, we're able to establish through authorities that it's original, where it came from, and who's had their hands on it through its lifetime. Provenance for a workflow result is the same thing. Minimally we want to be able to establish trust in a result. If you think about how that happens, it often starts by considering who wrote the workflow. So with Trident you can click on a result and interrogate the history of the workflow: who wrote it, who reviewed it, who revised it, when it first entered the system.

We do versioning as well, so you can look at an old result and know that it was created by an old version of the workflow. And then have the ability to run the new version on the old dataset to see if it makes a difference.

We capture execution provenance so you know exactly how your result was created. We capture provenance on the workflows themselves so you know who created them, and who's touched them.

You might be thinking about creating a community, where you click on a workflow and can say: "OK, I trust that post-doc."

JU: I've been reflecting on what Microsoft brings to the world of science, in yours and in other collaborations that I've been talking to MSR folks about. One is clearly the special competence and expertise in data management and processing. Even for computationally-oriented scientists, that data expertise isn't necessarily a core competence.

Another is the software tradition of version control. Again, that hasn't been a traditional strength of scientists. So this looks like a fruitful partnership on both fronts.

RB: Agreed. It would be nice to get Catharine van Ingen, or perhaps Alex Szalay to chime in how how this is being used for astronomy. Because we're giving drops of this code to our e-science researchers for use in other areas.

JU: I'd love talk with Alex. I had a couple of in-depth conversations about the WorldWide Telescope, one with Curtis Wong and the other with Jonathan Fay, and we touched on the work Alex has done. He's using your stuff as well?

RB: Not him personally, but his project -- Pan-STARRS -- is. Catharine van Ingen and Yogesh Simmhan are co-architects of that system along with Alex. And they're bringing workflow to the table. It's becoming the way scientists upload their data into Pan-STARRS and get it back out, and Trident is the workflow engine for that.

You've probably also heard about other activities here in External Research. Perhaps the scholarly communiations aspect?

JU: Yep. I've talked to Pablo Fernicola about the Word add-in for authoring scientific papers in the National Library of Medicine XML format. And recently I got the overview of External Research from Tony Hey.

RB: When you think about Trident in the context of scholarly communication -- and to your point about the importance of provenance, we see eye to eye on that -- not only can we use these tools for e-science data management, but we're focusing on reproducible research. When Trident has finished running a workflow, we'll create an XML structure that describes how to call back into Trident to recreate the result. We're really keen on the idea that not only is it easier to do the science, and publish the science, but actually reproduce it. And that XML description should be able to be embedded in the published work.

That's really exciting. It's been talked about in the computational sciences, but never addressed end to end with a tool that's instrumented, that produces an XML standard the community can own which describes how the science was done, and that gets carried along with the publication, either physically or by reference, and we store this execution script in a database somewhere.

JU: It's a really big idea.

RB: It is, I think it could be transformational.

JU: I do too.

RB: Right now, reproducibility means that that you happen to know the person who did the experiment, or you happen to capture enough stuff in your lab notebook or on your whiteboard, then you have a chance of being able to do it again. But imagine being able to click any result, and automatically and transparently reproduce that result.

JU: In reality it won't necessarily be the case that you can punch a button and have everything replayed exactly. But having the documentation, at that level of detail, and in that form, would be an incredible asset.

RB: Agreed. The hope is that here in External Research, because we're building these tools not just in the context of one science project, but many, you can have community tools that bridge communities. We're talking to people in the earth sciences doing atmospheric studies, and their workflows and analyses are so similar to what the oceanographers are doing. But right now, since those two communities aren't talking or sharing tools, it's very difficult for one community to interact with the other.

JU: That's a really nice point. Well, thanks Roger!

RB: See you later.

Lewis Shepherd discusses the Institute for Advanced Technology in Governments

JonUdell — Thu, 14 Aug 2008 17:01:00 GMT

Before joining Microsoft's Institute for Advanced Technology in Governments, Lewis Shepherd spent four years at the Defense Intelligence Agency where he helped usher in a new era of collaboration.

In this interview, he discusses how the Institute's small team of seven is exploring the nooks and crannies of Microsoft's research efforts and technology portfolios, looking for ways to help governments meet the diverse set of enterprise challenges they face.

Lewis Shepherd

JU: Microsoft's Institute for Advanced Technology in Government is a mysterious new organization that hasn't been heard from much. Readers of magazines like Government Computer News may have seen some notices about it, and may have noted that former CIA Assistant Director Jim Simon is the founder, and that it's attracted some other folks who formerly worked in government roles -- Aris Pappas from CIA, you from the Defense Intelligence Agency.

But not much else is known. So, what's this all about?

LS: Well, I'd say a better word than mysterious would be quiet. And that's because we're new and small. The Institute was set up by Bill Gates and Craig Mundie in 2004. They decided that Microsoft should play a more strategic role in the eyes of government.

Actually, in our title, there's a final letter, S. It's the Institute for Advanced Technology in Governments, plural. We're not strictly focusing on the U.S. federal government, which the backgrounds of the people involved would imply. It's actually governments at all levels.

We've worked a bit with state and local governments recently. In the past year we've increased our headcount to seven, and the seventh was an interesting addition. Bob Hayes is a British citizen, he lives and works in Cambridge UK, and he has experience at all levels of UK government. He began as a beat cop -- a bobby -- and has worked in and around the national security community in the UK for his entire career.

JU: You folks have close ties to Microsoft Research, but don't consider yourselves to be formally a research unit. Or do you?

LS: Not formally, but we do work closely with MSR, along with product groups. Jim Simon reports directly to Craig Mundie, so we have visibility into the entirety of strategic and future-oriented work that Microsoft is doing. Not just strictly MSR, but also incubation, Live Labs, Office Labs, forward-thinking people in various product groups.

A lot of it is personal. We're just seven people, I joined just seven months ago. It's been a wonderful way to see inside this tiny little 90,000-employee company.

JU: Governments are specialized kinds of large enterprises, so there are all sorts of potential applications for Microsoft's enterprise-oriented technologies.

LS: That's a big part of it. The core mission is to assist our federal government, state governments, and eventually we hope local governments and NGOs, to focus on their enterprise-wide problems. Of which there are many. As bureaucratic organizations they're a lot like commercial organizations, but they have particular unique challenges that you probably don't really understand unless you've had the pleasure and frustration of working inside a large federal government organization. If you have, as we have, you really understand the pain, particularly within our national security community. The intelligence community, the Department of Defense, these are massive bureaucratic constellations of organizations.

Five out of the seven in our group have some background in that national security community. I didn't have a career in it. But coming from a different kind of public sector background, and then a Silicon Valley background, I spent four years at the Defense Intelligence Agency where, post-9/11, I tried to bring some new thinking to the intelligence community. Along with a lot of other people, we were able to do some of that.

Along the way, as I looked out at the different strategic partners that government has in the technology world, we certainly viewed Microsoft as important, just because we -- like most others -- were on a Windows and Office platform, and were using a lot of other Microsoft products, but to be honest, it was limited to that. We thought Microsoft was a product vendor.

One thing that began to change my mind was, as a government executive, I used to visit Microsoft annually in Redmond. The account team that supported our agency began to hear from me that we'd noticed Microsoft spending six and seven billion dollars a year in R&D. I started to wonder: Where's that money going? And how much of it was focused on assisting with government problems? The answer to that was, at least consciously, in the minds of MSR, none of it.

Yet here I was in the intelligence community, working with the DoD fighting this long war on terror, surrounded by some of the keenest early adopters in the world who were looking to push the the limits of technology. So I began to talk to Microsoft and found they had indeed set up this quiet group in 2004 to consult with government, both inside the intelligence community and elsewhere, on these kinds of enterprise problems, and to bring to bear some of the more interesting and promising fruits of Microsoft Research.

JU: There are a bunch of Microsoft technology initiatives that intersect with the interests of governments as large IT-supported enterprises: identity management, data management, systems management, service-oriented architecture, application development. That's all playing out in governments as in other enterprises, but I suspect that's not what you mean by advanced technologies in governments?

LS: Correct. Although many of those are very much of interest, and frankly, federal and state governments aren't always aware of the leading edge in commercial software technology. So one of our roles is to make them aware of the leading edge, and of best practices. We do that in a way that doesn't come across like a sales pitch, because they don't need to hear that. And if it makes sense to advise government leaders to innovate in ways that don't necessarily require Microsoft products, that's a plus for the Institute and for Microsoft's role in assisting governments. So we've done that several times.

Our sales guys understandably focus on what they can sell today, and in the next quarter. But often government needs to know how to make better use of what it already has, or how to use something that isn't a Microsoft product.

But here's a case study that's more along the lines of what we mainly focus on: Microsoft Surface. It's gotten a lot of buzz as you know, and is now being commercially rolled out in the entertainment space.

When Surface was still a research project, Jim Simon -- who loves to poke into the nooks and crannies of MSR and incubation projects -- saw it, and talked with the team, and realized they were mainly focused on it as a gaming platform.

He thought about that for a while, and said there were two additional markets, and he knew people in each.

One is the big-G gaming world of casinos. In venues like Vegas and Atlantic City, the entertainment experience involves a holistic view of customers, from the moment they show up at the hotel, day and night on the casino floor, at the shows, at restaurants. The touch-enabled UI really supports that scenario.

The other, of course, is the national security world, particularly DoD. It so happened than when I came on board and first learned about Surface, I had previously, at the DIA, had experience with touch tables, a different kind of technology -- a touchscreen on a pool-table-sized device -- that we were one of the first customers for. It was sold to us by a large defense contractor, and it did a great job for us in 2004 and 2005. But each device was $250,000.

JU: And what were you able to do with it?

LS: Defense planners typically stand around a big sheet of paper, or a map, trying to collaboratively plan out a day's or month's or year's campaign. Doing it that way, or on a sandbox, is the traditional way, and there hadn't really been any innovation.

With the touch table device, you could show a map on this horizontal surface, and data layers. Think about Virtual Earth or Google Earth, the ability to do that on a 6-foot by 9-foot table becomes very appealing.

JU: Was this commercial software on a custom device, or was it all custom?

LS: It was all custom. $250,000 a pop. We bought two of them, and put them on two different floors, for two different teams to use. After a while we realized it'd be nice if the teams could collaborate, but they weren't networked, so we had to pay our contractor another $100,000 to connect them.

When I first saw Microsoft Surface, and realized the entire thing ran on essentially a state-of-the-art PC, and that the APIs were going to be open enough for developers to put any kind of Windows software onto it, and that it would all be networked...it really opened up the possibilities.

And then when you realized that, because of the scale Microsoft operates on, the price would be $10,000 instead of $250,000, it just blew my mind.

So now that same defense contractor has become one of the first to develop on the Surface platform. They know their customer, they know the scenarios for defense, intelligence, homeland security, state and local police. And I think the Institute played a small but important role in opening the eyes of a lot of people to the kind of difference a Microsoft platform could bring to that environment.

JU: What are some other kinds of connections like that that you're making, or want to make?

LS: Well, I've been extremely interested in robotics, and the emergine large-scale appeal of the Microsoft Robotics Studio. You and I have chatted about this. The appeal isn't so much robotics, per se, but rather the back-end architecture that takes advantage of advances in concurrency and high-performance computing and distributed services.

You mentioned service-oriented architecture, and SOA has been a buzzword in government and other kinds of enterprise circles for a while now. Well there really are multiple services being developed and deployed in lots of different environments. The ability to orchestrate enormous numbers of those services is something you can do natively with the Robotics Studio, whether or not you intend to develop a robot.

JU: It's a fascinating outgrowth of that project, and the implications are only beginning to sink in. The software infrastructure is extraordinarily general-purpose.

LS: Right. Among overnment early adopters, the people we've seen take a good deal of interest have been in DARPA, and also in a new organization in the intelligence community called IARPA. Once you get the right people looking at this, they understand what's really behind it, and the power of it.

JU: You've also been known as a proponent of Web 2.0 methods, and were responsible for bringing Intellipedia to life.

LS: I was one of the people who did. It was a great team.

JU: What are the opportunities in that realm?

LS: I've watched with great interest the rise of Enterprise 2.0. I think most people credit Andrew McAfee with coining that term. He and I have spoken on a number of panels, and we've talked about how government organizations can nurture bottom-up development of advanced capabilities using things like blogs and wikis, and take advantage of the emerging power of social media, without the kinds of constraints you inevitably get in a government bureaucracy.

Not only do you have all the usual bureaucratic problems of large organizations, but there's also a hypersensitivity to security, and also -- within the civil service -- the disincentive to innovation that happens when people are career civil servants.

So how do you nurture grassroots adoption of these technologies? It's very personal, you have to find the right people. I was lucky to have an inside role in the intelligence community.

When you say Intellipedia, people may or may not know about it, but it's been a phenomenal success story in the intelligence community as a community. That word, community, was openly mocked for decades because the sixteen different agencies -- and particularly the big ones everybody knows about, CIA, NSA, DIA, NRO, the alphabet soup of them all -- really didn't collaborate that well, if at all.

The 9/11 and WMD commissions went into great detail about this. What I and others were able to do was to begin working in small ways on identifiable chunks of value that we could create for community-wide use on shared networks.

This work began in 2004. I don't think there was any great flash of inspiration in deciding to basically plagiarize Wikipedia on a secure network, as a large-scale network for socially-authored and socially-maintained intelligence that had been kept in stovepiped databases.

The first pilot was in 2004, and it opened as an enterprise system for the whole intelligence community in 2006. Intellipedia has been a big success. There was an initial period of hockey-stick growth. That's leveled off some now, and the challenge -- as in any enterprise -- will be to continue to evangelize the business practices of social networking, and the value they bring within a large diverse set of organizations.

JU: Of course this wasn't an example of advanced technology. Wikis and blogs are just the kudzu of the Internet. It was more an exercise in social engineering than a deployment of any new or advanced technology, and appropriately so.

From a Microsoft perspective, then, is it about applying advanced technologies from MSR in this environment? Is it about bringing some of that grassroots sensibility into the Microsoft platform?

LS: I think it's both. In the federal space, particularly intelligence where you have knowledge workers on steroids, we have an interesting mindset within the account teams. They're not only focusing on what can be done with SharePoint 2007, with Office 2007's XML capabilities. They're also seeking out bits of code being worked on in Live Labs, in Office Labs, and elsewhere. Popfly, for example, It's being heartily evangelized by Microsoft teams within the federal government, and it's gaining enormous receptivity.

JU: I've talked about this with John Montgomery, the Popfly lead.

LS: In fact there's so much interest, he's almost to the point of being overwhelmed.

What I see is a changing mindset about Microsoft, and the role it can play in government. It's not just about are we on a Windows platform. It's about what can I use, on my computer or mobile device, that'll enable me to do things I couldn't do before. If those are Microsoft things with a Windows label, that's great. If they're not, if they're cool, funky, web-centric things like Popfly, that's great too.

JU: Tell me if this fits into your charter. A big aspect of what I think of as Government 2.0 is the emerging availability of various sources of government data. There's a growing consensus that data will be made available, and that's happening, but in a way that reminds me of how things were, and mostly still are, on the scientific web. Yeah, there's the data, go grab the gzipped tarball and have fun with it.

As opposed to offering a service layer interposed between both applications and human being.

I see an interesting possible role for Microsoft, and I see it as extension of something that's happening in the relationship between MSR and the scientific community. I've recently been talking to a lot of people in Tony Hey's area. These folks are what I'd call informaticians, and they're working closely with scientists in various fields.

In every branch of science, now, the work revolves around the collection and analysis of previously unimaginable quantities of data. One of the things I'm seeing Microsoft consistently doing in its partnerships with scientists is to provide both infrastructure and consulting expertise, to help people wrap their arms around large datasets and make them useful in ways they wouldn't otherwise be.

I'm wondering if there isn't scope for something analogous in the government space, as these datasets begin to be made available, but not necessarily in ways that enable citizens to ask and answer meaningful questions, or relate the raw information to policy.

LS: You've hit on something that's really important, and yes, it's an interest of ours. It's very hard to do, but if you do it, the value is tremendous.

I'll give some examples of things that we're thinking about, and one that we're working on.

One thing we're thinking about, as a model, comes from one member our group I want to mention, George Spix, because he's such a great guy, a lot of people around Microsoft know George. He's the only guy in our group who's been with the company for a long time, before that he worked with Seymour Cray.

JU: Here's something about George you may not know. The Microsoft Conference Center recently hosted the annual space elevator conference, and George was the guy who gave the go-ahead for that.

LS: That's closely related to the example I was going to mention, which is the WorldWide Telescope. George also did some work on that, and if you think about it, it exemplifies what you were talking about.

JU: Absolutely. The WorldWide Telescope is the paradigmatic example of a service layer that's been interposed between a previously available but practically inacessible dataset and a set of interoperable applications, on the one hand, and ordinary people, on the other. You're right. It's the perfect prototype.

LS: It is. And as such, it also serves as the perfect educational device for people who are in a position of authority over other large stovepiped datasets. So we've been using WorldWide Telescope as a teaching element: "Here's the future of what your world could be." It's not only an extremely appealing app -- people fall in love with it, Robert Scoble was famously moved to tears by it -- but there's also, as you said, the paradigmatic simplicity, and obviousness, and utility, of opening up data. So we've been using that in a number of ways to stretch the mental boundaries that government officials have about their data, about the accessibility they currently offer, about what new technologies and web-scale computing could bring to their data, and about what that would do for them, and their intent to serve their customers, their users, their citizens.

It really is a mind-blowing way to get them thinking creatively about what could be done.

Another example: machine translation, and some of the hybrid translation approaches that Microsoft Research is pushing the boundaries on. Here we have real examples, already offered within the Windows Live constellation, that people don't really know about.

There's the translator bot that is a Live Messenger client, you can have simultaneous translation among a dozen languages in your instant messaging.

JU: I hadn't seen that myself.

LS: Well, there you go. And there's the live translator plugin for Internet Explorer, I just blogged about that last week. It enables you to surf foreign language websites, with simultaneous translation. It's really changed things for me. I have a lot of interest in Russia, so being able to surf Russian-language sites, with good-enough machine translation appearing right in the browser, it's phenomenal.

And as we show that as a service to be exploited within service architectures, that's something governments find really intriguing. It helps them think about how they could provide better access for diverse populations.

JU: I'm glad to hear that. I've been pushing for a while on this theme, and have recently concluded that we're kind of stuck on the question of access to the data. But that's only the first step. It's great that we're getting to the point where that first step will be taken, but there's so much more that can be shown and done.

LS: I'll give you another example. I've had that same lingering feeling of frustration about the web, and the billions of pages and documents I can theoretically access. But there's no sense-making.

One of the most exciting things I saw last year, even before joining Microsoft, was Photosynth and Seadragon. We're working on making these and related technologies, like Deep Zoom, available to government organizations that have access to very large archives of images which are just sitting there. Yes, theoretically you have access to them, but something like Photosynth enables you to make sense out of them.

JU: What kind of images are we talking about?

LS: Well, some I can talk about and some I can't. But we have been talking to some state governments about their access to the world of Flickr and online collections like that, from the standpoint of homeland security, and the ability of first responders to make sense of the visual environment of today's world, in an up-to-the-minute way, just based on the open source information that's available.

This is something Microsoft has helped a lot of public sector groups, like the Los Angeles fire and police departments, to be real leaders on.

JU: So a lot of documentation of planet Earth is being done in a grassroots, ad-hoc way, for example in the form of photos on Flickr that are tagged and even geolocated. And there might be a government interest in those collections as the most up-to-date record of what exists.

LS: That's right, and it eventually works in a circular way on the provision of government services back to people. If you think about a government bureaucrat, like a building inspector, who goes to a site, takes a photo or two, and certifies that the site is being worked on in a way that conforms to local or county regulations.

Well, the ability to do all that in realtime, with a camera-equipped cellphone, and do it in a secure way, with timestamping and geocoding...

Or think about that capability deployed in child welfare scenarios where there certainly aren't enough government personnel to visit all the domiciles where trouble is reported.

When you think about large volumes of data being transmitted in both directions -- from citizens to governments, and from governments to citizens -- it really opens up the world. We haven't figured out all the ways, but it's fascinating to think about the diverse set of enterprise challenges that governments face, and about the technologies we have in the nooks and crannies of Microsoft that might be able to help.

JU: It sounds like you're having fun snooping around finding them.

LS: I'm having a blast!

Maurice Franklin reflects on the 2008 Space Elevator Conference

JonUdell — Fri, 08 Aug 2008 14:24:00 GMT

Maurice Franklin is a 12-year Microsoft veteran whose career has focused on performance engineering and server scalability. He's also passionate about the concept of a space elevator, and recently organized and hosted a conference held on that topic at the Microsoft Conference Center.

In this interview he discusses reasons to build a space elevator, and describes how the concept, first proposed by Arthur C. Clarke, is evolving toward a practical implementation.

The transcript for this interview appears below. Audio is available at ITConversations.

In a related interview Ted Semon, author of the Space Elevator Blog, reflects on the conference and on the goals and status of the effort.

Maurice Franklin

JU: Maurice, most folks don't know even that there's a serious plan to build a space elevator, and I'm sure even fewer know that those most closely involved gathered this past week for a conference hosted at Microsoft. How did that happen?

MF: Well, I'd qualify the word "plan" ...

JU: Maybe we should call it an intention.

MF: That's a good word. Or aspiration. So, I got involved in the space elevator world in 2002 when I discovered, quite by accident, that there was a conference in Seattle. It was hosted by an entrepeneur, Michael Lane, and a scientist, Bradley Edwards.

Dr. Edwards is the father of the 21st-century concept of the space elevator. He'd heard it was impossible, and didn't believe that, so he got a NASA grant, and came up with something that made everybody say: "Well, that's not how we were thinking about it at all. That's a decades plan instead of a centuries plan."

JU: What was different?

MF: I'd read a 1997 NASA study, it was big science and big engineering. It relied upon a spacefaring technology to get to a space elevator. Those of us who are fans of the space elevator see it as a bootstrapping mechanism to get to that spacefaring technology.

So, for example, one of the prerequisites assumed in the 1997 study was the ability to move an asteroid into earth orbit. And then to put a manned carbon-nanotube-manufacturing station in orbit. Well, you can't do any of that unless you have a space elevator.

As much as anybody has, Dr. Edwards cracked the chicken-and-egg problem. His proposal requires on the order of four or five heavy launches. After that, it self-bootstraps -- if the materials come along, and a lot of other ifs, but that's a game-changing proposal.

The NASA study also required a massive elevator, because there were going to be maglev trains running at high speeds, and that just adds more and more weight.

His plan is much more modest, it only goes 200 kilometers per hour, it's a bit slow, but it's practical, you could reasonably get stuff up pretty quickly.

And he added in remote power beaming, so you don't have to carry fuel for those days of climbing up towards geosynchronous orbit.

In all respects, it's much more practical. If we could get past the technical hurdles, it's something in our lifetime, or at least those of us who are older hope so.

JU: So your role at Microsoft was then, in 2002, and is now...

MF: I'm a performance engineer. No connection with space-related activites at all, just a personal interest. Although that 2002 conference was invitation-only, I showed up. I'd read the NASA study, I'd read Brad's study, I had intelligent questions, and I was accepted, which was very cool.

JU: So how did Microsoft come to be the sponsor and host of the 2008 conference?

MF: One of the collaborators on this project is Dr. Bryan Laubscher. He's an astrophysicist, most recently with Los Alamos National Laboratory. About a year ago he was invited to the visiting speaker program at Microsoft Research. I went to hear him, gave him my card, and invited him to contact me if there was any way I could help.

About a month later he got in touch and said, "I'd like to make Seattle and Microsoft the center of the space elevator universe, so let's get a conference."

Well, Microsoft employees themselves can't usually sign up the Microsoft conference center, so it became my job to find a sponsor. Through various happenstances I got connected with George Spix, now with the Microsoft Institute for Advanced Technology in Governments, who I actually I knew from some work we'd done together in the past, and he said "Sure."

JU: It's interesting how these things play out.

MF: Yes, it is.

JU: So on the face of it, this is a project that has a lot to do with power engineering, and...

MF: ...materials technology...

JU: ...right, and civil engineering. But there are computational spinoffs and synergies too.

MF: Right. One of the subjects that came up at the conference is that even though this ribbon, as we refer to it, will be under more stress and tension than anything ever built by man, it will also -- by virtue of being so darn long -- be very dynamic. It will flutter in the wind over miles rather than feet.

JU: So there will be a need to simulate the resonances.

MF: Yes. It's going to seem to be a simple object, but because of its length and its multiple environments -- gravitational, atmospheric -- it'll be very complex.

Also, it has to avoid what's already up there. If you put something on the equator and then span zero to 100,000 kilometers, you will intercept the orbit of everything, eventually. Repeat those words: Everything, eventually.

JU: Did you say 100,000 kilometers?

MF: Yes, it's very long.

JU: Wow.

MF: Yeah. It's a significant proportion of the way to the moon.

JU: That's way more than I realized.

MF: To touch the earth and stay in the same spot, it has to more or less orbit at geosynchronous. Once you establish that, you have to put enough mass above geosynchronous, call it a counterweight, that the earth is attempting to eject from orbit, to offset the mass, or in this case weight, below geosynchronous, that the earth is trying to pull down.

So you fix that geosynchronous point at 22,300 miles, I think it is. You then get to decide how big that counterweight is versus how far out it goes.

If you put an asteroid of appropriate side, it can be just the other side of geosynchronous.

JU: Ah. So that's what I'm remembering from the original proposals.

MF: Yes, in the NASA proposal, and also in Fountains of Paradise, the Arthur C. Clarke book, it was a large counterweight very close to geosynchronous.

JU: So that's "big engineering" as you've said, and the modern concept is to go smaller.

MF: Right.

JU: In terms of computational challenges, we have this carbon nanotube ribbon which is 100,000 kilometers long, and it's on a collision course with every piece of space junk.

MF: Right. So Dr. Edwards' proposal is make the base movable, by putting it out to sea on an oil-rig-like platform, and then to move it when the computers say it has to move. You induce movement of the base, and then up and up, to miss the space station.

So I think I just described a very complex thing. You've got to move this thing hours or days ahead of a collision that you have to avoide by, say, 5 kilometers to be safe.

JU: That's the safety buffer?

MF: Right. Part of the Air Force monitors every object up there.

JU: Even screwdrivers?

MF: I think they get down to the centimeter, but don't quote me, I'd have to look it up. Anyway they track all this, and call up the space shuttle and say, you need to fly a little bit that way.

But now you have this object that you can't just move on a whim. And when you move one part of it, you move all of it. The space shuttle has a buffer of a kilometer, or 10 kilometers, I don't know, but say it's a cubic kilometer they have to keep clear.

Well if the space elevator's square danger area is a kilometer, it's not a cube.

JU: It's a column.

MF: A column that's 100,000 cubic kilometers, projecting through every orbit known to man. And in order to move it, you have to move it all.

There was a NASA retiree of considerable note at the conference, Ivan Bekey, and he came in and said, "Have you guys really figured this out yet? I don't think you have."

It was a great keynote: "Potentially Fatal Elevator Flaws That Must Be Addressed".

JU: So the strategy you just discussed has been on the table.

MF: Yeah, but not enough to stand up to the scrutiny of somebody like Ivan Bekey, who's saying, no, no, you have to actually figure this out. And people are like, yeah, yeah, you're right, we have to figure it out.

Then there's the question of what kind of communications network runs on the space elevator. There will be radio waves, but you'll want to bring up your browser too, and the latencies will be very different than for a transatlantic cable, so there are interesting computational challenges there.

JU: Fascinating. So as an observer of this scene for some years, what was notable about this year's conference?

MF: Well, for one thing, to see people working through the ideas that Dr. Edwards came up with, for example the collision management problem. On the other hand, there was a presentation on how, having shot the arrow, you get the rest of the material across. He proposed a deployment. But an engineer showed up, somebody working purely on his own time, and he has software to simulate the dynamics, and he said, no, that's probably not going to work, but this might. His idea was to keep the middle and the ends at geosynchronous, then spool out two in-between parts, then let go of the ends.

JU: So the most basic deployment strategy is still very much being discussed and debated.

MF: Yeah. It's such a complex object. This isn't a satellite. It's a thing that sticks through a lot of gravity gradients, in its first 10 kilometers it's beat up by the atmosphere.

So people are digging in, and coming up with different ideas that thematically fit in and move things forward.

JU: People have a general sense of what it would mean to get a several order of magnitude reduction in the cost of moving stuff into space, and of what applications could flow from that, and of the benefits of those applications. What's your take?

MF: Cost is a big deal. We ship things across the country because it costs 15 dollars, but we don't ship things to orbit because it costs 15 million. The space elevator changes the market dynamics completely. Because it acts more like a railroad with high upfront investment and then low operating cost, versus rockets with continuously high operating costs, that drives potential market opportunity.

But as one gentleman pointed out very forcefully, in addition to cost, there's capacity. You can only build and launch rockets so fast. The space elevator not only gets you low price, it's always there, waiting to launch payloads every day. The baseline elevator has, say, a 10-ton capacity every day, versus 15 tons every 3 or 4 months with a fleet of four space shuttles.

He was pointing out that all these big post-Apollo dreams -- going to the moon and Mars, tugging asteroid into orbit to mine them...

JU: ... large-scale orbital solar power collectors and beamers...

MF: ... right, which is one of my favorite applications. So for these things it's not enough to have low cost, you have to lift a lot of stuff up there.

Solar power satellites sound wonderful. Of course there's a whole set of other engineering and environmental problems. But in order to even make a dent, you have to move a lot into orbit. That guy was right. Capacity, capacity, capacity.

JU: What other uses are people talking about?

MF: Space tourism. People go up the elevator for a few hours, see the rim of the earth, that might be a gangbuster business.

JU: How long does it take to get up?

MF: 200 kilometers/hour is kind of slow, I think geosynchronous works out to 6 days. You don't have to go to geosynchronous orbit, though, unless you want to step off and have nothing happen.

A critique that made it to the New York Times was: "If you go to low earth orbit and step off, you fall." True. Of course the comeback is, go past low earth orbit by a bit, then step off and you fall towards the atmosphere, but you don't actually hit it. You bring along a small rocket, fire it, and you're in low earth orbit around the earth.

So you don't have this big fiery explosive launch that gets you to orbit in 20 minutes, but you don't have the dangers that go along with such an enormous expenditure of energy in a short time.

Instead you go up in something that's slower than an Indy racecar but faster than most people drive. It might be several hours to an interesting spot, several days to a more interesting spot.

There are people going up now to look down on the earth at $20 million a pop. If the cost came down to $200K, $20K...

...and there's a sociological argument that it would be a great thing for people to observe the earth from space, because there are no maps.

JU: Yes. It's historically had a powerful effect on the privileged few who have been able to go up and see that.

So there's the solar satellite concept, there's space tourism, what else?

MF: Everything else in between. Things we could do in zero gravity. Today there are experiments with special drugs, special materials processing, but they're experiments. They have to fit in the bay of the space shuttle, it only goes up every 3 or 4 months, you're not going to build an industry that way. But if you could ship your goods up and down every day, to an orbiting manufacturing plant that you just carried up on one of the larger space elevators, that's not necessarily a dream, that could be a business plan.

Two related things I meant to mention. First, it's highly scalable.

JU: Meaning?

MF: If we can build a 20-ton elevator, we can build a 40-ton one. And by the way, the first thing you build with a space elevator is ... a space elevator. The very first payload will be the second space elevator.

JU: To be build in another location?

MF: Right.

JU: Which addresses the single point of failure vulnerability.

MF: Yes. It is vulnerable, for all sorts of unfortunate reasons. But while it costs a lot to get the first one up, the second and third are relatively a lot cheaper.

So they scale out. I'm a performance engineer, that's one of our favorite terms.

And they scale up. There's no reason you couldn't scale up to a million pound elevator. And the thing is, there's no particular limit on the size and shape of the thing you carry.

Today things have to fit in nosecones, basically. The shuttle is a long bay, but cylindrical and 15 feet wide. Here you're limited mostly by how much turbulence it can take during the first 10 kilometers of the climb.

After that it's: "Oh, you want to build the space station." Just launch the space space station, tomorrow, and it'll be there in six days. That's a totally different way of looking at it.

It's like container ships. They've totally changed global trade.

JU: I was going to mention that. There's a book called The Box about the innovation of the standard shipping container, which is the physical equivalent to packets in a packet-switching network.

As a result, although it seems silly that some of the things we buy have criss-crossed the world...

MF: ... interesting meta-discussion there...

JU: ... yeah, but because of that technology, it really is economical for a lot of stuff to move around.

MF: I've seen bottled water from New Zealand. It astounds me. But it probably cost more to make the plastic container, which I consider to be heavy and low-value, than to ship water across the ocean.

JU: So the question remains, is this a leap of faith, like with the early space program, where we don't know the benefits but we intuit that there will be all kinds of spinoffs.

This feels like that, so the more you can identify operations that benefit from arbitraging the different between earth and orbit...but it's not crystal clear to me there's a long list of those.

MF: No, I agree. A critic of the space tourism idea pointed out that, yes, airlines and before that steamships were all built on the desire of people to go somewhere interesting: Disneyland, or a business deal, or an exotic location. Currently space only qualifies on the last point. So the chicken/egg problem won't be cracked by space tourism.

There have to be business reasons. We mentioned solar power. Another possibility is mining asteroids for iron and nickel. Those two items -- a lot of energy, and all this material -- there's a whole chapter in Dr. Edward's book that starts by figuring out your individual share of that. It's a lot.

The way I look at it is that a quarter of the world lives at or near the US standard, there's another quarter trying to achieve that in the next generation, another quarter just starting, and then one behind the curve.

Energy and materials are resources that space is full of.

JU: Clean energy in particular.

MF: It's getting a lot of attention lately, yes.

JU: Of course a lot of folks will point out that there are lots of earth-based solutions for clean energy, so why go to space for that. But I guess the answer is that it's not necessarily either/or.

MF: Agreed. But consider this. Scientific American's January cover story is a proposal to do big solar by 2050. Part of it is to use 1/5 of the American desert. The environmentalists have to swallow that, there's an albedo change in the earth, and so on.

Plus, it turns out that having a desert close to your civilization is fairly unique the world. Europe doesn't have one.

Now factor in population growth.

Meanwhile, satellite advocates say that if you had a band of solar cells a kilometer wide -- and that's a lot of solar cells, no doubt about it -- they'd produce, annually, energy equal to all known remaining oil reserves.

JU: It brings to mind one of the Long Now talks, I think by Vernor Vinge, in which he plots human civilization with population on one axis, and the amount of energy available to be used by an individual in the population on the other axis...

MF: ... something similar was done at the conference, by the way...

JU: ... OK, so you get a stepwise progression where each major advance in the level of civilization is tied to the amount of energy that could be mobilized by an individual.

It's tricky to talk about that right now, in an era when it's critical that we conserve.

MF: But you only get to conserve once. Replace all the fluorescent bulbs with LEDs, and do everything else you can, and you bought yourself ten, twenty, maybe thirty percent. If you want to increase your standard of living you don't conserve, you use up more energy.

JU: Right. So, how cool is it to be a performance engineer at Microsoft, take an interest in this topic, and wind up bringing the conference to the Microsoft conference center?

MF: Yeah. By the way, about 10 percent of the attendees were employees.

JU: I was wondering about that. The conference was fairly small, right?

MF: Around 50 people.

JU: Most of whom are practitioners, engaged in the R&D.

MF: Yes.

JU: But you got to bring it to other folks at Microsoft who, like you, the first time you went to one of these, will make all kinds of connections, and start thinking about the computational aspects.

Well done! And what a treat for you to get the chance to do it.

MF: Thanks. One nice thing was that it got posted on the internal calendar, and a guy I know found it, and he sent his two sons, a junior in high school and a junior in college. One's interested in mechanical engineering, and the other in civil engineering. I think this counts as a fairly large mechanical and civil engineering project!

So I meant to mention one of the interesting side effects of the length of the elevator, which is: What happens when you let go? Remember, everything above geosynchronous is trying to tear the ribbon apart, it's trying to leave the earth. If you let go at the right time of year, and the right time of day, you'll be on your way to Mars really quickly.

JU: [laughs]

MF: I'm serious.

JU: If you climb way above geosynchronous and let go, you slingshot to Mars?

MF: That is the appropriate word. Slingshot. I'm talking about a couple of months. Faster than the fastest stuff we've sent with big heavy boosters for these little bitty probes. You just take the biggest thing you want to send, and just let go, it'll be there in two or three months, for free.

JU: [laughs]

MF: And as a result, guess what's one of the first things you send to Mars using the earth space elevator.

JU: A Mars space elevator.

MF: Yes. There's a book on that, by the way. Science fiction. It's called Red Mars. It has an unfortunate ending involving terrorism. And of course the first space elevator conference, where everybody had read that book, was early 2002. Dr. Edward's said: "We've all read Red Mars, and yes, we have to take this into consideration".

There's been a study of how to defend the space elevator.

JU: I wasn't going to mention this, but that is maybe the worst vulnerability. You're not going to move it out of the way of a 747.

MF: The author's conclusion was that a consortium of private companies would own the space elevator, and the US government would trade defense for access to it.

You'd spend a billion dollars a year parking American assets around the elevator's airspace, and it'd be like, do not fly here, just don't, you will be shot down with no questions asked.

JU: And unlike the problem of defending the ground, this is a relatively well-defined region of the sky that needs to be defended.

MF: Right, it's not the whole continent. But everybody agrees it'll have to be defended. And we have the infrastructure, for better or worse, to do that.

But anyway, the possibility of moving around the solar system using space elevators is a whole other thing. Is that because there's interesting stuff out there? Because we're going to colonize Mars? Because we need more material sent down the elevator?

You might say that's visionary, or you might say it's just being practical.

JU: Well thanks again, this has been fascinating and a lot of fun.

MF: You're welcome Jon!