At Full Stack Fest, I gave a talk on how to use Unison’s distributed programming API to write a simple search engine that runs on multiple nodes:
I didn’t have all the code working at the time I gave the talk, but promised it would be done Any Day Now™. Well, now after another few weeks of hacking, I have something running. This ~ 125 LOC file defines:
When I run this program, it actually works. Here’s the output:
waiting 2 minutes for indexing before issuing queries...: () indexing url: "http://unisonweb.org/design" indexing url: "http://lambda-the-ultimate.org/" indexing url: "http://www.cnn.com" finished indexing: "http://unisonweb.org/design" indexing url: "http://unisonweb.org/" finished indexing: "http://www.cnn.com" ...
A bunch of pages get indexed, and then two minutes later, the query
search 10 ["design", "unison", "programming"] ind
gets run and the results formatted:
http://www.zazzle.com/unisonweb/products Unison programming platform merchandise: Products on Zazzle Home Shop Create Sell Gifts My Account My Likes Collections Saved Designs Sign in » Shopping Cart (0 items) View Cart (0 items) 100% Satisfaction Guaranteed Shop Create Sell Gifts Shopping Cart (0 items) View Cart (0 items) 100% Satisfaction Guaranteed A community marketplace where you can collaborate with makers & designers to make almos... *** http://unisonweb.org/design Posts tagged 'design' Unison next-generation programming platform, currently in development about help fund the project swag Twitter . GitHub . RSS Posts tagged 'design' [ RSS ] These posts are design writeups for aspects of the Unison platform. The Unison codebase editing experience using plain-text tools Node containers and lightweight, zero-copy node provisioning Shared node storage and more ef... *** http://unisonweb.org/ Unison next-generation programming platform, currently in development about help fund the project swag Twitter . GitHub . RSS The Unison codebase editing experience using plain-text tools There are huge benefits to modeling the codebase as an immutable data structure, rather than a bag of text files that’s mutated in place. This post talks about these benefits while walking through a design for ho... ***
Nice! (Notice only 3 results—the crawling didn’t get very far in 2 minutes, maybe will try running it overnight sometime…)
This search engine is a proof of concept. It exercises a lot and shows what is possible, and it’s great that we’ve gotten this far. But, here are some ways in which this proof of concept is not yet realistic:
DIndexis too naive in various ways, and I doubt it would scale up to very large clusters—among other things, it needs to maintain the cluster state (the directory of nodes) in a decentralized fashion (Raft, Paxos), and it should probably implement the ‘skeleton’ variant for speeding up HRW hashing in large clusters. And that’s great! Raft consensus can be an ordinary Unison library, used by
DIndexand whatever other distributed data structures.
On the one hand, it’s lovely being able to use a single general purpose, typed programming language to describe all relevant aspects of building a software system. You don’t need to cobble together 15 different technologies just to build a multi-node software system! On the other hand, our programming tech is no longer enforcing and separation of concerns (the Unison language can talk about provisioning and deployment, so nothing stops you from intermingling these concerns in ad hoc ways with stuff like your ‘business logic’ that should be decoupled). Thus, we actually need to start thinking about how to carve things up and make it maintainable, upgradable, easy to monitor, and so on. This is probably the most ‘interesting’ problem (from Unison’s perspective): how do we build a highly available distributed Unison service and upgrade / maintain it over time?
However, I view this as a good problem to have—you aren’t forced by your programming tech into any particular decomposition of your system into a rigid set of phases, each managed by special-purpose tools. Instead, you can think about what decomposition makes the most sense for your use cases. And any common patterns you discover can be packaged up as regular Unison libraries and functions and reused!
This GitHub meta-issue tracks upcomming work on the project, and you can also check out the project board. But to summarize, the project is moving out of an R&D phase and will be focused on the known engineering work to make this a language you can use for real stuff!comments powered by Disqus