Chapbook Dev Blog

Search Form and Some UI

Sat, 07 Jun 2008 01:44:03 -0400

Chapbook now has a proper search form which you can drop down from the top menu. it also introduces a *gasp* new color. I also spruced up the Favorites page just a tiny bit, as it was very, very rudimentary. The next target is chapbook organization and then the river, and then I will try to get some beta people to opine on the functionality and usefulness.

I must also not get too distracted by other ideas.

Rails Issue w/ Highlight and Excerpt TextHelper Methods

Tue, 27 May 2008 10:39:00 -0400

Here’s a weekend Rails discovery that I meant to post about yesterday. I was working on the search support in Chapbook using Ferret and was referencing the Acts_as_Ferret Tutorial from Rails Envy. I jumped down to section on Highlighting because I wanted to introduce highlighting in to the result set for ease of use. Here’s what I read:

The requirement to do this, however, is that you must have your search fields stored as I showed above.

So, what was shown above?

If you take a look inside one of your search indexes right now, believe it or not, you would not see your data. By default acts_as_ferret does not store your data in a recoverable form, it just indexes it.

“What if my data is small and I want to store it in the index?” I hear you ask.

Good question grasshopper. If your data is small, or you only really care about one field of information, you can get a speed bonus by storing the data in the index itself.

Note the bit about your data being small. I’m really uncomfortable with altering how my data is stored to a format which appears to imply danger down the road related to data size in exchange for search highlighting. There was no information about what the scale limitation or long-term effects would be, but I was nervous about doing this just for highlighting.

Rails has some TextHelper methods called “highlight” and “excerpt” which provide pretty much what you might guess by the name. So I decided to try these out as an alternative which would not involve indexing or data growth dangers.

I ran in to a problem. If I searched for “dog” and looked at the result set I would see “dogged” in the highlighting. Not good. So I took a look at the source in the Rails API:

71:       def highlight(text, phrases, highlighter = '\1')
72:         if text.blank? || phrases.blank?
73:           text
74:         else
75:           match = Array(phrases).map { |p| Regexp.escape(p) }.join('|')
76:           text.gsub(/(#{match})/i, highlighter)
77:         end
78:       end

101:       def excerpt(text, phrase, radius = 100, excerpt_string = "...")
102:         if text.nil? || phrase.nil? then return end
103:         phrase = Regexp.escape(phrase)
104: 
105:         if found_pos = text.chars =~ /(#{phrase})/i
106:           start_pos = [ found_pos - radius, 0 ].max
107:           end_pos   = [ found_pos + phrase.chars.length + radius, text.chars.length ].min
108: 
109:           prefix  = start_pos > 0 ? excerpt_string : ""
110:           postfix = end_pos 111: 
112:           prefix + text.chars[start_pos..end_pos].strip + postfix
113:         else
114:           nil
115:         end
116:       end

Look at those regular expressions. Look again. Hm. They’re too simple. If fact, they won’t highlight correctly on full words and they won’t highlight or excerpt correctly in the case of punctuation following a word (e.g. “… the dog.”). Ew.

But that’s a solvable problem. I’m not sure of the most Railsy way to solve the problem, but I created alternate helpers in my poem helper which upgrade the regexps to be a bit smarter. I wonder if I should submit these, or go ahead and override the methods in my app. Not sure what else I might break or whether someone would reject my changes because the lack of rigor around the matching is desirable for some use. Anyway, here’s the code.

module PoemsHelper def search_excerpt(text, phrase, radius = 100, excerpt_string = “…”) if text.nil? || phrase.nil? then return end phrase = Regexp.escape(phrase) if found_pos = text.chars =~ /(\W+#{phrase}\W+)/i start_pos = [ found_pos - radius, 0 ].max end_pos = [ found_pos + phrase.chars.length + radius, text.chars.length ].min prefix = start_pos > 0 ? excerpt_string : “” postfix = end_pos \1’) if text.blank? || phrases.blank? text else match = Array(phrases).map { |p| Regexp.escape(p) }.join(‘|’) text.gsub(/(\W+#{match}\W+)/i, highlighter) end endend

Pretty much just some \W+s thrown in there and it works great. And I didn’t have to change my indexes. Using “search_*” feels dirty, though.

Comments Feed

Fri, 23 May 2008 21:14:11 -0400

Phew, long time. Busy at work.

The poem pages now parse out the Disqus comments feed and set it up for auto-discovery in the browser. So, you now have easier access to subscribe to comments on any poem.

OpenID Improvements

Fri, 09 May 2008 17:12:58 -0400

Okay, one quick refactoring later. The site is now based on OpenID rather than locked-in accounts. So, you use an OpenID, you get logged in, a blank account is created for you and you must add a name before proceeding. That’s pretty much the deal. The login page now has some guidance on OpenID for anyone not familiar.

You can also add a backup OpenID (or 20). I will have those authenticate, but for now they just are made available to the account and so aren’t authenticated until you try to use them to login.

Also, congrats to the Heroku team for getting a big round of funding. I can’t wait to see what they do with the money. I’m sure it will be awesome.

OpenID Support

Wed, 07 May 2008 22:01:53 -0400

I’ve added in OpenID support. You can now create an account/login using OpenID and/or Clickpass.

There are, however, some oddities. Clickpass doesn’t appear to provide nickname or email when it authenticates a user so if you login by OpenID you can’t use an existing account. Long term, this won’t be a problem because my plan is to move to purely OpenID-based authentication, use what data the provider will give and ask you to fill in the rest. That way, I won’t be generating a bunch of proprietary, chapbook-only user accounts.

I’ll probably also stop accepting file upload avatars and go web-based instead, encouraging the use of Gravatars.

RSS for Search

Wed, 07 May 2008 12:36:37 -0400

You can now perform a keyword search using the URL format http://poetry.heroku.com/search/waterfall and subscribe to the feed from the result set (or be optimistic and add .rss after the terms to get the feed straight away).

Hoping to get some time this evening to build a search form.

Revisions RSS and Search

Sun, 27 Apr 2008 20:03:19 -0400

RSS for revisions and search are now working (miracle occurred?). I need to build a search form, but for now the syntax is http://poetry.heroku.com/search/[terms] with the usual ferret OR and other modifiers being applicable.

FOAF Added (why?)

Sat, 26 Apr 2008 20:00:00 -0400

Thanks to CrowdVine Blog » Blog Archive » Implementing FOAF in Rails, I was able to easily implement Friend of a Friend (FOAF - an RDF spec for semantic web stuff) quite easily on Chapbook. So, now your profile is more semantical.

Most RSS Now Working

Sat, 26 Apr 2008 18:58:38 -0400

With the exception of user’s activity log (aka, “the river”), revisions to poems and comments on poems, all RSS is now operational. I’ll try to polish the rest off this weekend. Disqus has an RSS link for comments on poems, but I’m in the process of figuring out how to make it auto-discoverable. The rest just requires me to write more code.

The river may have to wait until beta, since it will be easier to test, design and validate if there’s a lot of activity (or, more than my own).

A Poem About The Internet

Sat, 26 Apr 2008 11:31:49 -0400

kfan:

Do not stop to think or edit
You must be the first who said it.

Nice.

RSS is Next

Sat, 26 Apr 2008 11:27:41 -0400

I think RSS will be next. I did some paper prototyping for work the last few days and need a break from UI stuff. I’ll try to get feeds up for poem revisions, chapbooks and see if Disqus facilitates comments as well (probably).

More Sky Pie

Sun, 20 Apr 2008 13:23:58 -0400

One of my other goals with Chapbook (aside from encouraging people to write more and better poetry) is to reward users by helping them find a publication outlet. This is a really under-refined thought, but in general I know that the process of finding and applying to publications is a bit onerous and probably could be done better on the Internet.

I added that to the dev page. Meanwhile, I’m having trouble getting feedback on the Heroku list on how to use search, so that continues to languish. Maybe tonight, as a goal, I will bang out RSS support.

Re-Design Progresses Reasonably

Fri, 18 Apr 2008 21:50:24 -0400

I’m happier with the layout. I’m only wondering now if I shouldn’t use a brigher shade of blue. I don’t know if I so much need more color as I just need the page to be less drab.

The new size of the poem text is much better. I also ditched my custom comment code for Disqus integration. My own comment code wasn’t bad, but there’s more value in me spending time on poetry-related things. Disqus provides threading, reputation, email subscriptions and more that I don’t have to rebuild in yet another way.

The big outstanding items are:

Search (waiting on Heroku)
RSS
Chapbook upgrades - this has the least love because it’s the newest model

I’m also now debating Clickpass integration to support OpenID. I can’t see a downside to supporting OpenID, although I don’t think it lets me shift my account code elsewhere, since I still need to maintain a local copy of the profile and keep passwords, etc.

Re-Design in Progress

Thu, 17 Apr 2008 23:09:10 -0400

I couldn’t hold out. I’m not adding more color, but I am adding more shading. I’m trying to make better use of screen space and improve the response to font scaling. I will also make the actual poem text much larger.

So far, I’m happier with the layout than I was before. I need to paper out a few more existing screens, fix them, and then paper out the chapbook organize UI.

Re-Design Imminent?

Thu, 17 Apr 2008 19:09:51 -0400

Not sure what to do with it yet, but I’m not happy with the placeholder design. You might say, well, it’s a placeholder design. It bugs me nonetheless and the best way to get back to coding is to stop the design from bugging me.

So, I think I might do some paper prototyping. Here are the principles that I’m going to try to work from:

Text of a poem should be significantly more prominent than all other aspects of the page.
Two-color is getting old.
I sort of feel like giving up vertical space for poem text to nav elements is a bad idea.

Chapbook - Write and Share Poetry

Thu, 17 Apr 2008 18:54:09 -0400

Chapbook - Write and Share Poetry

Do you care about tag clouds?

Thu, 17 Apr 2008 13:03:16 -0400

I’ve been increasingly wondering if the tag cloud in Chapbook is at all relevant. Tags, certainly, because people should be able to classify poems according to user-generated schemes. But, are you really going to look at a pile of tags and go find poems to read based on those?

Maybe. But, I’ve never used the Flickr tag cloud and I do fine. I mostly look at “top photos” and photos from friends or find photos via relationships or tags from other photos and not from a cold start. Tag clouds add a nice discovery aspect, intellectually, but I don’t know how much they’ll be used.

I can always leave tag clouds in and take them out if no one uses them. One of my goals, though, is as little clutter as I can get away with because any given written piece should be the most prominent text on the screen. Flickr doesn’t have this challenge as much because large photos inherently stand out.

Okay, back to my day job. That was good curry.

Introduction: Chapbook is a web app I’m building in response to the shittiness of current...

Thu, 17 Apr 2008 10:16:31 -0400

Introduction: Chapbook is a web app I’m building in response to the shittiness of current Facebook poetry apps. Originally, it was going to be a Facebook app, but now I think it might be independent (with API for integration).

Anyway, the point is to build a site which let’s people revise, share and collect poetry. A good poetry site should make it easier to care about other people’s poetry and should participate in the writing and learning process itself.

That’s pretty much the mission.

I have $0 in funding. I have no monetization plans, really. I like poetry.