I like the idea. I'd like to see a fairly steady amount of contests or otherwise community-wide things on the site for users to enjoy together. Being a creative writing website, this sort of contest simply makes sense. Does it have to wait until Fall though? Sooo faaarr awaayyy. Could be it gotten together and done during the month of August?
Thoughts on running it -- Two categories, Short Stories and Poems. One winner for each category. One entry per category, per user. I would suggest doing only one theme for now. If the contest proves popular, perhaps down the line it can be repeated with different themes in mind. Maybe it's just an interesting theme the entry has to be based on, or in addition to the theme there are guidelines that have to also be met: a specific word has to be used, or a specific item has to be implemented, or a specific setting has to be used. This could add some fun challenge to a so-so theme, though if the chosen theme is interesting enough on its own I wouldn't gild the lily.
As for judging it, and trying to keep things as unbiased as possible, the entries could be posted anonymously. A user would private message their entry to a specified mod/admin by a certain date, and that mod/admin can then enter all of the entries in one stickied post once the date has been reached, with the peoples' names omitted. If you want to do public voting, include a poll where people can read everything and then vote for their favorite. This won't really stop dudeguy07 from messaging all his friends telling them to vote for the eighth entry on the list though, which may or may not make a difference depending on the size of the turnout. Another idea on judging is for the mods/admin to judge the entries, as I think the integrity of the people we have in charge here goes without saying. Or in addition to that (or instead of), a few users with good writing skill and who can be trusted to be unbias can be asked to help judge anonymously.
Thoughts on prizes -- Perhaps temporary prizes similar to the Easter Egg Hunt: anyone who enters can be given a badge showing their participation and access to the VIP section, while the winner(s) receives a badge, VIP access and username color. For something more permanent, maybe some of our more talented artists on the site could be asked if they would volunteer their services to create a nice avatar/signature banner for the winner(s), or a piece of artwork; maybe something inspired by the winning entry itself.
Edit: Some additional thoughts. I'm not really feeling Mitsu's suggestion of votes from a mod/admin being worth a different amount than a standard user. I'm also not sure making a group for 'talented BM writers' is a good idea, as that's just asking to be taken the wrong way, good intentions or not. Maybe akin to the Superposters group there can be a 'Contest Winner' group/color if this is to become a repeating theme for the site? *shrugs* I can say that I, personally, was happy with the prize for the Easter Egg Hunt. Something to display permanently would be nice, but either way, I'm more interested in seeing the community inclusion this brings than the prize itself.