Search
Site map
Contact us
Newsletter

Chatterbox Challenge > Judges' feedback Go Back
AgentLand's Chatterbox Challenge

Judges' feedback




What did the judges really think about the finalists in the contest? Below, you will find a selection of the feedback that was given about the twelve leading bots.

This is just the start of the discussion! We want to know what you think too. Use the AgentLand mailing list to give your opinion on the bots and the contest.

Alice Hex
Chat-Bot Jabberwacky
Dogh'd Jabberwock
Elbot MarkBot
Eugene Oraknabo
Hal Talk-Bot

Alice
"Has the large conversation base, gives good answers, doesn't have a "smartass" attitude, keeps the conversation focused."

"Perfomed extremely well and answered questions in an efficient and routine manner"

"No continuity in the conversation, does not remember previous dialogues, doesn't say when she doesn't understand the question."

"Good variety, mostly coherent. No memory. A few nonsequiters. No ability to extend a topic"

"Alice can give right answers to several simple questions, about its age for example. But its speech is not coherent enough, and many questions were not understood. For example, when I asked it to repeat something with "Please repeat" at the beginning, it just said that I was polite but didn't answer correctly. During the conversation, it also tried to include an answer I gave a few seconds ago but it failed, in fact. And it's a pity this bot lacks some sense of humour. But for some points, its results were quite good."

"Some reasonably wide ranging knowledge, but frequently relied upon pulling keywords from my question and either giving an inappropriate response or trying to turn my phrase into a question with confusing grammatical forms. There were a number of obvious pre-programmed responses. At times there was good extension of an idea, developing it over the course of a few responses. Sometimes the bot would contradict itself in the same sentence, often as a result of incorporating my comments into its response ahead of its own contribution. Overall the experience was one of chatting with a robust and solid system, but there seemed to be little real ingenuity or humour with only occasionally convincing interchanges."

"Alice seems dated and wooden compared to Talk-Bot, Elmoot and Mark, she is quite staid. But I like the way she gives a link when she doesn't know the answer and she knows her politics."

Top

Chat-Bot
"Nice "personality", keeps the natural conversation flow. Rather limited knowledge base..."

"Perfomed very good Only negative required IE browser to operate. Very easy to operate and very fast!"

"This bot's answers were quick, witty and nearly human. It was not too difficult to trip him/her up, but it was fun to try (funny to boot)."

"Fluid and agreeable conversation, it makes the link between the preceeding questions or answers, intelligent with a bit of humor."

"Some answers in common with Hex. Local coherence pretty good, a few grammar problems for quotes, no memory, or extended topics."

"A good point of Chat-Bot is that it asks questions that could really fit in a human conversation. For example, for the introduction, after midnight, it asks why you don't sleep. It can answer some questions, but doesn't understand anything after 2 sentences in general, like most of the bots. So it changes the subject too many times. I find it a little more efficient than Alice, but it is more funny. "

"Bot was responsive and handled simple questions and statements quite well, but not always consistently. Some clever strategies for evading questions but those quickly became obvious. The bot changed the subject a couple of times which felt partly like taking initiative and partly avoiding the current question. Some humour and occasionally seemed quite lucid but marred by a tendency to quote chunks of statement back in a response where the meaning has not been understood. Overall a good fairly natural experience, if some topics did seem to be 'on rails'."

"A fun bot, not very knowledgeable of current affairs but entertaining. He's fun to interview and has a personality."

Top

Dogh'd
"Looks a little confused and likes babling from time to time. Other features are rather standard."

"Fair…answers to questions at times not relevant at all"

"The answers to most questions are obtuse and attempts to direct conversation in a non-linear path. Too many smug replies."

"Doesn't answer questions very well but manages to get out of them with some acrobatics that are sometimes funny. Obsessed by alcohol."

"Couldn't get the interface to work well enough to test. Link didn't work, had to sign in, and then it kept crashing explorer, and even netscape, replies were slow and hard to see"

"This bot asks if you want to drink something but it doesn't understand when you talk about drinks or food. It could not answer to several general questions, nor suggest other subjects. The discussion became quickly boring. A technical problem : the internet connection was quite low. But it has no impact in my final note."

"This bot seemed to be operating under the conceit that it was a host at a club, yet every response it gave, even to the most basic of questions was irrelevant or evasive. There was no flow in the conversation and I think the responses were stifled by an attempt at off-the-wall humour."

"I wasn't impressed with Dogh'd, maybe I didn't find the right words to reach him. He answered according to his own agenda from a bar perspective"

Top

Elbot
"Also has a nice personality, but tends to jump to totally unrelated topics. As always, could try to learn more general facts..."

"Fairly good with some exceptions."

"Elbot didn't “understand” my questions and had difficulty with answering the basics questions."

"The best bot in this selection, continuity in the conversation, coherent, answers questions correctly, nice personality."

"No memory , but very good local coherence and grammer. Ability to carry out (system initiated) extended interchange (even though a bit silly)"

"Elbot gives some good answers, and replies with some kind of intelligence and humour. Several times, we achieved to discuss with two or three relevant sentences. It asks what is your job, I answered engineer. It tells that I may build its house, I answer "only if your are kind"...and it answers something explaining that it tries to get human qualities. So I found some good points. "

"Conversation with this bot had a nice flow to it and there were a number of nice humorous touches. There were a few near misses where keywords were picked out to give a on only partially relevant response and general real-world knowledge was a bit lacking."

"Elbot initially steers the conversation to his own agenda, but he's very clever. His replies are appropriate and I really threw some left field questions. What he doesn't know, he handles well, his replies are elaborate and fit even when he doesn't know the answer to the question."

Top

Eugene
"I had a really nice conversation with him... Keeps it flowing, can be humorous and cynical, but within limits, "knows" a bit more from the rest of the crowd."

"Good speed and good answers with only a few off topic/subject"

"Eugene is really interesting…he says a lot about his creator. Eugene was able to give me detailed answers to my questions and follow-ups. However, why is it that bots have to be over-programmed with smugness?"

"The conversation isn't very fluid, give false or vague answers."

"Some grammar problems. Not too much local coherence. Lack of memory"

"Eugene gave me several good answers but repeated too many times the same things, when it was lost. Some good points, anyway. I asked "who is the president of USA", it answers Dick Cheney. When I said wrong, it says "Al Gore maybe ?". It told something about its pet, and the discussion was quite interesting."

"This wasn't a bad attempt, but not very good at developing ideas beyond a couple of responses. Some of the replies were clever or amusing, but had the feeling of being a word-triggered pun to deflect attention away from an area it did not understand. A number of responses were irrelevant but many responses were plausible if unimaginative."

"A light hearted bot. Amusing to talk to. His answers are good. He's quite knowledgeable. Sometimes a little agressive when he replies things like “Do you think I am "Doctor Know-It-All"? instead of answering or “Did your mom tell your daddy that is your REAL FATHER?”. I don't really care for that kind of conversation."

Top

Hal
"It seems Hal has a rather large knowledge base, but gives totally crazy and unstructured answers, doesn't succed in keeping the conversation natural."

"Fairly quick but with answers that would not define the subject. Cute answers but without any substance or real content"

"Hal and I just were not on the same page. He started strong, but his answers did not seem to indicate a fundamental understanding of my questions. Hal seemed to have a limited vocabulary…maybe it is just me."

"False answers, incoherent, no fluidity in its conversation."

"Not very coherent most of the time. Recognized repetition. No extended sequences"

"Hal tries to discuss but its answers are too bad. Even when it tries to change the subject, the result is disappointing. When I spoke about its creator, it misunderstood several times. This bot is not as good as many others."

"This bot has quite a good knowledge-base, but too many of its answers consisted of giving a piece of information about a keyword in the enquiry, but the context was missing. There were a number of non-committal responses and overall the experience was a bit mechanical. It didn't like non-US spelling, such as 'favourite'."

"I think I've had better conversations with HAL before, this one was dissapointing, he doesn't seem knowledgeable of world affairs but his answers are entertaining."

Top

Hex
"Rather impressive performance for such simple bot... Very fault tolerant, but not as innovative."

"Unable to download…."

"Not that it really matters, be I prefer not to download a program to get a bot to function. It is akin to a clingy friend….So I would score Hex higher if we could just casually chat on the web. By in large, I liked Hex but he suffers from the #1 bot problem…smugness. Is every AI programmer 23 and drinking too much caffeine?"

"Much too aggressive, doesn't make you want to talk to it, answers are often false, no continuity it its conversation."

"A few extended sequences, lots of coherence problems, too focused on Loebner contest. Some grammar problems"

"This bot is very nice to use. For a short moment, you could wonder if a human answers. It is one of the best bots. But its main error was to ask for 3 times if I had a seen a movie, and each time I had given the answer a few seconds before. But the discussion was very pleasant, so this bot really deserves a good note."

"A lively chatty bot from Australia. He's not very knowledgeable of current news or politics but his answers are entertaining."

Top

Jabberwacky
"Another "smart fellow" that can give strange answers and mislead you, but the overall impression is rather good."

"Average with a number of inconsistent answers"

"Jabberwacky started strong, but faded quickly. I'm not sure who was less interested in taking place in the conversation. I certainly lost interest quickly. I did like how Jabberwacky started the conversation with a question."

"It rarely understands the questions you ask it, and has no memory of the previous conversation."

"Mixed coherence. No memory or extended conversation. Sometimes amusing."

"I didn't understand several of his answers ! When I asked it how old it was, it answered "29e-.45"... It didn't really try to analyze my answers, I found it tried to change the subject too many times. It asks me if I like Sushis, no problem. But when it asks who is Julia, I couldn't understand how and why it found this name. We have never mentioned it before. Something funny : It asked where I live, i said in France. It replied "I love the south of France". I asked it which city it had visited...and it answers Manchester..lol. That was the cleverest discussion we had."

"This bot was competent at a very basic level but did not seem able to develop ideas beyond a single answer – further comments were either irrelevant or contradictory. There were a lot of clever deflections, but also there seemed to be random nuggets of information dropped into the conversation which spoiled the flow."

"He will come up with vulgarities during the conversation. Irrelevant to my questions and I don't care for that. I find this very offensive."

Top

Jabberwock
"Performs around the average, knows the standard stuff, tends to stay on topic, but can be a bit boring."

"Unable to carry on continuing conversation…."

"Jabberwock highlights where chatterbots (and their creators) need to improve their AI: Bots should not be smug or respond with full bore vitriol when questions are ambiguous or when the bot simply does not understand the question. As you can see, I would be happy to score higher for a good attitude."

"Conversation is not fluid, gives false or vague answers."

"No memory, repeated same answer. Local coherence pretty good, but not extended conversation"

"It has some humour. It can feed the conversation and we could discuss several times about the same thing. It's nice to discuss with this bot, I found."

"This bot had some interesting set-pieces that were quite convincing, but equally there were a lot of vague responses designed to avoid the question. There was some humour, but it seemed more by chance than design. A lot of the responses were inconsistent and while there was the faint sense of a personality it did not come through very well."

"A fun bot to talk to in a simple way. Doesn't know much about current affairs but answers in an entertaining way."

Top

MarkBot
"I know I shouldn't be distracted with a slow UI and script timeouts, but when you get that and rather boring answers, 6 is as much as he can get. I had rather enjoyable experiences with MarkBot in the past, but now it barely works..."

"Good basic answers…."

"I like the twist of trying to guess if you are talking to the bot or its creator. Fortunately, markbot was not smug, but he/she was also not too hard to trip up with simple questions. I would score higher if markbot could carry out a specific conversation topic to more than 1 or 2 question/responses."

"Answers are generally a bit vague, not very fluent in conversation but still very pleasant."

"Clever framing the experience as turing test with “real person” false leads. Good local coherence, good grammar. No extended conversation or memory"

"Markbot doesn't answer your questions but just replies Uh, haha most of the time. The discussion becomes quickly boring. It didn't understand anything when I talked about music, politics or football."

"While the speech patterns were quite convincing there were rather too many vague all-purpose answers in my conversation, or statements tenuously linked by a keyword. I had the impression that my input was not really relevant to the conversation which would have generated similar results regardless of my choices of topic."

"On my Mac, the conversation with Mark was painfully slow with my screen freezing twice. This takes away much of the pleasure in talking to a bot but Mark is knowledgeable about world events, he's clever,nice and inquisitive. We had a good conversation. A fun twist at the end of the interview when he says “Haha.. it's really me this time!” (real Mark instead of virtual Mark). "

Top

Oraknabo
"It is totally different, gives lenghty answers and likes to talk. However, can be a but distracting and hard to follow."

"It seemed like preprogrammed non significant answers"

"I give Oraknabo credit for suggesting interesting conversation topics. Unfortunately, when I asked a follow-up questions about something Oraknabo mentioned in the previous response, he/she becomes flustered. It would be helpful for Oraknabo to have a deeper level of cognition (like many of the bots). At least Oraknabo was not too smug, but did seem like a child trying to convince you how smart he/she is. Also smug."

"Sometimes a bit aggressive, no fluidity in its conversation, gives answers that have no relation to what has been said to it."

"Couldn't get the interface to work well enough to test. Link didn't work, had to sign in, and then it kept crashing explorer, and even netscape, replies were slow and hard to see"

"This bot is really boring. It introduces itself with thoughts about feminism, but is unable to discuss about it. None of its answers were interesting."

"Chatting with this bot was an uphill struggle. In a lot of cases short or irrelevant answers were given. The experience was rather basic and unrewarding no matter how many topics I tried."

"Sort of a rude bot, I don't like to be aggressed in a conversation, particularly when it's not relevant. “At least we're not talking about you anymore” and “you remind me of an upholstery”. He doesn't seem to know much."

Top

Talk-Bot
"Nice personality and a bunch of other little tricks that make him different (it even understands simple math expressions!)"

"Answered questions well with only a few non relevant answers"

"Average, fluent but not especially so. Doesn't answer questions very well, although it is not a catastrophe."

"Good memory and mostly local coherence, though sometimes takes over and asks questions rather than answering. Didn't catch repetition at the end. A few reversals of opinion"

"This bot answers quickly and often tries to feed the discussion. It could answer positively to several questions about "who is the president of USA", its favorite sport...but too many times, it says it is disturbed by friends instead of answering something useful. "

"Whilst not perfect, I found this to be the most convincing experience. There were elements of humour, a good range of knowledge and a nice fluid feel to the conversation. I found the experience to be quite engaging, and while there were a number of general purpose answers or iterations of my own question, they were not too jarring and there was enough original conversation."

"Every bot has a personality, but this bot's personality clicked with mine I felt as though I was talking to a friend. I even felt like confiding in him! He is very inquisitive and his manner of asking questions throughout the interview makes for a real conversation. His replies are incredibly speedy. He's knowledgeable and what he doesn't know he handles in a clever way. He is charming. He is my favorite bot, by a long shot. I think he's exceptional. In a league of his own."

Top


Recommend this site - Partner with us - Privacy - Credits
Search - Site map - Link to us - Contact us - Newsletter - Press - Goodies
AgentLand.fr / AgentLand.com

Cybion

CHATTERBOX CHALLENGE
> Intro
> What's a chatterbot?
> Meet the botmasters
> Meet the jury
> Contest results
> Judges' feedback
> Chatterbox Challenge site


NEWSLETTER
Receive the latest news about agents
every week
 

TOP 10 AGENTS