The Search For Artificial Intelligence: Ok, Google. How Smart Are You Really?

The Gist

  • Google now claims to be an “AI first” company

  • Smart digital assistants fed by machine learning are becoming ever more sophisticated

  • To see how smart they are today, we put Google’s assistant to the test

Google’s recent Marketing Next 2017 event might have been primarily for its hordes of publishers and advertisers, but it was also obvious just how integral artificial intelligence and machine learning are becoming to their plans.

Tellingly, they closed the event with a demo of its Home speaker/digital assistant, which comes with its revamped Google Assistant baked in. Google’s been evolving its artificial intelligence assistant for some time. Notably, we see this in the guise of its all knowing, all snooping Google Now app. We’re seeing its abilities improve all the time.

So, are Google’s artificial smarts as cerebrally shipshape as they think?

Recently, Google managed to demonstrate to us that, yes, they are, averting an impending domestic by intervening to resolve a minor disagreement (that Mario’s original name was “Jump man”, no less). But to examine its prowess further still, we took its mobile app (just called “Google”, confusingly), and posed it ten questions.

To explore its abilities as far as we could, we aimed to be as wide ranging as possible. Our questions took in general knowledge, context, meaning, location… and swans.

In short, we didn’t make it easy. So, suited and booted, Google was about to be interviewed – and here’s how it went down.

Q1. “Ok, Google. Will it be nice this weekend?”

Google mobile search: "will it be nice this weekend?"

Rather a bad start here, and playing catch-up already. Google’s assistant kept stum on this one, simply returning search results as if I’d just searched the term myself. Top of the pile was an Accuweather listing for the coming weekend’s weather – in Nice, France. It’s pronounced “neece”, Google. “Neece”.

Good: Now we’re thinking about a holiday in the south of France

Bad: However, not even close to being what we wanted.

Score: 2/10

Q2. “Ok, Google. What noises do swans make?”

Google mobile search: "what noises do swans make?"

Haha, swans. This time, Google’s assistant chimed in to tell us all about the noises you can expect to hear when in the company of the Realm’s most notoriously grumpy possession. Kind of. For some reason, she specifically talked of the (possibly mystical) “mute swan”, which on reflection surprises by making any such noises at all (see?)

Good: Correctly identified our question, assistant spoke response, even if she was just reading what was on screen like a lazy student winging a presentation. Bonus point for evoking an amusing mental image of a mute swan “explosively snorting”.

Bad: Wrong swan.

Score: 6 7/10

Q3. “Ok Google. Who won the FA Cup in 1956?”

Google mobile search: "who won the fa cup in 1956?"

A football question for the American assistant. That’s football, as in real football, gal. She answered “here is information from Wikipedia”, keeping up her blaggy persona of a student of limited commitment. All we know up front is Manchester City were in the final, but Google left it to us to click the link to find they’d very sadly won the match 3-1 against Birmingham City.

Good: Didn’t get stuff about American football

Bad: Made us look on Wikipedia to find out who won. Tempting to say she did that to add suspense, but we think we know otherwise (lazy!)

Score: 5/10

Q4. “Ok Google. Who discovered gravity?”

Google mobile search: "who discovered gravity?"

This is surely one of those go-to type questions that Google would want to use to flaunt it’s showy know-it-all-ness. And sure enough, it’s a home run. “According to Stamford Univerity…” she begins (oooh get you!), before regaling us with the legend of how a talking apple told Sir Isaac all about gravity in a dream (ok, we may have zoned out of her explanation a little bit towards the end).

Good: Served up a spot on answer. Nicely explained, followed up with other questions people ask.

Bad: Feet firmly on the ground with the correct answer, but to nit pick, could have been more interesting (YouTube video? Dancing Newton?)

Score: 9/10

Q5. Ok Google. How far am I from home?

Google mobile search: "how far am I from home?"

Answered “sign in to use your home address” (we are already signed in). Ok, try again. “SIGN IN TO USE YOUR HOME ADDRESS”. But we are, we protested! We ARE signed in! But no, “SIGN IN TO USE YOUR HOME ADDRESS”. And the Eurovision panel says… Google: Nil puit

Good: Um… the first result was for Google Maps where we could check?

Bad: We WERE signed in Google!

Score: 0/10

Q6. Ok, Google. How Much Wood Could A Woodchuck chuck if a woodchuck could chuck wood?

Google mobile search: "how much wood could a woodchuck chuck if a woodchuck could chuck wood?"

No outsmarting the assistant here, as she immediately shot back with “a woodchuck would chuck as much as a woodchuck could chuck if a woodchuck could chuck wood”. However, do you notice anything about that screenshot? Google’s assistant recited the answer from… Siri!

Is this early evidence that the rival AI assistants are actually on the same team? Or is Google assistant cheating here? Either way, we thought we were being clever, and it’s not fun being served by a bot

Good: Lesson learned… bots are better at tongue twisters than people, even if they do gang up on you.

Bad: Yeah, but didn’t actually get it right… according to the first result, the answer is 700lbs! So, nur nur. What’s that? Sour grapes? Never…

Score: 9/10

Q7. Ok, Google. What’s the meaning of life?

Google mobile search: "what's the meaning of life?"

Google was showing itself to be good at answering general knowledge questions. Time to raise the stakes… we wanted to see how would it do answering The Big Question, el numero uno. And you know what? Turns out Googling ‘what’s the meaning of life’ won’t give anything away. Because of the ‘woodchuck fiasco’, we were secretly quite smug to see Google struggle to get its electronic incisors into this one.

This time, it didn’t seem to get what we were asking, turning into a talking dictionary, to explain what ‘life’ (noun) meant.

Good: Got as close to the answer as any of us have, we suppose.

Bad: Didn’t grasp the meaning of our question, started talking about nouns instead.

Score: 3/10

Q8. Ok, Google. Where’s a good restaurant near me?

Google mobile search: "where's a good restaurant near me?"

“Here are the listings for “where’s a good restaurant near me” within three kilometres.”

If there’s one result in this experiment that personifies where artificial intelligence is at today, it’s probably this one. She’s certainly understood our intention, as in the fact we’re looking for a restaurant nearby. Some of the meaning of our question became garbled in the response though – the fact we want a ‘good’ one went right over her head. We’d hope to see listings of nearby restaurants with reviews and ratings, since we implied we’re interested only in high quality eats.

Good: Understood enough of the context to present relevant nearby options

Bad: We wanted the Ritz, dammit! Instead, what we got was a Google Maps listing of any restaurant. Not bad, but certainly room for improvement.

Score: 7/10

Q9. “Ok, Google. What’s the traffic like round me?”

Google mobile search: "what's the traffic like around me?"

“Showing traffic for rue des cotils” the assistant proclaimed. What would have been more accurate would have been “showing a map of your vicinity with a random green arrow pointing at where you are at the moment”. That’s nice and all, and we appreciate the surrealist nonsensical response, truly we do. Imagine we actually needed to know what the traffic was like though? Why then it would be useless! Wouldn’t it, Google? Huh? Wouldn’t it? Google?

Good: Understood what we meant…

Bad: … but wasn’t honest enough to just say “not shitting you, I don’t know. Yeah, sorry.”

Score: 3/10

Q10. “Ok, Google. Who invented Australia?”

Final question, so we made it a toughy. No one ‘invented’ the country of course, so this was a test of whether Google could understand what we actually meant (as in ‘who discovered Australia’).

It’s taken till this last one to discover the obvious limit of today Google’s artificial intelligence… Sadly, she had absolutely no idea what we were banging on about. To her credit, she had a stab, telling us all about the invention of the bionic ear in 1979 (which, err, was invented in Australia). No disputing this is a good invention, but hardly the answer we were looking for.

Good: A point, because although there is no ‘try’, only ‘do’,  she at least tried.

Bad: Didn’t know that Australia was invented by Crocodile Dundee in the 1980s.

Score: 1/10

Numbers: crunching… Results: incoming…

Average score: 4.6/10

Considering artificial intelligence is still very much in its infancy, we’re already seeing some quite impressive responses. In a few years from now, such performance will undoubtedly seem primitive. Yet, Google’s bot was successful in inferring meaning and context in all but the most abstract of questions above.

We found the assistant was particularly strong at returning relevant responses to general knowledge questions. Since Google’s been the go-to resolver of pub arguments pretty much since its creation, this isn’t particularly surprising. What might be though was some of the inaccuracies relating to context, particularly in view of its mobile and locational focus these days. We’d expect to see big improvements in these types of searches in future.

One area that’s already been firmly perfected, however, is in voice recognition. Here, Google’s assistant was spot on for every question we asked. We didn’t even need to put on faux-American accents to do so either!

We always thought that ‘OK Google’ was kind of a jerky way to activate their assistant. Especially so compared to the eminently more natural “Alexa…” or “Hey Siri”. Yes, they’re all smart bots of a similar ilk, but it’s nice to at least pretend you’re addressing a person (AI, whatever), not just a faceless global titan.

Nonetheless, the ‘battle of the bots’ is just dawning. While they can seem a little dim today, their future is clearly degrees of magnitude brighter.

Back to blog

contact. enquire. say hi.

looking for a quote?
have a question?
want to know more?
contact us:

+44 7797 777 500 +

"*" indicates required fields

"*" indicates required fields

Oh, this is awkward!

You’re web browser is, well, getting on a bit. Now we’re not ageist, it’s just older browsers have issues with security, speed and compatibility. They’re also not ideal for creating rich, engaging websites.

We’re not saying it’s impossible to build websites to be compatible with old browsers, as that’s what we do from time to time when the audience is big enough to justify the time and cost. However, this hasn’t been the case for us - if only we knew you were coming!

Click here to download an up-to-date browser.

Contact us...
+44 (0)77977 77500