Ask Cortana / Ask Bing - Discussion

[Elliot Kirk]

Hello Insiders! You have told us that you really want the Ask Cortana / Ask Bing feature in our next version of Microsoft Edge.

In the current version of Microsoft Edge, you can select text or click on an image and get a result from Cortana / Bing in a side panel, without leaving the page you are browsing.

We hear you and from your feedback we understand you would like similar contextual search functionality in a way that does not pull you out of your browsing flow.

We are investigating our ability to add this to the next version of Microsoft Edge and we would like to understand what makes this feature useful for you. 

How did you use this feature?  If you ever attempted to use Ask Cortana / Ask Bing and couldn’t get the answer you were looking for, what steps did you take, if any, to get the answer you were looking for?  

What aspects of the Ask Cortana experience do you like or appreciate?  What improvements would you like to see?

Please let us know, either in the comments below or by using the Send-A-Smile feedback. 

Thanks – Jared Brown and the Microsoft Edge Search Team

[#2]

In fact here is another revolutionary and controversial idea: 

Outside our developer community, Bing doesnt really feature as a search engine - everyone knows Google - its been verbed. Folk who haven't played Halo, or had a dinosaur phone (Lumia) have never heard of Cortana.

So, rename / splice "Bing" to Cortana - one unified search engine / AI, big marketing push, use some of the Cortana renders from the various Halo movielets - "Forward unto dawn" etc. Combine them with Donovan Brown's ads for Microsoft AI, walking round ancient ruins / on safari with Cortana on one shoulder acting as guide.  

Hey I should be in Marketing. I need to check my meds.

[#3]

@LegacyOfherot 

Spoiler

---

@LegacyOfherot wrote:

In fact here is another revolutionary and controversial idea: 

Outside our developer community, Bing doesnt really feature as a search engine - everyone knows Google - its been verbed. Folk who haven't played Halo, or had a dinosaur phone (Lumia) have never heard of Cortana.

So, rename / splice "Bing" to Cortana - one unified search engine / AI, big marketing push, use some of the Cortana renders from the various Halo movielets - "Forward unto dawn" etc. Combine them with Donovan Brown's ads for Microsoft AI, walking round ancient ruins / on safari with Cortana on one shoulder acting as guide.  

Hey I should be in Marketing. I need to check my meds.

---

more than 1 Billions people are using Windows 10, so Cortana is not unknown.

MSN,Live,Bing, all 3 words have been integrated in each other for years.

the difference is that google adds their prefix "google" to almost all of their products. google keep, google asssistance, google hangout, google whatever.

Microsoft has different style. "Word" , "OneNote" , "Skype" , "Bing" , "Cortana".

there is no problem with that. google thinks if they don't shove word "google" into everyone's throat, it means they lose.

[#4]

@HotCakeX ,  I said this was controversial. I agree there are a lot of Windows 10 machines out there (and a large number of older versions too, Win 7 refuses to go quietly into the night).

If you watch the news, whenever they talk about personal assistants, they quote Siri and Alexa.

Whenever they mention searching the interweb - its "I Googled for it".

Walk into a bar and ask anyone over 40 about Cortana, you'll get a blank look - as for explaining the difference between asking Bing and asking Cortana - I'm not sure I could.

My pitch was that Microsoft don't have a clear message, and I think having one point of interaction - Cortana - would be simpler, and a great marketing meme - "Ask Cortana" -  you needn't just get get a list of vaguely relevant web sites, complete with their GDPR "You cant do anything useful here unless you allow our cookies", She might actually give useful facts and figures.

On my Win 10 desktop - I click on the Cortana button and say "Tell me about the Tower of London". I get a link to the ToL's web site, a wikipedia entry, some other places to visit. I want her to TELL ME ABOUT the tower.

If I'm there (She knows I'm there because I have to have Location services turned on) She could tell me when it was built, and as I walk around, look through the camera on the Surface (not on my Lumia, obviously) and tell me about the various buildings. "Cortana, where is the bar?" and a map comes up.

BTW I hate Google too - In my view they are the new Evil Empire. I'm forming the Rebel Alliance. Come to the light side, we have cookies.

[#5]

Bing is a search engine, it crawls and indexes websites. Cortana is software that uses AI to operate. that's the difference.

Cortana is simply not good enough. it used to identify song lyrics, play songs on YouTube and Spotify and lots of other things, but it doesn't do it anymore. Microsoft plans to make it more useful to the businesses and enterprises rather than normal users.

so the solution is not merge, but improving Cortana to be actually a good AI.

https://blogs.windows.com/windowsexperience/2020/02/28/cortana-in-the-upcoming-windows-10-release-fo...

[#6]

@Elliot KirkI use both 'Search the web for' and 'ask Cortana about' all the time

I use ask Cortana when I need a quick definition or extra info about something without leaving the page, when the quck info will be enough and I don't need to explore further, for info about where an image comes from or as a quick way of seeing an image full size in its own tab.

I use 'search the web for' when I want to explore a topic in more depth and expect to open multiple links and go to multiple page and so want a persistent list of search results 

[#7]

@HotCakeX Thank you for clarifying that. As I said, we, as developers, can discuss the nuances of a user interface till the cows come home.

The other 900,000,000 users of Windows 10 are not quite so knowledgeable.

PCs are no longer the default for most people, they / we use phones, tablets, other devices. I 'tooth  my phone to the car, and if I receive a text message, Cortana asks if I'd like to reply, I say "Yes" and she turns the music back on. She used to respond to my voice commands, but now, I have to stop and get the phone out of my pocket.

On my Desktop, I asked Cortana "Tell me about the Tower of London", she gave me this - looks like she asked Bing on my behalf:

I opened a browser window and entered "Tower of London" in Bing, and got this:

Neither of which is exactly what I wanted. My Encarta CD has quite a lot of information on the Tower, original content, owned by MS - no copyright issues!

Walking around Hampstead, there are some fascinating buildings, some owned by the National Trust. It would be really cool to have Cortana tell me about the area AND inside the buildings, no button pushes or opening browser windows.  Just a foot-tall AI sitting on my right shoulder talking through my earpiece. Speak to the NT down in Swindon, Wiltshire UK, they are very IT savvy.

Back in 2015, some Microsoft guys had a device on show at Future Decoded, intended for blind people to wear on the head. (Just a 3D printed prototype).

It sensed direction with a small flux gate magnetometer, and toothed to an app running on a Lumia 950. The app had a haptic user interface - as the user dragged down a list, the phone vibrated simulating a resistance to scrolling, like notches. As far as I know it went no further, but now Microsoft do have an app for fruity phones to provide street guidance for blind people. I can't remember what they call it - more forgettable marketing from Redmond. If someone has an iPhone, maybe they could ask Siri?

My point is: this forum is about interfacing Bing / Cortana to Edge, but we need to remember Edge is just a UI - one of many - the UI is not the application.

[#8]

@LegacyOfherot 

Spoiler

---

@LegacyOfherot wrote:

@HotCakeX Thank you for clarifying that. As I said, we, as developers, can discuss the nuances of a user interface till the cows come home.

The other 900,000,000 users of Windows 10 are not quite so knowledgeable.

PCs are no longer the default for most people, they / we use phones, tablets, other devices. I 'tooth  my phone to the car, and if I receive a text message, Cortana asks if I'd like to reply, I say "Yes" and she turns the music back on. She used to respond to my voice commands, but now, I have to stop and get the phone out of my pocket.

On my Desktop, I asked Cortana "Tell me about the Tower of London", she gave me this - looks like she asked Bing on my behalf:

I opened a browser window and entered "Tower of London" in Bing, and got this:

Neither of which is exactly what I wanted. My Encarta CD has quite a lot of information on the Tower, original content, owned by MS - no copyright issues!

Walking around Hampstead, there are some fascinating buildings, some owned by the National Trust. It would be really cool to have Cortana tell me about the area AND inside the buildings, no button pushes or opening browser windows.  Just a foot-tall AI sitting on my right shoulder talking through my earpiece. Speak to the NT down in Swindon, Wiltshire UK, they are very IT savvy.

Back in 2015, some Microsoft guys had a device on show at Future Decoded, intended for blind people to wear on the head. (Just a 3D printed prototype).

It sensed direction with a small flux gate magnetometer, and toothed to an app running on a Lumia 950. The app had a haptic user interface - as the user dragged down a list, the phone vibrated simulating a resistance to scrolling, like notches. As far as I know it went no further, but now Microsoft do have an app for fruity phones to provide street guidance for blind people. I can't remember what they call it - more forgettable marketing from Redmond. If someone has an iPhone, maybe they could ask Siri?

My point is: this forum is about interfacing Bing / Cortana to Edge, but we need to remember Edge is just a UI - one of many - the UI is not the application.

---

if you're a developer then how can you say Edge is just a UI...

Edge is a full application, a browser.

it has a UI obviously but the code behind it is even more important (the framework it's built on, the engine etc).

I don't think that number is correct, it's just your personal opinion about Windows 10 users being educated or knowledgeable or not.

Microsoft clearly showed us that they intend to make Cortana only useful for businesses and enterprises with integration to Office/Microsoft 365 subscriptions.

[#9]

FYI:
here is 2 examples to show how UI is separate from the rest of the browser code:

Firefox:
Rendering engine: Gecko, C++, and in recent versions Rust language used too
JavaScript engine: SpiderMonkey, C
UI: Mostly XUL (a custom XML dialect), CSS, and JavaScript, with some C++.

Chrome:
Rendering engine: WebCore, C++
JavaScript engine: V8, C++
UI: Mostly C++, though the mac port uses Objective-C, and some features in all platforms use HTML, CSS, and JavaScript.

Edge uses the same rendering and Java script engine as Chrome, for the UI i think it uses HTML/JS/CSS too
Microsoft could've wrapped the actual code in XAML language and create a UWP UI for the Edge browser which would look great on Windows 10.

[#10]

---

@HotCakeX wrote:


Edge uses the same rendering and Java script engine as Chrome, for the UI i think it uses HTML/JS/CSS too
Microsoft could've wrapped the actual code in XAML language and create a UWP UI for the Edge browser which would look great on Windows 10.

---

+++++

[#11]

@HotCakeX 

Sigh.. one last try, then I'll go away.

I'm not talking about  MVC / MVVM /separation of concerns. I'm saying we should consider how Bing/Cortana work in the bigger picture. The physical UI in Edge is ok for users with a screen and keyboard/mouse/finger, could use some work as discussed here.

I was trying to point out that not all devices have those things: if I have a brick on my nose (Hololens/Samsung gear/Occulus/Sony thing) then selecting some text to get quick help doesn't really scan.

If I'm in my car I'd like to be able to tell Cortana to email/text someone, send a meeting request, set a reminder, find a restaurant / pub nearby. I can ask Frau Audi to call them, but her skills are limited, I must ask the right questions.

Microsoft abandoned their Band users, so that's no longer an issue.

Windows phone... uh huh.

Some of us bought Kinects, both for PC and XBox. I'm really not sure how I would differentiate between Bing and Cortana on XBox. I mean XBone - they abandoned XBox 360 users a while back.

That's my 2 pennies @Elliot Kirk  I'm outa here.

[#12]

Let me back to Internet Explorer, we had something call Accelerator and let say in one website I select address and use accelerator and it shows location in Bing map. I select non-English text and it translate it to English, so I don't have to copy them and open new tab and then go to translator, map or other website and paste it to see it.

Cortana in Microsoft Edge did similar task but it has some limitation. What I want is smart way and let say when I click on address, it shows location inside current page and not new tab , same for weather, translators and so on. It is also good to search name and something in Bing.

Therefore, we need this Cortana feature but with more flexibility of view , so user could select if show it in right side or in mini windows like accelerators in Internet Explorer.

[#13]

It seems the discussion is irrelevant: https://mspoweruser.com/microsoft-may-end-up-rebranding-cortana-as-microsoft-365-assistant/

"Ask Microsoft 365 assistant"? Someone needs to get fired.

[#14]

Yeah. Cortana is becoming useless for normal consumers. lots of its features are removed. 99% of times Cortana fails to do what I want her to do.

[#15]

I did send a feedback suggestion regarding Cortana. I'll repost it here so the community can expand and get creative on ways to integrate Cortana within the new Edge!

TAILORED EDGE UI.
If I said "Hey Cortana," and she popped out within Edge's window instead of the taskbar, that would be a convenient and clean way to involve her more while browsing the web. A small smart box that hovers within the tab itself would be great. Able to be dragged around within the Edge window, or outside its borders on top of the desktop for instance.

INTEREST SEARCH. Let's take a look at the problem first.
Sometimes, you'd be reading an article online, let's say I'm reading an article regarding Miley Cyrus on Buzzfeed.com (like the majority of Millennials and Gen Zs). When I finished reading the article, intriguing questions might pop in my head. Things like "What happened next?", "What happened before?", "I wonder if she recovered from that injury."
Today, you'd open a new tab and search for Miley Cyrus and include other keywords regarding the injury (she didn't have an injury, it's just an example). And try to sort things out and find out what happened next or before.

How can Cortana solve this? That's where Interest Search comes in.
Imagine instead of opening a new tab, you'd say "Hey Cortana, what's up with Miley?"... imagine the possibilities. I'll explain in detail below, but here's the basic explanation first;

Cortana would lookup the web using Bing technologies for articles similar to the one you're reading when gave the voice command. It will pull up the results in a clean side bar (remember the UI I mentioned above? It would expand vertically into a "Smart Card") And give you the results it found on multiple sources, sorting by relevance and Date-Match*

*It looks at the date of the published article and tries to find articles published within the same date window.

How can Cortana find similar articles? How Interest Search works?
1) Cortana would analyze the title of the article you're reading for keywords. E.g. (Miley Cyrus injured her leg in a skiing accident.) Cortana would ignore words like "the" and "her" and even "a". So the keywords extracted here are Miley + Cyrus + injured + leg + skiing + accident.
2) Since most websites nowadays include who published the article and when was it published, Cortana would take note of the date and create a "Search Filter" for use later in the upcoming steps.
3) It would take results from Step 01, and find matches within the article itself.
4) When matches are found in Step 03, Cortana would look into the sentences each match-found for additional keywords. When it reaches a comma or a full stop, Cortana would stop extending the sentence. Again, ignoring words like "the". But, this time, emphasize on Capital letter words in the middle of sentences (like names and places), creating another "Search Filter" for use in addition to the date in Step 02.
5) Cortana would have sentences of keywords after Step 04. Cortana would then try to find similarities within those sentences by using an actual dictionary. If two words from two different sentences have common synonyms (the same exact meaning), then Cortana would pick one of them and drop the Other Keyword (referred later as Dropped Keywords).
6) After Step 05, Cortana would now have "Search Keywords". She would take those "Search Keywords" and combine them Once with the "Search Filter" it created in Step 04. (E.g. Miley +Cyrus + Paris + France + skiing + injury). This is referred to as "Interest Phrase"
7) Cortana would take the "Dropped Keywords" and create another "Interest Phrase". Now, Cortana has TWO "Interest Phrases"
8) Cortana would apply the "Search Filter" it created in Step 02 (the Publishing Date) and consider your voice command to both "Interest Phrases". (e.g. "Hey Cortana, what's up with Miley?", by noticing "What's Up" in your command, Cortana would look for results AFTER the Publishing Date from Step 02.)
9) Each "Interest Phrase" would bring up certain results. Cortana would combine duplicate results and sort the final result according to Date-Match inside its Edge UI Smart Card.

[#16]

Two feature suggestions within Edge.

SMART ASSIST.
1) SMART TABS.
When you're in the middle of using the web (say few minutes into some research you're doing) and you click a new tab. An on-device Cortana-powered page should pop up. Similar to Siri Suggestions within Safari. However, more powerful since it already suggests pages from the web similar to what you have opened in other tabs because chances are that you're looking for something similar. However, if Cortana utilizes a suggestion I already mentioned in a different Reply (Interest Search), this would be powerful! Cortana would suggest articles from multiple sources in a layout similar to the Microsoft Store homepage on Windows 10. With the "Preview" feature, you can expand an article within the "Smart Tab" before fully loading its webpage. If you liked it, click a button within the expanded UI and it would open the full website in the same "Smart Tab", if you don't. Close the expanded preview and find another result. You can ALWAYS bypass this feature by typing whatever you want in the address bar like you would normally do today in a "New Tab".

2) SMART CONVERTER
By noticing you're location (processed on device) and language, Cortana can tell if you prefer the Imperial or Metric system. While reading a recipe that is using the Metric system, Cortana can hover-in you're Edge window with a question "Would you like me to convert units to Imperial?". Since Edge is already reading the website's source code, it can change the text if it needs to. So, if the user clicked yes, Cortana would then convert all Metric units on the website within the tab to Imperial.

INFO CARDS.
When reading online, if you highlight text, a small Cortana icon would animate-in next to it. If you click it, Cortana would expand vertically an Info Card experience on the right side (Latin language websites, on the left for right-to-left language website like Arabic).

The Info Card would be powered by Bing technologies to find relevant information (mostly from Wikipedia like Google cards), pictures, and maps if available. The Info Card would be expandable and resizable and can be dragged around, even outside Edge's window if possible. The "Experience" would allow the user to "read more" of the Wikipedia article within the Info Card itself (or click a button to read in a new tab from source). Same thing with maps and media content found.

[#17]

I loved the Ask Cortana feature. I mostly used it to quickly find the definition of a word I didn't know, which was quite often. I also used it for just about everything. If I needed to find out where a referenced location was, I would use "Ask Cortana". The ability to pin it to the side of the page was also very useful. The only thing I would change about it is the look. It would be great to have dark mode added, and it would also be nice if it looked more like Windows Search web results. That would make it more consistent and sleek. I'm really excited to see this feature in the future. The "search the for..." feature just isn't as convenient. 

[#18]

@Elliot Kirk 

The Cortana feature was very useful in the original Edge. It came in very useful when researching movies or something when Cortana would offer to bring up a page to view it, and it was very useful for online shopping. Times when you see the message to the right of the address bar were decently rare, but it was very useful the times that you did see it. It's hard to mention all the times I could use that feature. It was just very handy and very useful.

[#19]

@Elliot Kirk 

Ask Cortana wasn't the only Cortana feature that was great about Edge--it was the fact that Cortana could pop up in the URL bar and help with other things too. It was super useful to see her provide better deals when doing online shopping or providing more news articles, and I'd like to see that be reimplemented along with Ask Cortana.

Better yet, I'd like to see that feature be expanded to other features. I don't know exactly how she could be implemented, but I think a potentially great idea would be if Cortana could automatically suggest or create Collections based on sites you visit--the shopping feature, for example, could be expanded to not only suggest better deals, but to group them all in a Collection automatically so you can view them later. The same feature could be used for her suggestion of similar news articles. Collections could be a wonderful tool for expanding Cortana's abilities within Edge.

Also, one of the features I like most about Cortana was that she appeared in a sidebar that didn't take you out of your workflow. This was great about the Edge experience overall, and I'd love to see that implemented back in with Favorites, History, and Downloads.

[#20]

@martinnn 

Very well said. I couldn't really think of all the ways Cortana in Edge helped, but you explained them well. I agree with what you said completely.

[#21]

@Elliot Kirk a pop-up menu like on Edge on mobile phone would be extremely useful!