Cortana on PC and Windows Phone; Working Together

Cortana On PC and Windows Phone; Working Together

My Thoughts

Jason L Ward

Originally Sent Wed 8/13/14

Microsoft

It has been pondered in what ways the Cortana experience can be enhanced when the phone and PC come together?” This  outline expresses what has come to my mind as I have given the subject of  “what magic would happen” when the Windows Phone and the PC come together”.

First as we know the experiences of Cortana on the phone are inherently different from what one would look for on a PC or tablet.

There are different needs, limitations and advantages based on form factors and other characteristics.

Smartphone

1. Small and highly mobile and always with user.
2. Always connected via cell-phone network data connection or Wi-Fi
3. Used primarily for communication (texting, messaging, voice)
4. Used for Web-Surfing
5. Used for Gaming

Cortana on a Phone

1. Due to the highly mobile nature and the consistent presence on a user’s person of a smartphone Cortana’s ideal functions in that context are for reminders (Location, People and Time Based) and location based recommendations/suggestions (based on interests and habits) and searches (either web-based or search-type functions specific to specific apps).
2. She is great for dictation of messages.
3. She is great at offering suggestions for web-sites to visit based on interests.
4. As more third party developers embrace her I’m sure we will see more integration with Games, Messaging, Social, Productivity and other types of apps. Insteon for home automation is a great implementation.

PC/Tablet

1. Desktop(Not mobile) Laptop/Tablet(Less mobile Than Smartphone)
2. Limited Connectivity (Likely Wi-Fi Dependent in most cases
3. Desktops/Laptops

1. Desktops/Laptops : Used Primarily for Productivity Tasks
   1. Content Creation
   2. File Management
   3. Web-Based Search/Research (Web Searches/Research is primarily for Personal Purposes, Education and Work – finally Leisure/Pleasure)
   4.  Music
   5.  Entertainment (Videos/Movies), Gaming
      Tablets
      Used Primarily as Leisure tools (Content Consumption, Web surfing (Primarily for Pleasure/Leisure),
      Gaming, Videos,
      Music
      Limited productivity due to form factor/and market positioning.

Cortana on the Smartphone gets to KNOW a user via the Notebook
She Learns/and is given:

Interests:
Habits(Whereabouts): Using GPS
Can learn via Web Search’s
Facebook
Monitors Email

Again, because of the Windows Phones form factor and constant presence Cortana is great for reminders, notifications, recommendations, searching when out and about and moving around. On Windows Phone she is great at assisting a user based on location and time of day, and providing information based on interests.

Cortana on A PC (Desktop)

Should also employ the NOTEBOOK function to ensure that she gets to KNOW the user within other contexts in which she is now serving that user.
The focus of what Cortana knows or learns about the user in the context of a desktop however would be different from what she knows about the user on the Windows Phone. However this information would be added to Cortana’s knowledge of the user via the Notebook.
Whereas Cortana may sporadically proactively interject into the users activities or require direct activation by the user in the diverse and dynamic mobile Windows Phone environment her “presence” or function in the static task oriented PC context can be more intensive, continuous and collaborative.
Because the desktop environment is static and task oriented Cortana should know in REAL-TIME WHAT the user is DOING. She should be actively monitoring the task being conducted so that she can offer assistance when needed. (This level of “perception” of course would be optional and controlled via the Notebook)
Example: Cortana could be integrated into MS Word (and other desktop apps) and able to monitor in real-time what is being written. Such as if a user is writing a paper on the Civil War.

Via her interface which could be a “Windowed” card-based system akin to that present on the phone, she can offer relevant data (culled from the web or the users on data on the PC) proactively and unobtrusively in relation to what is being typed.

For instance if the last sentence typed was regarding President Abraham Lincoln and his signing of the Emancipation Proclamation, Cortana can offer in a card based format a brief synopsis about Lincoln and the effects of that proclamation. (This is just a single example. There are many other applications as one considers Cortana’s integration in various apps and the Windows OS in the desktop.

Listening Mode: Cortana can also be in a constant Listening Mode, so that as the user types and works on some task she can be directed to conduct a specific search, or via a verbal “Show Me More” command, provide more comprehensive information about what she proactively presented via the card in the previous example given above.
(The option for text based interaction as implemented on Windows Phone could also be present in the Desktop version)
Because she will be always assisting(watching) She can be instructed to conduct a supplemental task associated with a task that the user is currently performing and understand from context what is being requested.

For example: If reading an article about Robin Williams in Internet Explorer, the user can say, “Cortana, find another article on this topic and launch in another window.” Cortana can do so while the user still reading.

Or the user can say to Cortana (because she’ll be integrated in the app,

“Cortana, Copy the title and first three paragraphs and paste them into a Word document.”  Cortana will then execute this command. This is not only highly efficient but a powerful implementation for the disabled.

8.       Cortana can also begin to learn you’re writing style: If there is a long pause after writing, atypical of what she has “perceived” to be a user’s typical pause between writing she can deduce that the user has hit a wall or has writers block.

She can then, having been “watching” all along, and in the background (real-time) been curating relevant data from the web as the user typed (remember experience of Cortana on PC can be highly collaborative), the users own notes in other documents or media located in OneDrive and/or the Local PC), inquire if the user needs assistance having prepared a host of data related to the last topic/subject in the paragraph.

Ex: Cortana: Jason, can I be of assistance?

Me: Yes, I’m stuck.

Cortana: Ok, I found this in

·         Your documents: “Outline of the Civil War”

·         These websites that might help. (2-3 Links to Articles)

·         And this Video on OneDrive

·         I would also like to remind you that you visited the Civil War Museum last week. There may be information you learned there that can help.(This information would have been pulled from her location services utilized on Windows Phone)

Additionally: If Cortana is integrated with Kinect for Windows or a webcam, she can “see” a user’s inactivity as another prompt to offer assistance.

Cortana on PC and Phone
How Cortana can Work Across Platforms
Because the Cortana on the PC will be the SAME Cortana on Windows Phone (Exclusivity adds value to ecosystem) she will be accessing the same notebook and will know the user within various contexts and across various tools.
 

Example of Assisting Via Phone Based on Previous PC Activity:
Location Notification/Assist: the user who was writing the paper on the Civil War may now be driving through town. Via knowing the users location Cortana on Windows Phone will “realize” that the user has just driven past a key Civil War location or monument and proactively offer this information because she KNOW’s it is relevant to the user based on her recent collaboration and assisting of the user on the same subject on the PC.

Location Notification/Assist: A user may have been researching Natural Foods and a Healthy Diet on his/her PC/Tablet, so now while out and about they may come within proximity of a Health Food Store and Cortana on the Windows Phone can notify the user of this store.

Following the above example: A user may be researching Natural foods and a healthy diet on the PC. Cortana in collaboration notices that there is a Famers Market the coming weekend. Cortana can then alert the user on the PC about the Famers Market and ask if they would like her to add it to their calendar.

People Notification/Assist: A user may have been researching something about taxes on their PC.
Sometime later the user receives a call from a friend who is a CPA.  Cortana “sees” this job title in contacts and proactively suggests asking the caller for assistance utilizing the reminder format she currently does with People Based Reminders.

Scenario Two: Cortana on the PC can proactively suggest asking the friend who is a CPA for assistance while the user is at the PC.

Application Integration: Returning to our example of the user writing about the Civil War: If the user has a TV Listing type app such as the Comcast app, or if Cortana by default can be tied via the internet to TV listings for various services (Comcast/Dish Network/etc.)  she can then proactively suggest watching a program that may be scheduled to air on the history Channel related to the Civil War. This alert can appear on Desktop, Tablet or phone as a suggestion by Cortana.
Ex: Cortana: Jason, A History of the Civil War is scheduled or air 9pm tonight. Would you like me to add that to your schedule?
Me: Yes
Cortana: Ok. Got it! “History of the Civil War” has been added to your schedule.

Music: Naturally Cortana on PC should be able to play whatever music one has on their PC via a voice command. However she could also be able to play a song online per request that is not on a user’s device.
Because Cortana is the World’s First PERSONAL Digital Assistant, as she learns favorite places, and inner circle contacts, she can also learn FAVORITE MUSIC.
Via integration with music apps such Xbox Music (and others on Phone/PC) as a user plays music on any device she should learn what specific songs, as well as specific genre or artist the user prefers. This information would be culled from usage of the Music Apps on Windows Phone AND PC/Tablet creating a central knowledge base for the Notebook from multiple sources.
When a user is at the PC they can tell Cortana, “Play may favorite album.” Cortana will then begin to play the album.
If Cortana “sees” that the user is beginning to shut down or leave the PC (the user may begin closing apps, or shutting down stopping/pausing the music, or via webcam/Kinect for Windows she may “see” the person leaving the room, or if the user picks up the phone and Cortana “perceives” this action via the accelerometer and gyroscope) she can offer to continue playing the music via the Windows Phone. Thus providing continuity across devices.

3.       Cortana should also be able to move files via voice between folders. Since both the PC and Windows Phone have independent file systems BUT share OneDrive a user should be able to navigate files ON the devices via voice and BETWEEN the devices with ONEDRIVE as the mediator (or conduit) via Voice if desired.

Ex: A user may be working on a file on their Windows Phone while on the subway/bus after work.

User: “Cortana, Transfer this file to Folder Expense Reports (a folder on the PC) when I get home.

Cortana: Ok, I will transfer file, August Expenses, to expense reports when you get home.

(When the user gets home the location based event is activated (Cortana of course is already working in the background on the PC) Cortana transfers the file to OneDrive first in order to transfer it to the local folder on the PC, Expense Reports.

4.       Cortana should also be able to CONTINUE a document on one device that has been started on another device either by proactively offering to do so and/or by command.

1.       Ex: If a user starts a document on the subway/bus he can say,

User: “Cortana, let’s* work on this on my PC when I (we) get home (or at 6pm, or 2 hours after I get home or after dinner) (she can learn ones dinner schedule).

At the designated time Cortana will open the document on the PC (which can be in a different room) and remind the user via the phone that the document is open and ready to be worked on.

2.       This can work both ways PC to Phone or Tablet. Ex: If a user begins a document at home and has a long commute via bus/train,

User: “Cortana, let’s* start working on this document ten minutes after my commute tomorrow.”

Minutes into the train ride Cortana Alerts the User:

Cortana: Jason, our* document is ready; would you like to begin working on it now? (Can even have snooze option)     (Using words like let’s and our give a more personal and collaborative interaction)

5.       Cortana on PC should be able to transfer folders locally on the PC via voice, as well as perform other desktop functions such as opening and closing apps, selecting, copying and pasting text, shutting down the system etc.

6.       Cortana should be able to manage communication methods between Windows Phone and PC/Tablet.

When Windows Phone is in proximity of the PC or Tablet Cortana should offer (or have a default setting where she can always be permitted) to hand off calls, texting, messaging to the other device.

Temporary/Transient Interests for Active Support by Cortana on Temporary Projects

Cortana on PC/Tablet/Windows Phone should be able to assign/designate within the single Notebook temporary/transient interests based on a user’s recent activity on any device so that she can actively support the user by proactively offering information within all contexts (Static- PC or Mobile – Windows Phone/Tablet) to the user from all source to which she is connected:

·         The Internet

·         Apps

·         OneDrive

·         Windows Phone Memory

·         Local PC/Tablet

This category of interests would not be the permanent interests of the user which may be “learned” by Cortana over a longer period of activity surrounding that interest.

The temporary/transient interest category would be utilized by Cortana to cull information from all sources to which she is connected (Web, OneDrive, Apps, Local PC/Tablet/Windows Phone) to serve the user’s needs for a temporary task that she is collaborating with the user to complete.

Examples of a Temporary Task with an interest which has a limited lifespan:

·         A Project

·         A Term Paper/Presentation

·         Planning For a Family Trip Vacation

·         Planning an Event (Birthday Party/Wedding)

Cortana would proactively offer information due to the user’s activity on a subject on PC/Tablet and Windows Phone based on the user’s location, interaction with others (people based), TV listings, community event and a myriad host of other information she culls from diverse sources.

Learning Temporary/Transient Interests

Whatever measure is used now that allows Cortana to learn a user’s  permanent interests, could be applied using a shorter window and some other measure such as length of time researching a topic, writing on a topic, time spent in a location (Art Museum) etc., to prompt Cortana to ask if that topic should be added to temporary/transient interests.

To bypass a prompt. There can also be a default a setting to allow Cortana to add all activities (based on these and other parameters) for which an interest has been perceived, to Temporary/Transient Interests.

Interests would remain within this category only for as long as Cortana perceives the users activity (writing, researching, traveling to a location related to the subject etc.) in that subject or topic.

A designated period of inactive time in that activity could be set by default in the system (or options i.e. 3 days, 1 week, 2 weeks could be available to the user) after which Cortana removes that interest from the Temporary/Transient interest category having perceived her assistance is no longer required in that area because the task is complete.

_______________________________________________________________________________________

Thank you for your time. I know that the teams are working very hard on Cortana for phone and PC and realize that I am offering a user’s perspective without the benefit of knowing the full status of the progress the teams have made and internal ideas presented and implemented. So I offer these ideas fully respecting the hard work you and all others are engaged in.

I believe what I have shared gives PC users of ALL mobile ecosystems a great and valuable experience on the PC while providing an enhanced experience for users of Windows Phones. Thank you for your time and I look forward to talking to you more about these and other thoughts that are at the precipice of my mind.

Additional Suggestion Cortana and Voice Recognition