Microsoft has officially launched Copilot Vision on Windows, bringing a new level of AI-powered assistance and visual understanding to users. This feature, now available in the US for Windows 10 and 11, allows Copilot to "see" what's on your screen and provide contextual help, acting as a second set of eyes to analyze content, provide insights, and answer questions in real-time.

Copilot Vision represents a significant step forward in Microsoft's journey to make Copilot an everyday companion. It's designed to work with you, understand what you're seeing, and help you get things done more efficiently. This launch positions Microsoft directly against competitors like Google's Gemini Live and Apple Intelligence, marking a move towards proactive, ambient, and deeply integrated AI assistants.

How Copilot Vision Works

Copilot Vision gives the AI access to whatever you're currently doing on your PC. Whether you're browsing the web, editing a document, watching a video, or working in Excel, Copilot can offer assistance based on the screen content. Users can share up to two apps simultaneously, allowing Copilot to connect information between different programs.

To activate Copilot Vision, users need to open the Copilot app and click the glasses icon in the composer. They can then select the browser window or app they want to share. Microsoft emphasizes that this is a fully opt-in experience, putting users in control of their privacy. Copilot can only see what you actively choose to show it, and you can stop sharing at any time by pressing "Stop" or "X" in the composer.

Key Features and Benefits

• Real-time Assistance: Copilot Vision can analyze content, help you when you're lost, provide insights, and answer your questions as you go. Whether you're browsing, working, or deep in a project, Copilot Vision offers instant insights and answers, keeping your flow smooth and effortless.
• Highlights: This standout feature goes beyond basic screen reading. By asking Copilot "show me how" for any task, it will highlight where to click and what steps to follow within your apps. This hands-on approach aims to streamline user experience, allowing users to complete tasks more efficiently without needing to search for external tutorials.
• Multi-App Navigation: Copilot Vision can navigate multiple apps at once, serving as a guide when you want it to. By sharing two apps at a time, Copilot can travel with you as you work, gaining more context and connecting dots between different apps.
• Versatile Support: Copilot Vision can assist in various scenarios, such as providing tips while playing a game, suggesting photo enhancements, or reviewing travel itineraries.

Accessibility Features

Microsoft emphasizes that Copilot Vision can improve accessibility across different platforms. For individuals with visual impairments, Copilot can convert text to speech and generate text that can be read by a screen reader. For those with hearing impairments, it can transcribe audio and video content, adding captions and subtitles. Copilot also supports voice commands and dictation, providing an alternative to traditional input methods for those with mobility impairments. Furthermore, Copilot simplifies complex language and offers personalized learning support for individuals with cognitive or learning disabilities.

Use Cases

Copilot Vision can be used in many situations:

• Summarizing documents and web pages
• Explaining error messages
• Drafting email replies
• Learning new software
• Getting real-time tips while gaming
• Analyzing photos and suggesting improvements
• Reviewing travel itineraries

Availability

Copilot Vision on Windows is currently available in the US for Windows 10 and Windows 11 users. Microsoft plans to expand its availability to more non-European countries soon. It is part of Copilot Labs, where Microsoft continues to refine and enhance the experience.

Security and Privacy Considerations

Microsoft has taken a careful approach to privacy with Copilot Vision. The feature requires explicit permission for each app you want to share, and it only sees what you actively choose to show it. This opt-in design is a response to past controversies with features like Recall, ensuring users have control over their data.

Future Development

Microsoft is committed to refining and expanding Copilot Vision's capabilities, with new features and functionality on the horizon. As Copilot Vision evolves, it promises to become an increasingly valuable tool for Windows users, offering a new era of AI-powered assistance and visual understanding.