Projects – Page 3 – Gaurav Trivedi

Towards Making Kivy Apps Accessible, Part – 2

In this second part of my post of on making kivy apps accessible I would like to describe some of the existing libraries and APIs we can model our accessibility features upon. Last week we identified that the main obstacle for creating accessible kivy apps is a missing module that could communicate the widget states to screen-readers. I explored other frameworks that may have tried to solve these problems before us and will discus one such project in particular.

Kivy includes pygame as one of the supported window providers. While looking for accessible apps in Python I found a GUI engine for pygame called OcempGUI. As a part of this project they also worked upon an accessibility module named Papi:

Papi, the Python Accessibility Programming Interface, is a Python wrapper around the GNOME ATK toolkit. It allows a developer to make python objects and applications easily accessibility aware without the need to install PyGTK and the GNOME accessibility components. Instead it only depends on ATK and – on the developers behalf – the ATK/AT-SPI bridge shipped with AT-SPI.

There is also some support for Microsoft Active Accessibility (MSAA) on Microsoft Windows.

Papi is not limited to the apps you can build with OceampGUI graphical user interfaces but can help support accessibility for any python object. Here’s an example accessible app using Papi:

I created a gist as I couldn’t find a way to embed a Sourceforge file but you can find the complete project there. You can also read more about Papi and OcempGUI on their website. Unfortunately, the project looks like that it is no longer in active development. It’s last release was in 2008.

Assuming that we can build upon this module we’d be able to support accessible Kivy apps on Windows and Linux. We are still left with taking care of OS X Accessibility and also on the mobile devices.

MacOS has a very good accessibility support but only when you are writing Cocoa or Carbon apps. In order to provide a similar level of accessibility support you would likely need to hack around a bit. With inputs from our folks at #macdev on freenode, I set out to do just that. They suggested that I could subclass NSApplication and implement the NSAccessibility protocol within the Kivy app. This would involve creating an hierarchy of fake UI objects that provide accessibility implementations as on a Cocoa app. I did make some progress with by using our in-house project — PyObjus to access AppKit frameworks and subclass objective-c classes. But the situation became a little to overwhelming for me to handle within this one week and I haven’t succeeded in creating a working proof of concept as yet.

Fortunately, Apple folks have recently launched a new API starting OS X 10.10 that includes a NSAccesibilityElement class. I am hoping that this would help avoid creating fake UI objects to implement their accessibility protocols. Here are a couple of examples demonstrating that but I haven’t tried them out yet. Need to get access to their beta versions first. You can also watch their WWDC 2014 session videos on Accessibility on OS X describing it.

We are still left with discussing about the mobile platforms. Here are links to the relevant docs pages for Android (http://developer.android.com/guide/topics/ui/accessibility/apps.html#custom-views) and iOS (https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/iPhoneAccessibility/Accessibility_on_iPhone/Accessibility_on_iPhone.html). I would like use these bookmarks when I dive deeper into exploring mobile accessibility APIs. However this week, I plan to go back to my original plan and continue the Plyer work from where I left off.

Towards Making Kivy Apps Accessible

After last week‘s discussion with tshirtman about making making Kivy apps more accessible, I spent this week exploring the accessibility options on the devices I own. The idea was to investigate and plan a set of common features that we could implement for Kivy.

I ended up playing with all the settings I that could find under the accessibility menus on an Android phone and a Macbook. I will soon repeat this exercise with an iPhone as well.

In comparison to my Galaxy running Android 4.4.2, these features were easier to use and more responsive to interact with on my Mac 10.9. Even the tutorials were simpler to follow and avoided statements like:

“When this feature is on, the response time could be slowed down in Phone, Calculator and other apps.”

I will keep my rantings aside for now and would instead like to focus on how we could support these features within Kivy apps. Following Apple’s way of organizing these features, you can group accessibility features into three broad categories:

Here's TalkBack on Android reading out my PIN to me! — TalkBack on Android reading out my PIN to me!

Seeing: This includes display settings such as colors and contrast, making text larger and zooming the screen. A lot of these settings come free of cost to the app developers. However, an important feature in this category includes the screen reader app (Example: VoiceOver and TalkBack). This is the main area of interest to us as the onus of making this feature work correctly rests with us and the app developers. I will get back to this after we finish classifying the features.
Hearing: We must make sure all our alert widgets in Kivy are accompanied by visual cues (such as a screen flash) along with audio indicators. Another feature that might be of interest to us would be to ensure that we provide a subtitles option along in our video player widget.
Interacting: It includes features like the assistant menu on Android to provide easy access to a limited number of commonly accessed functions. Most of these features are again managed and controlled by the operating systems themselves. Apart from a few design choices to make sure that app doesn’t interfere with their functions, there’s nothing that the app developers have to do to support them.

So the most important missing piece in the Kivy framework is to provide support to the screen-reading apps. These apps are designed to provide spoken feedback to what the users select (touch or point) and activate. For example if I hover over a menu-item or select a button on the screen, the app must describe what it is and also say what is does, if possible.

While settings such as zoom, larger font-sizes etc. are already taken care of by the frameworks that Kivy builds upon, we must provide explicit support for the screen readers. Here’s an example of an hello world kivy app and how it is seen by the screen-readers. There is nothing apart from the main windows buttons, i.e. “Close”, “Minimize” and “Zoom” that is available to the VoiceOver program to describe:

The main challenge here is that none of the Kivy widgets are selectable at this point. Not only that makes it difficult to use them with screen-readers but it is also not possible to navigate them using a keyboard. We must provide a screen-reader support along with our feature on providing focus support for Kivy widgets.

Along with on_focus we must send a descriptive feedback that the screen-readers as a description about the widget. While most of the native OS UI elements have such content descriptors already defined, our kivy widgets lack such information. We must make an effort to make sure that common widgets such as text-inputs, select options etc. have a description consistent with the one natively provided by the platform.

So a major portion of this implementation must be dealt within Kivy itself. While we have previously adopted an approach for implementing the lowest common set of features in other Plyer facades, it would be a good idea to implement the content descriptors (or accessibility attributes) in a way that it covers all our target platforms. Plyer could then take on from there to make the the platform specific calls to support the native screen reading apps. We would need to provide a mechanism to transform these attributes into a form relevant on these platforms separately. We could safely ignore the extra set of attributes at this point.

In this post I have only outlined the first steps that we need to take for making Kivy apps more accessible. I will work on figuring out the finer details about implementing them in the coming weeks. Hopefully this would help in triggering off a meaningful discussion within the community and we can have a chance to listen to other opinions on this as well.

This is a two part post. You can continue reading on the topic here — Towards Making Kivy Apps Accessible – 2.

Kivy Idle Week

Bionoid has graciously agreed to ship an iOS development device to my place (Our Kivy community is awesome!). And while I was waiting for it, I had taken a break this week to attend a couple of interesting lectures happening in the city. Although I have been closely sticking with the plan otherwise, I will put in extra effort when that reaches me to cover up for this.

Also, I have decided to work on facades interfacing with accessibility systems on the various platforms. I didn’t have that as a part of my original summer plan but it seems like an interesting idea as suggested by tshirtman. I will investigate the common features and work on a plan to implement this.

Maintenance work in progress

Continuing my work from last week I included a PEP8 style checker in the Plyer repository. I was able to borrow most of the code from the main Kivy project. This also included a git hook which runs a script to check for style errors after every commit. I also included a make hook rule to make it easier for the contributors to set that up.

But I had to do a lot more work after checking in the code. Our script threw a whole bunch of errors for the previously checked-in code that was submitted without any checks in place. Unfortunately, I had to fix most of them by hand. The silver lining on the cloud was that I managed to memorize all the conventions and unlearn my bad habits while styling code in the process.

While I was setting up the Makefile, I also set up the rules for building the Plyer docs using Sphinx. I also took this opportunity to revise some of the docstrings to make them explain things better. Hopefully the documentation will now make more sense to the readers.

This is how my final pull request looks like with all of the above changes. I think I touched nearly every source file in the project and ended up changing 37 files with 2,417 additions and 125 deletions. I hope I haven’t introduced any silly errors while doing that though.

In other updates, I have started parallel efforts for iOS facades. I am trying to get my hands on a real device and also working on running Kivy on a simulator in the background. There has been some prior work in this direction but we don’t have a final solution for this as yet. Hopefully, I will be able to find something here in the coming weeks.

I can haz commit access and other updates

I can haz commit access In this week’s update on Kivy and Plyer, I have been granted commit access to the repository. Yay! And thanks everyone in the community for considering me responsible enough 😛

As a result I did some obligatory maintenance work and clean up the repo post my code merges. Doing that I also realized that we need a git hook to do all code style checking before any push. I think the standard Python PEP8 checker should suffice. I will work towards adding it in the coming week.

Also, I worked on a new battery status facade on Plyer. I have implemented the lowest common denominator set of features for now and we can query whether the device is connected to a power supply and its battery’s charge percentage. I also have access to some platform information which can be added as well if need. May be under extras or a specially marked category.

If you have been following my updates, you’d have noticed that I haven’t made much progress on iOS as yet. This is because we haven’t had any success running Kivy on a simulator and I don’t have access to a real device. Brousch and I have decided to get community help for any tips on this issue.

Mid-summer Progress on Kivy and Plyer

This week I worked on retrieving a Unique ID depending the the hardware your program is running on. This involves querying some platform specific unique IDs that are suitable for this purpose. For example, you could use the IMEI number on an Android device or the serial number of the machine when running on a desktop. While none of these methods are foolproof and can be spoofed, they can still be useful in primitive fingerprinting and probably even seeding a random number generator. You can check out my pull request here.

The week also marks the end of five weeks since I have started working on the Kivy project. I have been making steady progress towards improving Plyer, a cross-platform library to handle all the common platform dependent things you need while writing python applications that can run on Android / iOS / Linux / Mac or Windows.

Here are the links to my previous blog posts describing my progress each week:

I started my work with a critical bug that was blocking further progress in the first week. Although it is only recently that Inclement has been able to propose a fix for it, we were able to find a quick work-around that did not stop the development on Android platforms.

I also did some review of existing pull requests in Plyer that were pending to be merged. I also implemented the accelerometer facade on MacOS by writing a small library using ctypes to make the API calls. I then re-wrote some of the existing sensor facades on Android using the support provided by the PyJnius project. This made access to the platform specific APIs more simple and very clean. We decided against using the sl4a compatibility layer and stick with using Pyjnius for accessing Java classes on Android.

You can find a list of all the pull requests in Plyer here. In summary, these are the facades that were either added, modified or reviewed since the beginning of the project:

Platform	Android	iOS	Windows	OSX	Linux
Accelerometer	X			X	X
Camera (taking picture)	X		X	X	X
GPS	X	X
Email (open mail client)	X		X	X	X
Vibrator	X
SMS	X
Play Sound	X		X	X	X
Compass	X	X
Gyroscope	X	X
Unique ID	X		X	X	X

I also added some example applications using Kivy to demonstrate the use of these facades. And also made a point to include one with most of the new facades that I wrote.

I believe that I have made some significant progress till now but there are still some tricky facades that are left to be implemented before the end of summer. I hope to carry this momentum forward and add many more facades in the coming weeks.

Kivy Plyer Updates

This week I looked into a pending facade’s implementation on Mac and worked on some new ones as well. In other news, we have a functional #plyer channel on Freenode. It is not very active at the moment but is fully equipped with all the basic amenities (:D), ie. logging and a google-group to contact the folks offline. The idea behind making a separate channel was to reduce the noise on the main channel and also allow the relevant people to focus on project specific updates here.

We have had a camera facade in the queue for quite some time now. We wanted to model something on the lines of what is available on Android OS. It should provide a simple API to capture a picture from the default camera on the platform. We had tried two different unsatisfactory approaches previously and were on the lookout for a lighter (without many dependencies) method to get it working. I am of the view that we could do with tools in the public-domain like ImageSnap, for each platform. This is in line with the Plyer’s philosophy of trying not to re-invent the wheel and use external libraries wherever possible. I have created a pull request to get feedback on this approach and I am waiting to hear back on this from the community.

The other new facade that worked on is called “Play Sound”. As its name, it would be used to play a short clip of an audio file on command. It also relies on existing standard libraries and tools. I have made use of the Sound Pool API on android, NSSound on MacOS, WinSound on Windows and on Linux, I launch a command line player (play or aplay) for now. I may update that in future if I can find a better method to achieve that. Here’s a link to the PR.

I think slowly and steadily I have been reaching my mid-term goals and have been checking off items in my easy list. I think now would be a good time to gather some feedback on the PRs I have proposed till now. Some suggestions on them would help fix any issues with them and continue with new facades.