Google Summer of Code 2016 - Report

Firefox for Android (Fennec) is the build of Mozilla Firefox for Android smartphones and tablets. FatFennec is the code name of the ongoing project to reduce the size of the Fennec and make it use a fewer bytes. This involves the task of reducing the size of the APK file of the Firefox and reducing the size of the runtime overheads. Currently various aspects are being discussed to reduce the APK size of the Fennec. Smaller APK means faster downloads and lesser install failures. It all started with Mozilla being accepted as mentoring organization this year for the Google Summer of Code 2016. A part of this FatFennec is proposed as a project in Google Summer of Code 2016. The main aspect of the project revolves around the Meta Bug 1095719 - Download hyphenation dictionaries at runtime.

As a participant of Google Summer of Code 2015 I contributed to the community “Syslog-ng” for developing a “Monitor Application for Android”. Being a former participant of Google Summer of Code I kept myself updated about the happenings and timeline of the Google Summer of Code. This will be the last year for me to participate in Google Summer of Code as I have completed my studies and graduated on july. I waited for the day when the list of mentoring organizations was released to look for exciting opportunities. I selected a few organizations in which I thought I could contribute. As a developer who focuses on “Java” and “Android”, Firefox for Android was really a great project to contribute to. In between the announcement of mentoring organizations and proposal submission I decided to focus on one project rather than focusing on many projects / proposals (In keeping with my personal standpoint, I do not recommend the same to everybody). I decided to focus only on Mozilla and did much research to understand the project and to come up with a good proposal. My proposal can be downloaded from here

Pre-proposal

During “Pre-Proposal” stage I downloaded the Mozilla-Central codebase and started understanding the structure of the codebase. I have to admit that this codebase is one of the largest codebases I have ever seen and is made of millions of lines of code. It took me some time to get hold of it and understand the structuring of the code. Next thing I learned is about the build system - mach. Then I followed the simple yet highly effective guide. Firefox cannot be built on Windows. Fortunately I had Mac and Linux machine and mach build system is supported on both. With the help of that guide I successfully built the Android build of Firefox. I got an overall understanding as to how to build the Firefox and I started to explore further about the version control. Firefox uses Mercurial - a distributed source control management tool. Now with the code downloaded and ready, I understood the build system and version control. I decided to take this a step further and contribute something to the Fennec to complete the basic process. There were a few minor bugs listed in BugsAhoy! I decided to write a patch for something very simple. I chose Bug 1231655 - Remove GeckoApp.showButtonToast(). This bug was the tiniest of all. In this bug I removed an unused method from the build. This is my First Good Bug. This patch was accepted and later released with the Firefox 48 and I became a Mozillian. Yay!. I chose another bug as my “Good Next Bug” - Bug 1242624 - Use our color palette for toolbar and URL bar text which fixes the wrongful usage of the color code.

Having created patches for tiny bugs, I started exploring the code existing for the Downloadable Content. I had read about the Downloadable Fonts and understood that this is a similar project to that of Downloadable Fonts. The hyphenation algorithm is fundamentally the same as Knuth’s TeX algorithm. However, the implementation is a bit faster. The hyphenation files from TeX can be used directly. A hyphen, as we all know, is a puctuation mark used to join words. Gecko supports hyphens in the browser and it uses support files to display the hyphens. Those support files are referred as “Hyphenation Dictionaries”. We are currently packaging these “Hyphenation Dictionaries” along with the APK. There was a patch - Bug 1175555 which contains a flag used to remove the “Hyphenation Dictionary” from the build. I followed the steps mentioned in that bug tracker and managed to exclude the support files from the build. I created a Sample WebApp to verify the workings of the project. With the “Hyphenation Dictionary” removed from the build the Fennec build stopped showing the hyphens in the paragraphs.

Proposal and Acceptance

With this knowledge in the backdrop, I went on to write a [proposal]((http://krishnakannan.com/assets/moz-proposal.pdf). I checked and double checked the proposal and submitted and this was the only proposal I was able to prepare for the Google Summer of Code 2016. I waited till Google announced the successful proposals. I got selected for my Proposal - Download App assets at Runtime. Having been selected for the Google Summer of Code 2016 for Mozilla, I started to take up complex bugs from the Bugzilla and created patches for the same as we all progressed into the Community Bonding Period.

Community Bonding Period

When I submitted the patches for the previously mentioned bugs, I created patches using the Mercurial Export. It was alright to use the export tool for the minor bugs. But this could be very difficult to do and manage. I was told to get a Level 1 commit access to the Mozilla Central Repository and vouched by my mentor. Now after obtaining the commit access, I used Mozilla’s Mercurial Guide and other documentations to configure my machine to work with the version control system. I started working on the bugs related to the Downloadable Content. Bug 1209496 - Implementing proxy support is one of the bugs I was working on during the community bonding period. This bug is used to create proxies for the HTTP connections made by the DownloadContentService. Before this bug the HttpUrlConnection did not have any proxies and we had helper methods in ProxySelector class. I wrote code to make the connections use the proxies. This is the first bug I worked on during Community Bonding period. After the previous bug I moved on to work with Bug 1242385 - Removing unnecessary method markAsIgnored() in DownloadContentCatalog. I removed the methods and all the unnecessary constants and other unit test methods that made use of markAsIgnored(). Though the task was simple I was introduced to Try Server. This server is used to test a patch without checking the patch into the core repository - “Mozilla Central”. During this period I was also working on the Bug 1266156 which deals with the cruncher that is enabled in the gradle build file as a part of AAPT PNG Optimization. It turns out that enabling the optimization has a negative effect on the Fennec. In Fennec we already optimize the images that are included in the build. Hence while building with the Gradle which again applies AAPT Optimization increases the previously shrinked PNGs. This almost has an effect of about 100 Kilobytes in the APK size. This bug needs to be explored further.

Coding Period

The actual coding fun started during this period where I was involved in much more complex tasks of extending the existing DownloadAction to download the Hyphenation-Dictionary files and creating a scheduler to schedule the download actions by the project. The proposal and project were designed to have two major sections; one to be completed before the Mid-Term and another during the second part of the coding period.

Coding Period - Part 1 - Downloading hyphenation dictionaries at runtime

This is the first part of the project where I have extended the DownloadAction which was capable of downloading the Fonts at runtime. Now the extension has enabled the Fennec to support downloading the hyphenation-dictionaries at runtime. The first bug I worked on this period is Bug 1276577 - Use Switchboard.isInExperiment(). We use Switchboard to enable the synchronization of the downloadable content catalog for 50% of our Nightly users. In the previous implementation we used Experiment.isInExperimentLocal(). This implementation is used to decide whether the user is in experimental mode even before the first run of the application. For this purpose we will be using Switchboard a super light weight A/B Testing framework. After this I worked on one of the most important bugs for this project. Bug 1276588 - Modify DownloadAction to handle hyphenation dictionaries. For the development and testing purposes my mentor uploaded all the 36 dictionary files to a temporary location which can be found on the bug tracker. We have a static catalog of files and other informations like filename, checksum, size, type and kind. I added all the hyphenation-dictionaries to the static catalog - DownloadContentBootstrap. This catalog changes were shipped with the Bug 1276589. This catalog was in turn used as the part of Bug 1276588 where I have shipped the code to handle the files with a new kind - “Hyphenation”. This turned out to be simpler than our earlier estimation as we split the bug into two parts where we create and check for the new kind - “Hyphenation” as a part of Bug 1280769 - Add Hyphenation Dictionary as separate kind. Now with this bug I have also changed the way of checking the file that are about to be downloaded. Previously there was only one kind of file - “Fonts” but now we have multiple kind files and made necessary changes to check whether the file to be downloaded is a known content which checks for list of known kind and types.

Now I have successfully removed the “Hyphenation-Dictionary” files from the build so that it would be downloaded at runtime. Following this, we have various tests that are to be performed during the building process. One of the important tests is Reftest. It basically compares two display of two webpages and comparison is made at pixel level. If the bitmaps resulting from displaying the two files are identical then it passes. There are many reftests to be performed for Hyphenation-Dictionaries. Having removed the hypheantion-dictionary files from the build, the reftests for Hyphenation-Dictionary will check pixel wise and fail. Because without the support files in the build the Fennec will stop showing the hyphens in the webpage. If this reftests fail then no one can push the code to the central repository as passing the tests is mandatory before pushing the code. The solution is to package the reftests seperately and push it to the device. For this I have created a Bug 1276560 - Package hyphenation-dictionaries separately for reftests. In this bug I parse through the necessary directories in the source tree and collect all the necessary dictionary file - “.dic” files, then created a temporary directory and copied all the files to that temporary directory. Now after creating temporary directory and populating it with required files I have written the code to push the entire directory to the mobile device’s test profile directory.

After packing the dictionary files and pushing the same to the mobile while running the reftests, they fail again. While researching further about the problem I realized that the Gecko Rendering Engine looks for the hyphenation dictionaries in particular location and they dont look for the dictionaries in the test profile directory. I filed a Bug 1280757 - Load hyphenation dictionaries from writable location. I modified the Gecko to look for hyphenation dictionary in test profile directory. With these changes, the reftests passed despite excluding the support files from the build. This marks the completion of the first part of the project. Now I filed a bug to enable the exclusion of hyphenation in the nightly with a flag - Bug 1285752 - Enable hyphenation-dictionary download in Nightly.

Coding Period - Part 2 - Better Scheduling for downloadable contents

Currently when a person installs the Firefox, the code is designed and developed in such a way that both Fonts and Hyphenation Dictionaries are downloaded during the application start. The implementation is rudimentary that Synchronization and Verification of catalog will be performed during every application start. The aim is to schedule / reschedule the downloads in the downloadable-contents project. Various options exist for the scheduling of tasks in Android OS. AlarmManager is historically used for scheduling of jobs in Android OS. This is available from the API Level 1 in Android. This is a very primitive solution. Google released another API - JobScheduler from API Level 21. This is an advanced API. But Firefox for Android is available from API Level 15. Therefore we cannot use this solution directly. There is one more option GCMNetworkManager, which lets the Google Play Services to batch the network operations across the entire system. One problem with this solution is currently Firefox, provided by Mozilla can be downloaded and installed in any android device without having Google Play Services. Therefore we cannot have Firefox dependent on the Google Play Services. Another option is Firebase JobDispatcher which will replace the GCMNetworkManager in future.

There are also a few third-party libraries like Android-Job by Evernote Corporation, Android Priority Job Queue originally developed by Path Inc and Trigger. After much researching, experimentation and discussion we decided to go with the Android-Job instead of creating a new library ourselves.

For this purpose I was assigned the bug - Bug 1248901 - Create helper class for scheduling background tasks with best available system API. I created a few bugs and made this bug depend on those bugs to split the task into tiny pieces. As a first step I created a bug - Bug 1291180 - Add Android-Job library in the build. In this bug I have added the library to the Fennec Build. Adding a library in the Fennec is not a straight forward job. For android we have multiple build systems. Gradle and Mach. Therefore I have added the source code to the Fennec’s third-party library directory. The Android-Job transitively depends on a Logging library and Google Play Services. So as I have included the Logging library’s source code also to the third party library. In future we might be using a stripped down version of the Android-Job reducing it to absolute essentials. I have changed the moz.build to accomodate the newly introduced library and its dependencies for the mach based builds. I have also changed the app’s gradle and third-party gradle to compile the library in gradle based builds. Now we have a stable version of the library in our Fennec’s build.

Once after this Bug 1291180, I went on implementing the JobCreator class and created a new JobManager in the Application. The jobs will be created in JobCreator implementations. This is done in the bug - Bug 1291360 - Add a JobCreator class for Downloadable-Content. As a part of this bug, I have changed the mach build files and converted the DownloadContentCatalog as a Singleton to make sure that there exists only one instance at a given time. This bug is merely a basic implementation of JobCreator. This JobCreator will be further used in the future bugs to create individual jobs.

After working on the previous bugs I picked a new bug. Bug 1257492 - DLC Sync: Better scheduling of sync action. Here as a first step of migrating the downloadable content to use the Android-Job. SyncAction is the place where the entire process begins. It calls the StudyAction, CleanupAction. This is called during the start of the application every time at BrowserApp. Instead of initiating the entire process during every cold start we will now schedule the SyncAction as a periodic job. Then VerifyAction will be called only after the execution of SyncAction instead of calling it during app starts. Now we have successfully scheduled the Synchronization process in the downloadable content. I am also working on Bug 1209498 - DownloadContentService: Smarter rescheduling of failed downloads. In this I will be migrating the DownloadAction from the present state to job after which it can be scheduled and rescheduled.

List of Bugs

Many thanks to mentor - Sebastian Kaspari (:sebastian) who guided, encouraged and taught me all along this summer. No matter how busy he is, he always clears my doubts immediately. Without his guidance it would have been much more difficult. I also thank other Mozilla Engineers who reviewed my code submissions.

Future Works

Some of the identified areas in which I will be working in the future are

• Smartly reschedule the failed downloads [WIP]
• Uploading the Application assets to Kinto and make Fennec to download from Kinto
• Download localization files at runtime

Apart from this I will continue to look for opportunities to contribute to this awesome community.

I thank the entire community and Mozilla for giving me the opportunity. This is the most productive summer I have ever had.

Once a Mozillian, always a Mozillian