0xlab

Hacking at 0xlab: 簡介 0xlab 新的開放原始碼專案

noreply@blogger.com (jserv) — Wed, 01 Sep 2010 10:17:59 +0000

我們在今年八月中旬的 COSCUP 研討會中，很榮幸地向與會的朋友介紹 0xlab 若干新的開放原始碼專案，本文稍作整理回顧。0xdroid 是 0xlab 最早的一項整合性開放原始碼專案，目的是建構於 Beagleboard 開放的硬體之上，提供 Android 的參考實做環境，經過一年多的累積，裡頭的成果被帶到不同的移動裝置中，比方說手機與 Tablet。今年二月份，有幸在 Mobile World Congress 2010 展示基於 Android 的移動裝置，Tieto 也在 TI OMAP3 平台展示 0xdroid，這些都讓我們更有信心，擴展到多元的應用。

這半年間，延續 0xdroid 的經驗，我們發展 / 貢獻以下專案：

0xRobocat : 與 CATCAN 合作的專案，目的是打造一組控制伺服馬達與相關零組件的軟體框架，可提供給 Android 或 GNU/Linux 使用，以 Apache License 2.0 授權釋出。除了用於打造機器人外 (是真的可走動的 "Android")，本專案可用於 Android 裝置 (如手機) 的自動測試
0xBench : 有鑑於 Android 上的系統效能評比程式，不是不完整，就是非開放原始碼，所以我們決定重新打造一個。0xBench 可從 C 函式庫/系統呼叫的層面，一路從 Dalvik VM 評測到 Android framework，仍持續擴充，並且我們提供 Web service，允許貢獻評測的結果，用互動的方式來分析。程式主體是 Apache License 2.0 授權
android-toolchain : 0xlab 特製的 GNU Toolchain，以 Android team 的成果為基礎，整合來自 Linaro Toolchain Working Group 的改善。這些主要是 GNU GPLv3 授權
Android Open Source Project (AOSP) : 提交 0xlab 過去在 TI OMAP 與 Qualcomm 72xx 平台的一些修改，經過公開的 Review 機制，慢慢整合到 AOSP
CyanogenMod : 社群版本的 Android，在 Nexus One, HTC Dream 等裝置有相當優越的表現，收錄若干來自 0xlab 的改良，比方說 Bluetooth HID profile

除了以上的開放原始碼專案，我們內部正進行一個基於 Android 的新型態應用，未來也會將相關的專案程式碼，以開放原始碼的形式釋出，完整的列表可參考 0xlab Projects。至於 AOSP，我們現有更改的項目是：

build - 調整 Android build system、修正工具程式的瑕疵
bionic - 提供 ARM optimizations，針對 ARMv6/ARMv7
PixelFlinger - 基本的 refactoring，以及 ARM 實做的改良
libpng / libz - 利用 GCC visibility，進一步縮減 code size
elfcopy - ELF 工具的調整
bionic / apriori / build - 支援 DT_GNU_HASH，加快動態連結的速度
toolchain - 加強 build system 的彈性

以上，歡迎舊雨新知給我們支持與指教，謝謝。

Kanru's 探險日誌 » 0xlab: Android Service with dynamic arguments

Tue, 24 Aug 2010 08:01:34 +0000

Recently when I was preparing some Android training material I found this little trick.

Android “init” can do a lot of things, including “on property change” trigger; one can image some applications use this to implement an ad-hoc IPC. One example is the system/libnetutil that monitors the “init.svc.dhcpcd” property to detect whether dhcpcd is running or not.

Android framework can execute some predefined command (service) via the “ctl.start” property, the most interesting part is you can supply dynamic arguments to the command just like you were calling a command from shell.

The format is

setprop ctl.start service:arg1 arg2 arg3..

It must be a oneshot service, for example you can define a dhcpcd service like:

service dhcpcd /system/bin/dhcpcd -B
    group dhcp wifi system
    disabled
    oneshot

And set the target interface dynamically:

setprop ctl.start dhcpcd:eth0

“init” will then execute “dhcpcd -B eth0″ with appropriate permissions.

Kanru's 探險日誌 » 0xlab: [非官方] COSCUP 2010 Mobile App

Fri, 13 Aug 2010 23:36:53 +0000

今年 GoogleIO 時 Android team 不斷在推廣一個 Android UI Pattern，可以觀察到最早利用這個 Pattern 的應該是預設的 Facebook 程式，接下來發展的更為完備的是 Twitter 官方程式，處處可以見到該 Pattern 的影子。GoogleIO 前 Google 也推出一個程式叫作 IOSched 用來顯示議程相關資訊，同樣是使用該 Pattern 的實作。雖然 Twitter 承諾會釋出 source code，但是一直沒下文，因此想要使用這個 UI Pattern 的可以說完全沒有範例可以參考。

今年 COSCUP 就參考 IOSched，嘗試從頭打造一個類似的程式，順便練習一下怎麼使用這個 UI Pattern，結果寫好後才知道上官大神早有 HTML5 的版本，連 IOSched 不知道什麼時候也有 source code 了 (汗)

無論如何，總算是練習了一下 Android Programming，有參加 COSCUP 的人可以試用看看

Source code 在 http://github.com/kanru/Coscup2010

Insert smart title here: Introducing 0xBench

Fri, 13 Aug 2010 12:24:00 +0000

Today (Aug 14) we are announcing the release of 0xBench, an open source Android benchmarking app developed by 0xlab.

0xBench comes with several built-in benchmarks including Linpack, Scimark2, LibMicro, … etc. 0xBench can also be extended. Developers can add their own benchmarks (either in native C or Java) to suit their needs.

0xBench is now on the Android Market. If you wish to try it on your phone, please search for “0xBench” in the Market or scan the following QR code with your phone.

In addition to 0xBench, we are also introducing a website called 0xBenchWeb, a simple web service that stores and visualizes your benchmarking results. After finish running benchmarks on your phone, you can choose to upload your results to 0xBenchWeb.

You can find binary, source code, and documentation in following links:

Asleep from Day » 0xlab: 0xlab Annual Report, 2009.

Fri, 13 Aug 2010 03:00:51 +0000

I began writing this at February, but never have the chance to put in everything I wanted to put in. Now I decided that it’s better to publish it in its current form then not doing it at all. So, here it goes. You may also download the PDF version.

So much happened since 0xlab was established in April. It’s one of the few, if not only, open source lab whose operation depends solely on the sponsorship of Taiwanese companies. This is a new and bold attempt for both the businesses and the developers. In the beginning of 2010, this report tries to give a reflect on our ideas, and our works.

Our Ideas

We believe open source is the right thing to do. Its contribution could be shared by everyone around the globe and allows people to build upon each other’s work. Plus, as a developer, coding makes much more sense if you can see the whole source. It’s something that makes our job as a software developer fun and meaningful.

On the other hand, we believe there is a chance in Taiwan to make open source profitable. We’re not talking about the conventional thinking that business should lock up as much as possible and only release source code enforced by the GPL-like license. We are talking about being a good community member and opening up as much as possible, not only the source codes, but issue trackers, documents and discussions yet still making money because of openness.

This is not a new idea, but already a practice. Companies such as Apple, Google, IBM, Intel, Novell, Redhat, Sun, etc. open up to attract more brain power and get more publicity. Contradicting conventional wisdom, openness is actually becoming a good business strategy.¹

We believe that there exists an ecosystem that combines both the open source developers and the hardware manufacturers in Taiwan. Our focus is to strengthen this connection and make it profitable for both parties.

So, how are we going to do this?

Opening up the non-essential part of the software development and working with the communities. For the open source world, this means more contribution and feedbacks. For the business, this means minimized development and maintenance effort.
Focus on the unique advantage that is a combination of hardware and software. The actual profit comes from selling the physical product instead of selling the software, which is not a financially good idea anymore.

Businesses should cooperate on #1, so they could put more resource to build their own unique advantages #2.

A healthy ecosystem is the way to make open source prosper. So, by making this ecosystem strong, we can effectively make open source stronger in Taiwan, which is something we have long waited for.

Our Works

Timeline

2009/4/27 – 0xlab launched, sponsored by AzureWave Technologies, Inc.

2009/5/15 – Project Android Eee PC launched. It aims to bring Android to Eee PC but the current focus is 3D acceleration support.

2009/6/2 – Computex. 0xlab demonstrated Beagleboard and Samsung 6410 connected to the WiFi/Bluetooth combo module from AzureWave.

2009/6/10 – FreedomHEC. Talked about "Qi – Lightweight Boot Loader Applied in Mobile and Embedded Devices".

2009/6/22 – Project android obex launched. It provides obex file sharing via bluetooth in Android.

2009/7/1 – First code drop. 0xlab now has the following projects on gitorious: MadButterfly, Android Eee PC, 0xlab’s Linux Kernel Tree, esuit and android obex.

2009/7/14 – Project 0xdroid launched. It’s the 0xlab’s Android distribution with Beagleboard as the reference platform.

2009/8/7 – The first release of 0xdroid, Beagle-cupcake-0×1.

2009/8/15 – COSCUP, the biggest open source conference in Taiwan. 0xlab is a sponsor with 4 talks and 1 lightening talk.²

2009/10/9 – Beagle-cupcake-0×2 released.

2009/11/3 – Project FreeBroid created. It enables FreeBSD users to do Android development.

2009/11/6 – Created our branch of Qi supporting omap3530 (Beagleboard).

2009/12/18 – Beagle-donut-0×3 released.

2009/12/25 – Merry Christmas! Android with 3D Effects is our Christmas gift.

Business Review

Hardware is becoming more and more standardized and keeps converging to a few solution providers. From the technical point of view, businesses have to stand out in software and the hardware/software integration.

However, the cost of creating your own proprietary OS is sky high and it requires a dominating power on the market to create enough acceptance, such as Apple. Solutions like Windows Mobile require royalty and the room to customize is rather limited because it depends on the kindness of its provider. Open source solution is a way out, but it’s not without its problems.

The Impact of Open Source

There are mainly three obstacles that are often missed by project owners:

The technical understanding of the code: most of the time the existing projects can only partially meet the requirement, so a certain amount of engineering resource is required to fill the gap. However, just because the code is opened does not mean it could be modified easily. A big, complex project can be very difficult to trace and understand. Inappropriate modification to the source code leads to maintenance hell, while the ability to change properly is rare. Without a thorough understanding, the exact effort to develop/fix/maintain the code could be hard to estimate, as well as the risk of the project.
Lack of overview: a Linux based product usually consists of multiple projects. These components may base on different technologies and work in different system levels with different functionalities. The way they interact with each other and the rule they play in the system could be hard to understand without prior knowledge of these projects. Thus, to obtain the overview of the system and to get a grip of the project status could become a challenge. The project owner must understand what’s the core advantage/business value of the project and the rest should be opened to save development effort.
Don’t know how to work with upstream: the best way to understand the source code and to get support is maintaining a good relationship with upstream. The best practice is working directly with them and merging back new functionalities and bug fixes. However, communication could be difficult because open source communities have their own development model, tool and culture, and will react rather badly if you don’t follow.
Legal issues: many businesses intentionally or unintentionally violate open source license. This puts them under the risk of being sued by people like our friend Harald Welte. These businesses need experienced people to deal with license issues, especially when you mix codes with different licenses.

How to Deal with It

Work with businesses that provide open source services. These are people who not only understand open source, but also bear the business value in mind. They have experience in commercial projects before, so they are very likely easier to communicate with.
Hire experts. If you can’t directly hire upstream developers, hire developers who are familiar with your target projects.
Train your own experts. You still need to find some ‘seeds’ first. This takes longer but in the end you get your own group.
Managers with rich experience in open source is a must.
Seek legal consult familiar with different open source licenses.

New Challenges from Android

Multilayer debugging: Android is a complex combination of various technologies. Most of its platform services are a mix of different computer languages and run across multiple layers. Debugging (inside the platform itself, not the application layer) often requires tracing back and forth between the java framework and native libraries across different processes. This is significantly harder then tracing the old fashioned pure c/c++ codes.
Follow the fast paced upstream: Android evolves really fast, i.e. two major releases per year. This means your product will become outdated during the development phase and you will have to do the catching up and the product development (polishing, bugfixes, new features, etc.) at the same time. This will be a new challenge to a software team.
Knowledge of various technical areas: as mentioned before, Android is a complex mix of different technologies. Peripherals, power management, mobile phone protocols (2G/3G/…), graphic 2D/3D acceleration, virtual machine (Dalvik), toolchain (bionic, gcc, …), the design of user interface, certifications, etc. , each one of these could be a big topic. Android provides a basic platform to build on, but the entry barrier to build on top of it properly is still high, if not even higher.

Where We Stand

0xlab has an unique standing facing all all these new challenges. Most of us are both professional engineers experienced in shipping end products and active open source developers.

We consist of experts in different areas. Experience in major smartphone brand/ODM R&D, User interface, virtual machine, compiler, low-level kernel development, multimedia codecs, Mesa 3D contributor and Openmoko, the world’s first open phone, gives us everything we need to do Android platform development.

We offer the following:

Reduced development effort by working with the communities. Short time-to-market. Automatic testing suite to ensure quality.
Contribute back to the industry to make a bigger pie.
Cutting edge technology by following upstream development closely.
Commercial friendly license because we use the same license as Android.

Work with Us

We are accepting the following possibilities if they are technically related to Android:

Community: follow the convention. Join our mailing lists, chat on IRC, fire questions and share your knowledge just like any other open projects. Get helps for free by helping others.
Project: we’re accepting offers to work on a specific project as long as our work could be opened as well.
Consulting: contact us to get more info.

For Developers

0xlab is a virtual organization consists of full time employees and freelancers. We welcome passionate developers to join us. It doesn’t matter if you’re employed or self-employed as long as you share the same idea with us.

For Businesses

We encourage businesses to change mindset and open up non-essential parts to gain more developers’ eye balls. The more eye balls you get, the more powerful you’ll likely become. You don’t need to own every part of the software to sell the product. You just need your own unique advantage and that’s where you should really focus on.

Background

Why Android

Android is the event of the year with regards to mobile devices, which is our primary interest. Many OEM/ODM manufactures and brand names in Taiwan have put development resources into it, while the government agencies provide budgets to encourage Android research and development. We want to reuse as much resource as possible, so we picked it as our development focus.

Beagleboard

One of the problem we faced when 0xlab was just getting started is that it’s not easy to get a hardware platform. We picked Beagleboard because it’s modern, powerful, cheap and open with an active community. We do not have any business relationship with TI, but we do like their idea of opening up hardware. However, we are indeed the world’s no.1 RMA customer with the highest return rate since we broke a lot of boards while trying to do hardware modifications. We are software developers, after all…

Beagleboard is the platform we focused on, but it’s not the only platform we are interested to work on. We welcome people sending us their cool gadgets. If you want to find out what Android can do on your device, send us some sample. We may work on them and try to keep them in one piece.

More Info

We use git as our primary SCM, and use gitorious to host all of our projects. Some projects with the same name were created on google code as well for issue tracking and an easier to use wiki.

0xlab.org is hosted on our own cost and acts as an entry point. It provides basic information about us. The most interesting page is perhaps Planet, since it acts as a aggregation of our blogs.

Here is a list of all the rss feeds about our blogs and projects:

About 0xlab

0xlab is founded on 4/27, 2009 in Taiwan by a group of enthused FOSS³ developers, some of the finest. Most of its members are of Linux background with rich experience in embedded systems and created/contributed to various open source projects such as Linux kernel, opkg, OpenOCD, OpenEmbedded, QT Extended, Packagekit, FreeType, Kaffe and LXDE.

They met each other in the local FOSS community and at the startup Openmoko, from where they became familiar and connected with developers worldwide. This differentiated them in pan-Asian because they stay much closer with the mainstream and are actively attending the ongoing development.

The list of its current members could be found on our gitorious page. 0xlab aims to be a virtual organization, but at the moment it mainly operates in Taipei. 0xlab could be reached at:

Contact Us

IRC: #0xlab
mailto:contact@0xlab.org

Mailing Lists

1. http://googleblog.blogspot.com/2009/12/meaning-of-open.html

2. http://jserv.blogspot.com/2009/07/0xlab-coscup-2009.html

3. Free and open source software, http://en.wikipedia.org/wiki/Free_and_open_source_software

Filed under: 0xlab, Android

Hacking at 0xlab: 0xlab 在 COSCUP 2010 的議程分享

noreply@blogger.com (jserv) — Mon, 02 Aug 2010 03:06:02 +0000

一年一度的「COSCUP / 開源人年會」即將於八月 14 與 15 兩日舉辦，因為今年與 GNOME 亞洲峰會合辦，造成空前的盛大。0xlab 除了很榮幸能成為 COSCUP 贊助單位之外，也有四位開發者獲邀分享若干技術議程，以下摘錄 COSCUP 的演講摘要：
講題：Android Game Engine 比較

講者： walkingice (0xlab)

摘要：隨著 Android的蓬勃發展，遊戲亦如雨後春筍般冒出。避免重複造輪子，勢必要藉助現有的遊戲引擎，讓遊戲開發者可以專注在遊戲的最核心價值 — 樂趣

講題：Be "Android"

講者： tick (0xlab), matt (0xlab)
摘要：將 0xdroid 打造成為機器人平台。Robocat 是為 0xdroid 所設計的機器人控制系統，此議程將從硬體架構、系統設計、函式庫及應用程式等不同層面切入，介紹如何打造自己的 ”Android”

講題：打造特製的 Android Toolchain

講者：Jim Huang "jserv" (0xlab)
摘要：探討 0xlab 近來的實驗，從修改 Android 的 GNU Toolchain 到提供 source-to-source compiler，提出若干基於特定需求，從而發展的編譯器技術的新應用

涵蓋 Android Toolchain, Robot, Android Gaming 開發，依據慣例，屆時我們也會發布最近的成果，期待您的蒞臨指教，謝謝！

Reading, Writing and Arithmetic: 0xdroid release beagle-eclair-0x5, devkit8k-eclair-0x5 (with/without SGX)

icedventilatte17@gmail.com (erin) — Mon, 02 Aug 2010 01:21:02 +0000

0xlab team is glad to announce the fifth release of 0xdroid. 0xdroid is community-developed Android distribution by 0xlab, runs on omap3 based boards include beagleboard and devkit8000.

Version: beagle-eclair-0x5-sgx, beagle-eclair-0x5-no-sgx, devkit8k-eclair-0x5-sgx, devkit8k-eclair-0x5-no-sgx

Version: beagle-eclair-0x5
Date: July 30, 2010
Release Image:
beagle-eclair-0x5_no_sgx.zip
beagle-eclair-0x5_sgx.zip
devkit8k-eclair-0x5_no_sgx.zip
devkit8k-eclair-0x5_sgx.zip
MD5 sum:
beagle-eclair-0x5_no_sgx
beagle-eclair-0x5_sgx
devkit8k-eclair-0x5_no_sgx
devkit8k-eclair-0x5_sgx

Release Details

Eclair codebase
Feature: Major Linux kernel upgrade: from 2.6.29 to 2.6.32
Feature: Support TI's Android PowerVR SGX (hardware 2D/3D engine)
Feature: Support Android USB gadget: adb and mass-storage (issue #95)
Note: 0xdroid no longer support usbnet gadget driver. Use adb or USB Ethernet instead.
Feature: new Bluetooth Extensions -- HID (issue #102)
Feature: Make Launcher2 really usable (issue #76)
Feature: Make Gallery3D workable by fixing PixelFlinger regressions (issue #107)
Feature: Touchscreen Calibration shouldn't require reboot (issue #94, issue #99)
Feature: Robocat i2c port support (issue #134)
Feature: Improved libgralloc for non HW accelerated target (issue #108, issue #133)

For more detail, please reference our roadmap and issues tracking

Where to download images
You can grab the image from here and following the steps described on the wiki page to test the image.

This release comes with PowerVR SGX from Texas Instruments. If you try both two images with SGX and without SGX, you would tell how amazing it displays photos in Gallery 3D application. Like ususal, it also comes with a happy Installer and it would bring you an Android Eclair system in few seconds. If you are looking for more details about the integration, you could reference to this wiki page.

How to build from scratch


 $ repo init -u git://gitorious.org/0xdroid/manifest.git -b beagle-eclair
 $ echo "TARGET_PRODUCT := beagleboard" > buildspec.mk
 $ echo "TARGET_PRODUCT := devkit8000" > buildspec.mk ( if build for devkit8000 )
 $ echo "INSTALL_PREBUILT_DEMO_APKS := true" >> buildspec.mk ( optional for adding prebuilt demo apks )
 $ cd .repo/manifests/
 $ git checkout -b beagle-eclair-0x5-release beagle-eclair-0x5
 $ repo sync
 $ repo forall -c "git checkout -b beagle-eclair-0x5-release beagle-eclair-0x5"
 $ make (or make -j4 if there are 4 cores)

Give us feedback
Join the 0xlab Development mailing-list
Join the 0xlab General discussion mailing-list
Issue tracking

Abacus: Power is not enough

MattHsu1016@gmail.com (matt) — Tue, 13 Jul 2010 23:20:00 +0000

Eventually, I got damn-spare time to post something interesting.
Last update was posted in the last year. It's so obvious to see
how busy/lazy I was in the past half year.

Tick and I are managed to build a robot project based on Beagleboard. We plan to share this project and concepts behind this on the event of COSCUP.

Well, I can say so far, the most tough work is not writing software to control robot. Instead, it's all about hardware such as providing power to servos. The ultimate goal is to make everything on robot wireless such as communication and power supply. Of course high capacity battery becomes the only option. Initially, we tried Ni-MH battery bought from RC car store. Even battery is full, it can't drive all revolute joints (servos). So we need plan B. :(

Occasionally, I found lead acid battery might be the candidate. After go shopping in the Guang-Hua market, Its voltage and capacity and size are all satisfied with our requirement. Most of important, it could drive all servos, totally 18 servos. Here is snapshot of the battery and debut robot.

Reading, Writing and Arithmetic: Connect with Bluetooth keyboard and mice in Android Eclair

icedventilatte17@gmail.com (erin) — Mon, 05 Jul 2010 03:09:48 +0000

In Nexus one without HID profile:

In Qualcomm MSM72xx platform with HID profile support:

Apparently, it is more friendly and user can connect with bluetooth keyboard from Setting UI application. Use can type words from a real keyboard, not from a tiny touch screen. All code are in 0xdroid project based on Eclair.

Reading, Writing and Arithmetic: Add Bluetooth HID profile in Android

icedventilatte17@gmail.com (erin) — Mon, 21 Jun 2010 23:35:43 +0000

The HID profile defines the protocols, procedures and features to be used by Bluetooth HID such as keyboards, pointing devices, gaming devices and remote monitoring devices. When I tried bluetooth function in Android Dev Phone (G1) last year, I use a utility tool 'hidd' which is from native bluez. It is a command tool. After I can pair with Apple Wireless keyboard sucessfully, I type 'hidd ' from terminal, then I can start to use it. But it's not automatically, I cannot connect it from Setting UI application either. Therefore, we should add a new Bluetooth HID profile in Android, to make it be connectable, like a BT headset or speaker.

# hidd --connect 00:1D:4F:A7:9A:49

# hidd --show
00:1D:4F:A7:9A:49 Apple Inc. Keyboard [05ac:022c] connected

0xlab is a member of OESF (Oopen Embedded Software Foundation) and they have some very useful bluetooth extensions in their 'Embedded Master 1' project. Althought these features are based on Android Donuts, they are still a very good reference. Therefore, I use their code and modify few parts of code for newer Android Eclair version. The difference of Eclair from Donuts in Bluetooth is bluez dbus API changes and some java API changes in frameworks. I've committed my patches to 0xdroid project in gitouris. If you'd like to use my work, you could apply patches in Frameworks and Setting application.

Beside the code in Frameworks and Settings UI application, I have to enable some features in kernel, since I use Apple Wireless Keyboard for this experiment. If I don't enable them, I can't never create an input device for Apple keyboard in Android. It took me few days to debug this issue. I cannot tell why I can use hidd to connect with it without any problem, but cannot use Dbus method to connect with it correctly. It's very weird. It's said 'Connected', but I cannot receive any key events or cannot see a virtual input device in Android system. Furthermore, there is no problem on connecting with a BT mice at all. Both methods are fine.

When I tried to implement any features in Android, I would try it in my Ubuntu machine first. They have linux kenel and many native code are the same. Based on the experiences in Ubuntu, I can know how it works and the normal flow what it looks like. In Ubuntu, when I connect with an Apple Wireless Keyboard, I can see the device name from dmesg. But I cannot see the device name in Android at all, I can see BT mouse device name in Android. Therefore, I check menuconfig on building kernel, and then I see some HID special drivers. I just enable features like CONFIG_HID_APPLE=y, CONFIG_HID_DEBUG=y, CONFIG_HIDRAW=y, CONFIG_USB_HIDDEV=y. After using the new kenel image, I can connect/disconnect with a BT keyboard from Setting UI application normally. I've verified my work in Beagle board and Qualcomm MSM72xx platform.

Linux & Ocarina: Eclair Libgralloc Deadlock Problem

ticktock35@0xlab.org (Tick) — Thu, 10 Jun 2010 09:50:15 +0000

As we are developing 0xdroid beagle-eclair branch, we occasionally encounter screen flipping issue. This issue rarely happens however it bothers the user experience very much when running some resource eating applications. Last week in the Computex Taipei 2010 show, we demoed 0xdroid beagle-eclair connecting wireless modules and played games. I noticed that this issue happens very often while playing a game called "Frozen Bubble". (It's a good game, we all love this game a lot, and spent a lot of time on it. ;-) ) It's kind of embarrassing when the screen keeps flipping on the show. Therefore I decided to dig out this issue.

Beside of 0xdroid on beagleboard and Devkit8000, I tested some other platforms and find out actually almost all of them having this problem. Therefore I suspected it's not a hardware related problem. Maybe a framework or HAL issue. We noticed that when the screen is flipping, the logcat message will complain as following

E/SurfaceFlinger(  768): eglSwapBuffers: EGL error 0x3002 (EGL_BAD_ACCESS)
E/gralloc (  768): handle 0x13f8c0 not locked
~E/gralloc (  768): handle 0x13f8c0 already locked for write
E/libagl  (  768): eglSwapBuffers() failed to lock buffer 0x1368e0 (640x480)
E/SurfaceFlinger(  768): eglSwapBuffers: EGL error 0x3002 (EGL_BAD_ACCESS)
E/gralloc (  768): handle 0x13f8c0 not locked
E/gralloc (  768): handle 0x13f8c0 already locked for write
E/libagl  (  768): eglSwapBuffers() failed to lock buffer 0x1368e0 (640x480)
E/SurfaceFlinger(  768): eglSwapBuffers: EGL error 0x3002 (EGL_BAD_ACCESS)
E/gralloc (  768): handle 0x13f8c0 not locked
E/gralloc (  768): handle 0x13f8c0 already locked for write
E/libagl  (  768): eglSwapBuffers() failed to lock buffer 0x1368e0 (640x480)
E/SurfaceFlinger(  768): eglSwapBuffers: EGL error 0x3002 (EGL_BAD_ACCESS)
E/gralloc (  768): handle 0x13f8c0 not locked

Therefore I checked the libgralloc and adding some debug message. The libgralloc plugin 0xdroid used is branched from original eclair source tree. I took few hours created the omap3/libgralloc at the first day when I got eclair source code months ago. Since it works well for the most of time, I didn't pay too much attention to it, until I found the deadlock issue goes crazy on frozen bubble.
After noticing the lock log and swap error, I took a close look of the gralloc_lock and gralloc_unlock in hardware/omap3/libgralloc/mapper.c

int gralloc_lock(gralloc_module_t const* module,
        buffer_handle_t handle, int usage,
        int l, int t, int w, int h,
        void** vaddr)
{
    if (private_handle_t::validate(handle) < 0)
        return -EINVAL;

    int err = 0;
    private_handle_t* hnd = (private_handle_t*)handle;
    int32_t current_value, new_value;
    int retry;

    do {
        current_value = hnd->lockState;
        new_value = current_value;

        if (current_value & private_handle_t::LOCK_STATE_WRITE) {
            // already locked for write 
            LOGE("handle %p already locked for write", handle);
            return -EBUSY;
        } else if (current_value & private_handle_t::LOCK_STATE_READ_MASK) {
            // already locked for read
            if (usage & (GRALLOC_USAGE_SW_WRITE_MASK | GRALLOC_USAGE_HW_RENDER)) {
                LOGE("handle %p already locked for read", handle);
                return -EBUSY;
            } else {
                // this is not an error
                //LOGD("%p already locked for read... count = %d", 
                //        handle, (current_value & ~(1<<31)));
            }
        }

        // not currently locked
        if (usage & (GRALLOC_USAGE_SW_WRITE_MASK | GRALLOC_USAGE_HW_RENDER)) {
            // locking for write
            new_value |= private_handle_t::LOCK_STATE_WRITE;
        }
        new_value++;

        retry = android_atomic_cmpxchg(current_value, new_value, 
    } while (retry);

    if (new_value & private_handle_t::LOCK_STATE_WRITE) {
        // locking for write, store the tid
        hnd->writeOwner = gettid();
    }

    if (usage & (GRALLOC_USAGE_SW_READ_MASK | GRALLOC_USAGE_SW_WRITE_MASK)) {
        if (!(current_value & private_handle_t::LOCK_STATE_MAPPED)) {
            // we need to map for real
            pthread_mutex_t* const lock = &sMapLock;
            pthread_mutex_lock(lock);
            if (!(hnd->lockState & private_handle_t::LOCK_STATE_MAPPED)) {
                err = gralloc_map(module, handle, vaddr);
                if (err == 0) {
                    android_atomic_or(private_handle_t::LOCK_STATE_MAPPED,
                            (volatile int32_t*)&(hnd->lockState));
                }
            }
            pthread_mutex_unlock(lock);
        }
        *vaddr = (void*)hnd->base;
    }

    return err;
}

int gralloc_unlock(gralloc_module_t const* module, 
        buffer_handle_t handle)
{
    if (private_handle_t::validate(handle) < 0)
        return -EINVAL;

    private_handle_t* hnd = (private_handle_t*)handle;
    int32_t current_value, new_value;

    do {
        current_value = hnd->lockState;
        new_value = current_value;

        if (current_value & private_handle_t::LOCK_STATE_WRITE) {
            // locked for write
            if (hnd->writeOwner == gettid()) {
                hnd->writeOwner = 0;
                new_value &= ~private_handle_t::LOCK_STATE_WRITE;
            }
        }

        if ((new_value & private_handle_t::LOCK_STATE_READ_MASK) == 0) {
            LOGE("handle %p not locked", handle);
            return -EINVAL;
        }

        new_value--;

    } while (android_atomic_cmpxchg(current_value, new_value, 
            (volatile int32_t*)&hnd->lockState));

    return 0;
}

The code looks reasonably for the first look. Lock and unlock pair looks good. However there is a very tricky part "android_atomic_cmpxchg may fail". Understanding this, it is not hard to see there is a potential bug in gralloc_unlock. If android_atomic_cmpxchg fails, it will run the do while loop for more than once. However for the first run, the hnd->writeOwner will be changed to 0. And then the new_value will not be changed anymore. This lock will goes crazy here after.

The patch solves this problem.

diff --git a/mapper.cpp b/mapper.cpp
index 16ebcc2..1f3e722 100644
--- a/mapper.cpp
+++ b/mapper.cpp
@@ -267,13 +267,13 @@ int gralloc_unlock(gralloc_module_t const* module,
         if (current_value & private_handle_t::LOCK_STATE_WRITE) {
             // locked for write
             if (hnd->writeOwner == gettid()) {
-                hnd->writeOwner = 0;
                 new_value &= ~private_handle_t::LOCK_STATE_WRITE;
             }
         }
 
         if ((new_value & private_handle_t::LOCK_STATE_READ_MASK) == 0) {
             LOGE("handle %p not locked", handle);
+            hnd->writeOwner = 0;
             return -EINVAL;
         }
 
@@ -282,5 +282,6 @@ int gralloc_unlock(gralloc_module_t const* module,
     } while (android_atomic_cmpxchg(current_value, new_value, 
             (volatile int32_t*)&hnd->lockState));
 
+    hnd->writeOwner = 0;
     return 0;
 }

It make sure the value hnd->writeOwner is the same as the first loop, if android_atomic_cmpxchg fails.

This issue comes from the original eclair source tree, and it is still there, and had been inherited to many different platforms. If you encounter two frames crazily flipping and having the lock message, you may try to take a look of your libgralloc.

Hacking at 0xlab: 在 Android/Dalvik 環境引入 precompiled class 的實驗

noreply@blogger.com (jserv) — Wed, 26 May 2010 02:08:45 +0000

延續筆者去年的紀錄「當 Compiler 遇上 Mobile」，最近我們又獲得一些進展，是在 Android 的 Dalvik 虛擬機器環境中，引入 pre-compiled class 的實驗，除了可改善執行時期的效能、記憶體使用量外，另外就是避免在「反組譯並修改 Android 應用程式實例」一文可見的資訊保護議題。

其實十幾年前，在 Java 平台中就有相當多團隊提出可行的方案，而世界上第一個開放原始碼的 Kaffe 虛擬機器專案，早在 1999 年即提出實做 "Kaffe/GCJ integration"，成功地將 GCJ (GCC for Java) 自 Java class/source 編譯得到的機械碼，當作 pre-compiled shared library，讓 Kaffee VM 讀取，預期可有效改善執行效能並縮減起始時間。GNU Classpath 專案主持人 Mark Wielaard 在 LWN 的文章 "GCJ - past, present, and future" 提到相關的歷史與技術背景，最早可回溯到 Cygnus solutions 時期 (RedHat 尚未併購) 的 GCJ 計畫提案 -- "A Gcc-based Java Implementation"。

為了降低實做的複雜度，筆者最早採用 LLVM 與 IcedTea 作為系統框架，不過一直到今年春節，整體進度還是陷入膠著的狀態。現在則引入 Java2C 搭配 PGO (Profile-Guided Optimization) 與統計模型，嘗試分析運行於 Dalvik VM 的 Android activity / system server，佔用系統資源最頻繁的項目，並預先編譯 (pre-compile) 這些 class/method，居中透過 JNI (Java Native Interface) 讓 Dalvik VM 存取。不過，實務上來說，不是將所有 class/method 都編譯為原生機械碼，就會帶來效能提昇，相反地，後者往往讓系統陷入更差的效能。目前 Eclair 裡面的 Dalvik VM 雖然沒有完整的 JIT compiler，但其 fast interpreter for ARM 的確做了頗多 threaded interpreter 的改進，考量到 I-cache, paging, 與 branch prediction，若我們不思量 Java / Android Dalvik 應用程式的執行時期行為，只是一味作編譯轉換，很可能只是編譯極少被執行，或者完全不會被執行到的程式碼，因此加重執行時期的負擔，而最該被優化的部份，受限於不完整的 source-to-source 轉換，有顧此失彼的疑慮。

於是，找出系統中最該被優化的部份，並考量到正確性與相容度，就是我們的首要研究議題，筆者採用知名的 SciMark 2.0 作為量化分析的指標。以下是在 Qualcomm MSM7x25 硬體平台 (arm1136j-s) 的效能評比，ARM11 的時脈是 528 MHz，ARM Linux 版本為 2.6.29.6-0xlab：
由上可見，在 SciMark 2.0 的各項評比中，Pre-compiled class 執行效能都較 Android Dalvik fast interpreter for ARM 給予一定程度的改良，並且透過 FDO (Feedback Directed Optimization)，大多可進一步給予提昇。這僅是初步的整合實驗數據，還需要更多分析與進行實做。