0%

Welcome to Hexo! This is your very first post. Check documentation for more info. If you get any problems when using Hexo, you can find the answer in troubleshooting or you can ask me on GitHub.

Quick Start

Create a new post

1
$ hexo new "My New Post"

More info: Writing

Run server

1
$ hexo server

OR

1
$ hexo s

More info: Server

Generate static files

1
$ hexo generate

OR

1
$ hexo g

More info: Generating

Deploy to remote sites

1
$ hexo deploy

OR

1
$ hexo d

Generate Categories

1
$ hexo new page categories

Then open source/categories/index.md, Add type: "categories"

1
2
3
4
5
6
---
title: categories
date: 2019-03-09 13:31:42
type: "categories"
comments: false
---

Generate Tags

1
$ hexo new page tags

Then open source/tags/index.md, Add type: "tags"

1
2
3
4
5
6
---
title: categories
date: 2019-03-09 13:31:42
type: "tags"
comments: false
---

Using tags or categories

Adding categories or tags description on page title

1
2
3
4
5
6
---
title: Hexo tutorial
tag: [Hexo,Tutorial]
categories:
- Web
---

Clean caches and regenerated

1
2
$ hexo clean
$ hexo g

More info: Deployment

Andriod Camera 推荐使用YUV格式,在Camera API1 中,推荐使用 NV21 和 YV12,因为这两种格式支持所有的Camera设备。
在Camera API2 中,推荐试用 YUV_420_888

YUV简介

不同于RGB颜色模型,使用R,G,B三个颜色分量来表示颜色。YUV使用明亮度”Y”, 色度”U,V”来表示颜色。
其中明亮度”Y”(Luninance或Luma)就是我们常见的灰度值, 色度(Chrominance或Chroma)用于指定像素的颜色。
其中:U(Cb), 代表蓝色; V(Cr), 代表红色。所有YUV还有一种说法就是YCbCr

YUV的优点:

  1. 优化彩色视频信号传输,可以很方便的兼容老式黑白电视。黑白电视只需要处理Y分量,彩色在其基础上加上U(Cb), V(Cr)两个分量即可。
  2. 降低传输的数据量,以常见的YUV420来看,每4个Y共用一组UV分量给,一个YUV占用 8+2+2 = 12bits = 1.5Byte;而RGB需要8+8+8 = 24bits = 3Byte;节省了一半的数据。

常用的格式

  • YUV420SP

    • NV21

      1
      YYYYYYYY VUVU
    • NV12

      1
      YYYYYYYY UVUV
  • YUV420P 平面模式, Y,U,V分别在不同的平面,是标准的4:2:0

    • YV12

      1
      YYYYYYYY VVUU
    • I420(YU12)

      1
      YYYYYYYY UUVV

内存布局

在实际使用中,一个YUV420图像大小,往往会比按照图像宽高计算的来的大。主要的原因就是一个数据对齐。这里有下面几个概念:

  • width 图像宽度
  • rowStride 跨度,图像实际存储的宽度
  • height 图像高度
  • scanLines 扫描线。图像实际存储高度

在Android手机的Camera框架中,HAL层的Buffer,都是通过GraphicBufferMapper去分配处理的。我们调用GraphicBufferMapper的lock接口去获取GraphicBuffer时,传入的参数是图像的宽高,返回的实际内存大小是经过对齐后的。

高通 NV12

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
/* Venus NV12:
* YUV 4:2:0 image with a plane of 8 bit Y samples followed
* by an interleaved U/V plane containing 8 bit 2x2 subsampled
* colour difference samples.
*
* <-------- Y/UV_Stride -------->
* <------- Width ------->
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . ^ ^
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . Height |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | Y_Scanlines
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . V |
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . V
* U V U V U V U V U V U V . . . . ^
* U V U V U V U V U V U V . . . . |
* U V U V U V U V U V U V . . . . |
* U V U V U V U V U V U V . . . . UV_Scanlines
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . V
* . . . . . . . . . . . . . . . . --> Buffer size alignment
*
* Y_Stride : Width aligned to 128
* UV_Stride : Width aligned to 128
* Y_Scanlines: Height aligned to 32
* UV_Scanlines: Height/2 aligned to 16
* Extradata: Arbitrary (software-imposed) padding
* Total size = align((Y_Stride * Y_Scanlines
* + UV_Stride * UV_Scanlines
* + max(Extradata, Y_Stride * 8), 4096)
*/

高通NV21

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
/* Venus NV21:
* YUV 4:2:0 image with a plane of 8 bit Y samples followed
* by an interleaved V/U plane containing 8 bit 2x2 subsampled
* colour difference samples.
*
* <-------- Y/UV_Stride -------->
* <------- Width ------->
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . ^ ^
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . Height |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | Y_Scanlines
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . | |
* Y Y Y Y Y Y Y Y Y Y Y Y . . . . V |
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . V
* V U V U V U V U V U V U . . . . ^
* V U V U V U V U V U V U . . . . |
* V U V U V U V U V U V U . . . . |
* V U V U V U V U V U V U . . . . UV_Scanlines
* . . . . . . . . . . . . . . . . |
* . . . . . . . . . . . . . . . . V
* . . . . . . . . . . . . . . . . --> Padding & Buffer size alignment
*
* Y_Stride : Width aligned to 128
* UV_Stride : Width aligned to 128
* Y_Scanlines: Height aligned to 32
* UV_Scanlines: Height/2 aligned to 16
* Extradata: Arbitrary (software-imposed) padding
* Total size = align((Y_Stride * Y_Scanlines
* + UV_Stride * UV_Scanlines
* + max(Extradata, Y_Stride * 8), 4096)
*/

这是一个系列的文章,主要从HAL层,Framework层,应用层三个方面来刨析Android Camera框架的实现原理和设计理念。源码以AOSP android 11版本为例。主要参考Android官网的文档。

这里我会从上到下依次按层级的方式,以Camera2 API和HAL3来解读Android Camera 的源码,中间回掺杂一些额外的补充知识点。

首先看下SDK中的相关内容

相关的类

在Camera API2 中,主要用到以下几个Package/类

  • Package: android.hardware.camera2
  • Camera: API1的接口
  • SurfaceView: 处理预览,渲染的呈现
  • MediaRecorder: 用于录像
  • Intent: 提供MediaStore.ACTION_IMAGE_CAPTURE 或者 MediaStore.ACTION_VIDEO_CAPTURE 的Intent操作类型用于捕获图像或者视频,不需要直接使用Camera对象

清单声明

权限申请

  • 相机权限

    1
    <uses-permission android:name="android.permission.CAMERA" />

    使用现有相机来打开相机(Intent),不需要申请权限

  • 存储权限

    1
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
  • 录音权限

    1
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
  • 位置权限

    1
    2
    3
    4
    <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
    ...
    <!-- Needed only if your app targets Android 5.0 (API level 21) or higher. -->
    <uses-feature android:name="android.hardware.location.gps" />

功能声明

1
<uses-feature android:name="android.hardware.camera" />

非必需情况

1
<uses-feature android:name="android.hardware.camera" android:required="false" />

开发相机应用

在官方开发文档中,建议开发者使用Camera X来开发相机应用。

参考:
https://developer.android.com/guide/topics/media/camera

https://developer.android.com/training/camera2

最新得Windows 10 版本18917或更高版本中,可以安装WSL2子系统,相比于WSL1,WSL2相当于提供了更加完整的Linux支持,因此可以用来编译Android系统源码,也不用再单独安装虚拟机了。

系统要求

  • Windows 10 版本18917或更高版本中
  • 足够的存储空间

准备工作

启用安装WSL2

参考微软官方:https://docs.microsoft.com/zh-cn/windows/wsl/wsl2-install

安装Ubuntu

应用商店中搜索Ubuntu,安装Ubuntu 18.04发行版

修改安装源

默认的源更新起来慢,可以修改成阿里云的源

  • 备份原始源

    1
    sudo cp /etc/apt/sources.list /etc/apt/sourses.list.bak
  • 更新阿里源

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    sudo vim /etc/apt/sources.list
    # 将里面的内容替换成
    deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
    deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
    deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
    deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
    deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
  • 更新

    1
    2
    sudo apt update
    sudo apt upgrade

Windows Terminal

Windows Terminal是微软推出的一个现代化Terminal工具,很好用

  • 到应用商店搜索Windows Terminal安装

Git & Vim

  • 安装
    1
    sudo apt install git vim
  • 配置
    1
    2
    git config --global user.name "your name"
    git config --global user.email "your email"

    repo

    1
    2
    3
    4
    5
    6
    mkdir ~/bin
    PATH=~/bin:$PATH
    curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
    ## 如果上述 URL 不可访问,可以用下面的:
    ## curl -sSL 'https://gerrit-googlesource.proxy.ustclug.org/git-repo/+/master/repo?format=TEXT' |base64 -d > ~/bin/repo
    chmod a+x ~/bin/repo

JDK

1
sudo apt install openjdk-8-jdk

设置CCCache

1
2
3
4
vim ~/.bashrc
# 追加以下内容
export USE_CCACHE=1
export CCACHE_DIR=/mnt/e/workspace/CCACHE/.ccache

针对源码工作目录,开启大小写敏感(重要)

  • Windows系统中文件和文件夹默认是不区分大小写的,但是编译Android源码,需要区分大小的系统。
  • 为了给WSL提供更好的支持,微软从Windows10 18917更新开始,为NTFS文件系统新增了一个SetCaseSensitiveInfo标志。可以有选择的根据所需的文件夹启用此flag,启用之后,NTFS文件系统就会针对该文件夹及其子文件视为区分大小写。
  • 此功能不仅可以在WSL中起作用,也可以在Windows下起作用。

开启方式

  • 以管理员身份打开命令提示符或者Powershell
  • 执行以下命令开启
    1
    2
    fsutil file SetCaseSensitiveInfo E:\workspace\AOSP enable
    fsutil file SetCaseSensitiveInfo E:\workspace\CCACHE enable
  • 执行以下命令关闭
    1
    2
    fsutil file SetCaseSensitiveInfo E:\workspace\AOSP disable
    fsutil file SetCaseSensitiveInfo E:\workspace\CCACHE disable

    下载代码

    初始化仓库

    这里使用中科大的镜像:https://lug.ustc.edu.cn/wiki/mirrors/help/aosp
1
2
3
4
cd /mnt/e/workspace/AOSP
repo init -u git://mirrors.ustc.edu.cn/aosp/platform/manifest
## 如果提示无法连接到 gerrit.googlesource.com,可以编辑 ~/bin/repo,把 REPO_URL 一行替换成下面的:
## REPO_URL = 'https://gerrit-googlesource.proxy.ustclug.org/git-repo'

初始化特定版本

1
repo init -u git://mirrors.ustc.edu.cn/aosp/platform/manifest -b android-9.0.0_r40

同步源码

1
repo sync

也可以参考Google官方教程

编译

我这里选择编译Pixel版本,因为手里有一个Pixel手机

1
2
3
source build/envsetup.sh
lunch 45 # aosp_sailfish-userdebug
make

编译错误处理

  • dex2oatd F dex2oat did not finish after 2850 seconds

    修改build/core/目录下的两个文件

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    diff --git a/core/dex_preopt_libart.mk b/core/  dex_preopt_libart.mk
    index 9c4d55de7..3b1158d9c 100644
    --- a/core/dex_preopt_libart.mk
    +++ b/core/dex_preopt_libart.mk
    @@ -184,7 +184,7 @@ source build/make/core/ verify_uses_libraries.sh "$(1)" && \
    source build/make/core/construct_context.sh "$ (PRIVATE_CONDITIONAL_USES_LIBRARIES_HOST)" "$ (PRIVATE_CONDITIONAL_USES_LIBRARIES_TARGET)" && \
    ,) \
    ANDROID_LOG_TAGS="*:e" $(DEX2OAT) \
    - --runtime-arg -Xms$(DEX2OAT_XMS) --runtime-arg -Xmx$(DEX2OAT_XMX) \
    + --runtime-arg -Xms$(DEX2OAT_XMS) -j1 --runtime-arg -Xmx$(DEX2OAT_XMX) \
    $${class_loader_context_arg} \
    $${stored_class_loader_context_arg} \
    --boot-image=$ (PRIVATE_DEX_PREOPT_IMAGE_LOCATION) \
    diff --git a/core/dex_preopt_libart_boot.mk b/core/ dex_preopt_libart_boot.mk
    index a5e7e881a..fadd6d794 100644
    --- a/core/dex_preopt_libart_boot.mk
    +++ b/core/dex_preopt_libart_boot.mk
    @@ -102,7 +102,7 @@ $($(my_2nd_arch_prefix) DEFAULT_DEX_PREOPT_BUILT_IMAGE_FILENAME) : $(LIBART_TARGE
    @rm -f $(dir $($(PRIVATE_2ND_ARCH_VAR_PREFIX) LIBART_TARGET_BOOT_OAT_UNSTRIPPED))/*.art
    @rm -f $(dir $($(PRIVATE_2ND_ARCH_VAR_PREFIX) LIBART_TARGET_BOOT_OAT_UNSTRIPPED))/*.oat
    @rm -f $(dir $($(PRIVATE_2ND_ARCH_VAR_PREFIX) LIBART_TARGET_BOOT_OAT_UNSTRIPPED))/*.art.rel
    - $(hide) $(DEX2OAT_BOOT_IMAGE_LOG_TAGS) $ (DEX2OAT) --runtime-arg -Xms$(DEX2OAT_IMAGE_XMS) \
    + $(hide) $(DEX2OAT_BOOT_IMAGE_LOG_TAGS) $ (DEX2OAT) -j1 --runtime-arg -Xms$(DEX2OAT_IMAGE_XMS) \
    --runtime-arg -Xmx$(DEX2OAT_IMAGE_XMX) \
    $(PRIVATE_BOOT_IMAGE_FLAGS) \
    $(addprefix --dex-file=,$ (LIBART_TARGET_BOOT_DEX_FILES)) \

动态注册和静态注册在JNI开发中是必然会遇到的问题。在最新的AndroidStudio中,默认使用的是静态注册,但是动态注册确实一种更加灵活的方式。

静态注册

在AndroidStudio中,创建项目时选择Native C++项目,会帮我们默认创建以下几个文件。

  • MainActivity.java Java文件,包含JNI Java层接口

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    public class MainActivity extends AppCompatActivity {

    // Used to load the 'native-lib' library on application startup.
    static {
    System.loadLibrary("native-lib");
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    // Example of a call to a native method
    TextView tv = findViewById(R.id.sample_text);
    tv.setText(stringFromJNI());
    }

    /**
    * A native method that is implemented by the 'native-lib' native library,
    * which is packaged with this application.
    */
    public native String stringFromJNI();
    }
  • native-lib.cpp C++文件,包含JNI函数的实现

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    #include <jni.h>
    #include <string>

    extern "C" JNIEXPORT jstring JNICALL
    Java_com_vectoros_jnimethod_MainActivity_stringFromJNI(
    JNIEnv *env,
    jobject /* this */) {
    std::string hello = "Hello from C++";
    return env->NewStringUTF(hello.c_str());
    }
  • CMakeLists.txt CMake编译脚本

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    cmake_minimum_required(VERSION 3.4.1)

    add_library( # Sets the name of the library.
    native-lib
    SHARED
    native-lib.cpp)

    find_library( # Sets the name of the path variable.
    log-lib
    log)

    target_link_libraries( # Specifies the target library.
    native-lib
    ${log-lib})

动态注册

动态注册需要写的代码会多一些,我们可以参考Framework的很多JNI接口的实现。Java层的代码不需要变化,CMake也不需要修改(如果有引入其他头文件,需要在CMake里面增加对应的库)

  • JNINativeMethod
    动态注册过程中,需要使用结构体JNINativeMethod来记录java方法和jni函数的对应关系

    1
    2
    3
    4
    5
    typedef struct {
    const char* name; //Java方法名
    const char* signature; //方法的参数和返回值,使用字符串记录,格式形如`()V, (I)I`,括号内表示函数参数,括号右侧表示函数返回值
    void* fnPtr; // 指向JNI函数的函数指针
    } JNINativeMethod;
  • 数据类型映射

    • 基本数据类型

      Java类型Native类型域描述符补充
      booleanjbooleanZ
      bytejbyteB
      charjcharC
      shortjshortS
      intjintI
      longjlongJ
      floatjfloatF
      doublejdoubleD
      voidvoidV
    • 数组引用类型

      Java类型Native类型域描述符补充
      boolean[]jbooleanArray[Z
      byte[]jbyteArray[B
      char[]jcharArray[C
      short[]jshortArray[S
      int[]jintArray[I
      long[]jlongArray[J
      float[]jfloatArray[F
      double[]jdoubleArray[D
    • 对象引用类型

      Java类型Native类型域描述符补充
      ClassjobjectLcom.example.Class;"L"开头,以";"结尾,内部类使用"$"连接,String除外
      StringjstringLjava/lang/String;唯一的例外
    • 对象数组引用类型

      Java类型Native类型域描述符补充
      Classjobject[Lcom.example.Class;"[L"开头,以";"结尾,内部类使用"$"连接,String除外
      Stringjstring[Ljava/lang/String;唯一的例外
  • JNI函数默认参数

    • native方法默认参数
      普通的native方法,是Java类的成员方法,默认的参数有以下两个

      1
      JNIEnv *env, jobject thiz
      • JNIEnv *env

        指代当前的java环境,可以利用JNIEnv操作Java层代码

      • jobject thiz

        指代JNI函数对应的java native方法对应的类的实例

    • static native方法默认参数
      static native方法,是Java类的static方法,默认的参数有以下两个

      1
      JNIEnv *env, jclass classz
      • JNIEnv *env

        指代当前的java环境,可以利用JNIEnv操作Java层代码

      • jclass classz

        指代JNI函数对应的java static native方法对应的class对象

  • native-lib.cpp

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    #include <jni.h>
    #include <string>

    // 定义java层对应的package名称
    #define JNIREG_CLASS "com/example/MainActivity"

    // Native方法实现
    static jstring stringFromJNI(JNIEnv *env, jclass classz)
    {
    std::string hello = "Hello from C++";
    return env->NewStringUTF(hello.c_str());
    }

    // Java层和Native层的方法签名
    static JNINativeMethod nativeMethods[] = {
    {"stringFromJNI", "()Ljava/lang/String;", (jstring *)stringFromJNI},
    };

    // 注册native方法
    int register_native_methods(JNIEnv *env)
    {
    jclass clazz = env->FindClass(JNIREG_CLASS);
    if (clazz == NULL)
    {
    ALOGE("Native registeration unable to find class '%s'", JNIREG_CLASS);
    return JNI_FALSE;
    }

    if (env->RegisterNatives(clazz, nativeMethods, sizeof (nativeMethods) / sizeof(nativeMethods[0])) < 0)
    {
    env->DeleteLocalRef(clazz);
    ALOGE("RegisterNatives failed for '%s'", JNIREG_CLASS);
    return JNI_FALSE;
    }

    return JNI_TRUE;
    }

    // 重写JNI_OnLoad方法
    extern "C" JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved)
    {
    JNIEnv *env = NULL;
    if (vm->GetEnv((void **)&env, JNI_VERSION_1_6) != JNI_OK)
    {
    return JNI_ERR;
    }

    if (JNI_TRUE != register_native_methods(env))
    {
    return JNI_ERR;
    }

    return JNI_VERSION_1_6;
    }


两种注册方法的对比

注册方法优点缺点
静态注册支持自动生成的IDE,可以比较方便的编写代码在没有IDE自动生成的情况下,编写不方便
静态注册程序运行效率低,每次调用native函数时,需要根据函数名在JNI层搜索对应的本地函数,建立对应关系,比较耗时
动态注册Native层方法和Java层实现简单的代码分离,降低耦合度需要手动实现注册方法
动态注册大量的Native方法,注册起来更加方便

native方法和static native方法的区别

在上面的默认参数一节中,我们已经提到了关于native方法和static native方法的一些区别

  • 静态注册时,IDE帮我们生成对应的方法,我们可以省去编写函数签名的操作。
  • 动态注册时,函数签名需要我们自己编写,就需要知道native方法和static native方法的一些区别和差异,才能使得我们的函数能够正常运行。

常用的JNI方法

  • jclass获取jobject
    1
    2
    jmethod methodId = env->GetMethodID(classz, "<init>", "()V");
    jobject jobj = env->NewObject(classz, methodId);
  • jobject获取jclass
    1
    jclass classz = env->GetObjectClass(thiz);
  • 获取object的非静态字段
    1
    2
    3
    4
    5
    6
    7
    8
    9
    // 获取jclass,静态方法则不需要这步
    jclass classz = env->GetObjectClass(thiz);
    // 获取field id,
    jfieldID fid = env->GetFieldID(classz, "mName", "Ljava/lang/String;");
    // 获取field 值
    jstring mName = env->GetObjectField(thiz, fid);
    // 设置值
    std::string name = "myname";
    env->SetObjectField(thiz, fid, env->NewStringUTF(name.c_str()))
  • 获取class的静态字段
    1
    2
    3
    4
    5
    6
    7
    8
    9
    // 获取jclass,静态方法则不需要这步
    jclass classz = env->GetObjectClass(thiz);
    // 获取field id,
    jfieldID fid = env->GetStaticFieldID(classz, "sName", "Ljava/lang/String;");
    // 获取field 值
    jstring mName = env->GetStaticObjectField(thiz, fid);
    // 设置值
    std::string name = "myname";
    env->SetStaticObjectField(thiz, fid, env->NewStringUTF(name.c_str()))
  • 获取class的非静态方法
    1
    2
    3
    4
    5
    6
    // 获取jclass,静态方法则不需要这步
    jclass classz = env->GetObjectClass(thiz);
    // 获取method id,
    jmethodID methodId = env->GetMethodID(classz, "getName", "()Ljava/lang/String;");
    // 调用java方法
    jstring name = env->CallObjectMethod(thiz, methodId)
  • 获取class的静态方法
    1
    2
    3
    4
    5
    6
    // 获取jclass,静态方法则不需要这步
    jclass classz = env->GetObjectClass(thiz);
    // 获取method id,
    jmethodID methodId = env->GetStaticMethodID(classz, "getCount", "()I");
    // 调用java方法
    jint count = env->CallStaticIntMethod(thiz, methodId)
  • 访问构造方法
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    // 获取jclass,静态方法则不需要这步
    jclass classz = env->FindClass("java/util/Date");
    jmethod methodId = env->GetMethodID(classz, "<init>", "()V");
    // 实例化对象
    jobject date = env->NewObject(classz, methodId);
    // 获取调用方法ID
    jmethod getTimeId = env->GetMethodID(classz, "getTime", "()J");
    // 调用方法
    jlong time = env->CallLongMethod(date, getTimeId);
    std::out<<"current time: " << time <<endl;

在定制自己的Android嵌入式系统时候,有时我们需要创建一个Native Service用来和底层设备通信。

  • Native Service其实是一个Linux守护进程,提供一些必要的服务。
  • Android提供了Binder进程间通信机制,Native Service就需要遵循Android的规则来实现

整体框架

  • Server (后台Service服务)
  • Client (提供和Service通信的接口)
  • JNI (持有一个Client端,封装对应的接口,用于和Service通信)
  • Java (JNI层提供给Java层的接口)
  • Selinux修改(Android 8以后)
  • 预编译到系统中,供Framework调用(如何在framework.jar中调用我们自己生成的java接口)

代码实现框架

代码我们以一个LED开关接口为例,添加一个NativeService,用来控制一个独立的LED灯的开关。

  • 创建组织代码目录

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # root directory
    mkdir LEDService
    cd LEDService
    # headers
    mkdir include
    # hardware interface
    mkdir hal
    # service implement
    mkdir service
    # client test
    mkdir test
    # JNI interface
    mkdir jni
    # java interface
    mkdir java
    # summary
    ls -l

    编译脚本

  • 为什么先写编译脚本?

    • 编译脚本可以预先帮助我们组织代码
    • 我们可以提前知道需要编译输出什么
  • 我们需要生成哪几个文件

    • libledservice.so 共享库,提供给Server端和Client端调用
    • libledservicemanager_jni.so JNI共享库,供Java接口调用
    • ledservicemanager.jar Jar包,提供给其他java层调用
    • ledservice.rc 服务启动的rc文件
    • com.vectoros.led.xml permission文件
  • Android.mk

    我们在LEDService目录下创建一个Android.mk用于编译C/C++ so, 可执行文件以及rc,xml等资源文件的编译输出。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    ################ LIB_LED_SERVICE ############
    include $(CLEAR_VARS)
    LOCAL_MODULE:= libledservice

    LOCAL_SRC_FILES:= \
    service/LEDService.cpp \
    service/ILEDService.cpp \
    service/LEDServiceManager.cpp

    LOCAL_C_INCLUDES += \
    $(LOCAL_PATH)/include

    LOCAL_SHARED_LIBRARIES := \
    libbinder \
    libutils \
    libcutils \
    liblog

    # Export include dir
    LOCAL_EXPORT_C_INCLUDE_DIRS := $(LOCAL_PATH)/include

    # LOCAL_CFLAGS += -mhard-float
    # LOCAL_LDFLAGS += -Wl,--no-warn-mismatch

    include $(BUILD_SHARED_LIBRARY)

    ################ LED_SERVICE ############
    include $(CLEAR_VARS)
    LOCAL_MODULE:= ledservice

    LOCAL_SRC_FILES:= \
    service/main_ledservice.cpp

    LOCAL_SHARED_LIBRARIES := \
    libutils \
    liblog \
    libbinder \
    libledservice

    LOCAL_MODULE_CLASS := EXECUTABLES
    LOCAL_INIT_RC := service/ledservice.rc
    include $(BUILD_EXECUTABLE)

    ################ LED_SERVICE_MANAGER_JNI ############
    include $(CLEAR_VARS)
    LOCAL_MODULE:= libledservicemanager_jni

    LOCAL_SRC_FILES:= \
    jni/com_vectoros_ledservice_manager_jni.cpp

    LOCAL_SHARED_LIBRARIES := \
    libutils \
    liblog \
    libcutils \
    libledservice \
    libbinder

    LOCAL_CFLAGS += -Wno-unused-parameter
    LOCAL_LDFLAGS += -Wl,--no-warn-mismatch

    include $(BUILD_SHARED_LIBRARY)

    # Using Android.bp to build java library
    # ################# JAVA_LIBRARIES ############
    # include $(CLEAR_VARS)
    # LOCAL_MODULE:= ledservicemanager
    # LOCAL_SRC_FILES:= \
    # $(call all-java-files-under,java)

    # LOCAL_MODULE_TAGS := optional
    # LOCAL_NO_STANDARD_LIBRARIES := true
    # LOCAL_DX_FLAGS := --core-library

    # include $(BUILD_JAVA_LIBRARY)

    # ################ PERMISION_FILES ############
    include $(CLEAR_VARS)
    LOCAL_MODULE:= com.vectoros.led.xml
    LOCAL_MODULE_TAGS := optional
    LOCAL_MODULE_CLASS:= ETC

    LOCAL_SRC_FILES:= java/$(LOCAL_MODULE)
    LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/permissions
    include $(BUILD_PREBUILT)

    ################ CMD_TEST_CLIENT ############
    include $(CLEAR_VARS)
    LOCAL_MODULE:= ledserviceclient

    LOCAL_SRC_FILES:= \
    test/main_ledservice_client.cpp

    LOCAL_SHARED_LIBRARIES := \
    libbinder \
    libutils \
    libcutils \
    liblog \
    libledservice

    LOCAL_LDFLAGS += -Wl,--no-warn-mismatch
    LOCAL_MODULE_CLASS := EXECUTABLES
    include $(BUILD_EXECUTABLE)
  • Andorid.bp

    我们在java目录下,创建一个Android.bp文件,用于生成jar包。至于为什么必须要用Android.bp,会留在集成到framework里面的时候讲。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    java_library {
    name: "ledservicemanager",
    srcs: [
    "com/vectoros/led/*.java",
    ],

    hostdex: true,
    java_version: "1.7",
    no_framework_libs: true,
    }
  • 总结

    通过以上的编译脚本,我们知道了自己需要写哪些文件,会生成什么样的文件,接下来就可以往下写实现部分的代码了。

头文件实现

  • 主要有三个头文件

    • LEDService.h
    • LEDServiceManager.h
    • ILEDService.h
  • ILEDService.h

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    #ifndef ANDROID_ILED_NATIVE_SERVICE_H
    #define ANDROID_ILED_NATIVE_SERVICE_H

    #include <stdint.h>
    #include <sys/types.h>


    #include <utils/RefBase.h>
    #include <utils/Singleton.h>
    #include <utils/Errors.h>
    #include <utils/String16.h>

    #include <binder/IInterface.h>

    namespace android
    {

    class ILEDService : public IInterface
    {
    public:
    DECLARE_META_INTERFACE(LEDService);

    virtual int initHardware(void) = 0;
    virtual void releaseHardware(void) = 0;
    virtual void on(void) = 0;
    virtual void off(void) = 0;
    };

    class BnLEDService : public BnInterface<ILEDService>{
    public:
    virtual status_t onTransact(uint32_t code, const Parcel& data,
    Parcel* reply, uint32_t flags = 0);
    };

    enum LED_SERVICE_TYPE{
    BASE_START,
    ON,
    OFF,
    };

    } // namespace android
    #endif //ANDROID_ILED_NATIVE_SERVICE_H
  • LEDService.h

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    #ifndef ANDROID_LED_NATIVE_SERVICE_H
    #define ANDROID_LED_NATIVE_SERVICE_H

    #include <map>
    #include <stdint.h>
    #include <sys/types.h>

    #include <cutils/compiler.h>

    #include <utils/Atomic.h>
    #include <utils/Errors.h>
    #include <utils/KeyedVector.h>
    #include <utils/RefBase.h>
    #include <utils/SortedVector.h>
    #include <utils/threads.h>

    #include <binder/BinderService.h>

    #include "ILEDService.h"


    namespace android {

    class LEDService : public BinderService<LEDService>, public BnLEDService
    {
    public:
    static char const* getServiceName() {
    return "LEDService";
    }

    LEDService();
    ~LEDService();

    private:
    virtual int initHardware(void);
    virtual void releaseHardware(void);
    virtual void on(void);
    virtual void off(void);
    };
    }; // namespace android

    #endif // ANDROID_LED_NATIVE_SERVICE_H
  • LEDServiceManager.h

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    #ifndef ANDROID_LED_NATIVE_MANAGER_H
    #define ANDROID_LED_NATIVE_MANAGER_H

    #include <stdint.h>
    #include <sys/types.h>

    #include <binder/IBinder.h>

    #include <utils/RefBase.h>
    #include <utils/Singleton.h>
    #include <utils/threads.h>

    #include "ILEDService.h"

    namespace android
    {
    class LEDServiceManager : public Singleton<LEDServiceManager>
    {
    private:
    /* data */
    bool isDied;
    void ledServiceDied();

    mutable sp<ILEDService> mLEDService;
    mutable sp<IBinder::DeathRecipient> mDeathObserver;

    public:
    LEDServiceManager(/* args */);
    ~LEDServiceManager();

    int initHardware(void);
    void releaseHardware(void);
    void on(void);
    void off(void);

    status_t assertState();
    bool checkService() const;
    void resetServiceStatus();
    };
    } // namespace android
    #endif //ANDROID_LED_NATIVE_MANAGER_H

客户端和服务端实现

  • ILEDService.cpp
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    #include <stdio.h>
    #include <stdint.h>
    #include <malloc.h>
    #include <sys/types.h>

    #include <binder/Parcel.h>
    #include <binder/IMemory.h>
    #include <binder/IPCThreadState.h>
    #include <binder/IServiceManager.h>
    #include "ILEDService.h"

    #ifdef __ANDROID__
    #define LOG_TAG "ILEDService"
    #if ANDROID_API_LEVEL >= 26
    #include <log/log.h>
    #else
    #include <cutils/log.h>
    #endif
    #endif

    namespace android
    {
    enum{
    INIT_HARDWARE = IBinder::FIRST_CALL_TRANSACTION,
    RELEASE_HARDWARE,
    ON,
    OFF,
    };


    class BpLEDService : public BpInterface<ILEDService>
    {
    public:
    BpLEDService(const sp<IBinder>& impl) : BpInterface<ILEDService>(impl)
    {

    }

    int initHardware(void)
    {
    Parcel data, reply;
    data.writeInterfaceToken(ILEDService::getInterfaceDescriptor());
    remote()->transact(INIT_HARDWARE, data, &reply);
    return (int)reply.readInt32();
    }

    void releaseHardware(void)
    {
    Parcel data, reply;
    data.writeInterfaceToken(ILEDService::getInterfaceDescriptor());
    remote()->transact(RELEASE_HARDWARE, data, &reply);
    }

    int on(void)
    {
    Parcel data, reply;
    data.writeInterfaceToken(ILEDService::getInterfaceDescriptor());
    remote()->transact(ON, data, &reply);
    return (int) reply.readInt32();
    }

    int off(void)
    {
    Parcel data, reply;
    data.writeInterfaceToken(ILEDService::getInterfaceDescriptor());
    remote()->transact(OFF, data, &reply);
    return (int) reply.readInt32();
    }
    };

    IMPLEMENT_META_INTERFACE(LEDService, "android.vectoros.LEDService");

    status_t BnLEDService::onTransact(uint32_t code, const Parcel& data, Parcel* reply, uint32_t flags)
    {
    char *buff;
    int len, retval;
    status_t status;

    switch(code) {
    case INIT_HARDWARE:{
    CHECK_INTERFACE(ILEDService, data, reply);
    retval = initHardware();
    reply->writeInt32(retval);
    return NO_ERROR;
    } break;


    case RELEASE_HARDWARE:{
    CHECK_INTERFACE(ILEDService, data, reply);
    releaseHardware();
    return NO_ERROR;
    } break;

    case ON:{
    CHECK_INTERFACE(ILEDService, data, reply);
    on();
    return NO_ERROR;
    } break;

    case OFF:{
    CHECK_INTERFACE(ILEDService, data, reply);
    off();
    return NO_ERROR;
    } break;

    default:
    return BBinder::onTransact(code, data, reply, flags);
    }
    }
    } // namespace android
  • LEDService.cpp
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    #include <stdint.h>
    #include <stdio.h>
    #include <math.h>
    #include <sys/types.h>

    #include <utils/Errors.h>
    #include <utils/RefBase.h>
    #include <utils/Singleton.h>
    #include <utils/String16.h>

    #include <binder/BinderService.h>
    #include <binder/IServiceManager.h>

    #include "LEDService.h"

    #define BUFFER_SIZE 512
    #ifdef __ANDROID__
    #define LOG_TAG "LEDService"
    #if ANDROID_API_LEVEL >= 26
    #include <log/log.h>
    #else
    #include <cutils/log.h>
    #endif
    #endif

    namespace android {

    LEDService::LEDService()
    {
    ALOGI("LEDService()");
    }

    LEDService::~LEDService(){
    ALOGI("~LEDService()");
    }

    int LEDService::initHardware(void)
    {
    ALOGI("initHardware(), check camera env.");
    }

    void LEDService::releaseHardware(void)
    {
    ALOGI("releaseHardware()");
    }

    int LEDService::on(void)
    {
    ALOGI("on()");
    return 0;
    }

    int LEDService::off(void)
    {
    ALOGI("off()");
    return 0;
    }

    }; // namespace android

  • LEDServiceManager.cpp
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    #include <stdint.h>
    #include <sys/types.h>

    #include <utils/Errors.h>
    #include <utils/RefBase.h>
    #include <utils/Singleton.h>

    #ifdef __ANDROID__
    #define LOG_TAG "LEDServiceManager"
    #if ANDROID_API_LEVEL >= 26
    #include <log/log.h>
    #else
    #include <cutils/log.h>
    #endif
    #endif

    #include <binder/IBinder.h>
    #include <binder/IServiceManager.h>

    #include "ILEDService.h"
    #include "LEDServiceManager.h"

    namespace android
    {
    LEDServiceManager::LEDServiceManager() : isDied(false)
    {

    }

    LEDServiceManager::~LEDServiceManager()
    {

    }

    void LEDServiceManager::LEDServiceDied()
    {
    isDied = true;
    mLEDService.clear();
    }

    status_t LEDServiceManager::assertState() {
    if (mLEDService == NULL) {
    // try for one second
    const String16 name("LEDService");
    for (int i=0 ; i<4 ; i++) {
    status_t err = getService(name, &mLEDService);
    if (err == NAME_NOT_FOUND) {
    usleep(250000);
    continue;
    }
    if (err != NO_ERROR) {
    ALOGE("LEDService not found");
    return err;
    }
    break;
    }

    initLEDServiceNative();


    class DeathObserver : public IBinder::DeathRecipient {
    LEDServiceManager& mLEDServiceManager;
    virtual void binderDied(const wp<IBinder>& who) {
    ALOGW("LEDService died [%p]", who.unsafe_get());
    mLEDServiceManager.LEDServiceDied();
    }
    public:
    DeathObserver(LEDServiceManager& mgr) : mLEDServiceManager(mgr) { }
    };

    mDeathObserver = new DeathObserver(*const_cast<LEDServiceManager *>(this));
    mLEDService->asBinder(mLEDService)->linkToDeath(mDeathObserver);
    }

    return NO_ERROR;
    }

    bool LEDServiceManager::checkService() const
    {
    return isDied? true:false;
    }

    void LEDServiceManager::resetServiceStatus()
    {
    isDied = false;
    }

    int LEDServiceManager::initHardware(void)
    {
    return mLEDService->initHardware();
    }

    void LEDServiceManager::releaseHardware(void)
    {
    mLEDService->releaseHardware();
    }

    int LEDServiceManager::on()
    {
    return mLEDService->on();
    }

    int LEDServiceManager::off()
    {
    return mLEDService->off();
    }

    } // namespace android

  • main_LED_service.cpp
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    #include <binder/BinderService.h>
    #include <LEDService.h>
    #include <binder/IPCThreadState.h>
    #include <binder/ProcessState.h>
    #include <binder/IServiceManager.h>

    #include "ILEDService.h"

    using namespace android;

    int main(int argc, char** argv) {
    #if 1
    LEDService::publishAndJoinThreadPool(true);
    // Like the SurfaceFlinger, limit the number of binder threads to 4.
    ProcessState::self()->setThreadPoolMaxThreadCount(4);
    #else

    sp<ProcessState> proc(ProcessState::self());

    sp<IServiceManager> sm = defaultServiceManager();

    sm->addService(String16("LEDService"), new LEDService());

    ProcessState::self()->startThreadPool();
    ProcessState::self()->giveThreadPoolName();
    IPCThreadState::self()->joinThreadPool();
    ProcessState::self()->setThreadPoolMaxThreadCount(4);
    #endif
    return 0;
    }

客户端服务端测试程序

JNI接口

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
#include <jni.h>
#include <cstdio>
#include <map>
#include <stdlib.h>
#include <string.h>

#include "LEDServiceManager.h"

#ifdef __ANDROID__
#define LOG_TAG "LEDServiceManagerJNI"
#if ANDROID_API_LEVEL >= 26
#include <log/log.h>
#else
#include <cutils/log.h>
#endif
#endif

#define JNIREG_CLASS "com/vectoros/led/LEDServiceManager"

namespace android
{
static jint nativeOn(JNIEnv *env, jclass jcls)
{
int err;
if (ledmgr)
{
err = ledmgr->assertState();
if (err == NO_ERROR)
{
return ledmgr->on();
}
else
{
return err;
}
}
else
{
ALOGE("ledmgr is not init!\n");
return NO_INIT;
}
}

static jint nativeOff(JNIEnv *env, jclass jcls)
{
int err;
if (ledmgr)
{
err = ledmgr->assertState();
if (err == NO_ERROR)
{
return ledmgr->off();
}
else
{
return err;
}
}
else
{
ALOGE("ledmgr is not init!\n");
return NO_INIT;
}
}

static JNINativeMethod nativeMethods[] = {
{"nativeOn", "()I", (jint *)nativeOn},
{"nativeOff", "()I", (jint *)nativeOff},
};

int register_vectoros_LEDService(JNIEnv *env)
{
jclass clazz = env->FindClass(JNIREG_CLASS);
if (clazz == NULL)
{
ALOGE("Native registeration unable to find class '%s'", JNIREG_CLASS);
return JNI_FALSE;
}

if (env->RegisterNatives(clazz, nativeMethods, sizeof(nativeMethods) / sizeof(nativeMethods[0])) < 0)
{
env->DeleteLocalRef(clazz);
ALOGE("RegisterNatives failed for '%s'", JNIREG_CLASS);
return JNI_FALSE;
}
return JNI_TRUE;
}

} // namespace android

using namespace android;

extern "C" JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved)
{
JNIEnv *env = NULL;
if (vm->GetEnv((void **)&env, JNI_VERSION_1_6) != JNI_OK)
{
return JNI_ERR;
}

ALOG_ASSERT(env, "Could not retrieve the env!");

if (JNI_TRUE != register_vectoros_LEDService(env))
{
return JNI_ERR;
}

return JNI_VERSION_1_6;
}

Java接口

  • LEDServiceManager.java
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    package com.vectoros.led;

    /**
    *
    */
    public class LEDServiceManager{
    private static final String TAG = "LEDServiceManager";
    static{
    System.loadLibrary("LEDServiceManager_jni");
    }

    public static int on(){
    return nativeOn();
    }
    public static int off(){
    return nativeOff();
    }

    private static native int nativeOn();
    private static native int nativeOff();
    }

为什么要手动编译OpenCV

正常情况下,我们利用OpenCV来开发Android应用,使用预编译的so和jar包已经足够了,但是在某些特殊情形下,我们希望自己编译OpenCV,借此来提高OpenCV的性能。

准备条件

  1. Ubuntu 18.04 64Bit
  2. cmake
  3. NDK Reversion 17.2.4988734
  4. OpenCV 4.0.1
  5. Android API
  6. Target armeabi-v7a

下载和配置NDK

  1. 从Google官网下载

    1
    wget https://dl.google.com/android/repository/android-ndk-r17c-linux-x86_64.zip
  2. 下载后解压到特定目录

    1
    2
    unzip android-ndk-r17c-linux-x86_64.zip
    mv android-ndk-r17c /opt/ndk

下载OpenCV源码

  1. 从官网下载最新版本或者特定版本

    1
    unzip opencv.zip
  2. 从Github上下载

    1
    git clone https://github.com/opencv/opencv.git opencv

编译OpenCV

  1. 创建输出目录

    1
    2
    3
    $ cd opencv
    $ make build
    $ cd build
  2. 配置编译规则

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    cmake -DCMAKE_TOOLCHAIN_FILE=/opt/ndk/build/cmake/android.toolchain.cmake \
    -DANDROID_NDK=/opt/ndk \
    -DANDROID_NATIVE_API_LEVEL=android-21 \
    -DCMAKE_BUILD_TYPE=Release \
    -DENABLE_VFPV3=ON \
    -DANDROID_ARM_NEON=TRUE \
    -DENABLE_NEON=ON \
    -DBUILD_PNG=ON \
    -DBUILD_JASPER=ON \
    -DBUILD_JPEG=ON \
    -DBUILD_TIFF=ON \
    -DBUILD_ZLIB=ON \
    -DWITH_JPEG=ON \
    -DWITH_PNG=ON \
    -DWITH_JASPER=ON \
    -DWITH_TIFF=ON \
    -DSOFTFP=ON \
    -DBUILD_JAVA=OFF \
    -DBUILD_ANDROID_EXAMPLES=OFF \
    -DBUILD_ANDROID_PROJECTS=OFF \
    -DANDROID_STL=c++_shared \
    -DANDROID_ABI=armeabi-v7a \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_INSTALL_PREFIX=opencv/install \
    ..
  3. 编译

    1
    $ make -j24 && make install
  4. 检查生成文件

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    cd opencv/install/sdk
    # etc 目录,存放了人脸识别模块部分资源文件,licenses等
    vectoros@vectoros:~/opencv/install/sdk$ ll etc/
    total 28
    drwxrwxr-x 5 vectoros vectoros 4096 Oct 8 10:50 ./
    drwxrwxr-x 4 vectoros vectoros 4096 Oct 8 10:50 ../
    drwxrwxr-x 2 vectoros vectoros 4096 Oct 8 10:50 haarcascades/
    drwxrwxr-x 2 vectoros vectoros 4096 Oct 8 10:50 lbpcascades/
    drwxrwxr-x 2 vectoros vectoros 4096 Oct 8 10:50 licenses/
    -rw-r--r-- 1 vectoros vectoros 2593 Aug 24 23:19 valgrind_3rdparty.supp
    -rw-r--r-- 1 vectoros vectoros 4088 Aug 24 23:19 valgrind.supp

    # native/jni/include 目录,存放了对应的.h .hpp头文件
    vectoros@vectoros:~/opencv/install/sdk/native$ ll jni/include/
    drwxrwxr-x 16 vectoros vectoros 4096 Oct 8 10:50 opencv2/

    # native/libs/armeabi-v7a/ 目录,存放了jni的so库或者.a文件
    vectoros@vectoros:~/opencv/install/sdk/native$ ll libs/armeabi-v7a/*.so
    -rw-r--r-- 1 vectoros vectoros 13462060 Oct 8 10:48 libs/armeabi-v7a/libopencv_calib3d.so
    -rw-r--r-- 1 vectoros vectoros 25141056 Oct 8 10:46 libs/armeabi-v7a/libopencv_core.so
    -rw-r--r-- 1 vectoros vectoros 50133564 Oct 8 10:47 libs/armeabi-v7a/libopencv_dnn.so
    -rw-r--r-- 1 vectoros vectoros 6983552 Oct 8 10:47 libs/armeabi-v7a/libopencv_features2d.so
    -rw-r--r-- 1 vectoros vectoros 3502996 Oct 8 10:46 libs/armeabi-v7a/libopencv_flann.so
    -rw-r--r-- 1 vectoros vectoros 525868 Oct 8 10:47 libs/armeabi-v7a/libopencv_highgui.so
    -rw-r--r-- 1 vectoros vectoros 21684136 Oct 8 10:47 libs/armeabi-v7a/libopencv_imgcodecs.so
    -rw-r--r-- 1 vectoros vectoros 26117996 Oct 8 10:46 libs/armeabi-v7a/libopencv_imgproc.so
    -rw-r--r-- 1 vectoros vectoros 6184124 Oct 8 10:46 libs/armeabi-v7a/libopencv_ml.so
    -rw-r--r-- 1 vectoros vectoros 3730776 Oct 8 10:49 libs/armeabi-v7a/libopencv_objdetect.so
    -rw-r--r-- 1 vectoros vectoros 5078460 Oct 8 10:47 libs/armeabi-v7a/libopencv_photo.so
    -rw-r--r-- 1 vectoros vectoros 7637264 Oct 8 10:49 libs/armeabi-v7a/libopencv_stitching.so
    -rw-r--r-- 1 vectoros vectoros 3361504 Oct 8 10:47 libs/armeabi-v7a/libopencv_videoio.so
    -rw-r--r-- 1 vectoros vectoros 3026088 Oct 8 10:49 libs/armeabi-v7a/libopencv_video.so
  5. 讲生成的文件应用到AndroidStudio NDK Projects里面

编译配置解析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
-DCMAKE_TOOLCHAIN_FILE=/opt/ndk/build/cmake/android.toolchain.cmake # 指定编译工具链cmake脚本
-DANDROID_NDK=/opt/ndk # 指定NDK路径
-DANDROID_NATIVE_API_LEVEL=android-21 # 指定NDK API Level
-DCMAKE_BUILD_TYPE=Release # 指定编译类型,默认是Debug类型,Release会有性能上的提升
-DENABLE_VFPV3=ON # 开启VFP优化选项
-DANDROID_ARM_NEON=TRUE # 设置NDK支持NEON
-DENABLE_NEON=ON # 开启O彭CV NEON优化选项
-DBUILD_PNG=ON # 编译PNG模块
-DBUILD_JASPER=ON # 编译JASPER模块
-DBUILD_JPEG=ON # 编译JPEG模块
-DBUILD_TIFF=ON # 编译TIFF模块
-DBUILD_ZLIB=ON # 编译ZLIB模块
-DWITH_JPEG=ON # 生成JPEG模块
-DWITH_PNG=ON # 生成PNG模块
-DWITH_JASPER=ON # 生成JASPER模块
-DWITH_TIFF=ON # 生成TIFF模块
-DSOFTFP=ON # 开启SOFTFP
-DBUILD_JAVA=OFF # 编译Java,这里只需要C++版本,如果需要java,设置为ON
-DBUILD_ANDROID_EXAMPLES=OFF # 编译Android Examples
-DBUILD_ANDROID_PROJECTS=OFF # 编译Android Projects
-DANDROID_STL=c++_shared # 指定STL版本,新版NDK里面默认使用c++_shared
-DANDROID_ABI=armeabi-v7a # 指定生成ABI版本
-DBUILD_SHARED_LIBS=ON # 是否生成共享so库,如果不设置,默认生成.a静态库
-DCMAKE_INSTALL_PREFIX=opencv/install # 安装路径,用于生成需要的头文件和so或者.a静态库(可选)
.. # 指定OpenCV代码根目录(重要)

参考

  1. https://www.learnopencv.com/install-opencv-on-android-tiny-and-optimized/
  2. https://www.sisik.eu/blog/android/ndk/opencv-without-java

简介

  • OpenCV是目前计算机视觉里面非常常用的一个开源库,支持各大平台

准备条件

  • AndroidStudio 最新版本
  • SDK Manager里面安装SDK, NDK
  • OpenCV 3.4/4.0
    1
    wget https://codeload.github.com/OpenCV/OpenCV/zip/3.4.6

准备知识-CMake

AndroidStudio 支持cmake编译C/C++文件,通过编写CMakeList.txt来实现

常用变量

1
2
${CMAKE_SOURCE_DIR} # CMakeLists.txt文件所在位置
${ANDROID_ABI} # Android ABI架构 ("armeabi-v7a","arm64-v8a")

常用函数

  • 常用指令

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    # Set project
    project(demo)
    # Set variable
    set(VARIABLE, VALUE)
    # Show message
    message(${ANDROID_ABI})
    # if else
    if(${ANDROID_ABI} STREQUAL "areambi")
    message("armv5")
    elseif(${ANDROID_ABI} STREQUAL "areambi-v7a")
    message("armv7a")
    elseif(${ANDROID_ABI} STREQUAL "arm64-v8a")
    message("armv8a")
    elseif(${ANDROID_ABI} STREQUAL "x86_64")
    message("x86_64")
    elseif(${ANDROID_ABI} STREQUAL "x86")
    message("x86")
    else()
    message("unknown abi")
    endif()
  • 添加共享库

    1
    2
    3
    4
    5
    6
    7
    add_library( 
    # Sets the name of the library.
    native-lib
    # Sets the library as a shared library.
    SHARED
    # Provides a relative path to your source file(s).
    src/main/cpp/native-lib.cpp)
  • 添加可执行文件

    1
    2
    3
    4
    5
    add_executable( 
    # Sets the name of the executable file.
    test
    # Provides a relative path to your source file(s).
    src/main/cpp/main.cpp)
  • 查找NDK library

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    # Searches for a specified prebuilt library and stores the path as a
    # variable. Because CMake includes system libraries in the search path by
    # default, you only need to specify the name of the public NDK library
    # you want to add. CMake verifies that the library exists before
    # completing its build.

    find_library( # Sets the name of the path variable.
    log-lib

    # Specifies the name of the NDK library that
    # you want CMake to locate.
    log)
  • 指定链接库

    1
    2
    3
    4
    5
    6
    7
    8
    9
    # Specifies libraries CMake should link to your target library. You
    # can link multiple libraries, such as libraries you define in this
    # build script, prebuilt third-party libraries, or system libraries.

    target_link_libraries( # Specifies the target library.
    test
    # Links the target library to the log library
    # included in the NDK.
    ${log-lib})
  • 指定包含头文件

    1
    2
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include/)
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/OpenCV)
  • 手动指定共享库

    1
    2
    3
    add_library(libOpenCV_java3 SHARED IMPORTED)
    set_target_properties(libOpenCV_java3 PROPERTIES IMPORTED_LOCATION
    ${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libOpenCV_java3.so)
  • 查找所有源文件,并保存到IR_SRCS变量,但是不能查找子目录

    1
    aux_source_directory(${CMAKE_SOURCE_DIR}/src/main/cpp DIR_SRCS)

编译器设置

我们可以在CMakeLists.txt里面设置编译器参数,也可以在build.gradle脚本里面设置

  • CMakeLists.txt 设置

    1
    2
    3
    #支持-std=gnu++11
    set(CMAKE_VERBOSE_MAKEFILE on)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11")
  • build.gradle 脚本设置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    apply plugin: 'com.android.application'

    android {
    defaultConfig {
    externalNativeBuild {
    cmake {
    //设置cpp Flags
    cppFlags "-std=c++11 -frtti -fexceptions"
    //设置STL共享库,这里要注意,部分Android系统里面可能没有libc++_shared.so,这个时候需要在app里面打包进去
    //文件路径在NDK安装目录下 sources/cxx-stl/llvm-libc++/libs/${ANDROID_ABI}/libc++_shared.so
    arguments "-DANDROID_STL=c++_shared"
    }
    }
    }
    }

输出设置

  • 动态共享库 .so

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/OpenCV)
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include/)
    add_library(libOpenCV_java3 SHARED IMPORTED)
    set_target_properties(libOpenCV_java3 PROPERTIES IMPORTED_LOCATION
    ${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libOpenCV_java3.so)
    find_library( # Sets the name of the path variable.
    log-lib

    # Specifies the name of the NDK library that
    # you want CMake to locate.
    log)
    add_library( # Sets the name of the library.
    native-lib

    # Sets the library as a shared library.
    SHARED

    # Provides a relative path to your source file(s).

    src/main/cpp/native-lib.cpp)
    target_link_libraries( # Specifies the target library.
    native-lib
    libOpenCV_java3
    # Links the target library to the log library
    # included in the NDK.
    ${log-lib})
  • 静态共享库 .a

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/OpenCV)
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include/)
    add_library(libOpenCV_java3 SHARED IMPORTED)
    set_target_properties(libOpenCV_java3 PROPERTIES IMPORTED_LOCATION
    ${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libOpenCV_java3.so)
    find_library( # Sets the name of the path variable.
    log-lib

    # Specifies the name of the NDK library that
    # you want CMake to locate.
    log)
    add_library( # Sets the name of the library.
    native-lib

    # Sets the library as a static library.
    STATIC

    # Provides a relative path to your source file(s).

    src/main/cpp/native-lib.cpp)
    target_link_libraries( # Specifies the target library.
    native-lib
    libOpenCV_java3
    # Links the target library to the log library
    # included in the NDK.
    ${log-lib})
  • Executable

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/OpenCV)
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include/)
    add_library(libOpenCV_java3 SHARED IMPORTED)
    set_target_properties(libOpenCV_java3 PROPERTIES IMPORTED_LOCATION
    ${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libOpenCV_java3.so)
    find_library( # Sets the name of the path variable.
    log-lib

    # Specifies the name of the NDK library that
    # you want CMake to locate.
    log)
    # 设置可执行文件的输出目录
    set(EXECUTABLE_OUTPUT_PATH "${CMAKE_SOURCE_DIR}/src/main/assets/${ANDROID_ABI}")
    # 添加可执行文件的源文件
    add_executable(
    test

    src/main/cpp/main.cpp
    )
    target_link_libraries( # Specifies the target library.
    test
    libOpenCV_java3
    # Links the target library to the log library
    # included in the NDK.
    ${log-lib})

依赖设置

与Gradle结合

新建Project

用AndroidStudio创建项目,并选择Native C++支持

导入OpenCV头文件

  1. src/main/cpp目录下创建include目录
  2. 将OpenCV SDK解压后,sdk/native/jni/include/目录下的两个头文件文件夹OpenCV, OpenCV2复制到上面创建的include目录下
  3. CMakeList.txt添加对应引用
    1
    include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include/)

导入OpenCV so库

  1. 创建src/main/jniLibs目录,用于存放第三方共享库
  2. 将OpenCV SDK解压后,sdk/native/libs/目录下的so文件文件夹复制到上面创建的jniLibs目录下,这些文件夹是以abi为目录保存的。
  3. CMakeList.txt添加对应依赖
    1
    2
    3
    add_library(libOpenCV_java3 SHARED IMPORTED)
    set_target_properties(libOpenCV_java3 PROPERTIES IMPORTED_LOCATION
    ${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/libOpenCV_java3.so)

jni引用OpenCV方法

1
2
3
#include "OpenCV2/core.hpp"
using namespace cv;

编译验证

我们在部署模型到Android端的时候,需要先评估模型的性能,Tensorflow官方给我们提供了TFLite的benchmark工具

整体流程

  1. 准备编译环境
  2. 安装依赖项目
  3. 安装编译工具
  4. 下载tensorflow源码
  5. 编译benchmark
  6. 部署到Android设备上
  7. 运行

准备编译环境

建议用Linux系统,这个部分可以参考官方指南, 我这边使用的Ubuntu 18.04

1
2
3
4
5
6
7
8
# install python and prerequisites
sudo apt install python-dev python-pip
pip install -U --user pip six numpy wheel mock
pip install -U --user keras_applications==1.0.6 --no-deps
pip install -U --user keras_preprocessing==1.0.5 --no-deps

# install openjdk-8
sudo apt-get install openjdk-8-jdk

安装编译工具bazel

参考bazel官网,编译最新版本需要bazel版本>=0.24.1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# install the prerequisites
sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python

# download bazel from github
# Next, download the Bazel binary installer named bazel-<version>-installer-linux-x86_64.sh from the Bazel releases page on GitHub.
wget https://github.com/bazelbuild/bazel/releases/download/0.24.1/bazel-0.24.1-installer-linux-x86_64.sh


# run the installer 0.24.1
chmod +x bazel-0.24.1-installer-linux-x86_64.sh
./bazel-0.24.1-installer-linux-x86_64.sh --user

# setup environment
export PATH="$PATH:$HOME/bin"

安装Android SDK和NDK

编译Android平台的benchmark需要Android SDK和NDK(里面包含Build tools, Platform tools)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
mkdir ~/Android
cd Android
# NDK
wget https://dl.google.com/android/repository/android-ndk-r17c-linux-x86_64.zip
unzip android-ndk-r17c-linux-x86_64.zip
mkdir Sdk
mv android-ndk-r17c Sdk/ndk-bundle

# SDK tools
wget https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip
unzip sdk-tools-linux-4333796.zip
mv tools/ Sdk/


# Platform tools
wget https://dl.google.com/android/repository/platform-tools-latest-linux.zip
unzip platform-tools-latest-linux.zip
mv platform-tools Sdk/

# Using sdkmanager to install tools
cd Sdk
chmod a+x tools/bin/sdkmanager
./tools/bin/sdkmanager "platform-tools" "platforms;android-28" "platforms;android-29" "build-tools;29.0.0"

编译tflite benchmark

参考官方文档

1
2
3
4
5
6
7
8
9
10
11
# download tensorflow code
git clone https://github.com/tensorflow/tensorflow.git

# build benchmark for android
bazel build -c opt \
--config=android_arm \
--cxxopt='--std=c++11' \
tensorflow/lite/tools/benchmark:benchmark_model

# build benchmark for desktop
bazel build -c opt --cxxopt='--std=c++11' tensorflow/lite/tools/benchmark:benchmark_model

在Android上运行

编译好的benchmark_model是一个Android平台的可运行的二进制文件,可以在adb shell里面直接运行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
adb push bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model /data/local/tmp

adb shell chmod +x /data/local/tmp/benchmark_model

adb push mobilenet_v2_1.0_224_quant.tflite /data/local/tmp

/data/local/tmp/benchmark_model \
--graph=/data/local/tmp/mobilenet_v2_1.0_224_quant.tflite \
--num_threads=10

# tashset f0 是 Pixel系列手机上,让程序运行在Big core上的命令
adb shell taskset f0 /data/local/tmp/benchmark_model \
--graph=/data/local/tmp/mobilenet_v2_1.0_224_quant.tflite \
--enable_op_profiling=true

运行结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
# mobilenet
taskset f0 /data/local/tmp/benchmark_model \
--graph=/data/local/tmp/mobilenet_v2_1.0_224_quant.tflite \
--enable_op_profiling=true

STARTING!
Min num runs: [50]
Min runs duration (seconds): [1]
Inter-run delay (seconds): [-1]
Num threads: [1]
Benchmark name: []
Output prefix: []
Min warmup runs: [1]
Min warmup runs duration (seconds): [0.5]
Graph: [/data/local/tmp/mobilenet_v2_1.0_224_quant.tflite]
Input layers: []
Input shapes: []
Use nnapi : [0]
Use legacy nnapi : [0]
Use gpu : [0]
Allow fp16 : [0]
Enable op profiling: [1]
Loaded model /data/local/tmp/mobilenet_v2_1.0_224_quant.tflite
resolved reporter
INFO: Initialized TensorFlow Lite runtime.
Initialized session in 1.892ms
Running benchmark for at least 1 iterations and at least 0.5 seconds
count=6 first=91810 curr=85097 min=84834 max=91810 avg=86184.3 std=2519

Running benchmark for at least 50 iterations and at least 1 seconds
count=50 first=118810 curr=114219 min=88216 max=129193 avg=115062 std=7678

Average inference timings in us: Warmup: 86184.3, Init: 1892, no stats: 115062
============================== Run Order ==============================
[node type] [start] [first] [avg ms] [%] [cdf%] [mem KB] [times called] [Name]
CONV_2D 0.000 5.567 7.237 6.297% 6.297% 0.000 1 [MobilenetV2/Conv/Relu6]
DEPTHWISE_CONV_2D 7.241 3.518 3.726 3.243% 9.539% 0.000 1 [MobilenetV2/expanded_conv/depthwise/Relu6]
CONV_2D 10.971 2.399 3.711 3.229% 12.768% 0.000 1 [MobilenetV2/expanded_conv/project/add_fold]
CONV_2D 14.688 12.247 17.846 15.529% 28.297% 0.000 1 [MobilenetV2/expanded_conv_1/expand/Relu6]
DEPTHWISE_CONV_2D 32.539 2.594 3.486 3.033% 31.330% 0.000 1 [MobilenetV2/expanded_conv_1/depthwise/Relu6]
CONV_2D 36.028 1.460 2.462 2.142% 33.472% 0.000 1 [MobilenetV2/expanded_conv_1/project/add_fold]
CONV_2D 38.493 12.619 7.820 6.805% 40.277% 0.000 1 [MobilenetV2/expanded_conv_2/expand/Relu6]
DEPTHWISE_CONV_2D 46.318 4.750 3.683 3.205% 43.482% 0.000 1 [MobilenetV2/expanded_conv_2/depthwise/Relu6]
CONV_2D 50.005 4.853 3.305 2.876% 46.358% 0.000 1 [MobilenetV2/expanded_conv_2/project/add_fold]
ADD 53.313 0.279 0.293 0.255% 46.613% 0.000 1 [MobilenetV2/expanded_conv_2/add]
CONV_2D 53.607 12.993 7.827 6.811% 53.424% 0.000 1 [MobilenetV2/expanded_conv_3/expand/Relu6]
DEPTHWISE_CONV_2D 61.438 2.546 1.647 1.434% 54.857% 0.000 1 [MobilenetV2/expanded_conv_3/depthwise/Relu6]
CONV_2D 63.089 1.239 1.009 0.878% 55.735% 0.000 1 [MobilenetV2/expanded_conv_3/project/add_fold]
CONV_2D 64.099 2.261 2.151 1.872% 57.607% 0.000 1 [MobilenetV2/expanded_conv_4/expand/Relu6]
DEPTHWISE_CONV_2D 66.253 1.251 1.238 1.077% 58.685% 0.000 1 [MobilenetV2/expanded_conv_4/depthwise/Relu6]
CONV_2D 67.493 1.074 1.126 0.980% 59.665% 0.000 1 [MobilenetV2/expanded_conv_4/project/add_fold]
ADD 68.621 0.097 0.106 0.092% 59.757% 0.000 1 [MobilenetV2/expanded_conv_4/add]
CONV_2D 68.728 2.234 2.056 1.789% 61.546% 0.000 1 [MobilenetV2/expanded_conv_5/expand/Relu6]
DEPTHWISE_CONV_2D 70.786 1.135 1.180 1.027% 62.573% 0.000 1 [MobilenetV2/expanded_conv_5/depthwise/Relu6]
CONV_2D 71.968 1.073 1.174 1.022% 63.594% 0.000 1 [MobilenetV2/expanded_conv_5/project/add_fold]
ADD 73.145 0.096 0.098 0.086% 63.680% 0.000 1 [MobilenetV2/expanded_conv_5/add]
CONV_2D 73.244 1.992 2.172 1.890% 65.570% 0.000 1 [MobilenetV2/expanded_conv_6/expand/Relu6]
DEPTHWISE_CONV_2D 75.419 0.409 0.469 0.408% 65.978% 0.000 1 [MobilenetV2/expanded_conv_6/depthwise/Relu6]
CONV_2D 75.889 0.395 0.445 0.387% 66.366% 0.000 1 [MobilenetV2/expanded_conv_6/project/add_fold]
CONV_2D 76.335 1.277 1.172 1.020% 67.386% 0.000 1 [MobilenetV2/expanded_conv_7/expand/Relu6]
DEPTHWISE_CONV_2D 77.509 0.778 0.624 0.543% 67.929% 0.000 1 [MobilenetV2/expanded_conv_7/depthwise/Relu6]
CONV_2D 78.135 1.041 0.779 0.678% 68.606% 0.000 1 [MobilenetV2/expanded_conv_7/project/add_fold]
ADD 78.915 0.067 0.054 0.047% 68.653% 0.000 1 [MobilenetV2/expanded_conv_7/add]
CONV_2D 78.969 1.059 1.147 0.998% 69.651% 0.000 1 [MobilenetV2/expanded_conv_8/expand/Relu6]
DEPTHWISE_CONV_2D 80.118 0.639 0.619 0.539% 70.190% 0.000 1 [MobilenetV2/expanded_conv_8/depthwise/Relu6]
CONV_2D 80.738 0.620 0.767 0.667% 70.857% 0.000 1 [MobilenetV2/expanded_conv_8/project/add_fold]
ADD 81.506 0.052 0.051 0.044% 70.902% 0.000 1 [MobilenetV2/expanded_conv_8/add]
CONV_2D 81.558 1.271 1.139 0.991% 71.892% 0.000 1 [MobilenetV2/expanded_conv_9/expand/Relu6]
DEPTHWISE_CONV_2D 82.699 0.595 0.582 0.506% 72.399% 0.000 1 [MobilenetV2/expanded_conv_9/depthwise/Relu6]
CONV_2D 83.282 0.853 0.750 0.652% 73.051% 0.000 1 [MobilenetV2/expanded_conv_9/project/add_fold]
ADD 84.033 0.055 0.053 0.046% 73.097% 0.000 1 [MobilenetV2/expanded_conv_9/add]
CONV_2D 84.087 1.078 1.127 0.981% 74.078% 0.000 1 [MobilenetV2/expanded_conv_10/expand/Relu6]
DEPTHWISE_CONV_2D 85.216 0.635 0.595 0.518% 74.596% 0.000 1 [MobilenetV2/expanded_conv_10/depthwise/Relu6]
CONV_2D 85.813 1.022 1.106 0.963% 75.559% 0.000 1 [MobilenetV2/expanded_conv_10/project/add_fold]
CONV_2D 86.921 2.031 2.034 1.770% 77.328% 0.000 1 [MobilenetV2/expanded_conv_11/expand/Relu6]
DEPTHWISE_CONV_2D 88.957 0.999 0.980 0.852% 78.181% 0.000 1 [MobilenetV2/expanded_conv_11/depthwise/Relu6]
CONV_2D 89.939 1.476 1.588 1.382% 79.562% 0.000 1 [MobilenetV2/expanded_conv_11/project/add_fold]
ADD 91.529 0.074 0.077 0.067% 79.629% 0.000 1 [MobilenetV2/expanded_conv_11/add]
CONV_2D 91.607 2.472 2.057 1.790% 81.419% 0.000 1 [MobilenetV2/expanded_conv_12/expand/Relu6]
DEPTHWISE_CONV_2D 93.666 1.046 0.959 0.834% 82.253% 0.000 1 [MobilenetV2/expanded_conv_12/depthwise/Relu6]
CONV_2D 94.627 1.548 1.564 1.361% 83.614% 0.000 1 [MobilenetV2/expanded_conv_12/project/add_fold]
ADD 96.193 0.077 0.079 0.068% 83.683% 0.000 1 [MobilenetV2/expanded_conv_12/add]
CONV_2D 96.272 2.107 2.032 1.768% 85.451% 0.000 1 [MobilenetV2/expanded_conv_13/expand/Relu6]
DEPTHWISE_CONV_2D 98.306 0.318 0.319 0.278% 85.729% 0.000 1 [MobilenetV2/expanded_conv_13/depthwise/Relu6]
CONV_2D 98.626 0.717 0.732 0.637% 86.365% 0.000 1 [MobilenetV2/expanded_conv_13/project/add_fold]
CONV_2D 99.359 1.280 1.413 1.230% 87.595% 0.000 1 [MobilenetV2/expanded_conv_14/expand/Relu6]
DEPTHWISE_CONV_2D 100.774 0.332 0.364 0.316% 87.911% 0.000 1 [MobilenetV2/expanded_conv_14/depthwise/Relu6]
CONV_2D 101.142 1.191 1.238 1.077% 88.989% 0.000 1 [MobilenetV2/expanded_conv_14/project/add_fold]
ADD 102.382 0.037 0.039 0.034% 89.023% 0.000 1 [MobilenetV2/expanded_conv_14/add]
CONV_2D 102.421 1.305 1.459 1.270% 90.292% 0.000 1 [MobilenetV2/expanded_conv_15/expand/Relu6]
DEPTHWISE_CONV_2D 103.883 0.369 0.391 0.340% 90.633% 0.000 1 [MobilenetV2/expanded_conv_15/depthwise/Relu6]
CONV_2D 104.275 1.214 1.239 1.078% 91.711% 0.000 1 [MobilenetV2/expanded_conv_15/project/add_fold]
ADD 105.516 0.036 0.036 0.032% 91.743% 0.000 1 [MobilenetV2/expanded_conv_15/add]
CONV_2D 105.552 1.312 1.405 1.222% 92.965% 0.000 1 [MobilenetV2/expanded_conv_16/expand/Relu6]
DEPTHWISE_CONV_2D 106.960 0.352 0.401 0.349% 93.314% 0.000 1 [MobilenetV2/expanded_conv_16/depthwise/Relu6]
CONV_2D 107.362 2.322 2.311 2.011% 95.325% 0.000 1 [MobilenetV2/expanded_conv_16/project/add_fold]
CONV_2D 109.675 3.883 3.400 2.958% 98.283% 0.000 1 [MobilenetV2/Conv_1/Relu6]
AVERAGE_POOL_2D 113.078 0.129 0.125 0.109% 98.392% 0.000 1 [MobilenetV2/Logits/AvgPool]
CONV_2D 113.204 1.732 1.844 1.604% 99.996% 0.000 1 [MobilenetV2/Logits/Conv2d_1c_1x1/BiasAdd]
RESHAPE 115.050 0.005 0.005 0.004% 100.000% 0.000 1 [output]

============================== Top by Computation Time ==============================
[node type] [start] [first] [avg ms] [%] [cdf%] [mem KB] [times called] [Name]
CONV_2D 14.688 12.247 17.846 15.529% 15.529% 0.000 1 [MobilenetV2/expanded_conv_1/expand/Relu6]
CONV_2D 53.607 12.993 7.827 6.811% 22.340% 0.000 1 [MobilenetV2/expanded_conv_3/expand/Relu6]
CONV_2D 38.493 12.619 7.820 6.805% 29.145% 0.000 1 [MobilenetV2/expanded_conv_2/expand/Relu6]
CONV_2D 0.000 5.567 7.237 6.297% 35.442% 0.000 1 [MobilenetV2/Conv/Relu6]
DEPTHWISE_CONV_2D 7.241 3.518 3.726 3.243% 38.684% 0.000 1 [MobilenetV2/expanded_conv/depthwise/Relu6]
CONV_2D 10.971 2.399 3.711 3.229% 41.913% 0.000 1 [MobilenetV2/expanded_conv/project/add_fold]
DEPTHWISE_CONV_2D 46.318 4.750 3.683 3.205% 45.118% 0.000 1 [MobilenetV2/expanded_conv_2/depthwise/Relu6]
DEPTHWISE_CONV_2D 32.539 2.594 3.486 3.033% 48.151% 0.000 1 [MobilenetV2/expanded_conv_1/depthwise/Relu6]
CONV_2D 109.675 3.883 3.400 2.958% 51.109% 0.000 1 [MobilenetV2/Conv_1/Relu6]
CONV_2D 50.005 4.853 3.305 2.876% 53.985% 0.000 1 [MobilenetV2/expanded_conv_2/project/add_fold]

Number of nodes executed: 65
============================== Summary by node type ==============================
[Node type] [count] [avg ms] [avg %] [cdf %] [mem KB] [times called]
CONV_2D 36 92.625 80.619% 80.619% 0.000 36
DEPTHWISE_CONV_2D 17 21.256 18.501% 99.120% 0.000 17
ADD 10 0.882 0.768% 99.888% 0.000 10
AVERAGE_POOL_2D 1 0.125 0.109% 99.997% 0.000 1
RESHAPE 1 0.004 0.003% 100.000% 0.000 1

Timings (microseconds): count=50 first=118487 curr=113927 min=88158 max=129072 avg=114924 std=7657
Memory (bytes): count=0
65 nodes observed

常见问题

  1. C++11编译异常

bazel编译命令后,加上--cxxopt='--std=c++11' 选项

  1. Bazel版本过低或者过高

安装0.24.1版本的bazel

目前在移动端(Android)使用比较广泛的深度模型框架是TFLite,这个也是Google大力推广的。但是目前很多高效的网络都没有官方的TensorFlow版本,所以在使用的时候,我们需要将其他格式的模型,转换成TFLite格式。
好在Google提供了非常全的转换工具,在最新版本中,Keras已经成为了官方推荐的前端。

模型转换前的准备

  • 确定源模型的格式
  • 确定源模型的输入名称,输入尺寸
  • 确定源模型的输出名称,输出尺寸

下面以Yolov3的cfg和weights文件转换成TFLite为例

Yolo的Backbone是Darknet,首先需要转换成支持的keras模型

  1. 转换成Keras h5 格式

使用修改后的工具keras_yolov3, 用里面的convert.py文件,将模型转成keras的h5格式

1
2
wget https://pjreddie.com/media/files/yolov3.weights
python convert.py yolov3.cfg yolov3.weights 320 model_data/yolov3.h5
  1. 读取转换后模型的参数

转换模型函数需要提供输出和输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def get_output_names(keras_model):
outputs = keras_model.outputs
output_names = []
for _output in outputs:
output_names.append(_output.name[:-2])
return output_names


def get_input_names(keras_model):
inputs = keras_model.inputs
input_names = []
for _input in inputs:
input_names.append(_input.name[:-2])
return input_names
  1. 使用官方提供的lite.TFLiteConverter.from_keras_model_file转换模型

调用官方提供的方法转换模型

1
2
3
4
5
6
7
8
9
10
11
12
13
def keras_to_tflite(input_keras_model_file, output_tflite_file):
keras_model = load_model(input_keras_model_file)
input_names = get_input_names(keras_model)
output_names = get_output_names(keras_model)
print("input names:", input_names, " shapes:", keras_model.input_shape)
print("output names:", output_names, " shapes:", keras_model.output_shape)
converter = lite.TFLiteConverter.from_keras_model_file(model_file=input_keras_model_file,
input_arrays=input_names,
output_arrays=output_names)
model = converter.convert()
file = open(output_tflite_file, "wb")
file.write(model)
print("save tflite file to: ", output_tflite_file)
  1. 获取转换后的模型
1
2
3
4
5
6
7
8
def main():
keras_model_file = "./model_data/yolov3.h5"
tflite_model_file = "./model_data/yolov3.tflite"
keras_to_tflite(keras_model_file, tflite_model_file)


if __name__ == '__main__':
main()

预告

如何在PC端验证测试转换后的TFLite模型