Android hand tracking - starting tips ?

Android hand tracking - starting tips ?

Hello, 
im trying to start hand tracking module on my samsung s8, and I had not achieved anything yet,
even after manually enabling the camera permissions, there were no feedback .

What should one do to start android hand tracking hello world style?

Vive Staff

Re: Android hand tracking - starting tips ?

@VirtualPierogi,

 

I'm pretty sure that this SDK strictly provides support for Vive branded VR devices at this point in time - it's not designed with the intent to support other devices. @zzy will be able to confirm or correct me on this. 

To reach Viveport's customer service team, email customerservice@viveport.com

To connect with a live support agent for hardware support, go to www.vive.com/support, click on "contact us" in the top righthand corner, and then opt to speak with a Live Chat agent

David R

Technical Specialist - San Francisco, CA; Monday-Friday

Click here to send me a PM

Vive Staff

Re: Android hand tracking - starting tips ?

Hi @VirtualPierogi 

 

We have basic support for Android devices. Please read the Android support document for Unity plugin, Android plugin or C++ plugin. One thing to note, your screen orientation must be Landscape Left.

 

Best Regards,

zzy

Re: Android hand tracking - starting tips ?

Okay,
so ive compiled the sample provided from developer site along with the hands plugin, on daydream, removed aristo wavevr and referenced script define symbols, added google vr symbols, made sure that everything is landscape left
but

after compiling no hand tracking is seen.
Anybody can point me to the right direction?

Vive Staff

Re: Android hand tracking - starting tips ?

Hi @VirtualPierogi 

 

One thing first, if you are only using Vive Hand Tracking SDK with GoogleVR plugin and not use with WaveVR plugin, the default editor script should work for you. No need to manually remove libraries or modify script define symbols.

 

Now back to your question.

 

1. Please make sure camera permission is granted to your app.

2. Are you enabling Daydream as XR? If so, please make sure your cellphone's rear camera is not blocked by the Daydream viewer. From my knowledge, the daydream viewer blocks the rear camera and thus we cannot detect any hand. I would recommend test without daydream viewer, or use a cardboard viewer that does not block the camera.

 

If you still encounter the problem after you unblock the camera, please attach the adb log for us to investigate.

You can use the command `adb logcat -s Unity Aristo` to include all the logs from Unity and our native plugin.

 

Best Regards,

zzy

Re: Android hand tracking - starting tips ?

1. Camera is unobstructed and all permissions are given 

2. Nothing (no hand interaction or bones are seen)

3. adb log goes as follows :

 

05-26 15:41:35.288 17353 17387 I Unity   : SystemInfo CPU = ARMv7 VFPv3 NEON, Cores = 8, Memory = 3679mb
05-26 15:41:35.288 17353 17387 I Unity   : SystemInfo ARM big.LITTLE configuration: 4 big (mask: 0xf0), 4 little (mask: 0xf)
05-26 15:41:35.289 17353 17387 I Unity   : ApplicationInfo com.XRPierogi.XRtest version 0.1 build 3ef1d2df-e88b-4ffd-915c-760db1216eb8
05-26 15:41:35.289 17353 17387 I Unity   : Built from '2018.3/release' branch, Version '2018.3.6f1 (a220877bc173)', Build type 'Release', Scripting Backend 'mono', CPU 'armeabi-v7a'
05-26 15:41:35.631 17353 17387 E Unity   : Unable to find libaudioplugingvrunity
05-26 15:41:37.777 17353 17387 I Unity   : Created eye textures with a "separate" layout.  The "multi-pass" stereo mode will be used.
05-26 15:41:37.777 17353 17387 I Unity   :
05-26 15:41:37.777 17353 17387 I Unity   :
05-26 15:41:37.777 17353 17387 I Unity   : (Filename:  Line: 2264)
05-26 15:41:37.777 17353 17387 I Unity   :
05-26 15:42:14.731 17353 17387 E Unity   : NullReferenceException: Object reference not set to an instance of an object
05-26 15:42:14.731 17353 17387 E Unity   :   at Draw.OnEnable () [0x00006] in <24d534f4e3d94938bcab8bc32901764a>:0
05-26 15:42:14.731 17353 17387 E Unity   :
05-26 15:42:14.731 17353 17387 E Unity   : (Filename: <24d534f4e3d94938bcab8bc32901764a> Line: 0)
05-26 15:42:14.731 17353 17387 E Unity   :
05-26 15:42:15.027 17353 17387 E Unity   : AndroidJavaException: java.lang.NullPointerException: Attempt to invoke virtual method 'int android.app.Activity.checkSelfPermission(java.lang.String)' on a null object reference
05-26 15:42:15.027 17353 17387 E Unity   : java.lang.NullPointerException: Attempt to invoke virtual method 'int android.app.Activity.checkSelfPermission(java.lang.String)' on a null object reference
05-26 15:42:15.027 17353 17387 E Unity   :      at com.google.gvr.permissionsupport.PermissionsFragment.hasPermission(PermissionsFragment.java:88)
05-26 15:42:15.027 17353 17387 E Unity   :      at com.unity3d.player.UnityPlayer.nativeRender(Native Method)
05-26 15:42:15.027 17353 17387 E Unity   :      at com.unity3d.player.UnityPlayer.c(Unknown Source:0)
05-26 15:42:15.027 17353 17387 E Unity   :      at com.unity3d.player.UnityPlayer$e$2.queueIdle(Unknown Source:72)
05-26 15:42:15.027 17353 17387 E Unity   :      at android.os.MessageQueue.next(MessageQueue.java:395)
05-26 15:42:15.027 17353 17387 E Unity   :      at android.os.Looper.loop(Looper.java:181)
05-26 15:42:15.027 17353 17387 E Unity   :      at com.unity3d.player.UnityPlayer$e.run(Unknown Source:32)
05-26 15:42:15.027 17353 17387 E Unity   :   at UnityEngine.AndroidJNISafe.CheckException () [0x00091] in <303b1ed396ed420184be31eb8dc21cdb>:0
05-26 15:42:15.027 17353 17387 E Unity   :   at UnityEngine.AndroidJNISafe.CallBooleanMethod (System.IntPtr obj, System.IntPtr methodID, UnityEngine.jvalue[] args) [0x00011] in <303b1
Vive Staff

Re: Android hand tracking - starting tips ?

From the log, the problem is caused by the AndroidJavaException part, and the detection is never started.

 

If you have already granted camera permission, you can try to remove the ARISTO_WITH_GOOGLEVR script define symbol and try agin.

 

In the meanwhile, I'll try to reproduce it on my side and see if it can be fixed in next version.

Can you please share some of your project info:

1. Version of Unity

2. Version of Google VR plugin

3. Current active XRDevice (I assume it's daydream).

Vive Staff

Re: Android hand tracking - starting tips ?

Hi @VirtualPierogi 

 

I haved tested on my side and it seems to work with latest Unity and GoogleVR. Here are my steps:

1. Create a new 3D project using Unity 2019.1.3f1

2. Import Vive Hand Tracking SDK v0.8.0 and Google VR SDK v1.200.0. Please note after you imported both SDK, the ARISTO_WITH_GOOGLEVR define is added automatically.

3. Switch platform to Android and change some player settings to make sure build can succeed:

    1. Change company name, product name and Android package name

    2. Enable Daydream in XRSettings, and remove Vulkan from Android Graphics API (since Unity does not support Vulkan + XR yet)

    3. Change Android Min API level to 24

4. Build and run on Pixel. When the app is run for the first time, it calls daydream API to grant camera permission, you need to take the phone off the viewer, hold it as portrait, accept camera permission and put it back to viewer.

 

Note: You don't need to change the binaries or defines manually.

 

Please let me if this steps works on your side or not. If you are using different versions, please let me know.

Re: Android hand tracking - starting tips ?

Hey,
did everything as prescribed and added the sample scene from examples unitypackage for hand tracking, but still nothing happens when i use gestures shown on the infographics.
Should I change the the gesture provider type from skeleton to point 2d or 3d?
Or make empty scene with all the prefabs only just to test if it works on Samsung 8?

@zzy 

Vive Staff

Re: Android hand tracking - starting tips ?

Hi @VirtualPierogi 

 

Are you still experiencing the same error from adb log?

 

I think you might not have Daydream setup on your Samsung S8. You need to install the Daydream app from Google play and setup the viewer to make a daydream app running. If you don't have a daydream viewer, I suggest you use Cardboard as your XRDevice instead.

Re: Android hand tracking - starting tips ?

Thats probably not the case as I have both the controller and daydream view on my desk, but I've noticed the following problem in adb log :

05-29 12:03:26.071   948  1005 I Unity   : SystemInfo CPU = ARMv7 VFPv3 NEON, Cores = 8, Memory = 3679mb
05-29 12:03:26.071   948  1005 I Unity   : SystemInfo ARM big.LITTLE configuration: 4 big (mask: 0xf0), 4 little (mask: 0xf)
05-29 12:03:26.073   948  1005 I Unity   : ApplicationInfo com.XRInstitute.HandTracking version 0.1 build 5b1ec8d0-32fe-4103-88e1-ac25ff794964
05-29 12:03:26.073   948  1005 I Unity   : Built from '2019.1/staging' branch, Version '2019.1.4f1 (ffa3a7a2dd7d)', Build type 'Release', Scripting Backend 'mono', CPU 'armeabi-v7a', Stripping 'Disabled'
05-29 12:03:26.456   948  1005 E Unity   : Unable to find libaudiopluginresonanceaudio
--------- beginning of system
05-29 12:03:26.825   948  1005 D Unity   :  GL_EXT_debug_marker GL_ARM_rgba8 GL_ARM_mali_shader_binary GL_OES_depth24 GL_OES_depth_texture GL_OES_depth_texture_cube_map GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_EXT_read_format_bgra GL_OES_compressed_paletted_texture GL_OES_compressed_ETC1_RGB8_texture GL_OES_standard_derivatives GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_EGL_sync GL_OES_texture_npot GL_OES_vertex_half_float GL_OES_required_internalformat GL_OES_vertex_array_object GL_OES_mapbuffer GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_type_2_10_10_10_REV GL_OES_fbo_render_mipmap GL_OES_element_index_uint GL_EXT_shadow_samplers GL_OES_texture_compression_astc GL_KHR_texture_compression_astc_ldr GL_KHR_texture_compression_astc_hdr GL_KHR_texture_compression_astc_sliced_3d GL_KHR_debug GL_EXT_occlusion_query_boolean GL_EXT_disjoint_timer_query GL_EXT_blend_minmax GL_EXT_discard_framebuffer GL_OES_get_program_binary GL_OES_texture_3D GL_EXT_texture_storage GL_EXT_multisampled_render_
05-29 12:03:26.825   948  1005 D Unity   : to_texture GL_EXT_multisampled_render_to_texture2 GL_OES_surfaceless_context GL_OES_texture_stencil8 GL_EXT_shader_pixel_local_storage GL_ARM_shader_framebuffer_fetch GL_ARM_shader_framebuffer_fetch_depth_stencil GL_ARM_mali_program_binary GL_EXT_sRGB GL_EXT_sRGB_write_control GL_EXT_texture_sRGB_decode GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_RG8 GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent GL_OES_texture_storage_multisample_2d_array GL_OES_shader_image_atomic GL_EXT_robustness GL_EXT_draw_buffers_indexed GL_OES_draw_buffers_indexed GL_EXT_texture_border_clamp GL_OES_texture_border_clamp GL_EXT_texture_cube_map_array GL_OES_texture_cube_map_array GL_OES_sample_variables GL_OES_sample_shading GL_OES_shader_multisample_interpolation GL_EXT_shader_io_blocks GL_OES_shader_io_blocks GL_EXT_tessellation_shader GL_OES_tessellation_shader GL_EXT_primitive_bounding_box GL_OES_primitive_bounding_box GL_EXT_geometry_shader GL_OES_geometry_shader GL_ANDROID_extension_pack_es31a GL_EXT_gpu_sh
05-29 12:03:26.825   948  1005 D Unity   : ader5 GL_OES_gpu_shader5 GL_EXT_texture_buffer GL_OES_texture_buffer GL_EXT_copy_image GL_OES_copy_image GL_EXT_shader_non_constant_global_initializers GL_EXT_color_buffer_half_float GL_EXT_color_buffer_float GL_EXT_YUV_target GL_OVR_multiview GL_OVR_multiview2 GL_OVR_multiview_multisampled_render_to_texture GL_KHR_robustness GL_KHR_robust_buffer_access_behavior GL_EXT_draw_elements_base_vertex GL_OES_draw_elements_base_vertex GL_EXT_protected_textures GL_EXT_buffer_storage GL_EXT_external_buffer GL_EXT_EGL_image_array
05-29 12:03:29.324   948  1005 I Unity   : Created eye textures with a "separate" layout.  The "multi-pass" stereo mode will be used.
05-29 12:03:29.324   948  1005 I Unity   :
05-29 12:04:00.229   948  1005 I Unity   : GVR SDK for Unity version: 1.200.0
05-29 12:04:00.229   948  1005 I Unity   :
05-29 12:04:00.229   948  1005 I Unity   : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 48)
05-29 12:04:00.229   948  1005 I Unity   :
05-29 12:04:00.557   948  1005 I Aristo  : Running as 32-bit binary
05-29 12:04:00.663   948  1005 I Aristo  : Selected camera 0
05-29 12:04:02.783   948  2109 E Aristo  : Android NDK camera update frame failed!
05-29 12:04:02.783   948  2109 E Aristo  : Camera update failed, stop detection

It seems that the android ndk is failing to update camera, I dont have NDK installed but will try with it next, do I have to change from mono to il2cpp ?

Vive Staff

Re: Android hand tracking - starting tips ?

Hi @VirtualPierogi 

 

You are right, the problem is in Hand Tracking SDK now, since we cannot get camera frames from your rear camera. From the log, we can find a rear camera, but cannot get frames from it.

 

If you are building 32bit binary, using mono is definitely fine.

 

Please make sure:

1. Your app have camera permission (I recommend you grant permission manually in Settings)

2. Some phones may not support NDK Camera2 API, which is what we use to get frames. A simple check is to try this example and see if it works.

Re: Android hand tracking - starting tips ?

Allright, the example Native Android Camera2 shows black, is there any option to enable it or change the api from camera2 to other camera provider?
edit:


After a while it seems that native android camera actually does work, so that makes me wonder what doesnt work on the aristo side.

Vive Staff

Re: Android hand tracking - starting tips ?

OK, it seems that on your phone, it takes a longer time before the camera streaming starts to output frames. We internally have a 1.5 second timeout for waiting one frame. This seems to be the reason. I'll try to increase the timeout for the first frame in next version (expected to release in a few days).

Re: Android hand tracking - starting tips ?

Thanks for Your help, ill keep in touch for the nex version, 

Labels