Create your first AR project in Android studio using Sceneform

AR apps are possible using Unity 3D, but today we are going to use Android studio to build an AR application. Google has come up with ARCore library to enhance AR development. ARCore does motion tracking in the real world through the camera to create paths on planes/surfaces over which we can place our 3D Models and graphics. SceneForm is a 3D framework that’s come up recently and is a better alternative to OpenGL. OpenGL looks intimidating to code.

Thanks to new features in Google’s ARCore and Sceneform. Objects can now be placed on vertical planes such as walls or doors. 3D images render faster and more easily, and augmented reality scenes can be shared between devices. Overall, the AR experience has become much more advanced and easier to use compared to preceding versions.

SceneForm allows us to quickly render 3d objects without the need to learn graphics or OpenGL. We can download the Google Sceneform Tools plugin in our Android Studio to view and render the 3d models.

You can go to https://poly.google.com/ and download a sample model. The FBX and GLTX formats are used for rendering augmented images.

Now let’s build our first AR application!

Android AR Project Structure

In the above project, we’ve created a sampledata directory, where we add the fbx files that downloaded from any 3d model stores like cgtrader.

Add the following dependency in your project’s build.gradle:

classpath 'com.google.ar.sceneform:plugin:1.10.0'

Add the following in your app’s build.gradle :

implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.10.0'

You need to set the ar plugin in your app’s Gradle file as well.
Add the following below the dependencies:

apply plugin: 'com.google.ar.sceneform.plugin'

Add the following lines at the end of the app’s build.gradle:

sceneform.asset('sampledata/toy1.fbx',         
'default',
'sampledata/toy1.sfa',
'src/main/assets/toy1')

The build.gradle of the app looks like this finally:

apply plugin: 'com.android.application'
apply plugin: 'com.google.ar.sceneform.plugin'
android {
compileSdkVersion 29
buildToolsVersion "29.0.0"
defaultConfig {
applicationId "com.example.myapplication"
minSdkVersion 24
targetSdkVersion 29
versionCode 1
versionName "1.0"
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
}
}
compileOptions {
sourceCompatibility = '1.8'
targetCompatibility = '1.8'
}
}
dependencies {
implementation fileTree(dir: 'libs', include: ['*.jar'])
implementation 'androidx.appcompat:appcompat:1.0.2'
implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
testImplementation 'junit:junit:4.12'
androidTestImplementation 'androidx.test:runner:1.2.0'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
implementation "com.google.android.gms:play-services-location:+"
implementation "com.google.ar.sceneform.ux:sceneform-ux:1.10.0"
}
apply plugin: 'com.google.ar.sceneform.plugin'sceneform.asset('sampledata/toy1.fbx',
'default',
'sampledata/toy1.sfa',
'src/main/assets/toy1')
sceneform.asset('sampledata/toy2.fbx',
'default',
'sampledata/toy2.sfa',
'src/main/assets/toy2')
sceneform.asset('sampledata/toy3.fbx',
'default',
'sampledata/toy3.sfa',
'src/main/assets/toy3')
sceneform.asset('sampledata/toy4.fbx',
'default',
'sampledata/toy4.sfa',
'src/main/assets/toy4')
sceneform.asset('sampledata/toy5.fbx',
'default',
'sampledata/toy5.sfa',
'src/main/assets/toy5')
sceneform.asset('sampledata/toy6.fbx',
'default',
'sampledata/toy6.sfa',
'src/main/assets/toy6')
sceneform.asset('sampledata/toy7.fbx',
'default',
'sampledata/toy7.sfa',
'src/main/assets/toy7')
sceneform.asset('sampledata/toy8.fbx',
'default',
'sampledata/toy8.sfa',
'src/main/assets/toy8')
sceneform.asset('sampledata/toy9.fbx',
'default',
'sampledata/toy9.sfa',
'src/main/assets/toy9')

Android ARCore Sceneform requires Java 8 or higher version.

The sja and sjb are the Sceneform Asset Description and Sceneform Binary files. The sjb file is visible in the 3D viewer. It is shipped with the APK. and sja file is used to set properties for the sjb file.

Android AR Project Code

To configure ARCore in your application add the following permission and metadata to your AndroidManifest.xml file.

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature
android:name="android.hardware.camera.ar"
android:required="true" />

metadata goes inside the application tag.

<meta-data
android:name="com.google.ar.core"
android:value="required" />

The code for the AndroidManifest.xml is given below:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.myapplication">


<uses-permission android:name="android.permission.CAMERA" />

<uses-feature
android:name="android.hardware.camera.ar"
android:required="true" />

<application
android:allowBackup="true"
android:icon="@mipmap/ic_launcher"
android:label="@string/app_name"
android:roundIcon="@mipmap/ic_launcher_round"
android:supportsRtl="true"
android:theme="@style/AppTheme">
<meta-data
android:name="com.google.ar.core"
android:value="required" />

<activity
android:name=".MainActivity"
android:screenOrientation="portrait">
<intent-filter>
<action android:name="android.intent.action.MAIN" />

<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>

</manifest>

The code for the activity_main.xml class is given below:

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">

<fragment
android:id="@+id/ux_fragment"
android:name="com.google.ar.sceneform.ux.ArFragment"
android:layout_width="match_parent"
android:layout_height="match_parent"
/>


<SeekBar
android:id="@+id/sb_size"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:layout_alignParentBottom="true"
android:layout_marginBottom="100dp"
android:min="40"
android:max="100"
android:enabled="false"
android:progress="70"
/>


<RelativeLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:layout_margin="20dp"
android:layout_marginHorizontal="10dp"
android:background="#FFFFFF">
<Spinner
style="@style/Widget.AppCompat.DropDownItem.Spinner"
android:layout_width="match_parent"
android:layout_height="60dp"
android:id="@+id/spn_model"
/>
</RelativeLayout>

<Button
android:id="@+id/accelerate"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_marginLeft="15dp"
android:layout_marginRight="15dp"
android:layout_marginBottom="20dp"
android:layout_centerHorizontal="true"
android:text="@string/str_accelerate"
android:layout_alignParentBottom="true"
android:background="@drawable/dr_button_green"
android:textSize="15dp"
android:padding="5dp"
android:textColor="#FFF"
/>

<Button
android:id="@+id/r_left"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="bottom"
android:layout_marginBottom="20dp"
android:layout_alignParentBottom="true"
android:layout_toLeftOf="@+id/accelerate"
android:text="@string/str_left"
android:background="@drawable/dr_button_white"
android:textSize="15dp"
android:textColor="@color/blue"
android:padding="5dp"
/>

<Button
android:id="@+id/r_right"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignParentBottom="true"
android:layout_toRightOf="@+id/accelerate"
android:layout_marginBottom="20dp"
android:textColor="@color/blue"
android:background="@drawable/dr_button_white"
android:textSize="15dp"
android:padding="5dp"
android:text="@string/str_right"
/>
</RelativeLayout>

We’ve set the fragment as ArFragment. The ArFragment checks if the phone is compatible with ARCore. It checks if the camera permission is granted. If it isn’t, it’ll ask for it automatically. Also ask you to download the ARCore by Google Application.

The code for MainActivity.java class is given below:

package com.example.myapplication;

import android.app.Activity;
import android.app.ActivityManager;
import android.content.Context;
import android.os.Build;
import android.os.Build.VERSION_CODES;
import android.os.Bundle;
import androidx.appcompat.app.AppCompatActivity;
import android.util.Log;
import android.view.Gravity;
import android.view.MotionEvent;
import android.view.View;
import android.view.Window;
import android.view.WindowManager;
import android.widget.AdapterView;
import android.widget.ArrayAdapter;
import android.widget.Button;
import android.widget.ImageButton;
import android.widget.ListPopupWindow;
import android.widget.SeekBar;
import android.widget.Spinner;
import android.widget.TextView;
import android.widget.Toast;
import com.google.ar.core.Anchor;
import com.google.ar.core.HitResult;
import com.google.ar.core.Plane;
import com.google.ar.core.Pose;
import com.google.ar.sceneform.AnchorNode;
import com.google.ar.sceneform.math.Quaternion;
import com.google.ar.sceneform.math.Vector3;
import com.google.ar.sceneform.rendering.ModelRenderable;
import com.google.ar.sceneform.ux.ArFragment;
import com.google.ar.sceneform.ux.TransformableNode;

import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Consumer;

import static java.lang.Math.atan2;

/**
* This is an example activity that uses the Sceneform UX package to make common AR tasks easier.
*/
public class MainActivity extends AppCompatActivity {
private static final String TAG = MainActivity.class.getSimpleName();
private static final double MIN_OPENGL_VERSION = 3.0;
private float x=0f, y=0f, z=0f;
private ArFragment arFragment;
private AnchorNode myanchornode;
TransformableNode mytranode = null;

private SeekBar sb_size;
private Spinner spn_model;

private HitResult myhit;
private float mySize = 70f;
private float mytravel=0.01f, distance_x=0f, distance_z=0f, myangle=0f;


int[] sfb_source = {R.raw.toy1, R.raw.toy2, R.raw.toy3, R.raw.toy4, R.raw.toy5, R.raw.toy6, R.raw.toy7, R.raw.toy8, R.raw.toy9};
String[] arr_models = {"Toy 1", "Toy 2", "Toy 3", "Toy 4", "Toy 5", "Toy 6", "Toy 7", "Toy 8", "Toy 9" };
private ModelRenderable[] renderable_models = new ModelRenderable[sfb_source.length];

@Override
@SuppressWarnings({"AndroidApiChecker", "FutureReturnValueIgnored"})
// CompletableFuture requires api level 24
// FutureReturnValueIgnored is not valid
protected void onCreate(Bundle savedInstanceState) {
requestWindowFeature(Window.FEATURE_NO_TITLE);
super.onCreate(savedInstanceState);

if (!checkIsSupportedDeviceOrFinish(this)) {
return;
}
try
{
this.getSupportActionBar().hide();
}
catch (NullPointerException e){}
setContentView(R.layout.activity_main);
getWindow().addFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN);
arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);

Button r_left = (Button)findViewById(R.id.r_left);
Button r_right = (Button)findViewById(R.id.r_right);
Button accelerate = (Button)findViewById(R.id.accelerate);
spn_model = (Spinner) findViewById(R.id.spn_model);
sb_size = (SeekBar) findViewById(R.id.sb_size);
List<AnchorNode> anchorNodes = new ArrayList<>();

sb_size.setEnabled(false);

// Keep immersive view on spinner opening
Field popup = null;
try {
popup = Spinner.class.getDeclaredField("mPopup");
popup.setAccessible(true);
ListPopupWindow popupWindow = (ListPopupWindow) popup.get(spn_model);
popupWindow.setModal(false);
} catch (NoSuchFieldException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
}



sb_size.setOnSeekBarChangeListener(new SeekBar.OnSeekBarChangeListener() {
@Override
public void onProgressChanged(SeekBar seekBar, int progress, boolean fromUser) {
mySize = progress;
myanchornode.setLocalScale(new Vector3(progress/70f, progress/70f, progress/70f));
}

@Override
public void onStartTrackingTouch(SeekBar seekBar) {
}

@Override
public void onStopTrackingTouch(SeekBar seekBar) {
}
});

accelerate.setOnTouchListener(new View.OnTouchListener() {
@Override
public boolean onTouch(View v, MotionEvent event) {
if (mytranode != null) {
if (event.getAction() == MotionEvent.ACTION_DOWN) {
myangle = set(mytranode.getLocalRotation());
}
if (event.getAction() == MotionEvent.ACTION_UP) {
}
forward(myanchornode);
}
return true;

}
});

r_left.setOnTouchListener(new View.OnTouchListener() {
@Override
public boolean onTouch(View v, MotionEvent event) {
if(mytranode != null){
Quaternion q1 = mytranode.getLocalRotation();
Quaternion q2 = Quaternion.axisAngle(new Vector3(0, 1f, 0f), .5f);
mytranode.setLocalRotation(Quaternion.multiply(q1, q2));
myangle = set(mytranode.getLocalRotation());
}

return true;
}

});

r_right.setOnTouchListener(new View.OnTouchListener() {
@Override
public boolean onTouch(View v, MotionEvent event) {
if(mytranode != null){
myangle+=0.01f;
Quaternion q1 = mytranode.getLocalRotation();
Quaternion q2 = Quaternion.axisAngle(new Vector3(0, 1f, 0f), -.5f);
mytranode.setLocalRotation(Quaternion.multiply(q1, q2));
myangle = set(mytranode.getLocalRotation());
}

return true;
}

});

for(int i = 0 ; i < sfb_source.length ; i++) {
int finalI = i;
ModelRenderable.builder()
.setSource(this, sfb_source[i])
.build()
.thenAccept(renderable -> renderable_models[finalI] = renderable)
.exceptionally(
throwable -> {
Toast toast =
Toast.makeText(this, "Unable to load andy renderable", Toast.LENGTH_LONG);
toast.setGravity(Gravity.CENTER, 0, 0);
toast.show();
return null;
});
}


arFragment.setOnTapArPlaneListener(
(HitResult hitResult, Plane plane, MotionEvent motionEvent) -> {
if (renderable_models[spn_model.getSelectedItemPosition()] == null) {
return;
}

distance_x=0f;
distance_z=0f;
myangle=0f;

myhit = hitResult;

// Create the Anchor.
Anchor anchor = hitResult.createAnchor();

AnchorNode anchorNode = new AnchorNode(anchor);


anchorNode.setParent(arFragment.getArSceneView().getScene());
anchorNodes.add(anchorNode);
sb_size.setEnabled(true);

myanchornode = anchorNode;

// Create the transformable andy and add it to the anchor.
TransformableNode andy;
if(mytranode == null)
andy = new TransformableNode(arFragment.getTransformationSystem());
else andy = mytranode;

andy.setParent(anchorNode);
andy.setRenderable(renderable_models[spn_model.getSelectedItemPosition()]);
andy.select();

mytranode = andy;
mytranode.setLocalRotation(new Quaternion(0f, 0f, 0f, 1f));
myanchornode.setLocalScale(new Vector3(mySize/70f, mySize/70f, mySize/70f));
});

ArrayAdapter<String> adapter = new ArrayAdapter<String>(MainActivity.this,
android.R.layout.simple_spinner_dropdown_item,arr_models);

spn_model.setAdapter(adapter);
spn_model.setOnItemSelectedListener(new AdapterView.OnItemSelectedListener() {
@Override
public void onItemSelected(AdapterView<?> adapterView, View view, int i, long l) {
if(mytranode != null)
mytranode.setRenderable(renderable_models[i]);
}

@Override
public void onNothingSelected(AdapterView<?> adapterView) {
}
});

}

void ascend(AnchorNode an, float x, float y, float z){
Anchor anchor = myhit.getTrackable().createAnchor(
myhit.getHitPose().compose(Pose.makeTranslation(x/100f, z/100f, y/100f)));

an.setAnchor(anchor);
}

Quaternion rotate(AnchorNode an, float angle) {
//mytranode.setLocalRotation(Quaternion.axisAngle(new Vector3(0, 1f, 0), angle));

return mytranode.getLocalRotation();
}


public float set(Quaternion q1) {
Vector3 angles = new Vector3();
double sqw = q1.w*q1.w;
double sqx = q1.x*q1.x;
double sqy = q1.y*q1.y;
double sqz = q1.z*q1.z;
double unit = sqx + sqy + sqz + sqw; // if normalised is one, otherwise is correction factor
double test = q1.x*q1.y + q1.z*q1.w;
if (test > 0.499*unit) { // singularity at north pole
angles.x = (float) (2 * atan2(q1.x,q1.w));
angles.y = (float) Math.PI/2;
angles.z = 0;
return angles.x;
}
if (test < -0.499*unit) { // singularity at south pole
angles.x = (float) (-2 * atan2(q1.x,q1.w));
angles.y = (float) (-Math.PI/2);
angles.z = 0;
return angles.x;
}
angles.x = (float) atan2(2*q1.y*q1.w-2*q1.x*q1.z , sqx - sqy - sqz + sqw);
angles.y = (float) Math.asin(2*test/unit);
angles.z = (float) atan2(2*q1.x*q1.w-2*q1.y*q1.z , -sqx + sqy - sqz + sqw);
return angles.x;
}


void forward(AnchorNode an){
distance_x+=Math.sin(myangle)*mytravel;
distance_z+=Math.cos(myangle)*mytravel;

Anchor anchor = myhit.getTrackable().createAnchor(
myhit.getHitPose().compose(Pose.makeTranslation(-distance_x, 0f, -distance_z)));

an.setAnchor(anchor);
}

float getMetersBetweenAnchors(Anchor anchor1, Anchor anchor2) {
float[] distance_vector = anchor1.getPose().inverse()
.compose(anchor2.getPose()).getTranslation();
float totalDistanceSquared = 0;
for(int i=0; i<3; ++i)
totalDistanceSquared += distance_vector[i]*distance_vector[i];
return (float) Math.sqrt(totalDistanceSquared);
}

public static boolean checkIsSupportedDeviceOrFinish(final Activity activity) {
if (Build.VERSION.SDK_INT < VERSION_CODES.N) {
Log.e(TAG, "Sceneform requires Android N or later");
Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();
activity.finish();
return false;
}
String openGlVersionString =
((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))
.getDeviceConfigurationInfo()
.getGlEsVersion();
if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {
Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");
Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
.show();
activity.finish();
return false;
}
return true;
}
}

The anchor is the position where the model or the Node is placed in the screen. It’s the center of the screen here.

In the addObject() the ARFragment gets the hit points on the plane using motion tracking.

placeObject() asynchronously sets the node to the center of the screen.

There you have it! Your own AR App in Android studio :)

I hope you had fun reading and/or following along. In the next story, we will look into how to build more features and interactions into this App. Stay tuned!

If you are interested in further exploring, here are some resources I found helpful along the way:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store