Tutorials¶
Preparing a simple Detector project
Using your own events and callback function
- Write your own callback function
- Use your own event
- Test your application
Using your own input source
- Preparing your movie file for a streaming
- Use your own stream ready movie file
Using your own trained model binary
- Place your trained model binary and related files
- Change property configurations
Using your own trained Yolo model binary with IPlugin
- Follow the Yolo tutorial on DeepStream SDK
- Place your trained Yolo model binary and related files
- Change property configurations
Making a submission package
- What is a submission package?
- How to make a submission package?
Preparing a simple Detector project¶
In this tutorials, a simple vehicle detector will be customized. So, please create one as follows.
At first, create a simple vehicle detector from a vehicle template.
Then, modify the application name to reflect this application.
And change the resolution of the input.
The template has a tracker and a secondary model inference as well as a primary inference.
So, let’s remove all the properties related to a tracker and a secondary inference by selecting each property followed by Remove.
Then, modify event items for a detector as follows.
Please note that now this application raises events about a location of a detected object, instead of properties of each identified object by a tracker.
Now, it’s time to save these changes. Click Save button, then the sdk will reload applications.
Using your own events and callback function¶
In the last section, event items are modified for the detector.
So, you need to update the callback function as well.
The python file that defines a callback function exists as emi_signal_callback.py in an application folder as follows.
/mnt/nvme/toolkit_home$ ls -l applications/
total 4
drwxr-xr-x 3 nvidia nvidia 4096 Apr 24 11:41 'My Vehicle Detector'
/mnt/nvme/toolkit_home$ ls -l applications/My\ Vehicle\ Detector/
total 36
-rw-r--r-- 1 nvidia nvidia 6905 Feb 26 00:53 emi_signal_callback.py
-rw-r--r-- 1 nvidia nvidia 1317 Apr 24 11:41 emi_stream_config.json
-rw-r--r-- 1 nvidia nvidia 1543 Apr 24 11:41 emi_stream_config.json.bak
-rw-r--r-- 1 nvidia nvidia 13271 Dec 24 23:42 icon.png
drwxr-xr-x 3 nvidia nvidia 4096 Apr 10 14:36 resource
Write your own callback function¶
Copy the following content to the emi_signal_callback.py.
from datetime import datetime
ISO_FORMAT = '%Y-%m-%dT%H:%M:%S.%f%z'
'''
Vehicle Detector
Keys:
detected_datetime (string): The datetime when this object was detected
left (number): The left coordinate of this object
top (number): The top coordinate of this object
width (number): The width of this object
height (number): The height of this object
'''
class Car:
def __init__(self, detected_timestamp, left, top, width, height, class_id, confidence):
self.detected_timestamp = detected_timestamp
self.left = left
self.top = top
self.width = width
self.height = height
self.class_id = class_id
self.confidence = confidence
def to_event_item(self):
event_item = {
'detected_timestamp': self.detected_timestamp,
'left': self.left,
'top': self.top,
'width': self.width,
'height': self.height,
'class_id': self.class_id,
'confidence': self.confidence
}
return event_item
def iso_timestamp_to_datetime(timestamp):
return datetime.strptime(timestamp, ISO_FORMAT)
def update_tracking(signal):
""" a signal callback function """
debug_string = ''
detected_cars = []
frame_list = signal["frame"]
for frame in frame_list:
timestamp = frame['timestamp']
objects = frame["object"]
debug_string = debug_string + 'signal@' + timestamp + ':' + str(len(objects)) + 'objects\n'
for obj in objects:
class_id = obj['class_id']
confidence = obj['confidence']
rect_params = obj['rect_params']
left = rect_params['left']
top = rect_params['top']
width = rect_params['width']
height = rect_params['height']
car = Car(timestamp, left, top, width, height, class_id, confidence)
detected_cars.append(car.to_event_item())
return detected_cars, debug_string
The callback function name was left as update_tracking, but the whole content was replaced.
Let’s go back to the Toolkit, and check if this callback works correctly by pressing Spell Check.
Oops, failed. If you look at your console, you’ll see an output like this.
It says “confidence” does not exist in the produced event, which is based on a template. So, let’s create our own event and use it for this check.
Use your own event¶
Copy the following content to detector_signal.json in the signals folder under the Toolkit root folder.
If you happen to place such a file in an application folder, it wouldn’t work correctly.
{
"frame": [
{
"frame": 1,
"pts": 1,
"timestamp": "2000-01-01T00:00:00.000000+0900",
"object": [
{
"class_id": 0,
"confidence": 0.0,
"rect_params": {
"left": 0,
"top": 0,
"width": 0,
"height": 0
}
}
]
}
]
}
Note that another missing key, rect_params, was also added.
Then, try again Spell Check. This time, make sure to choose detector_signal.json.
By pressing Execute, you’ll see your application pass the check.
Test your application¶
So, finally, let’s test your application.
But, you need to create a stream folder to run this application.
Copy an existing vehicle_stream folder and name it mydetector_stream.
Now the folder structure should look like this.
/mnt/nvme/toolkit_home$ ls -l streams/
total 56
drwxr-xr-x 2 nvidia nvidia 4096 Jun 20 09:13 doubleeap_emcustom
drwxr-xr-x 2 nvidia nvidia 4096 May 13 04:13 face_net
drwxr-xr-x 2 nvidia nvidia 4096 Jun 20 09:13 line_stream
drwxr-xr-x 2 nvidia nvidia 4096 Jun 20 10:41 mydetector_stream
drwxr-xr-x 2 nvidia nvidia 4096 May 12 08:44 no_app_stream
drwxr-xr-x 5 nvidia nvidia 4096 Jun 20 09:13 pedestrian_stream
drwxr-xr-x 2 nvidia nvidia 4096 Jun 11 08:43 pedestrian_stream_bottomleft
drwxr-xr-x 2 nvidia nvidia 4096 Jun 11 08:43 pedestrian_stream_upperleft
drwxr-xr-x 2 nvidia nvidia 4096 Jun 11 08:43 pedestrian_stream_upperright
drwxr-xr-x 2 nvidia nvidia 4096 Jun 20 09:13 snmp_stream
drwxr-xr-x 5 nvidia nvidia 4096 Jun 20 09:13 vehicle_colorwatcher_stream
drwxr-xr-x 5 nvidia nvidia 4096 Jun 20 09:46 vehicle_stream
drwxr-xr-x 2 nvidia nvidia 4096 May 13 09:50 yolo_stream
drwxr-xr-x 2 nvidia nvidia 4096 Jun 11 08:43 yolo_stream_bottomright
/mnt/nvme/toolkit_home$ ls -l streams/mydetector_stream/
total 8
-rw-r--r-- 1 nvidia nvidia 1264 Jun 20 10:40 vehicle_counter_stream_configuration.json
-rw-r--r-- 1 nvidia nvidia 1515 Jun 20 10:40 vehicle_counter_stream_configuration_with_options.json
If you find any other files or folders when you come from the quickstart,
then remove all the files except for vehicle_by_make_counter_stream_configuration.json.
Rename vehicle_by_make_counter_stream_configuration.json as mydetector_stream_configuration.json,
then copy the following content.
{
"stream_id": "mydetector_stream",
"created": "2019-07-23T09:10:29.842496+09:00",
"last_updated": "2019-07-24T10:11:30.842496+09:00",
"revision": 3,
"stream_type": "rtsp",
"location": "rtsp://127.0.0.1:8554/test",
"mode": "sender",
"roi": {
"left": 0,
"right": 0,
"top": 0,
"bottom": 0
},
"action_rules": [
{
"rule_name": "Vehicle Recording",
"and": [
{
"key": "width",
"operator": ">",
"value": 100
},
{
"key": "height",
"operator": ">",
"value": 100
}
],
"or": [],
"action": {
"action_name": "record",
"duration_in_seconds": 3
}
},
{
"rule_name": "Upload to AWS Kinesis Firehose",
"and": [
{
"key": "width",
"operator": ">",
"value": 100
},
{
"key": "height",
"operator": ">",
"value": 100
}
],
"or": [],
"action": {
"action_name": "upload",
"deliveryStreamName": "trafficStream",
"accessKey": "",
"secretKey": "",
"region": ""
}
}
],
"application_package": {
"filename": "mydetector.zip",
"license": "ABC01234"
}
}
By executing this application in the mydetector_stream folder with the sample video file,
it will be shown as follows, which correctly produces upload actions for each event only when both of an width and an height are larger than 100.
Also, recording actions will be invoked, and leave some movie files in the recordings folder.
/mnt/nvme/toolkit_home/streams/mydetector_stream$ ls -l prerecordings/
total 17912
-rw-r--r-- 1 nvidia nvidia 18239015 Jun 20 10:45 mydetector_stream_21841_prerecord_0_2020-06-20T10:44:50+0900.mp4
-rw-r--r-- 1 nvidia nvidia 99418 Jun 20 10:45 mydetector_stream_21841_prerecord_0_2020-06-20T10:45:38+0900.mp4
Using your own input source¶
Using your own movie file is no more than choosing your own file when executing your application.
But making a movie file needs to follow some rules.
Preparing your movie file for a streaming¶
A movie file chosen at an execution is used internally as a source of local RTSP server.
Such a movie file contianer needs to be mp4. Other containers may work, but not tested well.
There are some requirements for making your movie file stream ready in the Toolkit.
- H.264 (YUV 4:2:0) video encoding
- faststart (MOOV atom at the beginning of a file instead of at the end)
- constant bit rate up to 4Mbps
This can be done with ffmpeg, not on the Toolkit box, but on your any host computer, with a command as follows.
$ ffmpeg -i INPUT -c:v libx264 -pix_fmt yuv420p -b:v 4M -maxrate 4M -bufsize 4M -movflags +faststart OUTPUT
Using your own trained Yolo model binary with IPlugin¶
If you have your own trained Yolo model, you can refer to the following guide by NVIDIA.
Custom YOLO Model in the DeepStream YOLO App
Here in this tutorial, you will see how to package a sample Yolo detector contained in the DeepStream.
Follow the Yolo tutorial on the DeepStream SDK¶
Sample files of the deepsteram are stored on /opt/nvidia/deepstream.
The Yolo sample project is located at /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo. You can build the project by simply following the README file as follows.
$ ./prebuild.sh
$ export CUDA_VER=10.0
$ make -C nvdsinfer_custom_impl_Yolo
Then, launch the deepstream-app to check if it correctly works. Also, at this initial launch, a TensorRT engine file will be created.
$ deepstream-app -c deepstream_app_config_yoloV3_tiny.txt
Note that the Tiny Yolo V3 application runs as fast as about 50 fps in FP32 mode on Jetson TX2. You can try different Yolo versions to see their performances.
The configuration of the tiny Yolo V3 will be used here in the following sections.
Change property configurations¶
The only property you have to change is config-file-path in the Primary.
After changing the property, save the config. Then, open config_infer_primary_yoloV3_tiny.txt, and update properties as follows. Please make sure to remove the comment on the model-engine-file property, and add “.gpg” suffixes.
/mnt/nvme/toolkit_home/applications/My Yolo Detector/resource$ diff config_infer_primary_yoloV3_tiny.txt /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/config_infer_primary_yoloV3_tiny.txt
65,68c65,68
< custom-network-config=models/Primary_Detector/yolov3-tiny.cfg
< model-file=models/Primary_Detector/yolov3-tiny.weights.gpg
< model-engine-file=models/Primary_Detector/model_b1_fp32.engine.gpg
< labelfile-path=models/Primary_Detector/labels.txt
---
> custom-network-config=yolov3-tiny.cfg
> model-file=yolov3-tiny.weights
> #model-engine-file=model_b1_fp32.engine
> labelfile-path=labels.txt
76c76
< custom-lib-path=libnvdsinfer_custom_impl_Yolo.so
---
> custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
By following the procedures as before, your application can be launched in the mydetector_stream as below.
Actions)
Debug Window)
Making a submission package¶
When you become confident that you app is ready to ship, you can make a submission package.
What is a submission package?¶
A submission package contains all the necessary files for EDGEMATRIX to prepare your app for sale.
This includes:
- an EAP file including encrypted model binaries
- the passphrase (used to encrpt your model binaries) encrypted with your device credential
- a stream config file used for the last test
- a movie file used for the last test
- a text file containing version information
How to make a submission package?¶
After successfully running your test, the Network button placed between the Stop button and the Close button becomes active.
Press this Network button to make a submission pacakge from the last test. Then, the save thread starts, and which will ask a sudo password in order to access the device credential to encrypt your pass phrase as below.
Then, you will find a folder that contains all the necessary files to submit as below.
/mnt/nvme/toolkit_home$ ls -l submissions/My\ Yolo\ Detector/
total 198928
-rw-r--r-- 1 nvidia nvidia 142 Jun 20 11:36 C0210001_encrypted.json
-rw-r--r-- 1 nvidia nvidia 129384358 Jun 20 11:36 ChuoHwy-720p-faststart.mp4
-rw-r--r-- 1 nvidia nvidia 1297 Jun 20 11:36 mydetector_stream_configuration.json
-rw-r--r-- 1 nvidia nvidia 74304890 Jun 20 11:36 mydetector.zip
-rw-r--r-- 1 nvidia nvidia 248 Jun 20 11:36 versions.json