Edit

Performance Tuning

Performance Test

OvenMediaEngine provides a tester for measuring WebRTC performance called OvenRtcTester. It is developed in Go language and uses the pion/webrtc/v3 and gorilla/websocket modules. Many thanks to the pion/webrtc and gorilla/websocket teams for contributing this wonderful project.

Getting Started OvenRtcTester

Install GO

Since OvenRtcTester is developed in Go language, Go must be installed on your system. Install Go from the following URL: https://golang.org/doc/install

OvenRtcTester was tested with the latest version of go 1.17.

Run

You can simply run it like this: -url is required. If the -life option is not used, it will run indefinitely until the user presses ctrl+c.

$ cd OvenMediaEngine/misc/oven_rtc_tester
$ go run OvenRtcTester.go
-url parameter is required and must be vaild. (input : undefined)
  -cint int
        [Optional] PeerConnection connection interval (milliseconds) (default 100)
  -life int
        [Optional] Number of times to execute the test (seconds)
  -n int
        [Optional] Number of client (default 1)
  -sint int
        [Optional] Summary information output cycle (milliseconds) (default 5000)
  -url string
        [Required] OvenMediaEngine's webrtc streaming URL (default "undefined")

You can also use go build or go install depending on your preference.

OvenRtcTester must test OvenMediaEngine 0.12.4 or higher as the target system. OvenMediaEngine versions below 0.12.4 have a problem with incorrectly calculating the RTP timestamp, so OvenRtcTester calculates the Video Delay value incorrectly.

Performance Tuning

Monitoring the usage of threads

Linux has various tools to monitor CPU usage per thread. We will check the simplest with the top command. If you issue the top -H -p [pid] command, you will see the following screen.

You can use OvenRtcTester to test the capacity of the server as shown below. When testing the maximum performance, OvenRtcTester also uses a lot of system resources, so test it separately from the system where OvenMediaEngine is running. Also, it is recommended to test OvenRtcTester with multiple servers. For example, simulate 500 players with -n 500 on one OvenRtcTester, and simulate 2000 players with four servers.

Building and running OvenMediaEngine in debug mode results in very poor performance. Be sure to test the maximum performance using the binary generated by make release && make install .

If the OvenMediaEngine's capacity is exceeded, you will notice it in OvenRtcTester's Summary report with Avg Video Delay and Avg Audio Delay or Packet loss.

On the right side of the above capture screen, we simulate 400 players with OvenRtcTester. <Summary> of OvenRtcTester shows that Avg Video Delay and Avg Audio Delay are very high, and Avg FPS is low.

And on the left, you can check the CPU usage by thread with the top -H -p command. This confirms that the StreamWorker threads are being used at 100%, and now you can scale the server by increasing the number of StreamWorker threads. If OvenMediaEngine is not using 100% of all cores of the server, you can improve performance by tuning the number of threads.

This is the result of tuning the number of StreamWorkerCount to 8 in config. This time, we simulated 1000 players with OvenRtcTester, and you can see that it works stably.

Tuning the number of threads

The WorkerCount in <Bind> can set the thread responsible for sending and receiving over the socket. Publisher's AppWorkerCount allows you to set the number of threads used for per-stream processing such as RTP packaging, and StreamWorkerCount allows you to set the number of threads for per-session processing such as SRTP encryption.

Scalable Threads and Configuration

Thread name

Element in the configuration

AW-XXX

<Application><Publishers><AppWorkerCount>

StreamWorker

<Application><Publishers><StreamWorkerCount>

SPICE-XXX

<Bind><Provider><WebRTC><IceCandidates><TcpRelayWorkerCount>

<Bind><Pubishers><WebRTC><IceCandidates><TcpRelayWorkerCount>

SPRtcSignalling

<Bind><Provider><WebRTC><Signalling><WorkerCount>

<Bind><Pubishers><WebRTC><Signalling><WorkerCount>

SPSegPub

<Bind><Pubishers><HLS><WorkerCount>

<Bind><Pubishers><DASH><WorkerCount>

SPRTMP-XXX

<Bind><Providers><RTMP><WorkerCount>

SPMPEGTS

<Bind><Providers><MPEGTS><WorkerCount>

SPOvtPub

<Bind><Pubishers><OVT><WorkerCount>

SPSRT

<Bind><Providers><SRT><WorkerCount>

AppWorkerCount

Type

Value

Default

1

Minimum

1

Maximum

72

With AppWorkerCount, you can set the number of threads for distributed processing of streams when hundreds of streams are created in one application. When an application is requested to create a stream, the stream is evenly attached to one of created threads. The main role of Stream is to packetize raw media packets into the media format of the protocol to be transmitted. When there are thousands of streams, it is difficult to process them in one thread. Also, if StreamWorkerCount is set to 0, AppWorkerCount is responsible for sending media packets to the session.

It is recommended that this value does not exceed the number of CPU cores.

StreamWorkerCount

Type

Value

Default

8

Minimum

0

Maximum

72

It may be impossible to send data to thousands of viewers in one thread. StreamWorkerCount allows sessions to be distributed across multiple threads and transmitted simultaneously. This means that resources required for SRTP encryption of WebRTC or TLS encryption of HLS/DASH can be distributed and processed by multiple threads. It is recommended that this value not exceed the number of CPU cores.

Use-Case

If a large number of streams are created and very few viewers connect to each stream, increase AppWorkerCount and lower StreamWorkerCount as follows.

If a small number of streams are created and a very large number of viewers are connected to each stream, lower AppWorkerCount and increase StreamWorkerCount as follows.

Last updated

Was this helpful?