JSON schema/example for hwa.jellyfin.org

[mrkz]

Hello,

I apologize if this is not the right place to ask about this topic.

I'm looking forward to populating values for an Intel N100 CPU in hwa.jellyfin.org, I did try 2 different repositories:

* joshuaboniface/hwatest
* BotBlake/pyTAB (which, from my understanding would be the official repository for the task).

I was not able to get them running as-is. After some debugging, I was able to get the first one running, but it appears that the JSON output I'm getting may be outdated from what the web page expects (I've attached the output to this post, please forgive the *.txt file extension).

I'm now looking for an updated schema or a JSON example of what the page may expect so that I can upload these results. I also did give preference to the first one because it uses the ffmpeg from the installed jellyfin package (which seems to be more up to date vs the version PyTAB downloads).

Thanks all for your time!

[crobibero]

You should join the HWA tool channel.

#jellyfin-hwa-tool:matrix.org

[jkboi]

Hello,

I apologize if this is not the right place to ask about this topic.

I'm looking forward to populating values for an Intel N100 CPU in hwa.jellyfin.org, I did try 2 different repositories:

* joshuaboniface/hwatest
* BotBlake/pyTAB (which, from my understanding would be the official repository for the task).

I was not able to get them running as-is. After some debugging, I was able to get the first one running, but it appears that the JSON output I'm getting may be outdated from what the web page expects (I've attached the output to this post, please forgive the *.txt file extension).

I'm now looking for an updated schema or a JSON example of what the page may expect so that I can upload these results. I also did give preference to the first one because it uses the ffmpeg from the installed jellyfin package (which seems to be more up to date vs the version PyTAB downloads).

Thanks all for your time!

Thank you so much for being excited to try this out! Like Crobibero pointed out above the place to talk about it would be in the either discord or matrix channels that he mentioned.

The pyTAB client is currently very unstable so this is to be expected. If you come to one of these rooms either b0tBlake, I (jkboy), or any others that are there will hopefully be able to help you out.

[mrkz]

Thank you so much for being excited to try this out! Like Crobibero pointed out above the place to talk about it would be in the either discord or matrix channels that he mentioned.

The pyTAB client is currently very unstable so this is to be expected. If you come to one of these rooms either b0tBlake, I (jkboy), or any others that are there will hopefully be able to help you out.

Thanks both for the quick reply and the direction!

I did look for a #jellyfin-hwa-tool:matrix.org channel and was able to find several #jellyfin-dev* and other rooms, but I had no luck with the one I got pointed out earlier. I'm looking forward to contributing bug reports (and patches), but I'd love to communicate over Matrix instead of Discord if it's possible.

If I'm still unsuccessful in connecting to a chat channel, I'll probably just start by creating the issue on PyTAB repo, as I already have some data to share about the failures I'm getting with the tool, but I'd rather put such details better refined after a short chat on what'd be useful info for you in the report.

Again, thanks for your replies and time.

[BotBlake]

Hello there

I thought I’d throw in my two cents as well.
It's awesome to have new testers/supporters on board! 😄

To answer your questions:

pyTAB should™ run automatically...
We’re aware of a few issues, but none that would block the entire program flow.

However, even pyTAB’s JSON files can’t be uploaded directly via the Web UI – the Web UI doesn’t accept None. Upload via API – specifically the Swagger Web UI – works, though! (pyTAB doesn’t support upload via API yet – PRs welcome 🗿)

Feel free to share more details about your issues with us!
For discussions, the Matrix/Discord channels mentioned earlier are the best places to go.


Now about hwatest:
Did you know that pyTAB started as a fork of hwatest?
https://github.com/joshuaboniface/hwates...est:master

hwatest is a standalone program that doesn’t work in combination with the Hardware Survey Server. It uses preconfigured test files and FFmpeg commands, whereas pyTAB pulls all that info directly from the hwa server.
(By the way, even the FFmpeg version to use! 😉)

So, it’s not possible to adjust hwatest’s output to make it compatible with the Server.
If you're still curious about the output format, you can find a description in the hwa server’s GitHub repository
https://github.com/JPVenson/Jellyfin.Har...esult-data.

Oh and or course what jkboy and cody where saying 😁

If you have any more questions, feel free to ask.
Kind regards, Blake
PS: Great, I think this is my first ever post on the Forum 😆

[mrkz]

Hello there
I thought I’d throw in my two cents as well.
It's awesome to have new testers/supporters on board! 😄

Thanks for the warm and positive welcome.

pyTAB should™ run automatically...
We’re aware of a few issues, but none that would block the entire program flow.
However, even pyTAB’s JSON files can’t be uploaded directly via the Web UI – the Web UI doesn’t accept None. Upload via API – specifically the Swagger Web UI – works, though! (pyTAB doesn’t support upload via API yet – PRs welcome 🗿)

It appears to be an error about something about the drm2 driver (pyTAB is obscuring the error as a generic ffmpeg error if I recall correctly). That may be because I'm using Fedora Silverblue + an Ubuntu 22.04 container (via distrobox). Either way, I'll try to reproduce either on a VM or direct ubuntu host OS.

Feel free to share more details about your issues with us!
For discussions, the Matrix/Discord channels mentioned earlier are the best places to go.

I'm still having issues finding the #jellyfin-hwa-tool:matrix.org channel (maybe it's a private one?), Either way, I'm reachable via matrix.org too (@mrkzboo:matrix.org)

Now about hwatest:
Did you know that pyTAB started as a fork of hwatest?
https://github.com/joshuaboniface/hwates...est:master
hwatest is a standalone program that doesn’t work in combination with the Hardware Survey Server. It uses preconfigured test files and FFmpeg commands, whereas pyTAB pulls all that info directly from the hwa server.
(By the way, even the FFmpeg version to use! 😉)

I did notice some similarities when I was looking at the code (i.e: I had to force-print the ffmpeg_stderr to see why it was failing),plus I did some git archeology to understand which was born first, etc. Good to know!
I did like hwatest because by default looked at the ffmpeg installed version (whereas pyTAB gets a custom one from the Internet if I'm not mistaken). Doing so also helped me to notice that there's some input/output ffmpeg changes for 7.1 (I did notice pyTAB used ffmpeg 6.x) .
Plus, I'm not a big fan of full cloud/Internet dependency for running tools (e.g: downloading content / uploading results). I do understand that this the the whole propose of the tool, so I can't say much in that aspect for this scenario

So, it’s not possible to adjust hwatest’s output to make it compatible with the Server.
If you're still curious about the output format, you can find a description in the hwa server’s GitHub repository
https://github.com/JPVenson/Jellyfin.Har...esult-data.
Oh and or course what jkboy and cody where saying 😁
If you have any more questions, feel free to ask.
Kind regards, Blake
PS: Great, I think this is my first ever post on the Forum 😆

Ah, that whitepaper it's gold. My first goal was to adapt JSON from hwatest to submit directly to the page. but if I'm to invest time, I'd rather do it on the proper tool, so I'm looking forward to keep the engagement on the helping!
PS: All this because a friend asked about my device's performance for jellyfin to compare and I wanted to have an apple-apple comparison between devices. Hopefully, when I get this running we'll be able to speak the same language (perf. data).

[BotBlake]

I messaged you on Matrix to point you in the right direction.
I hope that helps you find our dev room 😊

However I can already give some more detail about the features and issues behind pyTAB.
As jkboy already mentioned, it should be noted that pyTAB is still in development, therefore no release has been made yet.

It appears to be an error about something about the drm2 driver (pyTAB is obscuring the error as a generic ffmpeg error if I recall correctly).

This here is an issue we are currently fighting with. I dont know if you are experiencing the exact issue, but I am sure we will be abled to sort things out together in the Dev Chat at a later point. Generaly we currently have issues with the ffmpeg commands/ device selection - so a lot of current tests fail with weird reasons. We are on it, however the pyTAB crew and the hwa Server dev are currently pretty busy with other things 😅

I did notice some similarities when I was looking at the code (i.e: I had to force-print the ffmpeg_stderr to see why it was failing),plus I did some git archeology to understand which was born first, etc. Good to know!

If you are on the latest Version of pyTAB, it SHOULD create a seperate ffmpeg_err_log.txt file, that contains all Error Logs of ffmpeg.
Also pyTAB is trying to interpret the ffmpeg Error output and give a more detailed error reason itself. If the errors are too cryptic, it will default back to "generic_ffmpeg_error", hence your bad experience here.

I did like hwatest because by default looked at the ffmpeg installed version (whereas pyTAB gets a custom one from the Internet if I'm not mistaken). Doing so also helped me to notice that there's some input/output ffmpeg changes for 7.1 (I did notice pyTAB used ffmpeg 6.x) .
Plus, I'm not a big fan of full cloud/Internet dependency for running tools (e.g: downloading content / uploading results).

Now about some cool hidden features 😝
pyTAB could run fully local. It just expects SOMETHING to tell it what to do.
We just decided to not explicitly advertise that feature, as it would completely break compatibility with the hwa server.

Generaly pyTAB only needs the 3 informations to run:
1. a link to a ffmpeg binary to use
2. test files to use
3. commands to benchmark

All of theese things are provided by the hwa.jellyfin.org server by default.
(however the --server "URL" option could override it)

At the moment we use the exact same test files as hwatest and download them from the jellyfin repo:
https://repo.jellyfin.org/jellyfish/
When they already are present in the respective download folder, they will not automatically be downloaded again.

The ffmpeg binary we use also isnt "custom". We use jellyfin-ffmpeg and download from the jellyfin repo as well.
We only use the older ffmpeg release, because we didnt find the time to bump the versions yet. Also its totaly possible that a newer version MIGHT break our commands, so bumping the version could take some time.

We originaly wanted to make the output of pyTAB be interpretable without the server.
For now we just did not put our efforts into making it work.

However you will never HAVE TO upload your test results.
Also the Test results will NEVER contain personal Data.

Ah, that whitepaper it's gold. My first goal was to adapt JSON from hwatest to submit directly to the page. but if I'm to invest time, I'd rather do it on the proper tool, so I'm looking forward to keep the engagement on the helping!

Looking forward to work with you. Any collaboration is absolutely welcome.
Also glad if you found the tooling to be usefull or even potentialy usefull 😁

PS: All this because a friend asked about my device's performance for jellyfin to compare and I wanted to have an apple-apple comparison between devices. Hopefully, when I get this running we'll be able to speak the same language (perf. data).

I am realy curious: How did you even find this Project without being in our dev room? As far as I know, it was never specifically advertised yet, as its not even in Alpha.


Anyways, I hope we will hear each other soon on the Matrix 😆

PS: If you actualy want to compare an APPLE to an APPLE... Note that we are still working on apple support 😅
However we actualy HAVE put thought into how to support apple! - its on the ToDo List.

Oh I almost forgot 🗿

If you want to see what we are working on on pyTAB, you should defenetly take a look at our ToDo List:
https://github.com/users/BotBlake/projects/2