Jellyfin Organisational Enhancement Proposal (JOEP)


#1

So, to make all our stuff easier to manage and build in a homogeneous way. The idea was there to split out the packaging scripts into a separate repo.

This is an implementation proposal of that. Please leave your comments/suggestions/critique/other proposals below. This really adds some complexity for us now, but this should make the bar of entry for new developers and users that just want to build, much easier.


We would have one main entry repository with more ‘child’ repositories. There will be no submodules, anywhere.

jellyfin
\_ jellyfin-server
|_ jellyfin-web
|_ jellyfin-android
|_ etc.

Main repository jellyfin
The main repository contains the packaging scripts and everything required to build the full project. The main build script will either take an existing repository path or clone the required repository into its own directory. This gives existing developers the flexibility to have their repositories any place they want.[1]
The main projects script will handle cloning or building any dependencies of the chosen target, so for example if the server is being built and the web repository has not yet been cloned or build, it will do that first, given the configuration constraints specified on either the command line or a configuration file. This configuration file will also help with making sure that if you have your repository in a location that is not the default you will not have to enter it in the terminal every time.

Server repository jellyfin-server
This is the repository where all the server code lives. It contains all source files to build the server binaries and the build system (dotnet/MSBuild) that is required. A functioning install will need the files from the jellyfin-web repository. If people do not want to use the main repository to set it all up for them, they can manually point their server binaries to the place the files are located if the suggestion in [1] is implemented.

Web repository jellyfin-web
This is the repository where all the webclient code lives. It will contain all files required for it’s own build system (yarn+webpack).

Any other repository ex: jellyfin-android
Any other repository containing a specific client or plugin for example can be built using the same methods and the main server binaries or the web application. This would mean that in the end, and if run on a supported platform. There will be a make me everything command, building every supported project for the target platform or all supported platforms.

Active development options
The main build script by default should shallow clone every repository at the last tag, but for development a regular clone is required. This would be configurable when bootstrapping the main repository the initial time and be written to a configuration file. This could even be done interactively.
This would also be able to setup the correct paths for development on different parts, if required. [2] This ensures that testing for example web changes does not require one to recompile the whole server. But instead only a browser refresh.


[1] Adding a command line argument to configure the root of the web files to the server will help this along even more. An other option is using an environment variable, for this and other paths.
[2] Think about a docker-compose.yml file, to specify the correct paths to volumes automatically and to make developing as easy as calling docker-compose up. The same effect can be realized for non docker development aswell.


These are just some of my quick thoughts. I definitely missed stuff from the matrix discussion, so if you miss anything, just write below.


#2

I’m generally good with this idea. Some specific thoughts:

  1. Definitely in favour of a Python build setup. I don’t think there’s too much logical complexity in deciding how to build each type of package, since there’s really just 2: "direct dotnet package", and “Docker build with artifacts”. Sprinkle a bit of YAML config into each directory to give some specifics and it’s a perfect cross-platform build setup. I’m assuming though that Python 3’s subprocess module actually works on Windows…

  2. If we do it this way, I think SharedVersion.sh should definitely follow the pkg-src directories, i.e. be in the deployment repo rather than the Main repo. The downside is this prevents the main repo from building without being inside the package framework though, so I’m not sure if that’s a dealbreaker or not. I say this because otherwise it involves updating the versioning across two repos, which is a hassle. Would need to be better thought-out by the C# team.

  3. This is super powerful for building the apps and plugins - we need a centralized and efficient way to get the plugins built especially given how many there are. And if we’re Dockerizing all the app builds anyways, it fits in with #1 perfectly. Win-win.

  4. This would let us remove a few stale build types, for instance Debian/Ubuntu vs Linux amd64 builds - they’re almost exactly the same size artifact-wise, and aside from a shared library or two are basically identical. And on the flip side this makes it really easy to add additional Dockerized platform types, such as for individual Debian/Ubuntu or Fedora releases, and update these outside the main repository.

I’ve taken a look at how a few Debian source packages using git handle this, and almost all of them are manually importing upstreams from tarball, so I see no reason we can’t follow the same logic with git from a central build script.


#3

Definitely in favour of a Python build setup.

Agreed.

I’m assuming though that Python 3’s subprocess module actually works on Windows…

Certainly seems too:

All in I like @EraYaN 's suggestions and agree.


#4

I’ve actually slightly reconsidered by positioning on the naming and which repo is “main”.

The repo called “jellyfin” already has tons of traction - stars, forks, people using it, etc. So I think renaming or, or significantly altering it, would be a mistake and cause a lot of confusion. This does complicate things a bit with how to split it out, especially with my desire to keep the changelogs and sharedversion (and bump_version) in the same repo.

I want to step back and focus on the original X - what reasons do we have for wanting to split packaging out of this repo? I know I am an original proponent of doing this, but my main desire was to avoid some of the kerfuffle that the feature-branch model would have caused (e.g. usually wanting build changes on master quickly) - by moving to a master-primary setup, this disappears and packaging changes can be easily handled within the “server”/main repo easily. It’s also the primary build target.

With that said, I still want to move forward with the rest of it, the Python build script, dynamic/shallow clones of the various other repos without submodules, and the improvements to the current packaging schema.


#5

If the packaging is not one level higher then the rest of the repos. It might make the checkout of the (many many) repos pretty messy if you still want to work on the server itself. (although we could checkout one level up the tree. ../) This would mirror my personal layout, but might not be expected by most people.

And frankly renaming a repo only requires people to update their upstream repo. Although I’m also fine with just having jellyfin-packaging. The main “pull” or branding will be the website and the organization on github anyway.


#6

Yea that’s pretty similar to my layout right now as well. ~/Projects/Jellyfin/{jellyfin, jellyfin-web, etc.}. If we renamed the main repo we wouldn’t want anything to replace it then, and I’d be OK with that too. jellyfin-server and jellyfin-packaging, or maybe jellyfin-build instead since that’s the more general case. We’d just need docs to make clear how to get started with this, i…e “pull jellyfin-build first, then run script setup” or something.

I do worry that this might get convoluted though, if we’re not careful. That’s mainly why I asked the “why even split it” question earlier - if we don’t know why we want to do this we might fall into implementation traps that make normal users’ lives harder rather than easier. @nvllsvm I’d like to hear your input too.

For me, the main arguments either way are:

For a split: Easier to manage packaging on its own, cleaner implementation as as grow platforms, unification of multiple repository’s packaging easily.
Against a split: It’s more complex to know where to start, renaming main repo would break links etc. people already have, we have to deal with pulling repos in easily and consistently, huge overhead potentially to build certain parts.

I’m curious your folks’ thoughts.


#7

I personally am all for split packaging out - this just follows “normal” Unix principle, but for repos.
There’s no logical sense of packaging scripts in the main repo that build plugins.

That said, I don’t think we should rename or in any way alter main jellyfin repo. Simple set-aside jellyfin-build is fine (I don’t think we should name it jellyfin-package, as it might give some false assumptions because package can be both noun and verb, with example for noun would be client apk package; build can be seen as both, too, but that is less of a problem for me).

Putting SharedVersion.cs away and thus making main repo unbuildable is a no-go in my opinion. Ideally all parts of the puzzle should be buildable independently if wanted to. Plus moving SharedVersion makes it so we contribute the code that requires version change in one repo, but change the version in unrelated repo - ughh, don’t like that.

I would say that what I’d like to see is all repos that need to be built (server, clients, plugins) provide their own “instructions” (in terms of unified metadata manifest most likely) that describe their version and how to build them. Individual repos most likely should not contain build scripts to follow this metadata, those scripts should live in jellyfin-build IMHO and be as non-specific to what they build as possible.
Ideal solution for me would be some build script that could be fed a list of repos, then it would fetch their metadata and build all the stuff. That way we can let authors of 3rd party plugins to incorporate their work more easily, plus we move all the dealing with metadata in one place.

To sum up what I think: I think we need to make sure each repo serves only one purpose - storing the buildable code that solves this repo’s aim. Building such code is not a repo’s aim, though, and in my view should be handled separately.


#8

I agree with all the reasons you listed. The best solution in my opinion would be to rename jellyfin to jellyfin-server and create jellyfin-packaging, which would be more of a meta repo for us to easily build all of the projects than for casual users to build their favorite client.

You even mentioned that the scripts in jellyfin-packaging would not be building the projects directly but rather calling build scripts in the external repos. This works perfectly because it means when users want to build something they should always be able to go and build a single repo. So in general, don’t touch the individual projects but just create a meta repo for easily building and hosting everything on the website.


#9

Well I don’t think we should have nested build scripts, unless you view dotnet publish as a build script. If a user really want to build stuff separately and not follow the documentation they can figure the build command out themselves.

That makes it much easier for us to maintain. No need for more than one build script.