Search does not work after update to 10.9.x

[Kevin Nord]

Hello, since the update neither I nor my users have been able to use search function. The web client or device clients just sit there and spin perpetually with the infuriating blue circle.

I don't see any activity in the logs to indicate a problem. Kind of stuck at where to start troubleshooting this one with no logs, hoping someone has some ideas.

Ubuntu 22.04 / Jfin 10.9.2


   

[alan]

Permissions problem?

[TheDreadPirate]

It looks like you have a theme? This is a long shot, but is the theme the problem? I've seen themes do inexplicably weird things with Jellyfin.

[Kevin Nord]

It looks like you have a theme?  This is a long shot, but is the theme the problem?  I've seen themes do inexplicably weird things with Jellyfin.

I disabled the theme and restarted the service; no joy.

However, I found the following while tailing logs during a media scan

Code:

[2024-05-21 18:46:06.161 -04:00] [ERR] Error executing Scheduled Task
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 6: 'database table is locked'.
  at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
  at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
  at Microsoft.Data.Sqlite.SqliteConnectionExtensions.ExecuteNonQuery(SqliteConnection connection, String commandText, SqliteParameter[] parameters)
  at Microsoft.Data.Sqlite.SqliteTransaction..ctor(SqliteConnection connection, IsolationLevel isolationLevel, Boolean deferred)
  at Emby.Server.Implementations.Data.SqliteItemRepository.SaveImages(BaseItem item)
  at Emby.Server.Implementations.Library.LibraryManager.UpdateImagesAsync(BaseItem item, Boolean forceUpdate)
  at MediaBrowser.Controller.Entities.Folder.ValidateChildrenInternal2(IProgress`1 progress, Boolean recursive, Boolean refreshChildMetadata, Boolean allowRemoveRoot, MetadataRefreshOptions refreshOptions, IDirectoryService directoryService, CancellationToken cancellationToken)
  at MediaBrowser.Controller.Entities.Folder.ValidateChildrenInternal(IProgress`1 progress, Boolean recursive, Boolean refreshChildMetadata, Boolean allowRemoveRoot, MetadataRefreshOptions refreshOptions, IDirectoryService directoryService, CancellationToken cancellationToken)
  at MediaBrowser.Controller.Entities.Folder.<>c__DisplayClass69_0`1.<<RunTasks>b__1>d.MoveNext()
--- End of stack trace from previous location ---
  at MediaBrowser.Controller.Entities.Folder.RunTasks[T](Func`3 task, IList`1 children, IProgress`1 progress, CancellationToken cancellationToken)
  at MediaBrowser.Controller.Entities.Folder.ValidateChildrenInternal2(IProgress`1 progress, Boolean recursive, Boolean refreshChildMetadata, Boolean allowRemoveRoot, MetadataRefreshOptions refreshOptions, IDirectoryService directoryService, CancellationToken cancellationToken)
  at MediaBrowser.Controller.Entities.Folder.ValidateChildrenInternal(IProgress`1 progress, Boolean recursive, Boolean refreshChildMetadata, Boolean allowRemoveRoot, MetadataRefreshOptions refreshOptions, IDirectoryService directoryService, CancellationToken cancellationToken)
  at MediaBrowser.Controller.Entities.AggregateFolder.ValidateChildrenInternal(IProgress`1 progress, Boolean recursive, Boolean refreshChildMetadata, Boolean allowRemoveRoot, MetadataRefreshOptions refreshOptions, IDirectoryService directoryService, CancellationToken cancellationToken)
  at Emby.Server.Implementations.Library.LibraryManager.PerformLibraryValidation(IProgress`1 progress, CancellationToken cancellationToken)
  at Emby.Server.Implementations.Library.LibraryManager.ValidateMediaLibraryInternal(IProgress`1 progress, CancellationToken cancellationToken)
  at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)
[2024-05-21 18:46:06.163 -04:00] [INF] "Scan Media Library" Failed after 1 minute(s) and 55 seconds

I tried searching the SQLite error message but got a bunch of StackExchange threads that are above my head. Is this information helpful at all?

Thanks for your perpetual help!

[TheDreadPirate]

A bunch of people have had that log message and various issues as a result of it. We/I haven't been able to find any rhyme or reason.

Is your OS/database on a HDD?

[Kevin Nord]

A bunch of people have had that log message and various issues as a result of it.  We/I haven't been able to find any rhyme or reason.

Is your OS/database on a HDD?

I'm glad I am not alone. Is there any other data I can provide that would help?

My media is stored on several HDDs pooled with mergerfs but the OS is installed on a WD Red SSD, and /var/lib/jellyfin/metadata and /var/lib/jellyfin/transcodes are each on a separate  WD Red SSD.

I'll post a bit more about my setup here in case any of it is helpful:

Code:

$ lsblk
NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0  14.6T  0 disk /mnt/media8
sdb      8:16  0  14.6T  0 disk /mnt/media5
sdc      8:32  0  14.6T  0 disk /mnt/media1
sdd      8:48  0  14.6T  0 disk /mnt/media0
sde      8:64  0  12.7T  0 disk /mnt/media2
sdf      8:80  0  12.7T  0 disk /mnt/music0
sdg      8:96  0  9.1T  0 disk /mnt/media3
sdh      8:112  0  10.9T  0 disk /mnt/media4
sdi      8:128  0 465.8G  0 disk /var/lib/jellyfin/transcodes
sdj      8:144  0 465.8G  0 disk
├─sdj1  8:145  0    1G  0 part /boot/efi
└─sdj2  8:146  0 464.7G  0 part /
sdk      8:160  0 465.8G  0 disk /var/lib/jellyfin/metadata
sdl      8:176  0 931.5G  0 disk
sdm      8:192  0  12.7T  0 disk /mnt/media6
sdn      8:208  0  14.6T  0 disk /mnt/media7

$ sudo smartclt -a /dev/sdk
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-107-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:    WD Blue / Red / Green SSDs
Device Model:    WDC  WDS500G1R0A-68A4W0
Serial Number:    23411G800217
LU WWN Device Id: 5 001b44 8b347e878
Firmware Version: 411010WR
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:    Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed May 22 10:17:56 2024 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (  2) minutes.
Extended self-test routine
recommended polling time:        (  10) minutes.

SMART Attributes Data Structure revision number: 4
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct  0x0032  100  100  ---    Old_age  Always      -      0
  9 Power_On_Hours          0x0032  100  100  ---    Old_age  Always      -      3416
12 Power_Cycle_Count      0x0032  100  100  ---    Old_age  Always      -      28
165 Block_Erase_Count      0x0032  100  100  ---    Old_age  Always      -      2359319
166 Minimum_PE_Cycles_TLC  0x0032  100  100  ---    Old_age  Always      -      0
167 Max_Bad_Blocks_per_Die  0x0032  100  100  ---    Old_age  Always      -      51
168 Maximum_PE_Cycles_TLC  0x0032  100  100  ---    Old_age  Always      -      1
169 Total_Bad_Blocks        0x0032  100  100  ---    Old_age  Always      -      290
170 Grown_Bad_Blocks        0x0032  100  100  ---    Old_age  Always      -      0
171 Program_Fail_Count      0x0032  100  100  ---    Old_age  Always      -      0
172 Erase_Fail_Count        0x0032  100  100  ---    Old_age  Always      -      0
173 Average_PE_Cycles_TLC  0x0032  100  100  ---    Old_age  Always      -      0
174 Unexpected_Power_Loss  0x0032  100  100  ---    Old_age  Always      -      0
184 End-to-End_Error        0x0032  100  100  ---    Old_age  Always      -      0
187 Reported_Uncorrect      0x0032  100  100  ---    Old_age  Always      -      0
188 Command_Timeout        0x0032  100  100  ---    Old_age  Always      -      1
194 Temperature_Celsius    0x0022  058  048  ---    Old_age  Always      -      42 (Min/Max 21/48)
199 UDMA_CRC_Error_Count    0x0032  100  100  ---    Old_age  Always      -      0
230 Media_Wearout_Indicator 0x0032  001  001  ---    Old_age  Always      -      0x000300000003
232 Available_Reservd_Space 0x0033  100  100  004    Pre-fail  Always      -      100
233 NAND_GB_Written_TLC    0x0032  100  100  ---    Old_age  Always      -      129
234 NAND_GB_Written_SLC    0x0032  100  100  ---    Old_age  Always      -      239
241 Host_Writes_GiB        0x0030  253  253  ---    Old_age  Offline      -      198
242 Host_Reads_GiB          0x0030  253  253  ---    Old_age  Offline      -      346
244 Temp_Throttle_Status    0x0032  000  100  ---    Old_age  Always      -      0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

Selective Self-tests/Logging not supported

$ df -h /dev/sd{a..n} /var/media /
Filesystem    Type          Size  Used Avail Use% Mounted on
/dev/sda      ext4            15T  3.3T  11T  24% /mnt/media8
/dev/sdb      ext4            15T  12T  1.9T  87% /mnt/media5
/dev/sdc      ext4            15T  12T  1.9T  87% /mnt/media1
/dev/sdd      ext4            15T  12T  1.9T  87% /mnt/media0
/dev/sde      ext4            13T  11T  1.9T  85% /mnt/media2
/dev/sdf      ext4            13T  6.2T  5.9T  52% /mnt/music0
/dev/sdg      ext4          9.1T  6.8T  1.8T  80% /mnt/media3
/dev/sdh      ext4            11T  8.6T  1.8T  83% /mnt/media4
/dev/sdi      ext4          459G  781M  435G  1% /var/lib/jellyfin/transcodes
udev          devtmpfs        16G    0  16G  0% /dev
/dev/sdk      ext4          459G  35G  401G  8% /var/lib/jellyfin/metadata
udev          devtmpfs        16G    0  16G  0% /dev
/dev/sdm      ext4            13T  5.7T  6.4T  48% /mnt/media6
/dev/sdn      ext4            15T  3.3T  11T  24% /mnt/media7
Movies.fuse    fuse.mergerfs  118T  74T  39T  66% /var/media
/dev/sdj2      ext4          457G  135G  299G  32% /
total          -              249T  153T  84T  65% -

$ lsmem; lscpu
RANGE                                SIZE  STATE REMOVABLE  BLOCK
0x0000000000000000-0x000000007fffffff  2G online      yes  0-15
0x0000000100000000-0x000000087fffffff  30G online      yes 32-271

Memory block size:      128M
Total online memory:      32G
Total offline memory:      0B
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:        46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  56
  On-line CPU(s) list:  0-55
Vendor ID:              GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
    CPU family:          6
    Model:              79
    Thread(s) per core:  2
    Core(s) per socket:  14
    Socket(s):          2
    Stepping:            1
    CPU max MHz:        3500.0000
    CPU min MHz:        1200.0000
    BogoMIPS:            5193.44
    Flags:              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid ape
                        rfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cd
                        p_l3 invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_tota
                        l cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):
  L1d:                  896 KiB (28 instances)
  L1i:                  896 KiB (28 instances)
  L2:                    7 MiB (28 instances)
  L3:                    70 MiB (2 instances)
NUMA:
  NUMA node(s):          2
  NUMA node0 CPU(s):    0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54
  NUMA node1 CPU(s):    1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:        KVM: Mitigation: VMX disabled
  L1tf:                  Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                  Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:              Mitigation; PTI
  Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                Not affected
  Tsx async abort:      Mitigation; Clear CPU buffers; SMT vulnerable

[prahal]

I doubt storage matters. I am also affected by this bug but mostly have a working jellyfin at least the scanning and encoding since I forced to "1" in Settings > General the library scan and encoding number of parallel tasks.
COuld be the search does not work because the scan did not complete and thus the search has no index to search upon. Try the force number of parallel tasks, wait, or force in TaskScheduler a rescan and retry.
The search code is likely fine but it has an empty index to search upon.

[TheDreadPirate]

That's an idea. Maybe do a "find missing metadata" scan? I haven't been able to replicate on my system. Jellyfin search works as expected. My understanding is that it reads the database, which appears to be on /dev/sdj2 on your system, and not from the metadata or library drives.

[Kevin Nord]

I doubt storage matters. I am also affected by this bug but mostly have a working jellyfin at least the scanning and encoding since I forced to "1" in Settings > General the library scan and encoding number of parallel tasks.
COuld be the search does not work because the scan did not complete and thus the search has no index to search upon. Try the force number of parallel tasks, wait, or force in TaskScheduler a rescan and retry.
The search code is likely fine but it has an empty index to search upon.

Heard, I am also  doubtful storage is a factor, but TDP has consistently proven to be smarter than me, so if they ask about it I'm going down that road with them lol.

I was thinking the same thing about the empty index as well, but I am struggling to see how it would "disappear" between upgrades. (I'm also not a developer so much of it is just sorcery to me anyway, though)

Thanks for the tip about setting parallel scans to 1, I tried that this morning and it is running, albeit obviously slowly.  (how do people run JF on a Pi???) I'll report back when it is complete!

---

I haven't been able to replicate on my system.

hows about I just  give you $50 and add you to my sudoers and let you have a crack? jk jk

Running scans based on your and another commenters feedback, I'll update the thread when the scans are complete. Thanks for your perpetual help!

[Kevin Nord]

That's an idea.  Maybe do a "find missing metadata" scan?  I haven't been able to replicate on my system.  Jellyfin search works as expected.  My understanding is that it reads the database, which appears to be on /dev/sdj2 on your system, and not from the metadata or library drives.

I have tried doing  a scan for missing meta data, but no luck. Search is still non functional (perpetual spinning circle)