Information for those developing Aggregation Tools

[jer1956]

Due to demand from our users we decided to implement Aggregated JukeBox functionality for the X3D Series. The concept is new for Mede8er and it will take some time until a lot of scrapers and tools support this. Yadis supports this functionality for Dune Players so with the help of 'funkycochise' and his tool Yadis2 mede8er this can now be done on the X3D Series. This functionality is new but should be robust within the Mede8er as we still use all the existing elements of the Movie JukeBox and the Mede8er.db remains the same.

The way it works is to have a 'Workspace' or dummy Jukebox in a specific location. The dummy JukeBox can be on the Mede8er HDD or a USB Stick or USB Drive of on a Samba or NFS  Network share. The dummy JukeBox contains the full folder structure of the original Movie JukeBox with each movie in its own folder. Within each folder you will have the normal Mede8er folder.jpg and about.jpg and the Movie.name.xml or folder.name.xml. The dummy JukeBox also uses the Mede8er.db for xml search functionality so it also needs to be scanned. Practically you would navigate to this dummy JukeBox folder and add it to Favourites. In Favourites you would scan it and you now ready to go.

How does it work
When you try try to Play the movie the Mede8er firmware V2.0.0 - 24Jan or later will detect that there is no movie in the folder and it will open the movie.name.xml or folder.name.xml and check a new field entry called 'filelink'. This filelink contains information on the connection type and the path that Mede8er needs to use in order to play the movie.

Filelink details
Samba]
<filelink>smb://192.168.0.2/Share_Name/Folder/Sub Folder</filelink>
Samba within Login
<filelink>smb://User Name:Password@192.168.40.134/Share_Name/Folder/Sub Folder</filelink>
USB/HDD]
<filelink>storage_label://LABEL/Folder/Sub Folder</filelink>
NFS
<filelink>nfs://Host_IP/Path/To/Export:/Sub Folder</filelink>


xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Yadis Path Setup xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  

Yadis is used as the scraper and we will be using the 'Dune' profile.  

In Yadis select Setup
A. General Tab
1. In The Profile tab select Dune
2. Set the Workspace path (this is Yadis temp folder)
3. Dune Interface location - not required
4. Other settings according to preference
B. Folders Tab.  
1. Select 'Add'
2. Set the Folder Path
3. Set the Dune Path

Samba
smb://192.168.0.2/Share_Name/Folder/Sub Folder
Samba within Login
smb:/User Name:Password@192.168.40.134/Share_Name/Folder/Sub Folder
USB/HDD
storage_label://LABEL/Folder/Sub Folder
NFS
nfs://Host_IP/Path/To/Export:/Sub Folder

Samba Example
When you point to a share the Dune Path will look like this (Obviously the share name and folder will show the actual you have selected)
storage_label:///Share_Name/Folder
Replace the 'red' with smb://IP
ie
smb://192.168.0.2/Share_Name/Folder

Samba Example with Login
When you point to a share the Dune Path will look like this (Obviously the share name and folder will show the actual you have selected)
storage_label:///Share_Name/Folder
Replace the 'red' with smb:/User Name:Password@IP
ie
smb://User Name:Password@192.168.0.2/Share_Name/Folder

NSF
If the NFS share in on a NAS it will look like this when you enter the path
smb://192.168.0.100/Movie_Collection/Movies
Replace the 'red' with nfs://Host_IP/Path/To/Export:/Sub Folder

Actual example of my path to my Thecus nas
nfs://192.168.0.100/raid0/data/_NAS_NFS_Exports_/DVD_Music_Videos:

USB Drives and Internal HDD
We have a bug with internal HDD so please do not use this for the current round of testing. USB Drives are working fine.Will fix this asap Fixed V2 - 31Jan
This is correct so need need to change anything
ie
storage_label://Drive Name/Folder/ Sub Folder etc

Once you have set the 'storage_labels' as above and then scraped your movies you will be ready to convert the images and xml to Mede8er standard


xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Yadis2Mede8er xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


UPDATE BETA 12 - 28Jan2013
Download  Yadis2Mede8er beta12 and unrar and copy contents to a new folder in Program Files and make a shortcut

https://www.dropbox.com/s/x29enav4rl3ded1/Yadis2mede8er%20beta%2012.7z

Changelog
1. Fixes : unclosed tag when no genres, cast and so on.

Launch Yadis2Mede8er

1. Go to Settings
1. Fill in the Yadis Workspace folder
2. 'Aggregate jukebox' is the new 'dummy jukebox' that  Yadis2Mede8er will create so you need to make a folder and name it whatever you like and it can be on USB, Mede8er HDD, Samba Share, NFS share. Point to this folder
3. Check 'Use Dune profile in Yadis'
4. Check Aggregate jukebox  
5. Make sure 'Resize folder.jpg is checked and size is 160 x 237
6. Make sure foll are checked Folder.jpg, About.jpg, Movie.xml
7. Select Save
8 Back to main page
9. If you select 'send all' then Yadis2Mede8er will always send ALL images and xml to the dummy jukebox. If you leave it unchecked it will only save new updated information
10. Run 'Mede8erize' and a complete dummy JukeBox will be built in the Aggregate JukeBox folder

Treat the new Aggregate JukeBox like a normal Mede8er JukeBox
ie
1. Navigate to this new Dummy JukeBox in the X3D
2. Make a Favlink
3. Scan the Favlink


Good Luck  Grin


Mede8er1


Posted by Jer1956 on his behalf.

[jer1956]

The database is unchanged.  So offline scanning using a PC should remain the same, its just that you scan the agragated jukbox, not where the media is.

[Lupissimo]

Not being an expert on Network questions could you please help with the following:
I have a Synology NAS with 4 HDs each being a separate Volume (1..4) (no raid). On Volume 1 I have a Common Folder DE-Filme and in this 2 folders DE-Filme and HD-Filme each containing moviefolders (e.g. movie1, movie2 etc.)

The NAS is called Diskstation and has fixed IP 192.168.1.50. What is the correct <disklink> for an NFS connection ?

[jer1956]

Ive been told it's IP   - mount point.  So mede8er1 has a mount point of :-raid0/data/_NAS_NFS_Exports_/DVD_Music_Videos:

So  this is what yours should be like:-


nfs://192.168.0.100/volume1/DVD_Music_Videos:


The ":" is added when the filelink is created by Yadis2Mede8er. The guide only tells you  what to  set dunepath to in Yadis.

It's worth doing a  test using a small  test folder with Yadis/Yadis2mede8er to see what a filelink looks like.


you should deduce the following:-



nfs://IP/volume1/Export:/Sub Folder
Export = Share Name
Sub Folder would be Movie Folder
Unless there is another level
Mount for Synology = volume1

[Lupissimo]

sorry, but I am still confused, since I shall not use yadis but my own software. Do you mean for movie1 on volume1 folder DE-Filme/DE-Filme/movie1folder:
<dislink>nfs://192.168.1.50/volume1/DE-Filme/DE-Filme/movie1:movie1<disklink>

[jer1956]

Yadis/yadis2mede8er is the "reference". Even if making your own tool  to "convert", you can use the "reference" to see what it produces for small test samples.

Dunepath in the Yadis guide only makes up a part of the fielink.   

We posted the yadis guide to  get people going. It wasn't designed for the purpose of telling you what filelink looks like.

[Mede8er1]

sorry, but I am still confused, since I shall not use yadis but my own software. Do you mean for movie1 on volume1 folder DE-Filme/DE-Filme/movie1folder:<dislink>nfs://192.168.1.50/volume1/DE-Filme/DE-Filme/movie1:movie1<disklink>

I expect it would look more like this
<dislink>nfs://192.168.1.50/volume1/DE-Filme:/movie1<disklink>

[Lupissimo]

thank you for the example, but are you not missing one folder level DE-Filme/DE-Filme ??

and is it disklink ore filelink??

[Mede8er1]

thank you for the example, but are you not missing one folder level DE-Filme/DE-Filme ??

and is it disklink ore filelink??

Could be as I dont know your exact folder structure

Maybe its like this

<filelink>nfs://192.168.1.50/volume1/DE-Filme:/DE-Filme/movie1<filelink>

or this

<filelink>nfs://192.168.1.50/volume1/DE-Filme:/DE-Filme/movie1/movie1<filelink>

[fredberr]

Hi,
I apologize if this is not the right thread to post those considerations but to some extent I think it is on topic.
I am considering to get either one of the X3D devices in order to be able to play 3D contents in full definition (iso bluray and mkv mvc) on my brand new set, but after reading the Y2M user guide I am very scared about the current necessity to have a separate folder for each video !  I understand there is not the same behaviour as on the PCH where the device reads directly the prepared html or flash that YAMJ gets prepared in the Jukebox folder, so for X3D you currently have to copy a few image files and the video xml back to the video folder for internal scanning.  But this limitation would really be killing me and my 4.5TB video library.

Then, as you proposed, I beleive that scanning the Jukebox folder instead would allow not only to manage a multipath library but also to get rid of this stupid one video one folder...  as far as I understand YAMJ, there is already the aggregation function since you can give an xml library definition file as a moviejukebox input where for each library, <path> defines the video folder seen from your pc and <playerpath> the same path as seen by the player device. Then after scrapping you get included in the video xml the fileURL tag with the correct video path (playerpath + video filename).

In the end it is all about managing the library xml file from the Y2M GUI and the fileURL in each video xml on the player side, did I get something wrong ?


<libraries>

  <library>
    <path>N:\media\video</path>
    <playerpath>file:///opt/sybhttpd/localhost.drives/NETWORK_SHARE/NAS/media/video/</playerpath>
    <exclude name="sample,tmp/,temp/,RECYCLER/,RECYCLE.BIN,record/"/>
    <description></description>
    <prebuf></prebuf>
    <scrapeLibrary>true</scrapeLibrary>
  </library>

</libraries>

[jer1956]

In theory it's possible to have all  the fililinks pointing at one folder..even if the GUI elements are in the necessary separate folders. The problem is the only way to bypass converting a  conventional jukebox uses Yadis processing movies in separate folders.

Whats needed is an "Aggregating Scraper" which puts the scrape in the separate folders the  code needs, but will scrape a single folder creating links to that one folder.  

The issue is demand...this hasn't got the juices flowing amongst those who can create such a thing.

[fredberr]

Sorry, got the point now with your answer and after reading about the other scrapper Yadis that requires the folder hierarchy, and I understand also the X3D software expects this folder hierarchy.

Assumed the filelink functionality gets supported on the X3D this does not seem that complicated, I mean it should only impact the final step of copying the required files into the correct folder hierarchy but under a different root folder (e.g. Medebox) that could then be pointed to for scanning.

In the case of Y2M, I can immagine the following scenario:
1) My videos are under path1/videos, path2/videos (with or without separate folders for each video) and a library.xml is passed to moviejukebox.
2) The scrapping results are saved into the Jukebox folder.
3) A Medebox folder is created with all the folders and files necessary for scanning while videos remain in their original place(s).

It is a pity that nobody seems to be interested in such a solution, that would keep the video folder much cleaner and still usable when you browse it as a file listing.

Also for Yadis users only step3 needs to be updated.

[jer1956]

If you are aware of an "external" scraper which can work on a single folder and produces XML..then an aggregetion tool could be created to convert that into  the expected aggregated format.

It might be someone decided to improve on the internal popcorn scraper and created an external one to the popcorn "scrape a single folder" model..

[fredberr]

Here I don't follow you anymore, we already have that tool and this is YAMJ, it will scrap all the videos under your root folder(s) whether you have one file or one thousand files (i.e. as many as you want) under a single directory or subdirectory.

For example, I put my videos in the root folder (e.g. /video) and create a subfolder when I have several related videos such as a TV show or a set of films (e.g. Harry Potter):

NAS> ls Harry\ Potter/
[SET Harry Potter-1] Harry Potter e la pietra filosofale [2001].mkv
[SET Harry Potter-2] Harry Potter e la camera dei segreti [2002].mkv
[SET Harry Potter-3] Harry Potter e il prigioniero di Azkaban [2004].mkv
[SET Harry Potter-4] Harry Potter e il calice di fuoco [2005].mkv
[SET Harry Potter-5] Harry Potter e l'Ordine della Fenice [2007].mkv
[SET Harry Potter-6] Harry Potter e il principe mezzosangue [2009].mkv
[SET Harry Potter-7] Harry Potter e i Doni della Morte . Parte I [2010].mkv
[SET Harry Potter-8] Harry Potter e i Doni della Morte . Parte II [2010].mkv

But again you are totally free to put your videos wherever you want under the root folder.  Then passing a library.xml file as an input to moviejukebox you will get all your sources scrapped with all the images and xml files under the Jukebox directory. This is nothing different than what you can already get by running YAMJ from the YAMJ 2 Mede8er gui.

What needs to be modified in the Y2M is the Copy To Mede8er tool that should parse the Jukebox and generate the new folder hierarchy with all the images and xml. Also a new <filelink> tag has to be added in each xml if this is the tagname you expect (just need to duplicate the <fileURL> created by moviejukebox) or you could use directly the <fileURL>.

[fredberr]

Forgot to mention that Y2M also needs to support the configuration of multi source paths but this is only a GUI limitation since the underlying YAMJ scrapper does support the aggregation feature. In the end that would make a single tool to scrap and aggregate video sources without any constraint on the source folder hierarchy.

In the meantime I managed to read a few other threads and realized that some good job has already been done either on Yadis or by using a second tool to aggregate.  Did you check if the author of Y2M is willing to work on those enhancements ?