Recent Posts

Pages: 1 [2] 3 4 ... 10
11
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by livevil on February 23, 2018, 11:43:06 am »
It seems that mitmproxy can be installed under windows (if the your "server" is running windows).

Indeed, the same approach could be used to deal with the cloudfare/subtitles problems. I know I used my synology as a DNS server to forward requests to api.opensubtitles.org directly to their IP adress, but the query could be edited with mitmproxy (if performed in plain http, not sure it would works with https), without the need to use a specific DNS server.
12
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by jer1956 on February 23, 2018, 09:44:55 am »
Thanks for taking the time to post this.

Many of us have PC's as servers and could act as the gateway.
13
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by livevil on February 22, 2018, 08:43:10 pm »
So this is what I have done to get the images (it's a bit long). For this, I am using a Raspberry Pi 3, with Raspbian Stretch

The network needs to be properly configured first. It's better to have a fixed IP otherwise the Raspberry may change IP upon reboot. I will assume that it is done... So let's install mitmproxy.

First do a:
Code: [Select]
sudo apt-get update
sudo apt-get upgrade

Then install the dependencies for mitmproxy:
Code: [Select]
sudo apt-get install python3-dev python3-pip libffi-dev libssl-dev
You need python 3.5 or higher to install mitmproxy 2.0.2 (you can check your current version with: python -V)

I tried to install mitmproxy from apt-get, but I could only have version 0.8-2 (you can check which version will be installed by using apt-cache madison mitmproxy, apt-cache showpkg mitmproxy or apt-cache policy mitmproxy)
So I installed it directly with python:
Code: [Select]
sudo pip3 install mitmproxy
If everything went well, mitmproxy should be installed by now. By default, it uses port 8080 (I haven't tried to change it, but you could). We need to redirect http (80) and https (443) traffic towards it: these are the ports that the mede8er is using to send its queries. If you are running a web server or something else on these ports, you need to disabled it or change its port.
Then you need to allow port forwarding:
Code: [Select]
sysctl -w net.ipv4.ip_forward=1
And redirect the ports:
Code: [Select]
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080
(change "eth0" with your network interface, e.g. "wlan0" for wifi; check with ifconfig or ip link show)

mitmproxy is ready to run. Check if it is working by starting it with:
Code: [Select]
mitmproxy -T --host
(Note: if you want to use mitmproxy as an explicit proxy with your web browser, e.g. for testing purpose, you need to remove the "transparent" option "-T")

Now you should obtain a black window with a blue ribbon on the bottom. The application is ready to receive and process requests. Go to the mede8er and change the gateway to point to the raspberry. Try the scraper on your movie directory: you should have queries/reponses showing in the window (with the 200 code that says that it was processed successfully).
If so, mitmproxy is running fine. Right now, the query and response are left untouched. We need to attach a python script to the program. Quit mitmproxy (press ":", then "q", then "y").

Go to your home folder and open an editor:
Code: [Select]
cd /home/pi
sudo nano changedpi.py

And paste the following code:
Code: [Select]
"""
Written by livevil, version 1.2
Script for mitmproxy (tested with version 2.0.2)
This script intercepts the jpeg images sent by the server, checks whether the jfif
marker is present, if so, it gets the image density  and forces it to (72,72) dpi in
case it is set to (1, 1)
"""
import io
from PIL import Image
from mitmproxy import http
from mitmproxy import ctx

def response(flow: http.HTTPFlow) -> None:
    ctx.log.info("Script has started.")
    if ("image.tmdb.org" in flow.request.host_header) and flow.response.headers.get("content-type", "").startswith("image"):
        ctx.log.info("image sent by image.tmdb.org")
        s = io.BytesIO(flow.response.content)
        img = Image.open(s)
        try:
            imgmarker = img.info['jfif']
        except:
            ctx.log.info("jfif marker not found, nothing done...")
            pass
        else:
            ctx.log.info("image density is : {0} ".format(img.info['jfif_density']))
            if img.info['jfif_density'] == (1, 1):
                ctx.log.info("density is (1, 1), forcing it to (72, 72)")
                s2 = io.BytesIO()
                img.save(s2, "jpeg", dpi=(72,72))
                flow.response.content = s2.getvalue()
                flow.response.headers["content-type"] = "image/jpg"
            else:
                ctx.log.info("image density is OK, nothing done...")

Save and close by pressing "Ctrl+X" and validate with "Y". Now run mitmproxy again with the script attached:
Code: [Select]
mitmproxy -T --host -s changedpi.py
You should get the same black window, with a "[scripts:1]" at the bottom. Try again the scrapper: if everything goes well, the images with 1x1 dpi should be modified to 72x72 dpi.

Also port forwarding will be lost upon reboot, so you will need to enter again:
Code: [Select]
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080
when you restart. This is easily done automatically but I didn't have the time yet.

I didn't test everything extensively. I first installed mitmproxy 2 days ago, after trying some other things. So it's new to me :) Try it and tell me if it works for you :)
14
Customer Feedback & Reviews / Re: No picture
« Last post by mike_carton on February 22, 2018, 12:22:11 pm »
The first one is in 4k resolution - not supported by Mede8er.
The second one is using HEVC codec - not supported by Mede8er.
15
Customer Feedback & Reviews / No picture
« Last post by danie on February 22, 2018, 12:10:20 pm »
Please some one help me. . . . . Here are 2 examples of films in a hi standard that my Mede8er DX1000 does not want to play properly - no video but there is sound.
1.  The.Martian.2015.EXTENDED.2160p.BluRay.x265.10bit.HDR.TrueHD.7.1.Atmos-SWTYBLZ
2.  Whip It 2009 (1080p Bluray x265 HEVC 10bit AAC 5.1 Tigole)
Please send me an email to d@niel.co.za
16
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by mike_carton on February 22, 2018, 12:33:55 am »
This is what I like about the communities that build around hobbies like this.
17
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by livevil on February 21, 2018, 10:09:57 pm »
The problem that could be encountered with this solution would be if the mede8er uses https for the requests and verifies the certificate. In this case, it would detect the "man-in-the-middle" because it wouldn't provide the right certificate for the domain it was requesting.
During the tests I've made tonight, the mede8er only sent http requests towards image.tmdb.org, so these can be easily intercepted.

I will try to post more details later.
18
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by jer1956 on February 21, 2018, 09:24:43 pm »
Please post more details. Something similar could be done with other devices acting as a proxy.
19
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by livevil on February 21, 2018, 07:38:34 pm »
I've been tinkering with my mede8er and a raspberry this week and I found a way to make it work (it's not a fix though but it removes the need to manually edit images  :P).

I've installed mitmproxy on the raspberry and run it as a transparent proxy. I've attached a python script that intercepts answers from the server containing "image" in the header. The script edits the density of the images to force 72x72 and transmits them to the client.

I've then changed the gateway of the mede8er to point to the raspberry and now when I scrap, I get 72dpi images, which are accepted by the mede8er to build the about.jpg and the folder.jpg   :)

EDIT: it's weird tough because it seems that tmdb doesn't always send images with 1x1dpi. Some of the "wrong" pictures I got last week are now OK (72dpi)  :o
20
Technical Issues / Re: Corrupted pictures after scratching - single movies
« Last post by YyyNnn on February 21, 2018, 02:16:20 pm »
Propfax is right. Snaji have to fix it.
Pages: 1 [2] 3 4 ... 10