Setting your USBasp programmer to slow mode, if there is no jumper

I recently had the problem, that I was not able to program my Attiny anymore. I was a bit confused, because half a year ago, it just worked fine. Yet, as I couldn’t find any other solution (I checked the wiring about 10 times, the voltage for the attiny about 5 times and exchanged it with backup model), the only left reason seemed to be the fast mode. They say, that you can only program it with 250kHz, whereas the USBasp normally uses a bit more. However, my USBasp neither did allow setting the speed by software (firmware too old and no secondary programmer available), nor was there a slow jumper/switch.

So what’s the solution if the device costs only a few Euro and does not work at the moment anyway? Of course, the tough guys we are, we take a jumper cable, connect it with GND on the one side, and with the left hand hold it tightly onto PIN 25 while with the right hand we hit enter to execute the avrdude command!

People who are good at soldering, will manually solder GND and PIN 25 together.

Distribution of torrent files in categories (at kickass.to)

Just a quick chart: Since kickass.to provides its whole archive for download, I was able to analyze which category contains how many torrents.

Most files are movies and TV shows, followed by music and porn. Far behind then come applications, books, anime (which are not counted into TV or movies) and then games.

kickass.to categories

Scrapy: Fix wrong sitemap URLs with custom downloader middleware

On stackoverflow, the topic was discussed, how to solve the problem of some sitemaps having absolute URLs without a scheme. According to RFC this is fine, but as the maintainers of scrapy pointed out, sitemaps require the contents of <loc> to include a scheme (called protocol in the sitemap specs).

So it remains to the programmer of a spider to fix this issue, if he encounters some websites using the wrong format.

Overwriting the default spider does not work so well in this case, because one would have to copy a lot of code. So this seems like a good case for middlewares: Change the response to a valid format without the spider noticing.

The downloader middleware allows us to change the response by returning either a modified or a totally new Response object from process_response.

Thus, it’s rather easy to implement a middleware which takes care of replacing wrongly formatted URLs to the correct ones – at least for the most simplistic cases. I did not implement any sophisticated XML namespace parsing, nor did I implement support for Google’s alternate language pages. The XML namespace parsing would only be important in theory (and for alternate language pages), because the sitemap author could set an additional namespace for the normally default namespace and then the element would not be called <loc> but maybe <sitemap:loc>.

Ignoring these things, one can just use regular expressions to add the scheme where missing.

import re
import urlparse
from scrapy.http import XmlResponse
from scrapy.utils.gz import gunzip, is_gzipped
from scrapy.contrib.spiders import SitemapSpider
 
# downloader middleware
class SitemapWithoutSchemeMiddleware(object):
    def process_response(self, request, response, spider):
        if isinstance(spider, SitemapSpider):
            body = self._get_sitemap_body(response)
 
            if body:
                scheme = urlparse.urlsplit(response.url).scheme
                body = re.sub(r'<loc>\/\/(.+)<\/loc>', r'<loc>%s://\1</loc>' % scheme, body)
                return response.replace(body=body)
 
        return response
 
    def _get_sitemap_body(self, response):
        """Return the sitemap body contained in the given response, or None if the
        response is not a sitemap.
        """
        if isinstance(response, XmlResponse):
            return response.body
        elif is_gzipped(response):
            return gunzip(response.body)
        elif response.url.endswith('.xml'):
            return response.body
        elif response.url.endswith('.xml.gz'):
            return gunzip(response.body)

The newly created middleware can then be added to your project through the settings file (exact setting of course depends on where you saved the middleware).

DOWNLOADER_MIDDLEWARES = {                                                      
    'middlewares.SitemapWithoutSchemeMiddleware': 900
}