分类目录归档：python

socket.io-python-emitter 发送消息 socket.io-redis 无法收到的问题。

之前将服务迁移到 docker ，结果发现 socket.io-python-emitter 发送的消息全部发送不出去。经检查发现是 socket.io-redis 0.2.0 做了性能优化，与老版本不兼容了，无奈查源码自己打了个补丁(现已合并至主干)。

最开始是发现网站发起的推送延迟很厉害，而且推送 ping 日志显示丢包达到了80%以上。由于所有推送操作同时使用了极光推送及自建的 socket.io 推送。一开始没想到两套推送方案全部挂了，以为是测试设备出了问题，并且测试时设备日志记录了一次极光推送被强制关闭的log。同时发现极光推送控制台日志显示设备不在线(后来发现近期极光推送非常不稳定，这种延迟是常规状态了，由于原来自建推送可用，所以没有注意)。

但是经检查发现设备没有问题，而且设备状态上报部分同样使用的 socket.io，也很正常。所以又测试了一下python向网页端推送，发现还是收不到推送，确认python发出的推送出现故障了。

无奈进 redis 数据库查看推送是否进入了数据库，结果发现python发出的推送消息的确进入了redis数据库，但是和socket.io本身发出消息的频道名称不一致。找了一下，没有当时的记录了，大体是执行的redis-cli命令，通过 PSUBSCRIBE * 订阅所有频道。

于是觉得可能是频道名字的问题，所以仿着 socket.io 消息的格式通过 PUBLISH 频道名称消息发了个消息，无效，并且命令返回了1，只有手工PSUBSCRIBE * 的订阅，也就是 socket.io 过呢本没有接收消息。觉得是 socket.io 有问题，岁查看代码，发现socket.io-redis 会根据房间名订阅消息，并不会订阅 #emitter 频道的消息，就很纳闷之前的系统为什么能工作。检查了下修订历史，发现是 0.1.4 -> 0.2.0 为了性能只订阅相关房间的消息，看了一下发现 python 的代码是2年前了，和当前的实现不兼容…

无奈只能自己修改了，中间又发现新版本同时还修改了包格式，不过都问题不大。

这次修bug理清了大部分 socket.io-redis 的流程。

socket.io-redis 接收消息是通过订阅 redis 上面相关的频道来实现的，会检查 uid 看看是不是本机发出的，如果不是就转到 socket.io 处理。不过这里有个问题，uid 是随机生成的6位字符串，也就是有可能碰到相同的uid，socket.io-redis 会抛弃相同uid的消息造成相同uid的 socket.io 之间无法通行。

发送消息是通过向 redis 发送消息，然后其他 socket.io-redis 订阅并接收实现的。广播消息会直接发送到 prefix + ‘#’ + packet.nsp + ‘#’ ；发送到指定房间的消息会对涉及到的房间都发送一条 prefix + ‘#’ + packet.nsp + ‘#’ + room + ‘#’ 频道的消息。

删除 adb emulator-5554 device 的办法

发表评论

adb devices 多了一个 emulator-5554 设备，所有adb 的操作都需要手动指定设备名称，实在太麻烦了。

多次检查确定所有模拟器都关闭了，检查 5554 端口，确认并没有程序使用这个端口，最后终于找到了原因，adb 会搜索 5555 – 5683 端口，发现有打开的端口就会认为存在模拟器。我的5555端口是打开的，所以adb增加了一个迷你其，但是名字还是5554就比较坑人了。关闭5555端口后重启adb，emulator-5554 设备就消失了。

http://stackoverflow.com/questions/13017269/adb-devices-showing-dummy-device

解决 ElementTree 无法处理中文，UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 76-99: ordinal not in range(128)

发表评论

ET.fromstring 传进去 unicode 字符串提示 UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 76-99: ordinal not in range(128)。

encode(‘utf-8′) 后传进去提示 ValueError: multi-byte encodings are not supported,不支持多字节编码。

python 的中文支持真是头痛，这个如果用 sys.setdefaultencoding=’utf-8’ 可以解决，但是觉得直接为了这个直接修改了全局编码不太好，最后发现可以用如下代码解决：

try:
    import xml.etree.cElementTree as ET
except ImportError:
    import xml.etree.ElementTree as ET


utf8_parser = ET.XMLParser(encoding='utf-8')
tree = ET.parse(StringIO(r.text.encode('utf-8')), parser=utf8_parser)
root=tree.getroot()

try:

import xml.etree.cElementTree as ET

except ImportError:

import xml.etree.ElementTree as ET

utf8_parser = ET.XMLParser(encoding='utf-8')

tree = ET.parse(StringIO(r.text.encode('utf-8')), parser=utf8_parser)

root=tree.getroot()

https://gist.github.com/GameXG/89b92a9a94456ff2da85

稍微记一下，现在 django 关调试模式需要配置 ALLOWED_HOSTS = [] 否则 Bad Request (400)。

发表评论

稍微记一下，现在 django 关调试模式需要配置 ALLOWED_HOSTS = [] 否则 Bad Request (400)。

ALLOWED_HOSTS 填的是允许的域名列表，可以用 ‘*’ 来泛匹配。

今天早上部署 django 网站的时候被坑的不轻，根本找不到错误日志，最后无奈本地关调试模式 runserver 后看到提示才知道的。

python 使用 pycrypto‎ 实现 AES 加密解密

发表评论

http://www.voidspace.org.uk/python/modules.shtml#pycrypto 有 windows 下编译好的包，直接安装就可以。

import hashlib
from Crypto.Cipher import AES
from binascii import b2a_hex, a2b_hex
>>> def encrypt(key,text):
        # 密钥key 长度必须为16（AES-128）, 24（AES-192）,或者32 （AES-256）Bytes 长度
        # 所以直接将用户提供的 key md5 一下变成32位的。
	key_md5 = hashlib.md5(key).hexdigest()
        #  AES.MODE_CFB 是加密分组模式，详见 http://blog.csdn.net/aaaaatiger/article/details/2525561
        # b'0000000000000000' 是初始化向量IV ，16位要求，可以看做另一个密钥。在部分分组模式需要。
        # 那个 b 别漏下，否则出错。
	cipher = AES.new(key_md5,AES.MODE_CFB,b'0000000000000000')
        # AES 要求需要加密的内容长度为16的倍数，当密文长度不够的时候用 '\0' 不足。
	ntext = text + ('\0' * (16-(len(text)%16)))
        # b2a_hex 转换一下，默认加密后的字符串有很多特殊字符。
	return b2a_hex(cipher.encrypt(ntext))

>>> def decrypt(key,text):
	key_md5 = hashlib.md5(key).hexdigest()
	cipher = AES.new(key_md5,AES.MODE_CFB,b'0000000000000000')
	t=cipher.decrypt(a2b_hex(text))
	return t.rstrip('\0')

>>> encrypt('111','012340000000000000000000000000000000000000056')
'a37e0dda14a9678fcc82b8e16387c498c7206122195b1b91269a637322d776c48a28342d24b31879a35a0e77480a1dab'
>>> decrypt('111','a37e0dda14a9678fcc82b8e16387c498c7206122195b1b91269a637322d776c48a28342d24b31879a35a0e77480a1dab')
'012340000000000000000000000000000000000000056'
>>>

import hashlib

from Crypto.Cipher import AES

from binascii import b2a_hex, a2b_hex

>>> def encrypt(key,text):

# 密钥key 长度必须为16（AES-128）, 24（AES-192）,或者32 （AES-256）Bytes 长度

# 所以直接将用户提供的 key md5 一下变成32位的。

key_md5 = hashlib.md5(key).hexdigest()

# AES.MODE_CFB 是加密分组模式，详见 http://blog.csdn.net/aaaaatiger/article/details/2525561

# b'0000000000000000' 是初始化向量IV ，16位要求，可以看做另一个密钥。在部分分组模式需要。

# 那个 b 别漏下，否则出错。

cipher = AES.new(key_md5,AES.MODE_CFB,b'0000000000000000')

# AES 要求需要加密的内容长度为16的倍数，当密文长度不够的时候用 '\0' 不足。

ntext = text + ('\0' * (16-(len(text)%16)))

# b2a_hex 转换一下，默认加密后的字符串有很多特殊字符。

return b2a_hex(cipher.encrypt(ntext))

>>> def decrypt(key,text):

key_md5 = hashlib.md5(key).hexdigest()

cipher = AES.new(key_md5,AES.MODE_CFB,b'0000000000000000')

t=cipher.decrypt(a2b_hex(text))

return t.rstrip('\0')

>>> encrypt('111','012340000000000000000000000000000000000000056')

'a37e0dda14a9678fcc82b8e16387c498c7206122195b1b91269a637322d776c48a28342d24b31879a35a0e77480a1dab'

>>> decrypt('111','a37e0dda14a9678fcc82b8e16387c498c7206122195b1b91269a637322d776c48a28342d24b31879a35a0e77480a1dab')

'012340000000000000000000000000000000000000056'

>>>

XML-RPC 笔记

发表评论

XML-RPC 编码是个坑，只支持 ASCII 字符串（或二进制缓冲块）。为了安全，只能用 xmlrpclib.Binary 把数据包装一下了。

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> from SimpleXMLRPCServer import SimpleXMLRPCServer
>>> import xmlrpclib
>>> def a():
	return xmlrpclib.Binary({'a':1})

>>> server = SimpleXMLRPCServer(("localhost", 8000))
>>> print "Listening on port 8000..."
Listening on port 8000...
>>> server.register_function(python_logo, 'python_logo')

Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    server.register_function(python_logo, 'python_logo')
NameError: name 'python_logo' is not defined
>>> server.register_function(a, 'a')
>>> server.serve_forever()
127.0.0.1 - - [22/Apr/2014 22:36:32] "POST / HTTP/1.1" 200 -

Traceback (most recent call last):
  File "<pyshell#9>", line 1, in <module>
    server.serve_forever()
  File "C:\Python27\lib\SocketServer.py", line 236, in serve_forever
    poll_interval)
  File "C:\Python27\lib\SocketServer.py", line 155, in _eintr_retry
    return func(*args)
KeyboardInterrupt

>>> import json
>>> def a():
	return xmlrpclib.Binary(json.dumps({'a':1}))

>>> server.register_function(a, 'a')
>>> server.serve_forever()
127.0.0.1 - - [22/Apr/2014 22:37:27] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [22/Apr/2014 22:37:39] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [22/Apr/2014 22:38:04] "POST / HTTP/1.1" 200 -

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32

Type "copyright", "credits" or "license()" for more information.

>>> from SimpleXMLRPCServer import SimpleXMLRPCServer

>>> import xmlrpclib

>>> def a():

return xmlrpclib.Binary({'a':1})

>>> server = SimpleXMLRPCServer(("localhost", 8000))

>>> print "Listening on port 8000..."

Listening on port 8000...

>>> server.register_function(python_logo, 'python_logo')

Traceback (most recent call last):

File "<pyshell#7>", line 1, in <module>

server.register_function(python_logo, 'python_logo')

NameError: name 'python_logo' is not defined

>>> server.register_function(a, 'a')

>>> server.serve_forever()

127.0.0.1 - - [22/Apr/2014 22:36:32] "POST / HTTP/1.1" 200 -

Traceback (most recent call last):

File "<pyshell#9>", line 1, in <module>

server.serve_forever()

File "C:\Python27\lib\SocketServer.py", line 236, in serve_forever

poll_interval)

File "C:\Python27\lib\SocketServer.py", line 155, in _eintr_retry

return func(*args)

KeyboardInterrupt

>>> import json

>>> def a():

return xmlrpclib.Binary(json.dumps({'a':1}))

>>> server.register_function(a, 'a')

>>> server.serve_forever()

127.0.0.1 - - [22/Apr/2014 22:37:27] "POST / HTTP/1.1" 200 -

127.0.0.1 - - [22/Apr/2014 22:37:39] "POST / HTTP/1.1" 200 -

127.0.0.1 - - [22/Apr/2014 22:38:04] "POST / HTTP/1.1" 200 -

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import xmlrpclib
>>> proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
>>> proxy.a()

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    proxy.a()
  File "C:\Python27\lib\xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "C:\Python27\lib\xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "C:\Python27\lib\xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "C:\Python27\lib\xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "C:\Python27\lib\xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "C:\Python27\lib\xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:must be string or buffer, not dict">
>>> proxy.a()
<xmlrpclib.Binary instance at 0x02AEC508>
>>> proxy.a().data
'{"a": 1}'
>>> import json
>>> json.loads(proxy.a().data)
{u'a': 1}
>>>

Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32

Type "copyright", "credits" or "license()" for more information.

>>> import xmlrpclib

>>> proxy = xmlrpclib.ServerProxy("http://localhost:8000/")

>>> proxy.a()

Traceback (most recent call last):

File "<pyshell#2>", line 1, in <module>

proxy.a()

File "C:\Python27\lib\xmlrpclib.py", line 1224, in __call__

return self.__send(self.__name, args)

File "C:\Python27\lib\xmlrpclib.py", line 1578, in __request

verbose=self.__verbose

File "C:\Python27\lib\xmlrpclib.py", line 1264, in request

return self.single_request(host, handler, request_body, verbose)

File "C:\Python27\lib\xmlrpclib.py", line 1297, in single_request

return self.parse_response(response)

File "C:\Python27\lib\xmlrpclib.py", line 1473, in parse_response

return u.close()

File "C:\Python27\lib\xmlrpclib.py", line 793, in close

raise Fault(**self._stack[0])

Fault: <Fault 1: "<type 'exceptions.TypeError'>:must be string or buffer, not dict">

>>> proxy.a()

<xmlrpclib.Binary instance at 0x02AEC508>

>>> proxy.a().data

'{"a": 1}'

>>> import json

>>> json.loads(proxy.a().data)

{u'a': 1}

>>>

另一种注册方法：
http://soft.zdnet.com.cn/software_zone/2008/0527/887034.shtml

import SimpleXMLRPCServer

#定义自己的CMS类
class MyCMS:
    def getVersion(self):#向外公开版本的方法
        return "Powerd By python 0.1a"

cms = MyCMS()
server = SimpleXMLRPCServer.SimpleXMLRPCServer(("localhost", 8888))
server.register_instance(cms)

print "Listening on port 8888"
server.serve_forever()#服务器执行，并监听8888端口


import xmlrpclib

server = xmlrpclib.ServerProxy("http://localhost:8888";)

version = server.getVersion()

print "version:"+version

import SimpleXMLRPCServer

#定义自己的CMS类

class MyCMS:

def getVersion(self):#向外公开版本的方法

return "Powerd By python 0.1a"

cms = MyCMS()

server = SimpleXMLRPCServer.SimpleXMLRPCServer(("localhost", 8888))

server.register_instance(cms)

print "Listening on port 8888"

server.serve_forever()#服务器执行，并监听8888端口

import xmlrpclib

server = xmlrpclib.ServerProxy("http://localhost:8888";)

version = server.getVersion()

print "version:"+version

参考：
https://docs.python.org/2/library/xmlrpclib.html#fault-objects

lxml 笔记

发表评论

蜘蛛之前用的pyquery做采集，发现对非正常的网页并不适用，正则表达式更是不能用，目标网站网页乱的一团乱麻，打算重新用lxml的迭代解析功能来做吧。

Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import lxml

Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    import lxml
ImportError: No module named lxml
>>> import lxml
>>> class EchoTarget(object):
	def start(self, tag, attrib):
		print("start %s %r" % (tag, dict(attrib)))
		print attrib
	def end(self, tag):
		print("end %s" % tag)
	def data(self, data):
		print("data %r" % data)
	def comment(self, text):
		print("comment %s" % text)
	def close(self):
		print("close")
		return "closed!"

	
>>> from lxml import etree
>>> from io import StringIO, BytesIO

>>> parser = etree.XMLParser(target = EchoTarget())
>>> result = etree.XML("<element a='aaa' b='bbb' c=333 >some<!--comment-->text</element>",parser)
close

Traceback (most recent call last):
  File "<pyshell#18>", line 1, in <module>
    result = etree.XML("<element a='aaa' b='bbb' c=333 >some<!--comment-->text</element>",parser)
  File "lxml.etree.pyx", line 2723, in lxml.etree.XML (src/lxml/lxml.etree.c:52448)
  File "parser.pxi", line 1573, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:79932)
  File "parser.pxi", line 1452, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:78774)
  File "parser.pxi", line 960, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:75389)
  File "parsertarget.pxi", line 149, in lxml.etree._TargetParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:86190)
  File "parser.pxi", line 585, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:71955)
XMLSyntaxError: AttValue: " or ' expected, line 1, column 15
>>> result = etree.XML("<element a='aaa' b='bbb' >some<!--comment-->text</element>",parser)
start element {'a': u'aaa', 'b': u'bbb'}
{'a': u'aaa', 'b': u'bbb'}
data u'some'
comment comment
data u'text'
end element
close
>>> result
'closed!'
>>> class EchoTarget(object):
	def start(self, tag, attrib):
		print attrib
		print("start %s %r" % (tag, dict(attrib)))
	def end(self, tag):
		print("end %s" % tag)
	def data(self, data):
		print("data %r" % data)
	def comment(self, text):
		print("comment %s" % text)
	def close(self):
		print("close")
		return "closed!"

	
>>> result = etree.XML("<element a='aaa' b='bbb' >some<!--comment-->text</element>",parser)
start element {'a': u'aaa', 'b': u'bbb'}
{'a': u'aaa', 'b': u'bbb'}
data u'some'
comment comment
data u'text'
end element
close
>>> class EchoTarget(object):
	def start(self, tag, attrib):
		print attrib
		print("start %s %r" % (tag, dict(attrib)))
	def end(self, tag):
		print("end %s" % tag)
	def data(self, data):
		print("data %r" % data)
	def comment(self, text):
		print("comment %s" % text)
	def close(self):
		print("close")
		return "closed!"

	
>>> parser = etree.XMLParser(target = EchoTarget())
>>> 
>>> result = etree.XML("<element a='aaa' b='bbb' >some<!--comment-->text</element>",parser)
{'a': u'aaa', 'b': u'bbb'}
start element {'a': u'aaa', 'b': u'bbb'}
data u'some'
comment comment
data u'text'
end element
close
>>>

100

Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32

Type "copyright", "credits" or "license()" for more information.

>>> import lxml

Traceback (most recent call last):

File "<pyshell#0>", line 1, in <module>

import lxml

ImportError: No module named lxml

>>> import lxml

>>> class EchoTarget(object):

def start(self, tag, attrib):

print("start %s %r" % (tag, dict(attrib)))

print attrib

def end(self, tag):

print("end %s" % tag)

def data(self, data):

print("data %r" % data)

def comment(self, text):

print("comment %s" % text)

def close(self):

print("close")

return "closed!"

>>> from lxml import etree

>>> from io import StringIO, BytesIO

>>> parser = etree.XMLParser(target = EchoTarget())

>>> result = etree.XML("<element a='aaa' b='bbb' c=333 >sometext</element>",parser)

Traceback (most recent call last):

File "<pyshell#18>", line 1, in <module>

result = etree.XML("<element a='aaa' b='bbb' c=333 >sometext</element>",parser)

File "lxml.etree.pyx", line 2723, in lxml.etree.XML (src/lxml/lxml.etree.c:52448)

File "parser.pxi", line 1573, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:79932)

File "parser.pxi", line 1452, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:78774)

File "parser.pxi", line 960, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:75389)

File "parsertarget.pxi", line 149, in lxml.etree._TargetParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:86190)

File "parser.pxi", line 585, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:71955)

XMLSyntaxError: AttValue: " or ' expected, line 1, column 15

>>> result = etree.XML("<element a='aaa' b='bbb' >sometext</element>",parser)

start element {'a': u'aaa', 'b': u'bbb'}

{'a': u'aaa', 'b': u'bbb'}

data u'some'

comment comment

data u'text'

end element

>>> result

'closed!'

>>> class EchoTarget(object):

def start(self, tag, attrib):

print attrib

print("start %s %r" % (tag, dict(attrib)))

def end(self, tag):

print("end %s" % tag)

def data(self, data):

print("data %r" % data)

def comment(self, text):

print("comment %s" % text)

def close(self):

print("close")

return "closed!"

>>> result = etree.XML("<element a='aaa' b='bbb' >sometext</element>",parser)

start element {'a': u'aaa', 'b': u'bbb'}

{'a': u'aaa', 'b': u'bbb'}

data u'some'

comment comment

data u'text'

end element

>>> class EchoTarget(object):

def start(self, tag, attrib):

print attrib

print("start %s %r" % (tag, dict(attrib)))

def end(self, tag):

print("end %s" % tag)

def data(self, data):

print("data %r" % data)

def comment(self, text):

print("comment %s" % text)

def close(self):

print("close")

return "closed!"

>>> parser = etree.XMLParser(target = EchoTarget())

>>>

>>> result = etree.XML("<element a='aaa' b='bbb' >sometext</element>",parser)

{'a': u'aaa', 'b': u'bbb'}

start element {'a': u'aaa', 'b': u'bbb'}

data u'some'

comment comment

data u'text'

end element

>>>

使用目标解析器方法
目标解析器方法对于熟悉 SAX 事件驱动代码的开发人员来说应该不陌生。目标解析器是可以实现以下方法的类：
start 在元素打开时触发。数据和元素的子元素仍不可用。
end 在元素关闭时触发。所有元素的子节点，包括文本节点，现在都是可用的。
data 触发文本子节点并访问该文本。
close 在解析完成后触发。
清单 2 演示了如何创建实现所需方法的目标解析器类（这里称为 TitleTarget）。这个解析器在一个内部列表（self.text）中收集 Title 元素的文本节点，并在到达 close() 方法后返回列表。
清单 2. 一个目标解析器，它返回 Title 标记的所有文本子节点的列表
class TitleTarget(object):
def __init__(self):
self.text = []
def start(self, tag, attrib):
self.is_title = True if tag == ‘Title’ else False
def end(self, tag):
pass
def data(self, data):
if self.is_title:
self.text.append(data.encode(‘utf-8’))
def close(self):
return self.text
parser = etree.XMLParser(target = TitleTarget(),recover=True)
# This and most other samples read in the Google copyright data
infile = ‘copyright.xml’
results = etree.parse(infile, parser)
# When iterated over, ‘results’ will contain the output from
# target parser’s close() method
out = open(‘titles.txt’, ‘w’)
out.write(‘\n’.join(results))
out.close()
在运行版权数据时，代码运行时间为 54 秒。目标解析可以实现合理的速度并且不会生成消耗内存的解析树，但是在数据中为所有元素触发事件。对于特别大型的文档，如果只对其中一些元素感兴趣，那么这种方法并不理想，就像在这个例子中一样。能否将处理限制到选择的标记并获得较好的性能呢？

In [7]: f = StringIO.StringIO(r”””
…: ①
…: Mark
…: http://diveintomark.org/
…:
…:Dive into history, 2009 edition ②
…:
…: tag:diveintomark.org,2009-03-27:/archives/20090327172042 ④
…: 2009-03-27T21:56:07Z ⑤
…: 2009-03-27T17:20:42Z …: ⑥
…:
…:
…:

Putting an entire chapter on one page sounds ⑦
…: bloated, but consider this — my longest chapter so far
…: would be 75 printed pages, and it loads in under 5 seconds…
…: On dialup.

…: ⑧
…: “””)

http://lxml.de/parsing.html#parsers

The target parser interface
As in ElementTree, and similar to a SAX event handler, you can pass a target object to the parser:
>>> class EchoTarget(object):
… def start(self, tag, attrib):
… print(“start %s %r” % (tag, dict(attrib)))
… def end(self, tag):
… print(“end %s” % tag)
… def data(self, data):
… print(“data %r” % data)
… def comment(self, text):
… print(“comment %s” % text)
… def close(self):
… print(“close”)
… return “closed!”
>>> parser = etree.XMLParser(target = EchoTarget())
>>> result = etree.XML(“sometext“,
… parser)
start element {}
data u’some’
comment comment
data u’text’
end element
close
>>> print(result)
closed!
It is important for the .close() method to reset the parser target to a usable state, so that you can reuse the parser as often as you like:
>>> result = etree.XML(“sometext“,
… parser)
start element {}
data u’some’
comment comment
data u’text’
end element
close
>>> print(result)
closed!
Starting with lxml 2.3, the .close() method will also be called in the error case. This diverges(分歧) from the behaviour of ElementTree, but allows target objects to clean up their state in all situations, so that the parser can reuse them afterwards.
>>> class CollectorTarget(object):
… def __init__(self):
… self.events = []
… def start(self, tag, attrib):
… self.events.append(“start %s %r” % (tag, dict(attrib)))
… def end(self, tag):
… self.events.append(“end %s” % tag)
… def data(self, data):
… self.events.append(“data %r” % data)
… def comment(self, text):
… self.events.append(“comment %s” % text)
… def close(self):
… self.events.append(“close”)
… return “closed!”
>>> parser = etree.XMLParser(target = CollectorTarget())
>>> result = etree.XML(“some“,
… parser) # doctest: +ELLIPSIS
Traceback (most recent call last):
…
lxml.etree.XMLSyntaxError: Opening and ending tag mismatch…
>>> for event in parser.target.events:
… print(event)
start element {}
data u’some’
close
Note that the parser does not build a tree when using a parser target. The result of the parser run is whatever the target object returns from its .close() method. If you want to return an XML tree here, you have to create it programmatically in the target object. An example for a parser target that builds a tree is the TreeBuilder:
>>> parser = etree.XMLParser(target = etree.TreeBuilder())
>>> result = etree.XML(“sometext“,
… parser)
>>> print(result.tag)
element
>>> print(result[0].text)
comment

http://lxml.de/tutorial.html
http://www.ibm.com/developerworks/cn/xml/x-hiperfparse/index.html
http://sebug.net/paper/books/dive-into-python3/xml.html
http://alvinli1991.github.io/python/2013/11/12/python-xml-parser—elementtree/
http://pycoders-weekly-chinese.readthedocs.org/en/latest/issue6/processing-xml-in-python-with-element-tree.html

Eclipse + PyDev + 中文语言包

发表评论

1.到 http://www.oracle.com/technetwork/java/javase/downloads/index.html 下载 java jre 并安装，对了 32位的java只能运行32位的Eclipse ，64位的也只能运行64位的。

2.到 http://www.eclipse.org/downloads/ 下载 Eclipse Standard ，解压并执行 Eclipse.exe

3.安装中文包，到http://www.eclipse.org/babel/downloads.php 找到当前版本的中文包地址，例如 http://download.eclipse.org/technology/babel/update-site/R0.11.0/kepler 然后更新中文包。

3.pydev 的源的地址是 http://pydev.org/updates 加到eclipse里面直接安装就可以了

debian 下安装 nginx + uWSGI

发表评论

编译安装 nginx
sudo aptitude install gcc g++ libpcre3-dev libssl-dev

wget http://nginx.org/download/nginx-0.8.54.tar.gz
tar xzvf nginx-0.8.54.tar.gz
cd nginx-0.8.54/
./configure \
–prefix=/home/gamexg/web \
–sbin-path=/home/gamexg/web/sbin/nginx \
–conf-path=/home/gamexg/web/etc/nginx/nginx.conf \
–error-log-path=/home/gamexg/web/var/log/nginx/error.log \
–pid-path=/home/gamexg/web/var/run/nginx/nginx.pid \
–lock-path=/home/gamexg/web/var/lock/nginx.lock \
–user=nginx \
–group=nginx \
–with-http_gzip_static_module \
–http-log-path=/home/gamexg/web/var/log/nginx/access.log \
–http-client-body-temp-path=/home/gamexg/web/var/tmp/nginx/client/ \
–http-proxy-temp-path=/home/gamexg/web/var/tmp/nginx/proxy/ \
–http-fastcgi-temp-path=/home/gamexg/web/var/tmp/nginx/fcgi/

make && make install

配置nginx
增加一段记录
location /site1 {
uwsgi_pass 127.0.0.1:9000;
include uwsgi_params;
}

编译安装 uWSGI

sudo aptitude install python-dev libxml2-dev
wget http://projects.unbit.it/downloads/uwsgi-0.9.6.8.tar.gz
tar zxfv uwsgi-0.9.6.8.tar.gz
cd uwsgi-0.9.6.8/
make

配置 django 使用 uWSGI
gamexg@vps1:~/web$ mkdir site
gamexg@vps1:~/web$ cd site
gamexg@vps1:~/web/site$ mkdir site1
gamexg@vps1:~/web/site$ cd site1/
gamexg@vps1:~/web/site/site1$ mkdir www
gamexg@vps1:~/web/site/site1$ mkdir www/static
gamexg@vps1:~/web/site/site1$ django-admin startproject site1
gamexg@vps1:~/web/site/site1$ cd site1/
gamexg@vps1:~/web/site/site1/site1$ ./manage.py startapp app1
gamexg@vps1:~/web/site/site1/site1$ vi myapp.py
gamexg@vps1:~/web/site/site1/site1$ cat myapp.py
import os
os.environ[‘DJANGO_SETTINGS_MODULE’] = ‘test1.settings’
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

vi uwsgi.xml

127.0.0.1:9000
true /home/gamexg/web/var/uwsgi.pid 3 /home/gamexg/web/site/site1/site1 /home/gamexg/web/site/site1 myapp

启动
/home/gamexg/web/uwsgi-0.9.6.8/uwsgi -x /home/gamexg/web/site/site1/uwsgi.xml

UliPad 提示 There are some errors as importing mimxins, Please see the error.txt. 错误的解决办法。

发表评论

在winodws xp下安装wxpython后执行UliPad会提示下面的错误，这是因为wxpython新版本不再包含 comtypes ,自己下载安装即可。地址是http://sourceforge.net/projects/comtypes/

D:\我的文档\My Dropbox\ulipad.4.0\ulipad>UliPad.py
begin… 22:07:24
There are some errors as importing mimxins, Please see the error.txt.

wxpython下载地址：http://www.wxpython.org/download.php
UliPad下载地址：http://code.google.com/p/ulipad/downloads/list