Unicode - Revision history

Remi at 21:05, 19 September 2014

2014-09-19T21:05:31Z

← Older revision		Revision as of 21:05, 19 September 2014
Line 19:		Line 19:
	== Guidelines ==		== Guidelines ==

	Please familiarize yourself with unicode, [http://www.joelonsoftware.com/articles/Unicode.html encodings], and do not put casts in the code unless they are necessary. Be aware that some Python functions only accept str (like tarfile, zipfile, ...) so some compatibility might be used around these modules.		* Please familiarize yourself with unicode, [http://www.joelonsoftware.com/articles/Unicode.html encodings], and do not put casts in the code unless they are necessary. Be aware that some Python functions only accept str (like tarfile, zipfile, ...) so some compatibility might be used around these modules.
			* Stop using str() when unnecessary. Document why you use str(), unicode() or bytes() when you do.
			* Stop using type() at all.

	== Tasks ==		== Tasks ==

Remi at 16:39, 17 September 2014

2014-09-17T16:39:22Z

← Older revision		Revision as of 16:39, 17 September 2014
Line 21:		Line 21:
	Please familiarize yourself with unicode, [http://www.joelonsoftware.com/articles/Unicode.html encodings], and do not put casts in the code unless they are necessary. Be aware that some Python functions only accept str (like tarfile, zipfile, ...) so some compatibility might be used around these modules.		Please familiarize yourself with unicode, [http://www.joelonsoftware.com/articles/Unicode.html encodings], and do not put casts in the code unless they are necessary. Be aware that some Python functions only accept str (like tarfile, zipfile, ...) so some compatibility might be used around these modules.

	== ~~TODO~~ ==		== Tasks ==

	=== Mass-replacing str() casts ===		=== Mass-replacing str() casts ===
Line 37:		Line 37:

	=== Locators ===		=== Locators ===

			=== Tests ===

			We need to make sure everything works with unicode, so it's probably a good idea to insert non-ascii characters here and there in the tests (filenames, versions, parameters...)

Remi: Created page with 'This page discusses the steps and issues with getting proper unicode support in VisTrails. == Goal == VisTrails should be able to handle any kind of unicode string. This means …'

2014-09-17T16:06:35Z

Created page with 'This page discusses the steps and issues with getting proper unicode support in VisTrails. == Goal == VisTrails should be able to handle any kind of unicode string. This means …'

New page

This page discusses the steps and issues with getting proper unicode support in VisTrails.

== Goal ==

VisTrails should be able to handle any kind of unicode string. This means any string should be acceptable as input from console/API/GUI settings and should be serialized/deserialized correctly. Basically, every string in the program should be a unicode object, and although in some places we can accept a bytestring (e.g. from the API or modules) they should be casted to unicode.

Mentions of <tt>str</tt> should mainly disappear in favor of <tt>unicode</tt> and <tt>bytes</tt>.

== About Python 3 ==

Proper unicode support is a prerequisite to Python 3 support. Once VisTrails is fully unicode-safe and using unicode_literals, replacing unicode with six.text_type is very easy.

== Issues ==

The GUI has no problem with unicode, PyQt is fully unicode-safe. Since we are using API v2, PyQt already returns (and accepts) native unicode objects. It used to return QString objects which (I assume) is why we have str() casts everywhere in the code; these probably cannot be dealt with manually, which is why I propose to replace all of them with unicode() automatically (and deal with special cases afterwards).

Serialization to/deserialization from XML is not a problem since XML documents are already unicode. Properly encoding filenames in there might require some work (using proper url-encoded URLs instead is probably a good idea) -- see [[#locators|Locators]].

== Guidelines ==

Please familiarize yourself with unicode, [http://www.joelonsoftware.com/articles/Unicode.html encodings], and do not put casts in the code unless they are necessary. Be aware that some Python functions only accept str (like tarfile, zipfile, ...) so some compatibility might be used around these modules.

== TODO ==

=== Mass-replacing str() casts ===

{| class="wikitable"
|-
! From (regex)
!
! To
|-
| <tt>(?<!def )(?<![a-z_.])str\(</tt>
|        
| <tt>unicode(</tt>
|}

=== Locators ===