Should it address the potential side-effects of “undo-ocracy” and “redo-ocracy”?

-Evgeny.

(1) You can only pickle generators that generate the same sequence every time they are restarted.

I don’t know how you can persist state if you do not make this assumption.

(2) All the work the generator did prior to pickling must be performed again on unpickling.

Something faster would be to use file.tell() to get the state and file.seek() to set the state. Since the “unpickling” is not a bottleneck, I didn’t optimize this.

http://www.disinterest.org/resource/stackless/2.6.4-docs-html/library/stackless/pickling.html

http://www.dataspora.com/blog/xml-and-big-data/

Let me answer your last question first:

> 4. Why aren’t you using postgres ?

I was getting data from someone that uses MySQL.
Knowing what I know now, I believe should should have advised him to use the –xml flag.

> 1. I am friends with awk & pals, and stripping out INSERT .. VALUE from
> mysql dumps that I get from people is no biggy

I am friends with perl, and you cannot simply split using /,/ to get your fields. The comma might be right in the middle of a string.

With XML, though, it is simple to grep for <row>, because you know 100% that < will only be in the markup.

> 2. I’m pretty sure MySQL supports dumping table data as CSV (SELECT …
> OUTHOUSE ‘/tmp/file.csv’ …)

Same problems as above.

> 3. For big data, XML is just silly big.

Why? It gzips easily.
Not being able to load it all into memory is less of an issue if it is easy to split the data using regular expressions.

I'm the author of jsonlib, and I registered specifically to post this message. Please, please, please do not use cjson!

First, it is unmaintained. The latest version available was posted on August 24, 2007. When you encounter one of its myriad bugs, you'll either have to patch it yourself or pick another JSON library. Just skip the intermediate step and use another library to begin with.

Second, it is buggy. In some cases, parsing text it just generated will return a different value from what you passed in! It's almost entirely ignorant of Unicode, and what little it tries to parse it gets wrong.

Third, it's exceedingly non-compliant. The text it parses and generates bears only a passing resemblance to JSON. There are varying degrees of conformance to the spec between libraries, based on personal preference of the authors — I prefer strict conformance, others less strict — but cjson is so different as to be simply unusable.

Yes, it's fast. I know. I wrote jsonlib partly because I was unsatisfied with simplejson's performance, and one goal (never truly achieved) was always to surpass cjson. However, speed isn't everything. As the saying goes, “if I want my math performed fast and wrong I'll ask my cat”.

In my opinion, the only Python JSON libraries worth considering are:

* simplejson — it's in the standard library, and should therefore be considered first and most thoroughly.

* jsonlib — it's fast, well-tested, and standards-compliant.

* demjson — has several options for reliable parsing of invalid input.

Last time I checked, jsonlib and simplejson's C extensions are neck-and-neck performance-wise. In some quick, unscientific tests, jsonlib reads faster and simplejson writes faster. However, simplejson's extensions are only used for certain subsets of input — if you want to use an uncommon feature, performance will degrade. jsonlib has an implementation in pure C, which avoids this problem at the cost of complexity.

Apologies for the brain-dump, but even if you skip right over it, please remember: don't use cjson.

Compactness is very important for transferring data over a network.
However, during the development cycle, human readability is important and often overlooked. If all you need to do to read your data is type 'zcat', you are much more likely to be looking at your data, and hence more likely to catch bugs.