add sqlitedict
This commit is contained in:
parent
7ec3e8b26f
commit
6ab699c9c8
9 changed files with 779 additions and 0 deletions
|
|
@ -0,0 +1,135 @@
|
||||||
|
=================================================================
|
||||||
|
sqlitedict -- persistent ``dict``, backed-up by SQLite and pickle
|
||||||
|
=================================================================
|
||||||
|
|
||||||
|
|Travis|_
|
||||||
|
|Downloads|_
|
||||||
|
|License|_
|
||||||
|
|
||||||
|
.. |Travis| image:: https://img.shields.io/travis/piskvorky/sqlitedict.svg
|
||||||
|
.. |Downloads| image:: https://img.shields.io/pypi/dm/sqlitedict.svg
|
||||||
|
.. |License| image:: https://img.shields.io/pypi/l/sqlitedict.svg
|
||||||
|
.. _Travis: https://travis-ci.org/piskvorky/sqlitedict
|
||||||
|
.. _Downloads: https://pypi.python.org/pypi/sqlitedict
|
||||||
|
.. _License: https://pypi.python.org/pypi/sqlitedict
|
||||||
|
|
||||||
|
A lightweight wrapper around Python's sqlite3 database with a simple, Pythonic
|
||||||
|
dict-like interface and support for multi-thread access:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> from sqlitedict import SqliteDict
|
||||||
|
>>> mydict = SqliteDict('./my_db.sqlite', autocommit=True)
|
||||||
|
>>> mydict['some_key'] = any_picklable_object
|
||||||
|
>>> print mydict['some_key'] # prints the new value
|
||||||
|
>>> for key, value in mydict.iteritems():
|
||||||
|
>>> print key, value
|
||||||
|
>>> print len(mydict) # etc... all dict functions work
|
||||||
|
>>> mydict.close()
|
||||||
|
|
||||||
|
Pickle is used internally to (de)serialize the values. Keys are arbitrary strings,
|
||||||
|
values arbitrary pickle-able objects.
|
||||||
|
|
||||||
|
If you don't use autocommit (default is no autocommit for performance), then
|
||||||
|
don't forget to call ``mydict.commit()`` when done with a transaction:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> # using SqliteDict as context manager works too (RECOMMENDED)
|
||||||
|
>>> with SqliteDict('./my_db.sqlite') as mydict: # note no autocommit=True
|
||||||
|
... mydict['some_key'] = u"first value"
|
||||||
|
... mydict['another_key'] = range(10)
|
||||||
|
... mydict.commit()
|
||||||
|
... mydict['some_key'] = u"new value"
|
||||||
|
... # no explicit commit here
|
||||||
|
>>> with SqliteDict('./my_db.sqlite') as mydict: # re-open the same DB
|
||||||
|
... print mydict['some_key'] # outputs 'first value', not 'new value'
|
||||||
|
|
||||||
|
|
||||||
|
Features
|
||||||
|
--------
|
||||||
|
|
||||||
|
* Values can be **any picklable objects** (uses ``cPickle`` with the highest protocol).
|
||||||
|
* Support for **multiple tables** (=dicts) living in the same database file.
|
||||||
|
* Support for **access from multiple threads** to the same connection (needed by e.g. Pyro).
|
||||||
|
Vanilla sqlite3 gives you ``ProgrammingError: SQLite objects created in a thread can
|
||||||
|
only be used in that same thread.``
|
||||||
|
|
||||||
|
Concurrent requests are still serialized internally, so this "multithreaded support"
|
||||||
|
**doesn't** give you any performance benefits. It is a work-around for sqlite limitations in Python.
|
||||||
|
|
||||||
|
Installation
|
||||||
|
------------
|
||||||
|
|
||||||
|
The module has no dependencies beyond Python itself. The minimum Python version is 2.5, continuously tested on Python 2.6, 2.7, 3.3 and 3.4 `on Travis <https://travis-ci.org/piskvorky/sqlitedict>`_.
|
||||||
|
|
||||||
|
Install or upgrade with::
|
||||||
|
|
||||||
|
easy_install -U sqlitedict
|
||||||
|
|
||||||
|
or from the `source tar.gz <http://pypi.python.org/pypi/sqlitedict>`_::
|
||||||
|
|
||||||
|
python setup.py install
|
||||||
|
|
||||||
|
Documentation
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Standard Python document strings are inside the module:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> import sqlitedict
|
||||||
|
>>> help(sqlitedict)
|
||||||
|
|
||||||
|
(but it's just ``dict`` with a commit, really).
|
||||||
|
|
||||||
|
**Beware**: because of Python semantics, ``sqlitedict`` cannot know when a mutable
|
||||||
|
SqliteDict-backed entry was modified in RAM. For example, ``mydict.setdefault('new_key', []).append(1)``
|
||||||
|
will leave ``mydict['new_key']`` equal to empty list, not ``[1]``. You'll need to
|
||||||
|
explicitly assign the mutated object back to SqliteDict to achieve the same effect:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> val = mydict.get('new_key', [])
|
||||||
|
>>> val.append(1) # sqlite DB not updated here!
|
||||||
|
>>> mydict['new_key'] = val # now updated
|
||||||
|
|
||||||
|
|
||||||
|
For developers
|
||||||
|
--------------
|
||||||
|
|
||||||
|
Install::
|
||||||
|
|
||||||
|
# pip install nose
|
||||||
|
# pip install coverage
|
||||||
|
|
||||||
|
To perform all tests::
|
||||||
|
|
||||||
|
# make test-all
|
||||||
|
|
||||||
|
To perform all tests with coverage::
|
||||||
|
|
||||||
|
# make test-all-with-coverage
|
||||||
|
|
||||||
|
|
||||||
|
Comments, bug reports
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
``sqlitedict`` resides on `github <https://github.com/piskvorky/sqlitedict>`_. You can file
|
||||||
|
issues or pull requests there.
|
||||||
|
|
||||||
|
History
|
||||||
|
-------
|
||||||
|
|
||||||
|
**1.4.0**: fix regression where iterating over keys/values/items returned a full list instead of iterator
|
||||||
|
|
||||||
|
**1.3.0**: improve error handling in multithreading (`PR #28 <https://github.com/piskvorky/sqlitedict/pull/28>`_); 100% test coverage.
|
||||||
|
|
||||||
|
**1.2.0**: full python 3 support, continuous testing via `Travis CI <https://travis-ci.org/piskvorky/sqlitedict>`_.
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
|
``sqlitedict`` is open source software released under the `Apache 2.0 license <http://opensource.org/licenses/apache2.0.php>`_.
|
||||||
|
Copyright (c) 2011-now `Radim Řehůřek <http://radimrehurek.com>`_ and contributors.
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
pip
|
||||||
|
|
@ -0,0 +1,158 @@
|
||||||
|
Metadata-Version: 2.0
|
||||||
|
Name: sqlitedict
|
||||||
|
Version: 1.4.0
|
||||||
|
Summary: Persistent dict in Python, backed up by sqlite3 and pickle, multithread-safe.
|
||||||
|
Home-page: https://github.com/piskvorky/sqlitedict
|
||||||
|
Author: Radim Rehurek
|
||||||
|
Author-email: me@radimrehurek.com
|
||||||
|
License: Apache 2.0
|
||||||
|
Download-URL: http://pypi.python.org/pypi/sqlitedict
|
||||||
|
Keywords: sqlite,persistent dict,multithreaded
|
||||||
|
Platform: any
|
||||||
|
Classifier: Development Status :: 5 - Production/Stable
|
||||||
|
Classifier: Environment :: Console
|
||||||
|
Classifier: Intended Audience :: Developers
|
||||||
|
Classifier: License :: OSI Approved :: Apache Software License
|
||||||
|
Classifier: Operating System :: OS Independent
|
||||||
|
Classifier: Programming Language :: Python :: 2.5
|
||||||
|
Classifier: Programming Language :: Python :: 2.6
|
||||||
|
Classifier: Programming Language :: Python :: 2.7
|
||||||
|
Classifier: Programming Language :: Python :: 3.3
|
||||||
|
Classifier: Programming Language :: Python :: 3.4
|
||||||
|
Classifier: Topic :: Database :: Front-Ends
|
||||||
|
|
||||||
|
=================================================================
|
||||||
|
sqlitedict -- persistent ``dict``, backed-up by SQLite and pickle
|
||||||
|
=================================================================
|
||||||
|
|
||||||
|
|Travis|_
|
||||||
|
|Downloads|_
|
||||||
|
|License|_
|
||||||
|
|
||||||
|
.. |Travis| image:: https://img.shields.io/travis/piskvorky/sqlitedict.svg
|
||||||
|
.. |Downloads| image:: https://img.shields.io/pypi/dm/sqlitedict.svg
|
||||||
|
.. |License| image:: https://img.shields.io/pypi/l/sqlitedict.svg
|
||||||
|
.. _Travis: https://travis-ci.org/piskvorky/sqlitedict
|
||||||
|
.. _Downloads: https://pypi.python.org/pypi/sqlitedict
|
||||||
|
.. _License: https://pypi.python.org/pypi/sqlitedict
|
||||||
|
|
||||||
|
A lightweight wrapper around Python's sqlite3 database with a simple, Pythonic
|
||||||
|
dict-like interface and support for multi-thread access:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> from sqlitedict import SqliteDict
|
||||||
|
>>> mydict = SqliteDict('./my_db.sqlite', autocommit=True)
|
||||||
|
>>> mydict['some_key'] = any_picklable_object
|
||||||
|
>>> print mydict['some_key'] # prints the new value
|
||||||
|
>>> for key, value in mydict.iteritems():
|
||||||
|
>>> print key, value
|
||||||
|
>>> print len(mydict) # etc... all dict functions work
|
||||||
|
>>> mydict.close()
|
||||||
|
|
||||||
|
Pickle is used internally to (de)serialize the values. Keys are arbitrary strings,
|
||||||
|
values arbitrary pickle-able objects.
|
||||||
|
|
||||||
|
If you don't use autocommit (default is no autocommit for performance), then
|
||||||
|
don't forget to call ``mydict.commit()`` when done with a transaction:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> # using SqliteDict as context manager works too (RECOMMENDED)
|
||||||
|
>>> with SqliteDict('./my_db.sqlite') as mydict: # note no autocommit=True
|
||||||
|
... mydict['some_key'] = u"first value"
|
||||||
|
... mydict['another_key'] = range(10)
|
||||||
|
... mydict.commit()
|
||||||
|
... mydict['some_key'] = u"new value"
|
||||||
|
... # no explicit commit here
|
||||||
|
>>> with SqliteDict('./my_db.sqlite') as mydict: # re-open the same DB
|
||||||
|
... print mydict['some_key'] # outputs 'first value', not 'new value'
|
||||||
|
|
||||||
|
|
||||||
|
Features
|
||||||
|
--------
|
||||||
|
|
||||||
|
* Values can be **any picklable objects** (uses ``cPickle`` with the highest protocol).
|
||||||
|
* Support for **multiple tables** (=dicts) living in the same database file.
|
||||||
|
* Support for **access from multiple threads** to the same connection (needed by e.g. Pyro).
|
||||||
|
Vanilla sqlite3 gives you ``ProgrammingError: SQLite objects created in a thread can
|
||||||
|
only be used in that same thread.``
|
||||||
|
|
||||||
|
Concurrent requests are still serialized internally, so this "multithreaded support"
|
||||||
|
**doesn't** give you any performance benefits. It is a work-around for sqlite limitations in Python.
|
||||||
|
|
||||||
|
Installation
|
||||||
|
------------
|
||||||
|
|
||||||
|
The module has no dependencies beyond Python itself. The minimum Python version is 2.5, continuously tested on Python 2.6, 2.7, 3.3 and 3.4 `on Travis <https://travis-ci.org/piskvorky/sqlitedict>`_.
|
||||||
|
|
||||||
|
Install or upgrade with::
|
||||||
|
|
||||||
|
easy_install -U sqlitedict
|
||||||
|
|
||||||
|
or from the `source tar.gz <http://pypi.python.org/pypi/sqlitedict>`_::
|
||||||
|
|
||||||
|
python setup.py install
|
||||||
|
|
||||||
|
Documentation
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Standard Python document strings are inside the module:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> import sqlitedict
|
||||||
|
>>> help(sqlitedict)
|
||||||
|
|
||||||
|
(but it's just ``dict`` with a commit, really).
|
||||||
|
|
||||||
|
**Beware**: because of Python semantics, ``sqlitedict`` cannot know when a mutable
|
||||||
|
SqliteDict-backed entry was modified in RAM. For example, ``mydict.setdefault('new_key', []).append(1)``
|
||||||
|
will leave ``mydict['new_key']`` equal to empty list, not ``[1]``. You'll need to
|
||||||
|
explicitly assign the mutated object back to SqliteDict to achieve the same effect:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
>>> val = mydict.get('new_key', [])
|
||||||
|
>>> val.append(1) # sqlite DB not updated here!
|
||||||
|
>>> mydict['new_key'] = val # now updated
|
||||||
|
|
||||||
|
|
||||||
|
For developers
|
||||||
|
--------------
|
||||||
|
|
||||||
|
Install::
|
||||||
|
|
||||||
|
# pip install nose
|
||||||
|
# pip install coverage
|
||||||
|
|
||||||
|
To perform all tests::
|
||||||
|
|
||||||
|
# make test-all
|
||||||
|
|
||||||
|
To perform all tests with coverage::
|
||||||
|
|
||||||
|
# make test-all-with-coverage
|
||||||
|
|
||||||
|
|
||||||
|
Comments, bug reports
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
``sqlitedict`` resides on `github <https://github.com/piskvorky/sqlitedict>`_. You can file
|
||||||
|
issues or pull requests there.
|
||||||
|
|
||||||
|
History
|
||||||
|
-------
|
||||||
|
|
||||||
|
**1.4.0**: fix regression where iterating over keys/values/items returned a full list instead of iterator
|
||||||
|
|
||||||
|
**1.3.0**: improve error handling in multithreading (`PR #28 <https://github.com/piskvorky/sqlitedict/pull/28>`_); 100% test coverage.
|
||||||
|
|
||||||
|
**1.2.0**: full python 3 support, continuous testing via `Travis CI <https://travis-ci.org/piskvorky/sqlitedict>`_.
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
|
``sqlitedict`` is open source software released under the `Apache 2.0 license <http://opensource.org/licenses/apache2.0.php>`_.
|
||||||
|
Copyright (c) 2011-now `Radim Řehůřek <http://radimrehurek.com>`_ and contributors.
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -0,0 +1,10 @@
|
||||||
|
sqlitedict.py,sha256=9zIUg7xFsnHFXc07KklfLKKvtcF6Ph6VjHI7biEii98,17577
|
||||||
|
sqlitedict-1.4.0.dist-info/DESCRIPTION.rst,sha256=0DtdGW_JnxlMrNq3w5Z8X89LZR2GEmNUpc3bqUN-elY,4592
|
||||||
|
sqlitedict-1.4.0.dist-info/METADATA,sha256=U_yEc91oeI2LREBIsyZv9scFcKu4IF8XBxrBaRAjX_Y,5514
|
||||||
|
sqlitedict-1.4.0.dist-info/RECORD,,
|
||||||
|
sqlitedict-1.4.0.dist-info/WHEEL,sha256=AXucK5-TNISW1dQAdsyH02xCRCCE1VKjhXIOWMW_lxI,93
|
||||||
|
sqlitedict-1.4.0.dist-info/metadata.json,sha256=Gu_BazsbomIjBcE4XjoiZV6U7DUJpWklZudlIYduauI,1070
|
||||||
|
sqlitedict-1.4.0.dist-info/pbr.json,sha256=wraF_0ld56r3l9udmVdBYB-N7W8nh7Ax8-HRVqiGRFE,46
|
||||||
|
sqlitedict-1.4.0.dist-info/top_level.txt,sha256=gRsHHG_lHd0G92cPsIV8dhQS7yZfJUYW5GY_oqapYik,11
|
||||||
|
/tmp/p3k/lib/python3.5/site-packages/sqlitedict-1.4.0.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
|
||||||
|
__pycache__/sqlitedict.cpython-35.pyc,,
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
Wheel-Version: 1.0
|
||||||
|
Generator: bdist_wheel (0.29.0)
|
||||||
|
Root-Is-Purelib: true
|
||||||
|
Tag: cp35-none-any
|
||||||
|
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
{"classifiers": ["Development Status :: 5 - Production/Stable", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python :: 2.5", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Topic :: Database :: Front-Ends"], "download_url": "http://pypi.python.org/pypi/sqlitedict", "extensions": {"python.details": {"contacts": [{"email": "me@radimrehurek.com", "name": "Radim Rehurek", "role": "author"}], "document_names": {"description": "DESCRIPTION.rst"}, "project_urls": {"Home": "https://github.com/piskvorky/sqlitedict"}}}, "generator": "bdist_wheel (0.29.0)", "keywords": ["sqlite", "persistent", "dict", "multithreaded"], "license": "Apache 2.0", "metadata_version": "2.0", "name": "sqlitedict", "platform": "any", "summary": "Persistent dict in Python, backed up by sqlite3 and pickle, multithread-safe.", "version": "1.4.0"}
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
{"is_release": true, "git_version": "9351e48"}
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
sqlitedict
|
||||||
467
Shared/lib/python3.4/site-packages/sqlitedict.py
Normal file
467
Shared/lib/python3.4/site-packages/sqlitedict.py
Normal file
|
|
@ -0,0 +1,467 @@
|
||||||
|
#!/usr/bin/env python
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
#
|
||||||
|
# This code is distributed under the terms and conditions
|
||||||
|
# from the Apache License, Version 2.0
|
||||||
|
#
|
||||||
|
# http://opensource.org/licenses/apache2.0.php
|
||||||
|
#
|
||||||
|
# This code was inspired by:
|
||||||
|
# * http://code.activestate.com/recipes/576638-draft-for-an-sqlite3-based-dbm/
|
||||||
|
# * http://code.activestate.com/recipes/526618/
|
||||||
|
|
||||||
|
"""
|
||||||
|
A lightweight wrapper around Python's sqlite3 database, with a dict-like interface
|
||||||
|
and multi-thread access support::
|
||||||
|
|
||||||
|
>>> mydict = SqliteDict('some.db', autocommit=True) # the mapping will be persisted to file `some.db`
|
||||||
|
>>> mydict['some_key'] = any_picklable_object
|
||||||
|
>>> print mydict['some_key']
|
||||||
|
>>> print len(mydict) # etc... all dict functions work
|
||||||
|
|
||||||
|
Pickle is used internally to serialize the values. Keys are strings.
|
||||||
|
|
||||||
|
If you don't use autocommit (default is no autocommit for performance), then
|
||||||
|
don't forget to call `mydict.commit()` when done with a transaction.
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import random
|
||||||
|
import logging
|
||||||
|
import traceback
|
||||||
|
|
||||||
|
from threading import Thread
|
||||||
|
|
||||||
|
major_version = sys.version_info[0]
|
||||||
|
if major_version < 3: # py <= 2.x
|
||||||
|
if sys.version_info[1] < 5: # py <= 2.4
|
||||||
|
raise ImportError("sqlitedict requires python 2.5 or higher (python 3.3 or higher supported)")
|
||||||
|
|
||||||
|
# necessary to use exec()_ as this would be a SyntaxError in python3.
|
||||||
|
# this is an exact port of six.reraise():
|
||||||
|
def exec_(_code_, _globs_=None, _locs_=None):
|
||||||
|
"""Execute code in a namespace."""
|
||||||
|
if _globs_ is None:
|
||||||
|
frame = sys._getframe(1)
|
||||||
|
_globs_ = frame.f_globals
|
||||||
|
if _locs_ is None:
|
||||||
|
_locs_ = frame.f_locals
|
||||||
|
del frame
|
||||||
|
elif _locs_ is None:
|
||||||
|
_locs_ = _globs_
|
||||||
|
exec("""exec _code_ in _globs_, _locs_""")
|
||||||
|
|
||||||
|
exec_("def reraise(tp, value, tb=None):\n"
|
||||||
|
" raise tp, value, tb\n")
|
||||||
|
else:
|
||||||
|
def reraise(tp, value, tb=None):
|
||||||
|
if value is None:
|
||||||
|
value = tp()
|
||||||
|
if value.__traceback__ is not tb:
|
||||||
|
raise value.with_traceback(tb)
|
||||||
|
raise value
|
||||||
|
|
||||||
|
try:
|
||||||
|
from cPickle import dumps, loads, HIGHEST_PROTOCOL as PICKLE_PROTOCOL
|
||||||
|
except ImportError:
|
||||||
|
from pickle import dumps, loads, HIGHEST_PROTOCOL as PICKLE_PROTOCOL
|
||||||
|
|
||||||
|
# some Python 3 vs 2 imports
|
||||||
|
try:
|
||||||
|
from collections import UserDict as DictClass
|
||||||
|
except ImportError:
|
||||||
|
from UserDict import DictMixin as DictClass
|
||||||
|
|
||||||
|
try:
|
||||||
|
from queue import Queue
|
||||||
|
except ImportError:
|
||||||
|
from Queue import Queue
|
||||||
|
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def open(*args, **kwargs):
|
||||||
|
"""See documentation of the SqliteDict class."""
|
||||||
|
return SqliteDict(*args, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def encode(obj):
|
||||||
|
"""Serialize an object using pickle to a binary format accepted by SQLite."""
|
||||||
|
return sqlite3.Binary(dumps(obj, protocol=PICKLE_PROTOCOL))
|
||||||
|
|
||||||
|
|
||||||
|
def decode(obj):
|
||||||
|
"""Deserialize objects retrieved from SQLite."""
|
||||||
|
return loads(bytes(obj))
|
||||||
|
|
||||||
|
|
||||||
|
class SqliteDict(DictClass):
|
||||||
|
def __init__(self, filename=None, tablename='unnamed', flag='c',
|
||||||
|
autocommit=False, journal_mode="DELETE"):
|
||||||
|
"""
|
||||||
|
Initialize a thread-safe sqlite-backed dictionary. The dictionary will
|
||||||
|
be a table `tablename` in database file `filename`. A single file (=database)
|
||||||
|
may contain multiple tables.
|
||||||
|
|
||||||
|
If no `filename` is given, a random file in temp will be used (and deleted
|
||||||
|
from temp once the dict is closed/deleted).
|
||||||
|
|
||||||
|
If you enable `autocommit`, changes will be committed after each operation
|
||||||
|
(more inefficient but safer). Otherwise, changes are committed on `self.commit()`,
|
||||||
|
`self.clear()` and `self.close()`.
|
||||||
|
|
||||||
|
Set `journal_mode` to 'OFF' if you're experiencing sqlite I/O problems
|
||||||
|
or if you need performance and don't care about crash-consistency.
|
||||||
|
|
||||||
|
The `flag` parameter:
|
||||||
|
'c': default mode, open for read/write, creating the db/table if necessary.
|
||||||
|
'w': open for r/w, but drop `tablename` contents first (start with empty table)
|
||||||
|
'n': create a new database (erasing any existing tables, not just `tablename`!).
|
||||||
|
|
||||||
|
"""
|
||||||
|
self.in_temp = filename is None
|
||||||
|
if self.in_temp:
|
||||||
|
randpart = hex(random.randint(0, 0xffffff))[2:]
|
||||||
|
filename = os.path.join(tempfile.gettempdir(), 'sqldict' + randpart)
|
||||||
|
if flag == 'n':
|
||||||
|
if os.path.exists(filename):
|
||||||
|
os.remove(filename)
|
||||||
|
|
||||||
|
dirname = os.path.dirname(filename)
|
||||||
|
if dirname:
|
||||||
|
if not os.path.exists(dirname):
|
||||||
|
raise RuntimeError('Error! The directory does not exist, %s' % dirname)
|
||||||
|
|
||||||
|
self.filename = filename
|
||||||
|
self.tablename = tablename
|
||||||
|
|
||||||
|
logger.info("opening Sqlite table %r in %s" % (tablename, filename))
|
||||||
|
MAKE_TABLE = 'CREATE TABLE IF NOT EXISTS %s (key TEXT PRIMARY KEY, value BLOB)' % self.tablename
|
||||||
|
self.conn = SqliteMultithread(filename, autocommit=autocommit, journal_mode=journal_mode)
|
||||||
|
self.conn.execute(MAKE_TABLE)
|
||||||
|
self.conn.commit()
|
||||||
|
if flag == 'w':
|
||||||
|
self.clear()
|
||||||
|
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *exc_info):
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
def __str__(self):
|
||||||
|
return "SqliteDict(%s)" % (self.conn.filename)
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
return str(self) # no need of something complex
|
||||||
|
|
||||||
|
def __len__(self):
|
||||||
|
# `select count (*)` is super slow in sqlite (does a linear scan!!)
|
||||||
|
# As a result, len() is very slow too once the table size grows beyond trivial.
|
||||||
|
# We could keep the total count of rows ourselves, by means of triggers,
|
||||||
|
# but that seems too complicated and would slow down normal operation
|
||||||
|
# (insert/delete etc).
|
||||||
|
GET_LEN = 'SELECT COUNT(*) FROM %s' % self.tablename
|
||||||
|
rows = self.conn.select_one(GET_LEN)[0]
|
||||||
|
return rows if rows is not None else 0
|
||||||
|
|
||||||
|
def __bool__(self):
|
||||||
|
# No elements is False, otherwise True
|
||||||
|
GET_MAX = 'SELECT MAX(ROWID) FROM %s' % self.tablename
|
||||||
|
m = self.conn.select_one(GET_MAX)[0]
|
||||||
|
# Explicit better than implicit and bla bla
|
||||||
|
return True if m is not None else False
|
||||||
|
|
||||||
|
def iterkeys(self):
|
||||||
|
GET_KEYS = 'SELECT key FROM %s ORDER BY rowid' % self.tablename
|
||||||
|
for key in self.conn.select(GET_KEYS):
|
||||||
|
yield key[0]
|
||||||
|
|
||||||
|
def itervalues(self):
|
||||||
|
GET_VALUES = 'SELECT value FROM %s ORDER BY rowid' % self.tablename
|
||||||
|
for value in self.conn.select(GET_VALUES):
|
||||||
|
yield decode(value[0])
|
||||||
|
|
||||||
|
def iteritems(self):
|
||||||
|
GET_ITEMS = 'SELECT key, value FROM %s ORDER BY rowid' % self.tablename
|
||||||
|
for key, value in self.conn.select(GET_ITEMS):
|
||||||
|
yield key, decode(value)
|
||||||
|
|
||||||
|
def keys(self):
|
||||||
|
return self.iterkeys() if major_version > 2 else list(self.iterkeys())
|
||||||
|
|
||||||
|
def values(self):
|
||||||
|
return self.itervalues() if major_version > 2 else list(self.itervalues())
|
||||||
|
|
||||||
|
def items(self):
|
||||||
|
return self.iteritems() if major_version > 2 else list(self.iteritems())
|
||||||
|
|
||||||
|
def __contains__(self, key):
|
||||||
|
HAS_ITEM = 'SELECT 1 FROM %s WHERE key = ?' % self.tablename
|
||||||
|
return self.conn.select_one(HAS_ITEM, (key,)) is not None
|
||||||
|
|
||||||
|
def __getitem__(self, key):
|
||||||
|
GET_ITEM = 'SELECT value FROM %s WHERE key = ?' % self.tablename
|
||||||
|
item = self.conn.select_one(GET_ITEM, (key,))
|
||||||
|
if item is None:
|
||||||
|
raise KeyError(key)
|
||||||
|
return decode(item[0])
|
||||||
|
|
||||||
|
def __setitem__(self, key, value):
|
||||||
|
ADD_ITEM = 'REPLACE INTO %s (key, value) VALUES (?,?)' % self.tablename
|
||||||
|
self.conn.execute(ADD_ITEM, (key, encode(value)))
|
||||||
|
|
||||||
|
def __delitem__(self, key):
|
||||||
|
if key not in self:
|
||||||
|
raise KeyError(key)
|
||||||
|
DEL_ITEM = 'DELETE FROM %s WHERE key = ?' % self.tablename
|
||||||
|
self.conn.execute(DEL_ITEM, (key,))
|
||||||
|
|
||||||
|
def update(self, items=(), **kwds):
|
||||||
|
try:
|
||||||
|
items = [(k, encode(v)) for k, v in items.items()]
|
||||||
|
except AttributeError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
UPDATE_ITEMS = 'REPLACE INTO %s (key, value) VALUES (?, ?)' % self.tablename
|
||||||
|
self.conn.executemany(UPDATE_ITEMS, items)
|
||||||
|
if kwds:
|
||||||
|
self.update(kwds)
|
||||||
|
|
||||||
|
def __iter__(self):
|
||||||
|
return self.iterkeys()
|
||||||
|
|
||||||
|
def clear(self):
|
||||||
|
CLEAR_ALL = 'DELETE FROM %s;' % self.tablename # avoid VACUUM, as it gives "OperationalError: database schema has changed"
|
||||||
|
self.conn.commit()
|
||||||
|
self.conn.execute(CLEAR_ALL)
|
||||||
|
self.conn.commit()
|
||||||
|
|
||||||
|
def commit(self, blocking=True):
|
||||||
|
"""
|
||||||
|
Persist all data to disk.
|
||||||
|
|
||||||
|
When `blocking` is False, the commit command is queued, but the data is
|
||||||
|
not guaranteed persisted (default implication when autocommit=True).
|
||||||
|
"""
|
||||||
|
if self.conn is not None:
|
||||||
|
self.conn.commit(blocking)
|
||||||
|
sync = commit
|
||||||
|
|
||||||
|
def close(self, do_log=True):
|
||||||
|
if do_log:
|
||||||
|
logger.debug("closing %s" % self)
|
||||||
|
if hasattr(self, 'conn') and self.conn is not None:
|
||||||
|
if self.conn.autocommit:
|
||||||
|
# typically calls to commit are non-blocking when autocommit is
|
||||||
|
# used. However, we need to block on close() to ensure any
|
||||||
|
# awaiting exceptions are handled and that all data is
|
||||||
|
# persisted to disk before returning.
|
||||||
|
self.conn.commit(blocking=True)
|
||||||
|
self.conn.close()
|
||||||
|
self.conn = None
|
||||||
|
if self.in_temp:
|
||||||
|
try:
|
||||||
|
os.remove(self.filename)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def terminate(self):
|
||||||
|
"""Delete the underlying database file. Use with care."""
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
if self.filename == ':memory:':
|
||||||
|
return
|
||||||
|
|
||||||
|
logger.info("deleting %s" % self.filename)
|
||||||
|
try:
|
||||||
|
os.remove(self.filename)
|
||||||
|
except (OSError, IOError):
|
||||||
|
logger.exception("failed to delete %s" % (self.filename))
|
||||||
|
|
||||||
|
def __del__(self):
|
||||||
|
# like close(), but assume globals are gone by now (do not log!)
|
||||||
|
self.close(do_log=False)
|
||||||
|
|
||||||
|
# Adding extra methods for python 2 compatibility (at import time)
|
||||||
|
if major_version == 2:
|
||||||
|
SqliteDict.__nonzero__ = SqliteDict.__bool__
|
||||||
|
del SqliteDict.__bool__ # not needed and confusing
|
||||||
|
#endclass SqliteDict
|
||||||
|
|
||||||
|
|
||||||
|
class SqliteMultithread(Thread):
|
||||||
|
"""
|
||||||
|
Wrap sqlite connection in a way that allows concurrent requests from multiple threads.
|
||||||
|
|
||||||
|
This is done by internally queueing the requests and processing them sequentially
|
||||||
|
in a separate thread (in the same order they arrived).
|
||||||
|
|
||||||
|
"""
|
||||||
|
def __init__(self, filename, autocommit, journal_mode):
|
||||||
|
super(SqliteMultithread, self).__init__()
|
||||||
|
self.filename = filename
|
||||||
|
self.autocommit = autocommit
|
||||||
|
self.journal_mode = journal_mode
|
||||||
|
# use request queue of unlimited size
|
||||||
|
self.reqs = Queue()
|
||||||
|
self.setDaemon(True) # python2.5-compatible
|
||||||
|
self.exception = None
|
||||||
|
self.log = logging.getLogger('sqlitedict.SqliteMultithread')
|
||||||
|
self.start()
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
if self.autocommit:
|
||||||
|
conn = sqlite3.connect(self.filename, isolation_level=None, check_same_thread=False)
|
||||||
|
else:
|
||||||
|
conn = sqlite3.connect(self.filename, check_same_thread=False)
|
||||||
|
conn.execute('PRAGMA journal_mode = %s' % self.journal_mode)
|
||||||
|
conn.text_factory = str
|
||||||
|
cursor = conn.cursor()
|
||||||
|
cursor.execute('PRAGMA synchronous=OFF')
|
||||||
|
|
||||||
|
res = None
|
||||||
|
while True:
|
||||||
|
req, arg, res, outer_stack = self.reqs.get()
|
||||||
|
if req == '--close--':
|
||||||
|
assert res, ('--close-- without return queue', res)
|
||||||
|
break
|
||||||
|
elif req == '--commit--':
|
||||||
|
conn.commit()
|
||||||
|
if res:
|
||||||
|
res.put('--no more--')
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
cursor.execute(req, arg)
|
||||||
|
except Exception as err:
|
||||||
|
self.exception = (e_type, e_value, e_tb) = sys.exc_info()
|
||||||
|
inner_stack = traceback.extract_stack()
|
||||||
|
|
||||||
|
# An exception occurred in our thread, but we may not
|
||||||
|
# immediately able to throw it in our calling thread, if it has
|
||||||
|
# no return `res` queue: log as level ERROR both the inner and
|
||||||
|
# outer exception immediately.
|
||||||
|
#
|
||||||
|
# Any iteration of res.get() or any next call will detect the
|
||||||
|
# inner exception and re-raise it in the calling Thread; though
|
||||||
|
# it may be confusing to see an exception for an unrelated
|
||||||
|
# statement, an ERROR log statement from the 'sqlitedict.*'
|
||||||
|
# namespace contains the original outer stack location.
|
||||||
|
self.log.error('Inner exception:')
|
||||||
|
for item in traceback.format_list(inner_stack):
|
||||||
|
self.log.error(item)
|
||||||
|
self.log.error('') # deliniate traceback & exception w/blank line
|
||||||
|
for item in traceback.format_exception_only(e_type, e_value):
|
||||||
|
self.log.error(item)
|
||||||
|
|
||||||
|
self.log.error('') # exception & outer stack w/blank line
|
||||||
|
self.log.error('Outer stack:')
|
||||||
|
for item in traceback.format_list(outer_stack):
|
||||||
|
self.log.error(item)
|
||||||
|
self.log.error('Exception will be re-raised at next call.')
|
||||||
|
|
||||||
|
if res:
|
||||||
|
for rec in cursor:
|
||||||
|
res.put(rec)
|
||||||
|
res.put('--no more--')
|
||||||
|
|
||||||
|
if self.autocommit:
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
self.log.debug('received: %s, send: --no more--', req)
|
||||||
|
conn.close()
|
||||||
|
res.put('--no more--')
|
||||||
|
|
||||||
|
def check_raise_error(self):
|
||||||
|
"""
|
||||||
|
Check for and raise exception for any previous sqlite query.
|
||||||
|
|
||||||
|
For the `execute*` family of method calls, such calls are non-blocking and any
|
||||||
|
exception raised in the thread cannot be handled by the calling Thread (usually
|
||||||
|
MainThread). This method is called on `close`, and prior to any subsequent
|
||||||
|
calls to the `execute*` methods to check for and raise an exception in a
|
||||||
|
previous call to the MainThread.
|
||||||
|
"""
|
||||||
|
if self.exception:
|
||||||
|
e_type, e_value, e_tb = self.exception
|
||||||
|
|
||||||
|
# clear self.exception, if the caller decides to handle such
|
||||||
|
# exception, we should not repeatedly re-raise it.
|
||||||
|
self.exception = None
|
||||||
|
|
||||||
|
self.log.error('An exception occurred from a previous statement, view '
|
||||||
|
'the logging namespace "sqlitedict" for outer stack.')
|
||||||
|
|
||||||
|
# The third argument to raise is the traceback object, and it is
|
||||||
|
# substituted instead of the current location as the place where
|
||||||
|
# the exception occurred, this is so that when using debuggers such
|
||||||
|
# as `pdb', or simply evaluating the naturally raised traceback, we
|
||||||
|
# retain the original (inner) location of where the exception
|
||||||
|
# occurred.
|
||||||
|
reraise(e_type, e_value, e_tb)
|
||||||
|
|
||||||
|
def execute(self, req, arg=None, res=None):
|
||||||
|
"""
|
||||||
|
`execute` calls are non-blocking: just queue up the request and return immediately.
|
||||||
|
"""
|
||||||
|
self.check_raise_error()
|
||||||
|
|
||||||
|
# NOTE: This might be a lot of information to pump into an input
|
||||||
|
# queue, affecting performance. I've also seen earlier versions of
|
||||||
|
# jython take a severe performance impact for throwing exceptions
|
||||||
|
# so often.
|
||||||
|
stack = traceback.extract_stack()[:-1]
|
||||||
|
self.reqs.put((req, arg or tuple(), res, stack))
|
||||||
|
|
||||||
|
def executemany(self, req, items):
|
||||||
|
for item in items:
|
||||||
|
self.execute(req, item)
|
||||||
|
self.check_raise_error()
|
||||||
|
|
||||||
|
def select(self, req, arg=None):
|
||||||
|
"""
|
||||||
|
Unlike sqlite's native select, this select doesn't handle iteration efficiently.
|
||||||
|
|
||||||
|
The result of `select` starts filling up with values as soon as the
|
||||||
|
request is dequeued, and although you can iterate over the result normally
|
||||||
|
(`for res in self.select(): ...`), the entire result will be in memory.
|
||||||
|
"""
|
||||||
|
res = Queue() # results of the select will appear as items in this queue
|
||||||
|
self.execute(req, arg, res)
|
||||||
|
while True:
|
||||||
|
rec = res.get()
|
||||||
|
self.check_raise_error()
|
||||||
|
if rec == '--no more--':
|
||||||
|
break
|
||||||
|
yield rec
|
||||||
|
|
||||||
|
def select_one(self, req, arg=None):
|
||||||
|
"""Return only the first row of the SELECT, or None if there are no matching rows."""
|
||||||
|
try:
|
||||||
|
return next(iter(self.select(req, arg)))
|
||||||
|
except StopIteration:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def commit(self, blocking=True):
|
||||||
|
if blocking:
|
||||||
|
# by default, we await completion of commit() unless
|
||||||
|
# blocking=False. This ensures any available exceptions for any
|
||||||
|
# previous statement are thrown before returning, and that the
|
||||||
|
# data has actually persisted to disk!
|
||||||
|
self.select_one('--commit--')
|
||||||
|
else:
|
||||||
|
# otherwise, we fire and forget as usual.
|
||||||
|
self.execute('--commit--')
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
# we abuse 'select' to "iter" over a "--close--" statement so that we
|
||||||
|
# can confirm the completion of close before joining the thread and
|
||||||
|
# returning (by semaphore '--no more--'
|
||||||
|
self.select_one('--close--')
|
||||||
|
self.join()
|
||||||
|
#endclass SqliteMultithread
|
||||||
Loading…
Reference in a new issue