LocalFileIdentifiableStore: Fix concurrency issues

[hpoeche]

Previously the LocalFileIdentifiableStore had no access control for concurrent access through the same or multiple instances on the same directory.
This caused serious problems especially if this store is used as storage backend for a multi-worker WSGI server.

Issues with multiple instances

Multiple instances of the LocalFileIdentifiableStore were allowed to operate on the same directory. The main issue lies in the self._object_cache of each instance which does not get updated or invalidated if the other instance is modifying the underlying files through a write operation. In order for the caches to be coherent with the persistent content a inter-process cache synchronization would be necessary. This would introduce to much overhead so we decided to lock the access to a directory to a single instance of LocalFileIdentifiableStore . This also helps to simplify resolving the following issues.

Issues with concurrent calls to a single instance

1. Race condition on writes
   Concurrent calls to writing methods add() and commit() currently race in the _write_atomic() method on the os.replace(...) call. Whenever two threads invoke the method with an identifiable that has the same ID but different content both calls succeed. However, the persisted content depends on which thread gets scheduled to execute the os.replace(...) last.
2. TOCTOU for writes
   This occurs multiple times across the writing functions. For example in the add() function the time of check (TOC) for existence may be different that the time of use (TOU) where the new file is created. A concurrently executed add() might be scheduled such that at TOC both threads find no file for the ID so they both create a new file at TOU. Here again the last scheduled thread silently overwrites the content from the first.

Changes

In order to fix the multi-instance issue, these changes introduce a directory lock mechanism, that only allows one instance of the class to hold a lock on a .lock file in the directory. A context manager is used to secure operations on the directory and prevent concurrent release of the lock.
Caveat: For now this only work on POSIX machines, as it relies on flock functionality which is not provided by Windows.

To fix concurrent writes from the same store instance, all write accesses to a file need to be mutual exclusive.
In this implementation I chose to use a global _writing_lock to lock write operations to all files of the store, as locking individual files has no performance benefit, due to serialization of multiple threads through Pythons GIL.

This implementation prevents the following concurrency issues:

1. Two concurrent add(): Instead of silent overwriting, first call will second fails with KeyError.
2. commit() + discard(): Instead of commit() creating file after discard() deleted it, the file is now certainly discarded. Depending on the scheduled order commit() may raise a KeyError.

Fixes #554