brokvolli.multi
Multi-threaded variants of transduce and brokvolli.single/transduce-kv.
Warning: Use stateful transducers only with extreme caution.
Rules of thumb for using multi-threaded transducing functions in this namespace:
- Collections contain more than 100,000 elements.
- Per-element task involves more than ten computations.
- Partition threshold selected to induce at least ten partitions.
transduce
(transduce xform f coll)(transduce combine xform f coll)(transduce n combine xform f coll)Like clojure.core/transduce, but potentially multi-threaded. xform is a composed transducer ‘stack’ and f is a reducing function, each analogous to their clojure.core/transduce counterparts. This variant of transduce does not accept an explicit ‘init’ parameter; it must be provided by the arity-0 of f. Returns (combine) when coll is empty or nil.
xform has three arities:
- Zero args provides the ‘init’ value of the sub-reduction.
- One arg provides the ‘completing’ action on the sub-collection.
- Two args is the step function, applying the function to the accumulating value and the next element.
Warning: Use stateful transducers only with extreme caution.
combine is a function of three arities that governs how to gather the results of the sub-reductions:
- Zero args provides the output if
collis empty ornil. - One arg is the ‘completing’ action applied just before the final value.
- Two args combines the ‘left’ and ‘right’ sub-reduction results.
combine defaults to f.
coll must implement clojure.lang.IReduceInit. n is size at which coll is partitioned for multi-threaded processing, defaulting to 512. When n does not divide evenly into (size coll), the locations of the partition boundaries are an implementation detail.
Example, f also provides combine:
(transduce (map identity) + [1 2 3]) ;; => 6
Example with explicit partitioning,
- Partition
collinto partitions with max of two elements, - Compute (increment) on different threads, and
- Combine (concatenate):
(require '[brokvolli.core :refer [concatv]])
(transduce 2 concatv (map inc) conj [11 22 33]) ;; => [12 23 34]
See concatv for a helper function that concatenates eagerly and efficiently.
Example, ‘stack of xforms’:
(transduce 3
concatv
(comp (map inc)
(filter even?)
(remove #(< 70 %)))
conj
[11 22 33 44 55 66 77 88])
;; => [12 34 56]
Like clojure.core/reduce, elements of hash-maps are peeled off as map entries, 2-tuples of key and value.
Example, working with an associative collection (i.e., a hash-map):
(transduce merge
(comp (map #(update % 1 inc))
(filter #(even? (second %)))
(remove #(< 70 (second %))))
(completing
(fn
([] {})
([result value] (conj result value))))
{:a 11 :b 22 :c 33 :d 44 :e 55 :f 66 :g 77 :h 88 :i 99})
;; => {:e 56, :c 34, :a 12}
See transduce-kv for a transduce variant that behaves like reduce-kv.
transduce-kv
(transduce-kv xform f coll)(transduce-kv combine xform f coll)(transduce-kv n combine xform f coll)Multi-threaded variant of single/transduce-kv. xform, f, and combine are the similar to transduce. xform is a transducer stack, f is a reducing function with an additional arity three which consumes the accumulated value, the key/index, and the next element of coll.
As with single/transduce-kv, when used with a transducer stack adapted with kv-ize, the key/index is available as *keydex* at any “layer” within the transducer stack; it may be ignored to suit.
coll must implement clojure.lang.IKVReduce.
n is size at which coll is partitioned for multi-threaded processing, defaulting to 512. When n does not divide evenly into (size coll), the locations of the partition boundaries are an implementation detail.
Warning: Use stateful transducers only with extreme caution.
Examples:
(require '[brokvolli.core :refer [concatv tconj]]
'[brokvolli.transducers-kv :refer [map-kv]])
(transduce-kv (map-kv (fn [keydex x] (+ x (inc keydex)))) tconj [11 22 33])
;; => [12 24 36]
;; same result, but with explicit splitting into threads (also, must use a dedicated combining function, e.g., `concatv`)
(transduce-kv 2 concatv (map-kv (fn [keydex x] (+ x (inc keydex)))) tconj [11 22 33])
;; => [12 24 36]
;; value keydex (+ value (inc keydex)) eval result
;; 11 0 (+ 11 (inc 0)) 12 [12]
;; 22 1 (+ 22 (inc 1)) 24 [12 24]
;; 33 2 (+ 33 (inc 2)) 36 [12 24 36]
See also single/transduce-kv and multi/transduce.